Data processing method, system and equipment based on micro-service assembly line and medium

Abstract

The invention provides a data processing method, system and device based on a micro-service pipeline and a medium, and the method comprises the steps: obtaining inflow data, and carrying out the slicing processing of the inflow data, and obtaining a plurality of pieces of fragmented data; obtaining a business demand, and executing an instantiation operation of the assembly line according to the business demand and a preset micro-service assembly line model to obtain a business assembly line; wherein the instantiation operation comprises the steps of distributing a corresponding assembly line, carrying out capability arrangement on the micro-service capability of the assembly line, and configuring a capability processing sequence of each micro-service capability; and concurrently distributing all the fragmented data to a business assembly line, calling the corresponding micro-service capability according to the capability processing sequence, and performing data processing on the fragmented data to obtain a data processing result. The data processing efficiency can be improved, and an efficient pipeline calculation mode of data processing is achieved to the maximum extent.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, system, device and medium based on a microservice pipeline. Background Art

[0002] With the continuous expansion of the business development scale of communication enterprises and the popularization of the Internet in all aspects of life, telecommunications services no longer only include the package handling under traditional communication services. Smart home, home audio and video entertainment, etc. are all covered within the scope of telecommunications business billing. This leads to the diversity of data sources. Although there are data preprocessing steps in the current billing logic, it only stays at the simplified data classification level of "labeling" and cannot undertake the subsequent large-scale data diversion processing. In view of the current situation of numerous data sources and large data volumes, a data diversion processing mode similar to the assembly line process is urgently needed to divert the data source and match it to the corresponding business data processing process. Summary of the invention

[0003] The embodiment of the present invention provides a data processing method based on a microservice pipeline to solve the problems existing in the related technologies. The technical solution is as follows:

[0004] In a first aspect, an embodiment of the present invention provides a data processing method based on a microservice pipeline, comprising:

[0005] Obtain the incoming data, slice the incoming data, and obtain several shard data;

[0006] Obtain business requirements, and perform pipeline instantiation operations according to the business requirements and the preset microservice pipeline model to obtain a business pipeline; wherein the instantiation operation includes allocating corresponding pipelines, orchestrating the microservice capabilities of the pipeline, and configuring the capability processing order of each microservice capability;

[0007] All shard data are concurrently diverted to the business pipeline, and the corresponding microservice capabilities are called according to the capability processing order to process the shard data and obtain the data processing results.

[0008] In one implementation, the microservice pipeline model includes a business pipeline correspondence table, a pipeline orchestration table, and a capability definition table;

[0009] The business flow correspondence table is used to define the business types corresponding to different pipelines;

[0010] The pipeline orchestration table is used to configure the microservice capabilities and the capability processing order of each microservice capability;

[0011] The capability definition table is used to define the service programs that need to be called by different microservice capabilities.

[0012] In one embodiment, the slicing method includes:

[0013] The incoming data is evenly divided according to the hash modulus algorithm to obtain sharded data.

[0014] In one embodiment, the slicing method includes:

[0015] When the shard data is sliced, the shard data is stored in the message queue, and a business flow registration table is created based on the shard data stored in the message queue;

[0016] Identify the business requirements corresponding to each shard data and obtain the business type corresponding to each shard data;

[0017] Business tags are applied to the fragmented data of different business types, and the business tags of each fragmented data are written into the business flow registration table.

[0018] In one embodiment, the method of instantiating an operation includes:

[0019] Read the business tag corresponding to the shard data based on the business flow registration table to obtain the business type of the shard data;

[0020] Associate the microservice pipeline model according to the business type, match the pipeline corresponding to the business type from the business pipeline corresponding table, and obtain the business pipeline;

[0021] Configure the business pipeline according to the pipeline arrangement table and capability definition table to complete the configuration of the business pipeline.

[0022] In one embodiment, it further includes:

[0023] Record the processing information generated when the business pipeline processes each shard data, including the processing status;

[0024] Count the number of processes whose processing status is waiting and get the waiting number;

[0025] When the number of waiting nodes exceeds a preset threshold, an adjustment instruction is generated, and the adjustment instruction is used to optimize the distributed data processing capability.

[0026] In one implementation, the method for creating a business flow registration table is:

[0027] Record the processed data of the shard data during the data processing process, and write the processed data into the business flow registration table; the business flow registration table includes the slice primary key, slice primary key string, outlets, list data information business type, data arrival time, account period, slice, city and shard information.

[0028] In a second aspect, an embodiment of the present invention provides a data processing system based on a microservice pipeline, which executes the data processing method based on a microservice pipeline as described above.

[0029] In a third aspect, an embodiment of the present invention provides an electronic device, the device comprising: a memory and a processor. The memory and the processor communicate with each other through an internal connection path, the memory is used to store instructions, the processor is used to execute the instructions stored in the memory, and when the processor executes the instructions stored in the memory, the processor executes the method in any one of the above-mentioned embodiments.

[0030] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer, the method in any one of the above-mentioned embodiments is executed.

[0031] The advantages or beneficial effects of the above technical solution include at least:

[0032] After slicing the incoming data, the present invention can realize the concurrent operation of multiple slices and multi-threaded data processing, avoid the low efficiency of single-line execution, and improve data processing efficiency. Encapsulating the calling program into the capabilities of microservices and deploying them in the capability pool can highlight the characteristics of independent operation of capabilities, resource reuse, and capability orchestration according to business, and realize cross-pipeline calls. The pipeline composed of capabilities with microservice characteristics can realize arbitrary combination of capabilities and multiple reuse, construct a data processing mode with specific processing rules and processes, and maximize the efficient pipeline computing mode of data processing.

[0033] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present invention and should not be regarded as limiting the scope of the present invention.

[0035] Figure 1 It is a flow chart of the data processing method based on the microservice pipeline of the present invention;

[0036] Figure 2 The slicing algorithm table of the present invention;

[0037] Figure 3 The business flow correspondence table of the present invention;

[0038] Figure 4 Arrange a table for the pipeline of the present invention;

[0039] Figure 5 A table defining the capabilities of the present invention;

[0040] Figure 6 It is the service flow instantiation table of the present invention;

[0041] Figure 7 This is the business flow registration form of the present invention;

[0042] Figure 8 It is a pipeline arrangement table in the CRM example of the present invention;

[0043] Fig. 9 It is the business flow correspondence table in the CRM example of the present invention;

[0044] Fig.10 A capability definition table in the CRM instance of the present invention;

[0045] Fig.11 It is a business flow instantiation table in the CRM instance of the present invention;

[0046] Fig.12 It is a structural block diagram of the electronic device of the present invention. DETAILED DESCRIPTION

[0047] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0048] Embodiment 1

[0049] This embodiment provides a data processing method based on microservice pipelines, which processes large-volume, multi-source data through sharding, performs preliminary classification of data sources, and then inputs them into pipelines based on business classification through capability orchestration according to data features, closely integrating data and business, and then supports data processing on the pipeline through a shared capability module with microservice properties. In this way, the computing pressure brought by a large number of data sources is dispersed and the computing power of the system is improved.

[0050] refer to Figure 1 As shown, the data processing method based on the microservice pipeline specifically includes the following steps

[0051] Step S1: used to obtain incoming data, slice the incoming data, and obtain a number of sliced ​​data;

[0052] Step S2: used to obtain business requirements, and perform pipeline instantiation operations according to the business requirements and the preset microservice pipeline model to obtain a business pipeline; wherein the instantiation operation includes allocating corresponding pipelines, orchestrating the microservice capabilities of the pipelines, and configuring the capability processing order of each microservice capability;

[0053] Step S3: All shard data are concurrently diverted to the business pipeline, and the corresponding microservice capabilities are called according to the capability processing order to process the shard data and obtain the data processing results.

[0054] The incoming data is called inflow data. The inflow data is configured with a rule model according to the slicing algorithm. The data slicing process can be performed through the hash modulus algorithm to divert the data. According to different business rules, the business primary key is configured to make the data slice balanced; according to different business types configured, different slicing modes are selected, for example, Figure 2 As shown, the slicing algorithm table records the slicing algorithm ID, slicing primary key, business type, slicing size and status. For the slice with business type 0 (representing all business types), the corresponding slice is located according to the slice primary key, and 50 slices are evenly divided according to the Hash modulus Hash algorithm, and the sliced ​​data is processed concurrently.

[0055] In production, different businesses are composed of different business processing flows. According to different business rules, the business flow can be divided into multiple small functions for processing; in practical applications, these rules can often be reused. Therefore, this embodiment makes the small functions corresponding to the business flow into microservices, deploys them on different machines, and forms a resource-sharing capacity pool. The capacity pool is a cluster composed of multiple machines, so that the number of machines can be reasonably increased or decreased according to the business volume.

[0056] The microservice pipeline is to orchestrate the capabilities in the capability pool according to different business needs to form an independent business pipeline for processing different businesses. The microservice pipeline model includes a business pipeline mapping table, a pipeline orchestration table, and a capability definition table. Figure 3 As shown, Figure 3 It is a business flow correspondence table, which records the pipeline identification, business type, pipeline name and status. The business flow correspondence table defines the business types corresponding to different pipelines.

[0057] like Figure 4 As shown, Figure 4It is a pipeline scheduling table, which records the pipeline ID, microservice capability ID, microservice capability processing order and status. The pipeline scheduling table adds pipelines according to business needs. Each pipeline is configured with the microservice capability ID and processing order to be called, forming a configurable pipeline. A pipeline is fixed to process the same business, and the same business with different data can reuse the pipeline.

[0058] like Figure 5 As shown, Figure 5 It is a capability definition table, which records the capability identifier, microservice caller, microservice capability name and status. The capability definition table is the capability identifier primary key and the microservice corresponding to the capability. The service program required to call different microservice capabilities can be determined based on the capability definition table.

[0059] The microservice pipeline model consists of a business pipeline table, a pipeline orchestration table, and a capability definition table. The pipeline is instantiated according to SQL to solidify the pipeline. Figure 6 As shown, Figure 6 The service type, pipeline ID, called capability ID, service calling procedure, and capability processing order are recorded.

[0060] The establishment of a microservice pipeline model is to pre-integrate a large amount of pipeline data corresponding to different business types to form corresponding business pipeline tables, pipeline orchestration tables, and capability definition tables.

[0061] To perform instantiation operations, the business pipeline corresponding table of the microservice pipeline model is used to find the pipeline identifier corresponding to the business type according to the business requirements, the microservice capabilities configured for the pipeline and the capability processing order are found in the pipeline orchestration table, and the calling program corresponding to the microservice capabilities is found in the capability definition table. These programs are written into the business pipeline instantiation table to form a configured business pipeline. Different businesses execute different pipelines based on the business type as the entry point for division; the microservice capabilities configured for the pipeline are found according to the pipeline orchestration table, and different capabilities in the capability definition table are called in sequence to process the business.

[0062] According to the set segmentation rules, the segmented data is put into the message queue, and the different segmented data are assigned to different pipelines for data processing according to different business categories. The data entering the message queue is registered in the business flow registration table. The business flow registration table is as follows: Figure 7 As shown, the main functional fields include list information string, business type, processing status, slice primary key, slice primary key string, outlet, business type, data arrival time, account period, slice, city and slice information, etc.

[0063] According to the different registered business types, the business flow corresponding table is associated, and the pipeline corresponding to the business type is found from the business flow corresponding table, so as to know which pipeline the incoming data of the business type needs to be assigned to for data processing. During the processing of each shard data, the processing information is written into the business flow registration table. The processing information includes processing status, start time, end time, processing duration and slice, etc. Among them, the processing status includes four states: unprocessed, processed, processing and waiting.

[0064] In addition, after the incoming data of different business types are sliced, in order to facilitate the distinction of the business type corresponding to each sliced ​​data, the sliced ​​data of different business types can be business-tagged, and the business tag of each sliced ​​data can be written into the business flow registration table; subsequently, the business type corresponding to each sliced ​​data can be distinguished according to the business tag, and a large amount of sliced ​​data can be concurrently flowed into the pipeline of the corresponding business type for data processing.

[0065] In the process of recording the processing information generated when the business pipeline processes the data of each shard, the number of processes in the waiting state can also be counted in real time to obtain the waiting number; the waiting number is compared with the preset threshold, and the degree of blockage of the current business pipeline and capacity pool is judged by comparison. When the waiting number exceeds the preset threshold, it means that the current business pipeline and capacity pool are blocked. At this time, an adjustment instruction is generated. The adjustment instruction is used to increase or reduce the number of machines in the capacity pool and optimize the processing capacity of the capacity pool to ensure that the business processing flow can be completed on time.

[0066] The data processing method of the present invention is described with a specific embodiment:

[0067] like Figure 8 to Figure 10 , respectively configure the pipeline arrangement table lines_comp_process, the business flow correspondence table lines_attr_relation and the capability definition table pro_fun_block_define, and execute SQL to instantiate the pipeline. Among them, Figure 8 to Figure 10 The meaning of the Chinese fields corresponding to the Chinese and English field names can be found in Figure 3 to Figure 5 As shown, the description will not be repeated here.

[0068] Taking the card opening and recharge of a new user and the number change of another user in the CRM customer relationship management system as an example, the system will encode the three users as 20237570216028, 20237570218396 and 20237550108005 respectively.

[0069] The front-end interface Kafka message sent three data, which sliced ​​the three user codes respectively. According to the slice algorithm table, the slice primary key is serv_id, and the slice size is 50. According to the slice algorithm, the data is sliced ​​into 50 slices according to serv_id and placed on the corresponding slice. The sliced ​​data will be marked according to different service types. The service tags here are "1", "2", and "3". The service tag "1" is a new user service, the service tag "2" is a fee recharge service, and the service tag "3" is a number change service.

[0070] At the same time, the slice primary key, slice primary key string, outlets, inventory data information business type, data arrival time, account period, slice, city and slice information are registered into the business flow registration table, and the processing status is set to "N" unprocessed status, and then the processing status is updated to "W" waiting status.

[0071] The data starts to be processed, the processing status is updated to "R" (processing), and the start time is registered to start pipeline processing.

[0072] According to the business tag and the pipelines divided by business type in the business flow correspondence table, the sliced ​​data is arranged on the pipeline corresponding to the business type. Therefore, the data coded as 20237570216028 is arranged on the "user new installation" pipeline, the data coded as 20237570218396 is arranged on the "fee recharge" pipeline, and the data coded as 20237550108005 is arranged on the "number change" pipeline. According to the pipeline identification associated pipeline arrangement table, the microservice capabilities included in the "user new installation" pipeline are: capability 1-order attribution channel business processing, capability 2-user information processing, capability 4-batch price calculation processing, capability 5-settlement fee processing.

[0073] The microservice capabilities included in the "Fee Recharge" pipeline are: Capability 3 - Recharge fee processing, Capability 4 - Batch price calculation processing, Capability 5 - Settlement fee processing.

[0074] The microservice capabilities included in the "Number Change" pipeline are: Capability 2-User Information Processing, Capability 6-Number Information Change.

[0075] Instantiate the table according to the business flow, such as Fig.11 As shown, after the corresponding microservice capabilities are orchestrated to the corresponding pipeline according to the capability processing order of the microservice capabilities, the corresponding service program of the microservice capabilities is called to complete the data processing. Fig.11 The meaning of the Chinese fields corresponding to the English field names can be found in Figure 6 As shown, the description will not be repeated here.

[0076] When the data execution process is completed, the processing status is updated to "Y" and the completion time and processing duration are recorded. The processed data is placed in the table of the specified database for subsequent processing.

[0077] Embodiment 2

[0078] This embodiment provides a data processing system based on a microservice pipeline, which executes the data processing method based on a microservice pipeline as described in Embodiment 1. The system includes:

[0079] The slicing module is used to obtain the incoming data, slice the incoming data, and obtain several slicing data;

[0080] The orchestration module is used to obtain business requirements, and execute the instantiation operation of the pipeline according to the business requirements and the preset microservice pipeline model to obtain the business pipeline; wherein the instantiation operation includes allocating the corresponding pipeline, orchestrating the microservice capabilities of the pipeline, and configuring the capability processing order of each microservice capability;

[0081] The data processing module is used to concurrently distribute all shard data to the business pipeline, call the corresponding microservice capabilities according to the capability processing order, process the shard data, and obtain the data processing results.

[0082] After slicing the data, the present invention can realize the concurrent operation of multiple slices and multi-threaded data processing, avoid the low efficiency of single-line execution, and improve data processing efficiency. Encapsulating the calling program into the capabilities of microservices and deploying them in the capability pool can highlight the characteristics of independent operation of capabilities, resource reuse, and capability orchestration according to business, and realize cross-pipeline calls. The pipeline composed of capabilities with microservice characteristics can realize arbitrary combination of capabilities and multiple reuse, construct a data processing mode with specific processing rules and processes, and maximize the efficient pipeline computing mode of data processing.

[0083] The functions of the various modules of the system of the embodiment of the present invention can be found in the corresponding description of the above method, which will not be repeated here.

[0084] Embodiment 3

[0085] Fig.12 FIG. 2 shows a structural block diagram of an electronic device according to an embodiment of the present invention. Fig.12 As shown, the electronic device includes: a memory 100 and a processor 200, wherein the memory 100 stores a computer program that can be run on the processor 200. When the processor 200 executes the computer program, the data processing method based on the microservice pipeline in the above embodiment is implemented. The number of the memory 100 and the processor 200 can be one or more.

[0086] The electronic device also includes:

[0087] The communication interface 300 is used to communicate with external devices and perform data exchange transmission.

[0088] If the memory 100, the processor 200 and the communication interface 300 are implemented independently, the memory 100, the processor 200 and the communication interface 300 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0089] Optionally, in a specific implementation, if the memory 100, the processor 200 and the communication interface 300 are integrated on a chip, the memory 100, the processor 200 and the communication interface 300 can communicate with each other through an internal interface.

[0090] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present invention when executed by a processor.

[0091] An embodiment of the present invention further provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present invention.

[0092] An embodiment of the present invention also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided by the embodiment of the invention.

[0093] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the advanced RISC machines (ARM) architecture.

[0094] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).

[0095] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0096] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0097] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0098] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, segment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present invention includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0099] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0100] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0101] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a disk or an optical disk, etc.

[0102] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A data processing method based on a microservice pipeline, characterized in that: include: Obtaining incoming data, and slicing the incoming data to obtain a plurality of slicing data; Obtaining business requirements, and executing the instantiation operation of the pipeline according to the business requirements and the preset microservice pipeline model to obtain the business pipeline; wherein the instantiation operation includes allocating the corresponding pipeline, orchestrating the microservice capabilities of the pipeline, and configuring the capability processing order of each of the microservice capabilities; All the shard data are concurrently diverted to the business pipeline, the corresponding microservice capabilities are called according to the capability processing sequence, data processing is performed on the shard data, and a data processing result is obtained.

2. According to the data processing method based on microservice pipeline according to claim 1, it is characterized in that: The microservice pipeline model includes a business pipeline correspondence table, a pipeline orchestration table, and a capability definition table; The business flow correspondence table is used to define the business types corresponding to different pipelines; The pipeline orchestration table is used to configure the microservice capabilities and the capability processing order of each of the microservice capabilities; The capability definition table is used to define the service programs that need to be called by different microservice capabilities.

3. The data processing method based on microservice pipeline according to claim 1 is characterized in that: The slicing method comprises: The incoming data is evenly divided according to a hash modulus algorithm.

4. The data processing method based on microservice pipeline according to claim 1 is characterized in that: The slicing method comprises: When the slicing process of the slicing data is completed, the slicing data is stored in a message queue, and a business flow registration table is created according to the slicing data stored in the message queue; Identify the business requirements corresponding to each of the shard data, and obtain the business type corresponding to each of the shard data; The fragmented data of different business types are marked with business tags, and the business tag of each fragmented data is written into a business flow registration table.

5. The data processing method based on microservice pipeline according to claim 4 is characterized in that: The instantiation operation method includes: Read the service tag corresponding to the fragmented data based on the service flow registration table to obtain the service type of the fragmented data; Associating the microservice pipeline model according to the business type, matching the pipeline corresponding to the business type from the business pipeline corresponding table, and obtaining the business pipeline; The business pipeline is configured according to the pipeline orchestration table of the microservice pipeline model and the capability definition table of the microservice pipeline model to complete the configuration of the business pipeline.

6. The data processing method based on microservice pipeline according to claim 1 is characterized in that: Also includes: Recording processing information generated when the business pipeline processes each of the slice data, wherein the processing information includes a processing status; Counting the number of processes whose processing status is the waiting state to obtain the waiting number; When the waiting number exceeds a preset threshold, an adjustment instruction is generated, and the adjustment instruction is used to optimize the distributed data processing capability.

7. The data processing method based on microservice pipeline according to claim 4 is characterized in that: The method for creating the business flow registration table is: Record the processed data of the shard data during the data processing process, and write the processed data into the business flow registration table; the business flow registration table includes the slice primary key, slice primary key string, outlets, list data information business type, data arrival time, account period, slice, city and shard information.

8. A data processing system based on a microservice pipeline, characterized in that: Execute the data processing method based on the microservice pipeline as described in any one of claims 1 to 7.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the data processing method based on the microservice pipeline as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method based on a microservice pipeline according to any one of claims 1 to 7 is implemented.

Images