Data stream processing method, system, device and readable storage medium

Abstract

The present invention discloses a data stream processing method, system and device, the method comprising: obtaining initial file information and parsing to generate initial business data; performing correction processing based on preset configuration information to obtain corrected business model data; performing pipeline construction processing based on standardized historical samples to obtain a multi-level pipeline model based on the corrected business model data; obtaining a data stream processing task request to obtain the first user business identity; performing identification and matching based on the multi-level pipeline model to obtain the optimal business pipeline; executing business customization process processing based on the optimal business pipeline to obtain a unified data model. The present invention can train to obtain a unified data model, and after obtaining the unified data model, merchants can connect and fuse data from different channels. On the basis of data connectivity, data closed-loop analysis can be performed, and various types of merchant data can be connected and subsequently all-round analysis can be performed.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data stream processing method, system, device and readable storage medium. Background Art

[0002] In recent years, with the rise of SaaS, more and more companies choose to use SaaS services, so the business and volume of SaaS service providers are also gradually increasing, so large-scale data inflow and outflow has become the norm. In the e-commerce scenario, merchants may need to import and export a large amount of relevant data or import customer information, such as basic information, points, levels, membership cards, etc.; in addition, many merchants have more and more offline data and need to connect with many channels, which will inevitably lead to different industry information being produced in different channels. At present, this business model has multiple channels and multiple industries, so the amount of data has become not only more but also more complex than a single model, and all merchants also have the need to synchronize data across channels, so the amount of multi-channel data processed in a short time is extremely large.

[0003] In the existing technology, connecting multi-channel data involves a lot of development work, so most of them only focus on rule matching and process connection, and do not provide solutions for large-scale concurrent acceleration and rapid adaptation to various fields and scenarios. In actual operation, it is impossible to customize import for specified channels and specific file formats, and a lot of manpower is required for subsequent maintenance and development. Once there is a problem with the import process, not only is disaster recovery difficult, but the data inflow cannot be automatically analyzed afterwards, and the data closed loop is not considered.

[0004] In summary, the current problems are as follows:

[0005] First, due to differences in file formats and data formats, it is impossible to quickly adapt to different channel scenarios through the interface and configuration, and it is also impossible to modify parameters to take effect in real time.

[0006] Second, the data flow processing is single and fixed, and business process orchestration cannot be achieved for some scenarios with special business processes.

[0007] Third, in today’s distributed environment, due to network jitter and other reasons, tasks are interrupted and fail. Each business party needs to consider task reliability separately, which increases the cost of business development.

[0008] Fourth, there is a lack of closed-loop data processing such as data analysis, reporting, and suggestions.

[0009] In short, the existing solutions only focus on the process, require a lot of coding development work, and cannot meet the requirements of the SaaS industry to quickly support customized adaptation industries. There will also be technical problems such as disaster recovery, recovery, automatic data burial and core data closed loop in the case of large amounts of data. Summary of the invention

[0010] In view of the shortcomings of the prior art, the present invention provides a data stream processing method, system, device and readable storage medium.

[0011] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0012] A data stream processing method comprises the following steps:

[0013] Obtain the initial file information containing any external data and parse it to generate initial business data;

[0014] Performing correction processing on the initial service data based on preset configuration information to obtain corrected service mode data;

[0015] Based on the standardized historical samples in the standardized historical sample library, the corrected business model data is subjected to pipeline construction processing to obtain a multi-level pipeline model based on the corrected business model data, wherein the multi-level pipeline model includes multiple business pipelines;

[0016] Obtaining a data stream processing task request, initializing the data stream processing task request in combination with preset configuration information, and obtaining a first user service identity;

[0017] Based on the multi-level pipeline model, identifying and matching the first user service identity to obtain an optimal service pipeline corresponding to the first user service identity;

[0018] Business customization process processing is performed based on the optimal business pipeline to obtain a unified data model.

[0019] As an implementable method, obtaining the initial file information containing any external data and parsing it to generate the initial business data includes the following steps:

[0020] Obtain the initial file information and parse it to obtain the business characteristics of the original file;

[0021] Identifying the initial file information based on the business characteristics to obtain file data, wherein the file data includes channel data, business scenario data, and industry information data;

[0022] The file data are arranged and combined to obtain initial business data.

[0023] As an implementable method, the initial service data is corrected based on the preset configuration information to obtain the corrected service mode data, including the following steps:

[0024] Obtaining preset configuration information, wherein the preset configuration information at least includes channel information, business scenario information, and industry information;

[0025] The initial service data is corrected based on the preset configuration information to obtain corrected service mode data.

[0026] As an implementable method, the pipeline construction process is performed on the corrected business model data based on the standardized historical samples in the standardized historical sample library to obtain a multi-level pipeline model based on the corrected business model data, including the following steps:

[0027] Classify and process the standardized historical sample data in the standardized historical sample to obtain a multi-level business pipeline sample including multiple user business identities;

[0028] The corrected business model data is subjected to pipeline construction processing based on the multi-level business pipeline sample to obtain a multi-level pipeline model including multiple business pipelines, wherein the multiple user business identities and the multiple business pipelines have a one-to-one correspondence.

[0029] As an implementable method, the pipeline construction process is performed on the corrected business model data based on the standardized historical samples in the standardized historical sample library to obtain a multi-level pipeline model based on the corrected business model data, including the following steps:

[0030] Obtaining standardized historical samples in a standardized historical sample library, and extracting standardized historical sample information, wherein the standardized historical sample information includes multiple user service identities;

[0031] Classify and process the standardized historical sample information in combination with preset configuration information to obtain a multi-level service pipeline sample including the multiple user service identities;

[0032] The correction business mode data is subjected to multi-level classification processing by preset configuration information to obtain multi-level correction business mode data;

[0033] The multi-level correction business model data is constructed level by level through the multi-level business pipeline samples to obtain a multi-level pipeline model including multiple business pipelines.

[0034] As an implementable method, the data stream processing task request is obtained, and the data processing task request is initialized and processed in combination with preset configuration information to obtain the first user service identity, specifically:

[0035] Loading the file information included in the data stream processing task request based on the preset configuration information, and initializing the file information according to the preset configuration information to obtain first initialization file information;

[0036] The first initialization file information is parsed level by level to obtain a first user service identity.

[0037] As an implementable method, the identifying and matching the first user service identity based on the multi-level service pipeline model to obtain the optimal service pipeline corresponding to the first user service identity includes the following steps:

[0038] Based on matching the first user service identity with the multi-level pipeline model, a service pipeline corresponding to the user service identity with the highest matching value is selected, and the service pipeline corresponding to the user service identity with the highest matching value is the optimal service pipeline corresponding to the first user service identity.

[0039] As an implementable method, the business customization process processing includes the following steps:

[0040] Parsing the optimal business pipeline, loading field data corresponding to the project according to the configured field mapping rule, and obtaining parsed data;

[0041] The parsed data is classified into structured index data and unstructured data, and stored in the MySQL database and HBase database respectively;

[0042] Scan the data and perform type and business verification. After passing the verification, aggregate and reorganize the related data involved to obtain reorganized data;

[0043] The reorganized data is transformed to obtain a unified data model.

[0044] As an implementable method, it also includes the steps of inspecting the business customization process processing:

[0045] Delay detection of business customization process processing based on preset reasonable scheduling time to see if it is completed;

[0046] If it times out, it will be re-activated to continue the business customization process.

[0047] If the execution of the business customization process fails, the fault location will be automatically recorded and a unique identifier will be created, and then the failover recovery process will be performed.

[0048] A data stream processing system, comprising an acquisition and analysis module, a data correction module, a model building module, an initialization and analysis module, an identification and matching module and a customized process processing module;

[0049] The acquisition and parsing module is used to acquire and parse the initial file information containing any external data to generate initial business data;

[0050] The data correction module performs correction processing on the initial service data based on preset configuration information to obtain corrected service mode data;

[0051] The model building module performs pipeline building processing on the corrected business model data based on the standardized historical samples in the standardized historical sample library to obtain a multi-level pipeline model based on the corrected business model data, wherein the multi-level pipeline model includes multiple business pipelines;

[0052] The initialization parsing module is used to obtain a data stream processing task request, initialize the data stream processing task request in combination with preset configuration information, and obtain a first user service identity;

[0053] The identification and matching module identifies and matches the first user service identity based on the multi-level pipeline model to obtain an optimal service pipeline corresponding to the first user service identity;

[0054] The customized process processing module performs business customized process processing based on the optimal business pipeline to obtain a unified data model.

[0055] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following method steps are implemented:

[0056] Obtain the initial file information containing any external data and parse it to generate initial business data;

[0057] Performing correction processing on the initial service data based on preset configuration information to obtain corrected service mode data;

[0058] Based on the standardized historical samples in the standardized historical sample library, the corrected business model data is subjected to pipeline construction processing to obtain a multi-level pipeline model based on the corrected business model data, wherein the multi-level pipeline model includes multiple business pipelines;

[0059] Obtaining a data stream processing task request, initializing the data stream processing task request in combination with preset configuration information, and obtaining a first user service identity;

[0060] Based on the multi-level pipeline model, identifying and matching the first user service identity to obtain an optimal service pipeline corresponding to the first user service identity;

[0061] Business customization process processing is performed based on the optimal business pipeline to obtain a unified data model.

[0062] An electronic device comprises a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the following method steps when executing the computer program:

[0063] Obtain the initial file information containing any external data and parse it to generate initial business data;

[0064] Performing correction processing on the initial service data based on preset configuration information to obtain corrected service mode data;

[0065] Based on the standardized historical samples in the standardized historical sample library, the corrected business model data is subjected to pipeline construction processing to obtain a multi-level pipeline model based on the corrected business model data, wherein the multi-level pipeline model includes multiple business pipelines;

[0066] Obtaining a data stream processing task request, initializing the data stream processing task request in combination with preset configuration information, and obtaining a first user service identity;

[0067] Based on the multi-level pipeline model, identifying and matching the first user service identity to obtain an optimal service pipeline corresponding to the first user service identity;

[0068] Business customization process processing is performed based on the optimal business pipeline to obtain a unified data model.

[0069] The present invention has significant technical effects due to the adoption of the above technical solution:

[0070] Through the method, system and device proposed in the present invention, a unified data model can be trained. After obtaining the unified data model, merchants can connect and integrate data from different channels. On the basis of data connectivity, data closed-loop analysis can be performed, and various types of merchant data can be connected and subsequently analyzed in an all-round way, providing merchants with more favorable business guidance and various marketing reports, etc., making it more convenient for merchants to manage their own data. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0072] Figure 1 It is a schematic diagram of the overall process of the method of the present invention;

[0073] Figure 2 It is a flowchart of generating initial business data;

[0074] Figure 3 is a schematic diagram of a process for obtaining corrected business model data;

[0075] Figure 4 is a flow chart of an embodiment of a multi-level pipeline model;

[0076] Figure 5 is a flow chart of another embodiment of a multi-level pipeline model;

[0077] Figure 6 is a flow chart of obtaining the service identity of the first user;

[0078] Figure 7 It is a flowchart of the customized process processing;

[0079] Figure 8 It is a flowchart for inspecting the business customization process processing;

[0080] Fig. 9 It is a schematic diagram of the overall structure of the system of the present invention. DETAILED DESCRIPTION

[0081] The present invention is further described in detail below in conjunction with embodiments. The following embodiments are for explanation of the present invention but the present invention is not limited to the following embodiments.

[0082] With the development of science and technology, e-commerce has undoubtedly become the mainstream at present, and e-commerce of various channels has followed. Therefore, a merchant may choose e-commerce of multiple channels, so it is necessary to import and export a large amount of data or import customer information, such as basic information, points, levels, membership cards, etc.; In addition, many merchants have more and more offline data and need to be connected with many channels, which will inevitably lead to the output of different industry information under different channels. At present, this business model has multiple channels and multiple industries, so the amount of data becomes not only more but also more complex than that of a single model, and all merchants also have the need to synchronize data across channels. Therefore, the amount of multi-channel data processed in a short time is extremely large, which is particularly troublesome to operate with the current operation method. Therefore, the present invention is implemented in the following ways:

[0083] Embodiment 1:

[0084] A data stream processing method, such as Figure 1 As shown, the following steps are included:

[0085] S100, obtaining and parsing initial file information containing any external data to generate initial business data;

[0086] S200, performing correction processing on the initial service data based on preset configuration information to obtain corrected service mode data;

[0087] S300, performing pipeline construction processing on the corrected business model data based on the standardized historical samples in the standardized historical sample library to obtain a multi-level pipeline model based on the corrected business model data, wherein the multi-level pipeline model includes multiple business pipelines;

[0088] S400, obtaining a data stream processing task request, initializing the data stream processing task request in combination with preset configuration information, and obtaining a first user service identity;

[0089] S500: Based on the multi-level pipeline model, identify and match the first user service identity to obtain the optimal service pipeline corresponding to the first user service identity;

[0090] S600: Execute business customization process processing based on the optimal business pipeline to obtain a unified data model.

[0091] The present invention can actually be regarded as a process of model training and model application:

[0092] The process of model training is to obtain the initial business data by acquiring any external data, and then correct the channel information, business scenario information and industry information of the initial business data layer by layer in combination with the preset configuration information obtained by identifying the standardized historical sample library to obtain the corrected business model data; the corrected business model data is combined with the standardized historical sample library for training to obtain a multi-level pipeline model. During the model training process, any external data obtained is first parsed and then corrected, and during the correction process, it will be corrected in detail with the channel information, business scenario information and industry information obtained from the standardized historical sample library to obtain corrected business model data with more accurate levels and types, and then combined with the standardized historical sample library to train the model of multiple business pipelines, and finally obtain a multi-level pipeline model.

[0093] The process of model application: obtain the data flow processing task request and parse it to obtain the first user business identity. Since the multi-level pipeline model contains multiple business pipelines, and multiple user business identities and multiple business pipelines are in a one-to-one correspondence, the first user business identity is identified and matched through the multi-level pipeline model to obtain the optimal business pipeline corresponding to the first user business identity. Based on the optimal business pipeline, the business customization process processing is performed to obtain a unified data model. This unified data model is the data model to be used later, and this data model can be returned to the standardized historical sample library later. Through the method proposed in the present invention, a unified data model can be trained. After obtaining the unified data model, the merchant can connect and integrate the data from different channels. On the basis of data connectivity, data closed-loop analysis can be performed, and various types of merchant data can be connected and subsequently analyzed in an all-round manner, providing merchants with more favorable business guidance and various marketing reports, etc., so that merchants can manage their own data more conveniently.

[0094] In step S100, the initial file information containing any external data is obtained and parsed to generate initial business data, such as Figure 2 As shown, the following steps are included:

[0095] S110, obtaining initial file information and parsing it to obtain business features of the original file;

[0096] S120, identifying the initial file information based on the business characteristics to obtain file data, wherein the file data includes channel data, business scenario data, and industry information data;

[0097] S130: Arrange and combine the file data to obtain initial business data.

[0098] Here, any external data may be data contained in initial files of various formats generated by different merchants or ERP systems, such as data carried by files such as Excel, csv, txt, etc. These arbitrary external data may include channel data, business scenario data, industry information data, etc., but are not presented in a unified format or mode. Therefore, after obtaining the initial file information containing any external data, it is necessary to parse it to obtain the initial business data to be used by the present invention.

[0099] Business characteristics can be understood as the business features reflected in the initial file information. They can be obtained by business experts or expert systems through analysis of existing businesses, or they can be obtained in combination with historical experience. By identifying the initial file information in combination with business characteristics, we can obtain the channel data, business scenario data and industry information data of the initial file. For example, if there are platform dimension channels, ERP dimension channels and CRM manufacturer dimension channels, there will be channel data corresponding to these channels, such as Xiaohongshu, Alipay, Taobao, Douyin, Bojun, Yum! Brands, Regent, UFIDA, Yikr, etc.; if we get the business scenario, there will be corresponding business scenario data, such as: membership, stored value, orders, points, commodities, etc.; if we get the business scenario and industry information, there will be corresponding industry information data, such as: hotel and travel, fashion, baking, etc.

[0100] Permutations and combinations are actually multi-dimensional combinations to obtain fine-grained business models. For example, after parsing an initial file information and then performing permutations and combinations, we can obtain: Xiaohongshu, points, baking, and related information of baking members.

[0101] In addition, this step can be understood as splitting the initial file information into multiple different types of data, and then combining these file data according to different dimensions or different rules to obtain initial business data.

[0102] In step S200, the initial service data is corrected based on the preset configuration information to obtain corrected service mode data, such as Figure 3 As shown, the following steps are included:

[0103] S210, obtaining preset configuration information, where the preset configuration information at least includes channel information, business scenario information, and industry information;

[0104] S220: Correct the initial service data based on the preset configuration information to obtain corrected service mode data.

[0105] Here, the preset configuration information can be obtained by obtaining a standardized historical sample library and identifying the standardized historical sample information to obtain the preset configuration information. Of course, it can also be implemented in other ways, which will not be described in detail here;

[0106] For the initial business data, it may not meet the current needs, so it can be corrected according to the preset configuration information, such as adjusting the file format or data format that can be processed, the mapping rules of data fields, the verification rules of fields, data applications, closed-loop operations (data analysis, reports, suggestions), etc. according to the channel information, business scenario information and industry information, and then obtaining the corrected business model data. For example, a baking merchant believes that a member number is considered a member, while other merchants believe that a member + ID number is considered a member. Or the member information of this merchant includes mobile phone number + name, while the member information of another merchant includes mobile phone number + name + gender. Therefore, the initial business data needs to be corrected according to the preset configuration information to obtain the corrected business model data with consistent rules.

[0107] In step S300, the pipeline construction process is performed on the corrected business model data based on the standardized historical samples in the standardized historical sample library to obtain a multi-level pipeline model based on the corrected business model data, such as Figure 4 As shown, the following steps are included:

[0108] S310a, classifying and processing the standardized historical sample data in the standardized historical sample to obtain a multi-level service pipeline sample including multiple user service identities;

[0109] S310b. Perform pipeline construction processing on the corrected business model data based on the multi-level business pipeline sample to obtain a multi-level pipeline model including multiple business pipelines, wherein the multiple user business identities and the multiple business pipelines have a one-to-one correspondence.

[0110] Different merchants may have multiple user business identities. When establishing a multi-level pipeline model, multiple business pipelines will be constructed based on the multiple user business identities, and a multi-level pipeline model containing multiple business pipelines will be obtained.

[0111] In step S300, the pipeline construction process is performed on the corrected business model data based on the standardized historical samples in the standardized historical sample library to obtain a multi-level pipeline model based on the corrected business model data, such as Figure 5 As shown, the following steps are included:

[0112] S310, obtaining standardized historical samples in a standardized historical sample library, and extracting standardized historical sample information, wherein the standardized historical sample information includes multiple user service identities;

[0113] S320, classifying and processing the standardized historical sample information in combination with preset configuration information to obtain a multi-level service pipeline sample including the multiple user service identities;

[0114] S330, performing multi-level classification processing on the correction business mode data according to preset configuration information to obtain multi-level correction business mode data;

[0115] S340: construct the multi-level correction business model data level by level through the multi-level business pipeline sample to obtain a multi-level pipeline model including multiple business pipelines.

[0116] When constructing a pipeline, the present invention classifies the standardized historical sample information to obtain a multi-level business pipeline sample containing multiple user business identities, wherein there is a one-to-one correspondence between the user business identity and the business pipeline. At the same time, the correction business model data is also classified (specifically, a classification model can be used) to obtain multi-level correction business model data, and finally the multi-level business pipeline sample is used to match and construct the multi-level correction business model data step by step to form a multi-level pipeline model. This method can improve the universality of the data model and the accuracy of matching by performing fine-grained decomposition of user business identities and constructing business pipelines for business data.

[0117] In step S400, the data stream processing task request is obtained, and the data processing task request is initialized in combination with the preset configuration information to obtain the first user service identity, such as Figure 6 As shown, specifically:

[0118] S410, loading the file information included in the data stream processing task request based on the preset configuration information, and initializing the file information according to the preset configuration information to obtain first initialization file information;

[0119] S420: parse the first initialization file information level by level to obtain a first user service identity.

[0120] When a merchant or user initiates a data stream processing task request, it is necessary to load some core parameter configuration and process file information, so it is necessary to initialize the data stream processing task, that is, load the corresponding preset configuration information for initialization, and break these file information into sufficiently small execution units according to certain rules, and then save the file information and task dynamic variables. The initialization process is actually the process of preprocessing the contained file information according to the preset configuration information, which can be understood as "classification or identification", such as identifying a certain product or member information of a merchant on a certain e-commerce platform, and saving it after identification. Therefore, the first initialization file information is actually the data in a unified format loaded according to the preset configuration information.

[0121] In step S500, the first user service identity is identified and matched based on the multi-level service pipeline model to obtain the optimal service pipeline corresponding to the first user service identity, including the following steps:

[0122] Based on matching the first user service identity with the multi-level pipeline model, a service pipeline corresponding to the user service identity with the highest matching value is selected, and the service pipeline corresponding to the user service identity with the highest matching value is the optimal service pipeline corresponding to the first user service identity.

[0123] Since the multi-level pipeline model includes multiple business pipelines, and the multiple business pipelines correspond one-to-one to multiple user business identities, when matching the multi-level pipeline model, the first user business identity is matched with the multiple business pipelines in the multi-level pipeline model, and multiple matching results will appear. Of course, the matching results will be presented in the form of matching degree values. The business pipeline corresponding to the highest matching degree value is selected and this business pipeline is used as the optimal business pipeline.

[0124] In step S600, if Figure 7 As shown, the business customization process includes the following steps:

[0125] S610, parsing the optimal business pipeline, loading field data corresponding to the project according to the configured field mapping rule, and obtaining parsed data;

[0126] S620, classifying the parsed data according to structured index data and unstructured data, and storing them in a MySQL database and an HBase database respectively;

[0127] S630, scanning the data and performing type and business verification. After passing the verification, the related data involved are aggregated and reorganized to obtain reorganized data;

[0128] S640: Convert the reorganized data to obtain a unified data model.

[0129] The process of business customization process processing is actually the process of converting any external data into a unified data model, and the process of matching the optimal business pipeline is actually the process of data reorganization and matching identification. Since the original file data obtained is large and complex, this data will be reorganized and then analyzed.

[0130] After obtaining a unified data model, merchants can connect and integrate data from different channels. Based on data connectivity, data closed-loop analysis can be performed, and all types of merchant data can be connected and subsequently analyzed in an all-round manner.

[0131] Specifically, Figure 8As shown, it also includes the steps of inspecting the business customization process processing:

[0132] S650, delay detection of whether the business customization process is completed based on a preset reasonable scheduling time;

[0133] S660: If the timeout period has expired, the service customization process is restarted and continued to be executed;

[0134] S670: If the execution of the business customization process fails, the fault location is automatically recorded and a unique identifier is created, and then the failover recovery process is performed.

[0135] Conduct real-time inspections on business customized process processing, and automatically perform failover recovery if task execution fails: record the fault location of the current failed execution and create a unique identifier, load the marked business pattern recognition results and workflow execution until the data flow processing task is successfully executed or the execution limit is reached.

[0136] When the data flow processing task starts to execute, a reasonable scheduling time will be created based on the data volume of the data flow processing task (that is, based on the throughput, computing performance and historical import experience, the recommended splitting level of each task is generated. For example, if the total data volume is 8 million, it will be split into 16 subtasks with a data volume of 500,000 each). According to the preset reasonable scheduling time, a delay detection is performed to check whether the business customized process processing is completed. If it times out, it needs to be re-awakened to continue executing the task. This delay detection is actually to estimate the execution time of the main task based on the data volume and execution speed of a subtask. Since the execution process of the subtask takes a certain amount of time, it is necessary to set a reasonable scheduling time to perform delay detection in order to detect whether the business customized process processing is completed.

[0137] The purpose of establishing a unique identifier is to play a role in the failover recovery process. When the data flow processing task is restarted, the unique identifier position is queried. After finding it, the subsequent process is executed for subsequent tasks until the data flow processing task is successfully executed or the upper limit of the execution times is reached.

[0138] Specific examples:

[0139] A certain merchant has offline membership stores and online membership stores. Currently, it is necessary to connect the data of the online and offline membership stores, that is, to build different channel memberships for the same merchant. This will facilitate the management of the merchant and the unified implementation of various online and offline promotional activities or the unified issuance of coupons.

[0140] Obtain information of all original files online and offline, which contain data of a certain merchant and data of other merchants. Process and parse the online and offline data to obtain initial business data, and fine-tune these initial business data in combination with preset configuration information to obtain corrected business model data.

[0141] Pipeline the correction business model data to obtain a multi-level pipeline model;

[0142] Get a data flow processing task request from a certain merchant, perform initialization processing and parsing, obtain multiple user business identities of a certain merchant, obtain the most suitable business pipeline based on the multi-level pipeline model, and then execute customized process processing, that is, perform a series of processing on the data, and then obtain a unified data model. This data model is a trained data connectivity model that can be used by our technical staff and the corresponding merchants. These models are then added as samples to the standardized historical sample library.

[0143] It should be understood that although Figure 1-8 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-8 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0144] Embodiment 2:

[0145] A data stream processing system, such as Fig. 9 As shown, it includes an acquisition and analysis module 100, a data correction module 200, a model building module 300, an initialization and analysis module 400, an identification and matching module 500 and a custom process processing module 600;

[0146] The acquisition and parsing module 100 is used to acquire and parse the initial file information containing any external data to generate initial business data;

[0147] The data correction module 200 performs correction processing on the initial service data based on preset configuration information to obtain corrected service mode data;

[0148] The model building module 300 performs pipeline building processing on the corrected business model data based on the standardized historical samples in the standardized historical sample library to obtain a multi-level pipeline model based on the corrected business model data, wherein the multi-level pipeline model includes multiple business pipelines;

[0149] The initialization parsing module 400 is used to obtain a data stream processing task request, initialize the data stream processing task request in combination with preset configuration information, and obtain a first user service identity;

[0150] The identification and matching module 500 identifies and matches the first user service identity based on the multi-level pipeline model to obtain the optimal service pipeline corresponding to the first user service identity;

[0151] The customized process processing module 600 performs business customized process processing based on the optimal business pipeline to obtain a unified data model.

[0152] Embodiment 3:

[0153] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following method steps are implemented:

[0154] S100, obtaining and parsing initial file information containing any external data to generate initial business data;

[0155] S200, performing correction processing on the initial service data based on preset configuration information to obtain corrected service mode data;

[0156] S300, performing pipeline construction processing on the corrected business model data based on the standardized historical samples in the standardized historical sample library to obtain a multi-level pipeline model based on the corrected business model data, wherein the multi-level pipeline model includes multiple business pipelines;

[0157] S400, obtaining a data stream processing task request, initializing the data stream processing task request in combination with preset configuration information, and obtaining a first user service identity;

[0158] S500: Based on the multi-level pipeline model, identify and match the first user service identity to obtain the optimal service pipeline corresponding to the first user service identity;

[0159] S600: Execute business customization process processing based on the optimal business pipeline to obtain a unified data model.

[0160] Standardization History Examples The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other.

[0161] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] The present invention is described with reference to the flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0165] It should be noted that:

[0166] The "one embodiment" or "embodiment" mentioned in the specification means that the specific features, structures or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present invention. Therefore, the phrases "one embodiment" or "embodiment" appearing in various places throughout the specification do not necessarily refer to the same embodiment.

[0167] In addition, it should be noted that the shapes and names of the parts and components of the specific embodiments described in this specification may be different. Any equivalent or simple changes made based on the structure, features and principles described in the patent concept of the present invention are included in the protection scope of the patent of the present invention. The technicians in the technical field of the present invention can make various modifications or supplements to the specific embodiments described or replace them in a similar manner, as long as they do not deviate from the structure of the present invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A data stream processing method, characterized in that: The following steps are involved: Obtain the initial file information containing any external data and parse it to generate initial business data; Performing correction processing on the initial service data based on preset configuration information to obtain corrected service mode data; Based on the standardized historical samples in the standardized historical sample library, the corrected business model data is subjected to pipeline construction processing to obtain a multi-level pipeline model based on the corrected business model data, wherein the multi-level pipeline model includes multiple business pipelines; Obtaining a data stream processing task request, initializing the data stream processing task request in combination with preset configuration information, and obtaining a first user service identity; Based on the multi-level pipeline model, identifying and matching the first user service identity to obtain an optimal service pipeline corresponding to the first user service identity; Business customization process processing is performed based on the optimal business pipeline to obtain a unified data model.

2. The data stream processing method according to claim 1, characterized in that: The process of obtaining and parsing the initial file information containing any external data to generate initial business data includes the following steps: Obtain the initial file information and parse it to obtain the business characteristics of the original file; Identifying the initial file information based on the business characteristics to obtain file data, wherein the file data includes channel data, business scenario data, and industry information data; The file data are arranged and combined to obtain initial business data.

3. The data stream processing method according to claim 2, characterized in that: Correcting the initial service data based on the preset configuration information to obtain corrected service mode data includes the following steps: Obtaining preset configuration information, wherein the preset configuration information at least includes channel information, business scenario information, and industry information; The initial service data is corrected based on the preset configuration information to obtain corrected service mode data.

4. The data stream processing method according to claim 1, characterized in that: The pipeline construction process is performed on the corrected business model data based on the standardized historical samples in the standardized historical sample library to obtain a multi-level pipeline model based on the corrected business model data, including the following steps: Classify and process the standardized historical sample data in the standardized historical sample to obtain a multi-level business pipeline sample including multiple user business identities; The corrected business model data is subjected to pipeline construction processing based on the multi-level business pipeline sample to obtain a multi-level pipeline model including multiple business pipelines, wherein the multiple user business identities and the multiple business pipelines have a one-to-one correspondence.

5. The data stream processing method according to claim 4, characterized in that: The pipeline construction process is performed on the corrected business model data based on the standardized historical samples in the standardized historical sample library to obtain a multi-level pipeline model based on the corrected business model data, including the following steps: Obtaining standardized historical samples in a standardized historical sample library, and extracting standardized historical sample information, wherein the standardized historical sample information includes multiple user service identities; Classify and process the standardized historical sample information in combination with preset configuration information to obtain a multi-level service pipeline sample including the multiple user service identities; The correction business mode data is subjected to multi-level classification processing by preset configuration information to obtain multi-level correction business mode data; The multi-level correction business model data is constructed level by level through the multi-level business pipeline samples to obtain a multi-level pipeline model including multiple business pipelines.

6. The data stream processing method according to claim 1, characterized in that: The acquiring of the data stream processing task request, initializing the data stream processing task request in combination with the preset configuration information, and obtaining the first user service identity, is specifically: Loading the file information included in the data stream processing task request based on the preset configuration information, and initializing the file information according to the preset configuration information to obtain first initialization file information; The first initialization file information is parsed level by level to obtain a first user service identity.

7. The data stream processing method according to claim 4 or 5, characterized in that: The step of identifying and matching the first user service identity based on the multi-level pipeline model to obtain the optimal service pipeline corresponding to the first user service identity includes the following steps: Based on matching the first user service identity with the multi-level pipeline model, a service pipeline corresponding to the user service identity with the highest matching value is selected, and the service pipeline corresponding to the user service identity with the highest matching value is the optimal service pipeline corresponding to the first user service identity.

8. The data stream processing method according to claim 1, characterized in that: The business customization process includes the following steps: Parsing the optimal business pipeline, loading field data corresponding to the project according to the configured field mapping rule, and obtaining parsed data; The parsed data is classified into structured index data and unstructured data, and stored in the MySQL database and HBase database respectively; Scan the data and perform type and business verification. After passing the verification, aggregate and reorganize the related data involved to obtain reorganized data; The reorganized data is transformed to obtain a unified data model.

9. The data stream processing method according to claim 1 or 8, characterized in that: It also includes steps to inspect the business customization process processing: Delay detection of business customization process processing based on preset reasonable scheduling time to see if it is completed; If it times out, it will be re-activated to continue the business customization process. If the execution of the business customization process fails, the fault location will be automatically recorded and a unique identifier will be created, and then the failover recovery process will be performed.

10. A data stream processing system, characterized in that: It includes acquisition and analysis module, data correction module, model building module, initialization and analysis module, identification and matching module and customized process processing module; The acquisition and parsing module is used to acquire and parse the initial file information containing any external data to generate initial business data; The data correction module performs correction processing on the initial service data based on preset configuration information to obtain corrected service mode data; The model building module performs pipeline building processing on the corrected business model data based on the standardized historical samples in the standardized historical sample library to obtain a multi-level pipeline model based on the corrected business model data, wherein the multi-level pipeline model includes multiple business pipelines; The initialization parsing module is used to obtain a data stream processing task request, initialize the data stream processing task request in combination with preset configuration information, and obtain a first user service identity; The identification and matching module identifies and matches the first user service identity based on the multi-level pipeline model to obtain an optimal service pipeline corresponding to the first user service identity; The customized process processing module performs business customized process processing based on the optimal business pipeline to obtain a unified data model.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the data stream processing method according to any one of claims 1 to 9 are implemented.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the data stream processing method according to any one of claims 1 to 9 are implemented.

Images