A multidimensional data processing method and system based on stream-batch integration

Abstract

This application relates to the field of data processing technology and specifically discloses a multi-dimensional data processing method and system based on stream-batch integration. The method comprises: obtaining business requirement information, including the business scenario to which it belongs; parsing the business requirement information using a preset method to obtain data fields and data processing logic; extracting data sources and data sinks from a preset component library based on the data fields; matching corresponding operators from the preset component library based on the data processing logic; and process-based splicing of data sources, operators, and data sinks to obtain process configuration information. By componentizing business-related processes, relevant components can be selected based on business needs for process-based serial configuration, thereby achieving the purpose of rapid deployment and implementation.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a multi-dimensional data processing method and system based on stream-batch integration. Background Art

[0002] As the demand for market response time becomes increasingly higher, the value of real-time data brought by real-time marketing, precision marketing, real-time analysis, etc. is becoming increasingly obvious and occupies an important position. Technologies and methods for real-time processing and analysis of real-time data are also emerging in an endless stream.

[0003] Existing real-time data processing frameworks often build real-time batch multidimensional data processing models, or use mainstream message queues and stream data processing technologies to establish real-time multidimensional data models. However, these methods require too much modeling time, resulting in delayed business timeliness, affecting the final data-driven decision-making and pre-analysis, and the high dependency between modules leads to low reusability. Summary of the Invention

[0004] The purpose of this application is to provide a method for high-efficiency data processing and efficient implementation of integrated stream and batch components, which disassembles the relevant processes involved in the credit card business in the financial industry and forms relevant components, which are configured in series in a process-based manner and form scenario templates. Through rapid retrieval and configuration adjustment, the purpose of rapid deployment and implementation can be achieved.

[0005] In the first aspect, the present application provides a multi-dimensional data processing method based on stream-batch integration, which adopts the following technical solutions:

[0006] Obtain business demand information and parse it using preset methods to obtain data fields and data processing logic;

[0007] Based on the data fields, extract the data source and data sink from the preset component library;

[0008] Based on the data processing logic, the corresponding operator is matched from the preset component library;

[0009] Perform process-based splicing of data sources, operators, and data sinks to obtain process configuration information.

[0010] Through the above technical solution, business demand information can be parsed to obtain components related to the business, and then the process series configuration of the components can be performed to achieve rapid deployment and implementation. There is no need to manually sort out business needs, which reduces the time required for business data processing. In addition, the process configuration components are not fixedly connected and can be reorganized according to actual business needs, which increases reusability.

[0011] Optionally, the business requirement information further includes data execution conditions. The business requirement information is parsed using a preset method to obtain data fields and data processing logic, including:

[0012] For business demand information, keywords are extracted through a preset natural language processing model;

[0013] Based on keywords, generate associated data information through the preset data association rule model;

[0014] Based on the data execution conditions, matching is performed through preset operator rules to obtain data processing logic;

[0015] Get data fields based on associated data information and data processing logic.

[0016] Optionally, obtaining data fields based on associated data information and data processing logic includes:

[0017] Identify data producers and data consumers based on linked data information and data processing logic;

[0018] If there is more than one data producer, then data joint processing logic is obtained based on multiple data producers to serve as supplementary data processing logic;

[0019] adding the supplementary data processing logic to the data processing logic to form a new data processing logic;

[0020] Get data fields based on associated data information and new data processing logic.

[0021] Optionally, extracting a data source and a data sink from a preset component library based on the data field includes:

[0022] Determine the data types involved in the business based on data producers and data consumers;

[0023] Based on the data field, obtain the data transmission channel information;

[0024] According to the data transmission channel information and data type, the corresponding data source and data sink are matched from the preset component library.

[0025] Optionally, the data source includes attribute configuration items, and the data sink includes configuration field names. Process-based splicing of the data source, operator, and data sink to obtain process configuration information includes:

[0026] Build data parameter fields based on attribute configuration items;

[0027] Obtain data output information based on data parameter fields and operators;

[0028] Match the data output information with the configured field name using a preset method to obtain the matching result;

[0029] Determine whether the matching result reaches the preset threshold,

[0030] If so, the process is connected in the preset manner based on the current data source, operator, and data sink, and the corresponding attribute configuration is added to obtain the process configuration information;

[0031] If not, the name of the field that failed to match is obtained, and a first prompt message is input, where the first prompt message indicates that the process configuration has failed and corresponding adjustments need to be made.

[0032] Optionally, the business requirement information includes the business scenario to which it belongs, and after obtaining the process configuration information, the following is further included:

[0033] Verify the process configuration information using a preset verification method and obtain the data verification results;

[0034] Determine whether the verification result contains abnormal information.

[0035] If yes, obtain the abnormal information and output the second prompt information, which indicates that the data verification has failed and needs to be adjusted accordingly;

[0036] If not, the process configuration information is saved as a process configuration template corresponding to the corresponding scenario, and the process configuration template is stored in a preset process configuration management library.

[0037] Optionally, after storing the process configuration template in the preset process configuration management library, the following steps are also included:

[0038] The business requirement information corresponding to the process configuration template is used as a subordinate tag of the corresponding scenario;

[0039] Associate and store the process configuration template with the corresponding business requirement information;

[0040] Based on the business requirement information corresponding to the process configuration template, reorganize in a preset manner to obtain the reorganized business requirement information;

[0041] According to the reorganized business information, it is parsed through the preset method and the corresponding process configuration template is obtained;

[0042] The obtained process configuration template is associated with the corresponding reorganization business requirement information and stored.

[0043] Optionally, after obtaining the business requirement information, the process further includes:

[0044] Based on the business scenario, confirm the business scenario to which it belongs;

[0045] Determine whether there is a process configuration template for the corresponding scenario in the preset process configuration management library;

[0046] If a process configuration template exists, the current business requirement information is matched with the business requirement information corresponding to the existing process configuration template to determine whether the highest similarity reaches a preset threshold;

[0047] If the highest similarity reaches the preset threshold, the process configuration template corresponding to the business requirement information with the highest similarity is used as the target process configuration template;

[0048] According to the target process configuration template, obtain the corresponding process configuration information;

[0049] If there is no process configuration template or the highest similarity does not reach the preset threshold, the business requirement information is parsed using a preset method.

[0050] In a second aspect, the present application provides a multi-dimensional data processing system based on stream-batch integration, comprising:

[0051] An information acquisition module (101) is used to acquire business demand information and parse the business demand information using a preset method to obtain data fields and data processing logic;

[0052] A data component acquisition module (102) is used to extract a data source and a data sink from a preset component library based on a data field;

[0053] An operator component acquisition module (103) is used to match corresponding operators from a preset component library based on data processing logic;

[0054] The process configuration module (104) is used to process the data source, operator and data sink and obtain process configuration information.

[0055] In a third aspect, the present application provides a computer-readable storage medium storing a computer program that can be loaded by a processor and execute the above-mentioned multi-dimensional data processing method based on stream-batch integration.

[0056] In summary, this application decomposes the relevant processes involved in the business, that is, by analyzing the business needs, it can obtain the relevant components, and then form a process-based serial configuration from the components to achieve the purpose of rapid deployment and implementation; in addition, by saving the newly formed process configuration information as a template for the corresponding business scenario, the existing process configuration template can be used as a benchmark for the same or similar business processes to quickly deploy and release the business process. In addition, if a problem occurs, the component where the problem occurs can be optimized and adjusted without affecting the entire system, which shortens the implementation cycle and reduces the cost of trial and error. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of a multi-dimensional data processing method based on stream-batch integration provided by an embodiment of the present application;

[0058] Figure 2 This is a flowchart of obtaining data fields and data processing logic based on business demand information provided by an embodiment of the present application;

[0059] Figure 3 This is a flowchart for obtaining data fields based on associated data information and data processing logic provided by an embodiment of the present application;

[0060] Figure 4 This is a flowchart of an embodiment of the present application for performing process-based splicing of data sources, operators, and data sinks to obtain process configuration information;

[0061] Figure 5 This is a flowchart for reorganizing and expanding process configuration information provided by an embodiment of the present application;

[0062] Figure 6 This is a flowchart of determining whether there is an available template after obtaining business demand information provided by an embodiment of the present application;

[0063] Figure 7 This is a schematic diagram of a multi-dimensional data processing system based on stream-batch integration provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following is combined with Figure 1 -Attached Figure 5 , further details of this application are given.

[0065] The method provided in this application is applied in a credit card business system of the financial industry, which includes a data production module, a data processing center and a data consumption module.

[0066] The data production module is the source of data, such as transaction-related systems and card application systems of various channels, user information centers, etc.

[0067] The data processing center is used to process the data in the production data module, including the message processing center and the computing module. It can be specifically divided into marketing scenarios and transaction scenarios according to business scenarios. There will be corresponding data processing operations according to the relevant business scenarios.

[0068] The data consumption module is specifically used to put the data fed back by the data processing center into practical applications, including application consumption and business applications, such as marketing platforms, message push, big data processing, intelligent recommendations, etc.

[0069] By breaking down the relevant processes involved in the financial industry's business, relevant components can be formed. The components are divided into three major categories: data source, operator class, and data sink, which correspond to the data production module, data processing center, and data consumption module mentioned above.

[0070] Data sources, corresponding to the aforementioned consumer producers, are used for data access, integrating external data into real-time and batch data. As a container for data access, data sources support access to a variety of data sources, such as HDFS, Kafka message queues, JDBC, and HTTP sources. Different data sources have corresponding configuration properties based on their characteristics, such as data source type, address, account, and password.

[0071] Operator classes are used to process data and are divided into multiple subcategories, including aggregation classes, HTTP operation classes, machine learning classes, serialization operation classes, statistics classes, stream operation classes, data transformation classes, windows, and custom classes.

[0072] For example, the aggregation class performs window aggregation operations, similar to the GROUP BY in SQL statements. Specifically, it includes common functions, aggregate values, aggregate keys, watermarks, and business windows.

[0073] Data transformation class, which includes attribute association, conditional filters, etc.

[0074] Among them, attribute association: associate different transaction events through related field attributes, configure primary keys, field names, and field values, the field names will be paired with the field values, and the primary key is used to extract the content of the field name; conditional filter: filter data through conditional expressions, and data that meets the conditions will be retained.

[0075] The data sink supports multiple data access methods, such as HDFS, Kafka message queues, JDBC, and KV Store. The data sink has corresponding field names that are used to match the data output field names to better enable data flow within the constructed process configuration.

[0076] This application provides a multi-dimensional data processing method based on stream-batch integration. Figure 1 , including the following steps:

[0077] S100: Obtain business demand information, and parse the business demand information using a preset method to obtain data fields and data processing logic.

[0078] Business requirement information represents the business requirements involved in the credit card business system in the financial industry. Business requirement information is published in the form of a form and is mainly divided into three parts: the business scenario, data execution conditions, and data execution method.

[0079] For example, Business Requirement 1: Scenario: Real-time Marketing Scenario (Activation); Data Execution Condition: From the credit card perspective, the user activates their credit card for the first time; Data Execution Method: Real-time push of corresponding activity information. The entire business requirement indicates that when a user activates a credit card for the first time, the corresponding activity information will be pushed to the user, with the activity information accompanied by the corresponding activity number.

[0080] The default method represents a natural language processing algorithm and a semantic association rule model trained using historical business data. The natural language processing algorithm primarily performs keyword extraction based on business demand information, including operations such as word segmentation, word segmentation by part of speech, and function word removal.

[0081] The semantic association rule model represents the data processing method architecture associated with the business formed through the analysis of historical business procedures and big data. In addition to adding additional associated attributes to some commonly used keywords, it also models the semantic associations between different keywords. It can extract data associations between different keywords and obtain new association rule information by combining keywords.

[0082] In the embodiment of the present application, by obtaining business demand information and parsing the business demand information in a preset manner, automatic parsing is used to replace manual demand sorting, thereby increasing the efficiency of business data processing. By parsing the business demand information, the relevant data fields and related data processing logic can be obtained.

[0083] Specifically, the business requirement information is parsed using a preset method to obtain data fields and data processing logic. Figure 2 , specifically including the following steps:

[0084] S110. Extract keywords from the business demand information using a preset natural language processing model.

[0085] S120 : Generate associated data information based on the keywords through a preset data association rule model.

[0086] S130: Based on the data execution conditions, matching is performed through preset operator rules to obtain data processing logic.

[0087] S140: Acquire data fields based on associated data information and data processing logic.

[0088] The data field represents the name of the data involved in the business process, and the data processing logic represents the algorithm rules used in the data processing process.

[0089] By breaking down the processes involved in a business into components and storing all generated components in a component library, when faced with a new business, the required components can be obtained based on actual business needs. These components are then assembled into a process to obtain process configuration information. Finally, based on this process configuration information, the business can be quickly deployed and released. Therefore, the key lies in how to obtain the corresponding components based on actual business needs.

[0090] In an embodiment of the present application, keywords can be extracted from the business demand information through a preset natural language processing model.

[0091] For example, consider Business Requirement 1 above: Scenario: Real-time marketing (activation); Data execution condition: First-time credit card activation by a user; Data execution method: Real-time push of corresponding activity information. Natural language processing algorithms can be used to parse the business requirement information and extract relevant keywords, such as (user, first-time, activation, credit card, real-time push, activity information).

[0092] Then, based on keywords, a pre-set semantic association rule model can be used to retrieve associated data. For example, the keyword "user" is associated with attributes such as account number, password, user ID, and user behavior; the keyword "credit card" is associated with attributes such as card number, activation, cancellation, and query; and the keyword "activation" is associated with attributes such as "transaction code." Combining the keywords "activation" and "credit card" can yield association rules such as: a credit card can only be activated once, meaning that credit card activation is equivalent to first-time activation.

[0093] Furthermore, based on the data execution conditions, the preset operator rules can be used to match the corresponding data processing logic, namely the conditional filter (transaction code = activation, activation count = 1). Based on the associated data information obtained above, the "activation count = 1" condition can be discarded, resulting in the final data processing logic: conditional filter (transaction code = activation). The preset operator rules represent the classification and related rule descriptions generated for each operator component in the operator library.

[0094] Then, by associating data information and data processing logic, the data fields involved in the entire business process can be confirmed.

[0095] Specifically, based on the associated data information and data processing logic, obtain the data fields, see Figure 3 , specifically including the following steps:

[0096] S141. Based on the associated data information and data processing logic, confirm the data producer and data consumer.

[0097] S142. If there is more than one data producer, obtain data joint processing logic based on multiple data producers to serve as supplementary data processing logic.

[0098] S143. Add the supplementary data processing logic to the data processing logic to form a new data processing logic.

[0099] S144. Based on the associated data information and the new data processing logic, obtain the data field.

[0100] In an embodiment of the present application, firstly, all the database information involved can be determined by obtaining the associated data information, such as "user" corresponds to the user's information database, which records the user's personal identity information, account records, etc., and the key field is the user ID; "credit card" corresponds to the credit card information database, which stores information such as credit card usage records, and the key field is the credit card number; "activation" as a business form in the transaction system is usually expressed through a transaction code. In the transaction system, when the transaction code indicates activation, it means that the user activates the credit card for the first time; "real-time push activity message" as a data execution method is related to the decision system message queue.

[0101] Because transaction codes are the required data for data processing logic and are generated by the trading system, the source of business demand data, or the data producer, is the trading system. Because data execution is related to the decision-making system's message queue, the output of the data, or the data consumer, is the decision-making system.

[0102] Since the data producer only has a transaction system, the business data processing process can be completed according to the current data processing logic. That is, it is only necessary to perform conditional screening on the transaction code in the transaction system to determine whether the transaction code indicates activation.

[0103] This confirms the entire business requirement, and the data fields involved are (user ID, credit card number, transaction code).

[0104] However, if there is more than one data producer, the correlation between different data also needs to be considered. For example, business requirement 2: the scenario belongs to the real-time marketing scenario (lottery); the data execution condition is based on the date and the card number dimension, and the cumulative daily consumption amount reaches the preset value (such as >= 1000); the data execution method is to grant users lottery qualifications and promote activity messages.

[0105] Similarly, by applying a natural language parsing algorithm to business demand information, corresponding keywords can be extracted, such as (cumulative, current day, consumption amount, reached, user, lottery, push activity message). Then, through a semantic association rule model, further analysis can be performed to determine the associated data information and data processing logic. The associated data information can be used to confirm the database information involved.

[0106] To calculate the cumulative consumption amount, in addition to obtaining real-time consumption data from the transaction system, historical consumption data within a set timeframe is also required. Therefore, historical user data is required as support. This means connecting to local data sources to match and store user tag data. Therefore, the final confirmed data producers are the transaction system and the local data source that stores customer tag data.

[0107] Since there are two data producers and their data is related, a data union is needed. This is similar to database union, connecting the two databases and adding attribute associations. For example, the user ID in the transaction system equals the user ID in the local data source. This allows data accumulation calculations to be performed based on transaction data in the transaction system and the user's historical consumption data, implementing logical judgments during data processing. This adds additional data processing logic, namely union and attribute association, which is recorded as supplementary data processing logic.

[0108] Therefore, when there is more than one data producer, the associated data fields of the data producers are also extracted and used as the final data field. Therefore, for the entire business requirement 2, the final data fields obtained are (transaction code, transaction amount, transaction date, user ID, credit card number).

[0109] S200: Extracting a data source and a data sink from a preset component library based on the data field.

[0110] The pre-set component library stores components involved in various business processes and consists of three parts: data sources, data sinks, and operators. Each data source and data sink has a corresponding data access method, data reception method, or data storage method, effectively adding a usage tag and linking historical usage records. Operators have an operator library that stores the method logic used in various data processing processes, categorized by type, and generates corresponding algorithm rules or expressions for each operator.

[0111] In the embodiments of this application, by parsing the business requirements, the corresponding data fields are obtained, that is, the data required for the entire business requirement. From this, the data source and data sink can be determined. In other words, the required data and where the data is used are known. However, the problem to be solved is how to connect the data, that is, to build a data transmission bridge.

[0112] Data can be broadly divided into two categories: historical stock data and real-time streaming data. Historical stock data is business-related historical data, which is extracted through scheduling. Real-time streaming data is data generated by the business in real time and imported into the message queue. It is then retrieved from the message queue during analysis and calculation. Different data sources are configured for different data types.

[0113] Therefore, in the embodiment of the present application, the data type corresponding to the business is used to help select the required data source and data sink.

[0114] Specifically, based on the data field, extracting the data source and data sink from the preset component library includes the following steps:

[0115] S210: Determine the data types involved in the business based on the data producers and data consumers.

[0116] S220: Acquire data transmission channel information based on the data field.

[0117] S230: Match a corresponding data source and data sink from a preset component library according to the data transmission channel information and the data type.

[0118] Therefore, in the embodiment of the present application, when extracting data sources and data sinks from a preset component library based on a data field, the data type is first determined based on the data producer and data consumer associated with the data field. Based on the data type, the corresponding data source and data sink set is then obtained. Furthermore, based on the data information corresponding to the data field, the data access and reception format or transmission channel information is obtained. Then, based on the data access and reception format or transmission channel information, the desired data source and data sink are matched from the data source and data sink set.

[0119] For example, in business requirement 1 of the above example, the data producer is a real-time transaction system, the data consumer is a decision-making system, and the data type involved is real-time streaming data. The final matched data source is: Kafka real-time transaction streaming data, and the data sink is: Kafka sent to the message queue.

[0120] In business requirement 2 of the above example, the data producers are the real-time transaction system and user tag data stored in the local data source. The data consumer is the decision-making system. The data types involved include historical inventory data and real-time streaming data. The final matched data sources are: Kafka real-time transaction streaming data, Redis cumulative consumption amount cache data, and the data sink is: Kafka sends to the message queue.

[0121] S300: Based on the data processing logic, the corresponding operator is matched from the preset component library.

[0122] S400: Process-based splicing of data sources, operators, and data sinks to obtain process configuration information.

[0123] In the embodiment of the present application, after obtaining the data processing logic, the corresponding operator will be matched from the preset operator library according to the data processing logic. First, the data processing logic category is confirmed according to the data processing logic, and then targeted matching is performed to match the required operator.

[0124] After acquiring the data source, data sink, and operators, they are assembled in a process-based manner to form process configuration information. This process-based assembly is equivalent to building a complete data processing system, from data input to intermediate data processing steps to data output. In other words, the connection direction of each component is fixed: from the data source to the operator, from the operator to the data sink, or directly from the data source to the data sink.

[0125] In addition, the data source includes attribute configuration items, and the data sink includes configuration field names. After the components are spliced ​​together, corresponding data configuration will be performed to realize the flow of data in the entire process framework.

[0126] Specifically, the data source, operator and data sink are spliced ​​in a process-based manner to obtain process configuration information. Figure 4 , specifically including the following steps:

[0127] S410. Construct a data parameter field based on the attribute configuration item.

[0128] S420: Obtain data output information according to the data parameter field and the operator.

[0129] S430: Match the data output information with the configuration field name using a preset method to obtain a matching result.

[0130] S440: Determine whether the matching result reaches a preset threshold.

[0131] S450: If yes, then connect the process in a preset manner according to the current data source, operator, and data sink, and add corresponding attribute configuration to obtain process configuration information.

[0132] S460: If not, obtain the name of the field that failed to match successfully and enter a first prompt message, where the first prompt message indicates that the process configuration failed and corresponding adjustments are required.

[0133] The attribute configuration items represent the parameters required for data source access. Examples include address data, account passwords, and so on. Only after configuration is complete can data be accessed from the data source. The configuration field names represent the matching parameters for the corresponding received data in the data sink. Specifically, the data from the data source to the output data after operator processing must meet the requirements of the data sink, meaning that the output data must match the configuration field names.

[0134] First, data parameter fields are constructed based on the attribute configuration items of the data source to represent the data information that needs to be filled. Then, based on the data source and operator, the corresponding output data is obtained. The output data is matched with the configuration field name corresponding to the data sink to obtain the matching result.

[0135] Because there may be some uncertainty when matching field parameter names, or more specifically, data types, a corresponding threshold may be set as a reference standard. That is, if the matching result reaches the set threshold, the match is considered successful, and the process splicing is completed according to the current connection method, and the process configuration information is obtained.

[0136] If the matching result does not reach the set threshold, the match is considered to have failed. In this case, the name of the field that failed to match will be obtained and saved in the log, and the first prompt message will be entered to indicate that the process configuration failed and that corresponding adjustments need to be made.

[0137] After obtaining the process configuration information, corresponding data verification is required to confirm that there will be no abnormalities in the data flow process. Only in this way can the obtained process configuration information be published and deployed according to business needs.

[0138] Therefore, in the embodiment of the present application, after obtaining the process configuration information, the following steps are also included:

[0139] S510: Perform data verification on the process configuration information using a preset verification method, and obtain the data verification result.

[0140] S520: Determine whether the verification result contains abnormal information.

[0141] S530: If yes, obtain abnormal information and output second prompt information, where the second prompt information indicates that data verification has failed and corresponding adjustments are required.

[0142] S540: If not, save the process configuration information as a process configuration template corresponding to the scenario, and store the process configuration template in a preset process configuration management library.

[0143] Data verification of process configuration information primarily verifies that the data flows normally throughout the entire process. This means detecting data anomalies, such as data configuration errors or ineffective data conversion, which can lead to missing or erroneous data. By monitoring data at embedded points, data anomalies can be captured and stored in the verification results.

[0144] If abnormal information appears in the verification result, it means that the verification fails. Therefore, the abnormal information will be stored in the log and a second prompt message will be output to prompt that the data verification has failed and corresponding adjustments need to be made.

[0145] If abnormal information appears in the verification result, it means that the verification has passed. Therefore, the current process configuration information will be saved as a process configuration template corresponding to the scenario, and the process configuration template will be stored in the preset process configuration management library, so that if there are business needs for similar application scenarios, the data processing framework can be directly built according to the existing process configuration template.

[0146] Since the business scenarios to which business requirements belong cover a wide range, the business requirements of the same business scenario may show great differences. In this case, it is difficult to use the process configuration template of one party for the other party to use. Even if it can be used, there may be many places that need to be modified and adjusted, which will not achieve the purpose of reusing process templates and greatly reduce the time required to build a process framework.

[0147] Therefore, in the embodiment of the present application, after the process configuration template is stored in the preset process configuration management library, see Figure 5 , further comprising the following steps:

[0148] S610: Use the business requirement information corresponding to the process configuration template as a subordinate tag of the corresponding scenario.

[0149] S620: Associate and store the process configuration template with the corresponding business requirement information.

[0150] S630 : Based on the business requirement information corresponding to the process configuration template, reorganize in a preset manner to obtain reorganized business requirement information.

[0151] S640: Analyze the reorganized business information using a preset method and obtain a corresponding process configuration template.

[0152] S650: Associate and store the obtained process configuration template with the corresponding reorganization business requirement information.

[0153] In an embodiment of the present application, in order to better reuse the constructed process configuration template, the scenario will be regarded as a large category, and the newly generated process configuration template will be associated with the corresponding business needs and stored under the corresponding business scenario category.

[0154] In addition, considering that there are relatively few target process configuration templates in a certain scenario, or the associated business requirements are not very conventional, the possibility of being matched and used by new business requirements is relatively low. Therefore, after the process configuration template is stored in the preset process configuration management library, it will be expanded accordingly, that is, the business requirement information corresponding to the process configuration template will be reorganized accordingly, such as changing the relevant execution conditions, or adding new dimensions, etc. For example, the data execution condition in the business requirement information is: the first activation of a credit card. By changing the data execution condition, the changed data execution condition is: the first activation of a credit card and the first payment is completed. Then the reorganized business requirement information is obtained.

[0155] By reorganizing the business requirement information and following steps S100 to S400, the corresponding process configuration information can be obtained. Finally, the obtained process configuration template is associated and stored with the corresponding reorganized business requirement information. By reorganizing the business requirements and obtaining the corresponding process configuration template, the number of templates that can be matched to the business scenario can be expanded. This facilitates obtaining highly adaptable process configuration templates based on new business requirements, thereby accelerating the deployment of business data processing.

[0156] Since each time a new process configuration template is obtained, it will be associated with the corresponding business requirement information and stored in the preset process configuration management library, so after obtaining new business requirements, you can first determine whether there is a usable process configuration template. If there is, you can use it directly. If not, then select components in a preset way to obtain the corresponding process configuration information.

[0157] Specifically, after obtaining business requirement information, see Figure 6 , further comprising the following steps:

[0158] S710. Confirm the business scenario based on the business scenario.

[0159] S720: Determine whether there is a process configuration template for the corresponding scenario in the preset process configuration management library.

[0160] S730: If a process configuration template exists, the current business requirement information is matched with the business requirement information corresponding to the existing process configuration template to determine whether the highest similarity reaches a preset threshold.

[0161] S740: If the highest similarity reaches a preset threshold, the process configuration template corresponding to the business requirement information with the highest similarity is used as the target process configuration template.

[0162] S750. Obtain corresponding process configuration information according to the target process configuration template.

[0163] S760: If the process configuration template does not exist or the highest similarity does not reach the preset threshold, the business requirement information is parsed using a preset method.

[0164] In an embodiment of the present application, after obtaining the business demand information, it will be determined whether a process configuration template already exists in the corresponding scenario based on the business demand information. If not, the business demand will be analyzed step by step and the required components will be selected to obtain the process configuration information.

[0165] If a process configuration template already exists in the scenario, the business requirement information will be matched with the business requirement information corresponding to the existing process configuration template to obtain the corresponding similarity results. If the highest similarity fails to reach the preset threshold, the business requirement information will also be parsed using the preset method to select components and obtain the corresponding process configuration information.

[0166] If the highest similarity reaches the preset level, the process configuration template corresponding to the business requirement information with the highest similarity will be used as the target process configuration template, and then the process configuration information will be obtained based on the target process configuration template to perform corresponding data processing.

[0167] The present application also provides a multi-dimensional data processing system based on stream-batch integration. Figure 7 The system includes: an information acquisition module 101, a data component acquisition module 102, an operator component acquisition module 103, and a process configuration module 104.

[0168] The information acquisition module 101 is used to acquire business demand information, and parse the business demand information using a preset method to acquire data fields and data processing logic.

[0169] The data component acquisition module 102 is used to extract a data source and a data sink from a preset component library based on a data field.

[0170] The operator component acquisition module 103 is used to match the corresponding operator from the preset component library based on the data processing logic.

[0171] The process configuration module 104 is used to process the data source, operator and data sink and obtain process configuration information.

[0172] In an embodiment of the present application, the information acquisition module 101 is specifically used to obtain business demand information, and parse the business demand information through a preset method to obtain information such as keywords and data association rules, and then confirm the required key data fields and the algorithm logic used in the data processing process.

[0173] The data component acquisition module 102 is specifically used to select corresponding data sources and data sinks from the preset component library according to the data fields acquired by the information acquisition module 101 to determine the data source and data output of the data processing framework.

[0174] The operator component acquisition module 103 is specifically used to match the corresponding operator from the preset component library according to the data processing logic obtained by the information acquisition module 101, so as to determine what operation needs to be performed on the data during the data processing process.

[0175] The process configuration module 104 is specifically used to process the data source and data destination obtained by the data component acquisition module 102 and the operators obtained by the operator component acquisition module 103 to obtain process configuration information, which is equivalent to sorting out the data input, data output and intermediate data processing processes to form a complete data processing model.

[0176] An embodiment of the present application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and execute the above-mentioned multi-dimensional data processing method based on stream-batch integration.

[0177] The embodiments of this specific implementation method are all preferred embodiments of the present application and are not intended to limit the scope of protection of the present application. Therefore, all equivalent changes made based on the principles of the present application should be included in the scope of protection of the present application.

Claims

1. A multi-dimensional data processing method based on stream-batch integration, applied in the credit card business of the financial industry, characterized by: include: Obtain business demand information and parse it using preset methods to obtain data fields and data processing logic; Based on the data fields, extract the data source and data sink from the preset component library; Based on the data processing logic, the corresponding operator is matched from the preset component library; Perform process-based splicing of data sources, operators, and data sinks to obtain process configuration information; The business requirement information also includes data execution conditions. The business requirement information is parsed using a preset method to obtain data fields and data processing logic, including: For business demand information, keywords are extracted through a preset natural language processing model; Based on keywords, generate associated data information through the preset data association rule model; Based on the data execution conditions, matching is performed through preset operator rules to obtain data processing logic; Get data fields based on associated data information and data processing logic.

2. A multidimensional data processing method based on stream-batch integration according to claim 1, characterized in that: The obtaining of data fields based on the associated data information and data processing logic includes: Identify data producers and data consumers based on linked data information and data processing logic; If there is more than one data producer, then data joint processing logic is obtained based on multiple data producers to serve as supplementary data processing logic; adding the supplementary data processing logic to the data processing logic to form a new data processing logic; Get data fields based on associated data information and new data processing logic.

3. The multidimensional data processing method based on stream-batch integration according to claim 2 is characterized in that: The step of extracting a data source and a data sink from a preset component library based on the data field includes: Determine the data types involved in the business based on data producers and data consumers; Based on the data field, obtain the data transmission channel information; According to the data transmission channel information and data type, the corresponding data source and data sink are matched from the preset component library.

4. The multidimensional data processing method based on stream-batch integration according to claim 1 is characterized in that: The data source includes attribute configuration items, and the data sink includes configuration field names. The process of splicing the data source, operator, and data sink in a process-based manner to obtain process configuration information includes: Build data parameter fields based on attribute configuration items; Obtain data output information based on data parameter fields and operators; Match the data output information with the configured field name using a preset method to obtain the matching result; Determine whether the matching result reaches the preset threshold, If so, the process is connected in the preset manner based on the current data source, operator, and data sink, and the corresponding attribute configuration is added to obtain the process configuration information; If not, the name of the field that failed to match is obtained, and a first prompt message is input, where the first prompt message indicates that the process configuration has failed and corresponding adjustments need to be made.

5. The multidimensional data processing method based on stream-batch integration according to claim 1 is characterized in that: The business requirement information includes the business scenario to which it belongs. After obtaining the process configuration information, the following is also included: Verify the process configuration information using a preset verification method and obtain the data verification results; Determine whether the verification result contains abnormal information. If yes, obtain the abnormal information and output the second prompt information, which indicates that the data verification has failed and needs to be adjusted accordingly; If not, the process configuration information is saved as a process configuration template corresponding to the corresponding scenario, and the process configuration template is stored in a preset process configuration management library.

6. The multidimensional data processing method based on stream-batch integration according to claim 5 is characterized in that: After the process configuration template is stored in the preset process configuration management library, it also includes: The business requirement information corresponding to the process configuration template is used as a subordinate tag of the corresponding scenario; Associate and store the process configuration template with the corresponding business requirement information; Based on the business requirement information corresponding to the process configuration template, reorganize in a preset manner to obtain the reorganized business requirement information; According to the reorganized business information, it is parsed through the preset method and the corresponding process configuration template is obtained; The obtained process configuration template is associated with the corresponding reorganization business requirement information and stored.

7. The multidimensional data processing method based on stream-batch integration according to claim 6 is characterized in that: After obtaining the business demand information, the following steps are also included: Based on the business scenario, confirm the business scenario to which it belongs; Determine whether there is a process configuration template for the corresponding scenario in the preset process configuration management library; If a process configuration template exists, the current business requirement information is matched with the business requirement information corresponding to the existing process configuration template to determine whether the highest similarity reaches a preset threshold; If the highest similarity reaches the preset threshold, the process configuration template corresponding to the business requirement information with the highest similarity is used as the target process configuration template; According to the target process configuration template, obtain the corresponding process configuration information; If there is no process configuration template or the highest similarity does not reach the preset threshold, the business requirement information is parsed using a preset method.

8. A multi-dimensional data processing system based on stream-batch integration, characterized in that: include: An information acquisition module (101) is used to acquire business demand information and parse the business demand information using a preset method to obtain data fields and data processing logic; A data component acquisition module (102) is used to extract a data source and a data sink from a preset component library based on a data field; An operator component acquisition module (103) is used to match corresponding operators from a preset component library based on data processing logic; The process configuration module (104) is used to process the data source, operator and data sink and obtain process configuration information; The business requirement information also includes data execution conditions. The business requirement information is parsed using a preset method to obtain data fields and data processing logic, including: For business demand information, keywords are extracted through a preset natural language processing model; Based on keywords, generate associated data information through the preset data association rule model; Based on the data execution conditions, matching is performed through preset operator rules to obtain data processing logic; Get data fields based on associated data information and data processing logic.

9. A computer-readable storage medium storing a computer program capable of being loaded by a processor and executing a multi-dimensional data processing method based on stream-batch integration as claimed in any one of claims 1 to 7.

Images