Processing method and system supporting multi-source heterogeneous data

Abstract

The invention relates to the technical field of data processing, in particular to a processing method and system supporting multi-source heterogeneous data, and the method comprises the steps: collecting data from different target systems, and adding an identification tag corresponding to the target system for each piece of data; a plurality of real-time assembly lines parallel to the ETL path are constructed to process the collected data, analysis failure data are written into an error data area, and warning information is generated; the collected data are written into a temporary storage area after analysis failure data is removed; writing the output of the real-time data processing system and / or the output of the ETL path into a data warehouse; and creating a visual dynamic report by using the data in the data warehouse, analyzing and predicting the dynamic report data through a pre-trained machine learning model, and writing the prediction result into the dynamic report to predict the inventory demand and identify the supplier risk.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a processing method and system supporting multi-source heterogeneous data. Background Art

[0002] Heterogeneous data refers to a collection of data that differs in structure, format, type or storage method. This type of data usually comes from different data sources or systems and has a variety of organizational forms.

[0003] In the SPD business of the medical system, generating reports covering core data such as products, procurement, and inventory requires integrating data from different systems and structures through a multi-source heterogeneous data processing system. Existing systems usually use ETL (Extract-Transform-Load) and data warehouse technology to process multi-source heterogeneous data, combined with serialization frameworks and metadata management. However, traditional ETL tools rely on batch processing and have difficulty coping with real-time data streams; when integrating data from multiple systems, the permission model is inconsistent, which can easily lead to sensitive information leakage. Summary of the invention

[0004] The present invention combines the open source stream processing platform Kafka and the stream data processing engine Flink to build a real-time pipeline, and multiple real-time pipelines constitute a stream data processing system to improve the system's ability to process real-time data.

[0005] The technical solution proposed by the present invention is: a processing method supporting multi-source heterogeneous data, the method comprising: Collect data from different target systems and add identification tags of the corresponding target systems to each piece of data; Build multiple real-time pipelines in parallel with the ETL path to process the collected data, write the parsing failure data into the error data area, and generate warning information; remove the parsing failure data from the collected data and write it into the temporary storage area; Write the output of the real-time data processing system and / or the output of the ETL path into the data warehouse; Use the data in the data warehouse to create visual dynamic reports, analyze and predict the dynamic report data through pre-trained machine learning models, and write the prediction results into the dynamic report.

[0006] Preferably, the data that are successfully parsed constitute a correct data set, and the data that are unparsed constitute an error data set; according to the data structure type, a clustering algorithm is used to divide the data in the correct data set into multiple sub-data sets; Before acquiring data from different target systems, the steps also include: defining a global strategy to unify the permissions of each target system; Defining a global strategy includes: Register the target system in the console of the security management framework Ranger, and import the resource hierarchy of the target system, wherein the resource hierarchy includes tables and column families; Select the corresponding resources according to the resource level of each target system; specify the operations corresponding to the resources, including allowing operations and denying operations; The unified permissions of each target system include: Map the resources of each target system to the unified identifier of Ranger; convert the native operations of each target system to the standard operations of Ranger; Map the permission models of each target system into a unified "resource-operation-condition" model; Create a global role and associate the global role with a corresponding model to bind resources of a corresponding target system; the conditions include an IP range and a valid duration of the operation; The mapping of the permission models of each target system into a unified "resource-operation-condition" model includes: Use NTP protocol to synchronize the clock of Ranger and the target system; Set up extension plug-ins for the permission model of each target system, and use the polling of the extension plug-ins to implement the change of the permission model; Dynamically adjust the polling interval of the extension plug-in to ensure that the permission model of the target system can be mapped in time.

[0007] Preferably, the dynamically adjusting the polling interval of the extension plug-in to ensure that the permission model of the target system can be mapped in time includes: Record the start time of each model mapping; after the permission model is mapped, return the mapping completion time; The new polling interval is dynamically calculated through the sliding window weighted average delay and proportional integral control algorithm; Monitor the delay and replace the current polling interval with the new polling interval to adjust the polling interval to ensure that the permission model of the target system can be mapped in time.

[0008] Preferably, the dynamically calculating the new polling interval by using a sliding window weighted average delay and a proportional integral control algorithm comprises: After each permission model mapping is completed, calculate the time difference ;in Indicates the time when the mapping is completed. Indicates the start time of model mapping; Calculate weighted average delay ,in, represents the size of the sliding window, shows the weight coefficient; Indicates the current time The previous The time delay of a time point; Calculate control variables ;in, represents the target latency threshold, represents the integral gain of one, represents proportional gain of one; Indicates The average delay of a sliding window; Calculate new polling interval ;in, Indicates the current polling interval; Indicates the maximum polling interval. Maximum polling interval; if , replace the current polling interval with , and skip the calculation of the control variables.

[0009] Preferably, the defining of the global strategy further comprises: Create business semantic tags; Associating business semantic tags with resources in different target systems; Identify global role information and assign different business semantic tags to different global roles. Global roles access corresponding resources through corresponding business semantic tags.

[0010] Preferably, multiple real-time pipelines in parallel with the ETL path are constructed to process the collected data, including: Combine the open source stream processing platform Kafka and the stream data processing engine Flink to build a real-time pipeline. Multiple real-time pipelines constitute a stream data processing system to process multiple data streams. Get different types of data from different sub-datasets and convert the data into a format supported by Kafka. If the target system only supports batch file transfer, set up a batch file transfer plug-in in Kafka to pull data from the temporary storage area. After cleaning the acquired data set, data is collected in real time through a sliding time window to form a data stream; Write the data stream into the preset target storage area and lock it. After Flink completes the state point detection, unlock the target storage area and output the data. Maintain the original ETL path of the target system; build a gradual switching strategy to control the process of switching from ETL to real-time pipeline, and gradually switch the ETL data processing tasks of the target system to the real-time pipeline.

[0011] Preferably, the stepwise switching strategy is constructed to control the process of switching from ETL to real-time pipeline, and gradually switch the ETL data processing tasks of the target system to the real-time pipeline, including: Quantify the risk indicators in the switching process; the risk indicators include the data difference rate before and after the switching, the real-time pipeline processing delay time, the Flink detection failure rate and the downstream report generation error rate; Write data to Kafka and the original ETL path at the same time, compare the output of the ETL path and the output of the real-time pipeline, and stop switching if the data difference rate between the two outputs is greater than the preset difference rate threshold; Switch non-core task data to the real-time pipeline, while core task data is still transmitted through the original ETL path; Detect the core task data transmission volume. If the core task data transmission volume is lower than the preset transmission volume threshold within the preset time period, the core task data will be switched to the real-time pipeline and the original ETL path will be maintained. Month, ; When the real-time pipeline processing delay time exceeds the preset delay time threshold, the real-time pipeline is cut off and the original ETL path is automatically selected to transmit data.

[0012] Preferably, the step of constructing a progressive switching strategy to control the process of switching from ETL to real-time pipeline, and gradually switching the ETL data processing tasks of the target system to the real-time pipeline, further includes: Get real-time data processing system output and ETL path output; Calculate the relative entropy of the real-time data processing system output and the ETL path output: ,in, Represents the probability distribution of key data in the output of real-time data processing system, Indicates the probability distribution of the same key data in the ETL output; represents the Kullback-Leibler divergence; The switching process is dynamically adjusted by the PID controller, including the following sub-steps: Assume that the proportion of data flowing into the real-time data processing system is: ; in, ; represents proportional gain two, represents the integral gain of two, represents the differential gain; Set sliding time window 2 to collect output data from real-time data processing system and ETL path in real time, and calculate and ; use and Calculate the relative entropy; calculate the change in the proportion of data flowing into the real-time data processing system after a sliding time window. ,in, is the length of sliding time window 2; Dynamic optimization through reinforcement learning algorithm PPO , , , including the following sub-steps: Setting up the state space , Among them, the integral term , the differential term ; Setting up the action space ; Among them, the adjustment amount of proportional gain 2 ; Adjustment amount of integral gain 2 , Differential gain adjustment ; Setting up the reward function ; in, represents the error weight, represents the penalty for control quantity change, represents overshoot penalty; Overshoot , Indicates at the current moment The proportion of data that flowed into the real-time data processing system at a previous point in time; The Actor-Critic framework is used to build a reinforcement learning algorithm, which uses the elements in the action space to update the elements in the state space. Updates; The use of elements in the action space to update elements in the state space is specifically: , , ;in, , , Represents the updated proportional gain 2, integral gain 2, and differential gain.

[0013] Preferably, the method of creating a visual dynamic report using data in the data warehouse, analyzing and predicting the dynamic report data through a pre-trained machine learning model, and writing the prediction results into the dynamic report includes: Obtain data from different target systems from the data warehouse and create dynamic reports using the BI tool Tableau; Load the machine learning model, input data from different target systems to generate corresponding prediction results, and write them into dynamic reports; specifically, it includes: Obtain data from the data warehouse and identify data of different target systems through identification tags of target systems; The data of different target systems are subjected to feature value extraction and then standardized to form a feature set of the target system; Input the elements in the target system feature set as input variables into the machine learning model and output the prediction results; The prediction results are written into the data warehouse, and the prediction results are read from the data warehouse using BI tools and written into dynamic reports, and the dynamic reports are updated; the machine learning model includes one or more of the time series prediction model LSTM, the binary classification model XGBoost and the linear regression model.

[0014] A processing system supporting multi-source heterogeneous data includes a server, a processor, a communication module connected to the processor, a data acquisition interface and a memory. The processor is connected to the server through the communication module. The system is used to execute the processing method supporting multi-source heterogeneous data.

[0015] Beneficial effects of the present invention: The present invention replaces the traditional ETL with a real-time pipeline, constructs a progressive switching strategy, controls the process of switching from ETL to the real-time pipeline, and gradually switches the ETL data processing tasks of the target system to the real-time pipeline, thereby improving the real-time performance of the system in processing multi-source heterogeneous data; uses Ranger to unify permissions, thereby solving the problem of inconsistent permissions of multiple target systems; and optimizes the delay of model mapping by dynamically adjusting the polling interval. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The present invention is a flowchart of a method for processing multi-source heterogeneous data. DETAILED DESCRIPTION

[0017] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

[0018] It is to be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" should not be understood as a limitation on the quantity.

[0019] Embodiment 1: refer to Figure 1 The technical solution provided by the present invention is: a processing method and system supporting multi-source heterogeneous data, comprising the following steps: Step 1: Collect data from different target systems and add identification tags of the corresponding target systems to each piece of data; before this step, it also includes the steps of defining global policies and unifying the permissions of various target systems.

[0020] The purpose of setting this step is that traditional ETL tools cannot process some real-time data (such as sensor data), which can easily lead to inventory report delays. In addition, when managing cross-system permissions, the inconsistent permission models of ERP, WMS and logistics systems increase data access risks.

[0021] Defining a global policy includes: Register the target system in the console of the security management framework Ranger, import the resource hierarchy of the target system, and the resource hierarchy includes tables and column families; select corresponding resources according to the resource hierarchy of each target system; and specify operations corresponding to the resources, including allowing operations and denying operations.

[0022] Among them, unifying the permissions of various target systems includes: Map the resources of each target system to the unified identifier of Ranger; convert the native operations of each target system to the standard operations of Ranger; map the permission model of each target system to a unified "resource-operation-condition" model; create a global role, associate the global role with the corresponding model to bind the resources of the corresponding target system; the conditions include the IP range and the effective duration of the operation.

[0023] Map the permission models of each target system into a unified "resource-operation-condition" model, including: Use NTP protocol to synchronize the clock of Ranger and the target system; Reduces clock synchronization error and ensures The calculation is accurate.

[0024] Set up extension plug-ins for the permission model of each target system, and use the polling of the extension plug-ins to implement the change of the permission model; Dynamically adjust the polling interval of the extension plug-in to ensure that the permission model of the target system can be mapped in time.

[0025] For example, when generating the SPD inventory turnover report, unified permissions make it easier to extract daily inventory snapshots from the WMS (MySQL) and inventory records from the logistics system (MongoDB); Then, calculate the average daily inventory: (initial inventory + final inventory) / 2, aggregate the monthly outbound volume by drug ID; write the results to the inventory_turnover table of the data warehouse (Snowflake), and configure a line chart in Tableau to display the turnover trend of each drug.

[0026] In this embodiment, dynamically adjusting the polling interval of the extension plug-in to ensure that the permission model of the target system can be mapped in time includes the following steps: Record the start time of each model mapping; after the permission model is mapped, return the mapping completion time; dynamically calculate the new polling interval through the sliding window weighted average delay and proportional integral control algorithm; monitor the delay and replace the current polling interval with the new polling interval to adjust the polling interval to ensure that the permission model of the target system can be mapped in time.

[0027] Among them, the new polling interval is dynamically calculated through the sliding window weighted average delay and proportional integral control algorithm, including: After each permission model mapping is completed, calculate the time difference ;in Indicates the time when the mapping is completed. Indicates the start time of model mapping; Calculate weighted average delay ,in, represents the size of the sliding window, represents the weight coefficient; Indicates the current time The previous The time delay of a time point; Calculate control variables ;in, represents the target latency threshold, represents the integral gain, represents proportional gain; Indicates The average delay of a sliding window; Calculate new polling interval ;in, Indicates the current polling interval; Indicates the maximum polling interval. Maximum polling interval; if , replace the current polling interval with , and skip the calculation of the control variables.

[0028] For example, when Second, Second, Second, , .

[0029] If the delays of three consecutive handovers are 15 seconds, 18 seconds, and 20 seconds, the weighted average delay Seconds, control variables Second; Second.

[0030] If the switching delay is adjusted to 8 seconds, then Second, Second.

[0031] Step 2: Build multiple real-time pipelines in parallel with the ETL path to process the collected data, write the data that failed to be parsed into the error data area, and generate a warning message; remove the collected data that failed to be parsed and write it into the temporary storage area, and the data that successfully parsed will form the correct data set, and the data that failed to be parsed will form the error data set; according to the data structure type, use the clustering algorithm to divide the data in the correct data set into multiple sub-data sets to understand the number of each structure data.

[0032] Step 2 includes the following sub-steps: Combine the open source stream processing platform Kafka and the stream data processing engine Flink to build a real-time pipeline. Multiple real-time pipelines constitute a stream data processing system to process multiple data streams. Get different types of data from different sub-datasets and convert the data into a format supported by Kafka. If the target system only supports batch file transfer, set up a batch file transfer plug-in in Kafka to pull data from the temporary storage area. After cleaning the acquired data set, data is collected in real time through a sliding time window to form a data stream; Write the data stream into the preset target storage area and lock it. After Flink completes the state point detection, unlock the target storage area and output the data. Maintain the original ETL path of the target system; Build a gradual switching strategy to control the process of switching from ETL to real-time pipeline, and gradually switch the ETL data processing tasks of the target system to the real-time pipeline. Specifically: Quantify the risk indicators in the switching process; the risk indicators include the data difference rate before and after the switching, the real-time pipeline processing delay time, the Flink detection failure rate and the downstream report generation error rate.

[0033] In this embodiment, if the difference rate between the output results of the original system and the real-time pipeline system is greater than 1%, it is judged to be high risk when evaluating data consistency; if the Flink event time (real-time pipeline processing delay time) is greater than 5 minutes, it is considered to be high risk when evaluating system timeliness.

[0034] Write data to Kafka and the original ETL path at the same time, compare the output of the ETL path and the output of the real-time pipeline, and stop switching if the data difference rate of the two outputs is greater than the preset difference rate threshold.

[0035] Switch non-core task data to the real-time pipeline, while core task data is still transmitted through the original ETL path.

[0036] Detect the core task data transmission volume. If the core task data transmission volume is lower than the preset transmission volume threshold within the preset time period, the core task data will be switched to the real-time pipeline and the original ETL path will be maintained. Month, .

[0037] When the real-time pipeline processing delay time exceeds the preset delay time threshold, the real-time pipeline is cut off and the original ETL path is automatically selected to transmit data.

[0038] For example, if the real-time stream delay exceeds 10 minutes, the real-time pipeline is automatically disconnected and switched to ETL until the delay is recovered. The above functions can be implemented by embedding the fault-tolerant library Resilience4j or the service fuse degradation component Hystrix into the Flink job end.

[0039] The above method of building a real-time pipeline through Kafka and Flink can improve the timeliness of medical SPD services. At the same time, the use of dual-line transmission and progressive switching can reduce the risk of switching.

[0040] Step 3: Write the output of the real-time data processing system and / or the output of the ETL path into the data warehouse; Step 4: Use the data in the data warehouse to create a visual dynamic report, analyze and predict the dynamic report data through the pre-trained machine learning model, and write the prediction results into the dynamic report. It includes the following sub-steps: Obtain data from different target systems from the data warehouse and create dynamic reports using the BI tool Tableau; Load the machine learning model, input the data of different target systems to generate corresponding prediction results, and write them into the dynamic report; the machine learning model includes one or more of the time series prediction model LSTM, the binary classification model XGBoost and the linear regression model; specifically includes: Obtain data from the data warehouse and identify data of different target systems through identification tags of target systems; The data of different target systems are subjected to feature value extraction and then standardized to form the target system feature set; for example, the average drug purchase volume and inventory standard deviation over the past 7 days can constitute a time series feature set; the supplier's historical on-time delivery rate can constitute a risk classification feature set for the procurement and supply system, etc.

[0041] The elements in the target system feature set are input as input variables into the machine learning model to output the prediction results. For example, the LSTM model can output the inventory demand in a certain period of time in the future, which can be used to generate an inventory demand report. The binary classification model can output the supplier's default probability and add default warning data to the SPD report.

[0042] Write the forecast results into the data warehouse, use BI tools to read the forecast results from the data warehouse and write them into dynamic reports, and update the dynamic reports.

[0043] Traditional reports rely on static rules and cannot adapt to dynamic business changes (such as fluctuations in drug demand caused by emergencies). This embodiment uses the LSTM / XGBoost model to predict inventory demand and supplier risks, reduce reliance on manual experience, and improve prediction accuracy.

[0044] Embodiment 2: The difference between this embodiment and the first embodiment is that in the first embodiment, the global policy defined by Ranger is an explicit authorization method based on resources, while in this embodiment, an implicit authorization based on tags is provided.

[0045] The specific steps include: Create business semantic tags; associate business semantic tags with resources in different target systems; Identify global role information and assign different business semantic tags to different global roles. Global roles access corresponding resources through corresponding business semantic tags.

[0046] In this embodiment, by setting the business semantic tag to "query business tag", the tag allowed operation is set to "read" operation, the global role is set to "statistician", and the business semantic tag is associated with the system (HDFS, Hive, Kafka, etc.), system resources can be accessed through the business tag.

[0047] Embodiment three:

[0048] The difference between this embodiment and the first embodiment is that in the first embodiment, during the switching process between ETL and real-time pipeline, the switching control is performed by comparing the data difference rate of the two outputs with the difference rate threshold. This difference rate threshold is fixed and needs to be set manually, and cannot adapt to the dynamically changing data processing process. For this reason, a dynamic control switching strategy is provided in this embodiment, that is, by constructing a dynamic feedback system, the migration traffic ratio is adaptively adjusted in real time according to the behavioral differences between the ETL and real-time pipeline systems. The specific steps are as follows: Get real-time pipeline output and ETL path output; Calculate the relative entropy of the real-time pipeline output and the ETL path output: ,in, represents the probability distribution of key data in the real-time pipeline output, Indicates the probability distribution of the same key data in the ETL output; represents the Kullback-Leibler divergence.

[0049] The switching process is dynamically adjusted by the PID controller, including the following sub-steps: Assume that the proportion of data flowing into the real-time data processing system is: ; in, ; represents proportional gain two, represents the integral gain of two, represents the differential gain; Indicates the sensitivity of the PID controller to the current difference, Indicates the ability of the PID controller to correct long-term deviations. It indicates the PID controller's judgment on the trend of the difference change. When the difference suddenly increases, it reduces The value of .

[0050] Set sliding time window 2 to collect output data from real-time pipelines and ETL paths in real time and calculate and ; use and Calculate the relative entropy; calculate the change in the proportion of data flowing into the real-time pipeline after a sliding time window of 2 ,in, is the length of sliding time window 2; Dynamic optimization through reinforcement learning algorithm PPO , , , including the following sub-steps: Setting up the state space , Among them, the integral term , the differential term ; Setting up the action space ; Among them, the adjustment amount of proportional gain 2 ; Adjustment amount of integral gain 2 , Differential gain adjustment ; Setting up the reward function ; in, represents the error weight, represents the penalty for control quantity change, represents overshoot penalty; Overshoot ; Indicates at the current moment The proportion of data that flowed into the real-time data processing system at a previous point in time.

[0051] The Actor-Critic framework is used to build a reinforcement learning algorithm, which uses the elements in the action space to update the elements in the state space. Updates; The use of elements in the action space to update elements in the state space is specifically: , , ;in, , , Represents the updated proportional gain 2, integral gain 2, and differential gain.

[0052] For example, an implementation process of this embodiment may be: Set the initial flow rate to ,set up , , ; Start the PID controller every Calculate once , update once ; If within a period of time If the fluctuation amplitude is greater than the preset fluctuation amplitude threshold, or the fluctuation frequency exceeds the preset fluctuation frequency threshold, the reinforcement learning model is started to optimize , , ; when and , lasts for more than 24 hours, and the migration is judged to be complete, the switching control task is terminated.

[0053] The solution of this embodiment shortens the policy effectiveness delay and reduces the average delay through PID adaptiveness, can flexibly respond to sudden traffic or policy changes, optimizes the global policy effectiveness delay, and improves the switching efficiency.

[0054] The present invention also provides a processing system that supports multi-source heterogeneous data, including a server, a processor, and a communication module connected to the processor, a data acquisition interface and a memory, wherein the processor is connected to the server via the communication module, and the system is used to execute the processing method that supports multi-source heterogeneous data.

[0055] In the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. The embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above functions defined in the method of the present invention are executed. It should be noted that the above-mentioned computer-readable medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wire segments, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, device, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless segments, wire segments, optical cables, RF, etc., or any suitable combination of the above.

[0056] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0057] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and explained in the embodiments. Without departing from the principles, the implementation methods of the present invention may be subject to any changes or modifications.

Claims

1. A processing method supporting multi-source heterogeneous data, characterized in that: The method comprises: Collect data from different target systems and add identification tags of the corresponding target systems to each piece of data; Build multiple real-time pipelines in parallel with the ETL path to process the collected data, write the parsing failure data into the error data area, and generate warning information; remove the parsing failure data from the collected data and write it into the temporary storage area; Write the output of the real-time data processing system and / or the output of the ETL path into the data warehouse; Use the data in the data warehouse to create visual dynamic reports, analyze and predict the dynamic report data through pre-trained machine learning models, and write the prediction results into the dynamic report.

2. A processing method supporting multi-source heterogeneous data according to claim 1, characterized in that: The successfully parsed data constitutes a correct data set, and the unparsed data constitutes an incorrect data set; according to the data structure type, the clustering algorithm is used to divide the data in the correct data set into multiple sub-data sets; Before acquiring data from different target systems, the steps also include: defining a global strategy to unify the permissions of each target system; Defining a global strategy includes: Register the target system in the console of the security management framework Ranger, and import the resource hierarchy of the target system, wherein the resource hierarchy includes tables and column families; Select appropriate resources based on the resource level of each target system; Specify the operations corresponding to the resource, including allow operations and deny operations; The unified permissions of each target system include: Map the resources of each target system to the unified identifier of Ranger; Convert the native operations of each target system into standard operations of Ranger; Map the permission models of each target system into a unified "resource-operation-condition" model; Create a global role and associate the global role with the corresponding model to bind the resources of the corresponding target system; The conditions include the IP range and the validity period of the operation; The mapping of the permission models of each target system into a unified "resource-operation-condition" model includes: Use NTP protocol to synchronize the clock of Ranger and the target system; Set up extension plug-ins for the permission model of each target system, and use the polling of the extension plug-ins to implement the change of the permission model; Dynamically adjust the polling interval of the extension plug-in to ensure that the permission model of the target system can be mapped in time.

3. A processing method supporting multi-source heterogeneous data according to claim 2, characterized in that: The dynamically adjusting the polling interval of the extension plug-in to ensure that the permission model of the target system can be mapped in time includes: Record the start time of each model mapping; After the permission model completes the mapping, the mapping completion time is returned; The new polling interval is dynamically calculated through the sliding window weighted average delay and proportional integral control algorithm; Monitor the delay and replace the current polling interval with the new polling interval to adjust the polling interval to ensure that the permission model of the target system can be mapped in time.

4. A processing method supporting multi-source heterogeneous data according to claim 3, characterized in that: The new polling interval is dynamically calculated by using a sliding window weighted average delay and a proportional integral control algorithm, including: After each permission model mapping is completed, calculate the time difference ;in Indicates the mapping completion time, Indicates the start time of model mapping; Calculate weighted average delay ,in, represents the size of the sliding window, represents the weight coefficient; Indicates the current time The previous The time delay of a time point; Calculate control variables ;in, represents the target latency threshold, represents the integral gain of one, represents proportional gain of one; Indicates The average delay of a sliding window; Calculate new polling interval ;in, Indicates the current polling interval; Indicates the maximum polling interval. Maximum polling interval; if , replace the current polling interval with , and skip the calculation of the control variables.

5. A processing method supporting multi-source heterogeneous data according to claim 4, characterized in that: Defining a global strategy also includes: Create business semantic tags; Associating business semantic tags with resources in different target systems; Identify global role information and assign different business semantic tags to different global roles. Global roles access corresponding resources through corresponding business semantic tags.

6. A processing method supporting multi-source heterogeneous data according to claim 5, characterized in that: The construction of multiple real-time pipelines in parallel with the ETL path to process the collected data includes: Combine the open source stream processing platform Kafka and the stream data processing engine Flink to build a real-time pipeline. Multiple real-time pipelines constitute a stream data processing system to process multiple data streams. Get different types of data from different sub-datasets and convert the data into a format supported by Kafka. If the target system only supports batch file transfer, set up a batch file transfer plug-in in Kafka to pull data from the temporary storage area. After cleaning the acquired data set, data is collected in real time through a sliding time window to form a data stream; Write the data stream into the preset target storage area and lock it. After Flink completes the state point detection, unlock the target storage area and output the data. Maintain the original ETL path of the target system; Build a gradual switching strategy to control the process of switching from ETL to real-time pipeline, and gradually switch the ETL data processing tasks of the target system to the real-time pipeline.

7. A processing method supporting multi-source heterogeneous data according to claim 6, characterized in that: The step-by-step switching strategy is constructed to control the process of switching from ETL to real-time pipeline, and gradually switch the ETL data processing tasks of the target system to the real-time pipeline, including: Quantify the risk indicators in the switching process; the risk indicators include the data difference rate before and after the switching, the real-time pipeline processing delay time, the Flink detection failure rate and the downstream report generation error rate; Write data to Kafka and the original ETL path at the same time, compare the output of the ETL path and the output of the real-time pipeline, and stop switching if the data difference rate between the two outputs is greater than the preset difference rate threshold; Switch non-core task data to the real-time pipeline, while core task data is still transmitted through the original ETL path; Detect the core task data transmission volume. If the core task data transmission volume is lower than the preset transmission volume threshold within the preset time period, the core task data will be switched to the real-time pipeline and the original ETL path will be maintained. Month, ; When the real-time pipeline processing delay time exceeds the preset delay time threshold, the real-time pipeline is cut off and the original ETL path is automatically selected to transmit data.

8. A processing method supporting multi-source heterogeneous data according to claim 7, characterized in that: The step-by-step switching strategy is constructed to control the process of switching from ETL to real-time pipeline, and gradually switch the ETL data processing tasks of the target system to the real-time pipeline, and further includes: Get real-time pipeline output and ETL path output; Calculate the relative entropy of the real-time pipeline output and the ETL path output: ,in, represents the probability distribution of key data in the real-time pipeline output, Indicates the probability distribution of the same key data in the ETL output; represents the Kullback-Leibler divergence; The switching process is dynamically adjusted by the PID controller, including the following sub-steps: Assume that the proportion of data flowing into the real-time data processing system is: ； in, ; represents proportional gain two, represents the integral gain of two, represents the differential gain; Set sliding time window 2 to collect output data from real-time pipelines and ETL paths in real time and calculate ; use and Calculate the relative entropy; calculate the change in the proportion of data flowing into the real-time pipeline after a sliding time window of 2 ,in, is the length of sliding time window 2; Dynamic optimization through reinforcement learning algorithm PPO , , , including the following sub-steps: Setting up the state space , Among them, the integral term , the differential term ; Setting up the action space ; Among them, the adjustment amount of proportional gain 2 ; Adjustment amount of integral gain 2 , the differential gain adjustment amount ; Setting up the reward function ; in, represents the error weight, represents the penalty for control quantity change, represents overshoot penalty; Overshoot , Indicates at the current moment The proportion of data that flowed into the real-time data processing system at a previous point in time; The Actor-Critic framework is used to build a reinforcement learning algorithm, which uses the elements in the action space to update the elements in the state space. Updates; The use of elements in the action space to update elements in the state space is specifically as follows: , , ;in, , , Represents the updated proportional gain 2, integral gain 2, and differential gain.

9. A processing method supporting multi-source heterogeneous data according to claim 8, characterized in that: The method of creating a visual dynamic report using the data in the data warehouse, analyzing and predicting the dynamic report data through a pre-trained machine learning model, and writing the prediction results into the dynamic report includes: Obtain data from different target systems from the data warehouse and create dynamic reports using the BI tool Tableau; Load the machine learning model, input the data of different target systems to generate corresponding prediction results, and write them into the dynamic report; the machine learning model includes one or more of the time series prediction model LSTM, the binary classification model XGBoost and the linear regression model; specifically includes: Obtain data from the data warehouse and identify data of different target systems through identification tags of target systems; The data of different target systems are subjected to feature value extraction and then standardized to form a feature set of the target system; Input the elements in the target system feature set as input variables into the machine learning model and output the prediction results; Write the forecast results into the data warehouse, use BI tools to read the forecast results from the data warehouse and write them into dynamic reports, and update the dynamic reports.

10. A processing system supporting multi-source heterogeneous data, comprising a server, a processor, a plurality of data acquisition interfaces connected to the processor, a communication module and a memory, wherein the processor is connected to the server via the communication module, characterized in that: The system is used to execute a processing method supporting multi-source heterogeneous data as described in any one of claims 1 to 9.

Images