Data processing method and system based on ETL

Abstract

The invention provides an ETL-based data processing method and system, and the method comprises the following steps: constructing a first transmission queue between an extraction node and a conversion node, and constructing a second transmission queue between the conversion node and a loading node; acquiring system attributes of the first transmission queue, the second transmission queue and the data transmission system, and judging whether data extraction is performed or not according to the system attributes; if data extraction is carried out, the real-time extraction speed is obtained through the system attributes, the standby data packets are extracted from the heterogeneous systems through the extraction node based on the real-time extraction speed, and the standby data packets are sequentially transmitted to the conversion node and the loading node. By monitoring the data packet accumulation amount and the system occupation amount in real time, system imbalance caused by insufficient resources is avoided; according to the method, system attributes are taken as the basis of tradeoff, and the speeds of node extraction, node conversion and node loading are dynamically controlled and balanced, so that the efficiency and the stability of data processing are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to a data processing method and system based on ETL. Background Art

[0002] With the development of society and the advent of the digital age, a large amount of business data has been generated in all walks of life, and data has become a core production factor. There are usually a large number of heterogeneous systems such as CRM and ERP and various data sources among enterprise departments, making the data between systems relatively isolated and unable to be directly shared with each other, thus becoming data islands.

[0003] Through unified access by ETL technology, cross-system data connection can be realized, and the current situation of data islands between systems can be broken. However, ETL lacks control over dynamically adjusting the speed during the data processing process, resulting in an imbalance in the speeds of the extraction node, the transformation node, and the loading node, affecting the efficiency and stability of data processing. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the purpose of the present invention is to provide a data processing method and system based on ETL, aiming to solve the technical problem that in the prior art, ETL lacks control over dynamically adjusting the speed during the data processing process, resulting in an imbalance in the speeds of the extraction node, the transformation node, and the loading node, affecting the efficiency and stability of data processing.

[0005] To achieve the above purpose, in the first aspect, an embodiment of the present application provides a data processing method based on ETL, which is used to sequentially transfer data from an extraction node to a transformation node and a loading node. The extraction node, the transformation node, and the loading node constitute a data transmission system. The extraction node is connected to several heterogeneous systems. The data processing method based on ETL includes the following steps: Build a first transmission queue between the extraction node and the transformation node, and build a second transmission queue between the transformation node and the loading node; Obtain the system attributes of the first transmission queue, the second transmission queue, and the data transmission system, and determine whether to perform data extraction through the system attributes; If data extraction is to be performed, obtain the real-time extraction speed through the system attributes. Based on the real-time extraction speed, extract standby data packets from several heterogeneous systems through the extraction node, and sequentially transmit the standby data packets to the transformation node and the loading node.

[0006] Further, the system attributes include the first real-time packet quantity, the first real-time queue memory, the second real-time packet quantity, the second real-time queue memory, and the system memory. The step of determining whether to perform data extraction based on the system attributes includes: Comparing the first real-time packet quantity with the first maximum packet quantity, comparing the first real-time queue memory with the first maximum memory, comparing the second real-time packet quantity with the second maximum packet quantity, comparing the second real-time queue memory with the second maximum memory, and comparing the system memory with the maximum system memory; If the first real-time packet quantity is less than the first maximum packet quantity, and the first real-time queue memory is less than the first maximum memory, and the second real-time packet quantity is less than the second maximum packet quantity, and the second real-time queue memory is less than the second maximum memory, and the system memory is less than the maximum system memory, it is determined to perform data extraction.

[0007] Furthermore, the step of obtaining the real-time extraction speed based on the system attributes includes: Obtaining a preset extraction speed, obtaining a first adjustment parameter through the first real-time packet quantity and the first maximum packet quantity, obtaining a second adjustment parameter through the first real-time queue memory and the first maximum memory, obtaining a third adjustment parameter through the second real-time packet quantity and the second maximum packet quantity, obtaining a fourth adjustment parameter through the second real-time queue memory and the second maximum memory, and obtaining a fifth adjustment parameter through the system memory and the maximum system memory; Assigning parameter weights to the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter respectively to weight the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter into an adjustment coefficient; Comparing the adjustment coefficient with a coefficient threshold to obtain an adjustment amplitude, and adjusting the preset extraction speed to the real-time extraction speed based on the adjustment amplitude.

[0008] Furthermore, the formula for obtaining the first adjustment parameter is: , where, represents the first adjustment parameter, represents the first maximum packet quantity, represents the first real-time packet quantity; The formula for obtaining the adjustment coefficient is: , where, represents the adjustment coefficient, represents the second adjustment parameter, represents the third adjustment parameter, represents the fourth adjustment parameter, represents the fifth adjustment parameter, , , , , denotes parameter weights, and = , = , + + + + =1.

[0009] Furthermore, the formula for obtaining the real-time extraction speed is: , in, Indicates the real-time extraction speed, Indicates the preset extraction speed, represents the adjustment coefficient, Represents the coefficient threshold.

[0010] Furthermore, the step of extracting unused data packets from a plurality of the heterogeneous systems through the extraction node includes: Acquire a data extraction task, select a plurality of standby systems from the plurality of heterogeneous systems based on the data extraction task, and lock data sources in the standby systems to form a plurality of extraction paths; A plurality of standby data are acquired through the extraction nodes and the extraction paths, and the plurality of standby data and the plurality of extraction paths are combined into a standby data packet.

[0011] Furthermore, the step of sequentially transmitting the standby data packets to the conversion node and the loading node specifically includes: Determining whether the processing of the standby data packet at the conversion node and the loading node is successful; If unsuccessful, the abnormal situation is matched with the abnormal strategy, which includes the packet loss strategy and the processing strategy; If the abnormal situation matches the packet loss strategy, extracting the extraction path of the standby data packet, and re-extracting data based on the extraction path; If the abnormal situation matches the processing strategy, the standby data packet is transmitted to the screening unit.

[0012] Further, after the step of, if the abnormal situation matches the processing policy, transmitting the standby data packet to the screening unit, the method further includes: Performing error screening on the standby data packet to identify the error area; Correcting the error area to obtain an updated data packet, and re-depositing the updated data packet into the first transmission queue or the second transmission queue.

[0013] In a second aspect, an embodiment of the present application provides an ETL-based data processing system, which is applied to the ETL-based data processing method described in the first aspect above. The system includes: A construction module, configured to construct a first transmission queue between the extraction node and the transformation node, and construct a second transmission queue between the transformation node and the loading node; An execution module, configured to obtain the system attributes of the first transmission queue, the second transmission queue, and the data transmission system, and determine whether to perform data extraction based on the system attributes; An adjustment module, configured to, if data extraction is to be performed, obtain the real-time extraction speed through the system attributes, and based on the real-time extraction speed, extract standby data packets from several heterogeneous systems through the extraction node, and sequentially transmit the standby data packets to the transformation node and the loading node.

[0014] In a third aspect, an embodiment of the present application provides a computer, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the ETL-based data processing method described in the first aspect above is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the ETL-based data processing method described in the first aspect above is implemented.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: By constructing the first transmission queue and the second transmission queue, the transmission of the to-be-used data packets among the extraction node, the conversion node, and the loading node is realized; when performing data extraction, by considering the number and memory of the data packets that the two transmission queues can carry, and comprehensively considering the system memory, that is, real-time monitoring the data packet accumulation amount and system occupancy, and feeding back to the extraction node to avoid system imbalance caused by continuous data extraction when resources are insufficient; further, when the extraction of the to-be-used data packets can be performed, based on the system attributes as the basis for weighing, the preset extraction speed is dynamically controlled, reducing the extraction speed when the system occupancy rate is high and increasing the extraction speed when the system occupancy rate is low, that is, the balance among the speeds of the extraction node, the conversion node, and the loading node is realized, improving the efficiency and stability of data processing; by synchronizing the extraction path into the to-be-used data packets, if the packet loss strategy needs to be executed, indexing can be performed according to the extraction path without re-positioning and extracting the data again. If the processing strategy needs to be executed, the error repair of the to-be-used data packets can be directly performed and directly re-injected into the transmission queue without re-running the entire process, further improving the processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of the ETL-based data processing method in the first embodiment of the present invention; Figure 2 It is a structural block diagram of the ETL-based data processing system in the second embodiment of the present invention; The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] To facilitate the understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the disclosure of the present invention more thorough and comprehensive.

[0019] It should be noted that when an element is referred to as being "fixedly provided on" another element, it can be directly on the other element or there may also be an intermediate element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be an intermediate element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are only for the purpose of illustration.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this invention belongs. The terms used in the description of the present invention herein are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0021] Please refer to Figure 1 , the ETL-based data processing method provided by the first embodiment of the present invention is used to sequentially transfer data from an extraction node (E node) to a transformation node (T node) and a loading node (L node). The extraction node, the transformation node, and the loading node constitute a data transmission system. The extraction node is connected to a plurality of heterogeneous systems. The ETL-based data processing method includes the following steps: S10: Build a first transmission queue between the extraction node and the transformation node, and build a second transmission queue between the transformation node and the loading node; It can be understood that according to the system configuration of the data transmission system, the first maximum packet number and the first maximum memory of the first transmission queue, the second maximum packet number and the second maximum memory of the second transmission queue can be preset, and the maximum system memory of the data transmission system can be determined. After the extraction node completes data extraction, the extracted data is transmitted from the extraction node to the transformation node through the first transmission queue, and then transmitted from the transformation node to the loading node through the second transmission queue.

[0022] S20: Obtain the system attributes of the first transmission queue, the second transmission queue, and the data transmission system, and determine whether to perform data extraction based on the system attributes; The system attributes include the first real-time packet number, the first real-time queue memory, the second real-time packet number, the second real-time queue memory, and the system memory.

[0023] The step S20 includes: S210: Compare the first real-time packet number with the first maximum packet number, compare the first real-time queue memory with the first maximum memory, compare the second real-time packet number with the second maximum packet number, compare the second real-time queue memory with the second maximum memory, and compare the system memory with the maximum system memory; S220: If the first real-time packet number is less than the first maximum packet number, and the first real-time queue memory is less than the first maximum memory, and the second real-time packet number is less than the second maximum packet number, and the second real-time queue memory is less than the second maximum memory, and the system memory is less than the maximum system memory, it is determined to perform data extraction.

[0024] Understandably, if the number of the first real-time packets is greater than the first maximum packet number, or the first real-time queue memory is greater than the first maximum memory, or the number of the second real-time packets is greater than the second maximum packet number, or the second real-time queue memory is greater than the second maximum memory, or the system memory is greater than the maximum system memory, it is determined that data extraction needs to be paused. Wait until the conversion node and the loading node complete partial data processing and there is sufficient free space before performing data extraction. At the same time, judging the packet number and memory can avoid the situation where the system crashes due to a single data packet having too large a capacity or multiple small-capacity data packets being transmitted simultaneously, occupying a large amount of resource space.

[0025] S30: If data extraction is to be performed, obtain the real-time extraction speed through the system attributes. Based on the real-time extraction speed, extract standby data packets from several heterogeneous systems through the extraction node, and sequentially transmit the standby data packets to the conversion node and the loading node; The step 30 includes: S310: Obtain the preset extraction speed, obtain the first adjustment parameter through the number of the first real-time packets and the first maximum packet number, obtain the second adjustment parameter through the first real-time queue memory and the first maximum memory, obtain the third adjustment parameter through the number of the second real-time packets and the second maximum packet number, obtain the fourth adjustment parameter through the second real-time queue memory and the second maximum memory, and obtain the fifth adjustment parameter through the system memory and the maximum system memory; The formula for obtaining the first adjustment parameter is: , where, represents the first adjustment parameter, represents the first maximum packet number, represents the number of the first real-time packets. The acquisition methods of the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter are the same as those of the first adjustment parameter, and will not be elaborated here.

[0026] S320: Assign parameter weights to the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter respectively, so as to weight the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter into an adjustment coefficient; The formula for obtaining the adjustment coefficient is: , where, represents the adjustment coefficient, represents the second adjustment parameter, represents the third adjustment parameter, represents the fourth adjustment parameter, represents the fifth adjustment parameter, 、 、 、 、 all represent parameter weights, and = 、 = 、 + + + + = 1.

[0027] S330: Compare the adjustment coefficient with the coefficient threshold to obtain the adjustment amplitude, and adjust the preset extraction speed to the real-time extraction speed based on the adjustment amplitude; The formula for obtaining the real-time extraction speed is: , wherein, represents the real-time extraction speed, represents the preset extraction speed, represents the adjustment coefficient, represents the coefficient threshold. It can be understood that when the adjustment coefficient is greater than the coefficient threshold, it indicates that the space resources are relatively rich, and the preset extraction speed is increased. When the adjustment coefficient is less than the coefficient threshold, it indicates that the space resources are relatively scarce, and the preset extraction speed is decreased.

[0028] S340: Obtain a data extraction task, select a number of standby systems from several heterogeneous systems based on the data extraction task, and lock data sources in the standby systems to form a number of extraction paths; S350: Obtain a number of standby data through the extraction nodes and the extraction paths, and combine the number of standby data and the number of extraction paths into a standby data packet.

[0029] S360: Determine whether the processing of the standby data packet at the conversion node and the loading node is successful; It can be understood that when the standby data packet is transmitted from the extraction node to the conversion node, after the conversion node processes the data, it is necessary to make a judgment. And when the conversion node finishes processing, the standby data packet is transmitted from the conversion node to the loading node, and when the loading node processes the data, it is also necessary to make a judgment.

[0030] S370: If it is not successful, match the exception situation with the exception policy, where the exception policy includes a packet loss policy and a processing policy; S380: If the exception situation matches the packet loss policy, extract the extraction path of the standby data packet, and re - extract data based on the extraction path; S390: If the exception situation matches the processing policy, transmit the standby data packet to the screening unit; S391: Conduct error screening on the standby data packet to identify the error area; correct the error area to obtain an updated data packet, and re - place the updated data packet into the first transmission queue or the second transmission queue.

[0031] By constructing the first transmission queue and the second transmission queue, the transmission of the standby data packet between the extraction node, the conversion node, and the loading node is realized; when performing data extraction, by considering the number and memory of data packets that the two transmission queues can carry, and combining with the system memory for comprehensive consideration, that is, real - time monitoring of the data packet accumulation amount and system occupancy, and feedback to the extraction node, to avoid system imbalance caused by continuous data extraction when resources are insufficient; further, when the standby data packet can be extracted, based on the system attributes as the basis for weighing, dynamically control the preset extraction speed, reduce the extraction speed when the system occupancy rate is high, and increase the extraction speed when the system occupancy rate is low, that is, the balance between the speeds of the extraction node, the conversion node, and the loading node is realized, improving the efficiency and stability of data processing; by synchronizing the extraction path into the standby data packet, if the packet loss policy needs to be executed, indexing can be performed according to the extraction path, without re - positioning and extracting data, and if the processing policy needs to be executed, the standby data packet can be directly error - repaired and directly re - injected into the transmission queue, without re - running the entire process, further improving the processing efficiency.

[0032] Please refer to Figure 2 , the second embodiment of the present invention provides an ETL - based data processing system, which is applied to the ETL - based data processing method in the above - mentioned embodiment, and those that have been described will not be repeated. As used below, terms such as "module", "unit", "sub - unit", etc. can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0033] The system includes: A construction module 10, configured to construct a first transmission queue between the extraction node and the conversion node, and construct a second transmission queue between the conversion node and the loading node; An execution module 20, configured to obtain the first transmission queue, the second transmission queue, and system attributes of the data transmission system, and determine whether to perform data extraction based on the system attributes; The execution module 20 includes: A first unit, configured to compare the first real-time packet quantity with a first maximum packet quantity, compare the first real-time queue memory with a first maximum memory, compare the second real-time packet quantity with a second maximum packet quantity, compare the second real-time queue memory with a second maximum memory, and compare the system memory with a maximum system memory; A second unit, configured to determine to perform data extraction if the first real-time packet quantity is less than the first maximum packet quantity, the first real-time queue memory is less than the first maximum memory, the second real-time packet quantity is less than the second maximum packet quantity, the second real-time queue memory is less than the second maximum memory, and the system memory is less than the maximum system memory; An adjustment module 30, configured to, if data extraction is to be performed, obtain a real-time extraction speed through the system attributes, and based on the real-time extraction speed, extract standby data packets from several heterogeneous systems through the extraction nodes, and sequentially transmit the standby data packets to the conversion node and the loading node; The adjustment module 30 includes: A third unit, configured to obtain a preset extraction speed, obtain a first adjustment parameter through the first real-time packet quantity and the first maximum packet quantity, obtain a second adjustment parameter through the first real-time queue memory and the first maximum memory, obtain a third adjustment parameter through the second real-time packet quantity and the second maximum packet quantity, obtain a fourth adjustment parameter through the second real-time queue memory and the second maximum memory, and obtain a fifth adjustment parameter through the system memory and the maximum system memory; A fourth unit, configured to assign parameter weights to the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter respectively, so as to weight the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter into an adjustment coefficient; A fifth unit, configured to compare the adjustment coefficient with a coefficient threshold to obtain an adjustment amplitude, and adjust the preset extraction speed to a real-time extraction speed based on the adjustment amplitude; A sixth unit, configured to obtain a data extraction task, select several standby systems from several heterogeneous systems based on the data extraction task, and lock data sources in the standby systems to form several extraction paths; A seventh unit is used to obtain a plurality of standby data through the extraction nodes and the extraction paths, and combine the plurality of standby data and the plurality of extraction paths into a standby data packet; An eighth unit, used for determining whether the processing of the standby data packet at the conversion node and the loading node is successful; The ninth unit is used for matching the abnormal situation with the abnormal strategy if it fails, and the abnormal strategy includes the packet loss strategy and the processing strategy; A tenth unit is used for extracting the extraction path of the standby data packet if the abnormal situation matches the packet loss strategy, and re-extracting data based on the extraction path; An eleventh unit is configured to transmit the standby data packet to a screening unit if the abnormal situation matches the processing strategy; The twelfth unit is used to perform error screening on the standby data packets to identify error areas; correct the error areas to obtain update data packets, and re-place the update data packets into the first transmission queue or the second transmission queue.

[0034] The present invention also provides a computer, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the ETL-based data processing method as described in the above technical solution is implemented.

[0035] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the ETL-based data processing method as described in the above technical solution is implemented.

[0036] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0037] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An ETL-based data processing method for sequentially transmitting data from an extraction node to a transformation node and a loading node. The extraction node, the transformation node, and the loading node constitute a data transmission system. The extraction node is connected to a plurality of heterogeneous systems, and is characterized in that, The ETL-based data processing method includes the following steps: Construct a first transmission queue between the extraction node and the transformation node, and construct a second transmission queue between the transformation node and the loading node; Obtain the system attributes of the first transmission queue, the second transmission queue, and the data transmission system, and determine whether to perform data extraction based on the system attributes; If data extraction is to be performed, obtain the real-time extraction speed through the system attributes. Based on the real-time extraction speed, use the extraction node to extract standby data packets from several heterogeneous systems, and sequentially transmit the standby data packets to the transformation node and the loading node.

2. The ETL-based data processing method according to claim 1, wherein The system attributes include the first real-time packet quantity, the first real-time queue memory, the second real-time packet quantity, the second real-time queue memory, and the system memory. The step of determining whether to perform data extraction based on the system attributes includes: Compare the first real-time packet quantity with the first maximum packet quantity, compare the first real-time queue memory with the first maximum memory, compare the second real-time packet quantity with the second maximum packet quantity, compare the second real-time queue memory with the second maximum memory, and compare the system memory with the maximum system memory; If the first real-time packet quantity is less than the first maximum packet quantity, and the first real-time queue memory is less than the first maximum memory, and the second real-time packet quantity is less than the second maximum packet quantity, and the second real-time queue memory is less than the second maximum memory, and the system memory is less than the maximum system memory, then it is determined to perform data extraction.

3. The ETL-based data processing method according to claim 2, wherein The step of obtaining the real-time extraction speed through the system attributes includes: Obtain a preset extraction speed, obtain a first adjustment parameter through the first real-time packet quantity and the first maximum packet quantity, obtain a second adjustment parameter through the first real-time queue memory and the first maximum memory, obtain a third adjustment parameter through the second real-time packet quantity and the second maximum packet quantity, obtain a fourth adjustment parameter through the second real-time queue memory and the second maximum memory, and obtain a fifth adjustment parameter through the system memory and the maximum system memory; Assign parameter weights to the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter respectively, so as to weight the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter into an adjustment coefficient; Compare the adjustment coefficient with a coefficient threshold to obtain an adjustment amplitude, and adjust the preset extraction speed to the real-time extraction speed based on the adjustment amplitude.

4. The ETL-based data processing method according to claim 3, wherein The acquisition formula of the first adjustment parameter is: ， Among them, represents the first adjustment parameter, represents the first maximum packet quantity, represents the first real-time packet quantity; The acquisition formula of the adjustment coefficient is: ， Among them, represents a regulation coefficient, represents a second regulation parameter, represents a third regulation parameter, represents a fourth regulation parameter, represents a fifth regulation parameter, 、 、 、 、 all represent parameter weights, and = 、 = 、 + + + + = 1.

5. The ETL-based data processing method according to claim 3, wherein The acquisition formula of the real-time extraction speed is: ， Among them, represents the real-time extraction speed, represents the preset extraction speed, represents the adjustment coefficient, represents the coefficient threshold.

6. The ETL-based data processing method according to claim 1, wherein The step of using the extraction node to extract standby data packets from several heterogeneous systems includes: Obtain a data extraction task, select several standby systems from several heterogeneous systems based on the data extraction task, and lock data sources in the standby systems to form several extraction paths; A plurality of standby data are acquired through the extraction nodes and the extraction paths, and the plurality of standby data and the plurality of extraction paths are combined into a standby data packet.

7. The data processing method based on ETL according to claim 6, wherein The step of sequentially transmitting the standby data packets to the conversion node and the loading node specifically includes: Determining whether the processing of the standby data packet at the conversion node and the loading node is successful; If unsuccessful, the abnormal situation is matched with the abnormal strategy, which includes the packet loss strategy and the processing strategy; If the abnormal situation matches the packet loss strategy, extracting the extraction path of the standby data packet, and re-extracting data based on the extraction path; If the abnormal situation matches the processing strategy, the standby data packet is transmitted to the screening unit.

8. The ETL-based data processing method according to claim 7, wherein After the step of transmitting the standby data packet to the screening unit if the abnormal situation matches the processing strategy, the method further includes: Performing error screening on the standby data packet to identify an error area; The error area is corrected to obtain an update data packet, and the update data packet is re-placed into the first transmission queue or the second transmission queue.

9. An ETL-based data processing system, applied to the ETL-based data processing method according to any one of claims 1 to 8, characterized in that, The system comprises: A construction module, used to construct a first transmission queue between the extraction node and the conversion node, and to construct a second transmission queue between the conversion node and the loading node; an execution module, configured to obtain system properties of the first transmission queue, the second transmission queue and the data transmission system, and determine whether to perform data extraction according to the system properties; The adjustment module is used to obtain a real-time extraction speed through the system properties if data extraction is performed, extract standby data packets from a plurality of the heterogeneous systems through the extraction nodes based on the real-time extraction speed, and transmit the standby data packets to the conversion node and the loading node in sequence.

10. A computer, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, the ETL-based data processing method according to any one of claims 1 to 8 is implemented.

Images