A data processing method and system based on ETL

Abstract

The present invention provides an ETL-based data processing method and system, comprising the following steps: constructing a first transmission queue between an extraction node and a conversion node, and constructing a second transmission queue between a conversion node and a loading node; obtaining system properties of the first transmission queue, the second transmission queue, and the data transmission system, and determining whether to perform data extraction based on the system properties; if data extraction is to be performed, obtaining a real-time extraction speed based on the system properties, extracting standby data packets from a plurality of heterogeneous systems based on the real-time extraction speed through the extraction node, and sequentially transmitting the standby data packets to the conversion node and the loading node. By real-time monitoring of data packet accumulation and system occupancy, system imbalances caused by insufficient resources are avoided; using system properties as a basis for trade-offs, the speeds of the extraction node, conversion node, and loading node are dynamically controlled to balance, thereby improving the efficiency and stability of data processing.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an ETL-based data processing method and system. Background Art

[0002] With the development of society and the advent of the digital age, vast amounts of business data are generated across all industries, making data a core factor of production. Enterprise departments often have numerous heterogeneous systems, such as CRM and ERP, and various data sources. This results in data isolation between these systems, making it impossible to directly share data with each other, creating data silos.

[0003] Unified access through ETL technology enables cross-system data connectivity, breaking down data silos between systems. However, ETL lacks dynamic speed control during data processing, leading to imbalances in the speed of extracting, converting, and loading nodes, impacting data processing efficiency and stability. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a data processing method and system based on ETL, aiming to solve the technical problem that the existing technology lacks control over the dynamic adjustment speed of ETL during data processing, resulting in an imbalance in the speed of extracting nodes, converting nodes and loading nodes, affecting the efficiency and stability of data processing.

[0005] To achieve the above objectives, in a first aspect, embodiments of the present application provide an ETL-based data processing method for sequentially transferring data from an extraction node to a conversion node and a loading node. The extraction node, the conversion node, and the loading node constitute a data transmission system. The extraction node is connected to a plurality of heterogeneous systems. The ETL-based data processing method comprises the following steps:

[0006] Establishing a first transmission queue between the extraction node and the conversion node, and establishing a second transmission queue between the conversion node and the loading node;

[0007] Obtaining system properties of the first transmission queue, the second transmission queue, and the data transmission system, and determining whether to perform data extraction based on the system properties;

[0008] If data extraction is performed, a real-time extraction speed is obtained through the system properties. Based on the real-time extraction speed, standby data packets are extracted from several heterogeneous systems through the extraction node, and the standby data packets are transmitted to the conversion node and the loading node in sequence.

[0009] Furthermore, the system attributes include the number of first real-time packets, the memory of the first real-time queue, the number of second real-time packets, the memory of the second real-time queue, and the system memory. The step of determining whether to perform data extraction based on the system attributes includes:

[0010] Comparing the first real-time packet quantity with a first maximum packet quantity, comparing the first real-time queue memory with a first maximum memory, comparing the second real-time packet quantity with a second maximum packet quantity, comparing the second real-time queue memory with a second maximum memory, and comparing the system memory with a maximum system memory;

[0011] If the first real-time packet number is less than the first maximum packet number, and the first real-time queue memory is less than the first maximum memory, and the second real-time packet number is less than the second maximum packet number, and the second real-time queue memory is less than the second maximum memory, and the system memory is less than the maximum system memory, it is determined to perform data extraction.

[0012] Furthermore, the step of obtaining the real-time extraction speed through the system properties includes:

[0013] Obtaining a preset extraction speed, obtaining a first adjustment parameter based on the first real-time packet number and the first maximum packet number, obtaining a second adjustment parameter based on the first real-time queue memory and the first maximum memory, obtaining a third adjustment parameter based on the second real-time packet number and the second maximum packet number, obtaining a fourth adjustment parameter based on the second real-time queue memory and the second maximum memory, and obtaining a fifth adjustment parameter based on the system memory and the maximum system memory;

[0014] assigning parameter weights to the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter, respectively, so as to weight the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter into adjustment coefficients;

[0015] The adjustment coefficient is compared with a coefficient threshold to obtain an adjustment amplitude, and the preset extraction speed is adjusted to a real-time extraction speed based on the adjustment amplitude.

[0016] Furthermore, the formula for obtaining the first adjustment parameter is:

[0017] ,

[0018] in, represents the first adjustment parameter, Indicates the first maximum number of packets, Indicates the number of the first real-time packet;

[0019] The formula for obtaining the adjustment coefficient is:

[0020] ,

[0021] in, represents the adjustment coefficient, represents the second adjustment parameter, represents the third adjustment parameter, represents the fourth adjustment parameter, represents the fifth adjustment parameter, 、 、 、 、 Both represent parameter weights, and = 、 = 、 + + + + =1.

[0022] Furthermore, the formula for obtaining the real-time extraction speed is:

[0023] ,

[0024] in, Indicates the real-time extraction speed, Indicates the preset extraction speed, represents the adjustment coefficient, Indicates the coefficient threshold.

[0025] Furthermore, the step of extracting unused data packets from the plurality of heterogeneous systems through the extraction node includes:

[0026] Obtaining a data extraction task, selecting a plurality of standby systems from the plurality of heterogeneous systems based on the data extraction task, and locking data sources in the standby systems to form a plurality of extraction paths;

[0027] A plurality of standby data are acquired through the extraction nodes and the extraction paths, and the plurality of standby data and the plurality of extraction paths are combined into a standby data packet.

[0028] Furthermore, the step of sequentially transmitting the standby data packets to the conversion node and the loading node specifically includes:

[0029] Determining whether the standby data packet is successfully processed at the conversion node and the loading node;

[0030] If unsuccessful, the abnormal situation is matched with the abnormal strategy, which includes the packet loss strategy and the processing strategy;

[0031] If the abnormal situation matches the packet loss strategy, extracting the extraction path of the standby data packet and re-extracting data based on the extraction path;

[0032] If the abnormal situation matches the processing strategy, the standby data packet is transmitted to the screening unit.

[0033] Furthermore, after the step of transmitting the standby data packet to the screening unit if the abnormal situation matches the processing strategy, the method further includes:

[0034] performing error screening on the standby data packet to identify an error area;

[0035] The error area is corrected to obtain an update data packet, and the update data packet is re-placed into the first transmission queue or the second transmission queue.

[0036] In a second aspect, an embodiment of the present application provides an ETL-based data processing system, which is applied to the ETL-based data processing method described in the first aspect above. The system includes:

[0037] a construction module, configured to construct a first transmission queue between the extraction node and the conversion node, and to construct a second transmission queue between the conversion node and the loading node;

[0038] an execution module, configured to obtain system properties of the first transmission queue, the second transmission queue, and the data transmission system, and determine whether to perform data extraction based on the system properties;

[0039] The adjustment module is used to obtain a real-time extraction speed through the system properties when data extraction is performed, extract standby data packets from a plurality of the heterogeneous systems through the extraction node based on the real-time extraction speed, and transmit the standby data packets to the conversion node and the loading node in sequence.

[0040] In a third aspect, an embodiment of the present application provides a computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the ETL-based data processing method as described in the first aspect above is implemented.

[0041] In a fourth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the ETL-based data processing method as described in the first aspect above is implemented.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows: by constructing the first transmission queue and the second transmission queue, the transmission of the standby data packet between the extraction node, the conversion node and the loading node is realized; when extracting data, by considering the number of data packets and memory that the two transmission queues can carry, and combining with the memory of the system for comprehensive consideration, that is, real-time monitoring of the data packet accumulation and system occupancy, and feedback to the extraction node, to avoid system imbalance caused by continuous data extraction when resources are insufficient; further, when the extraction of the standby data packet can be carried, the system attributes are used as a basis for weighing and The preset extraction speed is dynamically controlled, reducing the extraction speed when the system occupancy rate is high and increasing the extraction speed when the system occupancy rate is low, thereby achieving a balance between the speed of the extraction node, the speed of the conversion node and the speed of the loading node, thereby improving the efficiency and stability of data processing; by synchronizing the extraction path to the standby data packet, if a packet loss strategy needs to be executed, it can be indexed according to the extraction path without the need to re-locate and extract the data; if a processing strategy needs to be executed, the standby data packet can be directly repaired and directly re-injected into the transmission queue without the need to re-run the entire process, thereby further improving the processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Flowchart of the data processing method based on ETL in the first embodiment of the present invention;

[0044] Figure 2 This is a structural block diagram of an ETL-based data processing system in a second embodiment of the present invention;

[0045] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0046] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The drawings illustrate several embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present invention.

[0047] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0049] See also Figure 1 The first embodiment of the present invention provides an ETL-based data processing method for sequentially transferring data from an extraction node (E node) to a transformation node (T node) and a loading node (L node). The extraction node, the transformation node, and the loading node constitute a data transmission system. The extraction node is connected to a plurality of heterogeneous systems. The ETL-based data processing method includes the following steps:

[0050] S10: constructing a first transmission queue between the extraction node and the conversion node, and constructing a second transmission queue between the conversion node and the loading node;

[0051] It is understood that, based on the system configuration of the data transmission system, a first maximum number of packets and a first maximum memory of the first transmission queue, a second maximum number of packets and a second maximum memory of the second transmission queue can be pre-set, and the maximum system memory of the data transmission system can be determined. After the extraction node completes data extraction, the extracted data is transmitted from the extraction node to the conversion node via the first transmission queue, and then transmitted from the conversion node to the loading node via the second transmission queue.

[0052] S20: Obtaining system properties of the first transmission queue, the second transmission queue, and the data transmission system, and determining whether to perform data extraction based on the system properties;

[0053] The system attributes include a first real-time packet quantity, a first real-time queue memory, a second real-time packet quantity, a second real-time queue memory, and a system memory.

[0054] The step S20 includes:

[0055] S210: Compare the first real-time packet quantity with a first maximum packet quantity, compare the first real-time queue memory with a first maximum memory, compare the second real-time packet quantity with a second maximum packet quantity, compare the second real-time queue memory with a second maximum memory, and compare the system memory with a maximum system memory;

[0056] S220: If the first real-time packet number is less than the first maximum packet number, and the first real-time queue memory is less than the first maximum memory, and the second real-time packet number is less than the second maximum packet number, and the second real-time queue memory is less than the second maximum memory, and the system memory is less than the maximum system memory, it is determined to perform data extraction.

[0057] Understandably, if the first real-time packet number is greater than the first maximum packet number, or the first real-time queue memory is greater than the first maximum memory, or the second real-time packet number is greater than the second maximum packet number, or the second real-time queue memory is greater than the second maximum memory, or the system memory is greater than the maximum system memory, then it is determined that data extraction needs to be suspended, and data extraction will resume when the conversion node and the loading node have completed partial data processing and have sufficient space. Simultaneously determining the number of packets and memory can avoid situations where a single data packet is too large or multiple small data packets are transmitted simultaneously, occupying a large amount of resource space and causing system crashes.

[0058] S30: If data extraction is performed, a real-time extraction speed is obtained through the system attribute, and based on the real-time extraction speed, standby data packets are extracted from the plurality of heterogeneous systems through the extraction node, and the standby data packets are sequentially transmitted to the conversion node and the loading node;

[0059] The step 30 includes:

[0060] S310: Obtain a preset extraction speed, obtain a first adjustment parameter based on the first real-time packet number and the first maximum packet number, obtain a second adjustment parameter based on the first real-time queue memory and the first maximum memory, obtain a third adjustment parameter based on the second real-time packet number and the second maximum packet number, obtain a fourth adjustment parameter based on the second real-time queue memory and the second maximum memory, and obtain a fifth adjustment parameter based on the system memory and the maximum system memory;

[0061] The formula for obtaining the first adjustment parameter is:

[0062] ,

[0063] in, represents the first adjustment parameter, Indicates the first maximum number of packets, The acquisition method of the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter and the fifth adjustment parameter is the same as that of the first adjustment parameter, and will not be described in detail here.

[0064] S320: Assigning parameter weights to the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter, respectively, so as to weight the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter into an adjustment coefficient;

[0065] The formula for obtaining the adjustment coefficient is:

[0066] ,

[0067] in, represents the adjustment coefficient, represents the second adjustment parameter, represents the third adjustment parameter, represents the fourth adjustment parameter, represents the fifth adjustment parameter, 、 、 、 、 Both represent parameter weights, and = 、 = 、 + + + + =1.

[0068] S330: Compare the adjustment coefficient with a coefficient threshold to obtain an adjustment amplitude, and adjust the preset extraction speed to a real-time extraction speed based on the adjustment amplitude;

[0069] The formula for obtaining the real-time extraction speed is:

[0070] ,

[0071] in, Indicates the real-time extraction speed, Indicates the preset extraction speed, represents the adjustment coefficient, It is understood that when the adjustment coefficient is greater than the coefficient threshold, it indicates that the spatial resources are relatively abundant, and the preset extraction speed is accelerated. When the adjustment coefficient is less than the coefficient threshold, it indicates that the spatial resources are relatively scarce, and the preset extraction speed is reduced.

[0072] S340: Obtaining a data extraction task, selecting a plurality of standby systems from the plurality of heterogeneous systems based on the data extraction task, and locking data sources in the standby systems to form a plurality of extraction paths;

[0073] S350: Acquire a plurality of standby data through the extraction nodes and the extraction paths, and combine the plurality of standby data and the plurality of extraction paths into a standby data packet.

[0074] S360: Determine whether the processing of the standby data packet at the conversion node and the loading node is successful;

[0075] It can be understood that when the standby data packet is transmitted from the extraction node to the conversion node, the conversion node needs to make a judgment after processing the data, and when the conversion node completes the processing, the standby data packet is transmitted from the conversion node to the loading node, and the loading node also needs to make a judgment when processing the data.

[0076] S370: If unsuccessful, matching the abnormal situation with the abnormal strategy, wherein the abnormal strategy includes a packet loss strategy and a processing strategy;

[0077] S380: If the abnormal situation matches the packet loss policy, extract the extraction path of the unused data packet and re-extract data based on the extraction path;

[0078] S390: If the abnormal situation matches the processing strategy, the standby data packet is transmitted to the screening unit;

[0079] S391: Perform error screening on the standby data packet to identify an error area; correct the error area to obtain an update data packet, and re-place the update data packet into the first transmission queue or the second transmission queue.

[0080] By constructing the first transmission queue and the second transmission queue, the transmission of the standby data packets between the extraction node, the conversion node and the loading node is realized; when extracting data, the number of data packets and the memory that can be carried by the two transmission queues are considered, and combined with the memory of the system, a comprehensive consideration is made, that is, the data packet accumulation and system occupancy are monitored in real time, and the feedback is fed back to the extraction node to avoid the system imbalance caused by continuous data extraction when resources are insufficient; further, when the extraction of the standby data packets can be carried, the preset extraction speed is adjusted based on the system properties. Dynamic control reduces the extraction speed when the system occupancy rate is high, and increases the extraction speed when the system occupancy rate is low, that is, it achieves a balance between the speed of the extraction node, the speed of the conversion node and the speed of the loading node, and improves the efficiency and stability of data processing; by synchronizing the extraction path to the standby data packet, if the packet loss strategy needs to be executed, it can be indexed according to the extraction path, without the need to re-locate and extract the data; if the processing strategy needs to be executed, the standby data packet can be directly repaired and directly re-injected into the transmission queue without the need to re-run the entire process, further improving the processing efficiency.

[0081] See also Figure 2 The second embodiment of the present invention provides an ETL-based data processing system, which is applied to the ETL-based data processing method described in the above embodiments. Details that have been described will not be repeated here. As used below, the terms "module," "unit," "subunit," etc. may refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0082] The system comprises:

[0083] A construction module 10 is configured to construct a first transmission queue between the extraction node and the conversion node, and to construct a second transmission queue between the conversion node and the loading node;

[0084] An execution module 20 is configured to obtain system properties of the first transmission queue, the second transmission queue, and the data transmission system, and determine whether to perform data extraction based on the system properties;

[0085] The execution module 20 includes:

[0086] a first unit, configured to compare the first real-time packet number with a first maximum packet number, compare the first real-time queue memory with a first maximum memory, compare the second real-time packet number with a second maximum packet number, compare the second real-time queue memory with a second maximum memory, and compare the system memory with a maximum system memory;

[0087] a second unit, configured to determine to perform data extraction if the first real-time packet number is less than the first maximum packet number, the first real-time queue memory is less than the first maximum memory, the second real-time packet number is less than the second maximum packet number, the second real-time queue memory is less than the second maximum memory, and the system memory is less than the maximum system memory;

[0088] an adjustment module 30 for, when data extraction is performed, obtaining a real-time extraction speed through the system attributes, extracting standby data packets from the plurality of heterogeneous systems through the extraction nodes based on the real-time extraction speed, and sequentially transmitting the standby data packets to the conversion node and the loading node;

[0089] The adjustment module 30 includes:

[0090] a third unit, configured to obtain a preset extraction speed, obtain a first adjustment parameter based on the first real-time packet number and the first maximum packet number, obtain a second adjustment parameter based on the first real-time queue memory and the first maximum memory, obtain a third adjustment parameter based on the second real-time packet number and the second maximum packet number, obtain a fourth adjustment parameter based on the second real-time queue memory and the second maximum memory, and obtain a fifth adjustment parameter based on the system memory and the maximum system memory;

[0091] a fourth unit, configured to assign parameter weights to the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter, respectively, so as to weight the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter into an adjustment coefficient;

[0092] a fifth unit, configured to compare the adjustment coefficient with a coefficient threshold to obtain an adjustment amplitude, and adjust the preset extraction speed to a real-time extraction speed based on the adjustment amplitude;

[0093] A sixth unit is configured to obtain a data extraction task, select a plurality of standby systems from the plurality of heterogeneous systems based on the data extraction task, and lock data sources in the standby systems to form a plurality of extraction paths;

[0094] a seventh unit, configured to obtain a plurality of standby data through the extraction nodes and the extraction paths, and combine the plurality of standby data and the plurality of extraction paths into a standby data packet;

[0095] An eighth unit is configured to determine whether the standby data packet is successfully processed at the conversion node and the loading node;

[0096] The ninth unit is configured to match the abnormal situation with an abnormal strategy if the abnormal situation fails, wherein the abnormal strategy includes a packet loss strategy and a processing strategy;

[0097] a tenth unit, configured to extract an extraction path of the standby data packet and re-extract data based on the extraction path if the abnormal situation matches the packet loss policy;

[0098] an eleventh unit, configured to transmit the standby data packet to a screening unit if the abnormal situation matches the processing strategy;

[0099] The twelfth unit is configured to perform error screening on the standby data packet to identify an error area; correct the error area to obtain an update data packet, and re-place the update data packet into the first transmission queue or the second transmission queue.

[0100] The present invention also provides a computer, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the ETL-based data processing method as described in the above technical solution is implemented.

[0101] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the ETL-based data processing method as described in the above technical solution is implemented.

[0102] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0103] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A data processing method based on ETL, for transferring data from an extraction node to a conversion node and a loading node in sequence, wherein the extraction node, the conversion node and the loading node constitute a data transmission system, and the extraction node is connected to a plurality of heterogeneous systems, characterized in that: The ETL-based data processing method includes the following steps: Establishing a first transmission queue between the extraction node and the conversion node, and establishing a second transmission queue between the conversion node and the loading node; Obtaining system properties of the first transmission queue, the second transmission queue, and the data transmission system, the system properties including the number of first real-time packets, the memory of the first real-time queue, the number of second real-time packets, the memory of the second real-time queue, and the system memory, and determining whether to perform data extraction based on the system properties; The step of determining whether to perform data extraction based on the system attributes includes: Comparing the first real-time packet quantity with a first maximum packet quantity, comparing the first real-time queue memory with a first maximum memory, comparing the second real-time packet quantity with a second maximum packet quantity, comparing the second real-time queue memory with a second maximum memory, and comparing the system memory with a maximum system memory; If the first real-time packet number is less than the first maximum packet number, the first real-time queue memory is less than the first maximum memory, the second real-time packet number is less than the second maximum packet number, the second real-time queue memory is less than the second maximum memory, and the system memory is less than the maximum system memory, then determining to perform data extraction; If data extraction is performed, a real-time extraction speed is obtained through the system attributes, and based on the real-time extraction speed, standby data packets are extracted from the plurality of heterogeneous systems through the extraction node, and the standby data packets are sequentially transmitted to the conversion node and the loading node; The step of obtaining the real-time extraction speed through the system properties includes: Obtaining a preset extraction speed, obtaining a first adjustment parameter based on the first real-time packet number and the first maximum packet number, obtaining a second adjustment parameter based on the first real-time queue memory and the first maximum memory, obtaining a third adjustment parameter based on the second real-time packet number and the second maximum packet number, obtaining a fourth adjustment parameter based on the second real-time queue memory and the second maximum memory, and obtaining a fifth adjustment parameter based on the system memory and the maximum system memory; assigning parameter weights to the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter, respectively, so as to weight the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter into adjustment coefficients; The adjustment coefficient is compared with a coefficient threshold to obtain an adjustment amplitude, and the preset extraction speed is adjusted to a real-time extraction speed based on the adjustment amplitude.

2. The ETL-based data processing method according to claim 1, characterized in that: The formula for obtaining the first adjustment parameter is: ， in, represents the first adjustment parameter, Indicates the first maximum number of packets, Indicates the number of the first real-time packet; The formula for obtaining the adjustment coefficient is: ， in, represents the adjustment coefficient, represents the second adjustment parameter, represents the third adjustment parameter, represents the fourth adjustment parameter, represents the fifth adjustment parameter, 、 、 、 、 Both represent parameter weights, and = 、 = 、 + + + + =1.

3. The ETL-based data processing method according to claim 1, characterized in that: The formula for obtaining the real-time extraction speed is: ， in, Indicates the real-time extraction speed, Indicates the preset extraction speed, represents the adjustment coefficient, Indicates the coefficient threshold.

4. The data processing method based on ETL according to claim 1, characterized in that: The step of extracting unused data packets from the plurality of heterogeneous systems through the extraction node includes: Obtaining a data extraction task, selecting a plurality of standby systems from the plurality of heterogeneous systems based on the data extraction task, and locking data sources in the standby systems to form a plurality of extraction paths; A plurality of standby data are acquired through the extraction nodes and the extraction paths, and the plurality of standby data and the plurality of extraction paths are combined into a standby data packet.

5. The ETL-based data processing method according to claim 4, characterized in that: The step of sequentially transmitting the standby data packets to the conversion node and the loading node specifically includes: Determining whether the standby data packet is successfully processed at the conversion node and the loading node; If unsuccessful, the abnormal situation is matched with the abnormal strategy, which includes the packet loss strategy and the processing strategy; If the abnormal situation matches the packet loss strategy, extracting the extraction path of the standby data packet and re-extracting data based on the extraction path; If the abnormal situation matches the processing strategy, the standby data packet is transmitted to the screening unit.

6. The ETL-based data processing method according to claim 5, characterized in that: After the step of transmitting the standby data packet to the screening unit if the abnormal situation matches the processing strategy, the method further includes: performing error screening on the standby data packet to identify an error area; The error area is corrected to obtain an update data packet, and the update data packet is re-placed into the first transmission queue or the second transmission queue.

7. An ETL-based data processing system, applied to the ETL-based data processing method according to any one of claims 1 to 6, characterized in that: The system comprises: a construction module, configured to construct a first transmission queue between the extraction node and the conversion node, and to construct a second transmission queue between the conversion node and the loading node; an execution module, configured to obtain system attributes of the first transmission queue, the second transmission queue, and the data transmission system, the system attributes including the number of first real-time packets, the memory of the first real-time queue, the number of second real-time packets, the memory of the second real-time queue, and the system memory, and determine whether to perform data extraction based on the system attributes; The execution module includes: a first unit, configured to compare the first real-time packet number with a first maximum packet number, compare the first real-time queue memory with a first maximum memory, compare the second real-time packet number with a second maximum packet number, compare the second real-time queue memory with a second maximum memory, and compare the system memory with a maximum system memory; a second unit, configured to determine to perform data extraction if the first real-time packet number is less than the first maximum packet number, the first real-time queue memory is less than the first maximum memory, the second real-time packet number is less than the second maximum packet number, the second real-time queue memory is less than the second maximum memory, and the system memory is less than the maximum system memory; an adjustment module, configured to, if data extraction is performed, obtain a real-time extraction speed through the system attributes, extract standby data packets from the plurality of heterogeneous systems through the extraction node based on the real-time extraction speed, and sequentially transmit the standby data packets to the conversion node and the loading node; The adjustment module includes: a third unit, configured to obtain a preset extraction speed, obtain a first adjustment parameter based on the first real-time packet number and the first maximum packet number, obtain a second adjustment parameter based on the first real-time queue memory and the first maximum memory, obtain a third adjustment parameter based on the second real-time packet number and the second maximum packet number, obtain a fourth adjustment parameter based on the second real-time queue memory and the second maximum memory, and obtain a fifth adjustment parameter based on the system memory and the maximum system memory; a fourth unit, configured to assign parameter weights to the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter, respectively, so as to weight the first adjustment parameter, the second adjustment parameter, the third adjustment parameter, the fourth adjustment parameter, and the fifth adjustment parameter into an adjustment coefficient; The fifth unit is configured to compare the adjustment coefficient with a coefficient threshold to obtain an adjustment amplitude, and adjust the preset extraction speed to a real-time extraction speed based on the adjustment amplitude.

8. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the ETL-based data processing method according to any one of claims 1 to 6 is implemented.

Images