Method, device and electronic equipment for optimizing data warehouse technology business

Abstract

This application provides a method, apparatus, and electronic device for optimizing data warehouse technology operations. The method includes: receiving a merge request from a user, the merge request indicating the merging of at least two ETL operations, the at least two ETL operations including a first ETL operation and a second ETL operation; determining whether the data sources of the first ETL operation and the second ETL operation are the same; if they are the same, merging the first ETL operation and the second ETL operation based on pre-configured operator merging rules to obtain a target ETL operation, wherein the target ETL operation is a single read data and multiple write data operations, and the multiple write data operations in the target ETL operation include write data from the first ETL operation and write data from the second ETL operation. This method solves the problem of redundancy in computational logic between ETL operations by merging operators of ETL operations with the same data source based on pre-configured operator merging rules.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus and electronic device for optimizing data warehouse technology operations. Background Technology

[0002] Data warehousing technology (Extract-Transform-Load, ETL) describes the process of extracting, transforming, and loading data from a source to a destination. Its purpose is to integrate scattered, disorganized, and inconsistent data within an enterprise, providing analytical support for decision-making. The source can be a business system database, a distributed file system, or other data warehouses, while the destination can be a target system database, a target file system, or other target data warehouses. The results input to the destination in an ETL process can include business metrics, statistical data, and consistent data.

[0003] Currently, with the rapid development of communication technology, user traffic and 5G services have experienced explosive growth in quantity. However, in the big data processing of operators, the customized business logic of ETL business systems is often not optimal, and there is often a lot of redundancy in the computational logic between various ETL services. This leads to a surge in the computing resources, input and output (IO) resources, and storage resources consumed by ETL services, resulting in a significant waste of resources.

[0004] Therefore, how to solve the redundancy problem of computational logic between ETL services is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides a method, apparatus, and electronic device for optimizing data warehouse technology services. By merging operators of ETL services from different data sources based on pre-configured operator merging rules, the redundancy problem of computational logic between ETL services is solved.

[0006] In a first aspect, embodiments of this application provide a method for optimizing data warehouse technology services. The method includes: receiving a merge request issued by a user, the merge request indicating the merging of at least two ETL services, the at least two ETL services including a first ETL service and a second ETL service; determining whether the data sources of the first ETL service and the second ETL service are the same, and if they are the same, merging the first ETL service and the second ETL service based on pre-configured operator merge rules to obtain a target ETL service, wherein the target ETL service is a service that performs a single read of data and multiple writes of data, and the multiple writes of data in the target ETL service include the write data of the first ETL service and the write data of the second ETL service.

[0007] Therefore, when multiple ETL services share the same data source, merging these services into a single ETL service that reads data once and writes data multiple times reduces computational logic redundancy and read / write redundancy among the multiple ETL services before merging, thus solving the problem of computational logic redundancy between ETL services.

[0008] In one possible implementation, the first ETL service and the second ETL service are merged based on a pre-configured operator merging rule, including: determining the operator depth of each operator in the first ETL service, where the operator depth is used to characterize the interval between the first operator and the first read operator in the first ETL service, the first read operator is used to read data, each operator includes the first operator, and the first operator includes the first read operator; and merging the first operator in the first ETL service and the second operator in the second ETL service in ascending order of operator depth according to the operator merging rule to obtain the target operator, where the first operator and the second operator have the same operator depth.

[0009] In one possible implementation, the method further includes: when the target operator includes a common operator, merging the third operator in the first ETL service and the fourth operator in the second ETL service based on the operator merging rule, wherein the third operator is adjacent to the first operator and the fourth operator is adjacent to the second operator, and the common operator is the operator that is commonly corresponding to the first ETL service and the second ETL service in the target ETL service.

[0010] In one possible implementation, the method further includes: when the target operator includes a common operator, a first branch operator, and a second branch operator, the order of the first target operator and the second target operator is swapped based on a pre-configured operator swapping rule. The common operator is the operator commonly corresponding to both the first and second ETL services in the target ETL service; the first branch operator is the operator corresponding to the first ETL service in the target ETL service; the second branch operator is the operator corresponding to the second ETL service in the target ETL service; the first target operator includes at least one of the first branch operator and the second branch operator; the second target operator is either the third operator in the first ETL service or the fourth operator in the second ETL service; the third operator is adjacent to the first operator; the fourth operator is adjacent to the second operator; and the first target operator and the second target operator correspond to the same ETL service. After swapping the order of the first target operator and the second target operator, the third operator and the fourth operator are merged based on an operator merging rule.

[0011] In one possible implementation, the method further includes: when the first operator and the second operator cannot be merged, or the first target operator and the second target operator cannot be swapped, or the number of target operations exceeds a preset threshold, the number of target operations includes at least one of the number of merges of operators of the same type and the number of swaps, and the merging of the first ETL service and the second ETL service is terminated; based on the operator depth, the obtained target operator is combined with the unmerged operators in the first ETL service to obtain the first branch ETL service, and the obtained target operator is combined with the unmerged operators in the second ETL service to obtain the second branch ETL service, wherein the target ETL service includes the first branch ETL service and the second branch ETL service.

[0012] In one possible implementation, the method further includes: when the target operator is obtained, determining the execution cost of the third ETL service, wherein the third ETL service consists of the obtained target operator and the unmerged operators in the first ETL service or the second ETL service; and determining the target ETL service based on the execution cost of the third ETL service.

[0013] In one possible implementation, the target ETL service is determined based on the execution cost of the third ETL service, including: when there are multiple third ETL services, if the execution cost of the fourth ETL service is less than or equal to the sum of the execution costs of the first ETL service and the second ETL service, then the fourth ETL service is taken as the target ETL service, and the fourth ETL service is the third ETL service with the lowest execution cost.

[0014] In one possible implementation, the fourth ETL service is taken as the target ETL service, including: setting a cache operator after the common operator contained in the third target operator in the fourth ETL service to obtain the target ETL service. The third target operator is the target operator with the largest operator depth in the fourth ETL service. The cache operator is used to cache the data output by the common operator and to provide data to the branch operators after the common operator.

[0015] In one possible implementation, after obtaining the target ETL business, the following steps are also included: modifying the target ETL business based on the sinking and floating rules, whereby the shrinking operator floats upwards towards the head of the ETL business and the expanding operator sinks downwards towards the tail of the ETL business; wherein, the shrinking operator is an operator that reduces the amount of data after processing, and the expanding operator is an operator that increases the amount of data after processing.

[0016] In one possible implementation, the target ETL service includes a common operator, a cache operator, and at least two sets of branch operators. The cache operator is located between the common operator and the at least two sets of branch operators. The cache operator is used to cache the data output by the common operator and to provide data to the first set of branch operators. The common operator includes a read operator, and the at least two sets of branch operators include a first set of branch operators. The first set of branch operators includes a write operator for the first ETL service. The read operator is used to read data, and the write operator is used to write data.

[0017] Secondly, embodiments of this application provide an optimization apparatus for data warehouse technology services. The apparatus includes: a receiving module, configured to receive a merge request issued by a user, the merge request indicating the merging of at least two ETL services, the at least two ETL services including a first ETL service and a second ETL service; and a processing module, configured to determine whether the data sources of the first ETL service and the second ETL service are the same, and if they are the same, merge the first ETL service and the second ETL service based on pre-configured operator merging rules to obtain a target ETL service, wherein the target ETL service is a service that performs a single read of data and multiple writes of data, and the multiple writes of data in the target ETL service include the write data of the first ETL service and the write data of the second ETL service.

[0018] In one possible implementation, the processing module is further configured to: determine the operator depth of each operator in the first ETL service, wherein the operator depth is used to characterize the interval between the first operator and the first read operator in the first ETL service, the first read operator is used to read data, each operator includes the first operator, and the first operator includes the first read operator; and according to the order of the operator depth, merge the first operator in the first ETL service and the second operator in the second ETL service in order from smallest to largest based on the operator merging rules to obtain the target operator, wherein the first operator and the second operator have the same operator depth.

[0019] In one possible implementation, the processing module is further configured to: when the target operator includes a common operator, merge the third operator in the first ETL service and the fourth operator in the second ETL service based on the operator merging rule, wherein the third operator is adjacent to the first operator and the fourth operator is adjacent to the second operator, wherein the common operator is the operator that is commonly corresponding to the first ETL service and the second ETL service in the target ETL service.

[0020] In one possible implementation, the processing module is further configured to: when the target operator includes a common operator, a first branch operator, and a second branch operator, exchange the order of the first target operator and the second target operator based on a pre-configured operator exchange rule, wherein the common operator is the operator commonly corresponding to the first ETL service and the second ETL service in the target ETL service, the first branch operator is the operator corresponding to the first ETL service in the target ETL service, the second branch operator is the operator corresponding to the second ETL service in the target ETL service, the first target operator includes at least one of the first branch operator and the second branch operator, the second target operator is the third operator in the first ETL service or the fourth operator in the second ETL service, the third operator is adjacent to the first operator, the fourth operator is adjacent to the second operator, and the first target operator and the second target operator correspond to the same ETL service; after exchanging the order of the first target operator and the second target operator, merge the third operator and the fourth operator based on the operator merging rule.

[0021] In one possible implementation, the processing module is further configured to: terminate the merging of the first ETL service and the second ETL service when the first operator and the second operator cannot be merged, or the first target operator and the second target operator cannot be swapped, or the number of target operations exceeds a preset threshold, wherein the number of target operations includes at least one of the number of merging operations of operators of the same type and the number of swapping operations; and, based on the operator depth, combine the obtained target operator with the unmerged operators in the first ETL service to obtain the first branch ETL service, and combine the obtained target operator with the unmerged operators in the second ETL service to obtain the second branch ETL service, wherein the target ETL service includes the first branch ETL service and the second branch ETL service.

[0022] In one possible implementation, the processing module is further configured to: determine the execution cost of the third ETL service when the target operator is obtained, wherein the third ETL service consists of the obtained target operator and the unmerged operators in the first ETL service or the second ETL service; and determine the target ETL service based on the execution cost of the third ETL service.

[0023] In one possible implementation, the processing module is further configured to: when there are multiple third ETL services, if the execution cost of the fourth ETL service is less than or equal to the sum of the execution costs of the first ETL service and the second ETL service, then the fourth ETL service is selected as the target ETL service, and the fourth ETL service is the third ETL service with the lowest execution cost.

[0024] In one possible implementation, the processing module is further configured to: set a cache operator after the common operator contained in the third target operator in the fourth ETL service to obtain the target ETL service, wherein the third target operator is the target operator with the largest operator depth in the fourth ETL service, and the cache operator is used to cache the data output by the common operator and to provide data to the branch operators after the common operator.

[0025] In one possible implementation, the processing module is further configured to: after obtaining the target ETL service, modify the target ETL service based on the sinking and floating rules, wherein the sinking and floating rules are: the shrinking operator floats upwards towards the head of the ETL service, and the expanding operator sinks downwards towards the tail of the ETL service; wherein the shrinking operator is an operator that reduces the amount of data after processing, and the expanding operator is an operator that increases the amount of data after processing.

[0026] In one possible implementation, the target ETL service includes a common operator, a cache operator, and at least two sets of branch operators. The cache operator is located between the common operator and the at least two sets of branch operators. The cache operator is used to cache the data output by the common operator and to provide data to the first set of branch operators. The common operator includes a read operator, and the at least two sets of branch operators include a first set of branch operators. The first set of branch operators includes a write operator for the first ETL service. The read operator is used to read data, and the write operator is used to write data.

[0027] Thirdly, embodiments of this application provide an electronic device, including: at least one memory and at least one processor; wherein, at least one memory is used to store a program, and at least one processor is used to execute the program stored in the memory, and when the program stored in the memory is executed, at least one processor is used to execute the method provided in the first aspect.

[0028] Fourthly, embodiments of this application provide a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect.

[0029] Fifthly, embodiments of this application provide a chip including at least one processor and an interface; wherein the interface is used to provide program instructions or data to at least one processor, and at least one processor is used to execute program line instructions to implement the method provided in the first aspect.

[0030] In a sixth aspect, embodiments of this application provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect. Attached Figure Description

[0031] The accompanying drawings used in the description of the embodiments or prior art are briefly introduced below.

[0032] Figure 1 This is a schematic diagram of the system architecture of an ETL service optimization method provided in an embodiment of this application;

[0033] Figure 2 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0034] Figure 3a This is a schematic diagram illustrating the changes before and after operator merging, as provided in an embodiment of this application.

[0035] Figure 3b This is a schematic diagram illustrating the changes before and after operator merging, as provided in another embodiment of this application.

[0036] Figure 3c This is a schematic diagram illustrating the changes before and after operator merging, as provided in another embodiment of this application.

[0037] Figure 4 This is a schematic diagram illustrating the change in the order of two serial operators before and after swapping, as provided in an embodiment of this application.

[0038] Figure 5 This is a schematic diagram of the display interface of an electronic device provided in an embodiment of this application;

[0039] Figure 6 This is a schematic diagram of a DAG of an operator in an ETL service provided in an embodiment of this application;

[0040] Figure 7a This is a schematic diagram of a model of an ETL service to be merged, provided in an embodiment of this application;

[0041] Figure 7b This is a schematic diagram of a merged ETL service model provided in an embodiment of this application;

[0042] Figure 7c This is a schematic diagram of a merged ETL service model provided in an embodiment of this application;

[0043] Figure 7d This is a schematic diagram of a merged ETL service model provided in an embodiment of this application;

[0044] Figure 7e This is a schematic diagram of a merged ETL service model provided in an embodiment of this application;

[0045] Figure 8 This is a schematic diagram of a merged ETL service model provided in an embodiment of this application;

[0046] Figure 9This is a schematic diagram of an ETL service that involves a single data read and multiple data writes, as provided in an embodiment of this application.

[0047] Figure 10 This is a flowchart illustrating an optimization method for ETL services provided in an embodiment of this application.

[0048] Figure 11 This is a schematic diagram illustrating the steps of merging a first ETL service and a second ETL service based on a pre-configured operator merging rule, as provided in an embodiment of this application.

[0049] Figure 12 This is a schematic diagram of the structure of an ETL service optimization device provided in an embodiment of this application;

[0050] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0051] Figure 14 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0053] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0054] In the description of the embodiments in this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple terminals refer to two or more terminals.

[0055] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0056] The following section introduces the relevant terms and concepts used in this plan.

[0057] (1) Operator

[0058] An operator is the smallest unit of computation in an ETL process.

[0059] (2) Reading operators

[0060] The read operator is the operator that reads data from the data source in an ETL process; it is the first operator in an ETL process.

[0061] (3) Write operators

[0062] The write operator is the operator used in ETL operations to load data, that is, to write data to the destination. It is the last operator in the ETL process.

[0063] (4) Expansion operator

[0064] An expansion operator is an operator that increases the amount of data after it has been processed.

[0065] (5) Shrinking operator

[0066] A shrinkage operator is an operator that reduces the amount of data after processing.

[0067] (6) Operator feature index set

[0068] The operator feature index set is a set of indices that can uniquely identify a certain operator.

[0069] (7) Operator dimension

[0070] Operator dimensions describe the characteristics of fact records in fact data. Some characteristics provide descriptive information, while others specify how to summarize the data in the fact data table so that analysts can provide useful information.

[0071] (8) Operator calculation formula

[0072] The calculation formula for operators is a fixed calculation model in ETL business, which is a formula for mathematical calculation of operator dimensions.

[0073] (9) Parent-child and child-child operators

[0074] In ETL operations, if the output of one operator is the input of another operator, then the one operator is called the parent operator of the other operator, and the other operator is called the child operator of the one operator.

[0075] (10) Operator depth

[0076] In ETL operations, operator depth can be defined as follows: the depth of a read operator is 0, and the depth of a child operator is the depth of its parent operator plus 1. For example, in an ETL operation, if the depth of the Nth operator is n, then the depth of the (N+1)th operator is n+1, where the Nth operator is the parent operator of the (N+1)th operator.

[0077] (11) ETL business with the same data source

[0078] For multiple ETL services, if the read operators of each ETL service read from the same data source, they are called ETL services with the same data source.

[0079] Next, we will introduce the application scenarios of the ETL business optimization methods provided in the solution.

[0080] Figure 1 This is a schematic diagram of the system architecture of an ETL service optimization method provided in an embodiment of this application. For example... Figure 1 As shown, the data acquisition module 11 can collect data, for example, the collected data can be usernames, protocol types, user types, cell information, etc. in telecommunications services; the data storage module 12 can store the data collected by the data acquisition module 11; the ETL service system 13 can extract data from the data storage module 12, then transform and load the extracted data, and finally store the processing results in a distributed file system or database, etc., wherein the ETL service system can include multiple ETL services; the data access service module 14 can display the processing results of the ETL service system to the user, and conduct human-computer interaction with the user, etc. In one example, the user can issue a request to merge multiple ETL services through the human-computer interaction interface provided by the data access service module 14. This request can instruct the ETL service system 13 to merge multiple ETL services, wherein the request can include the identifier of the ETL service to be merged selected by the user; then, the ETL service system 13 can merge the multiple ETL services selected by the user based on the operator merging rules pre-configured by the user, thereby enabling multiple ETL services to be combined into an ETL service that can read data once and write data multiple times. It is understandable that each write operation in the merged ETL service corresponds to a single write operation in the pre-merger ETL service; it can also be understood that the merged ETL service includes multiple write operators, each of which corresponds to a write operator in the pre-merger ETL service.

[0081] Understandable, Figure 1 The ETL business system 13 shown can be configured independently in an electronic device, or it can be configured together with one or more of the data acquisition module 11, data storage module 12 and data access service module 14 in the same electronic device, without any limitation.

[0082] The following is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0083] Figure 2 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. For example... Figure 2 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0084] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0085] Processor 110 may include one or more processing units. For example, processor 110 may include one or more of the following: application processor (AP), modem, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0086] The processor 110 may also include a memory for storing instructions and data. In some examples, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory to avoid repeated access, reduce the processor 110's waiting time, and improve system efficiency. In some examples, the processor 110 can be used to merge various ETL services based on user-issued ETL service merging requests.

[0087] In some examples, processor 110 may include one or more interfaces. Interfaces may include inter-integrated circuit (I2C) interfaces, inter-integrated circuit sound (I2S) interfaces, Pulse Code Modulation (PCM) interfaces, Universal Asynchronous Receiver / Transmitter (UART) interfaces, Mobile Industry Processor Interface (MIPI) interfaces, General Purpose I / O Ports (GPIO) interfaces, subscriber identity module (SIM) interfaces, and / or Universal Serial Bus (USB) interfaces, etc.

[0088] The charging management module 140 receives charging input from a charger, which can be either a wireless charger or a wired charger. In some wired charging examples, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging examples, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to other electronic devices via the power management module 141.

[0089] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other examples, the power management module 141 may be located within the processor 110. In still other examples, the power management module 141 and the charging management module 140 may be located in the same device.

[0090] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem, and baseband processor.

[0091] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In other examples, the antennas can be used in conjunction with a tuning switch.

[0092] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through at least two antennas, including antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem and convert it into electromagnetic waves for radiation via antenna 1. In some examples, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some examples, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0093] A modem may include a modulator and a demodulator. The modulator modulates a low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to a baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is passed to an application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through a display screen 194. In some examples, the modem may be a standalone device. In other examples, the modem may be independent of the processor 110 and housed within the same device as the mobile communication module 150 or other functional modules. In still other examples, the mobile communication module 150 may be a module within the modem.

[0094] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0095] In some examples, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), 5G, New Radio (NR), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0096] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0097] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some examples, electronic device 100 may include one or more display screens 194.

[0098] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0099] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, during shooting, when the shutter is opened, light is transmitted through the lens to the camera's image sensor. The light signal is converted into an electrical signal, and the camera's image sensor transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some examples, the ISP can be set within the camera 193 itself.

[0100] Camera 193 is used to capture still images or videos, such as capturing facial features, posture features, etc. An optical image of an object is generated by a lens and projected onto a photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some examples, electronic device 100 may include one or more cameras 193.

[0101] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0102] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0103] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0104] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0105] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0106] Audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Audio module 170 can also be used for encoding and decoding audio signals. In some examples, audio module 170 may be located in processor 110, or some functional modules of audio module 170 may be located in processor 110.

[0107] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0108] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0109] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some other examples, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0110] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0111] The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0112] The pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some examples, the pressure sensor 180A can be located on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to the display screen 194, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device 100 can also calculate the touch position based on the detection signal from the pressure sensor 180A. In some examples, touch operations applied to the same touch position but with different intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS message is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS message is executed.

[0113] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some examples, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 around three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the electronic device 100 is used to collect user feature information in the environment, the gyroscope sensor 180B detects the angle of the electronic device 100's shake, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 by moving in the opposite direction, thus achieving image stabilization.

[0114] The barometric pressure sensor 180C is used to measure air pressure. In some examples, the electronic device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0115] The 180E accelerometer can detect the magnitude of acceleration of electronic device 100 in various directions (typically three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices, and can be applied to applications such as screen orientation switching and pedometers.

[0116] A distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser. In some examples, when using electronic devices to collect user characteristic information about users in the environment, electronic device 100 can use the distance sensor 180F to measure distance for rapid focusing.

[0117] An ambient light sensor 180L is used to sense the ambient light intensity. Electronic device 100 can adaptively adjust the brightness of display screen 194 according to the sensed ambient light intensity.

[0118] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0119] Temperature sensor 180J is used to detect temperature. In some examples, electronic device 100 uses the temperature detected by temperature sensor 180J to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, electronic device 100 performs thermal protection by reducing the performance of a processor located near temperature sensor 180J to reduce power consumption. In other embodiments, when the temperature is below another threshold, electronic device 100 heats battery 142 to prevent abnormal shutdown of electronic device 100 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, electronic device 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0120] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0121] Buttons 190 include a power button, volume buttons, and an input keypad. Buttons 190 can be mechanical buttons or touch-sensitive buttons. The electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of the electronic device 100.

[0122] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations applied to different applications (such as video playback, audio playback, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations applied to different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0123] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0124] The following is based on Figure 1 The system architecture shown and Figure 2 The hardware structure of the electronic device is shown, and the technical solutions provided in the embodiments of this application are described in detail with reference to the accompanying drawings.

[0125] (1) Configure operator merging rules and operator swapping rules

[0126] In this scheme, the operator merging rule can be understood as the rule for merging operators in multiple ETL services, and the operator swapping rule can be understood as the rule for swapping the order of operators corresponding to one ETL service among multiple ETL services. For example, multiple ETL services include service A and service B. Service A includes operators a1 and b1, and service B includes operators a2 and b2. Then the operator merging rule can be the rule for merging a1 and a2, and the operator swapping rule can be the rule for swapping the order of a1 and b1. Furthermore, if merging a1 and a2 results in two operators c1 and c2, where c1 is the operator corresponding to service A and c2 is the operator corresponding to service B, then the operator swapping rule can be the rule for swapping the order of c1 and b1.

[0127] In one example, both operator merging rules and operator swapping rules can be pre-configured. This can be done manually or automatically by electronic devices, without limitation. For instance, during configuration, a set of feature metrics for each operator in the operator pool of the ETL business system can be pre-configured. This set of feature metrics may include the operator name, operator dimension, operator calculation formula, etc. Then, semantic analysis is performed on the feature metrics sets of each operator. Finally, based on the semantic analysis results, operator merging rules and operator swapping rules are configured.

[0128] The rules for merging operators and exchanging operators are described below.

[0129] a. Operator merging rules

[0130] In this scheme, the operator merging rule can be a merging rule constructed for multiple operators of the same type. Its purpose is to generate new equivalent operators by modifying the computational logic of two operators to be merged, thereby reducing the redundancy of computational logic between ETL processes. Specifically, when merging operators, a set of feature indicators corresponding to the common computational logic can be extracted from the operators to be merged to form a common operator. Based on the common operator, the feature indicator sets of the operators before merging are compared. While keeping the output of the operators before merging unchanged, the feature indicator sets of the operators before merging are modified to form branch operators.

[0131] It is understandable that since the read operators in different ETL services are all used to read data, the read operators in different ETL services can be directly merged.

[0132] It is understandable that a common operator and multiple branch operators can be serialized to form a forked operator, whose input and output are equivalent to the input and output of the operators before merging.

[0133] In this scheme, multiple single-input operators, multiple multi-input operators, and multiple multi-output operators can be merged. To facilitate understanding of the changes in operators before and after merging, the following sections explain the merging of multiple single-input operators, multiple multi-input operators, and multiple multi-output operators respectively.

[0134] like Figure 3a As shown, this illustrates the changes when merging two single-input operators. The merged operation produces a common operator and two branch operators, where branch operator 1 corresponds to operator 1 before merging, and branch operator 2 corresponds to operator 2 before merging. For example... Figure 3b As shown, this illustrates the changes when merging two multi-input operators. The merged operation produces a common operator and two branch operators, where branch operator 1 corresponds to operator 1 before merging, and branch operator 2 corresponds to operator 2 before merging. For example... Figure 3c As shown, this illustrates the changes when merging two multi-output operators. The merged operation produces one common operator and two branch operators. Branch operator 1 corresponds to operator 1 before the merge, and branch operator 2 corresponds to operator 2 before the merge. Figure 3a , 3b Each operator in 3c has a corresponding set of feature indices. The fork-shaped operator mentioned above can be understood as... Figure 3a The common operator is composed of branch operator 1 and branch operator 2.

[0135] To facilitate understanding, the following table describes a rule table for merging operators in this scheme. As shown in Table 1, when the two operators to be merged are of type aggregate, the feature index set of the operators before merging is groupby(dim3+dim4)sum(f(dim1)), groupby(dim3+dim5), sum(g(dim2)). The merging condition for these two operators is empty, meaning they can be merged directly. After merging, a common operator and two branch operators are formed. The feature index set of the common operator is groupby(dim3+dim4+dim5), sum(f(dim1)), sum(g(dim2)), and the feature index set of the branch operators is groupby(dim3+dim4)sum(f(dim1)), groupby(dim3+dim5), sum(g(dim2)).

[0136] Table 1

[0137]

[0138]

[0139] The specific meanings of the symbols in Table 1 are as follows:

[0140] Columns: Operators operate on fields in the input table.

[0141] Sum: Summes the corresponding fields. It is used in aggregation operators and operates on the aggregation dimension.

[0142] Select: Select the corresponding fields from the input table.

[0143] Condition: The filter expression for the corresponding field.

[0144] Jointype: refers to the type of join operator, common ones are leftjoin, innerjoin, and crossjoin.

[0145] dim1, dim2, dim3, and dim4 represent dimensions in telecommunications services, such as username, protocol type, user type, and cell.

[0146] b. Operator commutation rules

[0147] In this scheme, the operator exchange rule can be a rule constructed for two serial operators. The two serial operators are the preceding and following operators in a serial structure, and in this scheme, the output of the preceding operator is the input of the following operator. In this scheme, by exchanging the order (or sequence) of the preceding and following operators, and by modifying the feature index sets of the preceding and following operators, the overall input and output of the two operators can remain unchanged. For example, as shown... Figure 4 As shown, after the preceding and following operators satisfy the operator exchange condition, the order of the preceding and following operators can be swapped, and the feature index sets of the preceding and following operators can be modified at the same time, so that the overall input and output of the two operators remain unchanged after the order of the preceding and following operators is swapped.

[0148] To facilitate understanding, a rule table for operator exchange in this scheme is introduced below. This rule table consists of Table 2 and Table 3. The row containing the operator before the exchange in Table 2 corresponds to the row containing the operator after the exchange in Table 3. As shown in Table 2, before the exchange: if the feature index set of the former operator is mapping(f(dim1)→dim2), and the feature index set of the latter operator is mapping(g(dim2)→dim3); it can be seen from Table 3 that the exchange condition for these two operators is empty, so they can be directly exchanged; after the exchange, as shown in Table 3, the feature index set of the latter operator is mapping(fusion(f,g)), and the feature index set of the former operator is empty, that is, after the exchange, one operator can be used to replace the two operators. As shown in Table 2, before the exchange: if the feature index set of the former operator is mapping(f(dim1)→dim2), and the feature index set of the former operator is empty, then the exchange condition for these two operators is empty. 11 → dim 21 ,g(dim 12 → dim 22 The feature index set of the post-operator is aggregate(sum(h(dim)). 21 )),group(dim 22 As shown in Table 3, the exchange condition for these two operators is "the g function is one-to-one". Therefore, when the first and second operators satisfy the exchange condition "the g function is one-to-one", as shown in Table 3, the index sets after the exchange are as follows: The feature index set of the second operator is aggregate(sum(h(f(dim)). 11 )))→dim 31 ,group(dim 12 → dim 32 The feature index set of the pre-operator is mapping(dim) 31 ,g(dim 32 )).

[0149] Table 2

[0150]

[0151]

[0152] The specific symbols in the table are explained as follows:

[0153] dim L dim L1 dim represents the first input (also known as the left input) of a two-input operator. R dim R1 This represents the second input (also known as the right input) of a two-input operator.

[0154] Other definitions are shown in Table 1.

[0155] Table 3

[0156]

[0157]

[0158] The specific symbols in the table are explained as follows:

[0159] Fusion(f,g) is a composite function of functions f and g, equivalent to f(g(.)).

[0160] L and R represent the left and right input tables of the two input operators, respectively.

[0161] Other definitions are provided in Tables 1 and 2.

[0162] Understandably, in this solution, once the operator merging rules and operator exchange rules are configured, multiple ETL services can be merged.

[0163] (2) The user issues a merge request for the ETL service.

[0164] In this solution, the ETL service system can be configured on an electronic device. Users can then select the ETL services to be merged through the user interface on the electronic device. Users can select via, but is not limited to, touch or voice control. For example,... Figure 5As shown, the display screen of electronic device A displays the operation interface of the ETL business system. This interface shows various ETL services, such as service a, service b, and service c. The user can then select to merge services a and b from the ETL services. After selecting services a and b, clicking the "Confirm" button on the display screen completes the selection. At this point, the user has issued a merge request for services a and b from the ETL services. In one example, the merge request issued by the user can be used to instruct the merging of at least two ETL services, for example, merging... Figure 5 The merge request can include identifiers of the ETL services to be merged, for example, when the user selects to merge. Figure 5 If business a and business b are involved, then the merge request can carry the identifiers of business a and business b.

[0165] It is understandable that the ETL business system can also be configured on other devices in this solution, and no restrictions are imposed here.

[0166] (3) Determine whether the ETL services selected by the user can be merged.

[0167] After a user submits a merge request, the system can analyze at least two ETL services selected by the user for merging to determine if these ETL services can be merged. In one example, it can be determined whether the data sources of the selected ETL services are the same. If they are the same, it indicates that the inputs of each ETL service are identical, and they can be merged, allowing the process to proceed to the next step. If they are different, it indicates that the inputs of each ETL service are different, and they cannot be merged, thus ending the process. For example, in this solution, the data source can be a table in a database, such as a table showing the number of network outages in telecommunications services. Therefore, the system can determine whether the data sources of the various ETL services are the same based on the data source name or table name.

[0168] For example, if a user selects services a, b, and c as the ETL services to be merged, services a, b, and c can be merged if their data sources are the same; otherwise, services a, b, and c cannot be merged, or only two services with the same data source, such as services a and b, can be merged.

[0169] (4) Mark the operator depth of each operator in each ETL process.

[0170] After determining that at least two ETL services selected by the user can be merged, the operator depth of each operator within each ETL service can be marked. In one example, a directed acyclic graph (DAG) of the operators in each ETL service can be constructed, such as based on the data flow direction; during the construction of the DAG for the operators, the operator depth can be marked. For example, such as... Figure 6 As shown, in Figure 6 In the DAG of the constructed operators shown, operator 1 is the first operator (i.e., the read operator) of the ETL service, and operator 5 is the last operator (i.e., the write operator) of the ETL service. If the operator depth of operator 1 is marked as 0, then the operator depth of operator 2 is 1, the operator depth of operator 3 is 2, the operator depth of operator 4 is 3, and the operator depth of operator 5 is 4.

[0171] (5) Merge multiple ETL services

[0172] In this scheme, when merging multiple ETL services, operators with the same operator depth from the multiple ETL services are merged. Therefore, once the operator depth of each operator in each ETL service is marked, the merging of multiple ETL services can be performed.

[0173] When merging multiple ETL services, the pre-configured operator merging rules described above can be used to determine whether operators with the same depth in the ETL services to be merged can be merged. As described in the operator merging rules, read operators of the ETL services to be merged can be directly merged. Therefore, the read operators of the ETL services to be merged can be merged directly. Then, based on the operator depth, the operators with the same depth in the ETL services to be merged are judged sequentially from smallest to largest to see if they meet the merging conditions in the operator merging rules. If they do, they can be merged; otherwise, they cannot be merged. In this solution, when operators with the same depth in the ETL services to be merged do not meet the merging conditions in the operator merging rules, the process can be terminated, i.e., the merging operation ends.

[0174] In one example, when merging operators, if only one common operator is obtained, it can be directly determined whether operators at the next operator depth can be merged, and if so, the merging can be performed. For example, continuing to refer to Table 1 above, when the operator type of the two operators to be merged is mapping, after merging these two operators, only one common operator is obtained, and no branch operator is obtained. Therefore, at this time, the merging of operators at the next operator depth (such as operators of type aggregate) can begin.

[0175] In one example, when merging operators, if a common operator and multiple branch operators are obtained, the order of the resulting branch operator and the next operator depth operator in the corresponding ETL service can be swapped based on the pre-configured operator swapping rules. If they can be swapped, their order is swapped. For example, referring to Table 1 above, when the operator type of the two operators to be merged is `aggregate`, merging these two operators yields a common operator and two branch operators. If the type of the next operator depth operator of one of the branch operators is `filter`, referring to Table 2 above, the preceding operator is the branch operator, and the following operator is the next operator depth operator of the branch operator. Referring to Table 3 above, the swapping condition for these two operators is empty; therefore, their order can be swapped.

[0176] Furthermore, after swapping the order of the pre-operator and post-operator, the post-operator can be merged based on the operator merging rules. Here, the pre-operator can be understood as the branch operator generated after the above-mentioned merged operator, and the post-operator can be understood as the operator at the next operator depth in the ETL business corresponding to the branch operator.

[0177] Understandably, in this scheme, if the order of the two operators satisfies the exchange conditions in the operator exchange rules, the process can be terminated, i.e., the merging work can be ended. At this point, the obtained common operators and branch operators, as well as the unmerged operators in the ETL business corresponding to the branch operators, can constitute the merged ETL business.

[0178] To help you understand the merging process, we will use an example below.

[0179] like Figure 7a As shown, Figure 7a Consider the models of two ETL services to be merged, namely services a and b. These two ETL services share the same data source, therefore they can be merged. If merging operators a1 and b1 results in a common operator 1, but no branch operators are obtained, then after merging operators a1 and b1, the following can be obtained: Figure 7b The ETL business model shown is illustrated. Figure 7b In the process, operators a2 and b2 can be combined. After combining operators a2 and b2, the result is as follows: Figure 7c The ETL business model shown, where merging operators a2 and b2 yields common operator 2, branch operator 21, and branch operator 22, allows us to determine whether branch operator 21 and operator a3 can be interchanged, and whether branch operator 22 and operator b3 can be interchanged. If they can be interchanged, their order is swapped, resulting in the following... Figure 7d The ETL business model shown is illustrated. Figure 7dIn the process, operators a3 and b3 can be combined. After combining operators a3 and b3, the following can be obtained: Figure 7e The ETL business model shown, where operators a3 and b3 are merged, yields common operator 3, branch operator 31, and branch operator 32. At this point, it can be determined whether branch operator 31 and branch operator 21 can be swapped, and whether branch operator 32 and branch operator 22 can be swapped. If they cannot be swapped, the process ends, i.e., the merging operation is terminated. Figure 7e The ETL business model shown can be used to construct the model of the merged ETL business.

[0180] It should be noted that, in this scheme, in order to avoid continuous merging and / or continuous swapping of operators of the same type, the process can be terminated when the number of merging of operators of the same type reaches a preset threshold, and / or the number of swapping of operators of the same type reaches a preset threshold.

[0181] To make it easier to understand, the merging process will be described below with a more vivid example.

[0182] a. When merging multiple ETL services, the set of sub-operators of the read operators of each ETL service to be merged can be denoted as UnderMatched. UnderMatched is then partitioned into pairs of maximum mergeable elements: UnderMatched = M1∪M2∪…M k In this system, sub-operators of two mergeable read operators form a set M; sub-operators of read operators without mergeable operators individually form a set M. The maximum mergeable pairwise partition can be understood as the partition that minimizes the number of sets M. For example, the ETL services to be merged include services a, b, c, d, and e, and the set of sub-operators of the read operators of service a is F. a The set of suboperators for the read operator of business b is F. b The set of suboperators for the read operator of business c is F. c The set of suboperators for the read operator of business d is F. d The set of suboperators for the read operator of business e is F. e Then, after performing the maximum pairwise partitioning, it becomes: F a and F b It can be a combination M, F c and F d It can be a combination M, F e It can be a combination M on its own.

[0183] Set M i It can contain operator O i1 , For the operator O i1 and For M, two ETL services can be merged; i It contains only one operator O i Then the corresponding inclusion operator O i ETL operations can only be merged into read operators.

[0184] b. After performing the maximum mergeable pairwise partitioning on the set of sub-operators of the read operator for each ETL service to be merged, a set M can be... i Operator O included i1 , Merge.

[0185] First, the combineReader(R) operator. i1 ,R i2 Generate a new DAG:

[0186]

[0187] The depth of the Reader operator is 0, which can be expressed as i. The depth of the sub-operators increases sequentially, so the depth of the mapping operator is expressed as i1.

[0188] c. Based on the merging conditions of the mapping operator in the pre-configured operator branching rules, it can be seen that mapping merging is unconditional; therefore, the operator mapping... i1 , They can be merged. The merged DAG is:

[0189]

[0190] d. Determine the mapping cob Does it need to be swapped? This depends on the merged mapping. cob Since no branching operators are included, it is determined that no swap is needed.

[0191] c. For mapping i1 , The operators for subsequent ETL operations are then merged.

[0192] d. If, according to the operator merging rules, it is determined that operators of type aggregate can be merged, then the aggregate operator... i2 , Perform a merge, that is:

[0193]

[0194] The new ETL business after the merger is as follows:

[0195]

[0196] e. Determining the sub-operator aggregate ib and filter i3 Can the (post-operator) be commutative? Based on Tables 2 and 3 described above, regarding the operator aggregate... ib →filter i3 Execute: sink(aggregate) ib ,filter i3 ), for operators implement: That is, the aggregation operator ib and filter i3 The order, and the commutation operator and order

[0197] The new ETL service generated after the exchange order is as follows:

[0198]

[0199] f. Recursively execute steps c, d, and e until the output operator Writer (i.e., the write operator). Since the writer is the last operator in the ETL process, it cannot be merged or swapped.

[0200] g. Correction: For the merged ETL service DAG output in step e, adjust the order of operators based on the sinking and floating rules. The sinking and floating rules are as follows: expansion operators sink towards the end of the path (i.e., the direction of the write operator), and contraction operators float towards the beginning of the path (i.e., the direction of the read operator). After this, the corrected ETL service DAG can be output.

[0201] (6) Determine the execution cost of the ETL service generated after each merging operator.

[0202] In this scheme, the execution cost of the resulting ETL service can be determined after each merging operator. The reference dimension for the execution cost can be any business bottleneck dimension, such as execution time, peak I / O, memory usage, etc.

[0203] Furthermore, from the ETL services formed after merging operators, the ETL service with the lowest execution cost is selected as the ETL service obtained by merging multiple ETL services. Since the ETL service obtained by merging multiple ETL services includes common operators and branch operators, and the data output by the common operator with the largest operator depth is the input data of the branch operators corresponding to each ETL service before merging, a cache operator can be set between the common operator with the largest operator depth and the branch operators to facilitate the branch operators obtaining input data. This cache operator can cache the output data of the common operator and provide data to the branch operators. In this scheme, the ETL service with the cache operator can be used as the merged ETL service. For example, as shown... Figure 8 As shown, Figure 8 A schematic diagram of a merged ETL service model is shown. This ETL service is obtained by merging ETL service a and ETL service b. Its operators consist of common operators 1 and 2, cache operator 3, and branch operators 41, 42, 43, 51, 52 and 53. Among them, common operators 1 and 2 can be understood as the operators that correspond to both ETL services a and b. The string of operators composed of branch operators 41, 42 and 43 can be understood as the operators corresponding to ETL service a. The string of operators composed of branch operators 51, 52 and 53 can be understood as the operators corresponding to ETL service b.

[0204] In one example, the execution cost of the ETL process resulting from each merge operator, and the location for setting up cache operators, can be determined in the following way:

[0205] First, for each ETL transaction obtained by the merging operator, let CS denote the set of common operators. common ={C1,C2,…,C c The set of branch operators is BS. branch ={B1,B2,…,B b The set of operators added compared to the original ETL business is AS = {A1, A2, ..., A}. a The execution cost of this ETL service is then...

[0206]

[0207] Wherein, λ, ξ, θ O The execution cost per unit of data for the pre-configured corresponding operator. The set of branch operators BS. branch AS can be understood as the set of operators that have not been merged. The set of operators added compared to the original ETL business can be understood as the set of branch operators generated after the operators are merged.

[0208] Secondly, the set of common operators for the ETL service formed after multiple operator merging processes can be denoted as OS. common ={O 1c O 2c ,…,O kc The operator depth of the corresponding common operators is 1, 2, ..., k, that is, a total of k common operators are obtained.

[0209] Furthermore, when the operator depth of the common operators reaches j (1≤j≤k), operator merging stops, and the merged ETL generated at this point is denoted as ETL. j Then the optimal cache point O cache The cache point that minimizes the execution cost of the merged ETL process:

[0210]

[0211] Finally, in the common operator O cache Then, a caching operator is set up to form the ETL process with the minimum execution cost after merging ETL processes, denoted as ETL. cache .

[0212] (7) Determine whether to merge multiple ETL transactions.

[0213] This solution allows for cost analysis of the execution costs of multiple ETL services before the merger and the execution costs of the merged ETL service with cached operators, ultimately determining whether to proceed with the merger. Specifically:

[0214] a. If the sum of the execution costs of multiple ETL services before the merger is greater than the execution cost of the merged ETL service with cached operators, it indicates that merging multiple ETL services can significantly reduce the computational logic redundancy and read / write redundancy of the original ETL services. Therefore, multiple ETL services can be merged, i.e., the decision to merge multiple ETL services is executed, and ultimately multiple ETL services become a single ETL service with a single read and multiple write operations (e.g., ...). Figure 8 (The ETL process shown in the image).

[0215] b. If the sum of the execution costs of the multiple ETL services before the merger is less than the execution cost of the merged ETL service with cached operators, it indicates that merging the multiple ETL services has not significantly reduced the computational logic redundancy, read-write redundancy, etc., of the original ETL services. Therefore, merging is not necessary. However, in order to reduce read-write redundancy, only the read operators of the multiple ETL services can be merged, without merging the other operators. That is, the decision to merge multiple ETL services is executed, and ultimately the multiple ETL services become a single ETL service with a single read and multiple write operations.

[0216] c. If the sum of the execution costs of multiple ETL services before the merger equals the execution cost of the merged ETL service with cached operators, it indicates that merging multiple ETL services failed to significantly reduce the computational redundancy and read / write redundancy of the original ETL services. Although the execution costs of ETL services are the same before and after the merger, combining multiple ETL services into one ETL service can provide significant advantages in task scheduling, distribution, and management. Therefore, in this case, multiple ETL services can also be merged, i.e., the decision to merge multiple ETL services is executed, ultimately transforming multiple ETL services into a single ETL service with a single read operation and multiple write operations.

[0217] In one example, such as Figure 9 As shown, it illustrates a schematic diagram of an ETL service model involving a single data read and multiple data writes, by... Figure 9 As can be seen, the merged ETL service has one read operator, n common operators, one cache operator, m branch operators, and m write operators. Before the merger, each ETL service could correspond to a branch operator and a write operator connected to that branch operator. It can be understood that in this scheme, the read or cache operators in the merged ETL service can also be called common operators, and the write operators can also be called branch operators. In other words, in the merged ETL service, the operators commonly corresponding to the multiple ETL services before the merger can be called common operators, and the operators not commonly corresponding to the multiple ETL services before the merger can be called branch operators. Each ETL service before the merger could correspond to a branch operator. The branch operator mentioned in this scheme can be understood as multiple branch operators connected serially and having mutual relationships.

[0218] Understandably, when multiple ETL processes are merged into a single ETL process involving a single read and multiple write operations, the ETL business model within the ETL system will change. The number of ETL processes will decrease; that is, the multiple ETL processes before the merger will be removed, and the merged ETL process will be retained. Subsequent data processing will then utilize this merged ETL process.

[0219] Next, based on the merging process of multiple ETL services described above, an optimization method for ETL services provided in this application embodiment will be introduced. It is understood that this method is another expression of the merging process of multiple ETL services described above, and the two are combined. This method is proposed based on the merging process of multiple ETL services described above, and some or all of its content can be found in the above description of the merging process of multiple ETL services.

[0220] Please see Figure 10 , Figure 10 This is a flowchart illustrating an ETL service optimization method provided in an embodiment of this application. It is understood that this method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities.

[0221] like Figure 10 As shown, the optimization methods for this ETL service include:

[0222] Step S101: Receive a merge request sent by the user. The merge request is used to indicate the merging of at least two ETL services, including the first ETL service and the second ETL service.

[0223] Step S102: Determine whether the data sources for the first ETL service and the second ETL service are the same.

[0224] In this scheme, if the data sources of the ETL services to be merged are the same, then step S103 is executed; otherwise, step S104 is executed.

[0225] Step S103: Based on the pre-configured operator merging rules, merge the first ETL service and the second ETL service to obtain the target ETL service. The target ETL service is a service that reads data once and writes data multiple times. The multiple write data in the target ETL service includes the write data of the first ETL service and the write data of the second ETL service.

[0226] Step S104: End the merge.

[0227] It is understood that some or all of the descriptions in the methods provided in this solution can be found in the descriptions above, and will not be repeated here.

[0228] In one example, such as Figure 11 As shown, merging the first ETL service and the second ETL service based on pre-configured operator merging rules may include the following steps:

[0229] Step S201: Determine the operator depth of each operator in the first ETL service. The operator depth is used to characterize the interval between the first operator and the first read operator in the first ETL service. The first read operator is used to read data. Each operator includes the first operator, and the first operator includes the first read operator.

[0230] Step S202: Based on the operator depth order, from smallest to largest, merge the first operator in the first ETL service and the second operator in the second ETL service according to the operator merging rules to obtain the target operator. The first operator and the second operator have the same operator depth.

[0231] It is understood that some or all of the descriptions in the methods provided in this solution can be found in the descriptions above, and will not be repeated here.

[0232] In one example, when the target operator includes a common operator, the third operator in the first ETL service and the fourth operator in the second ETL service can be merged based on the operator merging rule. The third operator is adjacent to the first operator, and the fourth operator is adjacent to the second operator. The common operator is the operator that corresponds to both the first ETL service and the second ETL service in the target ETL service.

[0233] When the target operator includes a common operator, a first branch operator, and a second branch operator, the order of the first target operator and the second target operator can be swapped based on a pre-configured operator swapping rule. The common operator is the operator that corresponds to both the first ETL service and the second ETL service in the target ETL service. The first branch operator is the operator corresponding to the first ETL service in the target ETL service. The second branch operator is the operator corresponding to the second ETL service in the target ETL service. The first target operator includes at least one of the first branch operator and the second branch operator. The second target operator is either the third operator in the first ETL service or the fourth operator in the second ETL service. The third operator is adjacent to the first operator, and the fourth operator is adjacent to the second operator. The first target operator and the second target operator correspond to the same ETL service.

[0234] Furthermore, after swapping the order of the first and second target operators, the third and fourth operators are merged based on the operator merging rules.

[0235] It is understandable that after multiple operator swaps, newly generated branch operators may be followed by other branch operators, for example, as shown in the example. Figure 7e As shown, branch operator 31 is followed by branch operator 21. In this case, the third and fourth operators can be newly generated branch operators followed by branch operators.

[0236] In one example, if the first operator and the second operator cannot be merged, or the first target operator and the second target operator cannot be swapped, or the number of target operations exceeds a preset threshold, the number of target operations includes at least one of the number of merges of operators of the same type and the number of swaps, then the merging of the first ETL service and the second ETL service is terminated.

[0237] Furthermore, based on the operator depth, the obtained target operator can be combined with the unmerged operators in the first ETL service to obtain the first branch ETL service, and the obtained target operator can be combined with the unmerged operators in the second ETL service to obtain the second branch ETL service. The target ETL service includes the first branch ETL service and the second branch ETL service.

[0238] In one example, upon obtaining the target operator, the execution cost of the third ETL service can be determined. The third ETL service consists of the obtained target operator and the unmerged operators from the first or second ETL service. Then, based on the execution cost of the third ETL service, the target ETL service is determined.

[0239] As one possible implementation, when there are multiple third ETL services, if the execution cost of the fourth ETL service is less than or equal to the sum of the execution costs of the first ETL service and the second ETL service, then the fourth ETL service is taken as the target ETL service, where the fourth ETL service is the third ETL service with the lowest execution cost.

[0240] Furthermore, taking the fourth ETL service as the target ETL service can include: setting a cache operator after the common operator contained in the third target operator in the fourth ETL service to obtain the target ETL service. The third target operator is the target operator with the largest operator depth in the fourth ETL service. The cache operator is used to cache the data output by the common operator and to provide data to the branch operators after the common operator.

[0241] In one example, after obtaining the target ETL process, it can be modified based on the sinking and floating rules. The sinking and floating rules are as follows: the shrinking operator floats upwards towards the head of the ETL process, and the expanding operator sinks downwards towards the tail of the ETL process. Here, the shrinking operator is the one that reduces the amount of data after processing, and the expanding operator is the one that increases the amount of data after processing.

[0242] In one example, the target ETL service includes a common operator, a cache operator, and at least two branch operators. The cache operator is located between the common operator and the at least two branch operators. The cache operator is used to cache the data output by the common operator and to provide data to the first branch operator. The common operator includes a read operator, and the at least two branch operators include a first branch operator. The first branch operator includes the write operator for the first ETL service; the read operator is used to read data, and the write operator is used to write data.

[0243] It is understood that some or all of the descriptions in the methods provided in this solution can be found in the descriptions above, and will not be repeated here.

[0244] Based on the methods in the above embodiments, this application provides an optimization apparatus for ETL services. Please refer to... Figure 12 , Figure 12 This is a schematic diagram of the structure of an ETL service optimization device provided in an embodiment of this application. Figure 12 As shown, the ETL service optimization device 1200 includes a receiving module 1201 and a processing module 1202. The receiving module 1201 receives a merging request from a user, indicating the merging of at least two ETL services, including a first ETL service and a second ETL service. The processing module 1202 determines whether the data sources of the first and second ETL services are the same. If they are the same, it merges the first and second ETL services based on pre-configured operator merging rules to obtain a target ETL service. The target ETL service is a service involving a single read and multiple write operations, where the multiple write operations include write data from both the first and second ETL services.

[0245] In one example, the processing module 1202 is further configured to: determine the operator depth of each operator in the first ETL service, wherein the operator depth is used to characterize the interval between the first operator and the first read operator in the first ETL service, wherein the first read operator is used to read data, and each operator includes the first operator, and the first operator includes the first read operator; and according to the order of the operator depth, merge the first operator in the first ETL service and the second operator in the second ETL service in order from smallest to largest based on the operator merging rules to obtain the target operator, wherein the first operator and the second operator have the same operator depth.

[0246] In one example, the processing module 1202 is further configured to: when the target operator includes a common operator, merge the third operator in the first ETL service and the fourth operator in the second ETL service based on the operator merging rule, wherein the third operator is adjacent to the first operator and the fourth operator is adjacent to the second operator, wherein the common operator is the operator that is commonly corresponding to the first ETL service and the second ETL service in the target ETL service.

[0247] In one example, the processing module 1202 is further configured to: when the target operator includes a common operator, a first branch operator, and a second branch operator, swap the order of the first target operator and the second target operator based on a pre-configured operator swapping rule, wherein the common operator is the operator commonly corresponding to the first ETL service and the second ETL service in the target ETL service, the first branch operator is the operator corresponding to the first ETL service in the target ETL service, the second branch operator is the operator corresponding to the second ETL service in the target ETL service, the first target operator includes at least one of the first branch operator and the second branch operator, the second target operator is the third operator in the first ETL service or the fourth operator in the second ETL service, the third operator is adjacent to the first operator, the fourth operator is adjacent to the second operator, and the first target operator and the second target operator correspond to the same ETL service; after swapping the order of the first target operator and the second target operator, merge the third operator and the fourth operator based on the operator merging rule.

[0248] In one example, the processing module 1202 is further configured to: terminate the merging of the first ETL service and the second ETL service when the first operator and the second operator cannot be merged, or the first target operator and the second target operator cannot be swapped, or the number of target operations is greater than a preset threshold, wherein the number of target operations includes at least one of the number of merging operations of operators of the same type and the number of swapping operations; and, based on the operator depth, combine the obtained target operator with the unmerged operators in the first ETL service to obtain the first branch ETL service, and combine the obtained target operator with the unmerged operators in the second ETL service to obtain the second branch ETL service, wherein the target ETL service includes the first branch ETL service and the second branch ETL service.

[0249] In one example, the processing module 1202 is also used to: determine the execution cost of the third ETL service when the target operator is obtained, wherein the third ETL service consists of the obtained target operator and the unmerged operators in the first ETL service or the second ETL service.

[0250] The target ETL service is determined based on the execution cost of the third ETL service.

[0251] In one example, the processing module 1202 is further configured to: when there are multiple third ETL services, if the execution cost of the fourth ETL service is less than or equal to the sum of the execution costs of the first ETL service and the second ETL service, then the fourth ETL service is selected as the target ETL service, and the fourth ETL service is the third ETL service with the lowest execution cost.

[0252] In one example, the processing module 1202 is also used to: set a cache operator after the common operator contained in the third target operator in the fourth ETL service to obtain the target ETL service. The third target operator is the target operator with the largest operator depth in the fourth ETL service. The cache operator is used to cache the data output by the common operator and to provide data to the branch operators after the common operator.

[0253] In one example, the processing module 1202 is also used to: after obtaining the target ETL service, modify the target ETL service based on the sinking and floating rules, which are: the shrinking operator floats upward to the head of the ETL service, and the expanding operator sinks downward to the tail of the ETL service; wherein, the shrinking operator is the operator that reduces the amount of data after processing, and the expanding operator is the operator that increases the amount of data after processing.

[0254] In one example, the target ETL service includes a common operator, a cache operator, and at least two sets of branch operators. The cache operator is located between the common operator and the at least two sets of branch operators. The cache operator is used to cache the data output by the common operator and to provide data to the first set of branch operators. The common operator includes a read operator, and the at least two sets of branch operators include a first set of branch operators. The first set of branch operators includes a write operator for the first ETL service. The read operator is used to read data, and the write operator is used to write data.

[0255] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0256] Based on the methods described in the above embodiments, this application provides an electronic device. Please refer to... Figure 13 , Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 13 As shown in the embodiments of this application, the electronic device provided can be used to implement the methods described in the above method embodiments.

[0257] The electronic device includes at least one processor 1301, which can support the electronic device in implementing the methods provided in the embodiments of this application.

[0258] The processor 1301 can be a general-purpose processor or a special-purpose processor. For example, the processor 1301 may include a central processing unit (CPU) and / or a baseband processor. The baseband processor can be used to process communication data (e.g., determine the target screen terminal), and the CPU can be used to implement corresponding control and processing functions, execute software programs, and process data from the software programs.

[0259] Furthermore, the electronic device may also include a transceiver unit 1305 for receiving and transmitting signals. For example, the transceiver unit 1305 may include a transceiver or an RF chip. The transceiver unit 1305 may also include a communication interface.

[0260] Optionally, the electronic device may also include an antenna 1306, which can be used to support the transceiver unit 1305 in realizing the transceiver function of the electronic device.

[0261] Optionally, the electronic device may include one or more memories 1302 storing a program (or instructions or code) 1304. The program 1304 can be executed by a processor 1301, causing the processor 1301 to perform the methods described in the above method embodiments. Optionally, the memory 1302 may also store data. Optionally, the processor 1301 may also read data stored in the memory 1302 (e.g., pre-stored first feature information), which may be stored at the same storage address as the program 1304, or it may be stored at a different storage address than the program 1304.

[0262] The processor 1301 and memory 1302 can be configured separately or integrated together, for example, integrated on a single board or system on chip (SOC).

[0263] For a detailed description of the operations performed by the electronic device in the various possible designs described above, please refer to the description in the embodiments of the methods provided in this application, and will not be repeated here.

[0264] Based on the methods described in the above embodiments, this application also provides a chip. Please refer to... Figure 14 , Figure 14 This is a schematic diagram of a chip structure provided in an embodiment of this application. Figure 14 As shown, chip 1400 includes one or more processors 1401 and interface circuitry 1402. Optionally, chip 1400 may also include a bus 1403. Wherein:

[0265] Processor 1401 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed through integrated logic circuits in the hardware of processor 1401 or through software instructions. Processor 1401 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods and steps disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0266] The interface circuit 1402 can be used to send or receive data, instructions or information. The processor 1401 can use the data, instructions or other information received by the interface circuit 1402 to process the data, instructions or other information, and can send the processed information out through the interface circuit 1402.

[0267] Optionally, the chip may also include memory, which may include read-only memory and random access memory, and provide operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0268] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which may be stored in the operating system).

[0269] Optionally, the interface circuit 1402 can be used to output the execution results of the processor 1401.

[0270] It should be noted that the functions of the processor 1401 and the interface circuit 1402 can be implemented through hardware design, software design, or a combination of hardware and software; no restrictions are imposed here.

[0271] It should be understood that each step of the above method embodiments can be completed by hardware logic circuits or software instructions in a processor.

[0272] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0273] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0274] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0275] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

Claims

1. A method for optimizing ETL operations using data warehouse technology, characterized in that, The method includes: Receive a merge request from a user, the merge request being used to indicate the merging of at least two ETL services, the at least two ETL services including a first ETL service and a second ETL service; Determine whether the data sources of the first ETL service and the second ETL service are the same. If they are the same, then based on the pre-configured operator merging rules, merge operators with the same operator depth in the first ETL service and the second ETL service to obtain the target ETL service. The target ETL service is a service that reads data once and writes data multiple times. The multiple write data in the target ETL service includes the write data of the first ETL service and the write data of the second ETL service. The operator merging rule refers to the rule of generating a new equivalent operator by modifying the calculation logic of the two operators to be merged. The operator depth represents the interval between the operators and the read operators in the ETL service.

2. The method according to claim 1, characterized in that, The method of merging operators with the same operator depth in the first ETL service and the second ETL service based on pre-configured operator merging rules includes: Determine the operator depth of each operator in the first ETL service. The operator depth is used to characterize the interval between the first operator and the first read operator in the first ETL service. The first read operator is used to read data. Each operator includes the first operator. In the case of merging the first read operators, the first operator is the first read operator. Based on the order of operator depth, the first operator in the first ETL service and the second operator in the second ETL service are merged sequentially according to the operator merging rules to obtain the target operator, wherein the first operator and the second operator have the same operator depth.

3. The method according to claim 2, characterized in that, Also includes: When the target operator includes a common operator, based on the operator merging rule, the third operator in the first ETL service and the fourth operator in the second ETL service are merged. The third operator is adjacent to the first operator, and the fourth operator is adjacent to the second operator. The common operator is the operator that is commonly associated with both the first ETL service and the second ETL service in the target ETL service.

4. The method according to claim 2, characterized in that, Also includes: When the target operator includes a common operator, a first branch operator, and a second branch operator, the order of the first target operator and the second target operator is exchanged based on a pre-configured operator exchange rule. The common operator is the operator commonly corresponding to both the first ETL service and the second ETL service in the target ETL service. The first branch operator is the operator corresponding to the first ETL service in the target ETL service. The second branch operator is the operator corresponding to the second ETL service in the target ETL service. The first target operator includes at least one of the first branch operator and the second branch operator. The second target operator is either the third operator in the first ETL service or the fourth operator in the second ETL service. The third operator is adjacent to the first operator, and the fourth operator is adjacent to the second operator. The first target operator and the second target operator correspond to the same ETL service. After swapping the order of the first target operator and the second target operator, the third operator and the fourth operator are merged based on the operator merging rule.

5. The method according to claim 2 or 3, characterized in that, Also includes: When the first operator and the second operator cannot be merged, or when the target number of operations exceeds a preset threshold, the target number of operations includes the number of merges of operators of the same type, and the merging of the first ETL service and the second ETL service ends. Based on the depth of the operator, the obtained target operator is combined with the unmerged operators in the first ETL service to obtain the first branch ETL service, and the obtained target operator is combined with the unmerged operators in the second ETL service to obtain the second branch ETL service, wherein the target ETL service includes the first branch ETL service and the second branch ETL service.

6. The method according to claim 4, characterized in that, Also includes: When the first target operator and the second target operator cannot be swapped, or when the number of target operations exceeds a preset threshold, the number of target operations includes the number of swaps of operators of the same type, and the merging of the first ETL service and the second ETL service ends. Based on the depth of the operator, the obtained target operator is combined with the unmerged operators in the first ETL service to obtain the first branch ETL service, and the obtained target operator is combined with the unmerged operators in the second ETL service to obtain the second branch ETL service, wherein the target ETL service includes the first branch ETL service and the second branch ETL service.

7. The method according to any one of claims 2-4, characterized in that, Also includes: When the target operator is obtained, the execution cost of the third ETL service is determined. The third ETL service is composed of the obtained target operator and the unmerged operators in the first ETL service or the second ETL service. The target ETL service is determined based on the execution cost of the third ETL service.

8. The method according to claim 7, characterized in that, Determining the target ETL service based on the execution cost of the third ETL service includes: When there are multiple third ETL services, if the execution cost of the fourth ETL service is less than or equal to the sum of the execution costs of the first ETL service and the second ETL service, then the fourth ETL service is taken as the target ETL service, and the fourth ETL service is the third ETL service with the lowest execution cost.

9. The method according to claim 8, characterized in that, The step of using the fourth ETL service as the target ETL service includes: In the fourth ETL service, a cache operator is set after the common operator included in the third target operator to obtain the target ETL service. The third target operator is the target operator with the largest operator depth in the fourth ETL service. The cache operator is used to cache the data output by the common operator and to provide data to the branch operators after the common operator.

10. The method according to any one of claims 1-4, characterized in that, After obtaining the target ETL service, the process also includes: Based on the sinking and floating rules, the target ETL service is modified. The sinking and floating rules are as follows: the shrinking operator floats upward towards the head of the ETL service, and the expanding operator sinks downward towards the tail of the ETL service. The shrinkage operator is an operator that reduces the amount of data after processing, and the expansion operator is an operator that increases the amount of data after processing.

11. The method according to any one of claims 1-4, characterized in that, The target ETL service includes a common operator, a cache operator, and at least two sets of branch operators. The cache operator is located between the common operator and the at least two sets of branch operators. The cache operator is used to cache the data output by the common operator and to provide data to the first set of branch operators. The common operator includes a read operator, and the at least two branch operators include the first branch operator, which includes the write operator for the first ETL service. The read operator is used to read data, and the write operator is used to write data.

12. An optimization device for ETL (Extract, Transform, Load) operations in data warehouse technology, characterized in that, The device includes: The receiving module is used to receive a merge request sent by a user. The merge request is used to indicate the merging of at least two ETL services, including a first ETL service and a second ETL service. The processing module is used to determine whether the data sources of the first ETL service and the second ETL service are the same. If they are the same, it merges operators with the same operator depth in the first ETL service and the second ETL service based on a pre-configured operator merging rule to obtain a target ETL service. The target ETL service is a service that reads data once and writes data multiple times. The multiple write data in the target ETL service includes the write data of the first ETL service and the write data of the second ETL service. The operator merging rule refers to the rule of generating a new equivalent operator by modifying the calculation logic of two operators to be merged. The operator depth represents the interval between the operators and the read operators in the ETL service.

13. The apparatus according to claim 12, characterized in that, The processing module is further configured to: Determine the operator depth of each operator in the first ETL service. The operator depth is used to characterize the interval between the first operator and the first read operator in the first ETL service. The first read operator is used to read data. Each operator includes the first operator. In the case of merging the first read operators, the first operator is the first read operator. Based on the order of operator depth, the first operator in the first ETL service and the second operator in the second ETL service are merged sequentially according to the operator merging rules to obtain the target operator, wherein the first operator and the second operator have the same operator depth.

14. The apparatus according to claim 13, characterized in that, The processing module is further configured to: When the target operator includes a common operator, based on the operator merging rule, the third operator in the first ETL service and the fourth operator in the second ETL service are merged. The third operator is adjacent to the first operator, and the fourth operator is adjacent to the second operator. The common operator is the operator that is commonly associated with both the first ETL service and the second ETL service in the target ETL service.

15. The apparatus according to claim 13, characterized in that, The processing module is further configured to: When the target operator includes a common operator, a first branch operator, and a second branch operator, the order of the first target operator and the second target operator is exchanged based on a pre-configured operator exchange rule. The common operator is the operator commonly corresponding to both the first ETL service and the second ETL service in the target ETL service. The first branch operator is the operator corresponding to the first ETL service in the target ETL service. The second branch operator is the operator corresponding to the second ETL service in the target ETL service. The first target operator includes at least one of the first branch operator and the second branch operator. The second target operator is either the third operator in the first ETL service or the fourth operator in the second ETL service. The third operator is adjacent to the first operator, and the fourth operator is adjacent to the second operator. The first target operator and the second target operator correspond to the same ETL service. After swapping the order of the first target operator and the second target operator, the third operator and the fourth operator are merged based on the operator merging rule.

16. The apparatus according to claim 13 or 14, characterized in that, The processing module is further configured to: When the first operator and the second operator cannot be merged, or when the target number of operations exceeds a preset threshold, the target number of operations includes the number of merges of operators of the same type, and the merging of the first ETL service and the second ETL service ends. Based on the depth of the operator, the obtained target operator is combined with the unmerged operators in the first ETL service to obtain the first branch ETL service, and the obtained target operator is combined with the unmerged operators in the second ETL service to obtain the second branch ETL service, wherein the target ETL service includes the first branch ETL service and the second branch ETL service.

17. The apparatus according to claim 15, characterized in that, The processing module is further configured to: When the first target operator and the second target operator cannot be swapped, or when the number of target operations exceeds a preset threshold, the number of target operations includes the number of swaps of operators of the same type, and the merging of the first ETL service and the second ETL service ends. Based on the depth of the operator, the obtained target operator is combined with the unmerged operators in the first ETL service to obtain the first branch ETL service, and the obtained target operator is combined with the unmerged operators in the second ETL service to obtain the second branch ETL service, wherein the target ETL service includes the first branch ETL service and the second branch ETL service.

18. The apparatus according to any one of claims 13-15, characterized in that, The processing module is further configured to: When the target operator is obtained, the execution cost of the third ETL service is determined. The third ETL service is composed of the obtained target operator and the unmerged operators in the first ETL service or the second ETL service. The target ETL service is determined based on the execution cost of the third ETL service.

19. The apparatus according to any one of claims 12-15, characterized in that, The processing module is further configured to: after obtaining the target ETL service, modify the target ETL service based on the sinking and floating rules, wherein the sinking and floating rules are: the shrinking operator floats upward towards the head of the ETL service, and the expanding operator sinks downward towards the tail of the ETL service. The shrinkage operator is an operator that reduces the amount of data after processing, and the expansion operator is an operator that increases the amount of data after processing.

20. The apparatus according to any one of claims 12-15, characterized in that, The target ETL service includes a common operator, a cache operator, and at least two sets of branch operators. The cache operator is located between the common operator and the at least two sets of branch operators. The cache operator is used to cache the data output by the common operator and to provide data to the first set of branch operators. The common operator includes a read operator, and the at least two branch operators include the first branch operator, which includes the write operator for the first ETL service. The read operator is used to read data, and the write operator is used to write data.

21. An electronic device, characterized in that, include: At least one memory for storing programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-11.

22. A computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-11.

23. A chip, characterized in that, Includes at least one processor and interface; The interface is used to provide program instructions or data to the at least one processor; The at least one processor is configured to execute the program line instructions to implement the method as described in any one of claims 1-11.

24. A computer program product comprising instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-11.

Images