Data fusion method and system for multiple data sources in cross-cloud scene

Abstract

The invention discloses a data fusion method and system for multiple data sources in a cross-cloud scene. The method comprises the following steps: acquiring data from an API (Application Program Interface), a relational database and an object storage; scheduling and pulling the data by using a distributed scheduling tool; the data size to be processed at present is judged through the self-adaptive codes, and different schemes are selected to process the data; according to the method, a developer does not need to pay attention to the data magnitude in a non-uniformly distributed data scene, and directly compiles processing logic, so that a function that a scheme is automatically selected by codes is realized; and meanwhile, the development difficulty is simplified, and the system stability risk is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data fusion method and system for multiple data sources in a cross-cloud scenario. Background Art

[0002] In projects that provide store library services to brands, it is necessary to collect various data sources and aggregate them into the store library for centralized data processing and algorithm processing to form a final standard store library for use by the client's various business departments.

[0003] The data sources are diverse and the data volume is unevenly distributed. Most tasks have a moderate amount of data cleaning, while a small number of tasks are highly complex and have a large amount of data. The existing technology has the following shortcomings: It is difficult to effectively process various types of data sources (such as APIs, relational databases, and object storage) obtained from different cloud service providers, resulting in low efficiency in data integration and processing. For tasks with large amounts of data and high complexity, the existing technology is prone to excessive system load during processing, affecting the stability and performance of the overall system. There is a lack of flexibility in data pulling and scheduling, and it is impossible to perform adaptive scheduling and processing based on the characteristics of the data source and data volume, resulting in low resource utilization.

[0004] In order to rationally utilize resources and avoid large tasks causing excessive system load and affecting stability, an adaptive data processing solution is needed to evaluate the complexity and data volume before processing, and to choose whether to use trimmed data for batch processing. Summary of the invention

[0005] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0006] In view of the above existing problems, the present invention is proposed. Therefore, the present invention provides a data fusion method for multiple data sources in a cross-cloud scenario to solve the problems mentioned in the background technology.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In a first aspect, the present invention provides a data fusion method for multiple data sources in a cross-cloud scenario, comprising:

[0009] Get data from APIs, relational databases, and object stores;

[0010] Use a distributed scheduling tool to schedule and pull the data;

[0011] Through adaptive code, the current amount of data to be processed is determined, and different solutions are selected to process the data.

[0012] As a preferred solution of the data fusion method of multiple data sources in the cross-cloud scenario described in the present invention, obtaining data from the API includes:

[0013] Python interface task code, access the data interface after authentication, obtain data and write it into the database;

[0014] When authentication succeeds, the token obtained for the first time is persisted in the database for use by other nodes in the distributed system;

[0015] Access the data interface and download files, and save the data files to the local disk;

[0016] Read the data files in the local disk, determine the file size, slice them if they are larger than 2G, and persist the data to the database in batches.

[0017] As a preferred solution of the data fusion method of multiple data sources in the cross-cloud scenario described in the present invention, obtaining data from a relational database and an object storage includes:

[0018] To obtain data from relational databases, use the DataX tool to configure tasks for each database, periodically read data from upstream relational databases, and write them into the result database.

[0019] To obtain data from object storage, use the DataX tool, configure data pulling tasks, regularly read files in the specified directory location in the upstream object storage, parse the data, and write it to the result library.

[0020] As a preferred solution of the data fusion method of multiple data sources in the cross-cloud scenario described in the present invention, the data is scheduled and pulled using a distributed scheduling tool, including:

[0021] Define data pulling tasks and scheduling tasks;

[0022] The pulling tasks and scheduling tasks are monitored.

[0023] As a preferred solution of the data fusion method of multiple data sources in the cross-cloud scenario described in the present invention, wherein: defining data pulling tasks and scheduling tasks includes:

[0024] Create different data pulling tasks in DolphinScheduler, each of which corresponds to a pulling operation of a data source and data type;

[0025] Use the workflow definition function of DolphinScheduler to organize multiple data pulling tasks into a workflow, set the concurrent execution strategy of the task, and enable multiple data pulling tasks to be executed simultaneously.

[0026] As a preferred solution of the data fusion method of multiple data sources in the cross-cloud scenario described in the present invention, the pulling task and the scheduling task are monitored, including:

[0027] Through the monitoring function of DolphinScheduler, the execution status of tasks can be monitored in real time, and the scheduling strategy and resource allocation can be adjusted according to the execution status of tasks.

[0028] As a preferred solution of the data fusion method of multiple data sources in the cross-cloud scenario described in the present invention, wherein: the amount of data to be processed is determined by adaptive code, and different solutions are selected to process the data, including:

[0029] If the amount of data processed is less than 5 million, the processing code is executed;

[0030] The execution processing code includes determining whether the SQL contains JOIN and counting the table size and the number of records;

[0031] If the amount of data to be processed is greater than 5 million, one task is split for every 300,000 records, and the tasks are executed in batches;

[0032] Also includes:

[0033] Choose different processing strategies based on the complexity of the SQL statement and the size of the table involved;

[0034] The different processing strategies include single-table operation and multi-table operation;

[0035] Also includes:

[0036] Select different processing solutions based on the type of data source obtained;

[0037] The types of data sources include API, relational database and object storage.

[0038] In a second aspect, the present invention provides a data fusion system for multiple data sources in a cross-cloud scenario, which includes:

[0039] The data source acquisition module is configured to obtain data from APIs, relational databases, and object storage;

[0040] A distributed scheduling tool module, configured to schedule and pull the data using a distributed scheduling tool;

[0041] The adaptive code processing module is configured to determine the amount of data to be processed currently through the adaptive code and select different solutions to process the data.

[0042] In a third aspect, the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: the processor implements any step of the above method when executing the computer program.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: the computer program implements any step of the above method when executed by a processor.

[0044] Compared with the prior art, the invention has the following beneficial effects: the invention obtains data from API, relational database and object storage; uses distributed scheduling tools to schedule and pull data; uses adaptive code to determine the current amount of data to be processed and selects different solutions to process the data; the invention solves the problem of unevenly distributed data scenarios, so that developers do not have to pay attention to the data magnitude and can directly write processing logic, thereby realizing the function of the solution being decided by the code; at the same time, it simplifies the development difficulty and reduces the risk of system stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0046] Figure 1 This is an overall flow chart of a data fusion method for multiple data sources in a cross-cloud scenario according to an embodiment of the present invention;

[0047] Figure 2 This is a graph showing fluctuations in data processing volume and system stability of a data fusion method for multiple data sources in a cross-cloud scenario according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0049] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0050] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0051] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0052] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0053] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0054] Example 1

[0055] Reference Figure 1 , which is the first embodiment of the present invention, and provides a data fusion method for multiple data sources in a cross-cloud scenario, including:

[0056] S1, get data from API, relational database and object storage;

[0057] Furthermore, data is obtained from the API, and the Python interface task code is used to access the data interface after authentication, obtain data, and write it into the database;

[0058] It should be noted that the Python interface task code improves the efficiency and reliability of data processing, especially when processing frequent requests, it can ensure real-time data updates;

[0059] Specifically, authentication refers to verifying whether the user has the right to access the system;

[0060] Furthermore, when authentication is successful, the token obtained for the first time is persisted in the database for use by other nodes in the distributed system;

[0061] It should be noted that persisting the token obtained for the first time to the database for use by other nodes in the distributed system can enable the sharing of authentication information in the distributed system, avoid repeated authentication of each node, and improve the overall efficiency of the system;

[0062] Furthermore, the data interface is accessed and the file is downloaded, and the data file is saved to the local disk;

[0063] Furthermore, the data files in the local disk are read, the file size is determined, and if it is larger than 2G, the files are sliced ​​and persisted to the database in batches;

[0064] Specifically, the file size needs to be set according to the application scenario during slicing. If the size is not set, loading too much data at one time will cause memory overflow.

[0065] Furthermore, to obtain data from relational databases, use the DataX tool to configure tasks for each database, periodically read data from upstream relational databases, and write them into the result database;

[0066] Specifically, upstream relational database refers to obtaining data from other systems as one of the data sources;

[0067] Furthermore, to obtain data from the object storage, use the DataX tool to configure the data pull task, regularly read the files in the specified directory location in the upstream object storage, parse the data, and write it to the result library;

[0068] S2. Use distributed scheduling tools to schedule and pull data;

[0069] Furthermore, define data pulling tasks and scheduling tasks;

[0070] Furthermore, different data pulling tasks are created in DolphinScheduler, each pulling task corresponds to a pulling operation of a data source and data type;

[0071] Specifically, the data pulling tasks of various data sources are run independently and do not interfere with each other;

[0072] Furthermore, using the workflow definition function of DolphinScheduler, multiple data pulling tasks are organized into a workflow, and the concurrent execution strategy of the tasks is set so that multiple data pulling tasks are executed simultaneously;

[0073] It should be noted that organizing multiple data pulling tasks into one workflow optimizes the time for data pulling and improves the efficiency of data acquisition;

[0074] Furthermore, the pulling tasks and scheduling tasks are monitored;

[0075] Furthermore, through the monitoring function of DolphinScheduler, the execution status of tasks can be monitored in real time, and the scheduling strategy and resource allocation can be adjusted according to the execution status of tasks;

[0076] It should be noted that adjusting the scheduling strategy and resource allocation steps improves the resource utilization of the system, reduces the failure rate and number of retries of task execution, and ensures the stable operation of the system;

[0077] S3, through adaptive code, determine the current amount of data to be processed and select different solutions to process the data;

[0078] Furthermore, if the amount of data to be processed is less than 5 million, the processing code is executed;

[0079] Specifically, the execution processing code includes determining whether the SQL contains JOIN and counting the table size and the number of records;

[0080] Furthermore, different processing strategies are selected according to the complexity of the SQL statement and the size of the table involved;

[0081] Among them, different processing strategies include single-table operations and multi-table operations (including JOIN);

[0082] Specifically, if the SQL statement involves only a single table operation, the size and number of records of the table are counted; if the number of records is small and the table size is moderate, the code processing is directly executed;

[0083] Specifically, if the SQL statement involves multi-table operations (including JOIN), count the total number of records; based on the number of records, choose whether to process in batches;

[0084] Furthermore, if the amount of data to be processed is greater than 5 million, one task is split for every 300,000 records, and the tasks are executed in batches;

[0085] Specifically, if the number of records is large, the SQL statement is divided into pages (for example, 300,000 records per page) and executed page by page to ensure system stability;

[0086] Furthermore, different processing solutions are selected according to the type of data source obtained;

[0087] Specifically, the types of data sources include API, relational database, and object storage;

[0088] Specifically, for API data sources, authentication is performed through a distributed system to obtain data and write it into a database; depending on the amount of data, whether to obtain and process in batches is selected;

[0089] Specifically, use the DataX tool to periodically pull data from the relational database and synchronize the data to the centralized result database. Select the appropriate pulling frequency and synchronization strategy based on the amount of data.

[0090] Specifically, use the DataX tool to periodically pull data files from the object storage, parse the data and write it into the result library. Depending on the size of the data file, choose whether to split the file.

[0091] It should be noted that selecting different processing schemes based on the type of data source highlights the flexibility of the adaptive code; and by judging the data processing volume, the processing flow is simplified, allowing the system to quickly complete data processing tasks.

[0092] Furthermore, this embodiment also provides a data fusion system for multiple data sources in a cross-cloud scenario, including:

[0093] The data source acquisition module is configured to obtain data from APIs, relational databases, and object storage;

[0094] A distributed scheduling tool module, configured to schedule and pull the data using a distributed scheduling tool;

[0095] The adaptive code processing module is configured to determine the amount of data to be processed currently through the adaptive code and select different solutions to process the data.

[0096] This embodiment further provides a computer device, which is applicable to a data fusion method for multiple data sources in a cross-cloud scenario, including:

[0097] Memory and processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the data fusion method of multiple data sources in a cross-cloud scenario as proposed in the above embodiment.

[0098] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0099] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the data fusion method of multiple data sources in a cross-cloud scenario proposed in the above embodiment is implemented.

[0100] The storage medium proposed in this embodiment and the data storage method proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be found in the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0101] Example 2

[0102] Reference Figure 2 , which is the second embodiment of the present invention, and which provides a data fusion method for multiple data sources in a cross-cloud scenario, including: verifying the effects involved in the solution of the present invention by means of simulation experiments;

[0103] This experiment selected four typical data sources, namely API data source, relational database, object storage and local file system; each data source has different data volume and data complexity to simulate the diversity in actual application scenarios, and the processed data is stored in a unified result library to obtain comparative experimental results, as shown in Table 1 and Table 2;

[0104] Table 1 The method of the present invention

[0105]

[0106] Table 2 Traditional methods

[0107]

[0108]

[0109] It can be seen from Table 1 and Table 2 that the method of the present invention takes 40 minutes to process the API data source, which has a significant processing time advantage compared to the 60 minutes of the traditional method; the data processing time of the relational database and the object storage is also shortened by 30 minutes and 35 minutes respectively; the processing time of the local file system is reduced by 20 minutes; this shows that under the action of distributed scheduling and adaptive code, the present invention can more efficiently process multiple data sources with large amounts of data; when processing 5 million pieces of data, the system load becomes a key indicator; among them, the data processing system load of the object storage in the traditional method is as high as 98%, while the method of the present invention is only 85%; the system load of the API data source and the local file system is also reduced by 20% and 10% respectively; the method of the present invention effectively reduces the system load and improves the stability and performance of the system through adaptive code and slicing processing; in terms of data integrity, the method of the present invention has a high performance in each data source, and the data integrity of the API data source and the local file system is 98% and 99%, which is 3% higher than the traditional method; in terms of data accuracy, the API data source and the local file system are both 99%, which is 4% to 6% higher than the traditional method;

[0110] refer to Figure 2 It can be seen that when the data volume processed is between 5 million (ten thousand) and 10 million (ten thousand), the system can still maintain a high operating efficiency, and the fluctuation range is between 92.98% and 97.03%, indicating that the present invention reduces the risk of system stability.

[0111] Example 3

[0112] Referring to the following code, it is the third embodiment of the present invention, which provides a data fusion method for multiple data sources in a cross-cloud scenario, including: The adaptive code in the scheme of the present invention is as follows;

[0113]

[0114] def parse_insert_2count(insert_sql):

[0115] # Find the starting position of the SELECT statement

[0116] select_start=insert_sql.find("SELECT")

[0117] #Extract SELECT statement

[0118] select_sql=insert_sql[select_start:]

[0119] # Construct a new SELECT count(1) statement

[0120] count_sql = f"SELECT count(1) FROM ({select_sql}) as subquery"

[0121] return count_sql

[0122] def exec_query(conn, query):

[0123] with conn.cursor() as cursor:

[0124] cursor.execute(query)

[0125] result = cursor.fetchone()

[0126] if result:

[0127] cnt = result[1]

[0128] return cnt

[0129] else:

[0130] return None

[0131] def check_sql_for_join(sql):

[0132] # First, check if there is a JOIN keyword

[0133] join_pattern = re.compile(r'\bJOIN\b', re.IGNORECASE)

[0134] has_join = re.search(join_pattern, sql) is not None

[0135] if has_join:

[0136] return True

[0137] else:

[0138] return False

[0139] def parse_from_table(sql):

[0140] #Find the table name after the FROM keyword

[0141] table_name="

[0142] from_pattern=re.compile(r'\bFROM\b\s+([a-zA-Z0-9_]+)',re.IGNORECASE)

[0143]

[0144]

[0145] else:

[0146] exec_single(conn,sql)

[0147] conn.close()

[0148] Through the above adaptive code, the current amount of data to be processed is determined, and different solutions are selected to process the data.

[0149] Those skilled in the art will appreciate that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program codes. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0150] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0153] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0154] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data fusion method for multiple data sources in a cross-cloud scenario, characterized in that: include: Get data from APIs, relational databases, and object stores; Use a distributed scheduling tool to schedule and pull the data; Through adaptive code, the current amount of data to be processed is determined, and different solutions are selected to process the data.

2. The data fusion method for multiple data sources in a cross-cloud scenario according to claim 1, characterized in that: Get data from the API, including: Python interface task code, access the data interface after authentication, obtain data and write it into the database; When authentication succeeds, the token obtained for the first time is persisted in the database for use by other nodes in the distributed system; Access the data interface and download files, and save the data files to the local disk; Read the data files in the local disk, determine the file size, slice them if they are larger than 2G, and persist the data to the database in batches.

3. The data fusion method for multiple data sources in a cross-cloud scenario according to claim 1, characterized in that: Get data from relational databases and object stores, including: To obtain data from relational databases, use the DataX tool to configure tasks for each database, periodically read data from upstream relational databases, and write them into the result database. To obtain data from object storage, use the DataX tool, configure data pulling tasks, regularly read files in the specified directory location in the upstream object storage, parse the data, and write it to the result library.

4. The data fusion method for multiple data sources in a cross-cloud scenario as claimed in claim 1, characterized in that: Use distributed scheduling tools to schedule and pull the data, including: Define data pulling tasks and scheduling tasks; The pulling tasks and scheduling tasks are monitored.

5. The data fusion method for multiple data sources in a cross-cloud scenario as claimed in claim 4, characterized in that: Define data pulling tasks and scheduling tasks, including: Create different data pulling tasks in DolphinScheduler, each of which corresponds to a pulling operation of a data source and data type; Use the workflow definition function of DolphinScheduler to organize multiple data pulling tasks into a workflow, set the concurrent execution strategy of the task, and enable multiple data pulling tasks to be executed simultaneously.

6. The data fusion method for multiple data sources in a cross-cloud scenario as claimed in claim 4, characterized in that: The pulling task and the scheduling task are monitored, including: Through the monitoring function of DolphinScheduler, the execution status of tasks can be monitored in real time, and the scheduling strategy and resource allocation can be adjusted according to the execution status of tasks.

7. The data fusion method for multiple data sources in a cross-cloud scenario according to claim 2, 3 or 6, characterized in that: Through adaptive code, the amount of data to be processed is determined, and different solutions are selected to process the data, including: If the amount of data processed is less than 5 million, the processing code is executed; The execution processing code includes determining whether the SQL contains JOIN and counting the table size and the number of records; If the amount of data to be processed is greater than 5 million, one task is split for every 300,000 records, and the tasks are executed in batches; Also includes: Choose different processing strategies based on the complexity of the SQL statement and the size of the table involved; The different processing strategies include single-table operation and multi-table operation; Also includes: Select different processing solutions based on the type of data source obtained; The types of data sources include API, relational database and object storage.

8. A data fusion system for multiple data sources in a cross-cloud scenario, based on the data fusion method for multiple data sources in a cross-cloud scenario according to any one of claims 1 to 7, characterized in that: include: The data source acquisition module is configured to obtain data from APIs, relational databases, and object storage; A distributed scheduling tool module, configured to schedule and pull the data using a distributed scheduling tool; The adaptive code processing module is configured to determine the amount of data to be processed currently through the adaptive code and select different solutions to process the data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images