Multi-data source integration method and device, computer device and storage medium

Abstract

The embodiment of the application provides a kind of multi-data source integration method, device, computer equipment and storage medium, the method obtains the data source integration request of data warehouse, data source integration request includes to be handled business data and integration parameter, through service unit, data source integration request is checked, in the case where check passes, through analysis unit and integration parameter, to be handled business data is parsed, determines integration index and data source type, determines the target component matched with data source type from the multiple components of engine unit, according to integration index, through target component and service unit, to be handled business data is integrated, obtains standard integration data, realizes the normalization and unified processing of the data of different data sources, improves the integration efficiency of multiple data sources, improves the maintainability and expansibility of data warehouse, improves the ability of fast access and consumption to data.

Description

Technical Field

[0001] This application relates to the field of network database technology, specifically to a method, apparatus, computer equipment, and storage medium for integrating multiple data sources. Background Technology

[0002] With the development of information technology, the amount of data in enterprises or organizations is increasing rapidly. Taking the performance indicator data of bank credit cards as an example, when building a credit card OLAP system, it is often necessary to connect to multiple data sources, such as Oracle data, SQL Server data, API data, etc. Moreover, the storage specifications of different data sources are different, making it difficult to meet the requirements of OLAP systems to quickly connect to new data sources and consume data quickly. Therefore, it is necessary to provide an efficient method for integrating multiple data sources to improve the processing efficiency of data from multiple data sources.

[0003] Application content

[0004] This application provides a method, apparatus, computer device, and storage medium for integrating multiple data sources, in order to solve the technical problem of low processing performance of multiple data sources caused by different storage specifications of different data sources.

[0005] On one hand, this application provides a multi-data source integration method applied to a cloud platform server. The cloud platform server includes a data warehouse, which is configured as a service unit for validating and aggregating data, a parsing unit for parsing data, and an engine unit for routing to target components. The engine unit contains multiple components, each corresponding to a data source. The method includes:

[0006] Obtain a data source integration request for the data warehouse, the data source integration request containing the business data to be processed and integration parameters;

[0007] The service unit verifies the data source integration request.

[0008] If the verification passes, the data to be processed is parsed using the parsing unit and the integration parameters to determine the integration indicators and data source type;

[0009] Determine a target component that matches the data source type from among the multiple components of the engine unit;

[0010] Based on the integration metrics, the business data to be processed is integrated through the target component and the service unit to obtain standard integrated data.

[0011] On one hand, this application provides a multi-data source integration device applied to a cloud platform server. The cloud platform server includes a data warehouse, which is configured as a service unit for validating and aggregating data, a parsing unit for parsing data, and an engine unit for routing to target components. The engine unit includes multiple components, each corresponding to a data source. The device includes:

[0012] The receiving module is used to obtain a data source integration request for the data warehouse, wherein the data source integration request includes business data to be processed and integration parameters;

[0013] The verification module is used to verify the data source integration request through the service unit;

[0014] The parsing module is used to parse the business data to be processed through the parsing unit and the integration parameters, and determine the integration indicators and data source type, if the verification passes.

[0015] A determination module is used to determine a target component that matches the data source type from among multiple components of the engine unit;

[0016] The integration module is used to integrate the business data to be processed through the target component and the service unit according to the integration indicators to obtain standard integrated data.

[0017] On one hand, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps in the above-described multi-data source integration method.

[0018] On the one hand, this application provides a computer-readable medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps in the above-described multi-data source integration method.

[0019] This application provides a method for integrating multiple data sources. It obtains a data source integration request for a data warehouse, which includes business data to be processed and integration parameters. A service unit verifies the data source integration request. If the verification passes, a parsing unit parses the business data to be processed using the integration parameters to determine integration metrics and data source types. A target component matching the data source type is selected from multiple components of the engine unit. Based on the integration metrics, the business data to be processed is integrated through the target component and the service unit to obtain standard integrated data. This method achieves standardized and unified processing of data from different data sources, improving the efficiency of integrating multiple data sources. Furthermore, because the service unit, parsing unit, and engine unit are configured separately, each unit works independently, and each component is independent of each other, simplifying the internal code of each component. During the processing of the business data to be processed, other units and components are unaware of this process, improving the maintainability and scalability of the data warehouse and enhancing the ability to quickly access and consume data. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] in:

[0022] Figure 1 This is a schematic diagram illustrating an application scenario of a multi-data source integration method in one embodiment;

[0023] Figure 2 This is a flowchart of a method for integrating multiple data sources in one embodiment;

[0024] Figure 3 This is a schematic diagram of the structure of a cloud platform server in one embodiment.

[0025] Figure 4 This is a structural block diagram of a multi-data source integration device in one embodiment;

[0026] Figure 5 This is a structural block diagram of a computer device in one embodiment. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The multi-data source integration method provided in this application can be applied to, for example... Figure 1 In this application environment, terminal devices communicate with the cloud platform server via a network. These terminal devices can be, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The cloud platform server can be implemented using a standalone cloud platform server or a server cluster consisting of multiple cloud platform servers.

[0029] System framework 100 may include terminal devices 101, 102, and 103, network 104, and cloud platform server 105. Network 104 is used as a medium to provide a communication link between the terminal devices and the server. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0030] Users can use their terminal devices to interact with the cloud platform server over the network to receive or send messages, etc.

[0031] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Eperts Group Audio Layer III), MP4 players (Moving Picture Eperts Group Audio Layer IV), laptops, and desktop computers, etc.

[0032] The cloud platform server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0033] It should be noted that the multi-data source integration method provided in this embodiment of the invention is executed by a cloud platform server, and correspondingly, the multi-data source integration device is set in the cloud platform server.

[0034] It should be understood that Figure 1The number of terminal devices, networks, and cloud platform servers shown in this embodiment is merely illustrative. Depending on the implementation needs, there can be any number of terminal devices, networks, and cloud platform servers. The terminal devices in this embodiment can specifically correspond to the application systems in actual production.

[0035] like Figure 2 As shown, in one embodiment, a multi-data source integration method is provided. This method is applied to a cloud platform server, which includes a data warehouse. The data warehouse is configured as a service unit for validating and aggregating data, a parsing unit for parsing data, and an engine unit for routing to target components. The engine unit contains multiple components, each corresponding to a data source. The multi-data source integration method specifically includes the following steps:

[0036] Step 201: Obtain the data source integration request for the data warehouse. The data source integration request contains the business data to be processed and integration parameters.

[0037] Among them, a cloud platform server is a type of computing server with scalable and elastic processing capabilities. For example... Figure 3 The diagram shows the structure of the cloud platform server in this embodiment. The cloud platform server includes a data warehouse, which is configured as a service unit, a parsing unit, and an engine unit. The engine unit contains multiple components, each corresponding to a type of data source. Understandably, this embodiment configures the data warehouse as an independent multi-layered structure, with the service unit, parsing unit, and engine unit configured separately. Each unit works independently, ensuring that other layers are unaware of changes, thus avoiding interference to other layers when one or more layers change.

[0038] A data source integration request refers to an instruction sent by a terminal to a cloud platform server to integrate data from different types of data sources in a data warehouse. This data source integration request includes business data to be processed and integration parameters. Integration parameters refer to the attributes of the data to be integrated. For example, for performance indicator data to be processed in the form of bank credit cards, the integration parameters could be the total bill amount of credit cards issued by the same salesperson, credit card type, and application method.

[0039] Step 202: Verify the data source integration request through the service unit.

[0040] The validation mechanism is used to verify whether the data source integration request sent by the client is standard. For example, it verifies whether there are any missing business data to be processed in the data source integration request. Specifically, it can be validated using preset regular expressions, such as Regexp(REGEXP_INSTR, REGEXP_SUBSTR, and REGXP_REPLACE). REGEXP_INSTR extends the functionality of the INSTR function, supports regular expression patterns for searching strings, and DSC can accommodate REGEXP_INSTR with 2 to 6 parameters to determine whether the data source integration request is standard. If the format of the data source integration request meets the standard requirements, such as if the business data to be processed is complete, the validation is considered to have passed; otherwise, the validation is considered to have failed. This avoids redundant operations on non-standard data source integration requests and improves the efficiency of multi-data source integration.

[0041] Step 203: If the verification passes, the business data to be processed is parsed using the parsing unit and integration parameters to determine the integration indicators and data source type.

[0042] Among them, the integrated indicator refers to the indicator attribute of the business data to be processed after being associated and integrated according to the attribute logic of the integrated parameter. For example, when the integrated parameter A is a combination of a time period T, salesperson ID, and total bill amount ID, its representation is A = (T, ID, S). Then the corresponding integrated indicator L can be the total bill amount of the credit cards processed by the same salesperson during this time period, and L = sum(S).

[0043] Specifically, when the data source integration request passes verification, it indicates that the business data to be processed is complete and meets the integration conditions. The data source of the business data to be processed can be determined by matching the types of each business data in the business data to be processed with the preset standard data. Alternatively, if the data source integration request contains an address, the address in the data source integration request can be compared with the addresses corresponding to various data source types to determine the data source type of the business data to be processed.

[0044] Step 204: Determine the target component that matches the data source type from among the multiple components of the engine unit.

[0045] Components are plugins used to process data from different data source types, with one component corresponding to each data source type. Target components are plugins used to process data corresponding to their data source type; these are data models, such as machine learning models for data analysis, like Data Science or Vertica. Vertica uses massively parallel computing to process petabyte-scale data and performs internal machine learning through data parallelism. It has eight built-in algorithms for data preparation, three regression algorithms, four classification algorithms, two clustering algorithms, and multiple model management functions. It can also import trained TensorFlow and PMML models to other locations, or it can be a mathematical model for data analysis, i.e., a mathematical model pre-built using data modeling methods. Specifically, based on the data source type, routing is performed to the component corresponding to the data source, i.e., the target component. This allows the target component, matching the data source type, to standardize the different specifications of the business data to be processed, improving the uniformity and standardization of the business data and enhancing the fault tolerance performance of data from various data source types.

[0046] Step 205: Based on the integration metrics, integrate the business data to be processed through the target components and service units to obtain standard integrated data.

[0047] Integration refers to a data processing method that organizes and aggregates data. For example, based on the fields corresponding to the integration indicators, database query statements, such as SQL syntax, are used to aggregate and analyze the business data to be processed. More specifically, if the integration indicator is ES details, the target component converts the integration indicator into a DSL statement and sends it to the service unit. The service unit integrates the business data to be processed based on the received DSL statement. The DSL includes the following syntax: querying data that meets the conditions, and then performing aggregation analysis through nested-aggs to obtain standard integrated data. As a preferred embodiment, painless is used to format the aggregated data to obtain standard integrated data, realizing the standardization and unified processing of data from different data sources, improving the integration efficiency of multiple data sources. Furthermore, since the service unit, parsing unit, and engine unit are configured separately, each unit works independently, and each component is independent of each other, simplifying the internal code of each component. During the processing of the business data to be processed, other units and components are unaware of this, improving the maintainability and scalability of the data warehouse, and enhancing the ability to quickly access and consume data.

[0048] The aforementioned multi-data source integration method obtains a data source integration request for the data warehouse. This request includes the business data to be processed and integration parameters. The service unit verifies the data source integration request. If the verification passes, the parsing unit parses the business data to be processed using the integration parameters to determine the integration metrics and data source type. A target component matching the data source type is selected from multiple components in the engine unit. Based on the integration metrics, the business data to be processed is integrated through the target component and the service unit to obtain standard integrated data. This achieves standardized and unified processing of data from different data sources, improving the efficiency of multi-data source integration. Furthermore, because the service unit, parsing unit, and engine unit are configured separately, each unit works independently, and each component is independent of each other, simplifying the internal code of each component. During the processing of the business data to be processed, other units and components are unaware of this process, improving the maintainability and scalability of the data warehouse and enhancing the ability to quickly access and consume data.

[0049] In one embodiment, the service unit includes at least one of multiple standard data source integration requests or their corresponding verification expressions, wherein the verification expression is used to indicate the verification rules of the data source integration request; the service unit verifies the data source integration request, including: comparing the data source integration request with each standard data source integration request, and determining whether the verification passes based on the comparison result; or, using the verification expression to verify the data source integration request and determining whether the verification passes.

[0050] The standard data source integration request refers to a standard data source integration request pre-stored in the cloud platform server, such as a standard request URL. The validation expression refers to an expression that matches the standard data source integration request, such as a regular expression, used to indicate the validation rules for the data source integration request.

[0051] Specifically, the data source integration request is compared with each standard data source integration request. If the comparison result shows that the data source integration request matches one of the standard data source integration requests, the verification is considered successful; if the comparison result shows that the data source integration request does not match any of the standard data source integration requests, the verification is considered unsuccessful. Alternatively, the data source integration request is verified using the verification rules in the verification expression. If the result of the verification expression returns true, the verification is considered successful; if the result of the verification expression returns false, the verification is considered unsuccessful. In this embodiment, the data source integration request is verified using the standard data source integration requests in the service unit or their corresponding verification expressions, avoiding redundant processing of data source integration requests that fail verification and improving the integration efficiency of multiple data sources.

[0052] In one embodiment, the parsing unit includes a database address identification tool and an indicator parsing logical expression, wherein the indicator parsing logical expression is used to indicate the parsing rules of the integration parameters; if the verification passes, the parsing unit and the integration parameters are used to parse the business data to be processed to determine the integration indicators and data source type, including: using the database address identification tool to identify the business data to be processed to obtain the data source type; and using the indicator parsing logical expression to logically parse the definition of the integration parameters to obtain the integration indicators.

[0053] Among these, database address identification tools are plugins used to identify database addresses, such as mysqlbinlog, LogMiner, and fn_dblog. If identification is achieved through mysqlbinlog, the data source is determined to be a MySQL database; if it's achieved through LogMiner, the data source is determined to be an Oracle database; and if it's achieved through fn_dblog, the data source is determined to be a SQL Server database.

[0054] The indicator parsing logical expression is a parsing rule used to indicate the integration parameters. For example, the indicator parsing expression is sum(X), where X is one of the integration parameters, and the logic is a summation operation.

[0055] Specifically, a database address identification tool is used to identify the business data to be processed, thereby obtaining the data source type. The definition of the integration parameters is then logically parsed through the indicator parsing logical expression to obtain the integration indicators, thus realizing the parsing of the business data to be processed.

[0056] In one embodiment, determining a target component that matches the data source type from multiple components of the engine unit includes: inputting the data source type, the business data to be processed, and the integration metrics into the routing matching model to obtain the target component.

[0057] The routing matching model is a pre-trained machine learning model used to determine the target component corresponding to the data source type. Specifically, the data source type, the business data to be processed, and the integration metrics are used as inputs to the routing matching model, and the output of the routing matching model is the target component. Understandably, in this embodiment, by using a machine learning model and combining information about the data source type, the business data to be processed, and the integration metrics, the target component matching the data source type is determined, further improving the accuracy of the target component.

[0058] In one embodiment, based on the parsing and integration metrics, the business data to be processed is integrated through target components and service units to obtain standard integrated data, including: obtaining the fields contained in the business data to be processed through service units based on the integration metrics; obtaining the common fields contained in the business data to be processed from the fields; associating and merging multiple business data to be processed based on the common fields to obtain initial integrated data; and standardizing the initial integrated data through target components to obtain standard integrated data.

[0059] Standardization processing refers to the normalization of integrated data. For example, it involves unifying integrated metrics with the same meaning or fields with different names to achieve standardization. Specifically, based on the integrated metrics, the service unit obtains the fields contained in the business data to be processed. Then, it extracts the common fields contained in the business data to be processed. Next, it links and merges multiple pieces of business data to be processed with the same fields to obtain initial integrated data. Finally, the target component performs standardization processing on the initial integrated data to obtain standard integrated data. This achieves the integration of business data to be processed, making the standard integrated data more standardized and normalized, and realizing the unification of data from multiple data sources.

[0060] In one embodiment, each component includes a structured query statement corresponding to each field; standardizing the initial integrated data through the target component to obtain standard integrated data includes: using the initial integrated data as input parameters of the structured query statement to generate standard integrated data.

[0061] Each component includes a structured query statement corresponding to each field. Initial integrated data is input into the structured query statement of the target component, and the structured query statement outputs standard integrated data. In this embodiment, the standardization processing of the initial integrated data is achieved through the structured query statement, which is simple, convenient, and has simple code, thus improving the efficiency of standardization processing of multiple data sources.

[0062] In one embodiment, the multi-data source integration method further includes: when the data source type of the business data to be processed is detected to be a newly added data source type, constructing a data model matching the newly added data source type in the engine unit; determining the script of the data model; executing the script to obtain the target component matching the newly added data source type.

[0063] The script of the data model refers to the code of the data model. Specifically, when the data source type of the business data to be processed is a newly added data type, it is necessary to build a corresponding target component. Therefore, a data model matching the newly added data source type is built in the engine unit, the corresponding script is determined according to the data model, and the script is executed to generate a target component matching the newly added data source type, thereby realizing the rapid addition of target components for new data sources.

[0064] In one embodiment, the multi-data source integration method further includes: if a component in the engine unit is not a target component within a preset time period, determining that the component is a component to be deleted; and deleting the component to be deleted from the engine unit.

[0065] Specifically, if one or more components of the engine unit are not target components within a preset time period, such as one month, it indicates that the data source type of the business data to be processed does not match the component for a longer period of time, that is, the component is a redundant component. Therefore, such components are deleted as components to be deleted, which improves the memory of the data warehouse and further improves the efficiency of data processing.

[0066] like Figure 4 As shown, in one embodiment, a multi-data source integration device is proposed. The multi-data source integration device is applied to a cloud platform server, which includes a data warehouse. The data warehouse is configured as a service unit for validating and aggregating data, a parsing unit for parsing data, and an engine unit for routing to target components. The engine unit includes multiple components, each corresponding to a data source. The device includes:

[0067] The receiving module is used to obtain a data source integration request for the data warehouse, wherein the data source integration request includes business data to be processed and integration parameters;

[0068] The verification module is used to verify the data source integration request through the service unit;

[0069] The parsing module is used to parse the business data to be processed through the parsing unit and the integration parameters, and determine the integration indicators and data source type, if the verification passes.

[0070] A determination module is used to determine a target component that matches the data source type from among multiple components of the engine unit;

[0071] The integration module is used to integrate the business data to be processed through the target component and the service unit according to the integration indicators to obtain standard integrated data.

[0072] In one embodiment, the service unit includes at least one of multiple standard data source integration requests or their corresponding validation expressions, and the validation module includes:

[0073] The comparison unit is used to compare the data source integration request with each of the standard data source integration requests, and determine whether the verification passes based on the comparison result; or,

[0074] The verification unit is used to verify the data source integration request using the verification expression and determine whether the verification passes.

[0075] In one embodiment, the parsing unit includes a database address identification tool and an indicator parsing logical expression, and the parsing module includes:

[0076] The identification unit is used to identify the business data to be processed using the database address identification tool to obtain the data source type;

[0077] The parsing unit is used to logically parse the definition of the integrated parameters through the index parsing logical expression to obtain the integrated index.

[0078] In one embodiment, the determining module includes a matching unit, configured to input the data source type, the business data to be processed, and the integrated metrics into a routing matching model to obtain the target component.

[0079] In one embodiment, the integration module includes:

[0080] The first acquisition unit is used to acquire the fields contained in the business data to be processed through the service unit based on the integrated indicators;

[0081] The second acquisition unit is used to acquire the same fields contained in the business data to be processed from the fields;

[0082] The merging unit is used to associate and merge multiple pieces of the business data to be processed based on the same field to obtain initial integrated data;

[0083] The standardization unit is used to standardize the initial integrated data through the target component to obtain the standard integrated data.

[0084] In one embodiment, each component includes a structured query statement corresponding to each of the fields; the standardization unit includes a generation subunit, used to generate the standard integrated data by taking the initial integrated data as input parameters of the structured query statement.

[0085] In one embodiment, the multi-data source integration apparatus further includes:

[0086] The detection module is used to construct a data model matching the newly added data source type in the engine unit when it detects that the data source type of the business data to be processed is a newly added data source type.

[0087] The first determining module is used to determine the script for the data model;

[0088] An execution module is used to execute the script to obtain a target component that matches the newly added data source type.

[0089] In one embodiment, the multi-data source integration apparatus further includes:

[0090] The second determining module is used to determine that if a component in the engine unit is not the target component within a preset time period, the component is a component to be deleted.

[0091] A deletion module is used to delete the component to be deleted from the engine unit.

[0092] Figure 5 An internal structural diagram of a computer device in one embodiment is shown. This computer device may specifically be a server, including but not limited to high-performance computers and high-performance computer clusters. Figure 5 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program. When executed by the processor, this computer program enables the processor to implement a multi-data source integration method. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to implement the multi-data source integration method. Those skilled in the art will understand that… Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0093] In one embodiment, the multi-data source integration method provided in this application can be implemented as a computer program, which can be implemented in the form of, for example... Figure 5 The computer device shown runs on this device. The computer device's memory can store various program templates that make up the multi-data source integration device. For example, an acquisition module 301, an analysis module 302, a fusion module 303, an extraction module 304, and a determination module 305.

[0094] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the multi-data source integration method described above.

[0095] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the above-described multi-data source integration method.

[0096] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0097] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0098] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for integrating multiple data sources, characterized in that, The method is applied to a cloud platform server, which includes a data warehouse. The data warehouse is configured as a service unit for validating and aggregating data, a parsing unit for parsing data, and an engine unit for routing data to target components. The engine unit contains multiple components, each corresponding to a data source. Each component includes a structured query statement corresponding to various fields. Obtain a data source integration request for the data warehouse, the data source integration request containing the business data to be processed and integration parameters; The service unit verifies the data source integration request. If the verification passes, the data to be processed is parsed using the parsing unit and the integration parameters to determine the integration indicators and data source type; Determine a target component that matches the data source type from among the multiple components of the engine unit; Based on the integration metrics, the business data to be processed is integrated through the target component and the service unit. Specifically, the service unit obtains the fields contained in the business data to be processed; it then identifies the common fields contained in the business data to be processed from these fields; based on the common fields, multiple pieces of business data to be processed are associated and merged to obtain initial integrated data; and the initial integrated data is used as input parameters for the structured query statement, so that the target component performs standardization processing on the initial integrated data to obtain standard integrated data. When it is detected that the data source type of the business data to be processed is a newly added data source type, a data model matching the newly added data source type is constructed in the engine unit; Determine the script for the data model; Execute the script to obtain the target component that matches the newly added data source type.

2. The multi-data source integration method as described in claim 1, characterized in that, The service unit includes multiple standard data source integration requests and at least one of the verification expressions corresponding to each standard data source integration request. The verification expression is used to indicate the verification rules of the data source integration request. The step of verifying the data source integration request through the service unit includes: The data source integration request is compared with each of the standard data source integration requests, and the verification is determined based on the comparison results. or, The data source integration request is validated using the validation expression to determine whether the validation passes.

3. The multi-data source integration method as described in claim 1, characterized in that, The parsing unit includes a database address identification tool and an indicator parsing logic expression, wherein the indicator parsing logic expression is used to indicate the parsing rules of the integrated parameters; If the verification passes, the process involves parsing the business data to be processed using the parsing unit and the integration parameters to determine the integration metrics and data source type, including: The database address identification tool is used to identify the business data to be processed, thereby obtaining the data source type; The integrated index is obtained by logically parsing the definition of the integrated parameters through the index parsing logical expression.

4. The multi-data source integration method as described in claim 3, characterized in that, The step of determining the target component that matches the data source type from among the multiple components of the engine unit includes: The target component is obtained by inputting the data source type, the business data to be processed, and the integrated metrics into the routing matching model.

5. The multi-data source integration method as described in claim 4, characterized in that, The method further includes: If a component that is not the target component within a preset time period is detected in the engine unit, then the component is determined to be a component to be deleted. Remove the component to be removed from the engine unit.

6. A multi-data source integration device, characterized in that, The multi-data source integration device is applied to a cloud platform server. The cloud platform server includes a data warehouse, which is configured as a service unit for validating and aggregating data, a parsing unit for parsing data, and an engine unit for routing to target components. The engine unit contains multiple components, each corresponding to a data source. Each component includes a structured query statement corresponding to various fields. The device includes: The receiving module is used to obtain a data source integration request for the data warehouse, wherein the data source integration request includes business data to be processed and integration parameters; The verification module is used to verify the data source integration request through the service unit; The parsing module is used to parse the business data to be processed through the parsing unit and the integration parameters, and determine the integration indicators and data source type, if the verification passes. A determination module is used to determine a target component that matches the data source type from among multiple components of the engine unit; An integration module is used to integrate business data to be processed through the target component and the service unit according to the integration indicators. Specifically, the service unit obtains the fields contained in the business data to be processed; it extracts the common fields contained in the business data to be processed from the fields; it associates and merges multiple pieces of business data to be processed based on the common fields to obtain initial integrated data; it uses the initial integrated data as input parameters for the structured query statement to standardize the initial integrated data through the target component to obtain standard integrated data; when it detects that the data source type of the business data to be processed is a newly added data source type, it constructs a data model matching the newly added data source type in the engine unit; it determines the script for the data model; and it executes the script to obtain a target component matching the newly added data source type.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multi-data source integration method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the multi-data source integration method as described in any one of claims 1 to 5.

Images