Online fusion method, device and equipment for multi-source heterogeneous data and storage medium

Abstract

The invention provides an online fusion method, device and equipment for multi-source heterogeneous data and a readable storage medium. The method comprises the steps that the multi-source heterogeneous data associated with service information is converted into multiple sets of initial API interface data; fusing the multiple groups of initial API interface data in a preset virtual buffer layer based on a configuration file corresponding to a user service demand, and outputting service data; and packaging the service data into target API interface data. Through the implementation of the scheme of the invention, a virtual buffer layer can be constructed between the client and the server, the storage resources of the virtual buffer layer can be automatically released after the response is finished, and the data calling result in the whole data fusion process is not stored in a disk or a library, so that the data privacy can be effectively protected; and flexible configuration and development of the interface can be carried out according to business requirements, so that the practicability and applicability of data fusion are improved, the complexity of operation is reduced, and the millisecond-level response requirement of interactive service can be realized.

Description

Technical Field

[0001] The present application relates to the technical field of data fusion, and in particular to an online fusion method, device, equipment and readable storage medium for multi-source heterogeneous data. Background Art

[0002] Data fusion is a technology and method that integrates data from different sources. By integrating and analyzing information from multiple data sources, more comprehensive, accurate and useful information services can be provided. Data fusion can be applied to various fields, such as business intelligence, healthcare, finance, etc. The data used in public data development often comes from multiple departments and bureaus, and the unit data forms of mainstream data sources are diverse (for example: relational databases, files, API interfaces, etc.). Multi-dimensional and multi-form data fusion development can release the value of data elements. In related technologies, when fusing multi-source heterogeneous data, the data development link is relatively complicated, and usually only a response of seconds can be achieved. In addition, the data usually needs to be stored locally during the processing process, which poses a risk of data leakage. Summary of the invention The embodiments of the present application provide a method, apparatus, device and readable storage medium for online fusion of multi-source heterogeneous data, which can at least solve the problem of high risk of data leakage when performing data fusion processing in related technologies.

[0003] A first aspect of an embodiment of the present application provides an online fusion method for multi-source heterogeneous data, including: Convert multi-source heterogeneous data associated with business information into multiple sets of initial API interface data; Based on the configuration files corresponding to the user's business needs, multiple groups of initial API interface data are merged in the preset virtual buffer layer to output service data; Encapsulate service data into target API interface data.

[0004] A second aspect of an embodiment of the present application provides an online fusion device for multi-source heterogeneous data, including: A conversion module, used to convert multi-source heterogeneous data associated with business information into multiple sets of initial API interface data; A fusion module is used to fuse multiple groups of initial API interface data in a preset virtual buffer layer based on the configuration file corresponding to the user's business needs, and output service data; The encapsulation module is used to encapsulate service data into target API interface data.

[0005] A third aspect of an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the processor is used to execute a computer program stored in the memory, and when the processor executes the computer program, it implements each step of the online fusion method of multi-source heterogeneous data provided in the first aspect of the embodiment of the present application.

[0006] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step of the online fusion method for multi-source heterogeneous data provided in the first aspect of the embodiment of the present application is implemented.

[0007] As can be seen from the above, according to the online fusion method, device, equipment and readable storage medium of multi-source heterogeneous data provided by the present application, the multi-source heterogeneous data associated with the business information are converted into multiple sets of initial API interface data; based on the configuration file corresponding to the user's business needs, the multiple sets of initial API interface data are fused in the preset virtual buffer layer to output service data; the service data is encapsulated into target API interface data. Through the implementation of the present application, a virtual buffer layer is constructed to ensure that all data processing links (including access, conversion and processing) in the fusion process are completed in this layer, which reduces the complexity of the operation, shortens the development link, can achieve the millisecond-level response requirements of interactive services, and can also flexibly configure and develop interfaces according to business needs, effectively helping developers to develop and use; and the data in the entire process is not stored in the disk or the database, which can effectively protect data privacy, avoid the risk of data leakage caused by writing data into local archives during data processing, and avoid the risk of sensitive data being abused and used after being stored. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A basic flow chart of an online fusion method for multi-source heterogeneous data provided in the first embodiment of the present application; Figure 2 A schematic diagram of the composition of an API data online data fusion processing device provided in the first embodiment of the present application; Figure 3 A workflow diagram related to an online data fusion virtual group provided in the first embodiment of the present application; Figure 4 A detailed flowchart of an online fusion method for multi-source heterogeneous data provided in the second embodiment of the present application; Figure 5 A schematic diagram of program modules of an online fusion device for multi-source heterogeneous data provided in the third embodiment of the present application; Figure 6 A schematic diagram of the structure of an electronic device provided in the fourth embodiment of the present application. DETAILED DESCRIPTION

[0009] In order to make the purpose, features, and advantages of the invention of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0010] In the description of the embodiments of the present application, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0011] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0012] In the embodiments of the present application, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0013] There are two main data fusion methods in related technologies: 1. Read databases, files, and API interfaces, store relevant data resources locally, and then develop new services or interfaces based on the locally stored data for public use. However, this type of data fusion method has the following three problems: (1) The data needs to be stored locally, which is difficult to meet the management requirements of public management departments that "data should not be stored on disk"; (2) The real-time performance is poor. In the process of fusion processing between the interface and the library table, in order to form a unified standard, the interface content needs to be stored in the library. The processing flow is relatively long and it is difficult to meet the application requirements of real-time online conversion; (3) The development cost is high. Since the data needs to be stored on disk and the database needs to be built on the server, the cost is relatively high.

[0014] 2. Convert databases, files, etc. into API interfaces, and then use tools such as CSB bus to splice API interfaces. This can be done through mainstream tools (such as Postman, APISQL, Apifox, Apipost, Hoppscotch, etc.). However, these tools generally have the following problems: (1) They only achieve the primary fusion of interfaces (for example, simple addition, deletion and splicing of return parameters), which makes it difficult to solve the complex logical calculation problems required in the interface fusion process; (2) Interface splicing needs to be performed after reading, so the response time of the spliced ​​interface is generally long, which is difficult to meet the real-time requirements of users.

[0015] In order to solve the above defects in the related art, the first embodiment of the present application provides an online fusion method for multi-source heterogeneous data, such as Figure 1 A basic flow chart of the online fusion method for multi-source heterogeneous data provided in this embodiment, the online fusion method for multi-source heterogeneous data includes the following steps: Step 101: Convert multi-source heterogeneous data associated with business information into multiple sets of initial API interface data.

[0016] Specifically, multi-source heterogeneous data can be a collection of data from multiple data sources with different sources, formats, and structures. These data sources may include: relational databases: such as MySQL, PostgreSQL, etc. NoSQL databases: such as MongoDB, Cassandra, etc. Text files: such as CSV, JSON, XML files; Log files: such as Web server logs, application logs, etc.; IoT devices: such as sensor data, device status, etc. In the specific implementation process, multiple data sources can be uniformly converted into initial API interface data through the data source conversion component.

[0017] Step 102: Based on the configuration file corresponding to the user's business needs, multiple groups of initial API interface data are merged in a preset virtual buffer layer to output service data.

[0018] Specifically, in this embodiment, a virtual buffer layer may be constructed between the client and the server, so that multiple interface data may be calculated online in the space of the virtual buffer layer according to user needs, and desired service data may be output.

[0019] Step 103: Encapsulate the service data into target API interface data.

[0020] Specifically, during the encapsulation process, you can design API interface information according to your needs, that is, determine the interface type (such as REST, GraphQL), design the interface path and request method (such as GET, POST, PUT, DELETE, etc.), design the request parameters and response format, etc. According to the interface situation, the service data obtained after the fusion calculation is encapsulated into the target API interface data (for example: the return parameters of the fused interface S can be set to "total score" and "evaluation"). Figure 2 As shown, the data source conversion component and virtual buffer layer in this embodiment can be encapsulated into an API data online data fusion processing device to connect various data sources and perform rapid identification, query, and conversion calculations, and encapsulate the calculated service data into an interface service (i.e., target API interface data) in real time. Through the implementation of the above method steps of this embodiment, the millisecond-level response requirements of users for interactive services can be achieved, and the data call results of the entire process are not stored on the disk or in the database (only the data operation log can be retained for subsequent verification and supervision), which can effectively protect data privacy and avoid data leakage. In one implementation of the present embodiment, the above-mentioned step of converting multi-source heterogeneous data associated with business information into multiple sets of initial API interface data includes: configuring interface information for each data source in the multi-source heterogeneous data associated with business information; according to the interface information, obtaining corresponding data content from the database corresponding to each data source, and encapsulating the data content corresponding to each data source into initial API interface data.

[0021] Specifically, when configuring the interface information, you can configure the address, port, user name, password and other information of the data source in the CSB bus to read the data source. When performing API interface conversion, you can configure the interface information: interface name, interface input parameter, output parameter, URL, request method, etc., respectively call the data content corresponding to each data source and encapsulate it into the corresponding initial API interface data. The data source conversion component can convert multi-source heterogeneous data into a single API interface type data source for subsequent fusion calculations.

[0022] Furthermore, in one implementation of the present embodiment, before the above-mentioned step of converting the multi-source heterogeneous data associated with the business information into multiple sets of initial API interface data, it also includes: when the multi-source heterogeneous data includes a text file, reading the target data content of the text file; based on a preset database interface conversion model, converting the target data content into a target data source in a database format, and writing the target data source into the database.

[0023] Specifically, when multi-source heterogeneous data includes text files, that is, when the data source is in file format, it is necessary to first convert the format of the data source into a database format, read the file through a component with configured code, and store the file content in the database; then perform API interface conversion.

[0024] In one implementation of the present embodiment, before the step of fusing multiple sets of initial API interface data in a preset virtual buffer layer based on the above-mentioned configuration file corresponding to the user's business needs and outputting service data, it also includes: based on the interface document of each API interface, generating configuration parameters corresponding to each API interface; wherein the configuration parameters include at least one of the interface name, interface address, URL, request header, and query parameters; based on the configuration parameters, sending request parameters corresponding to each API interface to the server; parsing the response data returned by each API interface in response to the request parameters to obtain result parameters.

[0025] Specifically, Figure 2 As shown, this embodiment also provides an API data online data fusion processing device, and the virtual buffer layer includes an API reading component, an online data fusion virtual component, and a service encapsulation component. Among them, the API reading component is used to realize the function of reading the initial API interface data. More specifically, the API reading component is used to realize the functions of interface debugging, reading, responding, processing, etc. In this process, it is necessary to configure the API interface parameters of the call, and the interface parameters include information such as interface name, address, URL, request header, query parameters, etc., and parse the data content of the server response at the same time. The API call can be authenticated by appkey, rid, sid, and the key synchronization is authorized in 60 seconds, and it needs to be obtained again in 30 days; this implementation method refers to multi-factor authentication, improves the security of information calls, and controls user permissions and authorized access. The API reading component can use paramMap, headerMap, and bodyMap collections to record the request parameters corresponding to the API interface, and can use development tools such as httpclient or okhttp to read specific interface information. The online data fusion virtual component is used to realize the online fusion calculation of user configuration and data to obtain service data. The service encapsulation component is used to encapsulate the service data into the target API interface data.

[0026] It should be understood that different API interfaces correspond to required parameters, and specific parameters can be recorded before calling the API interface. Exemplarily, the social security interface (interface A) has data on the number of insured persons (the code for this data in interface A can be CBRS1), and the provident fund interface (interface B) has data on the number of provident fund payers (the code for this data can be GJJJNRS1), so the required parameters are inconsistent for different API interfaces, that is, specific parameters need to be set for different interfaces. During the specific implementation process, the interface documents of different interfaces can be obtained, and the code can be edited and set in advance in the API reading component according to the settings of the interface, that is, specific parameters are recorded; so as to realize reading "CBRS1" for interface A and reading "GJJJNRS1" for interface B.

[0027] like Figure 3 The figure shows a workflow diagram related to the online data fusion virtual group of this embodiment, in which a virtual conversion layer is established between the client and the server; wherein the user configuration module can generate a configuration file based on the user's business needs and business rules, so that the fusion rules (that is, the decision rule engine) can be determined according to the configuration file, and when it is necessary to call different data items of multiple API interfaces through the configuration input parameter module, the configuration instructions are unified, and multiple API interfaces are called, and finally the output parameter information of different interfaces is output, the service data is sorted out through the result arrangement component, and the service data is encapsulated into the target API interface data through the service encapsulation component.

[0028] Furthermore, in one implementation of the present embodiment, data contents of multiple groups of initial API interface data are respectively encapsulated in multiple API interfaces; accordingly, in the above-mentioned configuration file corresponding to the user's business needs, the multiple groups of initial API interface data are merged in a preset virtual buffer layer, and before the step of outputting service data, it also includes: based on the user's business needs, determining the target data items of each API interface and the configuration information corresponding to each target data item; wherein the configuration information includes the dimensional weight of the target data item in the dimension layer to which it belongs and the influencing factor in the factor layer to which it belongs; all configuration information is input into a preset configuration conversion model to generate a configuration file.

[0029] Specifically, in the virtual buffer layer, user configuration can be performed through the online data fusion virtual component. In the preset configuration conversion model, users can flexibly configure relevant fusion rules according to business rules and user service information acquisition requirements, and then generate corresponding configuration files. More specifically, users can flexibly configure parameters in the dimension layer, dimension weight, factor layer, and influencing factors through partially read and write configuration files. Among them, the configuration conversion model can be formed by big data analysis and feature value extraction. Taking the enterprise evaluation model as an example, through the comprehensive analysis and extraction of big data of enterprise evaluation indicators in different industries and different places, a comprehensive evaluation model matrix covering multiple dimensions and each dimension containing multiple factors (that is, the above-mentioned configuration conversion model) is formed. Therefore, in the specific implementation process, according to the actual business needs, for some enterprises, the dimensions and factor indicators involved in the evaluation can be extracted and their weights and influencing factors can be configured according to the needs of the business and users to form a configuration file.

[0030] For example, when a financial institution needs to evaluate the repayment ability of an enterprise based on the enterprise's social security information, actual amount of provident fund payment, number of people who actually paid provident fund, tax situation, and record of default, and give the enterprise a comprehensive score to evaluate its credit line, it can construct a configuration conversion model (enterprise evaluation model matrix) as shown in Table 1, and then obtain the corresponding configuration file (as shown in Table 2) through the configuration conversion model; when calling multiple API interface parameters, the obtained output parameter values ​​can be converted according to the settings in the configuration file.

[0031] Table 1:

[0032] Table 1 (continued):

[0033] Table 2:

[0034] Table 2 (continued):

[0035] Furthermore, in one implementation of the present embodiment, the above-mentioned step of fusing multiple groups of initial API interface data in a preset virtual buffer layer based on the configuration file corresponding to the user's business needs and outputting service data includes: in the preset virtual buffer layer, determining fusion rules according to the configuration file corresponding to the user's business needs; wherein the fusion rules are associated with the configuration information; calling the result parameters of the target data items returned from each API interface, and creating a MAP set based on the result parameters; fusing the MAP set based on the fusion rules, and outputting the service data.

[0036] Furthermore, in one implementation of the present embodiment, the above-mentioned step of fusing the MAP set based on the fusion rule and outputting the service data includes: extracting the first result value corresponding to each target parameter item from the MAP set based on the fusion rule; recording the first result value into a preset list set, traversing all the second result values ​​in the list set, and outputting the service data.

[0037] Specifically, the online data fusion virtual component can temporarily record the result parameters returned by multiple API interfaces through the MAP collection, extract specific parameters from the MAP collection, and add the specific parameters to the list collection, and finally traverse the specific parameter values, match the user's needs to perform simple logical operations, and finally form service data according to the results of user needs. Among them, the above-mentioned specific parameters can be determined according to the configuration file; when connecting with different API interfaces, by obtaining the interface document, understand the data parameter name, acquisition method and other information contained in the API interface; then send request parameters to the API interface according to the needs to obtain specific result parameters, for example: obtain CBRS1 (number of insured persons) from interface A (social security interface).

[0038] For example, when a financial institution needs to evaluate an enterprise's credit limit, the API interfaces used include: social security interface A, provident fund interface B, tax interface C, credit reporting interface D, and administrative interface E.

[0039] First, create a MAP set, which includes the following five sub-MAP sets: MAP 、MAP 、MAP <c>、MAP <d>、MAP <e> , used to record the result parameters returned by the 5 API interfaces, that is, MAP< / e> < / d> < / c> Record the result parameters returned by calling interface A: , MAP Record the result parameters returned by calling interface B: , MAP <c>Record the result parameters returned by calling interface C: , MAP <d>Record the result parameters returned by calling interface D: , MAP <e>Record the result parameters returned by calling interface E: , then:

[0040] Next, create a list collection to temporarily record the first result value extracted from the MAP collection. If a financial institution uses the target data items of interface A: the number of people to be paid, the total of the unit's payment base, and the three results of the unit's unified payment amount, the target data items of interface B: the actual amount paid and the number of depositors, the target data items of interface C: income tax expenses, taxes and additional results, the target data items of interface D: the default period, the default status code results, and the target data items of interface E: the administrative penalty decision document number results, then extract the first result value of the above target data items from MAP. to record it in the list set; finally, traverse the second result value recorded in the list set, and output the service data {A', B', C', D', E'} according to the fusion rule of the configuration file. The entire fusion process can be expressed by the following formula:

[0041] in, Represents the configuration transformation model, , , , , They respectively represent the policy rule engine parameters corresponding to the social security interface A, the provident fund interface B, the taxation interface C, the credit reporting interface D and the administrative interface E. Accordingly, the entire fusion process can also be represented by Table 3.

[0042] Table 3:

[0043] Based on the technical solution of the above-mentioned embodiment of the present application, the multi-source heterogeneous data associated with the business information is converted into multiple sets of initial API interface data; based on the configuration file corresponding to the user's business needs, the multiple sets of initial API interface data are integrated in the preset virtual buffer layer to output service data; the service data is encapsulated into target API interface data. Through the implementation of the solution of the present application, a virtual buffer layer is constructed to ensure that all data processing links (including access, conversion and processing) in the fusion process are completed in this layer, which reduces the complexity of the operation, shortens the development link, and can also flexibly configure and develop interfaces according to business needs, which can effectively help developers to develop and use; and the data in the whole process is not stored in the disk or the database, which can effectively protect data privacy, avoid the risk of data leakage caused by writing data into the local archive during the data processing process, and avoid the risk of sensitive data being abused and used after being stored.

[0044] < / e> < / d> < / c> Figure 4 The method in is a refined online fusion method for multi-source heterogeneous data provided in the second embodiment of the present application. The online fusion method for multi-source heterogeneous data includes: Step 401: configure interface information for each data source in multi-source heterogeneous data associated with business information.

[0045] Specifically, in this embodiment, when the multi-source heterogeneous data includes a text file, the target data content of the text file is read; based on a preset database interface conversion model, the target data content is converted into a target data source in a database format, and the target data source is written to the database, so as to encapsulate the data of the database formula into initial API interface data.

[0046] Step 402: According to the interface information, the corresponding data content is obtained from the database corresponding to each data source, and the data content corresponding to each data source is encapsulated into initial API interface data.

[0047] Step 403: In the preset virtual buffer layer, the initial API interface data is read through the API reading component.

[0048] Specifically, during the reading process, configuration parameters corresponding to each API interface can be generated based on the interface document of each API interface; wherein the configuration parameters include at least one of the interface name, interface address, URL, request header, and query parameters; request parameters corresponding to each API interface are sent to the server based on the configuration parameters; response data returned by each API interface in response to the request parameters is parsed to obtain result parameters.

[0049] Step 404: Based on user business requirements, determine the target data items of each API interface and the configuration information corresponding to each target data item.

[0050] Specifically, the configuration information includes the dimension weight of the target data item in the dimension layer to which it belongs and the influencing factor in the factor layer to which it belongs.

[0051] Step 405: Input all configuration information into a preset configuration conversion model to generate a configuration file.

[0052] Step 406: Determine fusion rules according to the configuration file corresponding to the user's business needs.

[0053] The fusion rule is associated with the configuration information.

[0054] Step 407: Call the result parameters of the target data items returned from each API interface, and create a MAP set based on the result parameters.

[0055] Step 408: Fuse the MAP set based on the fusion rule and output service data.

[0056] Step 409: Encapsulate the service data into target API interface data.

[0057] Specifically, this embodiment can achieve rapid fusion of interfaces through the use of a virtual buffer layer. On the basis of ensuring that no traces of data are left during the API fusion processing and that data is destroyed after use, secondary development of the interface can be carried out in combination with business needs. This is suitable for processing data with relatively complex interfaces and high privacy requirements.

[0058] The refinement method of this embodiment has the following advantages: 1. It can ensure that no traces are left in the API data fusion processing process, and the data is destroyed after use. All data processing is carried out in a virtual buffer layer, including access, conversion and processing; it effectively protects the privacy of the data and avoids the security risk of sensitive information being abused after being stored in the database. 2. It can meet the general user's requirements for millisecond-level response for interactive services, greatly improving the response efficiency. 3. It can effectively solve the problem that tools such as Postman and APISQL can only simply add, delete and splice interfaces, and have poor development flexibility, allowing users to configure files according to business rules and business needs, and flexibly set interface weights, numerical ranges and other indicators. It is suitable for application scenarios with more complex businesses.

[0059] It should be understood that the size of the serial number of each step in this embodiment does not mean the order of execution of the steps. The execution order of each step should be determined by its function and internal logic, and should not constitute a sole limitation on the implementation process of the embodiment of this application.

[0060] Figure 5 The third embodiment of the present application provides an online fusion device for multi-source heterogeneous data. The online fusion device for multi-source heterogeneous data can be applied to the online fusion method for multi-source heterogeneous data. Figure 5 As shown, the online fusion device of multi-source heterogeneous data mainly includes: A conversion module 501, used to convert multi-source heterogeneous data associated with business information into multiple groups of initial API interface data; A fusion module 502 is used to fuse multiple groups of initial API interface data in a preset virtual buffer layer based on a configuration file corresponding to the user's business needs, and output service data; The encapsulation module 503 is used to encapsulate the service data into target API interface data.

[0061] In some implementations of this embodiment, the conversion module is specifically used to: configure interface information for each data source in the multi-source heterogeneous data associated with the business information; obtain corresponding data content from the database corresponding to each data source according to the interface information, and encapsulate the data content corresponding to each data source into initial API interface data.

[0062] Furthermore, in some implementations of the present embodiment, when the conversion module executes the above-mentioned function of converting multi-source heterogeneous data associated with business information into multiple sets of initial API interface data, it is specifically used to: when the multi-source heterogeneous data includes a text file, read the target data content of the text file; based on a preset database interface conversion model, convert the target data content into a target data source in a database format, and write the target data source into the database.

[0063] In some implementations of the present embodiment, the data contents of multiple groups of initial API interface data are respectively encapsulated in multiple API interfaces; based on the configuration file corresponding to the user's business needs, the multiple groups of initial API interface data are fused in a preset virtual buffer layer, and before the service data is output, the fusion module is also used to: determine the target data items of each API interface and the configuration information corresponding to each target data item based on the user's business needs; wherein the configuration information includes the dimensional weight of the target data item in the dimension layer to which it belongs and the influencing factor in the factor layer to which it belongs; all configuration information is input into a preset configuration conversion model to generate a configuration file.

[0064] Furthermore, in some implementations of the present embodiment, the fusion module is specifically used to: determine the fusion rules in a preset virtual buffer layer according to the configuration file corresponding to the user's business needs; wherein the fusion rules are associated with the configuration information; call the result parameters of the target data items returned from each API interface, and create a MAP set based on the result parameters; fuse the MAP set based on the fusion rules, and output the service data.

[0065] Furthermore, in some implementations of the present embodiment, when the fusion module performs the above-mentioned function of fusing the MAP set based on the fusion rules and outputting the service data, it is also specifically used to: extract the first result value corresponding to each target parameter item from the MAP set based on the fusion rules; record the first result value into a preset list set, traverse all the second result values ​​in the list set, and output the service data.

[0066] In some implementations of the present embodiment, based on a configuration file corresponding to the user's business needs, multiple sets of initial API interface data are merged in a preset virtual buffer layer, and before the service data is output, the fusion module is also used to: generate configuration parameters corresponding to each API interface based on the interface document of each API interface; wherein the configuration parameters include at least one of the interface name, interface address, URL, request header, and query parameters; send request parameters corresponding to each API interface to the server based on the configuration parameters; parse the response data returned by each API interface in response to the request parameters to obtain result parameters.

[0067] According to the online fusion device of multi-source heterogeneous data provided by this embodiment, the multi-source heterogeneous data associated with the business information are converted into multiple groups of initial API interface data; based on the configuration file corresponding to the user's business needs, the multiple groups of initial API interface data are fused in the preset virtual buffer layer to output service data; the service data is encapsulated into target API interface data. Through the implementation of the scheme of this application, a virtual buffer layer is constructed to ensure that all data processing links (including access, conversion and processing) in the fusion process are completed in this layer, which reduces the complexity of the operation, shortens the development link, and can also flexibly configure and develop interfaces according to business needs, which can effectively help developers to develop and use; and the data in the whole process is not stored in the disk or the database, which can effectively protect data privacy, avoid the risk of data leakage caused by writing data into the local archive during the data processing process, and avoid the risk of sensitive data being abused and used multiple times after being stored.

[0068] Figure 6 An electronic device is provided in the fourth embodiment of the present application. The electronic device can be used to implement the online fusion method of multi-source heterogeneous data in the above-mentioned embodiment, mainly including: The memory 601, the processor 602, and the computer program 603 stored in the memory 601 and executable on the processor 602, the memory 601 and the processor 602 are connected by communication. When the processor 602 executes the computer program 603, the method in the above-mentioned embodiment 1 or 2 is implemented. The number of processors can be one or more.

[0069] The memory 601 may be a high-speed random access memory (RAM) memory, or a non-volatile memory, such as a disk memory. The memory 601 is used to store executable program codes, and the processor 602 is coupled to the memory 601 .

[0070] Furthermore, the embodiment of the present application also provides a computer-readable storage medium, which may be provided in the above-mentioned electronic device. The computer-readable storage medium may be the above-mentioned Figure 6 Memory in the illustrated embodiment.

[0071] The computer readable storage medium stores a computer program, and when the program is executed by the processor, the online fusion method of multi-source heterogeneous data in the aforementioned embodiment is implemented. Furthermore, the computer storable medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk, and other media that can store program codes.

[0072] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0073] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0074] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0075] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a readable storage medium, including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned readable storage medium includes: U disk, mobile hard disk, ROM, RAM, disk or CD-ROM and other media that can store program codes.

[0076] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0077] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0078] The above is a description of the online fusion method, device, equipment and readable storage medium for multi-source heterogeneous data provided by the present application. For technicians in this field, according to the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An online fusion method for multi-source heterogeneous data, characterized in that: include: Convert multi-source heterogeneous data associated with business information into multiple sets of initial API interface data; Based on the configuration file corresponding to the user's business needs, multiple groups of the initial API interface data are merged in a preset virtual buffer layer to output service data; The service data is encapsulated into target API interface data.

2. The online fusion method of multi-source heterogeneous data according to claim 1 is characterized in that: The converting of multi-source heterogeneous data associated with business information into multiple groups of initial API interface data includes: Configure interface information for each data source in multi-source heterogeneous data associated with business information; According to the interface information, corresponding data contents are respectively obtained from the databases corresponding to the data sources, and the data contents corresponding to the data sources are respectively encapsulated into initial API interface data.

3. The online fusion method of multi-source heterogeneous data according to claim 2 is characterized in that: Before converting the multi-source heterogeneous data associated with the business information into a plurality of groups of initial API interface data, the method further includes: When the multi-source heterogeneous data includes a text file, reading the target data content of the text file; Based on a preset database interface conversion model, the target data content is converted into a target data source in a database format, and the target data source is written into a database.

4. The online fusion method of multi-source heterogeneous data according to claim 1 is characterized in that: The data contents of the multiple groups of the initial API interface data are respectively encapsulated in multiple API interfaces; The configuration file corresponding to the user's business needs is used to merge the multiple groups of the initial API interface data in a preset virtual buffer layer, and before outputting the service data, it also includes: Based on user business needs, determine the target data items of each API interface and the configuration information corresponding to each target data item; wherein the configuration information includes the dimension weight of the target data item in the dimension layer to which it belongs and the influencing factor in the factor layer to which it belongs; All the configuration information is input into a preset configuration conversion model to generate a configuration file.

5. The online fusion method for multi-source heterogeneous data according to claim 4 is characterized in that: The configuration file corresponding to the user's business needs is used to merge multiple groups of the initial API interface data in a preset virtual buffer layer to output service data, including: In a preset virtual buffer layer, a fusion rule is determined according to a configuration file corresponding to a user service requirement; wherein the fusion rule is associated with the configuration information; Calling the result parameters of the target data item returned from each of the API interfaces, and creating a MAP set based on the result parameters; The MAP set is merged based on the merging rule, and service data is output.

6. The online fusion method for multi-source heterogeneous data according to claim 5 is characterized in that: The step of fusing the MAP set based on the fusing rule and outputting service data includes: Based on the fusion rule, extracting a first result value corresponding to each of the target parameter items from the MAP set; The first result value is recorded in a preset list set, all the second result values ​​in the list set are traversed, and the service data is output.

7. The online fusion method for multi-source heterogeneous data according to claim 5 is characterized in that: The configuration file corresponding to the user's business needs is used to merge the multiple groups of the initial API interface data in a preset virtual buffer layer, and before outputting the service data, it also includes: Based on the interface documents of the API interfaces, respectively generate configuration parameters corresponding to the API interfaces; wherein the configuration parameters include at least one of an interface name, an interface address, a URL, a request header, and a query parameter; Sending request parameters corresponding to each of the API interfaces to the server respectively based on the configuration parameters; Parse the response data returned by each of the API interfaces in response to the request parameters to obtain the result parameters.

8. An online fusion device for multi-source heterogeneous data, characterized in that: include: A conversion module, used to convert multi-source heterogeneous data associated with business information into multiple sets of initial API interface data; A fusion module, used to fuse the multiple groups of initial API interface data in a preset virtual buffer layer based on the configuration file corresponding to the user's business needs, and output service data; The encapsulation module is used to encapsulate the service data into target API interface data.

9. An electronic device, characterized in that: The device comprises a memory and a processor, wherein: The processor is used to execute the computer program stored in the memory; When the processor executes the computer program, the steps in the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

Images