Online fusion method, device and equipment of multi-source heterogeneous data and storage medium

Abstract

The application provides an online fusion method, device and equipment of multi-source heterogeneous data and a readable storage medium. The method comprises the following steps: converting multi-source heterogeneous data associated with business information into multiple groups of initial API interface data; fusing the multiple groups of initial API interface data in a preset virtual buffer layer based on a configuration file corresponding to a user business requirement, and outputting service data; and encapsulating the service data into target API interface data. Through implementation of the application, a virtual buffer layer can be constructed between a client and a server. Storage resources of the virtual buffer layer are automatically released after response ends. Data calling results of the entire data fusion process are not stored and saved in a database, so that data privacy can be effectively protected. In addition, interface flexible configuration development can be performed according to business requirements, the practicability and applicability of data fusion are improved, the complexity of operation is reduced, and millisecond-level response requirements of interactive services can be met.

Description

Technical Field

[0001] The present application relates to the technical field of data fusion, and in particular to an online fusion method, apparatus, device and readable storage medium for multi-source heterogeneous data. Background Art

[0002] Data fusion is a technology and method for integrating data from different sources. By integrating and analyzing information from multiple data sources, more comprehensive, accurate, and useful information services can be provided. Data fusion can be applied to various fields, such as business intelligence, healthcare, finance, etc. The data used in public data development often comes from multiple departments and units, and the unit data forms of mainstream data sources are diverse (for example: relational databases, files, API interfaces, etc.). Multi-dimensional and multi-form data fusion development can release the element value of data. In related technologies, when fusing multi-source heterogeneous data, the data development link is relatively complex, and usually only a response of seconds can be achieved. In the processing process, the data usually needs to be stored locally, which poses a risk of data leakage. Summary of the Invention

[0003] The embodiments of the present application provide a method, apparatus, device, and readable storage medium for online fusion of multi-source heterogeneous data, which can at least solve the problem of high risk of data leakage during data fusion processing in related technologies.

[0004] A first aspect of an embodiment of the present application provides an online fusion method for multi-source heterogeneous data, comprising:

[0005] Convert multi-source heterogeneous data associated with business information into multiple sets of initial API interface data;

[0006] Based on the configuration files corresponding to the user's business needs, multiple sets of initial API interface data are integrated in the preset virtual buffer layer to output service data;

[0007] Encapsulate service data into target API interface data.

[0008] A second aspect of an embodiment of the present application provides an online fusion device for multi-source heterogeneous data, including:

[0009] A conversion module, used to convert multi-source heterogeneous data associated with business information into multiple sets of initial API interface data;

[0010] The fusion module is used to fuse multiple sets of initial API interface data in a preset virtual buffer layer based on the configuration file corresponding to the user's business needs and output service data;

[0011] The encapsulation module is used to encapsulate service data into target API interface data.

[0012] A third aspect of an embodiment of the present application provides an electronic device comprising: a memory and a processor, wherein the processor is configured to execute a computer program stored in the memory. When the processor executes the computer program, the processor implements each step of the online fusion method for multi-source heterogeneous data provided in the first aspect of the embodiment of the present application.

[0013] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, each step of the online fusion method for multi-source heterogeneous data provided in the first aspect of the embodiment of the present application is implemented.

[0014] As can be seen from the above, according to the online fusion method, device, equipment and readable storage medium of multi-source heterogeneous data provided by the present application, the multi-source heterogeneous data associated with the business information are converted into multiple sets of initial API interface data; based on the configuration file corresponding to the user's business needs, the multiple sets of initial API interface data are fused in the preset virtual buffer layer to output service data; the service data is encapsulated into target API interface data. Through the implementation of the present application, a virtual buffer layer is constructed to ensure that all data processing links (including access, conversion and processing) in the fusion process are completed in this layer, which reduces the complexity of the operation, shortens the development link, can achieve the millisecond-level response requirements of interactive services, and can also flexibly configure and develop interfaces according to business needs, effectively helping developers to develop and use; and the data in the entire process is not written to the disk or stored in the database, which can effectively protect data privacy, avoid the risk of data leakage caused by writing data into the local archive during data processing, and avoid the risk of sensitive data being abused and multiple used after being stored. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of the basic flow of an online fusion method for multi-source heterogeneous data provided in the first embodiment of the present application;

[0016] Figure 2 A schematic diagram of the composition of an online data fusion processing device for API data provided in the first embodiment of the present application;

[0017] Figure 3 A workflow diagram related to online data fusion virtual grouping provided in the first embodiment of the present application;

[0018] Figure 4 A detailed flowchart of an online fusion method for multi-source heterogeneous data provided in the second embodiment of the present application;

[0019] Figure 5 A schematic diagram of program modules of an online fusion device for multi-source heterogeneous data provided in the third embodiment of the present application;

[0020] Figure 6 This is a schematic structural diagram of an electronic device provided in the fourth embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of this application.

[0022] In the description of the embodiments of the present application, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0023] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, the meaning of "plurality" is two or more, unless otherwise clearly specified.

[0024] In the embodiments of the present application, unless otherwise expressly specified or limited, the terms "installed," "connected," "connected," "fixed," etc. should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integration; mechanical connections, electrical connections; direct connections, or indirect connections through an intermediate medium; and internal connections between two components or interactions between two components. Those skilled in the art will understand the specific meanings of the above terms in the embodiments of the present application based on specific circumstances.

[0025] There are two main data fusion methods in related technologies:

[0026] 1. Read databases, files, and API interfaces, store relevant data resources locally, and then develop new services or interfaces based on the locally stored data for public use. However, this type of data fusion method has the following three problems: (1) The data needs to be stored locally, which makes it difficult to meet the public management department's "data not stored on disk" management requirements; (2) The real-time performance is poor. In the process of fusion processing between the interface and the library table, in order to form a unified standard, the interface content needs to be stored in the library. The processing flow is relatively long and it is difficult to meet the application requirements of real-time online conversion; (3) The development cost is high. Since the data needs to be stored on the disk and the database needs to be built on the server, the cost is relatively high.

[0027] 2. Convert databases, files, etc. into API interfaces, and then use tools such as CSB bus to splice the API interfaces. This can be done through mainstream tools (such as Postman, APISQL, Apifox, Apipost, Hoppscotch, etc.). However, these tools generally have the following problems: (1) They only achieve the primary fusion of interfaces (for example, simple addition, deletion, and splicing of return parameters), which makes it difficult to solve the complex logical calculation problems required in the interface fusion process; (2) Interface splicing needs to be performed after reading, so the response time of the spliced ​​interface is generally long, which makes it difficult to meet the real-time requirements of users.

[0028] In order to solve the above-mentioned defects in the related art, the first embodiment of the present application provides an online fusion method for multi-source heterogeneous data, such as Figure 1 This is a basic flow chart of the online fusion method for multi-source heterogeneous data provided in this embodiment. The online fusion method for multi-source heterogeneous data includes the following steps:

[0029] Step 101: Convert multi-source heterogeneous data associated with business information into multiple sets of initial API interface data.

[0030] Specifically, multi-source heterogeneous data can be a collection of data from multiple data sources with different origins, formats, and structures. These data sources may include: relational databases such as MySQL and PostgreSQL; NoSQL databases such as MongoDB and Cassandra; text files such as CSV, JSON, and XML files; log files such as web server logs and application logs; and IoT devices such as sensor data and device status. During implementation, data source conversion components can be used to uniformly convert multiple data sources into initial API interface data.

[0031] Step 102: Based on the configuration file corresponding to the user's business needs, multiple groups of initial API interface data are merged in a preset virtual buffer layer to output service data.

[0032] Specifically, in this embodiment, a virtual buffer layer may be constructed between the client and the server, so that multiple interface data may be calculated online in the space of the virtual buffer layer according to user needs, and desired service data may be output.

[0033] Step 103: Encapsulate the service data into target API interface data.

[0034] Specifically, during the encapsulation process, you can design the API interface information according to your needs, that is, determine the interface type (such as REST, GraphQL), design the interface path and request method (such as GET, POST, PUT, DELETE, etc.), design the request parameters and response format, etc. The service data obtained after the fusion calculation is encapsulated into the target API interface data according to the interface situation (for example, the return parameters of the fused interface S can be set to "total score" and "evaluation"). Figure 2 As shown, the data source conversion component and virtual buffer layer in this embodiment can be encapsulated into an API data online data fusion processing device to connect various data sources and perform rapid identification, query, conversion calculations, and encapsulate the calculated service data into an interface service (i.e., target API interface data) in real time. Through the implementation of the above-mentioned method steps of this embodiment, the user's millisecond-level response requirements for interactive services can be achieved, and the data call results of the entire process are not written to the disk or stored in the database (only the data operation log can be retained for subsequent verification and supervision), which can effectively protect data privacy and avoid data leakage.

[0035] In one implementation of this embodiment, the above-mentioned step of converting multi-source heterogeneous data associated with business information into multiple sets of initial API interface data includes: configuring interface information for each data source in the multi-source heterogeneous data associated with business information; according to the interface information, obtaining corresponding data content from the database corresponding to each data source, and encapsulating the data content corresponding to each data source into initial API interface data.

[0036] Specifically, when configuring the interface information, you can configure the data source address, port, username, password, and other information in the CSB bus to read the data source. When performing API interface conversion, you can configure the interface information: interface name, interface input parameters, output parameters, URL, request method, etc., respectively call the data content corresponding to each data source and encapsulate it into the corresponding initial API interface data. The data source conversion component can convert multi-source heterogeneous data into a single API interface type data source for subsequent fusion calculations.

[0037] Furthermore, in one implementation of this embodiment, before the above-mentioned step of converting multi-source heterogeneous data associated with business information into multiple sets of initial API interface data, it also includes: when the multi-source heterogeneous data includes a text file, reading the target data content of the text file; based on a preset database interface conversion model, converting the target data content into a target data source in a database format, and writing the target data source into the database.

[0038] Specifically, when multi-source heterogeneous data includes text files, that is, when the data source is in file format, it is necessary to first convert the format of the data source into a database format, read the file through a component with configured code, and store the file content in the database; then perform API interface conversion.

[0039] In one implementation of this embodiment, before the step of fusing multiple sets of initial API interface data in a preset virtual buffer layer based on the configuration file corresponding to the user's business needs and outputting the service data, it also includes: generating configuration parameters corresponding to each API interface based on the interface document of each API interface; wherein the configuration parameters include at least one of the interface name, interface address, URL, request header, and query parameters; sending request parameters corresponding to each API interface to the server based on the configuration parameters; parsing the response data returned by each API interface in response to the request parameters to obtain result parameters.

[0040] Specifically, such as Figure 2 As shown, this embodiment also provides an online data fusion processing device for API data. The virtual buffer layer includes an API reading component, an online data fusion virtual component, and a service encapsulation component. The API reading component is used to implement functions such as interface debugging, reading, responding, and processing. During this process, the API interface parameters to be called must be configured, including the interface name, address, URL, request header, query parameters, and other information, while also parsing the data content of the server response. API calls can be authenticated using appkey, rid, and sid. Authorization is performed with a 60-second synchronization key, which must be retrieved again after 30 days. This implementation utilizes multi-factor authentication, improving the security of information calls and controlling user permissions and authorized access. The API reading component can use the paramMap, headerMap, and bodyMap collections to record the request parameters corresponding to the API interface. Specific interface information can be read using development tools such as httpclient or okhttp. The online data fusion virtual component is used to implement online fusion calculations of user configuration and data to obtain service data. The service encapsulation component is used to encapsulate service data into target API interface data.

[0041] It should be understood that different API interfaces correspond to required parameters, and specific parameters can be recorded before calling the API interface. For example, the social security interface (interface A) has data on the number of insured persons (the code for this data in interface A can be CBRS1), and the provident fund interface (interface B) has data on the number of provident fund payers (the code for this data can be GJJJNRS1), so the required parameters are inconsistent for different API interfaces, that is, specific parameters need to be set for different interfaces. During the specific implementation process, the interface documents of different interfaces can be obtained, and the code can be edited and set in advance in the API reading component according to the settings of the interface, that is, specific parameters are recorded; so as to realize reading "CBRS1" for interface A and reading "GJJJNRS1" for interface B.

[0042] like Figure 3 The figure shows a workflow diagram related to the online data fusion virtual group of this embodiment, in which a virtual switching layer is established between the client and the server. Among them, the user configuration module can generate a configuration file based on the user's business needs and business rules, so that the fusion rules (i.e., the decision rule engine) can be determined based on the configuration file. When different data items of multiple API interfaces need to be called by configuring the input parameter module, unified configuration instructions are configured to call multiple API interfaces, and finally output the output parameter information of different interfaces. The service data is organized by the result orchestration component, and the service encapsulation component is used to encapsulate the service data into target API interface data.

[0043] Furthermore, in one implementation of this embodiment, the data contents of multiple groups of initial API interface data are respectively encapsulated in multiple API interfaces; accordingly, before the step of fusing the multiple groups of initial API interface data in a preset virtual buffer layer based on the above-mentioned configuration file corresponding to the user's business needs and outputting the service data, it also includes: based on the user's business needs, determining the target data items of each API interface and the configuration information corresponding to each target data item; wherein the configuration information includes the dimensional weight of the target data item in the dimension layer to which it belongs and the influencing factor in the factor layer to which it belongs; all the configuration information is input into the preset configuration conversion model to generate a configuration file.

[0044] Specifically, in the virtual buffer layer, user configuration can be performed through the online data fusion virtual component. In the preset configuration conversion model, users can flexibly configure relevant fusion rules based on business rules and user service information acquisition needs, and then generate corresponding configuration files. More specifically, users can flexibly configure parameters in the dimension layer, dimension weight, factor layer, and influencing factors through partially read-write configuration files. Among them, the configuration conversion model can be formed through big data analysis and feature value extraction. Taking the enterprise evaluation model as an example, through the comprehensive analysis and extraction of big data of enterprise evaluation indicators in different industries and different regions, a comprehensive evaluation model matrix covering multiple dimensions, each dimension containing multiple factors (i.e., the above-mentioned configuration conversion model) is formed. Therefore, in the specific implementation process, based on the actual business needs, for certain enterprises, the dimensions and factor indicators involved in the evaluation can be extracted and their weights and influencing factors can be configured according to the needs of the business and users to form a configuration file.

[0045] For example, when a financial institution needs to evaluate the repayment ability of an enterprise based on its social security information, actual amount of provident fund payment, number of provident fund payers, tax situation, and default record, and give the enterprise a comprehensive score to evaluate its credit limit, it can construct a configuration conversion model (enterprise evaluation model matrix) as shown in Table 1, and then obtain the corresponding configuration file (as shown in Table 2) through the configuration conversion model; when calling multiple API interface parameters, the obtained output parameter values ​​can be converted according to the settings in the configuration file.

[0046] Table 1:

[0047]

[0048] Table 1 (continued):

[0049]

[0050] Table 2:

[0051]

[0052] Table 2 (continued):

[0053]

[0054] Furthermore, in one implementation of the present embodiment, the above-mentioned step of fusing multiple sets of initial API interface data in a preset virtual buffer layer based on the configuration file corresponding to the user's business needs and outputting service data includes: in the preset virtual buffer layer, determining the fusion rules according to the configuration file corresponding to the user's business needs; wherein the fusion rules are associated with the configuration information; calling the result parameters of the target data items returned from each API interface, and creating a MAP set based on the result parameters; fusing the MAP set based on the fusion rules, and outputting the service data.

[0055] Furthermore, in one implementation of this embodiment, the above-mentioned step of fusing the MAP set based on the fusion rule and outputting the service data includes: extracting the first result value corresponding to each target data item from the MAP set based on the fusion rule; recording the first result value into a preset list set, traversing all the second result values ​​in the list set, and outputting the service data.

[0056] Specifically, the online data fusion virtual component can temporarily record the result parameters returned by multiple API interfaces through a MAP collection, extract specific parameters from the MAP collection, and add the specific parameters to the list collection. Finally, it traverses the specific parameter values, matches the user's needs to perform simple logical operations, and finally composes the service data according to the user's needs. Among them, the above-mentioned specific parameters can be determined based on the configuration file; when connecting with different API interfaces, by obtaining the interface documentation, the data parameter name, acquisition method and other information contained in the API interface can be understood; then, according to the needs, request parameters can be sent to the API interface to obtain specific result parameters, for example: obtaining CBRS1 (number of insured persons) from interface A (social security interface).

[0057] For example, when a financial institution needs to evaluate an enterprise's credit limit, the API interfaces used include: social security interface A, provident fund interface B, tax interface C, credit interface D, and administrative interface E.

[0058] First, create a MAP set, which includes the following five sub-MAP sets: MAP < A > , MAP < B > , MAP < C > , MAP < D > , MAP < E > , used to record the result parameters returned by the 5 API interfaces, that is, MAP < A > Record the result parameters returned by calling interface A:

[0059] {a1,a2,a3,a4,a5}, MAP < B > Record the result parameters returned by calling interface B:

[0060] {b1,b2,b3,b4,b5,b6}, MAP < C > Record the result parameters returned by calling interface C:

[0061] {c1,c2,c3,c4,c5}, MAP < D > Record the result parameters returned by calling interface D:

[0062] {d1,d2,d3,d4,d5,d6}, ​​MAP < E > Record the result parameters returned by calling interface E:

[0063] {e1,e2,e3,e4}, then:

[0064]

[0065] Next, create a list collection to temporarily record the first result value extracted from the MAP collection. For example, if a financial institution uses the target data items of interface A: the number of payable persons, the total unit payment base, and the unit payable unified amount, the target data items of interface B: the actual paid amount and the number of depositors, the target data items of interface C: income tax expenses, taxes and additional results, the target data items of interface D: the default period and the default status code results, and the target data item of interface E: the administrative penalty decision document number result, then extract the first result value {a1, a3, a4,, b2, b3, c1, c2, d3, d5, e1} of the above target data items from MAP and record them in the list collection; finally, traverse the second result value recorded in the list collection and output the service data {A', B', C', D', E'} according to the fusion rules of the configuration file. The entire fusion processing process can be expressed by the following formula:

[0066]

[0067] in, represents the configuration conversion model, and K1, K2, K3, K4, and K5 represent the policy rule engine parameters corresponding to the social security interface A, provident fund interface B, tax interface C, credit interface D, and administrative interface E, respectively. Accordingly, the entire fusion process can also be represented by Table 3.

[0068] Table 3:

[0069]

[0070]

[0071] Based on the technical solution of the embodiment of the present application, the multi-source heterogeneous data associated with the business information is converted into multiple sets of initial API interface data; based on the configuration file corresponding to the user's business needs, the multiple sets of initial API interface data are fused in the preset virtual buffer layer to output service data; the service data is encapsulated into target API interface data. Through the implementation of the solution of the present application, a virtual buffer layer is constructed to ensure that all data processing links (including access, conversion and processing) in the fusion process are completed in this layer, which reduces the complexity of the operation, shortens the development link, and can also flexibly configure and develop interfaces according to business needs, which can effectively help developers to develop and use; and the data in the entire process is not written to the disk or stored in the database, which can effectively protect data privacy, avoid the risk of data leakage caused by writing data into the local archive during data processing, and avoid the risk of sensitive data being abused and used multiple times after being stored.

[0072] Figure 4 The method in the second embodiment of the present application is a refined online fusion method for multi-source heterogeneous data, which includes:

[0073] Step 401: Configure interface information for each data source in multi-source heterogeneous data associated with business information.

[0074] Specifically, in this embodiment, when the multi-source heterogeneous data includes a text file, the target data content of the text file is read; based on a preset database interface conversion model, the target data content is converted into a target data source in a database format, and the target data source is written to the database, so as to encapsulate the data of the database formula into initial API interface data.

[0075] Step 402: According to the interface information, the corresponding data content is obtained from the database corresponding to each data source, and the data content corresponding to each data source is encapsulated into initial API interface data.

[0076] Step 403: In the preset virtual buffer layer, the initial API interface data is read through the API reading component.

[0077] Specifically, during the reading process, configuration parameters corresponding to each API interface can be generated based on the interface document of each API interface; wherein the configuration parameters include at least one of the interface name, interface address, URL, request header, and query parameters; based on the configuration parameters, request parameters corresponding to each API interface are sent to the server respectively; and the response data returned by each API interface in response to the request parameters is parsed to obtain result parameters.

[0078] Step 404: Based on the user's business needs, determine the target data items of each API interface and the configuration information corresponding to each target data item.

[0079] Specifically, the configuration information includes the dimension weight of the target data item in the dimension layer to which it belongs and the influencing factor in the factor layer to which it belongs.

[0080] Step 405: Input all configuration information into a preset configuration conversion model to generate a configuration file.

[0081] Step 406: Determine fusion rules based on the configuration file corresponding to the user's business needs.

[0082] The fusion rule is associated with the configuration information.

[0083] Step 407: Call the result parameters of the target data items returned from each API interface, and create a MAP set based on the result parameters.

[0084] Step 408: Fuse the MAP set based on the fusion rule and output service data.

[0085] Step 409: Encapsulate the service data into target API interface data.

[0086] Specifically, this embodiment can achieve rapid interface fusion through the use of a virtual buffer layer. On the basis of ensuring that no traces of data are left during the API fusion processing and that the data is destroyed after use, it can conduct secondary development of the interface in combination with business needs. It is suitable for processing data with relatively complex interfaces and high privacy requirements.

[0087] The refinement method of this embodiment has the following advantages: 1. It can ensure that no traces of data are left during the API data fusion processing process, and the data is destroyed after use. All data processing is carried out in a virtual buffer layer, including access, conversion and processing; it effectively protects the privacy of the data and avoids the security risk of sensitive information being abused after being stored in the database. 2. It can meet the general user's requirements for millisecond-level response for interactive services, greatly improving the response efficiency. 3. It can effectively solve the problem that tools such as Postman and APISQL can only simply add, delete and splice interfaces, and have poor development flexibility, allowing users to flexibly set interface weights, numerical ranges and other indicators according to business rules and business demand configuration files, and is suitable for application scenarios with more complex businesses.

[0088] It should be understood that the size of the serial numbers of the steps in this embodiment does not mean the order in which the steps are executed. The order in which the steps are executed should be determined by their functions and internal logic, and should not constitute a sole limitation on the implementation process of the embodiments of this application.

[0089] Figure 5 The third embodiment of the present application provides an online fusion device for multi-source heterogeneous data. The online fusion device for multi-source heterogeneous data can be applied to the online fusion method for multi-source heterogeneous data. Figure 5 As shown, the online fusion device of multi-source heterogeneous data mainly includes:

[0090] The conversion module 501 is used to convert multi-source heterogeneous data associated with business information into multiple sets of initial API interface data;

[0091] A fusion module 502 is configured to fuse multiple sets of initial API interface data in a preset virtual buffer layer based on a configuration file corresponding to a user's service needs, and output service data;

[0092] The encapsulation module 503 is used to encapsulate the service data into target API interface data.

[0093] In some implementations of this embodiment, the conversion module is specifically used to: configure interface information for each data source in the multi-source heterogeneous data associated with the business information; obtain corresponding data content from the database corresponding to each data source according to the interface information, and encapsulate the data content corresponding to each data source into initial API interface data.

[0094] Furthermore, in some implementations of this embodiment, when the conversion module performs the above-mentioned function of converting multi-source heterogeneous data associated with business information into multiple sets of initial API interface data, it is specifically used to: when the multi-source heterogeneous data includes a text file, read the target data content of the text file; based on a preset database interface conversion model, convert the target data content into a target data source in a database format, and write the target data source into the database.

[0095] In some implementations of this embodiment, the data contents of multiple sets of initial API interface data are respectively encapsulated in multiple API interfaces; based on the configuration file corresponding to the user's business needs, the multiple sets of initial API interface data are fused in a preset virtual buffer layer, and before outputting the service data, the fusion module is also used to: determine the target data items of each API interface and the configuration information corresponding to each target data item based on the user's business needs; wherein the configuration information includes the dimensional weight of the target data item in the dimension layer to which it belongs and the influencing factor in the factor layer to which it belongs; all configuration information is input into a preset configuration conversion model to generate a configuration file.

[0096] Furthermore, in some implementations of this embodiment, the fusion module is specifically used to: determine the fusion rules in a preset virtual buffer layer according to the configuration file corresponding to the user's business needs; wherein the fusion rules are associated with the configuration information; call the result parameters of the target data items returned from each API interface, and create a MAP set based on the result parameters; fuse the MAP set based on the fusion rules, and output service data.

[0097] Furthermore, in some implementations of this embodiment, when the fusion module performs the above-mentioned function of fusing the MAP set based on the fusion rules and outputting the service data, it is also specifically used to: extract the first result value corresponding to each target data item from the MAP set based on the fusion rules; record the first result value into a preset list set, traverse all the second result values ​​in the list set, and output the service data.

[0098] In some implementations of this embodiment, based on the configuration file corresponding to the user's business needs, multiple sets of initial API interface data are merged in a preset virtual buffer layer, and before the service data is output, the fusion module is also used to: generate configuration parameters corresponding to each API interface based on the interface document of each API interface; wherein the configuration parameters include at least one of the interface name, interface address, URL, request header, and query parameters; send request parameters corresponding to each API interface to the server based on the configuration parameters; parse the response data returned by each API interface in response to the request parameters to obtain result parameters.

[0099] According to the online fusion device of multi-source heterogeneous data provided by this embodiment, the multi-source heterogeneous data associated with the business information is converted into multiple sets of initial API interface data; based on the configuration file corresponding to the user's business needs, the multiple sets of initial API interface data are fused in the preset virtual buffer layer to output service data; the service data is encapsulated into target API interface data. Through the implementation of the scheme of this application, a virtual buffer layer is constructed to ensure that all data processing links (including access, conversion and processing) in the fusion process are completed in this layer, which reduces the complexity of the operation, shortens the development link, and can also flexibly configure and develop interfaces according to business needs, which can effectively help developers to develop and use; and the data in the entire process is not written to the disk or stored in the database, which can effectively protect data privacy, avoid the risk of data leakage caused by writing data into the local archive during the data processing process, and avoid the risk of sensitive data being abused and used multiple times after being stored.

[0100] Figure 6 This is an electronic device provided in the fourth embodiment of the present application. This electronic device can be used to implement the online fusion method of multi-source heterogeneous data in the above embodiments, mainly including:

[0101] Memory 601, processor 602, and computer program 603 stored in memory 601 and executable on processor 602. Memory 601 and processor 602 are connected via communication. When processor 602 executes computer program 603, the method of the aforementioned embodiment 1 or 2 is implemented. The number of processors may be one or more.

[0102] The memory 601 can be a high-speed random access memory (RAM) memory or a non-volatile memory such as a disk memory. The memory 601 is used to store executable program codes. The processor 602 is coupled to the memory 601 .

[0103] Furthermore, the embodiment of the present application also provides a computer-readable storage medium, which can be set in the above electronic device, and the computer-readable storage medium can be the above Figure 6 Memory in the illustrated embodiment.

[0104] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the online fusion method for multi-source heterogeneous data in the aforementioned embodiment. Furthermore, the computer-readable storage medium may be any medium capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), RAM, a magnetic disk, or an optical disk.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0106] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of these modules may be selected to achieve the purpose of this embodiment based on actual needs.

[0107] In addition, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or software functional modules.

[0108] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a readable storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned readable storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0109] It should be noted that for the aforementioned method embodiments, for ease of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0110] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0111] The above is a description of the online fusion method, device, equipment and readable storage medium for multi-source heterogeneous data provided by this application. For those skilled in the art, based on the ideas of the embodiments of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An online fusion method for multi-source heterogeneous data, characterized by: include: Converting multi-source heterogeneous data associated with business information into multiple sets of initial API interface data; wherein the data contents of the multiple sets of initial API interface data are respectively encapsulated in multiple API interfaces; Based on user business needs, determine the target data items of each API interface and the configuration information corresponding to each target data item; wherein the configuration information includes the dimension weight of the target data item in the dimension layer to which it belongs and the influencing factor in the factor layer to which it belongs; Inputting all the configuration information into a preset configuration conversion model to generate a configuration file; In a preset virtual buffer layer, a fusion rule is determined according to a configuration file corresponding to a user's service requirement; wherein the fusion rule is associated with the configuration information; Calling the result parameters of the target data item returned from each of the API interfaces, and creating a MAP set based on the result parameters; fusing the MAP set based on the fusion rule and outputting service data; Encapsulate the service data into target API interface data.

2. The online fusion method for multi-source heterogeneous data according to claim 1 is characterized in that: The step of converting multi-source heterogeneous data associated with business information into multiple sets of initial API interface data includes: Configure interface information for each data source in the multi-source heterogeneous data associated with business information; According to the interface information, corresponding data content is obtained from the database corresponding to each of the data sources, and the data content corresponding to each of the data sources is encapsulated into initial API interface data.

3. The online fusion method for multi-source heterogeneous data according to claim 2 is characterized in that: Before converting the multi-source heterogeneous data associated with the business information into multiple sets of initial API interface data, the method further includes: When the multi-source heterogeneous data includes a text file, reading the target data content of the text file; Based on a preset database interface conversion model, the target data content is converted into a target data source in a database format, and the target data source is written into the database.

4. The online fusion method for multi-source heterogeneous data according to claim 1 is characterized in that: The fusing the MAP set based on the fusing rule and outputting service data includes: Extracting a first result value corresponding to each of the target data items from the MAP set based on the fusion rule; The first result value is recorded in a preset list set, all the second result values ​​in the list set are traversed, and service data is output.

5. The online fusion method for multi-source heterogeneous data according to claim 1 is characterized in that: Before calling the result parameter of the target data item returned from each of the API interfaces, the method further includes: Based on the interface documents of each of the API interfaces, respectively generate configuration parameters corresponding to each of the API interfaces; wherein the configuration parameters include at least one of an interface name, an interface address, a URL, a request header, and a query parameter; Sending request parameters corresponding to each of the API interfaces to the server based on the configuration parameters; Parse the response data returned by each of the API interfaces in response to the request parameters to obtain the result parameters.

6. An online fusion device for multi-source heterogeneous data, characterized in that: include: A conversion module, configured to convert multi-source heterogeneous data associated with business information into multiple sets of initial API interface data; wherein the data contents of the multiple sets of initial API interface data are respectively encapsulated in multiple API interfaces; A fusion module is configured to determine, based on user business requirements, target data items of each of the API interfaces and configuration information corresponding to each of the target data items; wherein the configuration information includes the dimension weight of the target data item in the dimension layer to which it belongs and the influencing factor in the factor layer to which it belongs; input all of the configuration information into a preset configuration conversion model to generate a configuration file; determine, in a preset virtual buffer layer, a fusion rule based on the configuration file corresponding to the user business requirements; wherein the fusion rule is associated with the configuration information; call the result parameters of the target data items returned from each of the API interfaces and create a MAP set based on the result parameters; fuse the MAP set based on the fusion rule and output service data; The encapsulation module is used to encapsulate the service data into target API interface data.

7. An electronic device, characterized in that: Comprising a memory and a processor, wherein: The processor is configured to execute a computer program stored in the memory; When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images