Multi-source data fusion method, system, equipment and medium

Abstract

The invention relates to the technical field of data fusion, in particular to a multi-source data fusion method, system and device and a medium. The method comprises the following steps: firstly, according to multi-source data obtained from a multi-source data medium station, constructing a medium station configuration table; secondly, according to the obtained query request, configuring a data interface of the third-party system to a multi-source data middle station, and performing necessary item verification on interface parameters according to a front-end parameter configuration table of the middle station; then judging whether the check-passed query request is subjected to Cartesian integral splitting or not to obtain response messages of a plurality of third-party systems; and finally, carrying out data reorganization on the plurality of response messages to obtain reorganized data, calling a paging algorithm, and feeding the paged reorganized data back to the request end, so that the data access of a plurality of or a single system fused in a similar structure is realized, the interface is flexible to configure, the operation is simple and convenient, and the universality is high.

Description

Technical Field

[0001] The present invention relates to the technical field of data fusion, and in particular to a multi-source data fusion method, system, device and medium. Background Art

[0002] The current general solution for data fusion is to synchronize data or mine data locally across systems and query again. This approach requires hardware storage and software support, which is developed independently for different businesses. It has strong limitations, large investment in manpower and material resources, and high maintenance costs. For example, for big data of equipment performance monitoring of different systems of operators, the equipment has the characteristics of repeated collection of the same nature, and the data storage space requirements are huge. For front-end users who only need to view data, data synchronization and data mining are basically not feasible for this case. It is urgent to provide a feasible and reliable solution for this scenario. Summary of the invention

[0003] In view of the fact that existing data fusion methods require hardware storage and software supporting support, and the problems of strong limitations and high maintenance costs in the development of different businesses, the present invention proposes a multi-source data fusion method, system, equipment and medium; firstly, a middle platform configuration table is constructed according to the multi-source data obtained from the multi-source data middle platform; secondly, according to the obtained query request, the data interface of the third-party system is configured to the multi-source data middle platform, and the interface parameters are checked for required items according to the middle platform front-end parameter configuration table; then, it is determined whether the query request that passes the check is decomposed by Cartesian integrals to obtain response messages from multiple third-party systems; finally, multiple response messages are reorganized to obtain reorganized data, and a paging algorithm is called to feed back the reorganized data after paging to the request end, so as to realize the data access of multiple or single systems with the same structure fusion, and the interface has flexible configuration, simple operation and strong versatility.

[0004] The specific implementation content of the present invention is as follows: A multi-source data fusion method specifically includes the following steps:

[0005] Step S1: Construct a middle station configuration table based on the multi-source data obtained from the multi-source data middle station; the middle station configuration table includes the middle station access identification table SPR_MG_OUTPUT_IDENTITY, the middle station front-end parameter configuration table SPR_MG_INPUT_PARAMETER, the third-party system access configuration table SPR_MG_OUTPUT_IDEN_INTERFACE, the middle station parameter and third-party system field mapping table SPR_MG_OUTPUT_PARAMETER, and the middle station mapping table SPR_MG_REORGANIZE;

[0006] Step S2: According to the query request obtained, the interface parameters mapped with the third-party system are constructed, the data interface of the third-party system is configured to the multi-source data middle platform according to the interface parameters, and the interface parameters are checked for mandatory items according to the constructed middle platform front-end parameter configuration table SPR_MG_INPUT_PARAMETE;

[0007] Step S3: According to the third-party system access configuration table SPR_MG_OUTPUT_IDEN_INTERFACE, it is determined whether the query request that has passed the verification is to be decomposed by Cartesian integral;

[0008] Step S4: Obtain response messages from multiple third-party systems according to the query request decomposed by the Cartesian integral;

[0009] Step S5: reorganize the multiple response messages to obtain reorganized data, call the paging algorithm, and feed back the paging reorganized data to the requesting end.

[0010] The present invention has the following beneficial effects:

[0011] (1) The present invention is applicable to the integration of multiple or single system data access of similar structures in various industries, providing customized output fields and data export; access to a multi-source data center, flexible interface configuration, simple operation, and strong versatility; no storage requirements for target data, and immediate use.

[0012] (2) For big data fusion, compared with traditional data mining and data synchronization, it is economical, practical, efficient and easy to maintain. Economic means less investment in equipment, equipment deployment in the middle platform, and database sharing; practical means only configuration is required to meet the requirements of multi-system data fusion front-end visualization; efficient means using existing system interface resources, with low investment and quick results. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a schematic block diagram of the Cartesian product algorithm process provided by the present invention.

[0014] Figure 2 A schematic diagram of SPR_MG_OUTPUT_IDEN_INTERFACE table data provided in an embodiment of the present invention.

[0015] Figure 3 A schematic diagram of SPR_MG_INPUT_PARAMETER table data provided in an embodiment of the present invention.

[0016] Figure 4 A schematic diagram of the student status table data of a third-party system for escaping the user grade provided in an embodiment of the present invention.

[0017] Figure 5A schematic diagram of the configuration of the unescaped front-end mapping table provided in an embodiment of the present invention.

[0018] Figure 6 A schematic diagram of the un-escaped front-end access structure provided in an embodiment of the present invention.

[0019] Figure 7 A schematic diagram of the configuration of the middle station mapping table after the escape configuration provided in an embodiment of the present invention.

[0020] Figure 8 A schematic diagram of the middle station access structure after the escape configuration provided in an embodiment of the present invention.

[0021] Fig. 9 A schematic diagram of a table data structure after field redefinition provided in an embodiment of the present invention.

[0022] Fig.10 A schematic diagram of the IS_KEY field structure of the data center center mapping table provided in an embodiment of the present invention.

[0023] Fig.11 A schematic diagram of the data flow of a multi-source data center provided in an embodiment of the present invention.

[0024] Fig.12 A schematic diagram of a user submitting a data request access provided in an embodiment of the present invention.

[0025] Fig.13 A schematic diagram of downloading an EXCEL data file provided in an embodiment of the present invention.

[0026] Fig.14 A schematic diagram of the overall process flow of the multi-source data center provided in an embodiment of the present invention.

[0027] Fig.15 A schematic diagram of a third-party system interface access program flow provided in an embodiment of the present invention.

[0028] Fig.16 A schematic diagram of parameters for obtaining access requested by a third party provided in an embodiment of the present invention.

[0029] Fig.17 A schematic diagram of parameters of a third-party system corresponding to grade obtained by associating request parameters provided in an embodiment of the present invention.

[0030] Fig.18 A schematic diagram of a data reorganization process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be understood that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, and therefore should not be regarded as limiting the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technical personnel in this field without making creative work are within the scope of protection of the present invention.

[0032] Embodiment 1:

[0033] This embodiment proposes a multi-source data fusion method, which specifically includes the following steps:

[0034] Step S1: Construct a middle platform configuration table based on the multi-source data obtained from the multi-source data middle platform; the middle platform configuration table includes the middle platform access identification table SPR_MG_OUTPUT_IDENTITY, the middle platform front-end parameter configuration table SPR_MG_INPUT_PARAMETER, the third-party system access configuration table SPR_MG_OUTPUT_IDEN_INTERFACE, the middle platform parameter and third-party system field mapping table SPR_MG_OUTPUT_PARAMETER, and the middle platform mapping table SPR_MG_REORGANIZE.

[0035] Step S2: Based on the query request obtained, construct the interface parameters mapped to the third-party system, configure the data interface of the third-party system to the multi-source data middle platform according to the interface parameters, and verify the mandatory items of the interface parameters according to the constructed middle platform front-end parameter configuration table SPR_MG_INPUT_PARAMETE.

[0036] The step S2 specifically includes the following steps:

[0037] Step S21: Generate interface parameters mapped with the third-party system according to the acquired query request;

[0038] Step S22: configuring the third-party data interface to the multi-source data platform according to the interface parameters, and establishing an access identifier;

[0039] Step S23: configuring external access conditions of the third-party system according to the query field of the third-party data interface;

[0040] Step S24: configure the interface access address, receiving condition field, and message body information of the third-party system.

[0041] Step S3: According to the third-party system access configuration table SPR_MG_OUTPUT_IDEN_INTERFACE, it is determined whether the query request that has passed the verification is to be decomposed by Cartesian integral.

[0042] The step S3 specifically comprises the following steps:

[0043] Step S31: Determine whether to perform Cartesian integral decomposition on the query request that has passed the verification according to the third-party system access configuration table SPR_MG_OUTPUT_IDEN_INTERFACE;

[0044] Furthermore, the specific operation of the Cartesian product splitting in step S31 is: first, according to the third-party system access configuration table SPR_MG_OUTPUT_IDEN_INTERFACE, the type value of the OUT_SOURCE field is determined; if the type value is the URL_MUTI value, a system-level Cartesian product splitting is performed, and a third-party URL is dynamically synthesized; then, the conditional field type of the dynamically synthesized third-party URL is determined; if the conditional field type is the middle platform parameter and the third-party system field mapping table SPR_MG_INPUT_PARAMETER table, and the IN_PARAMTER_TYPE value is stringMuti, a field-level Cartesian product splitting is performed based on the system Cartesian product; otherwise, it is indicated as a non-multiple-choice option or the third-party system can pass multiple-choice values ​​without the need for Cartesian product splitting, and the input parameter mode is maintained to pass the parameters to the third-party interface; wherein, the Cartesian product splitting method is to split the parameters with commas.

[0045] Step S32: If the third-party system is a single-system single-option interface, a first return value is generated and encapsulated into a result set; otherwise, multiple options are split into a single-system single option, a second return value is generated, and encapsulated into a result set.

[0046] Step S4: Obtain response messages from multiple third-party systems according to the query request after decomposition of the Cartesian integral.

[0047] Furthermore, the step S4 specifically includes the following steps:

[0048] Step S41: Determine the interface source type according to the OUT_SOURCE field of the third-party system access configuration table SPR_MG_OUTPUT_IDEN_INTERFACE;

[0049] Step S42: If the interface source type is URL, query the SOURCE_VALUE value of the third-party system access configuration table SPR_MG_OUTPUT_IDEN_INTERFACE from the identityId parameter and the system parameter of the query request, and associate the parameters of the third-party system corresponding to the input parameters according to the access request parameters of the obtained third-party system, and generate a URL response message;

[0050] Step S43: If the interface source type is SOCKET, query the SOURCE_VALUE value of the third-party system access configuration table SPR_MG_OUTPUT_IDEN_INTERFACE according to the connected port IP, associate the input parameter configuration with the actual field, transmit it to the other end according to the protocol link, and receive the SOCKET response message according to the interface;

[0051] Step S44: If the interface source type is CLASS, query the SOURCE_VALUE value of the third-party system access configuration table SPR_MG_OUTPUT_IDEN_INTERFACE, and through the interface method of the instance class, associate the input parameter with the actual field input, and the interface receives the CLASS response message.

[0052] Step S5: reorganize the multiple response messages to obtain reorganized data, call the paging algorithm, and feed back the paging reorganized data to the requesting end.

[0053] Furthermore, the step S5 specifically includes the following steps:

[0054] Step S51: According to the obtained response message, determine whether the request end transmits clear=1. If so, determine whether the result needs to be cleaned. If not, return the unreorganized data. If the result needs to be cleaned, read the JSON_KEY field of IS_KEY=Y in the middle station mapping table SPR_MG_REORGANIZE for cleaning.

[0055] Step S52: Add the cleaned data to the temporary cache and store it, and match it to the records of the middle station mapping table SPR_MG_REORGANIZE according to the system field to redefine the multi-source middle station output message;

[0056] Step S53: determine whether the field needs to be escaped. If not, update the job field in the cache set. If escaped is required, find out whether there is a field to be escaped in the cache set. If so, escape the cached value directly. If not, escape the field to be escaped according to the SQL library and store the escaped value in the temporary cache set.

[0057] Step S54: declare the instance variable resultJson to uniformly store and output the results after redefining the fields in the temporary cache set and escaping the data;

[0058] Step S55: paging and merging the data according to the set paging conditions, and transmitting the paging data to the requesting end.

[0059] In order to better implement the present invention, further, the specific operation of step S52 is: according to the system field, match the record of the middle station mapping table SPR_MG_REORGANIZE table, redefine the multi-source middle station output message, if the corresponding configuration field is JSON_KEY, then after cleaning, convert the JSON_KEY field into the MG_KEY field.

[0060] The step S53 specifically includes the following steps:

[0061] Step S531: Declare the field cache variable according to the SYSTEM value, MG_KEY field and JSON_KEY message value of the middle station mapping table SPR_MG_REORGANIZE table, and use it as the key value of the cache;

[0062] Step S532: Determine whether the field needs to be escaped. If not, update the job field in the cache set. If escape is required, check whether there is a field to be escaped in the cache set. If so, escape the cached value directly. If not, query the field to be escaped from the SQL database to obtain the third-party request source, and execute the SQL query entered from the DICT_CONTENT field in the middle station mapping table SPR_MG_REORGANIZE table to obtain the escaped value, and store the escaped value in the temporary cache set.

[0063] In order to better implement the present invention, further, the step S54 specifically includes the following steps:

[0064] Step S541: Declare the instance variable resultJson, and call the SPR_MG_REORGANIZE table to obtain the IS_KEY field according to the escape value obtained from the third-party system;

[0065] Step S542: If the IS_KEY field is Y, call the cleanData function to match the data of the fields that are not empty. If the match is not satisfied, clear the row of data.

[0066] Step S543: According to the cleaned first-layer key, clean the non-compliant items and save them.

[0067] Furthermore, the specific operation of step S55 is: paging and merging the data according to the set paging condition offset, and transmitting the paging result set to the requesting end.

[0068] Working principle: This embodiment is suitable for the integration of multiple or single system data access of similar structures in various industries, providing customized output fields and data export; access to a multi-source data center, with flexible interface configuration, simple operation, and strong versatility; there is no storage requirement for target data, and it can be used immediately after query.

[0069] Embodiment 2:

[0070] This embodiment is based on the above-mentioned embodiment 1, and takes the student registration system of school A and the student registration system of school B as examples for detailed description, and specifically includes the following steps.

[0071] The middle station configuration table constructed in this embodiment includes: middle station access identification table SPR_MG_OUTPUT_IDENTITY, middle station front-end parameter configuration table SPR_MG_INPUT_PARAMETER, third-party system access configuration table SPR_MG_OUTPUT_IDEN_INTERFACE, middle station parameter and third-party system field mapping table SPR_MG_OUTPUT_PARAMETER, and middle station mapping table SPR_MG_REORGANIZE.

[0072] The middle station access identification table SPR_MG_OUTPUT_IDENTITY is shown in Table 1 below.

[0073] Table 1 Middle platform access identification table

[0074]

[0075] The middle platform front-end parameter configuration table SPR_MG_INPUT_PARAMETER is shown in Table 2 below.

[0076] Table 2 Middle platform front-end parameter configuration table

[0077]

[0078] The third-party system access configuration table SPR_MG_OUTPUT_IDEN_INTERFACE is shown in Table 3.

[0079] Table 3 Third-party system access configuration table

[0080]

[0081] The middle platform parameters and third-party system field mapping table SPR_MG_OUTPUT_PARAMETER is shown in Table 4 below.

[0082] Table 4 Mapping table between middle platform parameters and third-party system fields

[0083]

[0084]

[0085] The middle station mapping table SPR_MG_REORGANIZE is shown in Table 5.

[0086] Table 5 Middle station mapping table

[0087]

[0088] Multi-source data center query interface address: / web / mg.db2v? action=reorganize. Data source: front-end user access request. After configuring the center interface, the student data of the student registration system of school A and the student registration system of school B generate interface parameters mapped to the parameters of the two systems, access the student data of the systems of schools A and B, and merge the output. The following table 6 shows the input parameter table.

[0089] Table 6 Input parameters

[0090]

[0091] Message body example:

[0092] http: / / 172.16.30.96:8180 / web / mg.db2v?action=reorganize&identityId=userInfo&system=01,02&offset=1&limit=10&grade=9. Table 7 below shows an output parameter table.

[0093] Table 7 Output parameter table

[0094]

[0095] The result message description is shown in Table 8 below.

[0096] Table 8 result message correspondence table

[0097]

[0098] List message description is shown in Table 9 below.

[0099] Table 9 result message correspondence table

[0100]

[0101] Sample message:

[0102]

[0103]

[0104]

[0105]

[0106] The sample input of school A is as follows:

[0107] The student data interface address of school A: / topic / webservice / sprWebsMgr.db2v? action=zwjson. Data source: The middle platform configuration parameters are dynamically combined into access requests. The interface provides the query input conditions of grade, student number, and name, and returns the result set.

[0108] The input parameters of School A are shown in Table 10.

[0109] Table 10A School input parameter table

[0110] Field Name Field Description Data Types Remark azGrade grade String studentCode Student ID String userName Name String

[0111] Message body example:

[0112] http: / / 10.180.74.128:8082 / topic / webservice / sprWebsMgr.db2v?action=zwjson&xmlName=asUserInfo&xmlName=asUserInfo&grade=9. The output parameters of school A are shown in Table 11 below.

[0113] Table 11A School output parameter table

[0114] Field Name Field Description Data Types Remark code Return code int Service processing success: 200, service processing exception: 400; Result Message Collection InfoList Message Collection

[0115] The result message description is shown in Table 12 below.

[0116] Table 12A School result message correspondence table

[0117] Field Name Field Description Data Types createTime Creation time datatime studentCode Student ID string userGrade age int userName Name string userId Student ID string userAge Student Age int

[0118] A sample message from school A is shown below.

[0119]

[0120]

[0121] The input sample of school B is as follows:

[0122] The interface address of the student data of School B is: / web / webservice / sprWebsMgr.db2v? action=zwjson. Data source: The configuration parameters of the middle platform are dynamically combined into access requests. The interface provides the query input conditions of grade, student number, and name, and returns the result set. The input parameters of School B are shown in Table 13 below.

[0123] Table 13B School Input Parameters Table

[0124] Field Name Field Description Data Types azGrade grade String studentCode Student ID String userName Name String

[0125] An example message body is as follows:

[0126] http: / / 2.0.0.1:8180 / web / webservice / sprWebsMgr.db2v? action=zwjson&xmlName=azUserInf o&azGrade=9.

[0127] Output the same type of response data as the same type of JSON as that of school A (for details, please refer to the data with the name B in the above-mentioned middle station message example)

[0128] Step S3: According to the third-party system access configuration table SPR_MG_OUTPUT_IDEN_INTERFACE, it is determined whether the query request that has passed the verification is to be decomposed by Cartesian integral;

[0129] like Figure 1 As shown, Cartesian integral decomposition is applicable to scenarios where the front-end needs multiple or one multiple options, while the third-party interface only accepts single value transmission. It is necessary to perform Cartesian integral decomposition on the multiple options, and then request access to the third-party system multiple times to retrieve the value returned; the Cartesian product algorithm is described in Table 14 below.

[0130] Table 14 Cartesian product algorithm corresponding table

[0131]

[0132] SPR_MG_OUTPUT_IDEN_INTERFACE table data, such as Figure 2 shown.

[0133] SPR_MG_INPUT_PARAMETER table data, such as Figure 3 shown.

[0134] Cartesian product splitting formula = system parameter value (OUT_SOURCE is URL_MUTI, the system comma splitting parameter of the request) * SPR_MG_INPUT_PARAMETER table field value (IN_PARAMTER_TYPE is the comma splitting parameter of the stringMuti request field); For example, in this example, the system and grade need to be checked:

[0135] http: / / 172.16.30.96:8180 / web / mg.db2v?action=reorganize&identityId=userInfo&system=01,02&offset=1&limit=10&grade=8,9

[0136] The program will automatically decompose 01 (A school student registration system) and 02 (B school student registration system) according to the configuration information OUT_SOURCE value URL_MUTI. After the decomposition is completed, it will dynamically synthesize the dynamic third-party URL in a loop.

[0137] If the dynamically synthesized condition field type is the IN_PARAMTER_TYPE value of the SPR_MG_INPUT_PARAMETER table and is stringMuti, Cartesian product splitting is performed again.

[0138] According to the formula, 1 and 2 together form four groups of access request parameters:

[0139] A school student registration system request:

[0140] http: / / 10.180.74.128:8082 / topic / webservice / sprWebsMgr.db2v?action=zwjson&xmlName=asUserInfo&xmlName=asUserInfo&system=01&grade=8

[0141] http: / / 10.180.74.128:8082 / topic / webservice / sprWebsMgr.db2v?action=zwjson&xmlName=asUserInfo&xmlName=asUserInfo&system=01&grade=9

[0142] B School Student Registration System Request:

[0143] http: / / 2.0.0.1:8180 / web / webservice / sprWebsMgr.db2v?action=zwjson

[0144] &xmlName=azUserInfo&system=02&azGrade=8

[0145] http: / / 2.0.0.1:8180 / web / webservice / sprWebsMgr.db2v?action=zwjson

[0146] &xmlName=azUserInfo system=02&azGrade=9

[0147] Note: If the front end passes 3 studentCodes and 3 names, it will form 2*2*3*3=36 third-party request access addresses.

[0148] This embodiment takes the applicable scenario of multiple systems with data merging query requirements as an example to illustrate the specific process of the fusion algorithm. The operator implements a single code for the site, collects the performance of equipment under the energy system, operation and maintenance monitoring system, and intelligent control cloud system sites, and queries the performance results of the equipment at the same site at a certain time based on the site ID; it is suitable for system data integration display with fusion requirements in various industries.

[0149] Fusion algorithm description: includes flexible third-party data source definition and implementation methods, data escape methods, field redefinition methods, and data union algorithms.

[0150] Third-party data source definition: The TRAN_SQL_DB field in the SPR_MG_OUTPUT_IDEN_INTERFACE table declares the database mapping information, through which you can link to the configured database; for example, the value of TRAN_SQL_DB in the SPR_MG_OUTPUT_IDEN_INTERFACE table is db, and the value db from the configuration information defulatDb corresponds to TRAN_SQL_DB, so you can connect to the corresponding database;

[0151] defulatDb=db

[0152] db.datasource=

[0153] db.type=mysql

[0154] db.username = dbname

[0155] db.password = dbpassword

[0156] db.driver=org.gjt.mm.mysql.Driver

[0157] db.url=jdbc:mysql: / / xx.xx.xx.xx:3306 / db? useUnicode=true&characterEncoding=UTF-8

[0158] The data escape method is as follows: SPR_MG_REORGANIZE table MG_KEY is an output field. Some field values ​​are numbers, such as the value of the status class. The front-end readability is poor, so numerical escape is required. For example, gender 0 represents female and 1 represents male. Data escape is to convert the numeric value of the field into Chinese.

[0159] Data escape business logic: In order to improve performance, the middle platform declares a field cache variable, the content of which is the SYSTEM value of the SPR_MG_REORGANIZE table + _ + MG_KEY field + JSON_KEY message value as the cache key value, and the escape value is value. For example, the corresponding escape value of 01_userGrade_9 is the third grade of junior high school. The field that needs to be escaped is first checked in the escape cache to see if it exists. If it exists, directly take the value of the escape cache and save it to the temporary cache set. If there is no escape cache, query the database SQL for escape (see the following example of escape), and place the value in the escape cache, and then save the escape value to the temporary cache set.

[0160] There are two ways to escape: 1. Set up a data dictionary and save the fields that need to be escaped in the dictionary in advance, and replace them from the cache when used; 2. Query the escape word from the database, obtain the above-mentioned third-party data source, and then execute the query from the SQL entered in DICT_CONTENT in the SPR_MG_REORGANIZE table to obtain the escape value.

[0161] For example: the student status table data of a third-party system (the word that needs to be escaped is the user's grade), such as Figure 4 shown.

[0162] Before escaping, the SPR_MG_REORGANIZE table configuration is as follows: Figure 5 As shown, the access results of the middle platform are as follows: Figure 6 After the escape configuration, the SPR_MG_REORGANIZE table configuration is as follows Figure 7 As shown, the access results of the middle platform are as follows Figure 8 As shown in the following example, after escaping, the value of grade=9 is returned as "junior high school grade 9". Fig. 9 The following is a method for redefining fields. According to system=01 (or 02), the corresponding records in the SPR_MG_REORGANIZE table are matched to redefine the output message of the middle station. The data returned by the third party is:

[0163]

[0164] The corresponding configuration field is JSON_KEY. After cleaning (see the details of the data union algorithm), JSON_KEY is then transformed into MG_KEY so that after the multi-source (third-party) system messages are redefined, they can be provided to the front-end users for browsing and display in a unified format;

[0165] The data union algorithm is as follows. The data union algorithm has two characteristics: 1. First, declare an instance variable resultJson, whose function is to merge the json data of each access result; 2. Second, the json formats returned by all requests to the third-party systems are not unified, and the levels may be multi-level, while the data that needs to be merged only requires the key main body. In order to accurately obtain the main body data, it needs to be cleaned by a general method before it can be merged. The specific approach is as follows:

[0166] Data returned by the third party:

[0167]

[0168]

[0169] In the data middle platform, the role of the IS_KEY field in the SPR_MG_REORGANIZE table is to clean the key. As Fig.10 shown.

[0170] If IS_KEY is Y, the cleaning function cleanData performs data matching on the fields that cannot be empty. If not satisfied, this row of data is cleared. After traversing the json data, json data will be obtained. In the second step, according to the studentCode in the middle platform configuration where IS_KEY is Y, the first-level key after cleaning is matched, and the non-conforming items are cleared and saved in resultJson:

[0171] [{

[0172] studentCode = 201407310001,

[0173] userGrade = 6,

[0174] createTime = 2024-04-28 16:18:54.0,

[0175] userName = Li Siyi A,

[0176] userId = 11e cdc9821a8a3eb8a6800163e010111,

[0177] userAge = 12

[0178] }]

[0179] This process is repeated until all interface data are combined and output.

[0180] A complete data union execution: The data center accesses multiple source systems. After being split by the Cartesian product algorithm, there will be multiple accesses to the third-party system and multiple return values ​​will be collected. Take the request to access the center as an example:

[0181] http: / / 172.16.30.150:8180 / web / mg.db2v?action=reorganize&identityId=userInfo&system=01,02&offset=1&limit=10&grade=6,9

[0182] After Cartesian product splitting, execute:

[0183] http: / / 10.180.74.128:8082 / topic / webservice / sprWebsMgr.db2v?action=zwjson&xmlName=asUserInfo&userGrade=6

[0184] Access request results:

[0185]

[0186] http: / / 10.180.74.128:8082 / topic / webservice / sprWebsMgr.db2v?action=zwjson&xmlName=asUserInfo&userGrade=9

[0187] Access request results:

[0188]

[0189] http: / / 2.0.0.1:8180 / web / webservice / sprWebsMgr.db2v?action=zwjson&xmlName=azUserInfo&azGrade=6

[0190] Access request results:

[0191]

[0192]

[0193] http: / / 2.0.0.1:8180 / web / webservice / sprWebsMgr.db2v?action=zwjson&xmlName=azUserInfo&azGrade=9

[0194] Access request results:

[0195]

[0196]

[0197] Data union result (after escaping):

[0198]

[0199] The paging algorithm is as follows: After the data is combined, it needs to be sent to the front end after the paging algorithm is completed. The resultJson is set according to the paging condition offset starting at the number of data, and the limit is N per page. For example, offset = 6, limit = 5. Then, the result set (9 data) of resultJson should be taken out; records 6-9 should be parsed and returned according to the paging results specified by the middle platform.

[0200] The application of multi-source data middle platform needs to go through two steps: first, log in to the multi-source data middle platform to configure the interface; then, based on the identifier generated by the configuration, the terminal can access the application.

[0201] This embodiment uses the example of merging students from the student registration system of school A and the student registration system of school B to illustrate the usage process.

[0202] First, configure the two school data interfaces to the middle platform and establish the access identifier of userInfo. After the record is configured, it has the ability to provide access to the terminal system.

[0203] Middle platform data interface:

[0204] / mg.db2v?action=reorganize&identityId=userInfo&system=01,02&offset=1&limit=10&grade=9.

[0205] Download the data from the middle platform:

[0206] / mg.db2v?action=mgJsonExcel&identityId=userInfo&system=01,02&offset=1&limit=100&grade=9.

[0207] Next, configure the external access conditions. This condition must have a common query field for multiple source systems (the query conditions of the student registration system A and the student registration system B are consistent). The sample provides three search conditions: grade, studentCode, and userName. Finally, configure the third-party access source. Configure the interface access address of the third-party system.

[0208] Configure the condition fields received by the third-party system. After the external parameter request conditions for accessing the middle station are processed, the conditions, namely grade, studentCode, and userName, will forward the request to the third-party system; Fig.14 As shown, configure the access parameters for school A.

[0209] Configure the access parameters of school B. They may not be completely consistent with the [External Access Conditions] of the middle office, and they must be consistent with the query field definitions required by the actual interface, but the number and correspondence of query conditions under the multi-source system must be consistent. For example, the middle office interface parameter is grade, and the query condition of the school B interface is azGrade. Then configure the field information of the json message body of the student registration system A.

[0210] Indicator standardization is the field used by the middle station MG_KEY. The purpose of MG_KEY is to escape the unique identifier of the cache and the message definition of the json field of the middle station output front end. Since the fields defined by multiple source systems are consistent, the system defaults to only taking the records of the 01 system (the sample 01 system is the student status system of School A) as the standard output field of the middle station.

[0211] The interface indicator JSON_KEY is the json message field of the student registration system; whether to clean the field IS_KEY, see the data union algorithm in the fusion algorithm; escape sql, DICT_CONTENT, see the data escape method in the fusion algorithm; value source ORG_PARA value is 1: data is taken from the third-party interface response message; value is 2: data is taken from the input parameter; download indicator IS_DOWN The field with value Y is used as the EXCEL export field, and the field with value N is ignored. The system defaults to only using the 01 system as the export configuration; indicator order DOWN_ORD The order of exporting EXCEL table fields; download display name DOWN_TITLE Customize the EXCEL export display name; configure the field information of the json message body of school B.

[0212] The original intention of the multi-source data middle platform program design is to solve the big data integration needs under multiple systems. The industry's common data json format specification is used as the communication standard. The access source is not limited in form, and the data matching the business can be integrated and output by the present invention. The design concept of the middle platform is elaborated in detail from the two aspects of business process overview and program design.

[0213] The data flow of multi-source data platform is as follows Fig.11 As shown, the specific process is described as follows: A user submits a data request to access: http: / / 172.16.30.150:8180 / web / mg.db2v?action=reorganize&identityId=userInfo&system=01,02&offset=6&limit=5&grade=6,9.

[0214] The data center performs Cartesian integration after configuring the information, accesses the student status systems A and B, obtains the return results of the two external systems, and integrates the feedback results. The terminal receives the result set returned by the interface call, such as Fig.12 shown.

[0215] User submits download request:

[0216] http: / / 172.16.30.150:8180 / web / mg.db2v?action=mglsonExcel&identityld=userinfo&system=01,02&offset=6&limit=5&grade=6,9.

[0217] like Fig.13 As shown, downloading EXCEL data files, the middle platform download control has the following characteristics: the data source inherits the middle platform interface, uses the jxls component, and outputs the middle platform result set to the EXCLE file; the data volume is less than or equal to 65535 records, and the download format is a .xls suffix file; the data volume is greater than 65535 records, and the program automatically generates multiple EXCLE files separated by every 65535 records. The file name is named after the IDEN_TITLE value of SPR_MG_OUTPUT_IDENTITY (middle platform access identification table), and then adds "_1", "_2" and other numbers for accumulation; when the number of downloaded files is greater than 1, all data files are automatically packaged into zip, and the output download is named with the IDEN_TITLE value.zip.

[0218] The overall program design of the multi-source data center includes three algorithms: Cartesian product, fusion, and paging, and two sub-program designs: accessing third-party systems and data reorganization. The overall program flow of the multi-source data center is as follows: Fig.14 The specific process is as follows:

[0219] 1) User initiates a query request:

[0220] http: / / 172.16.30.150:8180 / web / mg.db2v?action=reorganize&identityId=userInfo&system=01,02&offset=6&limit=5&grade=6,9;

[0221] 2) The middle station obtains the link parameters and makes a logical judgment with the mandatory field (IS_KEY) configured in SPR_MG_INPUT_PARAMETER;

[0222] Table 15 Logical judgment table

[0223]

[0224] 3) If the required items fail to be checked, the process ends and the error reason is returned; if the required items pass the check, proceed to 4);

[0225] 4) According to the SPR_MG_OUTPUT_IDEN_INTERFACE table, if the OUT_SOURCE value is not URL_MUTI, no splitting is performed; if the OUT_SOURCE value is URL_MUTI, a system-level Cartesian integral split is performed;

[0226] 5) Then, check whether the IN_PARAMTER_TYPE value in the SPR_MG_INPUT_PARAMETER table is stringMultiple. If so, perform field-level Cartesian integral decomposition based on the system Cartesian product.

[0227] 6) No, indicating that the field is not a multiple-choice field or the third-party system can pass multiple-choice values ​​without Cartesian integral decomposition, and the input parameter method is maintained to pass the parameters to the third-party interface;

[0228] 7) Access the third-party system interface to obtain a response message;

[0229] 8) Reorganize all third-party system interface messages;

[0230] 9) The reorganized data is output in accordance with the paging algorithm of the middle station;

[0231] 10) The paginated data is fed back to the front end, ending the platform process.

[0232] like Fig.15 The following is a flowchart of accessing the third-party system interface. The specific process is as follows:

[0233] The third-party system interface access is mainly about the design of the access method. There are two core points: First, determine which of the interface source is CLASS / URL / URL_MUTI / SOKET according to the SPR_MG_OUTPUT_IDEN_INTERFACE table OUT_SOURCE. Second, how to generate the interface access address in the four ways and determine the complete address of the access, including the construction of parameter integrity. The following describes the implementation design of the four methods respectively.

[0234] (1) URL (URL_MUTI): The URL and URL_MUTI address generation methods are exactly the same, the difference is that the URL does not have Cartesian integral decomposition. This example URL_MUTI is used to illustrate how to build a third-party http link. User request:

[0235] http: / / 172.16.30.150:8180 / web / mg.db2v?action=reorganize&identityId=userInfo&system=01,02&offset=6&limit=5&grade=6,9.

[0236] Generate one of the addresses accordingly:

[0237] http: / / 2.0.0.1:8180 / web / webservice / sprWebsMgr.db2v? action=zwjson&xmlName=azUserInf o&azGrade=9.

[0238] First, query the SOURCE_VALUE value of the SPR_MG_OUTPUT_IDEN_INTERFACE table from the identityId and system parameters of the access request:

[0239] http: / / 2.0.0.1:8180 / web / webservice / sprWebsMgr.db2v?action=zwjson&xmlName=azUserInfo.

[0240] Secondly, obtain the parameters requested by the third party, the input parameters of the third-party interface corresponding to the grade input parameter, such as Fig.16 As shown in the following figure. Through the request parameter, the parameter of the third-party system corresponding to grade is azGrade. Fig.17 As shown. So the final result is:

[0241] http: / / 2.0.0.1:8180 / web / webservice / sprWebsMgr.db2v? action=zwjson&xmlName=azUserInf o&azGrade=9.

[0242] (2) SOCKET: SOCKET corresponds to the client connection IP: port, such as the SOURCE_VALUE value.

[0243] 192.168.1.3:10001, after the program is connected, the input parameter configuration above is associated with the actual field, and transmitted to the other end according to the protocol link, and the response message is received according to the interface.

[0244] (3) CLASS: CLASS corresponds to the Java localization implementation class of the third-party system interface, such as SOURCE_VALUE value. com.gzdec.spire.md.third.hisperformance.HisPerformance. Through the interface method of the instance class, the above configuration of the input parameter is associated with the actual field input, and an interface is provided to receive the response message.

[0245] like Fig.18 The figure shows a schematic diagram of the data reorganization process flow, and the process flow is described as follows:

[0246] 1) The middle station receives a message returned by the third-party system;

[0247] 2) According to whether the front end transmits clear = 1, determine whether the result needs to be cleaned. If not, return the unreorganized data to the next link;

[0248] 3) If cleaning is required, read the JSON_KEY(studentCode) of IS_KEY=Y in the SPR_MG_REORGANIZE table, perform cleaning and perform the fusion algorithm - data union algorithm;

[0249] 4) After cleaning is completed, the data is added to the temporary cache set for storage and fusion algorithm - data union algorithm;

[0250] 5) According to system=01 (or 02), the corresponding record of the SPR_MG_REORGANIZE table is matched, and the middle station output message is redefined to perform the fusion algorithm-field redefinition method;

[0251] 6) Determine whether the field needs to be escaped. If not, update the job field in the cache set;

[0252] 7) If escape is required, perform fusion algorithm-data escape method;

[0253] 8) The instance variable resultJson stores the results of redefining the fields in the temporary cache set and escaping the data, thus ending this process.

[0254] Working principle: This embodiment is suitable for the integration of multiple or single system data access in the same structure in various industries, providing custom output fields and data export; access to multi-source data middle platform, flexible interface configuration, simple operation, and strong versatility;

[0255] There is no storage requirement for the target data, which can be used immediately after query; projects with huge data volume and multi-source system data fusion requirements, such as operator big data Mongdb, hbase and other data types.

[0256] The other parts of this embodiment are the same as those of the above-mentioned embodiment 1, and thus will not be described in detail.

[0257] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A multi-source data fusion method, characterized in that: The specific steps include: Step S1: construct a middle platform configuration table based on the multi-source data obtained from the multi-source data middle platform; The middle station configuration table includes a middle station access identification table, a middle station front-end parameter configuration table, a third-party system access configuration table, a middle station parameter and third-party system field mapping table, and a middle station mapping table; Step S2: According to the query request obtained, the interface parameters mapped with the third-party system are constructed, the data interface of the third-party system is configured to the multi-source data middle platform according to the interface parameters, and the interface parameters are checked for mandatory items according to the constructed middle platform front-end parameter configuration table; Step S3: according to the third-party system access configuration table, determine whether to perform Cartesian integral decomposition on the query request that has passed the verification; Step S4: Obtain response messages from multiple third-party systems according to the query request decomposed by the Cartesian integral; Step S5: reorganize the multiple response messages to obtain reorganized data, call the paging algorithm, and feed back the paging reorganized data to the requesting end.

2. A multi-source data fusion method according to claim 1, characterized in that: The step S2 specifically includes the following steps: Step S21: Generate interface parameters mapped with the third-party system according to the acquired query request; Step S22: configuring the third-party data interface to the multi-source data platform according to the interface parameters, and establishing an access identifier; Step S23: configuring external access conditions of the third-party system according to the query field of the third-party data interface; Step S24: configure the interface access address, receiving condition field, and message body information of the third-party system.

3. A multi-source data fusion method according to claim 2, characterized in that: The step S3 specifically comprises the following steps: Step S31: Determine whether to perform Cartesian integral decomposition on the query request that has passed the verification according to the third-party system access configuration table; Step S32: If the third-party system is a single-system single-option interface, a first return value is generated and encapsulated into a result set; otherwise, multiple options are split into a single-system single option, a second return value is generated, and encapsulated into a result set.

4. A multi-source data fusion method according to claim 3, characterized in that: The specific operation of the Cartesian product splitting in step S31 is: first, according to the third-party system access configuration table, the type value of the OUT_SOURCE field is determined. If the type value is the URL_MUTI value, a system-level Cartesian product splitting is performed, and a third-party URL is dynamically synthesized; then, the conditional field type of the dynamically synthesized third-party URL is determined. If the conditional field type is the middle platform parameter and the IN_PARAMTER_TYPE value of the third-party system field mapping table is stringMuti, a field-level Cartesian product splitting is performed based on the system Cartesian product. Otherwise, it is indicated as a non-multiple-choice option or the third-party system can pass multiple-choice values ​​without Cartesian product splitting, and the input parameter mode is maintained to pass the parameters to the third-party interface; wherein, the Cartesian product splitting method is to split the parameters with commas.

5. The multi-source data fusion method according to claim 1, characterized in that: The step S4 specifically comprises the following steps: Step S41: Determine the interface source type according to the OUT_SOURCE field of the third-party system access configuration table; Step S42: If the interface source type is URL, query the SOURCE_VALUE value of the third-party system access configuration table from the identityId parameter and the system parameter of the query request, and associate the parameters of the third-party system corresponding to the input parameters according to the access request parameters of the obtained third-party system, and generate a URL response message; Step S43: If the interface source type is SOCKET, query the SOURCE_VALUE value of the third-party system access configuration table according to the connected port IP, associate the input parameter configuration with the actual field, transmit it to the other end according to the protocol link, and receive the SOCKET response message according to the interface; Step S44: If the interface source type is CLASS, query the SOURCE_VALUE value of the third-party system access configuration table, and associate the input parameter configuration with the actual field input through the interface method of the instance class, and the interface receives the CLASS response message.

6. The multi-source data fusion method according to claim 1, characterized in that: The step S5 specifically comprises the following steps: Step S51: According to the obtained response message, determine whether the request end clear is empty. If it is not empty, determine whether the result needs to be cleaned. If it does not need to be cleaned, return the unreorganized data. If the result needs to be cleaned, read the JSON_KEY field of IS_KEY=Y in the middle station mapping table for cleaning; Step S52: Add the cleaned data to the temporary cache and store it, and match it to the records in the middle station mapping table according to the system field to redefine the multi-source middle station output message; Step S53: determine whether the field needs to be escaped. If not, update the job field in the cache set. If escaped is required, find out whether there is a field to be escaped in the cache set. If so, escape the cached value directly. If not, escape the field to be escaped according to the SQL library and store the escaped value in the temporary cache set. Step S54: declare the instance variable resultJson to uniformly store and output the results after redefining the fields in the temporary cache set and escaping the data; Step S55: paging and merging the data according to the set paging conditions, and transmitting the paging data to the requesting end.

7. A multi-source data fusion method according to claim 6, characterized in that: The specific operation of step S52 is: according to the system field, match the record of the middle station mapping table, redefine the multi-source middle station output message. If the corresponding configuration field is JSON_KEY, after cleaning, the JSON_KEY field is converted into the MG_KEY field.

8. The multi-source data fusion method according to claim 6, characterized in that: The step S53 specifically includes the following steps: Step S531: Declare a field cache variable according to the SYSTEM value, MG_KEY field and JSON_KEY message value of the middle station mapping table, and use it as the key value of the cache; Step S532: Determine whether the field needs to be escaped. If not, update the job field in the cache set. If escape is required, check whether there is a field to be escaped in the cache set. If so, directly escape the cached value. If not, query the field to be escaped from the SQL database to obtain the third-party request source, execute the query from the SQL entered in the DICT_CONTENT field of the middle platform mapping table to obtain the escaped value, and store the escaped value in the temporary cache set.

9. The multi-source data fusion method according to claim 6, characterized in that: The step S54 specifically includes the following steps: Step S541: Declare the instance variable resultJson, and call the middle platform mapping table to obtain the IS_KEY field according to the escape value obtained from the third-party system; Step S542: If the IS_KEY field is Y, call the cleanData function to match the data of the fields that are not empty. If the match is not satisfied, clear the row of data. Step S543: According to the cleaned first-layer key, clean the non-compliant items and save them.

10. A multi-source data fusion method according to claim 6, characterized in that: The specific operation of step S55 is: paging and merging the data according to the set paging condition offset, and transmitting the paging result set to the request end.

Images