A multi-source heterogeneous data interaction and fusion method and system for a shield scenario

Abstract

The application provides a multi-source heterogeneous data interaction and fusion method and system for a shield scene, and relates to the technical field of data fusion.The application comprises the following steps: integrating multi-source heterogeneous data sources from different systems by using a network service Webservice technology, establishing databases under different construction scenes, and forming a multi-source database; presetting a micro-service granularity, and performing data transfer on data of the multi-source database by using multiple micro-service components; performing unified semantic query calling on output data after transfer by using a unified data interaction service gateway, and completing multi-source heterogeneous data interaction and fusion.The application divides the whole system into multiple micro-services, and simultaneously communicates between the micro-services by using a unified RestAPI interface, thereby improving the expansibility of the system and reducing the difficulty of operation and maintenance; the core data integration micro-service integrates multi-source heterogeneous data sources by using Webservice and ontology technology, thereby effectively improving the semantic integration degree of data and data interaction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data fusion, in particular to a multi-source heterogeneous data interaction and fusion method and system for a shield scenario. BACKGROUND

[0002] As a kind of high-end intelligent equipment, compared with traditional engineering machinery equipment, the components and system composition of shield machine are various, in order to ensure the stability of the running state, it is necessary to monitor each component and system in real time, and these data will also grow in geometric progression over time. Then due to the wide data sources and semantic heterogeneity of multi-scene data, it will cause data island problem, which cannot effectively utilize these valuable data resources, and cannot provide efficient data interaction application for users, so it is necessary to effectively fuse multi-source heterogeneous data. At present, data fusion includes data level fusion, feature level fusion and decision level fusion, especially data level fusion, also known as data integration, which can fuse and integrate multi-source heterogeneous data at the bottom layer, and then provide a unified data interaction interface for upper layer application.

[0003] At present, data integration mainly has three modes of federal database system, middleware and data warehouse. Federal database system is composed of multiple semi-autonomous databases through confederation, all databases in the federation add interfaces for mutual access to realize interoperation, with the increase of the number of databases, the task of maintaining a large number of interfaces will become very huge; data warehouse integrates the data mode of heterogeneous data sources into a unified global mode, and extracts, converts and loads the data in the heterogeneous data sources into the data warehouse to form the data copy after integration of each data source, complete data integration, however, the data in the data warehouse is new storage, and the internal data is mainly historical data, so there is not only data redundancy, but also poor real-time performance; middleware integrates each data source logically by providing a global mode, the data of each data source is still saved in independent data source, and each type of data request operation of each data source is performed in the global mode, while the middleware can decompose the operation request for the global mode into sub-requests for the local mode of each data source according to the predefined mapping rule, and dispatch them to the corresponding data source adapter, the adapter converts each sub-request into a request that can be recognized and processed by each data source, and returns the execution result after encapsulation, thereby providing a unified data interaction interface for users, the middleware does not need to maintain a large number of interfaces, and has strong data real-time performance and low redundancy. Therefore, the data integration based on middleware is the current mainstream way.

[0004] Nowadays, the implementation technology adopts the concept architecture of SOA (Service-Oriented Architecture), adopts the Web service technology to complete the fusion and interaction of multi-source heterogeneous data, and provides data services for users through an ESB (Enterprise Service Bus), abstracts the complexity of the bottom layer, and then users can access independent services without needing to understand the service bottom layer platform implementation. The SOA is a design method of a software system, and the Web service technology is a technology for implementing the architecture, which realizes the efficient integration of multi-source heterogeneous data sources through three elements of SOAP (Simple Object Access Protocol), WSDL (Web Services Description Language) and UDDI (Universal Description Discovery and Integration).

[0005] However, with the advent of the big data era, the existing technology mainly has the following defects: the enterprise service bus is relatively complex, lacks effective service management, the system has poor expansibility, operation and maintenance are complex, and the semantic integration degree is not high. SUMMARY

[0006] In view of the problems that the enterprise service bus is relatively complex, lacks effective service management, the system has poor expansibility, operation and maintenance are complex, and the semantic integration degree is not high in the prior art, the application provides a multi-source heterogeneous data interaction and fusion method and system for a shield scene.

[0007] To solve the above technical problems, the application provides the following technical solutions.

[0008] On one hand, a multi-source heterogeneous data interaction and fusion method for a shield scene is provided, which is applied to an electronic device and includes the following steps.

[0009] S1: Multi-source heterogeneous data sources from different systems are integrated by using a network service Web service technology, a database under different construction scenes is established, and a multi-source database is formed.

[0010] S2: A micro-service granularity is preset, and data in the multi-source database is transferred through a plurality of micro-service components.

[0011] S3: Unified semantic query calling is performed on the output data after the transfer through a unified data interaction service gateway, and the multi-source heterogeneous data interaction and fusion are completed.

[0012] Optionally, in step S1, the multi-source heterogeneous data sources from different systems are integrated by network service Web service technology, databases under different construction scenes are established, and a multi-source database is formed, including:

[0013] S11: unified calling and integration of multi-source heterogeneous data sources across platforms by network service Web service technology;

[0014] S12: introducing a hybrid ontology idea to automatically build a local ontology for different data sources;

[0015] S13: building a global ontology using domain expert knowledge, mapping the local ontology and the global ontology, and integrating data structures; on the basis of data structure integration, databases under different construction scenes are established, and a multi-source database is formed.

[0016] Optionally, the multi-source database types include: relational database management system MySQL, Oracle database, DB2, and SQL Server; and the data type formats include: comma-separated value file format CSV, file transfer protocol FTP, electronic spreadsheet EXCEL, document, and relational data.

[0017] Optionally, in step S2, a micro-service granularity is preset, and a plurality of micro-service components are used for data transfer of the multi-source database, including:

[0018] S21: presetting a micro-service granularity and dividing a plurality of micro-service components; the micro-service components are registered in a service registration center;

[0019] S22: according to the application of the user data request, the corresponding micro-service component is called for the user in the service registration center;

[0020] S23: the plurality of micro-service components communicate with the micro-service core module through a Rest API interface to obtain the request data of the user;

[0021] S24: the micro-service core module is used for fusing the obtained request data;

[0022] S25: according to the fused data, the micro-service core module is used for data redistribution.

[0023] Optionally, the plurality of micro-service components include: a data visualization micro-service component, a data analysis micro-service component, and a fault prediction micro-service component.

[0024] Optionally, each micro-service component is preset with a fixed function and can be independently run.

[0025] Optionally, in step S24, the micro-service core module fuses the obtained request data, including:

[0026] S241: Obtain relational schema information of a data source through Java Database Connectivity (JDBC), and formulate a mapping rule; automatically construct a local ontology for the data source by using the relational schema information through jena technology; the data source is request data of each user; the relational schema information of the data source includes table name metadata, data type metadata, and constraint relationship metadata;

[0027] S242: Construct a global ontology by using a seven-step method and combining with shield machine field knowledge; and perform mapping and fusion of the local ontology and the global ontology by using an ontology mapping algorithm;

[0028] S243: Construct a Web Service framework by using CXF technology, create a Web Service for each data source, and register the Web Service with a service registry center through Web Service Description Language (WSDL); send a search request to the service registry center, select a suitable data source, and obtain corresponding data from the data source through Simple Object Access Protocol (SOAP) protocol;

[0029] S244: Fuse CXF technology with a lightweight open source framework Spring framework to form a data integration micro-service group.

[0030] Optionally, in step S25, the micro-service core module performs data redistribution according to the fused data, including:

[0031] S251: Receive a query request issued by a user through a query generator; and extract the received user information and transmit the user information to an ontology manager;

[0032] S252: Decompose query data of a global ontology that meets a requirement into query data of each local ontology through a query decomposer;

[0033] S253: Find a location of the corresponding local ontology in a service registry center, and issue a query request to different data sources;

[0034] S254: According to the query request, convert query data of each local ontology into query data of a data source through a query decomposer, and obtain the query data;

[0035] S255: Centralize the query data of different data sources to a result processor, obtain a data structure suitable for application through processing of the result processor, and perform data redistribution on the data structure suitable for application.

[0036] Optionally, step S2 further includes:

[0037] Before the application-adapted data structure is redistributed to the application, the user operation legitimacy is verified; the verification includes: verifying the validity of the user's query, and verifying the user's authority.

[0038] In an aspect, a multi-source heterogeneous data interaction and fusion system for a shield scenario is provided, the system is applied to an electronic device, and the system comprises:

[0039] A data integration subsystem integrates multi-source heterogeneous data sources from different systems by using Web service technology, establishes databases in different construction scenarios, and forms a multi-source database.

[0040] A data transfer subsystem is configured to preset a micro-service granularity and transfer data of the multi-source database through a plurality of micro-service components.

[0041] A data interaction subsystem is configured to perform unified semantic query calling on the output data after the transfer through a unified data interaction service gateway, and complete multi-source heterogeneous data interaction and fusion.

[0042] In an aspect, an electronic device is provided, and the electronic device comprises a processor and a memory, the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned multi-source heterogeneous data interaction and fusion method for a shield scenario.

[0043] In an aspect, a computer-readable storage medium is provided, and the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned multi-source heterogeneous data interaction and fusion method for a shield scenario.

[0044] The above technical solution of the embodiment of the present application has at least the following beneficial effects:

[0045] In the above solution, the present application provides a data interaction and fusion method, based on the micro-service architecture idea, the whole system is divided into a plurality of micro-services, the micro-service can be implemented by using a suitable programming language and technology inside, and meanwhile, the micro-services communicate with each other through a unified Rest API interface, the expansibility of the system is improved, and the difficulty of operation and maintenance is reduced; the core data integration micro-service integrates multi-source heterogeneous data sources by using Web service and ontology technology, and the semantic integration degree and data interaction of the data are effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to make the technical solutions in the embodiments of the present application clearer, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0047] Figure 1 is a flow chart of a multi-source heterogeneous data interaction and fusion method for a shield scenario provided by an embodiment of the present application.

[0048] Figure 2 is a flow chart of a multi-source heterogeneous data interaction and fusion method for a shield scenario provided by an embodiment of the present application.

[0049] Figure 3 is a system block diagram of a multi-source heterogeneous data interaction and fusion system for a shield scenario provided by an embodiment of the present application.

[0050] Figure 4 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the technical problems, technical solutions and advantages of the present application clearer, the following will be described in detail in combination with the drawings and specific embodiments.

[0052] An embodiment of the present application provides a multi-source heterogeneous data interaction and fusion method for a shield scenario, which can be realized by an electronic device, and the electronic device can be a terminal or a server. Figure 1 As shown in the flow chart of the multi-source heterogeneous data interaction and fusion method for a shield scenario, the method comprises the following steps.

[0053] S101: multi-source heterogeneous data sources from different systems are integrated by using a network service Web service technology, databases under different construction scenarios are established, and a multi-source database is formed;

[0054] S102: a micro-service granularity is preset, and data in the multi-source database is transferred by using a plurality of micro-service components;

[0055] S103: unified semantic query calling is performed on the output data after the transfer by using a unified data interaction service gateway, and multi-source heterogeneous data interaction and fusion are completed.

[0056] Optionally, in step S101, the multi-source heterogeneous data sources from different systems are integrated by using the network service Web service technology, the databases under different construction scenarios are established, and the multi-source database is formed, and the step comprises the following steps.

[0057] S111: unified calling and integration of multi-source heterogeneous data sources across platforms through network service Web service technology;

[0058] S112: introducing a hybrid ontology idea to automatically build local ontologies for different data sources;

[0059] S113: constructing a global ontology using domain expert knowledge, mapping the local ontology and the global ontology, and integrating data structures; on the basis of data structure integration, establishing databases under different construction scenarios to form a multi-source database.

[0060] Optionally, the multi-source database types include: a relational database management system MySQL, an Oracle database, a DB2 database (IBM DB2 relational database management system), and a SQL Server; and the data type formats include: a comma-separated values (CSV) file format, a file transfer protocol (FTP), an Excel file, a document, and relational data.

[0061] Optionally, in step S102, a micro-service granularity is preset, and a plurality of micro-service components are used to transit data of the multi-source database, including:

[0062] S121: presetting a micro-service granularity and dividing a plurality of micro-service components; the micro-service components are registered in a service registration center;

[0063] S122: according to an application of a user data request, calling corresponding micro-service components for the user in the service registration center;

[0064] S123: the plurality of micro-service components communicate with a micro-service core module through a Rest API (REpresentational State Transfer Application Programming Interface) interface to obtain request data of the user;

[0065] S124: the micro-service core module is used to fuse the obtained request data;

[0066] S125: according to the fused data, the micro-service core module is used to perform data redistribution.

[0067] Optionally, the plurality of micro-service components include: a data visualization micro-service component, a data analysis micro-service component, and a fault prediction micro-service component.

[0068] Optionally, each micro-service component is preset with a fixed role and can be run independently.

[0069] Optionally, in step S124, the micro-service core module is used to fuse the obtained request data, including:

[0070] S1241: Obtain relational schema information of a data source through JDBC (Java Database Connectivity), and formulate a mapping rule; automatically construct a local ontology for the data source using the relational schema information through jena technology; the data source is request data of each user; the relational schema information of the data source includes table name metadata, data type metadata, and constraint relationship metadata;

[0071] S1242: Construct a global ontology using a seven-step method combined with shield machine domain knowledge; and perform mapping and fusion of the local ontology and the global ontology using an ontology mapping algorithm;

[0072] S1243: Build a Web Service framework using CXF (Apache CXF) technology, create a Web Service for each data source, and register with a service registry center through WSDL (Web Services Description Language); send a search request to the service registry center, select a suitable data source, and obtain corresponding data from the data source through a SOAP protocol (Simple Object Access Protocol);

[0073] S1244: Fuse the CXF technology with a lightweight open source framework, Spring framework, to form a data integration micro-service group.

[0074] Optionally, in step S125, the micro-service core module is used to perform data re-distribution according to the fused data, including:

[0075] S1251: Receive a query request issued by a user through a query generator; and extract the received user information and transmit it to an ontology manager;

[0076] S1252: Decompose the query data of the global ontology that meets the requirements into query data of each local ontology through a query decomposer;

[0077] S1253: Find the location of the corresponding local ontology in the service registry center, and issue a query request to different data sources;

[0078] S1254: According to the query request, the query data is obtained by querying the data source of each local ontology through the query resolver.

[0079] S1255: The query data of different data sources is centralized to the result processor, and the data structure adapted to the application is obtained through the processing of the result processor, and the data structure adapted to the application is redistributed.

[0080] Optionally, step S102 further comprises:

[0081] Before the application-adapted data structure is redistributed to the application, the user operation legality is verified, and the verification comprises: validity verification of the user query, and verification of the user authority.

[0082] In the embodiment of the application, a data interaction and fusion method is provided, based on the micro-service architecture idea, the whole system is divided into multiple micro-services, the micro-services can be implemented by using appropriate programming languages and technologies, and the micro-services communicate with each other through a unified Rest API interface, so that the expansibility of the system is improved, and the difficulty of operation and maintenance is reduced; the core data integration micro-service integrates multiple source heterogeneous data sources by using Web service and ontology technology, so that the semantic integration degree and data interaction of the data are effectively improved.

[0083] The embodiment of the application provides a multi-source heterogeneous data interaction and fusion method for a shield scene, which can be realized by an electronic device, which can be a terminal or a server. Figure 2 As shown in the flowchart of the multi-source heterogeneous data interaction and fusion method for the shield scene, the method comprises the following steps:

[0084] S201: Unified calling and integration of multiple source heterogeneous data sources are performed through network service Web service technology across platforms.

[0085] In a feasible implementation manner, the multiple source heterogeneous data sources from different systems are integrated by using Web service technology. First, the Web service technology is used to take advantage of cross-platform, to eliminate the inconsistency of communication protocols and interfaces between different systems, and to realize unified calling and integration of multiple source heterogeneous data sources.

[0086] S202: A hybrid ontology idea is introduced to automatically construct a local ontology for different data sources.

[0087] S203: A global ontology is constructed by using domain expert knowledge, mapping between the local ontology and the global ontology is performed, data structure integration is performed, and on the basis of the data structure integration, a database under different construction scenes is established, and a multi-source database is formed.

[0088] In an embodiment, a hybrid ontology is introduced, local ontologies are automatically constructed for different data sources, a global ontology is constructed using domain expert knowledge, and mapping between the local ontologies and the global ontology is implemented, thereby improving semantic integration of data on the basis of data structure integration.

[0089] In an embodiment, the multi-source database types include a relational database management system MySQL, an Oracle database, a DB2 database, and a SQL Server; and the data type formats include a comma-separated values (CSV) file format, a file transfer protocol (FTP), an EXCEL file, a document, and relational data.

[0090] S204: A microservice granularity is preset, and a plurality of microservice components are divided; the microservice components are registered in a service registration center.

[0091] In an embodiment, the microservice architecture is taken as a core, and the data transfer part is composed of a plurality of different microservice components, which are registered in a service registration center.

[0092] In an embodiment, the plurality of microservice components include a data visualization microservice component, a data analysis microservice component, and a fault prediction microservice component.

[0093] S205: According to an application requested by user data, a corresponding microservice component is called for the user in the service registration center.

[0094] S206: The plurality of microservice components communicate with a microservice core module through a Rest API interface to obtain request data of the user.

[0095] In an embodiment, each microservice component is preset with a fixed function and can be independently run.

[0096] S207: The microservice core module is used to fuse the obtained request data.

[0097] In an embodiment, the whole core microservice component can fuse and redistribute data. The specific steps include:

[0098] S271: Obtain relational schema information of a data source through JDBC, formulate a mapping rule, and automatically construct a local ontology for the data source by using jena technology according to the relational schema information; the data source is requested data of each user; the relational schema information of the data source includes table name metadata, data type metadata, and constraint relationship metadata;

[0099] S272: Construct a global ontology by using a seven-step method and combining with shield machine field knowledge, and perform mapping and fusion of the local ontology and the global ontology by using an ontology mapping algorithm.

[0100] S273: Construct a Web Service framework by using CXF technology, create a Web Service for each data source, and register the Web Service with a service registry center through WSDL; send a search request to the service registry center, select a suitable data source, and obtain corresponding data from the data source through a simple object access protocol SOAP protocol.

[0101] S274: Fuse CXF technology with a lightweight open source framework Spring framework to form a data integration micro-service group.

[0102] After the data is fused, the following steps are further included:

[0103] S208: Redistribute data by using the micro-service core module according to the fused data.

[0104] Optionally, in step S208, the data is redistributed by using the micro-service core module according to the fused data, including:

[0105] S281: Receive a query request sent by a user through a query generator; and extract the received user information and transmit the user information to an ontology manager.

[0106] S282: Decompose query data of a global ontology that meets a requirement into query data of each local ontology by using a query decomposer.

[0107] S283: Find a location of the local ontology in a service registry center, and send a query request to different data sources.

[0108] S284: According to the query request, convert query data of each local ontology into query data of a data source by using the query decomposer, and obtain the query data.

[0109] S285: Centralize the query data of different data sources to a result processor, obtain a data structure adapted to an application by processing of the result processor, and redistribute the data structure adapted to the application.

[0110] In the embodiment of the application, the ontology technology is introduced for the fusion and redistribution process of the data of the micro-service core module, thereby improving the semantic integration of the system; the local ontology of each data source is constructed by using the automatic construction method, thereby eliminating the manual operation and greatly reducing the workload; and the fusion of the local ontology and the global ontology is realized by using the ontology mapping algorithm.

[0111] In an available implementation, the user operation legality is verified before the data structure adapted to the application is redistributed to the application; the verification includes: validity verification of the user query and verification of the user authority.

[0112] In the embodiment of the application, the verification of the user operation legality improves the security of the system. On the one hand, the validity of the user query is verified to ensure that the user can query the valid data; on the other hand, the user authority is verified, the user includes an administrator and a staff, the authority of the administrator is higher than that of the staff, and the administrator can perform the functions of adding, deleting, modifying and inquiring the data source; and the staff can only perform the query operation on the data. Meanwhile, the operation of other illegal users is shielded.

[0113] S209: The unified semantic query calling is performed on the output data after the transfer by using the unified data interaction service gateway, and the multi-source heterogeneous data interaction and fusion are completed.

[0114] In an available implementation, a unified semantic query interface is provided for the user. After the user inputs the corresponding semantic query, the legality of the semantic query is verified first; then, the system further verifies the user operation authority by using the data interaction security module; and finally, once the verification of both is passed, the data request operation is legal, and then the core step is continued.

[0115] In the embodiment of the application, a data interaction and fusion method is provided. The ontology technology is introduced on the basis of the Webservice technology, thereby improving the semantic integration degree of the data; in the thought of the micro-service, the appropriate programming language and technology can be used in the micro-service, the system is divided into micro-service components, each component is an independent application service, although each application service can be implemented by using different technologies, but each component has a unified communication interface for mutual access, thereby improving the expansibility of the system and reducing the difficulty of operation and maintenance; the system can also communicate with the upper service gateway to provide the data interaction for the user. The service calling process is simplified; the user can obtain the integrated information of the entity from each system database through the service gateway, thereby effectively improving the semantic integration degree of the data and the data interaction.

[0116] Figure 3 Fig. 3 is a multi-source heterogeneous data interaction and fusion system for a shield scenario according to an example embodiment.

[0117] The data integration subsystem 310 is configured to integrate multi-source heterogeneous data sources from different systems by using a Web service technology, establish databases in different construction scenarios, and form a multi-source database.

[0118] The data transfer subsystem 320 is configured to preset a micro-service granularity and transfer data of the multi-source database by using a plurality of micro-service components.

[0119] The data interaction subsystem 330 is configured to perform unified semantic query and calling on the output data after the transfer by using a unified data interaction service gateway, and complete multi-source heterogeneous data interaction and fusion.

[0120] Optionally, the data integration subsystem 310 is further configured to:

[0121] perform unified calling and integration of multi-source heterogeneous data sources across platforms by using a Web service technology.

[0122] introduce a hybrid ontology idea to automatically construct a local ontology for different data sources.

[0123] construct a global ontology by using domain expert knowledge, perform mapping between the local ontology and the global ontology, and perform data structure integration; on the basis of the data structure integration, establish databases in different construction scenarios, and form a multi-source database.

[0124] Optionally, the multi-source database types include a MySQL, an Oracle, a DB2, and a SQL Server.

[0125] The data types include a CSV, a FTP, an EXCEL, a document, and relational data.

[0126] Optionally, the data transfer subsystem 320 is further configured to preset a micro-service granularity and divide a plurality of micro-service components 321; the micro-service components 321 are registered in a service registration center.

[0127] According to an application of a user data request, the service registration center is used to call corresponding micro-service components 321 for the user.

[0128] The plurality of micro-service components 321 communicate with a micro-service core module 322 through a Rest API interface to obtain request data of the user.

[0129] The micro-service core module 322 is used to fuse the obtained request data.

[0130] According to the fused data, the micro-service core module 322 performs data redistribution.

[0131] Optionally, the plurality of micro-service components comprise a data visualization micro-service component 3211, a data analysis micro-service component 3212, and a fault prediction micro-service component 3213.

[0132] Optionally, each micro-service component is preset with a fixed function and can be independently operated.

[0133] Optionally, the data relay subsystem 320 is further configured to obtain relational schema information of a data source by Java Database Connectivity (JDBC) and formulate a mapping rule; and automatically construct a local ontology for the data source by using the relational schema information through jena technology, wherein the data source is request data of each user, and the relational schema information of the data source comprises table name metadata, data type metadata, and constraint relationship metadata.

[0134] The global ontology is constructed by using a seven-step method and related domain knowledge, and the mapping and fusion of the local ontology and the global ontology are performed by using an ontology mapping algorithm.

[0135] A Web Service framework is constructed by using CXF technology, a Web Service is created for each data source, and the Web Service is registered in a service registry center through Web Service Description Language (WSDL); a search request is sent to the service registry center, a suitable data source is selected, and corresponding data is obtained from the data source through Simple Object Access Protocol (SOAP) protocol.

[0136] The CXF technology is fused with a lightweight open source framework, Spring framework, to form a data integration micro-service group.

[0137] Optionally, the data relay subsystem 320 is further configured to receive a query request sent by a user through a query generator; and extract and transmit user information received to an ontology manager.

[0138] The query data of the global ontology that meets the requirements is decomposed into query data of each local ontology by a query decomposer.

[0139] The location of the corresponding local ontology is searched in the service registry center, and a query request is sent to different data sources.

[0140] According to the query request, the query data source of each local ontology is converted to obtain query data by the query decomposer.

[0141] The query data of different data sources is centralized to a result processor, the data structure adapted to the application is obtained through processing of the result processor, and the data structure adapted to the application is subjected to data redistribution.

[0142] Optionally, the data relay subsystem 320 further comprises a data interaction security module 323, configured to verify the legality of user operation before the application-adapted data structure is redistributed to the application; the verification comprises verifying the validity of user query and verifying the authority of the user.

[0143] In the embodiment of the application, the ontology technology is introduced on the basis of the Webservice technology, thereby improving the semantic integration degree of data; in the thought of micro service, a suitable programming language and technology can be used in the micro service to divide the system into micro service components, each component is an independent application service, although each application service can be implemented by different technologies, but each component has a unified communication interface to access each other, thereby improving the expansibility of the system and reducing the difficulty of operation and maintenance; the micro service can also communicate with the service gateway in the upper layer to provide data interaction for users; the service calling process is simplified; the users can obtain integrated information of entities from each system database through the service gateway, thereby effectively improving the semantic integration degree of data and data interaction.

[0144] Figure 4 is a structural schematic diagram of an electronic device 400 provided by the embodiment of the application, the electronic device 400 can have great difference due to different configurations or performances, and can comprise one or more than one processor (central processing unit, CPU) 401 and one or more than one memory 402, wherein the memory 402 stores at least one instruction, the at least one instruction is loaded and executed by the processor 401 to realize the steps of the following multi-source heterogeneous data interaction and fusion method for a shield scene:

[0145] S1: the multi-source heterogeneous data sources from different systems are integrated by using the network service Web service technology, databases in different construction scenes are established, and a multi-source database is formed;

[0146] S2: presetting the micro service granularity, the data of the multi-source database is relayed by using a plurality of micro service components;

[0147] S3: the output data after the relay is uniformly queried and called by using a unified data interaction service gateway, and the multi-source heterogeneous data interaction and fusion are completed.

[0148] In the example embodiment, a computer readable storage medium, such as a memory including instructions executable by a processor in a terminal to perform the above-mentioned method of multi-source heterogeneous data interaction and fusion for a shield scene is also provided. For example, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0149] Those of ordinary skill in the present art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, such as a read-only memory, a magnetic disk or an optical disk.

[0150] The above description is merely preferred embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for multi-source heterogeneous data interaction and fusion for a shield scenario, characterized in that, The method is applied to a multi-source heterogeneous data interaction and fusion system, and comprises the following steps: S1: integrating multi-source heterogeneous data sources from different systems by using a network service Web service technology, establishing databases under different construction scenes, and forming a multi-source database; S2: presetting a micro-service granularity, and performing data transfer on data of the multi-source database by using multiple micro-service components; In the step S2, the preset micro-service granularity is used to perform data transfer on data of the multi-source database by using multiple micro-service components, comprising: S21: presetting a micro-service granularity, and dividing multiple micro-service components; the micro-service components are registered in a service registration center; S22: according to an application of a user data request, calling corresponding micro-service components for the user in the service registration center; S23: the multiple micro-service components communicate with a micro-service core module through a Rest API interface to obtain request data of the user; S24: the micro-service core module is used to fuse the obtained request data; S25: according to the fused data, the micro-service core module is used to perform data redistribution; In the step S24, the micro-service core module is used to fuse the obtained request data, comprising: S241: obtaining relational schema information of a data source through a Java database connection (JDBC), formulating a mapping rule, automatically constructing a local ontology for the data source by using a jena technology, wherein the data source is request data of each user, and the relational schema information of the data source comprises table name metadata, data type metadata, and constraint relationship metadata; S242: constructing a global ontology by using a seven-step method and combining with field knowledge of a shield tunneling machine; and mapping and fusing the local ontology and the global ontology by using an ontology mapping algorithm; S243: constructing a Web Service framework by using a CXF technology, creating a Web Service for each data source, registering the Web Service in a service registration center through a Web Service Description Language (WSDL), sending a search request to the service registration center, selecting a suitable data source, and obtaining corresponding data from the data source through a Simple Object Access Protocol (SOAP) protocol; S244: fusing the Web Service framework constructed by using the CXF technology and a lightweight open source framework (Spring framework) to form a data integration micro-service group; S3: establishing a unified data interaction service gateway, performing unified semantic query calling on output data after the transfer, and completing multi-source heterogeneous data interaction and fusion.

2. The method of claim 1, wherein, In the step S1, the network service Web service technology is used to integrate multi-source heterogeneous data sources from different systems, establish databases under different construction scenes, and form a multi-source database, comprising: S11: performing unified calling and integration of multi-source heterogeneous data sources across platforms by using a network service Web service technology; S12: introducing a hybrid ontology idea to automatically construct a local ontology for different data sources; S13: global ontology is constructed by using domain expert knowledge, mapping between local ontology and global ontology is performed, and data structure integration is performed; on the basis of data structure integration, databases under different construction scenarios are established, and a multi-source database is formed.

3. The method of claim 2, wherein, The multi-source database types include: a relational database management system MySQL, an Oracle database, a DB2 database, and a SQL Server database; and data type formats include: a comma-separated values file format CSV, a file transfer protocol FTP, an Excel file, a document, and relational data.

4. The method of claim 3, wherein, The plurality of micro-service components include a data visualization micro-service component, a data analysis micro-service component, and a fault prediction micro-service component.

5. The method of claim 4, wherein, Each micro-service component is preset with a fixed function and can be independently run.

6. The method of claim 5, wherein, In the step S25, according to the fused data, data is redistributed by the micro-service core module, including: S251: receiving a query request issued by a user through a query generator; and extracting the received user information and transmitting the extracted user information to an ontology manager; S252: decomposing, by a query decomposer, the queried data of the global ontology that meets the requirements into query data of each local ontology; S253: finding the location of the corresponding local ontology in a service registry center, and issuing a query request to different data sources; S254: according to the query request, the query decomposer is used to query and convert the data sources of each local ontology, and query data is obtained; S255: the query data of different data sources is centralized to a result processor, and the result processor is used to process the query data to obtain a data structure suitable for application, and the data structure suitable for application is used for data redistribution.

7. The method of claim 6, wherein, The step S2 further includes: Before the data structure suitable for application is redistributed to the application, the legality of the user operation is verified; the verification includes validity verification of the user query and verification of the user rights.

8. A multi-source heterogeneous data interaction and fusion system for a shield scenario, characterized in that, The system includes: A data integration subsystem that integrates multi-source heterogeneous data sources from different systems by using Web service technology, establishes databases under different construction scenarios, and forms a multi-source database; A data relay subsystem that is used to preset micro-service granularity, and relays data of the multi-source database by using a plurality of micro-service components; The data relay subsystem is also used to preset micro-service granularity, and divide the plurality of micro-service components; the micro-service components are registered in a service registry center; According to the application of the user data request, the corresponding micro-service components are called for the user in the service registry center; The plurality of micro-service components communicate with the micro-service core module through a Rest API interface to obtain the request data of the user; The micro-service core module is used to fuse the obtained request data; According to the fused data, the micro-service core module is used to redistribute the data; and The data relay subsystem is also used for obtaining relational schema information of a data source through a Java database connection (JDBC) and formulating a mapping rule, and automatically constructing a local ontology for the data source through jena technology by using the relational schema information; the data source is request data of each user; the relational schema information of the data source includes table name metadata, data type metadata and constraint relationship metadata; A global ontology is constructed by using a seven-step method and combining relevant domain knowledge; and a local ontology and the global ontology are mapped and fused by using an ontology mapping algorithm; A Web Service framework is constructed by using CXF technology, a Web Service is created for each data source, and the Web Service is registered in a service registry center through a Web Service Description Language (WSDL); a search request is sent to the service registry center, a suitable data source is selected, and corresponding data is obtained from the data source through a Simple Object Access Protocol (SOAP) protocol; The CXF technology is fused with a lightweight open source framework (Spring framework) to form a data integration micro-service group; The data interaction subsystem is used for uniformly querying and calling the output data after the relay through a unified data interaction service gateway, and completing multi-source heterogeneous data interaction and fusion.

Images