Data integration system and method

Abstract

The invention provides a data integration system and method, and the system comprises a data collection unit, a data verification unit, a data cleaning unit, and a data integration display unit. The data acquisition unit is used for acquiring each piece of initial data, and the initial data comprises instance data, associated data of each piece of instance data and hardware equipment data; the data verification unit is used for verifying the initial data on the basis of verification rules and determining abnormal data failed in verification and normal data successful in verification; the data cleaning unit is used for correcting and supplementing instance attributes of the abnormal data to obtain optimized data; and the data integration and display unit is used for integrating the optimized data and the normal data to obtain target data, and displaying the target data to a target user. By adopting the scheme, the manpower cost consumed for managing the data can be reduced, and meanwhile, the accuracy, practicability and effective utilization rate of the data are improved.

Description

Technical Field

[0001] The present invention relates to the field of data management, and in particular to a data integration system and method. Background Art

[0002] At present, in the field of data management, CMDB (Configuration Management Database) is an important data management platform that bears the responsibility of storing, maintaining and managing various configuration information. However, it is found in the research that the existing CMDB platform mainly relies on manual data maintenance, which has a series of obvious problems and shortcomings.

[0003] First, due to the intervention of manual maintenance, data omissions and errors often occur. Manual operations are difficult to avoid negligence and errors, which directly leads to a decrease in data accuracy. In practical applications, such inaccurate data may cause a series of problems, such as decision-making errors and system failures. Secondly, the effectiveness of data entry is limited by manual execution. The speed of manual data processing is relatively slow and is affected by personnel capabilities and work status. This means that the update of data may not reflect the actual situation in a timely manner, thereby reducing the practical value of the data. Especially in a market environment that requires rapid response, this lag may become a competitive disadvantage for enterprises. Finally, the existing CMDB platform has obvious deficiencies in the correlation of multi-platform data. Due to the lack of effective data integration and association mechanisms, it is difficult to form a unified view of multi-platform data. This makes it impossible for users to display the correlation information of data at all levels from a panoramic perspective, resulting in data dispersion, which not only increases the complexity of data management, but also reduces the effective utilization of data.

[0004] In summary, the existing CMDB platform has obvious problems and shortcomings in data maintenance, effectiveness and multi-platform data correlation, which will not only increase the manpower cost required for data management, but also reduce the accuracy, practicality and effective utilization of data. Summary of the invention

[0005] In view of this, an object of the present invention is to provide a data integration system and method to reduce the manpower cost required for data management, while improving the accuracy, practicality and effective utilization of data.

[0006] In a first aspect, an embodiment of the present application provides a data integration system, the system comprising a data acquisition unit, a data verification unit, a data cleaning unit and a data integration display unit;

[0007] The data collection unit is used to collect various initial data, wherein the initial data includes instance data, associated data of each instance data and hardware device data;

[0008] The data verification unit is used to verify each initial data based on the verification rule, and determine the abnormal data that failed the verification and the normal data that succeeded the verification;

[0009] The data cleaning unit is used to correct and supplement the instance attributes of each abnormal data to obtain optimized data;

[0010] The data integration and display unit is used to integrate the optimized data and the normal data to obtain target data, and display the target data to the target user.

[0011] Optionally, the system further comprises a data output unit;

[0012] The data output unit is used to share and output the target data.

[0013] Optionally, the data output unit includes an external data providing module and a daily data push module;

[0014] The external data providing module is used to send the target data to a third-party system to achieve data sharing;

[0015] The daily data push module is used to periodically count the target data and push the counted target data to the user to achieve data output.

[0016] Optionally, the data collection unit includes an automatic data collection module and a manual data management module;

[0017] The automatic data collection module is used to automatically collect each instance data and the associated data of each instance data based on a preset interface or an internal agent of the instance;

[0018] The manual data management module is used to manually collect data of each hardware device based on the data entered by the target user.

[0019] Optionally, the automatic data collection module further includes an instance data collection submodule, an instance relationship collection submodule, and an instance internal resource and configuration situation collection submodule;

[0020] The instance data acquisition submodule is used to connect with each target platform or target system through the preset interface to obtain each instance data;

[0021] The instance-to-instance relationship collection submodule is used to collect dependency, communication and association relationship data of each instance data and association information data between the product and each instance data from the cloud platform interface;

[0022] The instance internal resource and configuration status collection submodule is used to collect the configuration information and resource usage information data inside the virtual machine by using the instance internal agent.

[0023] Optionally, the data verification unit further includes a process management module and a configuration checking module;

[0024] The process management module is used to check whether the format of each hardware device data is correct, determine the hardware device data with incorrect format as abnormal data, and determine the hardware device data with correct format as normal data;

[0025] The configuration checking module is used to check whether each instance data and the associated data of each instance data are missing, conflicting or repeated, determine the missing, conflicting or repeated instance data and the associated data as abnormal data, and determine the instance data and the associated data that are not missing, conflicting or repeated as normal data.

[0026] Optionally, the configuration checking module is further used to:

[0027] Automatically send email alerts when instance data or associated data is missing, conflicting, or duplicated.

[0028] Optionally, the data cleaning unit includes an instance attribute correction module and an instance attribute supplementation module;

[0029] The instance attribute correction module is used to determine a first actual attribute value of the instance attribute of each abnormal data based on the instance attribute table, and update the instance attribute of the abnormal data using the first actual attribute value;

[0030] The instance attribute supplement module is used to determine the extended attribute of each abnormal data based on the extended attribute table, determine the second actual attribute value of the extended attribute of the abnormal data based on the instance attribute table, and update the extended attribute of the abnormal data using the second actual attribute value.

[0031] Optionally, the data integration and display unit includes a personal resource integration module, a team data integration module, a product data integration module, a resource type integration module, a cloud resource pool data integration module and a data display module;

[0032] The personal resource integration module is used to integrate various optimization data and normal data related to the target user into the target data;

[0033] The team data integration module is used to integrate various optimization data and normal data related to the team to which the target user belongs into the target data;

[0034] The product data integration module is used to integrate the optimized data and normal data corresponding to each instance data belonging to the same product into the target data;

[0035] The resource type integration module is used to integrate the optimized data and normal data corresponding to each instance data belonging to the same resource type into the target data;

[0036] The cloud resource pool data integration module is used to integrate the optimized data and normal data corresponding to each instance data collected from the same cloud platform interface into the target data.

[0037] The technical solution provided by this application includes but is not limited to the following beneficial effects:

[0038] In the data integration system provided by the present application, the data acquisition unit can automatically connect to multiple source platforms and collect data. This feature greatly improves the efficiency and breadth of data collection. At the same time, for hardware equipment data that cannot be automatically collected, the data acquisition unit can also manually enter instance data to ensure the comprehensiveness and integrity of the data. The data verification unit uses advanced process management technology to ensure the accuracy of manually entered data and effectively avoid human errors. The configuration inspection module regularly checks the data and intelligently identifies faulty, missing, and conflicting data according to preset rules, further improving the accuracy and reliability of the data. The data cleaning unit corrects the erroneous attributes according to the existing attributes, supplements the missing attributes, and calculates the new height attributes. This series of operations significantly improves the data quality. Through the data integration display unit, the instance data can be comprehensively and meticulously displayed from multiple dimensions such as individuals, teams, products, resource types, and resource platforms, providing users with a richer and more three-dimensional data view. And through the data output unit, the data interface is provided to the outside, so that the system can be seamlessly connected with other systems or platforms, realizing the extensive sharing and application of data. The function of pushing data daily enables users to grasp the data dynamics in a timely manner and provides strong data support for decision-making.

[0039] By adopting the above solution, through refined module design and efficient data processing flow, comprehensive collection, cleaning, integration and management of various resource information can be achieved, which can effectively reduce the manpower cost required for data management, while improving the accuracy, practicality and effective utilization of data.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A schematic diagram of the structure of a data integration system provided by the first embodiment of the present invention is shown;

[0043] Figure 2 A schematic diagram showing the structure of a second data integration system provided by the first embodiment of the present invention is shown;

[0044] Figure 3 A schematic diagram showing the structure of a third data integration system provided by the first embodiment of the present invention is shown;

[0045] Figure 4 A schematic diagram showing the structure of a fourth data integration system provided by the first embodiment of the present invention is shown;

[0046] Figure 5 A schematic diagram showing the structure of a fifth data integration system provided by the first embodiment of the present invention is shown;

[0047] Figure 6 A schematic diagram showing the structure of a sixth data integration system provided by the first embodiment of the present invention is shown;

[0048] Figure 7 A schematic diagram showing the structure of a seventh data integration system provided by the first embodiment of the present invention is shown;

[0049] Figure 8 A schematic diagram showing the structure of an eighth data integration system provided by the first embodiment of the present invention is shown;

[0050] Fig. 9 A schematic diagram of the architecture of a panoramic information platform provided by the first embodiment of the present invention is shown;

[0051] Fig.10 A flow chart of a data integration method provided by the second embodiment of the present invention is shown. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0053] Embodiment 1

[0054] To facilitate understanding of this application, Figure 1 The content described in the structural schematic diagram of a data integration system provided by the first embodiment of the present invention is described in detail for the first embodiment of the present application.

[0055] See also Figure 1 As shown, Figure 1 The structure diagram of a data integration system provided by the first embodiment of the present invention is shown, wherein the system includes a data acquisition unit 101, a data verification unit 102, a data cleaning unit 103 and a data integration display unit 104;

[0056] The data collection unit is used to collect various initial data, wherein the initial data includes instance data, associated data of each instance data and hardware device data.

[0057] Specifically, the initial data includes instance data, associated data of each instance data, and hardware device data. Instance data is the host IP, including the virtual machine, storage, network, database, and product operating system instance IP. The associated data of each instance data includes the dependency, communication, and association relationship data between instances, as well as the association information between the product and each instance. Hardware device data is the out-of-band IP of the hardware device.

[0058] The data verification unit is used to verify each initial data based on the verification rule, and determine the abnormal data that failed the verification and the normal data that succeeded the verification.

[0059] Specifically, the verification content includes but is not limited to whether it meets the format requirements, is missing, conflicting or repeated. Initial data that does not meet the format requirements, is missing, conflicting or repeated is determined as abnormal data that fails the verification.

[0060] The data cleaning unit is used to correct and supplement the instance attributes of each abnormal data to obtain optimized data.

[0061] Specifically, the known instance attributes of each abnormal data (including but not limited to the region to which the IP belongs, the branch company to which it belongs, and the member unit) are corrected and expanded according to the custom rules. The instance attributes are part of the instance data. After the instance attributes are corrected and expanded, the instance attributes are updated.

[0062] The data integration and display unit is used to integrate the optimized data and the normal data to obtain target data, and display the target data to the target user.

[0063] Specifically, the target data is obtained by integrating various optimization data based on the preset integration strategy, and the integration strategy includes but is not limited to personal resource integration, team data integration, product data integration, resource type integration and cloud resource pool data integration. The target data is displayed to the target user so that the user can obtain accurate and comprehensive instance-related data in a timely manner through the system.

[0064] In an alternative embodiment, see Figure 2 As shown, Figure 2 It shows a schematic diagram of the structure of a second data integration system provided by the first embodiment of the present invention, wherein the system further includes a data output unit 201;

[0065] The data output unit is used to share and output the target data.

[0066] Specifically, the data output unit provides a data interface to the outside, so that the system provided in this application can be seamlessly connected with other systems or platforms, thereby realizing the wide sharing and application of data.

[0067] In an alternative embodiment, see Figure 3 As shown, Figure 3 The structure diagram of the third data integration system provided by the first embodiment of the present invention is shown, wherein the data output unit 201 includes an external data providing module 2011 and a daily data push module 2012;

[0068] The external data providing module is used to send the target data to a third-party system to achieve data sharing.

[0069] Specifically, a data interface is provided for external systems or applications to obtain platform data, thereby realizing data sharing and reuse.

[0070] The daily data push module is used to periodically count the target data and push the counted target data to the user to achieve data output.

[0071] Specifically, resource information is statistically generated daily and pushed regularly via email to keep users informed of data changes in a timely manner.

[0072] In an alternative embodiment, see Figure 4 As shown, Figure 4 The structure diagram of the fourth data integration system provided by the first embodiment of the present invention is shown, wherein the data acquisition unit 101 includes an automatic data acquisition module 1011 and a manual data management module 1012;

[0073] The automatic data collection module is used to automatically collect each instance data and the associated data of each instance data based on a preset interface or an internal agent of the instance.

[0074] Specifically, each instance data and the associated data of each instance data are automatically collected through the automatic collection module. By automatically connecting to multiple source platforms and collecting data, this feature greatly improves the efficiency and breadth of data collection.

[0075] The manual data management module is used to manually collect data of each hardware device based on the data entered by the target user.

[0076] Specifically, for hardware equipment data that cannot be collected automatically, manual entry and management functions are provided to ensure the integrity and accuracy of the data.

[0077] In an alternative embodiment, see Figure 5 As shown, Figure 5 The structure diagram of the fifth data integration system provided by the first embodiment of the present invention is shown, wherein the automatic data collection module 1011 further includes an instance data collection submodule 10111, an instance relationship collection submodule 10112 and an instance internal resource and configuration situation collection submodule 10113;

[0078] The instance data acquisition submodule is used to connect with each target platform or target system through the preset interface to obtain each instance data.

[0079] Specifically, it connects with HCS cloud platform, FeiTian cloud platform, VM cluster, database cluster and product management platform through preset interfaces, and uses Java programs to realize the automatic collection of instance data such as virtual machines, storage, network, database and products.

[0080] The instance-to-instance relationship collection submodule is used to collect dependency, communication and association relationship data of each instance data and association information data between the product and each instance data from the cloud platform interface.

[0081] Specifically, it automatically collects dependency, communication and association relationship data between instances from the cloud platform interface, and obtains the association information between products and instances from the product management platform, providing necessary association data support for the panoramic view.

[0082] The instance internal resource and configuration status collection submodule is used to collect the configuration information and resource usage information data inside the virtual machine by using the instance internal agent.

[0083] Specifically, use agents such as zabbix and hids installed inside the instance to automatically collect configuration information and resource usage inside the virtual machine.

[0084] In an alternative embodiment, see Figure 6 As shown, Figure 6 1 shows a schematic diagram of the structure of the sixth data integration system provided by the first embodiment of the present invention, wherein the data verification unit 102 further includes a process management module 1021 and a configuration checking module 1022;

[0085] The process management module is used to verify whether the format of each hardware device data is correct, determine the hardware device data with incorrect format as abnormal data, and determine the hardware device data with correct format as normal data.

[0086] Specifically, this module implements strict process restrictions and verifies whether the format of data of each hardware device is correct based on a pre-configured standard format. It performs subsequent data correction work for abnormal data with incorrect format, and directly stores normal data with correct format.

[0087] Ensure that the format of the entered data meets the requirements and undergoes a multi-person review process to achieve accurate management of the manually entered data interface and data format. This measure aims to ensure the accuracy and standardization of the data and lay a solid foundation for subsequent data processing and analysis.

[0088] The configuration checking module is used to check whether each instance data and the associated data of each instance data are missing, conflicting or repeated, determine the missing, conflicting or repeated instance data and the associated data as abnormal data, and determine the instance data and the associated data that are not missing, conflicting or repeated as normal data.

[0089] Specifically, this module is responsible for regularly checking data and accurately identifying data items that do not meet requirements, are missing, conflicting, or duplicated based on custom rules. It also performs subsequent data correction work on abnormal data with incorrect formats, and directly stores normal data with correct formats.

[0090] In an optional implementation, the configuration checking module is further used to:

[0091] Automatically send email alerts when instance data or associated data is missing, conflicting, or duplicated.

[0092] Specifically, once data items that do not meet requirements, are missing, conflicting, or duplicated are found, the configuration check module will automatically send an email alert and promptly notify relevant personnel to handle the issue to ensure the integrity and accuracy of the data.

[0093] In an alternative embodiment, see Figure 7 As shown, Figure 7 The structure diagram of the seventh data integration system provided by the first embodiment of the present invention is shown, wherein the data cleaning unit 103 includes an instance attribute correction module 1031 and an instance attribute supplementation module 1032;

[0094] The instance attribute correction module is used to determine a first actual attribute value of the instance attribute of each abnormal data based on the instance attribute table, and update the instance attribute of the abnormal data using the first actual attribute value.

[0095] Specifically, known attributes (IP region, branch, member unit) are processed according to custom rules, and abnormal data is corrected to ensure data accuracy and consistency. This is intended to enhance the system's data self-healing capabilities, so that when data anomalies occur, the system can self-repair, thereby effectively avoiding the impact of anomalies on data use.

[0096] Furthermore, the first actual attribute value of the instance attribute of each abnormal data is determined by querying the instance attribute table, and the first actual attribute value is used as the new instance attribute of the abnormal data to replace the original instance attribute, so as to update the instance attribute of the abnormal data.

[0097] The instance attribute supplement module is used to determine the extended attribute of each abnormal data based on the extended attribute table, determine the second actual attribute value of the extended attribute of the abnormal data based on the instance attribute table, and update the extended attribute of the abnormal data using the second actual attribute value.

[0098] Specifically, this module deeply mines and utilizes known attribute information based on a series of carefully designed custom rules. Furthermore, by querying the extended attribute table, the extended attributes of each abnormal data other than the existing instance attributes are determined, and the second actual attribute value of the extended attribute is determined by querying the instance attribute table, and the extended attribute is added as a new attribute of the abnormal data, and the second actual attribute value is added as the attribute value of the extended attribute, so as to update the extended attribute of the abnormal data. Through the above-mentioned precise calculations and logical reasoning, new attribute values ​​with actual value can be derived. This function not only enhances the richness and diversity of data, but also provides users with a more convenient and flexible data screening perspective. Users can locate the required data more quickly and accurately based on these newly generated attribute values, thereby greatly improving the convenience and efficiency of data use and reducing the difficulty and complexity of users in the use process.

[0099] In an alternative embodiment, see Figure 8 As shown, Figure 8 The structure diagram of the eighth data integration system provided by the first embodiment of the present invention is shown, wherein the data integration and display unit 104 includes a personal resource integration module 1041, a team data integration module 1042, a product data integration module 1043, a resource type integration module 1044, a cloud resource pool data integration module 1045 and a data display module 1046;

[0100] The personal resource integration module is used to integrate various optimization data and normal data related to the target user into the target data.

[0101] Specifically, the optimization data and normal data related to the target user are obtained, and the optimization data and normal data related to the target user are integrated into the target data, so as to focus on providing the individual user with a view of all resources related to the individual user. This brings great convenience to the individual user, enabling them to conveniently conduct overall resource troubleshooting and operation and maintenance work, thereby improving their personal work efficiency.

[0102] The team data integration module is used to integrate various optimization data and normal data related to the team to which the target user belongs into the target data.

[0103] Specifically, the optimization data and normal data related to the target user's team are obtained, and the optimization data and normal data related to the target user's team are integrated into the target data, so that all resources within a team can be integrated and displayed uniformly. This allows team managers to easily and comprehensively understand the occupation of team resources, and then make more global and reasonable resource allocation and optimize the use of team resources.

[0104] The product data integration module is used to integrate the optimized data and normal data corresponding to each instance data belonging to the same product into the target data.

[0105] Specifically, for different products, we obtain the optimized data and normal data corresponding to each instance data of each product, and integrate the optimized data and normal data corresponding to each instance data of each product into target data. Each product corresponds to a set of target data, which can focus on comprehensively and systematically integrating core resource information such as host, storage, and network based on product-related information, and show the resource usage of a product in detail from a global perspective, providing users with an intuitive and easy-to-understand resource view. In this way, users can not only fully control resources, but also quickly locate the resources and fault points involved when a fault occurs, thereby significantly improving fault response and processing efficiency.

[0106] The resource type integration module is used to integrate the optimized data and normal data corresponding to each instance data belonging to the same resource type into the target data.

[0107] Specifically, for different resource types (such as physical machines and virtual machines), the optimization data and normal data corresponding to each instance data belonging to each resource type are obtained, and the optimization data and normal data corresponding to each instance data belonging to each resource type are integrated into target data. Each resource type corresponds to a set of target data, and data of the same type are integrated on the same page for display according to the resource type. This enables users to more conveniently view and manage resources by resource type, further improving the efficiency and accuracy of resource management and optimizing the user experience.

[0108] The cloud resource pool data integration module is used to integrate the optimized data and normal data corresponding to each instance data collected from the same cloud platform interface into the target data.

[0109] Specifically, for different cloud platform interfaces, the optimized data and normal data corresponding to each instance data collected from each cloud platform interface are obtained, and the optimized data and normal data corresponding to each instance data collected from the same cloud platform interface are integrated into target data. Each cloud platform interface corresponds to a set of target data, and the usage of multiple different cloud resource pools is summarized and displayed uniformly. In this way, users can view the usage of all resources from a global perspective without logging into different cloud platforms separately, which greatly improves the convenience and efficiency of resource management.

[0110] Based on the data integration system provided by this application, a panoramic information platform can be constructed. Fig. 9 As shown, Fig. 9The schematic diagram of the architecture of a panoramic information platform provided by the first embodiment of the present invention is shown, wherein the panoramic information platform architecture includes a panoramic database, a data collection layer, a data verification layer, a data cleaning layer, a data integration and display layer, and a data external output layer. The data collection layer includes an automatic collection module and a manual management module. The automatic collection module is connected to each cloud platform (including a database cluster, an HCS cloud platform, a FeiTian cloud platform, and a VMware cluster), each instance internal agent (including Zabbix and HIDS: titan), and other external data source systems. The data verification layer includes a configuration check module and a process management module, and the process management module is used to manage the hardware device data input by the manual management module. The data cleaning layer includes an attribute correction module and an attribute supplement module. The data integration and display layer includes a personal resource module, a team resource module, a product resource module, a resource type module, and a cloud resource pool module. The data external output layer includes an external interface module and a data daily report module, and finally outputs the instance data to an external data system. The specific working principle and process of each architecture layer and each module in the platform can be found in the above description of each module in the system.

[0111] Embodiment 2

[0112] Embodiment 2 of the present invention provides a data integration method, see Fig.10 As shown, Fig.10 A flow chart of a data integration method provided by Embodiment 2 of the present invention is shown, wherein the method is applied to a data integration system, the system includes a data acquisition unit, a data verification unit, a data cleaning unit and a data integration display unit, and the method includes steps S1001 to S1004:

[0113] S1001: The data collection unit collects various initial data, wherein the initial data includes instance data, associated data of each instance data, and hardware device data;

[0114] S1002: The data verification unit verifies each initial data based on the verification rule, and determines abnormal data that fails the verification and normal data that succeeds the verification;

[0115] S1003: The data cleaning unit corrects and supplements the instance attributes of each abnormal data to obtain optimized data;

[0116] S1004: The data integration and display unit integrates the optimized data and the normal data to obtain target data, and displays the target data to the target user.

[0117] In an optional embodiment, the system further comprises a data output unit, and the method comprises:

[0118] The data output unit performs data sharing and data output on the target data.

[0119] In an optional implementation, the data output unit includes an external data providing module and a daily data push module, and the method includes:

[0120] The external data providing module sends the target data to a third-party system to achieve data sharing;

[0121] The daily data push module periodically counts the target data and pushes the counted target data to the user to achieve data output.

[0122] In an optional embodiment, the data collection unit includes an automatic data collection module and a manual data management module, and the method includes:

[0123] The automatic data collection module automatically collects each instance data and the associated data of each instance data based on a preset interface or an internal agent of the instance;

[0124] The manual data management module manually collects data of each hardware device based on the data entered by the target user.

[0125] In an optional embodiment, the automatic data collection module further includes an instance data collection submodule, an instance relationship collection submodule, and an instance internal resource and configuration collection submodule, and the method includes:

[0126] The instance data acquisition submodule interfaces with each target platform or target system through the preset interface to obtain each instance data;

[0127] The instance-to-instance relationship collection submodule collects dependency, communication and association relationship data of each instance data and association information data between the product and each instance data from the cloud platform interface;

[0128] The instance internal resource and configuration status collection submodule uses the instance internal agent to collect configuration information and resource usage information data inside the virtual machine.

[0129] In an optional implementation, the data verification unit further includes a process management module and a configuration checking module, and the method includes:

[0130] The process management module verifies whether the format of each hardware device data is correct, determines hardware device data with incorrect format as abnormal data, and determines hardware device data with correct format as normal data;

[0131] The configuration checking module verifies whether each instance data and its associated data are missing, conflicting or repeated, determines the missing, conflicting or repeated instance data and associated data as abnormal data, and determines the non-missing, non-conflicting and non-repeated instance data and associated data as normal data.

[0132] In an alternative embodiment, the method comprises:

[0133] The configuration checking module automatically sends an email alert when instance data or associated data is missing, conflicting or duplicated.

[0134] In an optional implementation, the data cleaning unit includes an instance attribute correction module and an instance attribute supplementation module, and the method includes:

[0135] The instance attribute correction module determines a first actual attribute value of the instance attribute of each abnormal data based on the instance attribute table, and updates the instance attribute of the abnormal data using the first actual attribute value;

[0136] The instance attribute supplement module determines the extended attribute of each abnormal data based on the extended attribute table, determines the second actual attribute value of the extended attribute of the abnormal data based on the instance attribute table, and updates the extended attribute of the abnormal data using the second actual attribute value.

[0137] In an optional implementation scheme, the data integration and display unit includes a personal resource integration module, a team data integration module, a product data integration module, a resource type integration module, a cloud resource pool data integration module and a data display module, and the method includes:

[0138] The personal resource integration module integrates various optimization data and normal data related to the target user into the target data;

[0139] The team data integration module integrates various optimization data and normal data related to the team to which the target user belongs into the target data;

[0140] The product data integration module integrates the optimized data and normal data corresponding to each instance data belonging to the same product into the target data;

[0141] The resource type integration module integrates the optimized data and normal data corresponding to each instance data belonging to the same resource type into the target data;

[0142] The cloud resource pool data integration module integrates the optimized data and normal data corresponding to each instance data collected from the same cloud platform interface into the target data.

[0143] The data integration system provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the system provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the system embodiment can refer to the corresponding contents in the aforementioned method embodiment. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0144] In the embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0147] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0148] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0149] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or perform equivalent replacements on some of the technical features thereof; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A data integration system, characterized in that: The system includes a data acquisition unit, a data verification unit, a data cleaning unit and a data integration and display unit; The data collection unit is used to collect various initial data, wherein the initial data includes instance data, associated data of each instance data and hardware device data; The data verification unit is used to verify each initial data based on the verification rule, and determine the abnormal data that failed the verification and the normal data that succeeded the verification; The data cleaning unit is used to correct and supplement the instance attributes of each abnormal data to obtain optimized data; The data integration and display unit is used to integrate the optimized data and the normal data to obtain target data, and display the target data to the target user.

2. The system according to claim 1, characterized in that The system also includes a data output unit; The data output unit is used to share and output the target data.

3. The system according to claim 2, characterized in that The data output unit includes an external data providing module and a daily data pushing module; The external data providing module is used to send the target data to a third-party system to achieve data sharing; The daily data push module is used to periodically count the target data and push the counted target data to the user to achieve data output.

4. The system according to claim 1, characterized in that The data collection unit includes an automatic data collection module and a manual data management module; The automatic data collection module is used to automatically collect each instance data and the associated data of each instance data based on a preset interface or an internal agent of the instance; The manual data management module is used to manually collect data of each hardware device based on the data entered by the target user.

5. The system according to claim 4, characterized in that The automatic data collection module also includes an instance data collection submodule, an instance relationship collection submodule, and an instance internal resource and configuration situation collection submodule; The instance data acquisition submodule is used to connect with each target platform or target system through the preset interface to obtain each instance data; The instance-to-instance relationship collection submodule is used to collect dependency, communication and association relationship data of each instance data and association information data between the product and each instance data from the cloud platform interface; The instance internal resource and configuration status collection submodule is used to collect the configuration information and resource usage information data inside the virtual machine by using the instance internal agent.

6. The system according to claim 1, characterized in that The data verification unit also includes a process management module and a configuration checking module; The process management module is used to check whether the format of each hardware device data is correct, determine the hardware device data with incorrect format as abnormal data, and determine the hardware device data with correct format as normal data; The configuration checking module is used to check whether each instance data and the associated data of each instance data are missing, conflicting or repeated, determine the missing, conflicting or repeated instance data and the associated data as abnormal data, and determine the instance data and the associated data that are not missing, conflicting or repeated as normal data.

7. The system according to claim 6, characterized in that The configuration checking module is also used for: Automatically send email alerts when instance data or associated data is missing, conflicting, or duplicated.

8. The system according to claim 1, characterized in that The data cleaning unit includes an instance attribute correction module and an instance attribute supplement module; The instance attribute correction module is used to determine a first actual attribute value of the instance attribute of each abnormal data based on the instance attribute table, and update the instance attribute of the abnormal data using the first actual attribute value; The instance attribute supplement module is used to determine the extended attribute of each abnormal data based on the extended attribute table, determine the second actual attribute value of the extended attribute of the abnormal data based on the instance attribute table, and update the extended attribute of the abnormal data using the second actual attribute value.

9. The system according to claim 1, characterized in that The data integration and display unit includes a personal resource integration module, a team data integration module, a product data integration module, a resource type integration module, a cloud resource pool data integration module and a data display module; The personal resource integration module is used to integrate various optimization data and normal data related to the target user into the target data; The team data integration module is used to integrate various optimization data and normal data related to the team to which the target user belongs into the target data; The product data integration module is used to integrate the optimized data and normal data corresponding to each instance data belonging to the same product into the target data; The resource type integration module is used to integrate the optimized data and normal data corresponding to each instance data belonging to the same resource type into the target data; The cloud resource pool data integration module is used to integrate the optimized data and normal data corresponding to each instance data collected from the same cloud platform interface into the target data.

10. A data integration method, characterized in that: Applied to a data integration system, the system includes a data acquisition unit, a data verification unit, a data cleaning unit and a data integration display unit, and the method includes: The data collection unit collects various initial data, wherein the initial data includes instance data, associated data of each instance data, and hardware device data; The data verification unit verifies each initial data based on the verification rule, and determines abnormal data that fails the verification and normal data that succeeds the verification; The data cleaning unit corrects and supplements the instance attributes of each abnormal data to obtain optimized data; The data integration and display unit integrates the optimized data and the normal data to obtain target data, and displays the target data to target users.

Images