Multi-source heterogeneous data routing and management method based on ecological environment data center

By constructing multi-source heterogeneous data configuration information and calculating server weights, the problems of data consistency and high concurrency pressure in the ecological environment are solved, data routing and induction are realized, and the stability and reliability of the system are ensured.

CN118820348BActive Publication Date: 2024-11-26SICHUAN GUOLAN ZHONGTIAN ENVIRONMENTAL TECH GRP CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411307454.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-19
Publication Date
2024-11-26
Estimated Expiration
2044-09-19

AI Technical Summary

Technical Problem

Due to the multi-source heterogeneous characteristics of ecological environment data, data consistency, synchronization and integration difficulty increases, and dynamic query server resources are unbalanced when high concurrent requests are used, and the pressure of a single server is too high, and the existing methods have failed to effectively solve these problems.

Method used

By constructing configuration information of multi-source heterogeneous data, obtaining dynamic query server list, calculating server weights and routing to the optimal server, realizing data routing and insulating management, solving the problems of data consistency and high concurrency pressure.

Benefits of technology

It realizes standardized management of different types of data sources, solves data consistency and synchronization problems, and maintains server resources balanced under high concurrency, avoids data loss and response hysteresis, and ensures business continuity and user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118820348B_ABST
    Figure CN118820348B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for routing and managing multi-source heterogeneous data based on an ecological environment data center, and relates to the field of routing and managing multi-source heterogeneous data. The method comprises the following steps: constructing configuration information of multi-source heterogeneous data; obtaining an initial list of dynamic query servers; determining a final dynamic query server to achieve routing of multi-source heterogeneous data; obtaining dynamic query results, and encapsulating the dynamic query results in a unified format based on an ecological environment data center to achieve management of multi-source heterogeneous data. The present invention can solve the problems of data consistency, data synchronization and integration brought about by multi-source heterogeneous data in an ecological environment, and can solve the problems of unbalanced use of dynamic query server list resources and excessive pressure on a single dynamic query server under high concurrent requests, maintain a high degree of stability and reliability, and avoid problems such as data loss and response delays that seriously affect business continuity and user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of multi-source heterogeneous data routing and management, and in particular to a multi-source heterogeneous data routing and management method based on an ecological environment data center. Background Art

[0002] There are numerous ecological and environmental business systems, and different business systems have different database types. The data is multi-source and heterogeneous, which greatly increases the difficulty of data access and management.

[0003] Due to the above-mentioned characteristics of ecological and environmental data, whenever the business system needs to interact with data across different types of databases, the existing methods have to be customized according to the specificity of each type of database, which increases the complexity of system development and increases the cost of data application. It also fails to effectively abstract and encapsulate the underlying database characteristics and the specific implementation logic of data queries, and lacks an effective method to manage multi-source heterogeneous data.

[0004] In addition, as the business scale continues to expand, especially during peak hours or in the case of high-concurrency requests caused by sudden system events, the dynamic query server will face severe high-concurrency pressure. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multi-source heterogeneous data routing and management method based on the ecological environment data center, which can solve the data consistency, data synchronization and integration problems brought about by multi-source heterogeneous data in the ecological environment, and can solve the problems of unbalanced use of dynamic query server list resources and excessive pressure on a single dynamic query server under high concurrent requests.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0007] The multi-source heterogeneous data routing and management method based on the ecological environment data center includes the following steps:

[0008] S1. Build configuration information for multi-source heterogeneous data;

[0009] S2, obtaining a dynamic query code request task, and obtaining an initial dynamic query server list according to the dynamic query code request task and the configuration information of the multi-source heterogeneous data in step S1;

[0010] S3, according to the initial dynamic query server list in step S2, the weights of the dynamic query servers are calculated, and the final dynamic query servers are determined according to the weights of the dynamic query servers to realize the routing of multi-source heterogeneous data;

[0011] S4. According to the configuration information of multi-source heterogeneous data in step S1, the dynamic query coding request task in step S2 and the final dynamic query server in step S3, the dynamic query results are obtained, and the dynamic query results are encapsulated in a unified format based on the ecological environment data middle platform to realize the management of multi-source heterogeneous data.

[0012] Furthermore, in step S1, the configuration information of the multi-source heterogeneous data includes data source connection information of the multi-source heterogeneous data, data source library table metadata information of the multi-source heterogeneous data, dynamic query coding of the multi-source heterogeneous data, dynamic query server of the multi-source heterogeneous data, and a mapping relationship between dynamic query coding and dynamic query server.

[0013] Furthermore, step S1 includes the following steps:

[0014] S11. Configure data source connection information for multi-source heterogeneous data;

[0015] S12, according to the data source connection information of the multi-source heterogeneous data configured in step S11, establish a data source connection for the multi-source heterogeneous data, synchronize the metadata information of the database and the metadata information of the data table in the data source, so as to configure the data source library table metadata information of the multi-source heterogeneous data;

[0016] S13, configuring dynamic query coding of multi-source heterogeneous data according to the data source connection information of the multi-source heterogeneous data configured in step S11 and the data source library table metadata information of the multi-source heterogeneous data configured in step S12;

[0017] S14. According to the dynamic query code of the multi-source heterogeneous data configured in step S13, a mapping relationship between the dynamic query code and the dynamic query server is configured.

[0018] Furthermore, in step S11, the data source connection information of the multi-source heterogeneous data includes the identification number, name, network interconnection protocol and port number of the data source.

[0019] Further, in step S13, according to the data source connection information of the multi-source heterogeneous data configured in step S11 and the data source library table metadata information of the multi-source heterogeneous data configured in step S12, the dynamic query coding of the multi-source heterogeneous data is configured. The specific process is: select the table fields in the metadata information of the data table as the condition fields, return fields and sorting fields of the dynamic query, generate a data table of dynamic query coding, and store the data source connection information and data source library table metadata information of the multi-source heterogeneous data in the data table of dynamic query coding to configure the dynamic query coding of the multi-source heterogeneous data.

[0020] Furthermore, in step S14, according to the dynamic query coding of multi-source heterogeneous data configured in step S13, a mapping relationship between the dynamic query coding and the dynamic query server is configured. The specific process is as follows: based on the identity identification number of the data source, according to the configured dynamic query coding of the multi-source heterogeneous data, the correspondence between the network interconnection protocol of the data source and the dynamic query coding is obtained; based on the request frequency of the data source and the performance of the dynamic query server, according to the correspondence between the network interconnection protocol of the data source and the dynamic query coding, a mapping relationship between the dynamic query coding and the dynamic query server is established and configured.

[0021] Further, in step S3, according to the initial dynamic query server list in step S2, the weight of the dynamic query server is calculated, including the following steps:

[0022] A1. Determine the performance index data of the dynamic query server according to the initial dynamic query server list in step S2;

[0023] A2. Determine the weight of the performance indicator of the dynamic query server;

[0024] A3. Calculate the weight of the dynamic query server according to the performance indicator data of the dynamic query server in step A1 and the weight of the performance indicator of the dynamic query server in step A2.

[0025] Furthermore, in step A1, the performance indicator data of the server dynamically queried includes the total utilization of all CPU cores, memory utilization, disk I / O occupancy, and network I / O occupancy.

[0026] Furthermore, in step A2, the weights of the performance indicators of the dynamic query server are determined, specifically: the weight of all CPU cores is determined to be 0.4, the weight of the memory is determined to be 0.3, the weight of the disk I / O is determined to be 0.2, and the weight of the network I / O is determined to be 0.1.

[0027] Further, in step A3, the weight of the dynamic query server is calculated according to the performance indicator data of the dynamic query server in step A1 and the weight of the performance indicator of the dynamic query server in step A2, which is expressed as:

[0028]

[0029] in: To dynamically query the server weight, The number of the performance indicator for the dynamic query server. To dynamically query the total number of performance indicators of the server, To dynamically query the server Performance indicator data, To dynamically query the server The weight of a performance indicator.

[0030] The present invention has the following beneficial effects:

[0031] (1) The present invention realizes standardized management of connection information of different types of data sources by constructing configuration information of multi-source heterogeneous data, configuring data source connection information of multi-source heterogeneous data, data source library table metadata information of multi-source heterogeneous data, dynamic query coding of multi-source heterogeneous data, dynamic query server of multi-source heterogeneous data, and mapping relationship between dynamic query coding and dynamic query server, thereby shielding the differences in the use of different types of databases, thereby providing a unified connection method for database connection requests of application programs, as well as a unified metadata management entry and data sharing interface, which can solve the data consistency, data synchronization and integration problems caused by the distribution of ecological environment data in different databases, storage systems or services;

[0032] (2) The present invention calculates the routing weight value of the dynamic query server through a weighted average algorithm, and selects the dynamic query server with the largest routing weight value as the server that responds to requests from multiple data sources. This can solve the problem of unbalanced use of dynamic query server list resources and excessive pressure on a single dynamic query server under high concurrent requests, thereby maintaining a high degree of stability and reliability and avoiding problems such as data loss and response delay that seriously affect business continuity and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flowchart of the multi-source heterogeneous data routing and management method based on the ecological environment data center. DETAILED DESCRIPTION

[0034] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0035] like Figure 1 As shown, the multi-source heterogeneous data routing and management method based on the ecological environment data center includes steps S1-S4, which are as follows:

[0036] S1. Build configuration information for multi-source heterogeneous data.

[0037] In an optional embodiment of the present invention, the configuration information of multi-source heterogeneous data includes data source connection information of the multi-source heterogeneous data, data source library table metadata information of the multi-source heterogeneous data, dynamic query coding of the multi-source heterogeneous data, dynamic query server of the multi-source heterogeneous data, and a mapping relationship between dynamic query coding and dynamic query server.

[0038] Specifically, the present invention constructs configuration information of multi-source heterogeneous data, and then writes the configuration information of the multi-source heterogeneous data into a cache. The cache in the present invention uses a non-relational database.

[0039] Step S1 includes the following steps:

[0040] S11. Configure data source connection information of multi-source heterogeneous data.

[0041] The data source connection information of multi-source heterogeneous data includes the data source's identification number, name, network interconnection protocol and port number.

[0042] S12. According to the data source connection information of the multi-source heterogeneous data configured in step S11, a data source connection of the multi-source heterogeneous data is established, and the metadata information of the database and the metadata information of the data table in the data source are synchronized to configure the data source library table metadata information of the multi-source heterogeneous data.

[0043] S13: According to the data source connection information of the multi-source heterogeneous data configured in step S11 and the data source library table metadata information of the multi-source heterogeneous data configured in step S12, dynamic query coding of the multi-source heterogeneous data is configured.

[0044] The present invention configures dynamic query coding of multi-source heterogeneous data according to the data source connection information of the multi-source heterogeneous data configured in step S11 and the data source library table metadata information of the multi-source heterogeneous data configured in step S12. The specific process is: selecting table fields in the metadata information of the data table as condition fields, return fields and sorting fields for dynamic query, generating a data table of dynamic query coding, and storing the data source connection information and data source library table metadata information of the multi-source heterogeneous data in the data table of dynamic query coding to configure dynamic query coding of multi-source heterogeneous data.

[0045] S14. According to the dynamic query code of the multi-source heterogeneous data configured in step S13, a mapping relationship between the dynamic query code and the dynamic query server is configured.

[0046] The present invention configures a mapping relationship between a dynamic query code and a dynamic query server according to the dynamic query code of multi-source heterogeneous data configured in step S13. The specific process is: based on the identity identification number of the data source, according to the configured dynamic query code of the multi-source heterogeneous data, the corresponding relationship between the network interconnection protocol of the data source and the dynamic query code is obtained; based on the request frequency of the data source and the performance of the dynamic query server, according to the corresponding relationship between the network interconnection protocol of the data source and the dynamic query code, a mapping relationship between the dynamic query code and the dynamic query server is established and configured.

[0047] The data table of dynamic query coding contains the data source connection information of multi-source heterogeneous data, that is, it contains the identification number of the data source. The identification number of the data source is determined according to the dynamic query code, and then the network interconnection protocol of the corresponding data source is determined according to the identification number of the data source.

[0048] S2. Obtain a dynamic query coding request task, and obtain an initial dynamic query server list according to the dynamic query coding request task and the configuration information of the multi-source heterogeneous data in step S1.

[0049] In an optional embodiment of the present invention, the present invention obtains a dynamic query code request task, and then obtains an initial dynamic query server list from a mapping relationship between dynamic query codes and dynamic query servers in configuration information of multi-source heterogeneous data.

[0050] S3. According to the initial dynamic query server list in step S2, the weights of the dynamic query servers are calculated, and the final dynamic query servers are determined according to the weights of the dynamic query servers to achieve routing of multi-source heterogeneous data.

[0051] In an optional embodiment of the present invention, the present invention calculates the weights of the dynamic query servers based on the initial dynamic query server list, selects the dynamic query server with the largest weight, determines it as the final dynamic query server, and routes the dynamic query encoding request task to the final dynamic query server, thereby realizing the routing of multi-source heterogeneous data.

[0052] The present invention calculates the weight of the dynamic query server according to the initial dynamic query server list in step S2, including the following steps:

[0053] A1. According to the initial dynamic query server list in step S2, performance indicator data of the dynamic query server is determined.

[0054] The performance indicator data of the dynamic query server includes the total utilization of all CPU cores, memory utilization, disk I / O utilization, and network I / O utilization.

[0055] A2. Determine the weight of the performance indicator of the dynamic query server.

[0056] The present invention determines the weights of the performance indicators of the dynamic query server, specifically: the weights of all CPU cores are determined to be 0.4, the weight of the memory is determined to be 0.3, the weight of the disk I / O is determined to be 0.2, and the weight of the network I / O is determined to be 0.1.

[0057] Based on the historical data of the actual performance contribution of the server, the present invention determines the weight of all CPU cores to be 0.4, the weight of the memory to be 0.3, the weight of the disk I / O to be 0.2, and the weight of the network I / O to be 0.1.

[0058] A3. Calculate the weight of the dynamic query server according to the performance indicator data of the dynamic query server in step A1 and the weight of the performance indicator of the dynamic query server in step A2.

[0059] The present invention calculates the weight of the dynamic query server according to the performance indicator data of the dynamic query server in step A1 and the weight of the performance indicator of the dynamic query server in step A2, which is expressed as:

[0060]

[0061] in: To dynamically query the server weight, The number of the performance indicator for the dynamic query server. To dynamically query the total number of performance indicators of the server, To dynamically query the server Performance indicator data, To dynamically query the server The weight of a performance indicator.

[0062] S4. According to the configuration information of multi-source heterogeneous data in step S1, the dynamic query coding request task in step S2 and the final dynamic query server in step S3, the dynamic query results are obtained, and the dynamic query results are encapsulated in a unified format based on the ecological environment data middle platform to realize the management of multi-source heterogeneous data.

[0063] In an optional embodiment of the present invention, the present invention matches the dynamic query coding request task with the configuration information of multi-source heterogeneous data, obtains the configuration information of the dynamic query coding request task, and generates an SQL (Structured Query Language) statement from the configuration information of the dynamic query coding request task, and then selects the corresponding data-driven class according to the configuration information of the dynamic query coding request task, executes the SQL statement according to the data-driven class, obtains the dynamic query result, and then encapsulates the dynamic query result in a unified format based on the ecological environment data middle platform to realize the management of multi-source heterogeneous data.

[0064] The ecological environment data center in the present invention refers to an ecological environment data management system in an enterprise or organization. Based on the ecological environment data center, the present invention encapsulates the dynamic query results in a unified format, realizes the management of multi-source heterogeneous data, and can encapsulate all multi-source heterogeneous data in the ecological environment data together to improve the efficiency of subsequent multi-source heterogeneous data queries.

[0065] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0066] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0067] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0068] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0069] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A multi-source heterogeneous data routing and management method based on an ecological environment data center, characterized in that: The following steps are involved: S1. Build configuration information for multi-source heterogeneous data; The configuration information of multi-source heterogeneous data includes data source connection information of multi-source heterogeneous data, data source library table metadata information of multi-source heterogeneous data, dynamic query coding of multi-source heterogeneous data, dynamic query server of multi-source heterogeneous data, and mapping relationship between dynamic query coding and dynamic query server; Step S1 includes the following steps: S11. Configure data source connection information for multi-source heterogeneous data; S12, according to the data source connection information of the multi-source heterogeneous data configured in step S11, establish a data source connection for the multi-source heterogeneous data, synchronize the metadata information of the database and the metadata information of the data table in the data source, so as to configure the data source library table metadata information of the multi-source heterogeneous data; S13, according to the data source connection information of the multi-source heterogeneous data configured in step S11 and the data source library table metadata information of the multi-source heterogeneous data configured in step S12, the dynamic query coding of the multi-source heterogeneous data is configured, and the specific process is: select the table field in the metadata information of the data table as the condition field, return field and sorting field of the dynamic query, generate the data table of the dynamic query coding, and store the data source connection information and the data source library table metadata information of the multi-source heterogeneous data in the data table of the dynamic query coding, so as to configure the dynamic query coding of the multi-source heterogeneous data; S14, according to the dynamic query code of the multi-source heterogeneous data configured in step S13, configure a mapping relationship between the dynamic query code and the dynamic query server; S2, obtaining a dynamic query code request task, and obtaining an initial dynamic query server list according to the dynamic query code request task and the configuration information of the multi-source heterogeneous data in step S1; S3, according to the initial dynamic query server list in step S2, the weights of the dynamic query servers are calculated, and the final dynamic query servers are determined according to the weights of the dynamic query servers to realize the routing of multi-source heterogeneous data; S4. According to the configuration information of multi-source heterogeneous data in step S1, the dynamic query coding request task in step S2 and the final dynamic query server in step S3, the dynamic query results are obtained, and the dynamic query results are encapsulated in a unified format based on the ecological environment data middle platform to realize the management of multi-source heterogeneous data.

2. The multi-source heterogeneous data routing and management method based on the ecological environment data center according to claim 1 is characterized in that: In step S11, the data source connection information of the multi-source heterogeneous data includes the identification number, name, network interconnection protocol and port number of the data source.

3. The multi-source heterogeneous data routing and management method based on the ecological environment data center according to claim 1 is characterized in that: In step S14, according to the dynamic query coding of multi-source heterogeneous data configured in step S13, the mapping relationship between the dynamic query coding and the dynamic query server is configured. The specific process is: based on the identity identification number of the data source, according to the configured dynamic query coding of the multi-source heterogeneous data, the correspondence between the network interconnection protocol of the data source and the dynamic query coding is obtained; based on the request frequency of the data source and the performance of the dynamic query server, according to the correspondence between the network interconnection protocol of the data source and the dynamic query coding, a mapping relationship between the dynamic query coding and the dynamic query server is established and configured.

4. The multi-source heterogeneous data routing and management method based on the ecological environment data center according to claim 1 is characterized in that: In step S3, the weights of the dynamic query servers are calculated according to the initial dynamic query server list in step S2, including the following steps: A1. Determine the performance index data of the dynamic query server according to the initial dynamic query server list in step S2; A2. Determine the weight of the performance indicator of the dynamic query server; A3. Calculate the weight of the dynamic query server according to the performance indicator data of the dynamic query server in step A1 and the weight of the performance indicator of the dynamic query server in step A2.

5. The multi-source heterogeneous data routing and management method based on the ecological environment data center according to claim 4 is characterized in that: In step A1, the performance indicator data of the server is dynamically queried including the total utilization of all CPU cores, memory utilization, disk I / O utilization, and network I / O utilization.

6. The multi-source heterogeneous data routing and management method based on the ecological environment data center according to claim 4 is characterized in that: In step A2, the weights of the performance indicators of the dynamic query server are determined, specifically: the weights of all CPU cores are determined to be 0.4, the weight of memory is determined to be 0.3, the weight of disk I / O is determined to be 0.2, and the weight of network I / O is determined to be 0.

1.

7. The multi-source heterogeneous data routing and management method based on the ecological environment data center according to claim 4 is characterized in that: In step A3, the weight of the dynamic query server is calculated according to the performance indicator data of the dynamic query server in step A1 and the weight of the performance indicator of the dynamic query server in step A2, which is expressed as: in: To dynamically query the server weight, The number of the performance indicator for the dynamic query server. To dynamically query the total number of performance indicators of the server, To dynamically query the server Performance indicator data, To dynamically query the server The weight of a performance indicator.

Citation Information

Patent Citations

  • Big data resource scheduling method and system and storage medium

    CN113806066A

  • Dynamic query method and system

    CN115114321A

  • Local area power grid data management method, device and equipment based on data medium station

    CN115439015A

  • Business processing method based on data lake system and computer readable storage medium

    CN115757460A

  • Comprehensive intelligent platform data management method and system based on spatio-temporal system

    WO2024011829A1