Visualization method and device for multi-source heterogeneous data

Abstract

The invention belongs to the technical field of information, and particularly relates to a multi-source heterogeneous data visualization method and device. The method provided by the invention comprises the steps of obtaining a first data interface based on a first data source corresponding to a first data name associated with a first visual component; obtaining first data source metadata based on the first data interface; generating a first request according to the parameters of the first visual component and the metadata configuration of the first data source; retrieving in a first database based on parameters in the first request to obtain first data; obtaining a first request message based on the first request and the first data; the first request message is sent to a first data interface; and in response to a first response entity of the first data interface, updating second data used by the first visualization component for rendering. According to the invention, the influence on normal business can be reduced while the visualization of enterprise information is realized.

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and in particular relates to a method and device for visualizing multi-source heterogeneous data. Background Art

[0002] An enterprise's different business systems are often scattered in different databases. When conducting business analysis and information presentation, it is often necessary to link the data tables of different business systems to form new tables or views, and then convert the data in the tables or views into corresponding data sets for visual presentation.

[0003] Currently, enterprises often use multiple business systems at the same time, and the corresponding database types are inconsistent. According to the requirements of developers or projects, there are databases such as Mysql, PostgreqSQL, ClickHouse and MongoDB in the enterprise. These databases are generally distributed in different networks and have inconsistent rule configurations. Different database loads and hardware configurations are inconsistent. Some databases can return query results quickly, while some databases may affect the normal operation of the business when querying.

[0004] In addition, most current application software is built on heterogeneous data storage, which also brings about a series of complex data management issues such as data storage, data security, and data conversion. The process of converting the original data format to the target data structure has high requirements for relevant personnel and low work efficiency. When collecting statistical business information, it is necessary to associate data tables of different business systems without affecting the normal use of the enterprise's database (such as row locks or time-consuming query processes, which will directly affect the normal business development). This is a technical problem that needs to be solved.

[0005] It should be noted that the information disclosed in the background technology is only intended to increase the understanding of the overall background of the present invention, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art already known to those skilled in the art. Summary of the invention

[0006] The present invention can realize enterprise information visualization while reducing the impact on normal business, and the query speed is also improved, the structure of the system is simplified, and support for multi-source data is achieved.

[0007] According to an embodiment of the present invention, a method for visualizing multi-source heterogeneous data includes: Obtaining a first data interface based on a first data source corresponding to a first data name associated with the first visualization component; Based on the first data interface, obtaining first data source metadata; generating a first request according to parameters of the first visualization component and metadata configuration of the first data source; Retrieving first data from a first database based on the parameters in the first request; Obtaining a first request message based on the first request and the first data; The first request message is sent to the first data interface; In response to the first response entity of the first data interface, updating the second data used by the first visualization component for rendering; The information associated with the first data name and included in the first response entity of the first data interface is stored in the first database.

[0008] According to an embodiment of the present invention, the first data source metadata includes data table definition, query template and database information; The process of generating the first request includes: determining parameters for the query template based on the parameters of the first visualization component; A first request is constructed based on the first database and the parameters of the query template.

[0009] According to an embodiment of the present invention, the first data does not include data that has been stored in the first database.

[0010] According to an embodiment of the present invention, the process of acquiring the first data includes: Based on the first data name and the parameters of the first visualization component, determining a first data set associated with the first data name that has been stored in the first database; Determine, based on the first data set, a number of time ranges corresponding to the first data set; Based on the time information and the multiple time ranges included in the first request, determine a first data time range associated with the first data name as the first data; The first data is not included in the first database.

[0011] According to an embodiment of the present invention, the first request message is constructed in the following manner: Determine an array of time ranges included in the query based on the first data; A first request message is constructed based on the parameter information and the time range array in the query template.

[0012] According to one embodiment of the present invention, after the first request message is sent, a second request message is created; The second request message is used to indicate that after the query associated with the first request message is started, a first query process associated with the first request message is determined; The second request message is also used to instruct to destroy the unfinished first query process after a first preset time period.

[0013] According to one embodiment of the present invention, determining a first response duration of a first request message and a first association between a time window in the first request message; Determine an abnormal value range of the response time based on the first association, wherein the abnormal value range is when the time window is greater than the abnormal value range, the query response time is greater than a preset value; A third request message is constructed based on the abnormal value range, and the first database is updated based on the third request message.

[0014] According to one embodiment of the present invention, a second data set category is obtained in the first database where the execution time is not greater than a first threshold from the current timestamp; acquiring a second data set based on the second data set category; determining a time window to be updated based on a time range of the second data set; A plurality of second request messages are constructed based on the time window to be updated and the update strategy, wherein the time window size included in each request message is not greater than a lower limit of the abnormal value.

[0015] According to one embodiment of the present invention, the updating process of the second data includes: Acquire third data in the first response entity, where the third data includes a data type, a value of the data, and a corresponding time range; saving the third data to the first database; The third data is obtained from the first database according to the parameters of the first visualization component, and the value of the second data used for rendering in the first visualization component is set to the third data.

[0016] According to one embodiment of the present invention, a visualization device for multi-source heterogeneous data includes: The metadata acquisition unit obtains a first data interface based on a first data source corresponding to a first data name associated with the first visualization component; Based on the first data interface, obtaining first data source metadata; A first request acquisition unit generates a first request according to a parameter of the first visualization component and a first data source metadata configuration; A first data acquisition unit, which retrieves first data from a first database based on the parameters in the first request; A first request message generating unit, which obtains a first request message based on the first request and the first data; A data updating rendering unit, configured to send the first request message to a first data interface, and update second data used by a first visualization component for rendering in response to a first response entity of the first data interface; The information associated with the first data name and included in the first response entity is stored in the first database. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a multi-source heterogeneous data visualization method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a visualization device for multi-source heterogeneous data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0019] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. Some backgrounds involved in the present invention are introduced as follows.

[0020] The visualization component used in the present invention is a component that is rendered based on the operating environment. For example, when the operating environment is a browser, visualization components such as antv, echarts, g3d, etc. can be selected for rendering. For such components, after configuring the rendering options, the corresponding data set Dataset or the array contained in the data, such as an array (Number[]), can be set to refresh the graph. Similarly, if the rendered component is a table, the source data corresponding to the table component is set to refresh the table content. If the corresponding operating environment is for a certain development environment, such as using jfreechart for rendering, the update process is also similar. When maintaining the corresponding source data, the corresponding chart will also be updated. Since the rendering and update mechanisms of the two are similar, when introducing the following embodiments, they will be mainly described in conjunction with the browser environment. Those skilled in the art should be aware that it is applicable to a variety of environments.

[0021] like Figure 1As shown, according to one embodiment of the present invention, a method for visualizing multi-source heterogeneous data includes: Obtaining a first data interface based on a first data source corresponding to a first data name associated with the first visualization component; Based on the first data interface, obtaining first data source metadata; generating a first request according to parameters of the first visualization component and metadata configuration of the first data source; Retrieving first data from a first database based on the parameters in the first request; Obtaining a first request message based on the first request and the first data; The first request message is sent to the first data interface; In response to the first response entity of the first data interface, updating the second data used by the first visualization component for rendering; The information associated with the first data name and included in the first response entity of the first data interface is stored in the first database.

[0022] As mentioned above, when the execution environment in the present invention is a browser, progressive rendering can be performed on the server based on Node.js or other containers, or data can be updated by passing json. Here, the server for providing data or rendered data to the client is named the first server, and the first server can be configured with several application services, database services, file services or cache services.

[0023] When the first server receives the rendering data request from the first client, or performs rendering based on component loading initialization, its context can be determined, that is, the corresponding data category can be obtained. For example, when the data category corresponding to the initialized chart is Type 1, an interface is obtained based on a local or server-side query.

[0024] For example, a rendering data request is associated with an echart bar chart, and the data name corresponding to the first data object in the bar chart is "Type1". The first data interface corresponding to the data type of Type1 is determined by caching or querying the database or configuration file. That is, based on this data interface, an API query can be performed or a query can be performed by loading a driver to obtain an address URL for query.

[0025] Since the configuration of the database may change, the actual data is avoided from being saved on the first server side, but the latest configuration is obtained by accessing the first data interface, such as the type of database, database information, driver file information, table information used for query, and template information used for query, etc. The above information is not always fully configured or fully open to the first server. Due to the existence of authority or security issues, the first server cannot always access the required resources. At this time, the first data interface, as an information providing unit, can accept the information used for query and return the corresponding results to the first server. At this time, the returned first data source metadata can provide database information, table information, and parameter configuration information, etc., to assist the first server in identifying changes in the data source based on the first database accessed. For example, when a database type changes, accurate business information can be obtained by executing a data update task. In addition, considering that data may come from multiple places, the complexity of the system can be reduced by configuring the first data interface instead of the first server to access the corresponding data.

[0026] After obtaining the metadata of the first data source, a data request can be generated based on it, that is, a data query request is constructed according to the information of the first visualization component. The request can be queried through the first data interface and the corresponding query results can be obtained; since this process may ignore the data already stored locally, for example, the first database caches the data between dates D1 and D2, when querying, it is only necessary to query the time period other than D1 and D2. At this time, the first request message of the actual request is obtained by comparing with the data in the first database. The first request message is actually executed to reduce the impact of the query on normal business. The first database is designed to design a table, wherein the table includes at least information such as data category, value, size of time window, start time, end time, source database identifier, update time, etc. Based on the above information, it can be determined whether the corresponding information is stored in the first database, thereby determining whether the corresponding data needs to be updated.

[0027] Furthermore, the query result is obtained through the first data interface. The query result is contained in the response entity ContentBody of the response message. By parsing its content, the data on the server side can be updated to the first database, and the corresponding Type1 statistical data can be obtained by querying the first database. Since the first database does not save actual data but only retains statistical information, it takes up little space and has a fast query speed. The update of the first visualization component can be achieved by returning the data that meets the requirements queried in the first database to the first client or setting it as the value of the data in the component.

[0028] Obviously, this method can avoid the interference of queries on existing businesses, avoid the additional costs introduced by migration, and improve the response speed. According to an embodiment of the present invention, the first data source metadata includes data table definition, query template and database information; The process of generating the first request includes: determining parameters for the query template based on the parameters of the first visualization component; A first request is constructed based on the first database and the parameters of the query template.

[0029] The data table definition may include the name of the data table, the type of each column, whether it has an index, default value and other information; the query template is used to execute the query string, and parameter information may be set in the string. When performing an actual query, the parameter information is set to the actual parameter information, so as to execute the expected query and return the query result. Since only parameter information needs to be provided in the template information, when the data source changes, the first server does not need to pay attention to the details of the corresponding query template. For scenarios where attention needs to be paid to the impact of the query process, the query template can be used to track the query process; the database information is used to return the actual database information of the saved data. The database information saved in the first database can be compared with the returned database information to determine whether the data source has changed. When a change occurs, the data can be updated by clearing the data in the first database or updating it step by step, or only updating the new data. According to one embodiment of the present invention, the first data does not include data that has been saved in the first database.

[0030] When the first data does not include data already stored in the first database, the existing local data is not queried by default.

[0031] In another implementation of the present invention, the first data includes data that has been saved in the first database. This scenario is used in which the data may be updated or deleted. In this case, the content of the first database needs to be consistent with the actual data. In order to avoid invalid queries, a time threshold can be set, and data above this time threshold is not displayed to reduce the number of queries; or the query results are cached on the side providing the first data interface, so that the query process is accelerated. In this case, the first data can be set to include the data of the first database. According to one embodiment of the present invention, the process of obtaining the first data includes: Based on the first data name and the parameters of the first visualization component, determining a first data set associated with the first data name that has been stored in the first database; Determine, based on the first data set, a number of time ranges corresponding to the first data set; Based on the time information and the multiple time ranges included in the first request, determine a first data time range associated with the first data name as the first data; The first data is not included in the first database.

[0032] In this way, it is possible to remove the data that has been stored locally before requesting data from the first data interface, that is, based on the first data name and the parameters of the first visualization component (especially the time range), determine the first data set associated with the first data name that has been stored in the first database; Based on the time start and end points corresponding to each data in the first data set, the data with adjacent start times are merged to obtain several time ranges. The statistical data within these time ranges are excluded when constructing the query request. For example, when the original time range is [{"start":"20240101 00:00:00", "end":"20240105 00:05:00"}], if a time range [{"start":"20240102 00:00:00", "end":"20240102 00:05:00"}] has been saved in the first database, the original time range is adjusted to [{"start":"20240101 00:00:00", "end":"20240102 00:00:00"}, {"start":"20240102 00:05:00", "end":"20240105 00:05:00"}], thereby subdividing the query process and reducing resource usage. According to one embodiment of the present invention, the first request message is constructed in the following manner: Determine an array of time ranges included in the query based on the first data; A first request message is constructed based on the parameter information and the time range array in the query template.

[0033] In this way, a query can be split and implemented in several sub-queries, wherein the query is executed by the application server associated with the first data interface, reducing the details that the client is concerned about, thereby reducing the workload of configuration. According to an embodiment of the present invention, after the first request message is sent, a second request message is created; The second request message is used to indicate that after the query associated with the first request message is started, a first query process associated with the first request message is determined; The second request message is also used to instruct to destroy the unfinished first query process after a first preset time period.

[0034] In some cases, such as when the database is performing a large number of IO tasks, or the host's resources are occupied, or the available resources of the container / virtual machine are insufficient, query jams may occur. If it does not involve the production environment, the aforementioned problems can be solved by hardware updates. However, in the actual production process, it is not realistic to solve the signing problem in the aforementioned way. At this time, by tracking the query process created and recording the query time to avoid the interference of slow queries on normal production, the impact of visualization on production can be reduced.

[0035] The specific implementation can be, when executing a query, adding a marker text in the query template, such as A12 as the name of the temporary table, and associating it with the query request through a status viewing tool provided by the database such as show processlist, and then associating the query process with the message, and regularly refreshing the query process to determine whether the corresponding process is completed. If it is not completed within the set time range, the query process can be canceled through the kill command (or other similar commands). The canceled query process can be resubmitted after narrowing the time window range and querying, which can avoid row locks or table locks that may be caused by slow queries and avoid affecting production. According to one embodiment of the present invention, the first response duration of the first request message and the first association of the time window in the first request message are determined; Determine an abnormal value range of the response time based on the first association, wherein the abnormal value range is when the time window is greater than the abnormal value range, the query response time is greater than a preset value; A third request message is constructed based on the abnormal value range, and the first database is updated based on the third request message.

[0036] As mentioned above, the query time window has little impact on the production environment when the data volume is small and the database hardware configuration is high. However, for high-load scenarios, the impact of the selected time window needs to be considered. In this case, outlier tracking can be used to determine whether a slow query has occurred. Outliers can be specified or confirmed based on an anomaly discovery algorithm.

[0037] Taking the clustering algorithm as an example, clustering is performed based on the ratio of the response time and the size of the query time window. The number of clusters specified by the clustering is 2. The response time corresponding to the cluster with a larger mean is determined as the upper and lower limits of the abnormal value range based on the time size corresponding to the members in the cluster; conversely, the response time of the cluster with a smaller mean can be used as the upper and lower limits of the normal response time. When the mean value of a query response time is higher than the lower limit of the abnormal value range, or higher than the upper limit of the normal response time, it is considered that the query has an impact on the production environment, and the time window used for the query can be readjusted based on this. For example, when the time window is 10 minutes, the query response time is too high, and the window size is reduced to 8 minutes. It should be understood that the time window can also be used to associate the response time to determine the upper and lower limits of the query time window corresponding to each type of data. According to one embodiment of the present invention, a second data set category whose execution time in the first database is not greater than a first threshold from the current timestamp is obtained; acquiring a second data set based on the second data set category; determining a time window to be updated based on a time range of the second data set; A plurality of second request messages are constructed based on the time window to be updated and the update strategy, wherein the time window size included in each request message is not greater than a lower limit of the abnormal value.

[0038] In this way, queries can be performed to reduce the impact on the production environment. Among them, the category of the second data set whose execution time is not greater than the first threshold from the current timestamp is generally the data that has been queried currently, but some data is incomplete; the second data is obtained through the category of the second data set, and the data already saved in the first database is counted, and the time window of the saved information is determined based on the statistical information, and then the time window to be updated is determined; on this basis, a number of second request messages are constructed based on the time window to be updated and the update strategy, wherein the time window size contained in each request message is not greater than the lower limit of the abnormal value. According to one embodiment of the present invention, the update process of the second data includes: Acquire third data in the first response entity, where the third data includes a data type, a value of the data, and a corresponding time range; saving the third data to the first database; The third data is obtained from the first database according to the parameters of the first visualization component, and the value of the second data used for rendering in the first visualization component is set to the third data.

[0039] By first saving the data to the first database and querying it in the first database, the burden on the server side and the actual production database can be effectively reduced, so that the speed of local update and statistical data can be accelerated. Figure 2As shown, according to an embodiment of the present invention, a visualization device for multi-source heterogeneous data is also provided, the device comprising: The metadata acquisition unit obtains a first data interface based on a first data source corresponding to a first data name associated with the first visualization component; Based on the first data interface, obtaining first data source metadata; A first request acquisition unit generates a first request according to a parameter of the first visualization component and a first data source metadata configuration; A first data acquisition unit, which retrieves first data from a first database based on the parameters in the first request; A first request message generating unit, which obtains a first request message based on the first request and the first data; A data updating rendering unit, configured to send the first request message to a first data interface, and update second data used by a first visualization component for rendering in response to a first response entity of the first data interface; The information associated with the first data name and included in the first response entity is stored in the first database.

[0040] According to an embodiment of the present invention, the first server is an application server, which accepts HTTP requests and responds to them. The first client is a browser, and when the visualization module is started, the browser is configured with a timestamp range of T1 to T2, and the corresponding data category is DataType1. The request parameters in the request message received by the first server from the first client include a start timestamp T1 and an end timestamp T2, as well as a data category; The first server sends a request to the first data interface. The value of the data category in the query request is DataType1. The first data interface returns the following content: { "DatabaseId":"ID-String-Value", "DatabaseType":"PostgresSQL16", "Scheme":"schemeName", "Table":"tableName", "args":[ {"name":"start","type","ticks"}, {"name":"start","type","ticks"}] "template":"*" } The corresponding parameter names from top to bottom are database ID, database type, database name, table name and parameter array type. The query template is empty and the actual address of the database is not provided.

[0041] The first server retrieves data of type DateType1 in the first database, and constructs several time intervals according to the upper and lower limits of the data, and then merges these time intervals according to whether they are adjacent or not to obtain first data; compares the first data with timestamps T1 and T2, removes consistent time intervals, obtains an array for representing time, and uses the array and data type as parameters to construct a first request message and sends it to the first data interface; After the first data interface executes the corresponding query, it sends the query result to the first server. The first server parses the query result message and saves the data in the first database. For example, if a query result is the following array ["DataType1", T1, T2, 3123, 1732237836000], which is data name, start timestamp, end timestamp, data value and query time in order, it is saved to the first database based on the database information contained in the metadata of the first data source. After that, the first server performs another query to obtain the corresponding result and returns the corresponding result to the first client in the form of an array. The first client (taking echarts5.2 as an example) sets it as the data value of the corresponding series in the chart data object to set the data and completes the data refresh by means of setOption.

[0042] According to an embodiment of the present invention, the first server is an application server, which accepts HTTP requests and responds to them. The first client is a browser, and when the visualization module is started, the browser is configured with a timestamp range of T1 to T2, and the corresponding data category is DataType1. The request parameters in the request message received by the first server from the first client include a start timestamp T1 and an end timestamp T2, as well as a data category; The first server sends a request to the first data interface. The value of the data category in the query request is DataType1. The first data interface returns the following content: { "DatabaseId":"ID-String-Value", "DatabaseType":"PostgresSQL16", "Scheme":"schemeName", "Table":"tableName", "args":[ {"name":"start","type","ticks"}, {"name":"start","type","ticks"}], "template":"select v,count(v)as result from tableName group by vwhere v>=? and v <?" } The corresponding parameter names from top to bottom are database ID, database type, database name, table name and parameter array type. The query template has actual values, but the actual address of the database is not provided, so debugging and data acquisition cannot be achieved locally.

[0043] The first server retrieves data of type DateType1 in the first database, and constructs several time intervals according to the upper and lower limits of the data, and then merges these time intervals according to whether they are adjacent or not to obtain the first data; compares the first data with timestamps T1 and T2, removes consistent time intervals, obtains an array for representing time, divides the time range in the array according to the first time window (30 minutes), and uses the divided array and data type as parameters to construct a first request message and send it to the first data interface; After the first data interface executes the corresponding query, it sends the query result to the first server. The first server parses the query result message and saves the data in the first database. For example, a query result is the following array ["DataType1", T1, T2, 3123, 1732237836000, 1200], which is the data name, start timestamp, end timestamp, data value, query time and query duration in order. It is saved to the first database based on the database information contained in the metadata of the first data source. Then the first server performs another query to obtain the corresponding result and returns the corresponding result to the first client in the form of an array. The first client (taking echarts5.2 as an example) sets it to the data value of the corresponding series in the chart data object to set the data and completes the data refresh by means of setOption.

[0044] According to an embodiment of the present invention, the first server is an application server, which accepts HTTP requests and responds to them. The first client is a browser, and when the visualization module is started, the browser is configured with a timestamp range of T1 to T2, and the corresponding data category is DataType1. The request parameters in the request message received by the first server from the first client include a start timestamp T1 and an end timestamp T2, as well as a data category; The first server sends a request to the first data interface. The value of the data category in the query request is DataType1. The first data interface returns the following content: { "DatabaseId":"ID-String-Value", "DatabaseType":"Mysql5.6", "Scheme":"schemeName", "Table":"tableName", "args":[ {"name":"start","type","ticks"}, {"name":"start","type","ticks"}], "template":"*" } The corresponding parameter names from top to bottom are database ID, database type, database name, table name and parameter array type. The query template does not have an actual value and the actual address of the database is not provided, so debugging and data acquisition cannot be achieved locally.

[0045] The first server retrieves data of type DateType1 in the first database, and constructs several time intervals according to the upper and lower limits of the data, and then merges these time intervals according to whether they are adjacent or not to obtain the first data; compares the first data with timestamps T1 and T2, removes consistent time intervals, obtains an array for representing time, divides the time range in the array according to the first time window (30 minutes), and uses the divided array and data type and the maximum response time of 500 ms as parameters to construct a first request message and send it to the first data interface; the first data interface obtains the command (ActionCommand) contained in the currently executed process, extracts the identifier contained therein, determines the process associated with the query based on the identifier, sets the current timestamp as the starting timestamp, and periodically (at intervals of 100 ms) queries whether the current process is completed. If not completed, the query is terminated by killing the corresponding process, and the query is re-performed by reducing the time window of the unfinished query. If it is still not completed, the size of the time window can be further reduced; After the first data interface executes the corresponding query, it sends the query result to the first server. The first server parses the query result message and saves the data in the first database. For example, a query result is the following array ["DataType1", T1, T2, 3123, 1732237836000, 1200], which is the data name, start timestamp, end timestamp, data value, query time and query duration in order. It is saved to the first database based on the database information contained in the metadata of the first data source. Then the first server performs another query to obtain the corresponding result and returns the corresponding result to the first client in the form of an array. The first client (taking echarts5.2 as an example) sets it to the data value of the corresponding series in the chart data object to set the data and completes the data refresh by means of setOption.

[0046] According to an embodiment of the present invention, the first server is an application server, which accepts HTTP requests and responds to them. The first client is a browser, and when the visualization module is started, the browser is configured with a timestamp range of T1 to T2, and the corresponding data category is DataType1. The request parameters in the request message received by the first server from the first client include a start timestamp T1 and an end timestamp T2, as well as a data category; The first server sends a request to the first data interface. The value of the data category in the query request is DataType1. The first data interface returns the following content: { "DatabaseId":"ID-String-Value", "DatabaseType":"Mysql5.6", "Scheme":"schemeName", "Table":"tableName", "args":[ {"name":"start","type","ticks"}, {"name":"start","type","ticks"}], "template":"*" } The corresponding parameter names from top to bottom are database ID, database type, database name, table name and parameter array type. The query template does not have an actual value and the actual address of the database is not provided, so debugging and data acquisition cannot be achieved locally.

[0047] The first server retrieves data of type DateType1 in the first database to obtain the first data set, and constructs several time intervals according to the upper and lower limits of the data, and then merges these time intervals according to whether they are adjacent or not to obtain the first data; compares the first data with timestamps T1 and T2, removes consistent time intervals, and obtains an array for representing time; classifies the query process according to the average of the query time window and the response time, and obtains 20 minutes as the upper limit of the query process time window; based on the first data, the time range in the array is divided according to the first time window (20 minutes), and the divided array and data type are used as parameters to construct a first request message and send it to the first data interface; wherein 20 minutes is the time window size, which is the upper limit of the corresponding time window when the query completion time in the first data set is within 50 ms; After the first data interface executes the corresponding query, it sends the query result to the first server. The first server parses the query result message and saves the data in the first database. For example, a query result is the following array ["DataType1", T1, T2, 3123, 1732237836000, 1200], which is the data name, start timestamp, end timestamp, data value, query time and query duration in order. It is saved to the first database based on the database information contained in the metadata of the first data source. Then the first server performs another query to obtain the corresponding result and returns the corresponding result to the first client in the form of an array. The first client (taking echarts5.2 as an example) sets it to the data value of the corresponding series in the chart data object to set the data and completes the data refresh by means of setOption.

[0048] According to one embodiment of the present invention, after completing the first query, the data type saved in the first database class is obtained, the time range to be queried is determined based on the data type and the saved time window, a second request message is constructed with 20 minutes as the time window for query, and the first database is updated based on the data contained in the second request message.

[0049] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed on an electronic device, the electronic device executes the aforementioned method.

[0050] An embodiment of the present application further provides a computer program product, including: a computer program code, when the computer program code is executed on an electronic device, the electronic device executes the aforementioned method.

[0051] An embodiment of the present application further provides a chip, comprising: a processor, configured to call and run a computer program from a memory, so that an electronic device equipped with the chip executes the aforementioned method.

[0052] Through the description of the above implementation mode, those skilled in the art can understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0053] It should be understood that the devices and processes disclosed in the several embodiments provided in the present application can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device. In addition, some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0054] The units described as separate components may or may not be physically separated. The components shown as units may be one physical unit or multiple physical units. That is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected to achieve the purpose of this solution according to actual needs.

[0055] In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit; or may exist physically separately; or some units may be integrated into one unit, and some units may exist physically separately. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0056] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, all or part of the technical solution of the embodiment of the present application can be embodied in the form of a software product. The software product is stored in a storage medium. The software product includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0057] It should be noted that all or part of the above-mentioned embodiments provided in the present application (for example, part or all of any feature) can be arbitrarily combined or used in combination with each other.

[0058] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A visualization method for multi-source heterogeneous data, characterized in that: include: Obtaining a first data interface based on a first data source corresponding to a first data name associated with the first visualization component; Based on the first data interface, obtaining first data source metadata; generating a first request according to parameters of the first visualization component and metadata configuration of the first data source; Retrieving first data from a first database based on the parameters in the first request; Obtaining a first request message based on the first request and the first data; The first request message is sent to the first data interface; In response to the first response entity of the first data interface, updating the second data used by the first visualization component for rendering; The information associated with the first data name and included in the first response entity of the first data interface is stored in the first database.

2. The method for visualizing multi-source heterogeneous data according to claim 1, characterized in that: The first data source metadata includes data table definition, query template and database information; The process of generating the first request includes: determining parameters for the query template based on the parameters of the first visualization component; A first request is constructed based on the first database and the parameters of the query template.

3. The method for visualizing multi-source heterogeneous data according to claim 2, characterized in that: The first data does not include data that has been stored in the first database.

4. The method for visualizing multi-source heterogeneous data according to claim 2, characterized in that: The process of obtaining the first data includes: Based on the first data name and the parameters of the first visualization component, determining a first data set associated with the first data name that has been stored in the first database; Determine, based on the first data set, a number of time ranges corresponding to the first data set; Based on the time information and the multiple time ranges included in the first request, determine a first data time range associated with the first data name as the first data; The first data is not included in the first database.

5. The method for visualizing multi-source heterogeneous data according to claim 2, characterized in that: The first request message is constructed in the following manner: Determine an array of time ranges included in the query based on the first data; A first request message is constructed based on the parameter information and the time range array in the query template.

6. The method for visualizing multi-source heterogeneous data according to claim 5, characterized in that: After the first request message is sent, creating a second request message; The second request message is used to indicate that after the query associated with the first request message is started, a first query process associated with the first request message is determined; The second request message is also used to instruct to destroy the unfinished first query process after a first preset time period.

7. The method for visualizing multi-source heterogeneous data according to claim 2, characterized in that: Determine a first association between a first response duration of the first request message and a time window in the first request message; Determine an abnormal value range of the response time based on the first association, wherein the abnormal value range is when the time window is greater than the abnormal value range, the query response time is greater than a preset value; A third request message is constructed based on the abnormal value range, and the first database is updated based on the third request message.

8. The method for visualizing multi-source heterogeneous data according to claim 7, characterized in that: Obtain a second data set category in the first database whose execution time is not greater than a first threshold from the current timestamp; acquiring a second data set based on the second data set category; Determining a time window to be updated based on a time range of the second data set; A plurality of second request messages are constructed based on the time window to be updated and the update strategy, wherein the time window size included in each request message is not greater than a lower limit of the abnormal value.

9. The method for visualizing multi-source heterogeneous data according to claim 1, characterized in that: The updating process of the second data includes: Acquire third data in the first response entity, where the third data includes a data type, a value of the data, and a corresponding time range; saving the third data to the first database; The third data is acquired from the first database according to the parameters of the first visualization component, and the value of the second data used for rendering in the first visualization component is set to the third data.

10. A visualization device for multi-source heterogeneous data, characterized in that: include: The metadata acquisition unit obtains a first data interface based on a first data source corresponding to a first data name associated with the first visualization component; Based on the first data interface, obtaining first data source metadata; A first request acquisition unit generates a first request according to a parameter of the first visualization component and a first data source metadata configuration; A first data acquisition unit, which retrieves first data from a first database based on the parameters in the first request; A first request message generating unit, which obtains a first request message based on the first request and the first data; A data updating rendering unit, configured to send the first request message to a first data interface, and update second data used by a first visualization component for rendering in response to a first response entity of the first data interface; The information associated with the first data name and included in the first response entity is stored in the first database.

Images