Data table view management method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a data table view management method and device, equipment and a storage medium, and belongs to the technical field of data processing. The method comprises the following steps: receiving a view creation instruction, wherein the view creation instruction is used for creating a view in a first data source; under the condition that the view creation instruction indicates that the created view is a cross-source view, a second data source accessed by the cross-source view in the view creation instruction is determined, and the cross-source view is a data table created based on data from at least one second data source; acquiring data of each data source table in the view creation instruction from each second data source; and generating the cross-source view according to the data of the data source table. By adopting the scheme provided by the embodiment of the invention, the data in different data sources are subjected to conjoint analysis by creating the cross-source view crossing the plurality of data sources, so that the data query and management efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a data table view management method, apparatus, device, and storage medium. Background Art

[0002] With the development of big data technology, more and more businesses rely on database systems. Therefore, a wide variety of database systems exist in the big data field to handle various types of big data businesses. In practice, multiple database systems are often used to store data to meet the needs of different scenarios.

[0003] In related technologies, cross-data source queries are implemented by creating database system views through the same syntax statements or using specialized query tools. These tools have high usage thresholds and low usage rates, and the supported scenarios are also relatively limited, thus affecting the efficiency of cross-source data queries. Summary of the Invention

[0004] The present invention provides a data table view management method, apparatus, device, and storage medium. By creating a cross-source view across multiple data sources, it enables joint analysis of data from different data sources, thereby improving data query and management efficiency. The technical solution is as follows:

[0005] In one aspect, an embodiment of the present application provides a data table view management method, the method comprising:

[0006] receiving a view creation instruction, where the view creation instruction is used to create a view in a first data source;

[0007] In a case where the view creation instruction indicates that the created view is a cross-source view, determining a second data source accessed by the cross-source view in the view creation instruction, where the cross-source view is a data table created based on data from at least one second data source;

[0008] Acquire data from each second data source table in the view creation instruction;

[0009] The cross-source view is generated according to the data in the data source table.

[0010] On the other hand, an embodiment of the present application provides a data table view management device, the device comprising:

[0011] An instruction receiving module, configured to receive a view creation instruction, wherein the view creation instruction is used to create a view in the first data source;

[0012] a data source determining module, configured to determine, when the view creation instruction indicates that the view to be created is a cross-source view, a second data source accessed by the cross-source view in the view creation instruction, wherein the cross-source view is a data table created based on data from at least one second data source;

[0013] A data acquisition module, configured to acquire data of each data source table in the view creation instruction from each second data source;

[0014] A view generation module is used to generate the cross-source view according to the data in the data source table.

[0015] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the data table view management method described in the above aspects.

[0016] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the data table view management method as described in the above aspects.

[0017] In another aspect, embodiments of the present application provide a computer program product comprising at least one instruction stored in a computer-readable storage medium. A processor of a computer device reads the at least one instruction from the computer-readable storage medium and executes the at least one instruction, causing the computer device to perform the data table view management method described in the above aspects.

[0018] In an embodiment of the present application, when the received view creation instruction indicates that a cross-source view is to be created in a first data source, by determining the second data source accessed by the cross-source view, the data of each data source table in the view creation instruction is directly obtained from each second data source. That is, a cross-source view can be generated based on the data in the data source table without the need to manually adapt and adjust the syntax of the data in different data sources. That is, joint analysis of data from different data sources can be achieved, which reduces the threshold for data query across data sources and improves data query and management efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 A schematic diagram showing an implementation environment provided by an exemplary embodiment of the present application is shown;

[0021] Figure 2 A flowchart of a data table view management method provided by an exemplary embodiment of the present application is shown;

[0022] Figure 3 A schematic diagram of a view execution plan tree corresponding to a cross-source view provided by an exemplary embodiment of the present application is shown;

[0023] Figure 4 A schematic diagram of merging data source subtrees corresponding to equivalent data sources provided by an exemplary embodiment of the present application is shown;

[0024] Figure 5 A schematic diagram of merging data source subtrees corresponding to equivalent data sources provided by another exemplary embodiment of the present application is shown;

[0025] Figure 6 A schematic diagram of creating a temporary view provided by an exemplary embodiment of the present application is shown;

[0026] Figure 7 A flowchart of executing a data processing task including a cross-source view provided by an exemplary embodiment of the present application is shown;

[0027] Figure 8 A schematic diagram of the structure of a cross-source and cross-engine integrated view framework provided by an exemplary embodiment of the present application is shown;

[0028] Figure 9 A structural block diagram of a data table view management device provided by an exemplary embodiment of the present application is shown;

[0029] Figure 10 A schematic structural diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0031] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0032] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0033] It should be understood that although the terms first, second, etc. may be used in this application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0034] It should be noted that this application can display a prompt interface, pop-up window or output voice prompt information before collecting relevant user data (including but not limited to user device information, user personal information, etc.) and during the process of collecting relevant user data. The prompt interface, pop-up window or voice prompt information is used to remind the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining user-related data after obtaining the user's confirmation operation on the prompt interface or pop-up window. Otherwise (that is, when the user's confirmation operation on the prompt interface or pop-up window is not obtained), the relevant steps of obtaining user-related data are terminated, that is, the user's relevant data is not obtained. In other words, all user data collected by this application are collected with the user's consent and authorization, and the collection, use and processing of relevant user data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0035] First, a brief introduction to the terms involved in the embodiments of this application is given:

[0036] Data source: A system that stores and processes data, such as traditional relational database management systems (RDBMS) such as MySQL, PostgreSQL (PG), Click House (CK), and Oracle, or big data NoSQL systems such as Hive, HBase, and Elasticsearch.

[0037] Computing Engine: A distributed computing system that supports access to different types of data sources through customized connectors, aggregates data and performs joint analysis.

[0038] View: A view in a data source or computing engine is a virtual table defined by SQL (Structured Query Language), also known as a logical view. Like a real relational table, a view contains named columns and rows of data. However, the view table does not actually store data. Instead, when the view is used, data is dynamically retrieved from the referenced table set through the query that defines the view.

[0039] Materialized Views: A materialized view (MV) in a data source or computing engine is a pre-calculated view that materializes and stores the results of time-consuming operations (such as joins and aggregates) for direct reuse in subsequent SQL executions, accelerating queries. A materialized view can be considered a view table that stores real data, or a special real table used to improve computing efficiency.

[0040] Cross-source view: A data table created based on data from at least two data sources, which can be either a logical view or a materialized view.

[0041] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing.

[0042] Cloud technology is a general term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on the cloud computing business model. It can form a resource pool that can be used flexibly and conveniently on demand. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as video websites, image websites, and more portals, require extensive computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identifier, requiring transmission to backend systems for logical processing. Different levels of data will be processed separately, and data from various industries will require a strong system backend, which can only be achieved through cloud computing.

[0043] A database, in short, can be thought of as a digital filing cabinet—a place where electronic files are stored, allowing users to add, query, update, and delete data. A database is a collection of data stored in a specific way, shared by multiple users, with minimal redundancy, and independent of applications.

[0044] A database management system (DBMS) is a computer software system designed for managing databases, typically providing basic functions such as storage, retrieval, security, and backup. DBMSs can be categorized by the database model they support, such as relational or XML (Extensible Markup Language); by the type of computer they support, such as server clusters or mobile phones; by the query language used, such as SQL (Structured Query Language) or XQuery; by performance priorities, such as maximum scale or maximum speed; or by other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, supporting multiple query languages ​​simultaneously.

[0045] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application. In some embodiments, the implementation environment in the embodiments of the present application may include a terminal 120 and a server 140. Data communication between the terminal 120 and the server 140 is performed via a communication network. Optionally, the communication network may be a wired network or a wireless network, and the communication network may be at least one of a local area network, a metropolitan area network, and a wide area network.

[0046] The terminal 120 is an electronic device installed with an application program having a database query function. The database query function can be a function of a native application in the terminal or a function of a third-party application. The terminal can be a smartphone, tablet computer, laptop computer, desktop computer, smart TV, wearable device, or vehicle-mounted terminal, etc. Figure 1 The terminal 120 is taken as an example as a laptop computer, but this is not a limitation.

[0047] Server 140 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. In the embodiment of the present application, the server can be a backend server for an application with database query capabilities.

[0048] Optionally, the terminal 120 can be configured with multiple computing engines 160 , and each computing engine supports access to different types of data sources 180 and can aggregate and jointly analyze data. The server 140 can access data in the data source 180 by selecting one or more computing engines 160 .

[0049] In some embodiments, there is data interaction between the server and the terminal, and a data table view management method proposed in an embodiment of the present application can be executed by the terminal, or by the server, or by both the terminal and the server. Figure 1 Taking the case where a terminal and a server jointly execute the data table view management method as an example, the terminal 120 receives a view creation instruction input by a user in a big data computing platform for creating a view in a first data source, and determines the view to be created indicated by the view creation instruction. When the view to be created indicated by the view creation instruction is a cross-source view, the terminal 120 further determines the second data source accessed by the cross-source view in the view creation instruction, and simultaneously sends the view creation instruction to the server 140. The server 140 uses the computing engine 160 to obtain data from each data source table from each second data source 180 based on the second data source accessed by the cross-source view, thereby generating a cross-source view based on the data in the data source table, and returns the cross-source view to the terminal 120. The terminal 120 then displays the data query results in the cross-source view to the user.

[0050] Please refer to Figure 2 , which shows a flow chart of a data table view management method provided by an exemplary embodiment of the present application. This embodiment is described by taking the method applied to a computer device (including a terminal and / or a server) as an example. The method includes the following steps:

[0051] Step 201: Receive a view creation instruction, where the view creation instruction is used to create a view in a first data source.

[0052] A view is a data table created by performing conditional queries and calculations on the data in a data source table. This includes logical views and materialized views. For example, to query and calculate employee gender and age in a database to obtain gender-related data, you can create a view and query the database for relevant data.

[0053] Optionally, view creation includes determining a view definition and defining the view in a corresponding data source. In the embodiment of the present application, the view creation instruction is used to define the cross-source view in the first data source.

[0054] Optionally, the first data source can be a traditional relational database (RDBMS) such as MySQL, PostgreSQL (PG), Click House (CK), Oracle, etc., or a big data NoSQL system such as Hive, HBase, Elasticsearch, etc., which is not limited in the embodiments of the present application.

[0055] In some embodiments, a computer device may provide a user with a window for view creation and data query via a big data computing platform, so that the user can create a view by entering a view creation instruction in the view creation window. In one possible implementation, the computer device receives a view creation instruction for creating a view in a first data source.

[0056] Optionally, the view creation instruction can be an SQL statement. For example, the SQL statement "CREATE VIEW hive_ds.v1 AS" creates a view named "hive_ds" in the Hive data source.

[0057] Step 202 : When the view creation instruction indicates that the created view is a cross-source view, determine a second data source accessed by the cross-source view in the view creation instruction, where the cross-source view is a data table created based on data from at least one second data source.

[0058] Optionally, views can be divided into single-source views and cross-source views based on the source of the data required when creating the view. A single-source view is a data table created based on data from the current first data source, while a cross-source view is a data table created based on data from at least one second data source.

[0059] Optionally, different data sources have different data source names. For example, "mysql_ds1" and "mysql_ds2" are two different MySQL data sources, and "pg_ds" is a PG data source.

[0060] Optionally, a data source may include one data source table or multiple data source tables. For example, the data source "mysql_ds1" may include the data source table t1 "mysql_ds1.t1" and the data source table t2 "mysql_ds1.t2".

[0061] In some embodiments, a computer device may determine, based on the table name of a data source table included in the view creation instruction, whether the view to be created by the view creation instruction is a single-source view or a cross-source view. Furthermore, if the view to be created by the view creation instruction is a cross-source view, the computer device determines the second data source accessed by the cross-source view in the view creation instruction.

[0062] Step 203: Acquire data of each data source table in the view creation instruction from each second data source.

[0063] In some embodiments, after determining the second data source accessed by the cross-source view, in order to generate the cross-source view, the computer device needs to obtain data of each data source table in the view creation instruction from each second data source respectively.

[0064] In one possible implementation, when defining a cross-source view using a view creation instruction, the view creation instruction may include not only the name of the data source table accessed by the cross-source view but also a data query requirement. For example, the SQL statement SELECT t1.cnt, t3.col2 FROM mysql_ds1.t1 requests the selection of data in the cnt column of row t1 and the col2 column of row t3 from the data source table named mysql_ds1.t1. ​​Furthermore, the computer device may obtain data from each of the data source tables referenced in the view creation instruction from each of the second data sources based on the view creation instruction.

[0065] The data in a data source table includes the table's definition, structure, column data types, index information, and constraint information. It may also include the data source definition for the data source to which the table belongs. This data source definition is a detailed description of the data source, including its location, access method, authentication information, data structure, and access permissions. It should be noted that the data in a data source table can also be understood as metadata for the data source table; that is, the data in a data source table does not include the actual data stored within the table.

[0066] Optionally, the computer device may connect to the second data source by using a JDBC (Java Database Connectivity) interface to obtain data from each data source table from the second data source.

[0067] Step 204: Generate a cross-source view based on the data in the data source table.

[0068] In some embodiments, after obtaining the data of each data source table in the view creation instruction, the computer device can generate a cross-source view based on the data of each data source table and display the data query results to the user.

[0069] Optionally, when the cross-source view is a logical view, the computer device directly generates the cross-source view based on the data in the data source table; when the cross-source view is a materialized view, after obtaining the data from each data source table, the computer device needs to calculate and process the data in each data source table according to the data processing method indicated by the view creation instruction, thereby generating a cross-source view based on the data in the data source table and the data calculation results.

[0070] To summarize, in an embodiment of the present application, when the received view creation instruction instructs to create a cross-source view in the first data source, by determining the second data source accessed by the cross-source view, the data of each data source table in the view creation instruction is directly obtained from each second data source. That is, a cross-source view can be generated based on the data in the data source table without the need to manually adapt and adjust the syntax of the data in different data sources. That is, joint analysis of data from different data sources can be achieved, which reduces the threshold for data queries across data sources and improves data query and management efficiency.

[0071] In some embodiments, for a single-source view that only accesses a single data source, the computer device can store it in the corresponding data source; and for a cross-source view that accesses at least two data sources, in order to improve the management efficiency of the cross-source view, the computer device can store and manage it through a unified cross-source view database, that is, the cross-source view database is used to store view data of the cross-source view.

[0072] Optionally, view data includes view definitions, view objects, data source definitions of various data sources accessed by cross-source views, etc. Among them, the view definition refers to the SQL statement used when creating the view, which describes how to select, transform and combine data from underlying tables or other views to generate the content of the view. The view definition contains the structure and logic of the view, such as the tables involved, selected columns, connection conditions, filter conditions, aggregate functions, etc. The view object refers to the virtual table created in the database based on the view definition. It does not store data itself, but dynamically applies the view definition to generate data when the view is queried. The view object allows users to query the view like querying an actual table and can be used for further SQL operations.

[0073] In one possible implementation, after receiving a view creation instruction, the computer device first obtains the table name of the data source table contained in the view creation instruction, and when the table name of the data source table indicates different data sources, determines that the view to be created indicated by the view creation instruction is a cross-source view.

[0074] Furthermore, the computer device generates a view definition for the cross-source view based on the view creation instruction. The view definition may include metadata such as the view name, SQL text, and view properties. Furthermore, to uniformly manage the cross-source view, the computer device uniformly stores the view definition for the cross-source view and the data source definition of the second data source accessed by the cross-source view in a cross-source view database.

[0075] In one possible implementation, in order to enable data query and processing across computing engines on a big data computing platform, before storing the data in a cross-source view database, the computer device may also perform unified format processing on the view definition of the cross-source view and the data source definition of the second data source accessed by the cross-source view, so that the cross-source view processed in the unified format supports queries from different computing engines, and then the view definition and data source definition processed in the unified format are stored in the cross-source view database.

[0076] Optionally, the computing engine may be Spark / Presto / Hive / StarRocks, etc., which is not limited in the embodiments of the present application.

[0077] In some embodiments, after the view data of the cross-source view is uniformly stored in the cross-source view database, when a data processing task containing the cross-source view is received, the computer device can directly obtain the view data corresponding to the cross-source view from the cross-source view database.

[0078] Optionally, a data processing task refers to a task that requires data processing, specifically a computational task that requires joint analysis of data from different data sources. A data processing task can be sent by a user to a server via a terminal, such as by sending an SQL statement to indicate the data processing task to be executed. In other words, the data processing task can specifically take the form of an SQL statement. In different application scenarios, data processing tasks can perform different types of computational processing on data from different data sources.

[0079] In one possible implementation, upon receiving a data processing task containing a cross-source view, the computer device can obtain the view name of the cross-source view by parsing the data processing task, and then obtain the view data of the cross-source view from the cross-source view database based on the view name.

[0080] In the above embodiment, when the created view is a cross-source view, the view definition is stored in the cross-source view database for persistent storage, and the data source definitions of multiple data sources accessed by the cross-source view are also stored in the cross-source view database. Subsequently, when a data processing task containing a cross-source view is received, the view data can be directly obtained from the cross-source view database. This process is the same as the process of obtaining view data of a single-source view from a single data source, which optimizes the management of cross-source views and enables users to use cross-source views seamlessly and transparently.

[0081] In some embodiments, when a data processing task includes a data processing subtask for accessing a cross-source view, each time a cross-source view is accessed, the computer device needs to obtain the view data of the cross-source view from a cross-source view database. The cross-source view database includes a data access interface (Application Programming Interface, API), and the computer device needs to read the view data of the cross-source view by calling the data access interface. The response time for a single call to the data access interface is generally tens of milliseconds.

[0082] However, in real-world business scenarios, complex data statistical analysis logic may be involved, leading to multiple accesses to a cross-source view within a short period of time. This means multiple calls to the data access interface to read the view data, wasting network and computing resources and increasing data processing time. Therefore, to improve data query efficiency, after initially reading the cross-source view data, the computer device can cache the view data to enable reuse.

[0083] Optionally, the caching method for view data can be divided into three levels, and the optimization scope of different cache levels is different. The first-level caching method is the cache based on data processing tasks (query cache), that is, no matter how many times a data processing task accesses a cross-source view, only the first time will the data access interface be called to obtain view data from the cross-source view database; the second-level caching method is the cache based on session processing tasks (session cache), that is, in a session processing task within a period of time, only the first time will the data access interface be called to obtain view data from the cross-source view database; the third-level caching method is the cache based on global processing tasks (global cache), that is, in a global processing task, only the first time will the data access interface be called to obtain view data from the cross-source view database.

[0084] Regarding Query Cache, in one possible implementation, when a data processing task includes multiple data processing subtasks for accessing cross-source views, when a computer device first accesses the cross-source view, it obtains the view data of the cross-source view from the cross-source view database through the data access interface according to the view name, and after obtaining the view data, it creates a cache object corresponding to the view data. Then, when the cross-source view is accessed for the i-th time (i is greater than 1), the view data can be obtained from the cache object without having to use the data access interface to obtain data from the cross-source view database.

[0085] For Query cache, its cache optimization scope is the smallest. View data is cached only in a single data processing task. The cache object is not shared between different data processing tasks. Therefore, there is no need to consider the issue of view definition expiration during long data processing. The currently obtained view data is valid data.

[0086] Regarding session caching, a session processing task within a period of time includes multiple data processing tasks, and the data processing tasks are executed serially within the session processing task. In one possible implementation, when a cross-source view is accessed for the first time during session task processing, the computer device retrieves the view data of the cross-source view from the cross-source view database based on the view name. After obtaining the view data, the computer device creates a cache object corresponding to the view data. Then, when the cross-source view is accessed for the i-th time (i is greater than 1) during the session processing task, the computer device can directly retrieve the view data from the cache object without having to use the data access interface to retrieve data from the cross-source view database.

[0087] For Session cache, its cache optimization scope is medium. View data is cached only within a session processing task. Cache objects are not shared between different session processing tasks. In addition, the life cycle of most session processing tasks is only at the minute or hour level. Therefore, there is usually no problem of view definition expiration.

[0088] Regarding the Global Cache, global task processing includes multiple task processing tasks. In one possible implementation, when a cross-source view is first accessed in global task processing, the computer device retrieves the view data of the cross-source view from the cross-source view database based on the view name. After obtaining the view data, the computer device creates a cache object corresponding to the view data. Then, when the cross-source view is accessed for the i-th time (i is greater than 1) in global task processing, the computer device can directly retrieve the view data from the cache object without having to use the data access interface to retrieve data from the cross-source view database.

[0089] For the Global cache, its cache optimization scope is the largest, and sharing is achieved in global processing tasks. After the cache object of the cross-source view is created, the computer device in subsequent data processing tasks directly obtains the view data from the cache object.

[0090] Combining the three levels of caching mentioned above, although the session cache and global cache have a larger optimization scope, they may both have the problem of cross-source view definition expiration. In addition, the deployment and operation of different service instances in the same cluster are independent. A service instance cannot detect the view modification command executed on another service instance in real time. Therefore, in order to avoid the problem of view definition expiration, the computer device can also adopt two refresh mechanisms for cache objects of cross-source views.

[0091] In a possible implementation, when accessing a cross-source view for the i-th time, the computer device may obtain only the latest view definition of the cross-source view from the cross-source view database based on the view name, without obtaining all view data, and thereby compare the latest view definition with the historical view definition in the cache object. If the latest view definition matches the historical view definition, it is determined that the view definition in the cache object is not expired, and the computer device may obtain the view data from the cache object. If the latest view definition does not match the historical view definition, the computer device needs to obtain the view data again from the cross-source view database and rebuild the cache object.

[0092] In the above refresh mechanism, the computer device still needs to call the data access interface when obtaining the view definition, which means that the network overhead of the interface call cannot be saved, but there is no need to re-obtain the view data and rebuild the cache object, which relatively saves the computational overhead of cache object reconstruction.

[0093] In another possible implementation, the computer device may set a data expiration date (TTL) for each cross-source view. After creating a cache object and when the TTL for the cross-source view expires, the cache object for the cross-source view automatically expires, and the view data for the cross-source view is reloaded and rebuilt. Furthermore, when accessing the cross-source view for the i-th time, the computer device may first determine the data expiration date of the view data in the cache object, and if the data expiration date has not expired, retrieve the view data from the cache object.

[0094] Optionally, the TTL time of the view data may be set by the user or by the computer device according to the frequency of data access, which is not limited in this embodiment of the present application.

[0095] In the above refresh mechanism, the cache object is automatically invalidated and reloaded according to the TTL time of the view data. Although it cannot guarantee the strong consistency of the view data, it saves the network overhead of each call to the data access interface and improves the efficiency of data query.

[0096] In some embodiments, when applying the three-level cache mechanism described above, the computer device can determine whether to apply the cache mechanism at each level based on the switch parameters corresponding to that level of cache. Furthermore, when applying session cache or global cache, cached objects can be periodically refreshed based on the set TTL time. For example, if the global cache is enabled, the query cache and session cache are disabled, and the TTL time is set to 1 hour, the SQL statement can be set to:

[0097] set `supersql.crossDsView.queryCache.enabled` = false / / Disable Query cache

[0098] set `supersql.crossDsView.sessionCache.enabled` = false / / Disable Session cache

[0099] set `supersql.crossDsView.globalCache.enabled` = true / / Enable Global cache

[0100] set `supersql.crossDsView.globalCache.ttl` = 3600 / / Unit: seconds

[0101] In the above embodiment, when executing a data processing task that includes accessing a cross-source view, a cache object corresponding to the view data is created after obtaining the view data from the cross-source view database for the first time. When the cross-source view is accessed subsequently, the view data is directly obtained from the cache object, thereby saving the network overhead of calling the data access interface and the computational overhead of rebuilding the cache object, improving the access efficiency to the cross-source view, and optimizing the data query efficiency.

[0102] Furthermore, by setting up three levels of caching mechanisms and two refresh mechanisms, users can adjust the caching mechanism and refresh mechanism according to current data query requirements, thereby improving the flexibility of obtaining view data of cross-source views.

[0103] In some embodiments, since cross-source views are created by users based on data processing needs, the view creator has all permissions to the cross-source view. After creating a cross-source view, he or she has the right to query the view, modify the view definition, or delete the view. Other users can only process the cross-source view based on the view permissions granted by the view creator.

[0104] In one possible implementation, a user can set view permissions for a cross-source view in a view creation instruction, where the view permissions include at least one of view query permissions, view modification permissions, and view deletion permissions. Furthermore, after receiving the view creation instruction for creating a cross-source view, the computer device can determine the view permissions for the cross-source view based on the view creation instruction and store the view permissions for the cross-source view in a cross-source view database.

[0105] Optionally, you can grant other users or user groups the right to delete, modify, or query the cross-source view by using the GRANT (grant) or REVOKE (revoke) commands. For example, GRANT SELECT, ALTER on hive_ds.v1 to user1 grants user user1 the right to query and modify the cross-source view hive_ds.v1; GRANT SELECT, ALTER on hive_ds.v1 to userGroup1 grants all users in userGroup1 the right to query and modify the cross-source view hive_ds.v1; and REVOKE DROP on hive_ds.v1 to user2 revokes the right to delete the cross-source view hive_ds.v1 from user user2.

[0106] In some embodiments, upon receiving a data processing task containing a cross-source view, in order to determine whether the user who issued the data processing task has view permissions, the computer device can parse the data processing task to determine the view name of the cross-source view, and then obtain the view permissions of the cross-source view from the cross-source view database based on the view name, and obtain the view data of the cross-source view from the cross-source view database if the view permissions indicate that the cross-source view is allowed to be queried.

[0107] In a possible implementation, a permission interface is provided in the cross-source view database. After the computer device queries the view permission in the cross-source view database through the data access interface, the database can return the view permission to the computer device through the permission interface.

[0108] Optionally, since the cross-source view includes multiple data source tables and needs to access multiple data sources, the query permission for the cross-source view can also be subdivided into query permission for each data source table, that is, when the user has query permission for the cross-source view, the computer device can obtain the view data of the cross-source view from the cross-source view database; or, when the user has query permission for each data source table corresponding to the cross-source view, the computer device can also obtain the view data of the cross-source view from the cross-source view database.

[0109] In the above embodiment, when creating a cross-source view, the view creator sets the view permissions of the cross-source view, and stores the view permissions and view data together in the cross-source view database. Therefore, when executing a data processing task involving a cross-source view, it is possible to first determine whether the view permissions for the cross-source view are possessed, and then obtain the view data from the cross-source view database based on the view permissions, thereby optimizing the security of accessing the cross-source view and improving data management efficiency.

[0110] In some embodiments, since a cross-source view corresponds to multiple data sources, including a first data source that defines the cross-source view and a second data source that the cross-source view accesses, in order to improve the efficiency of obtaining data from each data source, the computer device can first generate a view execution plan tree corresponding to the cross-source view according to the view creation instruction that indicates the creation of the cross-source view, and then obtain the data of each data source table in the view creation instruction from each data source according to the view execution plan tree.

[0111] Optionally, the execution plan tree may include a projection operation, a join operation, an enumeration operation, a filter operation, a limit operation, an aggregation operation, a scan operation, etc. An execution plan tree can be obtained by combining multiple operations.

[0112] Optionally, the view execution plan tree of the cross-source view includes data source subtrees corresponding to multiple data sources, and connection relationships are established between different data source subtrees through a Join operation.

[0113] In one possible implementation, after determining the second data source accessed by the cross-source view in the view creation instruction, the computer device can first generate an initial view execution plan tree for accessing the cross-source view according to the view creation instruction, and in order to improve data query efficiency, the computer device can optimize the initial view execution plan tree for the cross-source view according to the execution optimization strategy, thereby obtaining a target view execution plan tree, wherein the execution cost of the target view execution plan tree is no greater than the execution cost of the initial view execution plan tree, and then the computer device can obtain data of each data source table in the view creation instruction from each second data source according to the target view execution plan tree.

[0114] For example, taking the DDL sample of the cross-source view Q1 as shown below, the initial view execution plan tree corresponding to the cross-source view can be as follows: Figure 3 shown.

[0115] CREATE VIEW hive_ds.v1 AS / / Define a cross-source view in the hive_ds data source

[0116] SELECT t1.cnt,t3.col2 FROM mysql_ds1.t1 / / Query the data in the .cnt column of row t1 and the col2 column of row t3 from the data source table mysql_ds1.t1

[0117] JOIN(SELECT col1,count(*)AS cnt FROM pg_ds.t2 / / Select col1 column data and a count expression count(*) from the data source table pg_ds.t2

[0118] WHERE col3>0 GROUP BY col1 LIMIT 100) t2 / / Filter out the records where col3>0, and group the query results by the value of col1, while limiting the number of rows to 100

[0119] ON t1.col1 = t2.col1 JOIN mysql_ds2.t3 ON t1.col1 = t3.col1 / / Joins tables t1 and t2 based on the equality between t1.col1 and t2.col1, and joins tables t1 and t3 based on the equality between t1.col1 and t3.col1

[0120] WHERE t1.col4 between 1 AND 10 AND t3.col5 < 1000 / / Select all rows in the t1 table where the value of the col4 column is between 1 and 10, and all rows in the t3 table where the value of the col5 column is less than 1000

[0121] Optionally, the computer device can optimize the initial view execution plan tree across the source view based on at least one execution optimization strategy of RBO (Rule-Based Optimization) / CBO (Cost-Based Optimization) / HBO (Heuristic-Based Optimization) to obtain the target view execution plan tree.

[0122] Regarding the optimization process of the execution plan tree, in one possible embodiment, when there are at least two second data sources with an equivalent relationship in the second data sources accessed by the cross-source view, in order to reduce the number of data processing times for data in different second data sources, the computer device can merge the data source subtrees corresponding to the at least two second data sources with an equivalent relationship in the initial view execution plan tree, thereby obtaining a candidate view execution plan tree corresponding to the cross-source view. Furthermore, the computer device estimates the cost of the candidate view execution plan tree and the initial view execution plan tree, respectively obtaining a first execution cost corresponding to the initial view execution plan tree and a second execution cost corresponding to the candidate view execution plan tree, and by comparing the first execution cost and the second execution cost, the target view execution plan tree can be determined from the initial view execution plan tree and the candidate view execution plan tree.

[0123] Optionally, the computer device may determine the view execution plan tree with the lowest execution cost as the target view execution plan tree to improve data query efficiency. If the first execution cost is greater than the second execution cost, the computer device determines the candidate view execution plan tree as the target view execution plan tree. If the first execution cost is not greater than the second execution cost, the computer device determines the initial view execution plan tree as the target view execution plan tree.

[0124] The equivalence relationship between the second data sources in the second data source accessed by the cross-source view is determined based on the data source type and data source definition of the second data source. Optionally, if the two data sources are of the same type and the same database host name (domain name or IP) / port and the same (or equal) JDBC username / password are specified in the data source definition, it can be determined that the two data sources have an equivalence relationship, so that the computer device can treat the two data sources with an equivalence relationship as the same data source for data processing.

[0125] Regarding the cost estimation method, in one possible implementation, a computer device may perform cost estimation on the initial view execution plan tree and the candidate view execution plan tree corresponding to the cross-source view through a CBO optimizer to obtain a first execution cost corresponding to the initial view execution plan tree and a second execution cost corresponding to the candidate view execution plan tree.

[0126] Indicative, such as Figure 4 As shown, in the data sources accessed by the cross-source view, mysql_ds1 and mysql_ds2 are equivalent, corresponding to two different databases, db1 and db3, in the same MySQL Server instance. Therefore, to improve data query efficiency, the computer device can adjust the connection relationships between the data source subtrees in the initial view execution plan tree, merge mysql_ds1 and mysql_ds2, and push down the more efficient Join operator to a single data source JDBC execution, using the union of the parameters of the two equivalent sources. Furthermore, by reordering the joins in the initial view execution plan tree of the cross-source view, transforming t1 join t2 join t3 into t1 join t3 join t2, this can save network transmission overhead and reduce the execution cost of the view execution plan tree.

[0127] Furthermore, after determining the target view execution plan tree corresponding to the cross-source view, the computer device can obtain data from each data source table from each second data source according to the target view execution plan tree. In one possible embodiment, considering that when creating a cross-source view based on a view creation instruction, the computer device can process the view definition and the data source definition in a unified format so that the cross-source view can support queries from different computing engines, when executing the target view execution plan tree, the computer device can first obtain the engine properties of different types of computing engines, and then determine the target computing engine for querying the cross-source view based on the engine properties of different types of computing engines and the target view execution plan tree, and compared with other computing engines, the execution cost consumed by the target computing engine when executing the target view execution plan tree corresponding to the cross-source view is the lowest.

[0128] The engine attributes represent the data query methods and capabilities of the computing engine. Optionally, the engine attributes may include the data source table types, SQL operator and function types, source table data volume, etc. supported by the computing engine, which are not specifically limited in this embodiment of the application.

[0129] In some embodiments, after determining the target computing engine used to query the data source, the computer device can perform syntax adaptation processing on the target view execution plan tree based on the engine properties of the target computing engine to obtain a target view execution plan tree adapted to the target computing engine, and then use the target computing engine to execute the target view execution plan tree, that is, the data of each data source table in the view creation instruction can be obtained from each second data source.

[0130] Optionally, the syntax adaptation process may include rewriting equivalent function names or input parameters, changing implicit data type conversion to explicit cast, changing Spark lateral view explode to Presto / StarRocks cross join unnest, etc., which is not limited in the embodiments of the present application.

[0131] In the above embodiment, an initial view execution plan tree of a cross-source view is generated according to a view creation instruction, and the initial view execution plan tree is optimized based on an execution optimization strategy to obtain a candidate view execution plan tree. Then, based on the cost estimation result, a target execution plan tree is determined for querying the data source corresponding to the cross-source view, thereby reducing the network transmission overhead during the data query process and improving the data query efficiency.

[0132] In addition, when using a computing engine to execute a view execution plan tree, the target computing engine that is adapted to the current target view execution plan tree is determined based on the engine properties of different types of computing engines, and the syntax of the target view execution plan tree is adapted to execute the target view execution plan tree, thereby improving the query efficiency of the computing engine accessing the data source.

[0133] In some embodiments, the optimization process of the execution plan tree can be performed not only in the process of querying the data source accessed by the cross-source view, but also in the process of executing the data processing task containing the cross-source view. In one possible embodiment, when receiving the data processing task containing the cross-source view, the computer device can first generate an initial task execution plan tree for the data processing task, and then, when the cross-source view in the initial task execution plan tree has a connected data source, and the connected data source is equivalent to the second data source accessed by the cross-source view, the computer device can adjust the connection relationship between the cross-source view and the connected data source, merge the data source subtree of the connected data source with the data source subtree of the second data source, obtain the candidate task execution plan tree for the data processing task, and determine the target task execution plan tree from the initial task execution plan tree and the candidate task execution plan tree based on the cost estimation result.

[0134] Optionally, the connected data source is a data source that has a join relationship with the cross-source view in the initial task execution plan tree. Furthermore, a cross-source view in the initial task execution plan tree can have a join relationship with one data source or with multiple data sources.

[0135] Regarding the cost estimation process of the task execution plan tree, in one possible implementation, the computer device can use the CBO optimizer to perform cost estimation on the initial task execution plan tree and the candidate task execution plan tree, and obtain the first execution cost corresponding to the initial task execution plan tree and the second execution cost corresponding to the candidate task execution plan tree, thereby determining the task execution plan tree with the minimum execution cost as the target task execution plan tree.

[0136] Indicative, such as Figure 5 As shown in the figure, in the initial task execution plan tree corresponding to the data processing task, the cross-source view hive_ds.v1 and the data source table mysql_ds3.t4 have a connection relationship, and the data sources mysql_ds3 and mysql_ds1 are equivalent data sources. Therefore, to improve task execution efficiency, the connection relationship between the cross-source view hive_ds.v1 and the data source table mysql_ds3.t4 can be pushed down to the view execution plan tree corresponding to the cross-source view, that is, t1 join t4 is pushed down to the equivalent source execution.

[0137] In one possible implementation, when a data processing task containing a cross-source view is received, and the cross-source views in the initial task execution plan tree of the data processing task have connection relationships with multiple data sources respectively, if the data processing task is executed directly based on the initial task execution plan tree, it is necessary to process the connection relationship between the cross-source view and each data source separately, resulting in waste of computing resources and low data query efficiency. Therefore, in order to improve data query efficiency, the computer device can directly create a temporary view corresponding to the cross-source view, and connect each data source with the temporary view, thereby generating a target task execution plan tree for the data processing task, in which the temporary view is connected to multiple data sources.

[0138] Optionally, the computer device may create a temporary view corresponding to the cross-source view based on the CTE (Common Table Expression) SQL optimization principle, that is, directly "solidify" and map the cross-source view into a temporary view of the computing engine.

[0139] Indicative, such as Figure 6 As shown, when a data processing task containing a cross-source view is received, and the cross-source view hive_ds.v1 in the initial task execution plan tree of the data processing task has a connection relationship with the data source table ck_ds1.t1, the data source table ck_ds2.t2, and the data source table ck_ds3.t3, respectively, the computer device can create a temporary view hive_ds.v1' corresponding to the cross-source view hive_ds.v1, thereby connecting each data source with the temporary view to generate a target task execution plan tree for the data processing task.

[0140] In the above embodiment, when executing a data processing task involving a cross-source view, if a data source having a connection relationship with the cross-source view in the task execution plan tree has an equivalence relationship with the data source corresponding to the cross-source view, the task execution plan tree is optimized by merging the data source subtrees corresponding to the data sources having the equivalence relationship. This allows the data processing task to be executed based on the optimized task execution plan tree, effectively saving network transmission overhead and reducing the execution cost of the data processing task.

[0141] In addition, for the case where the cross-source views in the initial task execution plan tree of the data processing task have connection relationships with multiple data sources, by creating temporary views corresponding to the cross-source views and using the temporary views to connect with each data source, the task execution plan tree is simplified and the task execution efficiency is improved.

[0142] In combination with the above embodiments, the following describes the process of executing a data processing task involving cross-source views. Figure 7As shown, it shows a flowchart of executing a data processing task including a cross-source view provided by an exemplary embodiment of the present application.

[0143] When a data processing task (user SQL) containing a cross-source view is received, the computer device parses the data processing task through an SQL parser / validator to obtain the view name of the cross-source view, and obtains the view definition of the cross-source view from the cross-source view database based on the view name, and queries the view permissions of the cross-source view and the source table permissions of the data source table accessed by the cross-source view from the cross-source view database based on the view name. When the view permissions returned by the permission API in the cross-source view database are obtained, and the view permissions indicate that cross-source views are allowed to be queried, a task execution plan tree for the data processing task is generated based on the view execution plan tree corresponding to the cross-source view.

[0144] Furthermore, the RBO optimizer is used to optimize the task execution plan tree of the data processing task, wherein the optimization process includes expanding the view execution plan tree of the cross-source view, creating a temporary view of the cross-source view through CTE solidification, judging the data source equivalence relationship existing in the task execution plan tree, merging and calculating the data source subtrees corresponding to the data sources with the equivalence relationship, and in this process, the CBO optimizer is used to estimate the cost of the generated candidate task execution plan tree, thereby obtaining a target task execution plan tree with the lowest cost.

[0145] Ultimately, the computer device determines the computing engine that is adapted to the target task execution plan tree based on the engine properties of different computing engines, and performs syntax / function adaptation on the target task execution plan tree to obtain the adapted target task execution plan tree, thereby executing the target task execution plan tree through the computing engine to achieve access to various data sources and complete the data processing task.

[0146] Please refer to Figure 8 , which shows a structural diagram of a cross-source and cross-engine integrated view framework provided by an exemplary embodiment of the present application.

[0147] Optionally, a cross-source and cross-engine integrated view framework is applied to the big data computing platform. The cross-source and cross-engine integrated view framework can include four parts, namely SQL parsing / validation, SQL optimizer, unified metadata database (cross-source view database) and view data cache.

[0148] When a data processing task (user SQL) containing a cross-source view is received through a unified SQL interface, the computer device first uses an SQL parser / validator to parse the data processing task to obtain the view name of the cross-source view, and based on the view name, obtains the view definition of the cross-source view and the data source definition of each data source accessed by the cross-source view from the unified metadata database through a data access interface (cross-source view read and write API), and queries the view permissions of the cross-source view and the source table permissions of the data source table accessed by the cross-source view from the cross-source view database according to the view name. When the view permissions returned by the permission API in the cross-source view database are obtained, and the view permissions indicate that the query on the cross-source view is allowed, a task execution plan tree for the data processing task is generated based on the view execution plan tree corresponding to the cross-source view.

[0149] At the same time, the computer equipment uses the RBO optimizer to optimize the task execution plan tree of the data processing task, including view expansion, view authentication, engine selection, and syntax / function rewriting, and uses the CBO optimizer to estimate the cost of the generated candidate task execution plan tree, including merging and pushing down data source subtrees based on data source equivalence sets, creating temporary views based on common table expressions (CTE), etc.

[0150] In addition, after obtaining the view data of the cross-source view from the unified metadata repository for the first time, the computer device creates a cache object corresponding to the view data and caches the cache object of the view data using three levels of caching, including query cache, session cache, and global cache, to improve the efficiency of obtaining view data. At the same time, a TTL refresh mechanism is used to perform real-time timeout and invalidation detection on the cache object to ensure the validity of the view data.

[0151] Please refer to Figure 9 , which shows a structural block diagram of a data table view management device provided by an exemplary embodiment of the present application, the device comprising:

[0152] An instruction receiving module 901 is configured to receive a view creation instruction, wherein the view creation instruction is configured to create a view in a first data source;

[0153] a data source determining module 902, configured to determine, when the view creation instruction indicates that the view to be created is a cross-source view, a second data source accessed by the cross-source view in the view creation instruction, where the cross-source view is a data table created based on data from at least one second data source;

[0154] A first data acquisition module 903 is configured to acquire data of each data source table in the view creation instruction from each second data source;

[0155] The view generation module 904 is configured to generate the cross-source view according to the data in the data source table.

[0156] Optionally, the device further includes:

[0157] a definition generation module, configured to generate a view definition of the cross-source view based on the view creation instruction;

[0158] a format processing module, configured to perform unified formatting on the view definition of the cross-source view and the data source definition of the second data source accessed by the cross-source view, so that the cross-source view after unified formatting supports queries from different computing engines;

[0159] A storage module is used to store the view definition and the data source definition processed in a unified format into a cross-source view database, and the cross-source view database is used to store view data of the cross-source view.

[0160] Optionally, the view definition includes a view name, and the apparatus further includes:

[0161] a first task parsing module, configured to, upon receiving a data processing task containing the cross-source view, parse the data processing task to obtain a view name of the cross-source view;

[0162] The second data acquisition module is configured to acquire the view data of the cross-source view from the cross-source view database based on the view name.

[0163] Optionally, the second data acquisition module is used to:

[0164] In a case where the data processing task includes a plurality of data processing subtasks for accessing the cross-source view, when accessing the cross-source view for the first time, acquiring the view data of the cross-source view from the cross-source view database based on the view name;

[0165] The device further comprises:

[0166] A first object creation module, configured to create a cache object corresponding to the view data after acquiring the view data;

[0167] A third data acquisition module is configured to acquire the view data from the cache object when accessing the cross-source view for the i-th time, where i is greater than 1.

[0168] Optionally, the second data acquisition module is used to:

[0169] When accessing the cross-source view for the first time in a session processing task, obtaining the view data of the cross-source view from the cross-source view database based on the view name, the session processing task including a plurality of data processing tasks; or

[0170] When accessing the cross-source view for the first time in a global processing task, acquiring the view data of the cross-source view from the cross-source view database based on the view name, the global processing task including a plurality of session processing tasks;

[0171] The device further comprises:

[0172] A second object creation module is used to create a cache object corresponding to the view data after obtaining the view data;

[0173] The third data acquisition module is configured to acquire the view data from the cache object when accessing the cross-source view for the i-th time, where i is greater than 1.

[0174] Optionally, the third data acquisition module is used to:

[0175] When accessing the cross-source view for the i-th time, obtaining the latest view definition of the cross-source view from the cross-source view database based on the view name; if the latest view definition matches the historical view definition in the cache object, obtaining the view data from the cache object; or

[0176] When accessing the cross-source view for the i-th time, determining the data validity period of the view data in the cache object; and obtaining the view data from the cache object if the data validity period has not been reached.

[0177] Optionally, the view creation instruction includes view permissions of the cross-source view, and the view permissions include at least one of view query permissions, view modification permissions, and view deletion permissions;

[0178] The device further comprises:

[0179] a permission determination module, configured to determine the view permission of the cross-source view based on the view creation instruction;

[0180] The permission storage module is used to store the view permission of the cross-source view in the cross-source view database.

[0181] Optionally, the device further includes:

[0182] a second task parsing module, configured to, upon receiving a data processing task containing the cross-source view, parse the data processing task to obtain a view name of the cross-source view;

[0183] A permission acquisition module, configured to acquire the view permission of the cross-source view from the cross-source view database based on the view name;

[0184] A fourth data acquisition module is configured to acquire view data of the cross-source view from the cross-source view database if the view permission indicates that the cross-source view is allowed to be queried.

[0185] Optionally, before acquiring the data of each data source table in the view creation instruction from each second data source, the apparatus further includes:

[0186] A first plan tree generating module, configured to generate an initial view execution plan tree for accessing the cross-source view based on the view creation instruction;

[0187] a plan tree optimization module, configured to optimize the initial view execution plan tree according to an execution optimization strategy to obtain a target view execution plan tree, wherein an execution cost of the target view execution plan tree is not greater than an execution cost of the initial view execution plan tree;

[0188] The first data acquisition module 903 is configured to:

[0189] Based on the target view execution plan tree, data of each data source table in the view creation instruction is obtained from each second data source.

[0190] Optionally, the plan tree optimization module is used to:

[0191] If there are at least two second data sources with an equivalence relationship among the second data sources accessed by the cross-source view, merging the data source subtrees corresponding to the at least two second data sources with an equivalence relationship in the initial view execution plan tree to obtain a candidate view execution plan tree, wherein the equivalence relationship is determined based on the data source type and data source definition of the second data source;

[0192] Performing cost estimation on the candidate view execution plan tree and the initial view execution plan tree to obtain a first execution cost corresponding to the initial view execution plan tree and a second execution cost corresponding to the candidate view execution plan tree;

[0193] In a case where the first execution cost is greater than the second execution cost, determining the candidate view execution plan tree as the target view execution plan tree;

[0194] When the first execution cost is not greater than the second execution cost, the initial view execution plan tree is determined as the target view execution plan tree.

[0195] Optionally, the first data acquisition module 903 is further configured to:

[0196] Obtaining engine properties of different types of computing engines, wherein the engine properties represent data query methods and capabilities of the computing engines;

[0197] determining a target computing engine for querying the cross-source view based on the engine properties of the different types of computing engines and the target view execution plan tree;

[0198] Based on the engine properties of the target computing engine, performing syntax adaptation processing on the target view execution plan tree to obtain the target view execution plan tree adapted to the target computing engine;

[0199] The target view execution plan tree is executed by using the target computing engine to obtain data of each data source table in the view creation instruction from each second data source.

[0200] Optionally, the device further includes:

[0201] A second plan tree generating module is configured to generate an initial task execution plan tree for the data processing task when receiving the data processing task including the cross-source view;

[0202] a subtree merging module, configured to, when the connected data source of the cross-source view is equivalent to the second data source accessed by the cross-source view, merge the data source subtree of the connected data source with the data source subtree of the second data source to obtain a candidate task execution plan tree for the data processing task, wherein the connected data source is a data source having a connection relationship with the cross-source view in the initial task execution plan tree;

[0203] The plan tree determination module is used to determine a target task execution plan tree from the initial task execution plan tree and the candidate task execution plan trees according to a cost estimation result.

[0204] Optionally, the device further includes:

[0205] a view creation module, configured to, upon receiving a data processing task including the cross-source view and in a case where the cross-source view in an initial task execution plan tree of the data processing task has connection relationships with multiple data sources, create a temporary view corresponding to the cross-source view;

[0206] The third plan tree generating module is configured to generate a target task execution plan tree for the data processing task based on the temporary view, wherein the temporary view in the target task execution plan tree is connected to the multiple data sources.

[0207] Optionally, the device further includes:

[0208] A table name acquisition module is used to acquire the table name of the data source table contained in the view creation instruction after receiving the view creation instruction;

[0209] The view determination module is configured to determine, when the table name of the data source table indicates different data sources, that the view to be created indicated by the view creation instruction is the cross-source view.

[0210] To summarize, in an embodiment of the present application, when the received view creation instruction instructs to create a cross-source view in the first data source, by determining the second data source accessed by the cross-source view, the data of each data source table in the view creation instruction is directly obtained from each second data source. That is, a cross-source view can be generated based on the data in the data source table without the need to manually adapt and adjust the syntax of the data in different data sources. That is, joint analysis of data from different data sources can be achieved, which reduces the threshold for data queries across data sources and improves data query and management efficiency.

[0211] It should be noted that the apparatus provided in the above embodiments is merely exemplified by the division of the above functional modules. In actual applications, the above functions can be distributed among different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The implementation process is detailed in the method embodiments and will not be repeated here.

[0212] Please refer to Figure 10 , which shows a schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application. Specifically, the computer device 1000 includes a central processing unit (CPU) 1001, a system memory 1004 including a random access memory 1002 and a read-only memory 1003, and a system bus 1005 connecting the system memory 1004 and the central processing unit 1001. The computer device 1000 may also include a basic input / output system (I / O system) 1006 that helps transmit information between various components within the computer, and a large-capacity storage device 1007 for storing an operating system 1013, application programs 1014, and other program modules 1015.

[0213] In some embodiments, the basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009, such as a mouse or keyboard, for user input. Both the display 1008 and the input device 1009 are connected to the central processing unit 1001 via an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may also include an input / output controller 1010 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, printer, or other types of output devices.

[0214] The mass storage device 1007 is connected to the central processing unit 1001 via a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer-readable media provide non-volatile storage for the computer device 1000. In other words, the mass storage device 1007 may include a computer-readable medium (not shown) such as a hard disk or drive.

[0215] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, tape cassette, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage medium is not limited to the above-mentioned ones. The above-mentioned system memory 1004 and mass storage device 1007 can be collectively referred to as memory.

[0216] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1001. The one or more programs contain instructions for implementing the above-mentioned method. The central processing unit 1001 executes the one or more programs to implement the data table view management method provided by the above-mentioned various method embodiments.

[0217] According to various embodiments of the present application, the computer device 1000 may also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 1000 may be connected to the network 1011 via the network interface unit 1012 connected to the system bus 1005, or the network interface unit 1012 may be used to connect to other types of networks or remote computer systems (not shown).

[0218] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction. The at least one instruction is loaded and executed by a processor to implement the data table view management method described in the above embodiment.

[0219] Optionally, the computer-readable storage medium may include: ROM, RAM, solid state drives (SSDs) or optical disks, etc. Among them, RAM may include resistance random access memory (ReRAM) and dynamic random access memory (DRAM).

[0220] An embodiment of the present application provides a computer program product, comprising at least one instruction stored in a computer-readable storage medium. A processor of a computer device reads the at least one instruction from the computer-readable storage medium and executes the at least one instruction, causing the computer device to perform the data table view management method described in the above embodiment.

[0221] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0222] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A data table view management method, characterized in that: The method comprises: receiving a view creation instruction, where the view creation instruction is used to create a view in a first data source; In a case where the view creation instruction indicates that the created view is a cross-source view, determining a second data source accessed by the cross-source view in the view creation instruction, where the cross-source view is a data table created based on data from at least one second data source; Acquire data from each second data source table in the view creation instruction; The cross-source view is generated according to the data in the data source table.

2. The method according to claim 1, characterized in that The method further comprises: generating a view definition of the cross-source view based on the view creation instruction; Performing unified format processing on the view definition of the cross-source view and the data source definition of the second data source accessed by the cross-source view, so that the cross-source view after unified format processing supports queries from different computing engines; The view definition and the data source definition processed in a unified format are stored in a cross-source view database, and the cross-source view database is used to store view data of the cross-source view.

3. The method according to claim 2, characterized in that The view definition includes a view name, and the method further includes: When receiving a data processing task including the cross-source view, parsing the data processing task to obtain a view name of the cross-source view; Based on the view name, view data of the cross-source view is acquired from the cross-source view database.

4. The method according to claim 3, characterized in that The acquiring, based on the view name, view data of the cross-source view from the cross-source view database includes: In a case where the data processing task includes a plurality of data processing subtasks for accessing the cross-source view, when accessing the cross-source view for the first time, acquiring the view data of the cross-source view from the cross-source view database based on the view name; The method further comprises: After obtaining the view data, creating a cache object corresponding to the view data; When accessing the cross-origin view for the i-th time, the view data is obtained from the cache object, where i is greater than 1.

5. The method according to claim 3, characterized in that The acquiring, based on the view name, view data of the cross-source view from the cross-source view database includes: When accessing the cross-source view for the first time in a session processing task, obtaining the view data of the cross-source view from the cross-source view database based on the view name, the session processing task including a plurality of data processing tasks; or When accessing the cross-source view for the first time in a global processing task, acquiring the view data of the cross-source view from the cross-source view database based on the view name, the global processing task including a plurality of session processing tasks; The method further comprises: After obtaining the view data, creating a cache object corresponding to the view data; When accessing the cross-origin view for the i-th time, the view data is obtained from the cache object, where i is greater than 1.

6. The method according to claim 5, characterized in that The acquiring the view data from the cache object when accessing the cross-origin view for the i-th time includes: When accessing the cross-source view for the i-th time, obtaining the latest view definition of the cross-source view from the cross-source view database based on the view name; if the latest view definition matches the historical view definition in the cache object, obtaining the view data from the cache object; or When accessing the cross-source view for the i-th time, determining the data validity period of the view data in the cache object; and obtaining the view data from the cache object if the data validity period has not been reached.

7. The method according to claim 2, characterized in that The view creation instruction includes the view permission of the cross-source view, and the view permission includes at least one of a view query permission, a view modification permission, and a view deletion permission; The method further comprises: Determining the view permission of the cross-source view based on the view creation instruction; The view permission of the cross-source view is stored in the cross-source view database.

8. The method according to claim 7, characterized in that The method further comprises: When receiving a data processing task including the cross-source view, parsing the data processing task to obtain a view name of the cross-source view; Based on the view name, obtaining the view permission of the cross-source view from the cross-source view database; In a case where the view permission indicates that querying the cross-source view is allowed, the view data of the cross-source view is acquired from the cross-source view database.

9. The method according to claim 1, characterized in that Before acquiring data of each data source table in the view creation instruction from each second data source, the method further includes: generating an initial view execution plan tree for accessing the cross-source view based on the view creation instruction; Optimizing the initial view execution plan tree according to the execution optimization strategy to obtain a target view execution plan tree, wherein the execution cost of the target view execution plan tree is not greater than the execution cost of the initial view execution plan tree; The step of obtaining data from each data source table in the view creation instruction from each second data source includes: Based on the target view execution plan tree, data of each data source table in the view creation instruction is obtained from each second data source.

10. The method according to claim 9, characterized in that Optimizing the initial view execution plan tree according to the execution optimization strategy to obtain a target view execution plan tree includes: If there are at least two second data sources with an equivalence relationship among the second data sources accessed by the cross-source view, merging the data source subtrees corresponding to the at least two second data sources with an equivalence relationship in the initial view execution plan tree to obtain a candidate view execution plan tree, wherein the equivalence relationship is determined based on the data source type and data source definition of the second data source; Performing cost estimation on the candidate view execution plan tree and the initial view execution plan tree to obtain a first execution cost corresponding to the initial view execution plan tree and a second execution cost corresponding to the candidate view execution plan tree; In a case where the first execution cost is greater than the second execution cost, determining the candidate view execution plan tree as the target view execution plan tree; When the first execution cost is not greater than the second execution cost, the initial view execution plan tree is determined as the target view execution plan tree.

11. The method according to claim 9, characterized in that The acquiring, from each second data source, data of each data source table in the view creation instruction based on the target view execution plan tree includes: Obtaining engine properties of different types of computing engines, wherein the engine properties represent data query methods and capabilities of the computing engines; determining a target computing engine for querying the cross-source view based on the engine properties of the different types of computing engines and the target view execution plan tree; Based on the engine properties of the target computing engine, performing syntax adaptation processing on the target view execution plan tree to obtain the target view execution plan tree adapted to the target computing engine; The target view execution plan tree is executed by using the target computing engine to obtain data of each data source table in the view creation instruction from each second data source.

12. The method according to any one of claims 9 to 11, characterized in that: The method further comprises: Upon receiving the data processing task including the cross-source view, generating an initial task execution plan tree for the data processing task; In a case where the connected data source of the cross-source view is equivalent to the second data source accessed by the cross-source view, merging the data source subtree of the connected data source with the data source subtree of the second data source to obtain a candidate task execution plan tree for the data processing task, where the connected data source is a data source having a connection relationship with the cross-source view in the initial task execution plan tree; According to the cost estimation result, a target task execution plan tree is determined from the initial task execution plan tree and the candidate task execution plan trees.

13. The method according to any one of claims 9 to 11, characterized in that: The method further comprises: Upon receiving a data processing task including the cross-source view, and in an initial task execution plan tree of the data processing task, the cross-source view is respectively connected to multiple data sources, creating a temporary view corresponding to the cross-source view; Based on the temporary view, a target task execution plan tree of the data processing task is generated, wherein the temporary view in the target task execution plan tree is connected to the multiple data sources.

14. The method according to claim 1, wherein The method further comprises: After receiving the view creation instruction, obtaining the table name of the data source table included in the view creation instruction; In a case where the table name of the data source table indicates different data sources, it is determined that the view to be created by the view creation instruction is the cross-source view.

15. A data table view management device, characterized in that: The device comprises: An instruction receiving module, configured to receive a view creation instruction, wherein the view creation instruction is used to create a view in the first data source; a data source determining module, configured to determine, when the view creation instruction indicates that the view to be created is a cross-source view, a second data source accessed by the cross-source view in the view creation instruction, wherein the cross-source view is a data table created based on data from at least one second data source; A data acquisition module, configured to acquire data of each data source table in the view creation instruction from each second data source; A view generation module is used to generate the cross-source view according to the data in the data source table.

16. A computer device, characterized in that: The computer device includes a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement the data table view management method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, and the at least one instruction is used to be executed by a processor to implement the data table view management method according to any one of claims 1 to 14.

18. A computer program product, characterized in that The computer program product includes at least one instruction, which is stored in a computer-readable storage medium; the processor of the computer device reads the at least one instruction from the computer-readable storage medium, and the processor executes the at least one instruction, so that the computer device implements the data table view management method as described in any one of claims 1 to 14.

Images