Multi-source remote data query method, apparatus, device and computer-readable medium

Abstract

This disclosure provides embodiments of a multi-source remote data query method, apparatus, device, and computer-readable medium. One specific implementation of the method includes: in response to receiving multi-source data query information containing user equipment configuration information, generating various trimmed multi-source list data based on the multi-source data query information; obtaining a distributed query instruction set; and for each trimmed multi-source list data, performing the following steps: performing predicate pushdown processing on the trimmed multi-source list data to obtain a multi-source data pushdown instruction information set; generating multi-structured rewritten query language data; sending each multi-structured query language data to a preset execution node to obtain a remote data trimming query result; generating a data query result based on the non-pushdown instruction information set in the multi-source data pushdown instruction information set and at least one obtained remote data trimming query result; and returning the data query result to a preset user terminal. This implementation reduces the waste of transmission resources.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to multi-source remote data query methods, apparatus, devices, and computer-readable media. Background Technology

[0002] As enterprises deepen their digital transformation, the scale of data storage is growing exponentially. A large amount of enterprise data is scattered across multiple heterogeneous data sources (such as relational databases, big data warehouses, real-time log systems, and cloud storage). Business systems often need to cross-domain correlate these scattered data, frequently initiating multi-source remote queries to obtain a comprehensive view of the information. Multi-source remote data query refers to a technology that dynamically retrieves structured or semi-structured information relevant to the current business scenario from multiple distributed data sources (such as cloud databases, edge nodes, and third-party platforms). Currently, the common approach to querying multi-source remote data is as follows: first, query the database to obtain a list of matching identifiers; then, iterate through each identifier in the list and sequentially initiate query requests to multiple other databases to obtain corresponding detailed information; finally, integrate and combine the scattered data to form the final complete query result.

[0003] However, when using the above method to query multi-source remote data, the following technical problems often arise: The process involves querying a database to obtain a list of matching identifiers. Each identifier in this list is then iterated through, and queries are sent to multiple other databases to retrieve corresponding detailed information. Finally, the scattered data is integrated and combined to form the final complete query result. However, different data sources use different SQL execution languages, which may prevent some query information from being directly executed. Furthermore, when sending queries to multiple databases for detailed information, entire rows of data are retrieved, and some unnecessary column information is also imported (for example, if the target query is for the cumulative performance of "10", the data row corresponding to "10" in the database may contain unnecessary information such as "contact information" and "join time" in addition to performance, which will also be imported). This results in data redundancy and a large amount of data being transmitted, leading to some information redundancy and wasted transmission resources.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not form prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure provide multi-source remote data query methods, apparatuses, devices, and computer-readable media to address one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a multi-source remote data query method, the method comprising: responding to receiving multi-source data query information containing user equipment configuration information sent by a preset user terminal; generating various trimmed multi-source list data based on the multi-source data query information; obtaining a distributed query instruction set corresponding to the user equipment configuration information; and performing the following steps on each trimmed multi-source list data in the various trimmed multi-source list data: performing predicate pushdown processing on the trimmed multi-source list data based on the distributed query instruction set to obtain a multi-source data pushdown instruction information set, wherein the multi-source data pushdown instruction information set includes a pushdown instruction information set and a non-pushdown instruction information set. The process involves: 1) Rewriting the multi-source list data using a query language based on the push-down instruction set included in the multi-source data push-down instruction set, resulting in multi-structured rewritten query language data. 2) Sending each multi-structured rewritten query language data to a preset execution node for data query trimming, resulting in remote data trimming query results. 3) Generating a data query result based on the non-push-down instruction set included in the multi-source data push-down instruction set and at least one remote data trimming query result. 4) Returning the data query result to the preset user terminal.

[0008] Secondly, some embodiments of this disclosure provide a multi-source remote data query apparatus, comprising: a first generation unit configured to, in response to receiving multi-source data query information containing user equipment configuration information sent by a preset user terminal, generate various trimmed multi-source list data based on the multi-source data query information; an acquisition unit configured to acquire a distributed query instruction set corresponding to the user equipment configuration information; and an execution unit configured to, for each trimmed multi-source list data in the various trimmed multi-source list data, perform the following steps: based on the distributed query instruction set, perform predicate pushdown processing on the trimmed multi-source list data to obtain a multi-source data pushdown instruction information set, wherein the multi-source data pushdown instruction information set includes a pushdown instruction information set and a non-pushdown instruction information set. Based on the pushable instruction information set included in the multi-source data pushdown instruction information set, the above-mentioned trimmed multi-source list data is rewritten using a query language to obtain multi-structured rewritten query language data; the sending unit is configured to send each of the obtained multi-structured rewritten query language data to a preset execution node, so that the preset execution node can perform data query trimming processing on the multi-structured rewritten query language data to obtain remote data trimming query results; the second generation unit is configured to generate data query results based on the non-pushable instruction information set included in the multi-source data pushdown instruction information set and at least one obtained remote data trimming query result; the return unit is configured to return the data query results to the preset user terminal.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0011] The above-described embodiments of this disclosure have the following beneficial effects: the multi-source remote data query method of some embodiments of this disclosure reduces the amount of data transmitted and the waste of transmission resources. Specifically, the reason for the large amount of data transmitted and the waste of transmission resources is that: by querying the database to obtain a list of identifiers that meet the conditions, traversing each identifier in the list and sequentially initiating query requests to multiple other databases to obtain the corresponding detailed information, and finally integrating and splicing the scattered data to form the final complete query result, the SQL execution languages ​​of different data sources are different when executing the query action, which may cause some query information to be unable to be executed directly. Moreover, when initiating query requests to multiple other databases to obtain the corresponding detailed information, the corresponding whole row of information will be obtained, and some unnecessary column information will also be imported along with it (for example, if the target query is the cumulative performance of "10", the data row corresponding to "10" in the database may contain unnecessary information such as "contact information" and "join time" in addition to performance, which will also be imported and returned, resulting in data redundancy and a large amount of transmitted data, with some information redundancy, causing waste of transmission resources. Based on this, the multi-source remote data query method of some embodiments of this disclosure firstly, in response to receiving multi-source data query information containing user equipment configuration information sent by a preset user terminal, generates various trimmed multi-source list data based on the multi-source data query information. This allows for the decomposition and analysis of the user's multi-source data query information to obtain various trimmed multi-source list data representing each data table, the target query data column, and the filtering data conditions for different data tables. Next, a distributed query instruction set corresponding to the aforementioned user equipment configuration information is obtained. This yields a distributed query instruction set containing locally executable instruction functions and remotely executable instruction functions of the execution subject. Then, for each trimmed multi-source list data, the following steps are performed: First, based on the aforementioned distributed query instruction set, predicate pushdown processing is performed on the trimmed multi-source list data to obtain a multi-source data pushdown instruction information set, wherein the multi-source data pushdown instruction information set includes a pushdown instruction information set and a non-pushdown instruction information set. Therefore, the various filtering conditions representing the user's desired data acquisition in the trimmed multi-source list data can be divided into instructions that can be executed at a preset execution node (push-down instruction information set) and instructions that must be executed locally (non-push-down instruction information set), resulting in a multi-source data push-down instruction information set, which is used to subsequently generate multi-structured rewritten query language data. Next, based on the push-down instruction information set included in the multi-source data push-down instruction information set, the trimmed multi-source list data is rewritten using query language to obtain multi-structured rewritten query language data.Therefore, multi-source list data from different data sources can be rewritten using a single execution language, avoiding the possibility that different SQL execution languages ​​from different data sources might prevent some queries from being executed directly. This data is then sent to a preset execution node to retrieve the target data (satisfying the filtering conditions in the query information) from the target data list using multi-structured rewritten query language data. Each multi-structured rewritten query language data is then sent to the preset execution node for data pruning, resulting in remote data pruning query results. This yields remote data pruning query results returned by the preset execution node, containing only the data needed from each target query table, with useless data removed. Next, based on the non-pushdown instruction set included in the multi-source data pushdown instruction information set and at least one remote data pruning query result, a data query result is generated. This provides a data query result that meets the user's query requirements. Finally, the data query result is returned to the preset user terminal. Because the multi-source data query information sent by users is broken down, various tailored multi-source list data are obtained for different data tables and different target filtering conditions. At the same time, combined with the distributed query instruction set (representing whether some filtering conditions and operations are executed locally or can be executed by the execution end of the stored data table), the filtering conditions represented in each tailored multi-source list data are distinguished, and step instructions that can be operated on at the execution end of the stored data table (i.e., multi-structured rewritten query language data) are obtained. Furthermore, the fused perception and knowledge vector are sent to the preset execution node (i.e., the execution end of the stored data table). This allows the aforementioned preset execution nodes to filter target data using multi-structured rewritten query language data (for example, in the "10" data table, only the "ID" and "sign" columns with the "sign" value of "RED" are obtained; even if other data columns exist, they are not extracted or transmitted). The preset execution nodes return information containing only the target query content (i.e., remote data pruning query results), while data in the data table unrelated to the target query content is not transmitted or returned. Furthermore, based on the user device configuration information, the execution steps in the query data requirements are assessed to determine if local execution is suitable. This avoids directly loading large amounts of detailed information from multiple databases locally, preventing data redundancy and excessive data transmission, thus reducing wasted transmission resources. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of some embodiments of the multi-source remote data query method according to the present disclosure; Figure 2 These are schematic diagrams illustrating the structure of some embodiments of the multi-source remote data query apparatus according to this disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1 A flow 100 of some embodiments of the multi-source remote data query method according to the present disclosure is shown. The multi-source remote data query method includes the following steps: Step 101: In response to receiving multi-source data query information containing user equipment configuration information sent by a preset user terminal, generate various trimmed multi-source list data based on the multi-source data query information.

[0021] In some embodiments, the execution entity (e.g., a computing device) of the multi-source remote data query method can receive multi-source data query information containing user device configuration information sent by a preset user terminal via a wired or wireless connection. Based on the multi-source data query information, various trimmed multi-source list data are generated. The preset user terminal can be a hardware device for submitting multi-source data query information or an execution program (software page) equipped on the execution entity. For example, a user sends a query request to the execution entity via a computer or submits a query request using a software page, and the execution entity equipped with the software page performs data query processing in the background. The multi-source data query information can represent the information the user wants to query, including the content requirements of the data the user wants to query. For example, the multi-source data query information can be "SELECT t.user_name, t.salary; FROM remote_db.employees t; WHERE t.department='IT'", requiring data from the "t.user_name, t.salary" columns of the "remote_db.employees t" data table where "t.department" is "IT". The aforementioned user equipment configuration information may be data representing the relevant hardware parameters of the aforementioned preset user terminal, used to determine the local processing capabilities of the preset user terminal, such as storage capacity, cache size, etc.

[0022] In some optional implementations of certain embodiments, the aforementioned execution entity can generate a trimmed multi-source list data based on the aforementioned multi-source data query information through the following steps: The first step is to parse the multi-source data query information to obtain various Abstract Syntax Trees (ASTs). In practice, the execution entity uses an SQL (Structured Query Language) parser to parse the multi-source data query information, obtaining various ASTs. The SQL parser can be Calcite, Antlr4, or Vitess-sqlparse. The multi-source data query information may include requests to query multiple data tables; each AST, after decomposition, represents the query information for a single data table.

[0023] The second step is to perform metadata trimming on each of the above abstract syntax trees to obtain trimmed multi-source list data.

[0024] In some optional implementations of certain embodiments, the aforementioned execution entity may perform metadata trimming on the aforementioned abstract syntax trees through the following steps to obtain trimmed multi-source list data: The first step is to perform the following steps for each of the above abstract syntax trees: The first sub-step involves generating a set of referenced columns based on the abstract syntax tree (AST) described above. In practice, the execution entity can traverse each part of the AST, collect all appearing column names, and determine the set of referenced columns as the final set. For example, the AST could be "SELECT list: name, age; WHERE condition: age; ORDER BY list: name;", and the merged set of referenced columns would be: {name, age}.

[0025] The second sub-step involves obtaining the target metadata information based on the aforementioned multi-source data query information. This target metadata can represent the names of the specific data columns contained in the target query data table. For example, the query data table could be a "user" table, which includes five columns: (id, name, age, email, and address). In practice, the executing entity can retrieve the target metadata corresponding to the list names in the multi-source data query information from a pre-set storage device.

[0026] The third sub-step involves generating a trimmed multi-source list data based on the aforementioned target metadata information and the aforementioned set of reference columns. This trimmed multi-source list data can be a query instruction rewritten from the specific content of the queried data table. In practice, the executing entity can rewrite the aforementioned AST into a query instruction, using the column names of each reference list element in the set of reference columns and information from the target metadata information, as the trimmed multi-source list data. As an example, the above AST could be "SELECT id, name, age, email, address; FROM external_db.user; WHERE age > 18; ORDER BY name;". The set of referenced columns could be {*, name, age}, where "*" represents all columns in the query list. The multi-source data query information could be (id, name, age, email, address). Expanding the "*" columns and replacing "FROM" from "external_db.user" with the local execution result in a rewritten AST that could be "SELECT id, name, age, email, address; FROM user; WHERE age > 18; ORDER BY name;". This serves as a way to trim the multi-source list data.

[0027] It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other currently known or future wireless connection methods.

[0028] Step 102: Obtain the distributed query instruction set corresponding to the user equipment configuration information.

[0029] In some embodiments, the execution entity can obtain a distributed query instruction set corresponding to the user equipment configuration information. In practice, the execution entity can obtain distributed query instructions corresponding to the user equipment configuration information from a preset storage device. The preset storage device may include, but is not limited to, at least one of the following: a conventional hard disk drive (HDD), a SATA solid-state drive (SATA SSD), and an NVMe M.2 solid-state drive. The distributed query instruction set may be a pre-defined set of instructions that match the user equipment configuration information, consisting of instructions executed locally and instructions executed remotely.

[0030] Step 103: For each piece of clipped multi-source list data in each clipped multi-source list data set, perform the following steps: Step 1031: Based on the distributed query instruction set, perform predicate pushdown processing on the trimmed multi-source list data to obtain a multi-source data pushdown instruction information set.

[0031] In some embodiments, the aforementioned execution entity can perform predicate pushdown processing on the aforementioned trimmed multi-source list data based on the aforementioned distributed query instruction set to obtain a multi-source data pushdown instruction information set. This multi-source data pushdown instruction information set includes a pushable instruction information set and a non-pushable instruction information set. Each pushable instruction in the pushable instruction information set can be an instruction step that can be performed by a preset execution node from the filtering conditions represented in the trimmed multi-source list data (e.g., content that restricts target data such as "age > 25" or "salary < 8000"). For example, the pushable instruction can be "age > 25," and the preset execution node can filter data with "age > 25." This execution by the preset execution node constitutes a pushable instruction. Each non-pushable instruction in the non-pushable instruction information set can be an instruction step that cannot be performed by a preset execution node from the filtering conditions represented in the trimmed multi-source list data (e.g., calculating the final information volume). The preset execution node can be a pre-defined computing device used to perform data query transmission.

[0032] In some optional implementation manners of some embodiments, the above-mentioned execution subject may perform the following steps to perform predicate pushdown processing on the above-mentioned trimmed multi-source list data based on the above-mentioned distributed query instruction set, so as to obtain a multi-source data pushdown instruction information set: First step, generate a predicate data set based on the above-mentioned trimmed multi-source list data. In practice, the above-mentioned execution subject may split the conditions in the Where instruction in the above-mentioned trimmed multi-source list data into one predicate each, and obtain a data set composed of the split predicates as the predicate data set. As an example, the Where instruction in the above-mentioned trimmed multi-source list data may be "WHERE age>18 AND name LIKE 'Zhang%' AND city='Beijing'", and after splitting, three predicates are obtained as "age>18", "name LIKE 'Zhang%'", "city='Beijing'", and the obtained predicate data set may be {age>18, nameLIKE 'Zhang%', city='Beijing'}.

[0033] Second step, perform pushdown discrimination processing on the above-mentioned predicate data set based on the above-mentioned distributed query instruction set, so as to obtain a pushdownable instruction information set and a non-pushdownable instruction information set. In practice, for each predicate data in the above-mentioned predicate data set, the above-mentioned execution subject may perform the following steps: When the filtering condition to be executed in the above-mentioned predicate data is represented as local execution in the above-mentioned distributed query instruction set, the above-mentioned predicate data is determined as a non-pushdownable instruction. When the filtering condition to be executed in the above-mentioned predicate data is represented as remote execution in the above-mentioned distributed query instruction set, the above-mentioned predicate data is determined as a pushdownable instruction. Finally, the obtained pushdownable instructions are determined as the pushdownable instruction set, and the obtained non-pushdownable instructions are determined as the non-pushdownable instruction set.

[0034] Third step, determine the above-mentioned pushdownable instruction information set and the above-mentioned non-pushdownable instruction information set as the multi-source data pushdown instruction information set.

[0035] Step 1032, perform query language rewriting processing on the trimmed multi-source list data based on the pushdownable instruction information set included in the multi-source data pushdown instruction information set, so as to obtain multi-structured rewritten query language data.

[0036] In some embodiments, the above-mentioned execution entity may perform query language rewriting processing on the above-mentioned trimmed multi-source list data based on the push-down instruction information set included in the above-mentioned multi-source data push-down instruction information set, and obtain multi-structured rewritten query language data. In practice, the above-mentioned execution entity may connect each push-down instruction in the above-mentioned push-down instruction information set with a logical operator (for example, "AND"), and combine it with the above-mentioned trimmed multi-source list data to obtain a multi-structured rewritten query language data. As an example, the trimmed multi-source list data may be "SELECT name, age; FROM user; WHERE age>18; AND name LIKE 'Zhang%'; AND bitmap_contains(user_tag_bitmap, 1001);", where "bitmap_contains(user_tag_bitmap, 1001)" is a non-push-down instruction, and "age>18" and "name LIKE 'Zhang%'" are push-down instructions, and the obtained multi-structured rewritten query language data is "SELECT name, age, user_tag_bitmap; FROM user; WHERE age>18 AND name LIKE 'Zhang%';".

[0037] Step 104: Send each multi-structured rewritten query language data obtained to a preset execution node for the preset execution node to perform data query trimming processing on the multi-structured rewritten query language data, and obtain a remote data trimming query result.

[0038] In some embodiments, the above-mentioned execution entity may send each multi-structured rewritten query language data obtained to a preset execution node for the preset execution node to perform data query trimming processing on the multi-structured rewritten query language data, and obtain a remote data trimming query result. Among them, the above-mentioned preset execution node may be a pre-set computing device for executing data query transmission. The corresponding data table queried in the above-mentioned multi-structured rewritten query language data is stored in the above-mentioned preset execution node.

[0039] In addressing the technical problems mentioned above by adopting technical solutions, the following technical issues often arise in the application scenario: information retrieval of cross-provincial and multi-regional joint data (such as social security data and medical insurance data). When querying cross-provincial and multi-regional joint data, due to the large amount of data stored on each node distributed across various regions, returning all queried data to the relevant querying device would result in a massive amount of transmitted data, including a large amount of useless data (e.g., querying payment amounts would include personal identification numbers, place of origin, and other information). Each node directly returns all the queried information without removing unnecessary fields, leading to a significant waste of transmission resources and potential resource overload (such as memory overflow), causing query failures or interruptions. Therefore, this application scenario requires the following characteristics: reducing resource overload and query failures caused by large data transmissions. Faced with these technical problems, we decided to adopt the following solution: In some optional implementations of certain embodiments, the aforementioned preset execution node can perform data query processing on the aforementioned multi-structured rewritten query language data through the following steps to obtain remote data pruning query results: The first step involves retrieving the target processing dataset from a pre-defined database based on the aforementioned multi-structured rewritten query language data. This pre-defined database can be a database storing various data tables. In practice, the execution node can retrieve the data table representing the target query from the multi-structured rewritten query language data within the pre-defined database. Then, this target query data table is determined as the target processing dataset.

[0040] The second step involves reading the target processing dataset from a preset database based on a preset batch sample value, resulting in the dataset to be processed. This preset batch sample value can be a pre-defined maximum number of data rows to read per batch; for example, 1000 means a maximum of 1000 rows can be read at a time. In practice, the execution node can start from the first data row in the target processing dataset and read the number of rows represented by the preset batch sample value, using each row as the dataset to be processed.

[0041] Third, based on the above multi-structured rewritten query language data, perform the following processing steps on the dataset to be processed: The first sub-step involves pruning the dataset to be processed based on the aforementioned multi-structured rewritten query language data, resulting in a pruned dataset. In practice, for each piece of data to be processed in the dataset, the executing entity can delete the data corresponding to columns not represented in the multi-structured rewritten query language data, obtaining the pruned dataset as the pruned data. As an example, the data table corresponding to the above data to be processed has the following columns: (employee_id, full_name, age, salary, department, phone_number). Each data entry contains data corresponding to these columns. The above multi-structured rewritten query language data can be expressed as "SELECT full_name, department, salary; FROM employees; WHERE department='R&D Department';". The above multi-structured rewritten query language data only contains the three columns "full_name", "department", and "salary". "employee_id", "age", and "phone_number" are not represented. Deleting these unrepresented columns from the above data to be processed results in data containing only the three columns "full_name", "department", and "salary".

[0042] The second sub-step involves performing type conversion on each clipped data in the aforementioned clipped dataset to obtain the respective conversion target data. In practice, for each clipped data in the aforementioned clipped dataset, the execution node can convert the format of the data to be processed into an internal format adapted by Apache Doris, and determine the converted data as the conversion target data.

[0043] The third sub-step involves logically mapping the aforementioned target data to obtain a target mapping data table. In practice, the execution node can fill the target data into a preset field data table according to preset rules, and then define the filled data table as the target mapping data table. The preset rules could be to fill the target data according to its position in the target processing dataset; for example, after conversion, `emp_name` would be filled into the `user_name` field of the table. The preset field data table could be an empty table created based on the field names in the target processing dataset.

[0044] The fourth sub-step involves generating a data group to be returned based on the aforementioned target mapping data table. In practice, the execution node can validate each piece of data in the target mapping data table (e.g., checking for missing fields (e.g., required fields being empty); checking for data format errors (e.g., date fields not in YYYY-MM-DD format, numeric fields containing letters, etc.). If validation fails, the target mapping data table is discarded, and the dataset to be processed is reread. If validation succeeds, the target mapping data table is identified as the data group to be returned.

[0045] The fifth sub-step involves reading the target processing dataset from the preset database again, based on the aforementioned preset batch sample values, to obtain the dataset to be processed. In practice, the execution node can continue reading data representing the number of rows of values ​​from the preset batch sample values, starting from the end of the previous reading position, as the dataset to be processed.

[0046] The fifth sub-step involves, in response to determining that the target processing dataset meets the preset reading requirements, executing the above processing steps again based on the dataset to be processed. The preset reading requirements may include the ability to successfully read the dataset, indicating that there is still unread data in the target processing dataset.

[0047] The fourth step is to determine that the target processing dataset does not meet the preset reading requirements, and then identify each data group to be returned as the remote data trimming query result.

[0048] The above-described technical solution and its related content, as an inventive point of this disclosure, solve the technical problem of "query failure or interruption". Factors leading to query failure or interruption often include: when querying joint data across multiple regions such as provinces and cities, due to the large amount of data stored by nodes distributed across various regions, returning all queried data to the relevant querying entity device would result in a massive amount of transmitted data, including a large amount of useless data (e.g., querying payment amounts would include personal identification numbers, place of origin, and other information). Each node directly returns all the queried information without removing unnecessary fields, leading to a significant waste of transmission resources and potential resource overload (such as memory overflow), resulting in query failure or interruption. Solving these factors can reduce query failures or interruptions. To achieve this, firstly, based on the aforementioned multi-structured rewritten query language data, a preset database is retrieved to obtain the target processing dataset. This yields the target query data set. Next, based on preset batch sample values, the target processing dataset is read from the preset database to obtain the dataset to be processed. This allows for the batch reading of a certain amount of data to be processed, reducing the amount of data processed per session. Then, based on the aforementioned multi-structured rewritten query language data, the following processing steps are performed on the aforementioned dataset to be processed: First, based on the aforementioned multi-structured rewritten query language data, the dataset to be processed is pruned to obtain a pruned dataset. This yields a pruned dataset that excludes irrelevant data except for the fields to be retained. Next, each pruned data in the pruned dataset undergoes type conversion to obtain various conversion target data. Then, logical mapping is performed on each of the conversion target data to obtain a target mapping data table. This allows for the correct arrangement of target mapping data by filling the corresponding fields into their corresponding positions. Next, based on the target mapping data table, a data group to be returned is generated. This allows for the format verification of each data in the target mapping data table (whether the format meets the requirements, whether it is missing), deleting data that does not meet the conditions, resulting in a data group that can be used as returned data. Finally, based on the aforementioned preset batch sample values, the target processing dataset is read again from the preset database to obtain the dataset to be processed. Then, in response to determining that the target processing dataset meets the preset reading requirements, based on the aforementioned multi-structured rewritten query language data, the above processing steps are executed again on the dataset to be processed. This allows for batch reading of the target processing dataset, reducing the amount of data processed per batch. Finally, in response to determining that the target processing dataset does not meet the preset reading requirements, each resulting data group to be returned is identified as the remote data trimming query result.Because by rewriting the query language data using a multi-structured approach, only an array of fields contained within the rewritten query language data is retrieved as the target processing dataset. Unnecessary or unmentioned data from the query requirements is directly pruned and deleted. Simultaneously, data with errors or that is empty is filtered from the target processing dataset, reducing the transmission of useless information and the amount of data that needs to be transmitted at each execution node. This reduction in the amount of data returned decreases significant waste of transmission resources and reduces the frequency of query failures or interruptions.

[0049] Step 105: Generate a data query result based on the set of non-pushdown instruction information included in the multi-source data pushdown instruction information set and at least one remote data trimming query result obtained.

[0050] In some embodiments, the execution entity may generate a data query result based on the non-pushdown instruction information set included in the multi-source data pushdown instruction information set and at least one remote data trimming query result obtained. The data query result may be a dataset that integrates various data sets desired by the user.

[0051] In some optional implementations of certain embodiments, the aforementioned execution entity can generate data query results based on the non-pushdown instruction information set included in the multi-source data pushdown instruction information set and at least one remote data trimming query result obtained: The first step is to obtain the non-pushdown aggregation information. This non-pushdown aggregation information can be a list of non-pushdown function instructions that need to be executed locally.

[0052] The second step involves performing a validity check on the non-pushdown aggregated information, the non-pushdown instruction information set, and the results of at least one remote data pruning query, based on the aforementioned non-pushdown instruction information set included in the multi-source data pushdown instruction information set. This validity check results indicative of whether existing instructions and information can continue execution; for example, "erroneous" indicates that execution cannot continue, while "correct" indicates that execution can continue. In practice, the executing entity can verify whether each non-pushdown function instruction in the aforementioned non-pushdown aggregated information is a locally supported instruction. If an unsupported instruction exists, a "erroneous" validity check is generated. If all instructions are locally supported, it further determines whether all data fields included in the non-pushdown instruction information set exist and match the type in the results of the aforementioned at least one remote data pruning query. If any data field is missing or its type does not match, a "erroneous" validity check is generated. If all data fields exist and their types match, a "correct" validity check is generated. As an example, the above non-pushdown instruction information set can be "grouped by dept_id, calculate SUM(salary) and COUNT(DISTINCT user_name)". The data fields required in the non-pushdown instruction information set are "dept_id, user_name, salary". If at least one remote data pruning query result contains all of "dept_id", "user_name", and "salary" and the data types match, it is considered "correct"; otherwise, it is considered "incorrect".

[0053] The third step involves terminating the generation of data query results and returning an error message to the user upon determining that the aforementioned discrimination information does not meet the preset discrimination conditions. The preset discrimination conditions can be information representing "correctness." The error message can indicate an error in generating the query results, such as "query failed." The data query results can be the final summarized data returned to the user after the data query.

[0054] The fourth step involves generating a query result data set based on the at least one remote data trimming query result, in response to the determination that the above-mentioned discrimination information meets the preset discrimination conditions. In practice, the executing entity can group the data from the at least one remote data trimming query result according to a preset grouping rule to obtain the query result data set. The preset grouping rule can be based on the data being classified according to the same field; for example, grouping by "dept_id" means that data with the same "dept_id" information are grouped together.

[0055] The fifth step involves aggregating information for each query result data group in the aforementioned query result data set to obtain aggregated information for each group. In practice, for each query result data group in the aforementioned query result data set, the executing entity can summarize the data from each query result data group (for example, add the values ​​of the same field, or merge the information of each field into one) into a single data entry, and the summarized data entry will serve as the aggregated information for each group. As an example, under "dept_id" of "101", there are five pieces of information with the structure (user_name, salary, join_date), namely (01, 5600, 2023-01-05), (01, 5600, 2023-01-05), (02, 6500, 2003-02-25), (01, 5600, 2023-01-05), and (02, 6500, 2003-02-25). Summarizing the information of "01" and "02" (summing the values ​​at "salary" and summing the information at "join_date"), we get the grouped aggregated information as {(01, 16800, 2023-01-05) and (02, 13000, 2003-02-25)}.

[0056] The sixth step is to generate an aggregation result set based on the obtained aggregation information for each group. In practice, the executing entity can determine the aggregation result set based on the aggregation information for each group.

[0057] Step 7: Perform aggregation result validation on the above aggregation result set and generate aggregation result validation information. This validation information can characterize whether the format of each aggregation result in the aggregation result set meets the requirements, whether the number of data rows is reasonable (e.g., there are five possible identifiers for `dept_id`, and eight pieces of information in the aggregation result set), etc. For example, the validation information can be "normal" or "abnormal". In practice, the executing entity can determine whether the number of information in the aggregation result set is reasonable (e.g., there are five possible identifiers for `dept_id`, and eight pieces of information in the aggregation result set), and determine whether there are any numerical anomalies in the aggregation result set (e.g., negative numbers appear in data that should be all positive), and generate the corresponding aggregation result validation information.

[0058] Step 8: In response to determining that the above-mentioned aggregation result verification information meets the preset verification conditions, a data query result is generated based on the above-mentioned query result data set. The above-mentioned preset verification conditions may be that the above-mentioned aggregation result verification information indicates "normal". In practice, the above-mentioned executing entity can arrange the various query result data in the above-mentioned query result data set according to a preset sorting rule, and obtain the sorted query result data as the data query result.

[0059] Step 106: Return the data query results to the preset user terminal.

[0060] In some embodiments, the execution entity may return the data query results to the preset user terminal.

[0061] In addressing the technical problems mentioned above by adopting technical solutions, the application scenario of information retrieval at mobile stations in remote areas (e.g., mobile medical stations) often presents the following technical challenges: In information retrieval scenarios at mobile stations in remote areas (such as mobile medical stations), limited by the low power consumption, low computing power, and small storage of the devices, each node directly returns all the large amounts of queried information, easily leading to resource overload (such as memory overflow), resulting in query failures or interruptions. The following requirements are necessary for this application scenario: reducing resource overload and query failures caused by receiving large amounts of data under limited device conditions. Faced with these technical problems, we decided to adopt the following solution: Optionally, the aforementioned implementing entity may also perform the following steps: The first step is to generate the predicted result row values ​​based on the aforementioned trimmed multi-source list data and preset table-level summary information. The preset table-level summary information can include various summary data. These summary data can reflect the total number of rows in the data table and the size of the compressed data. Each summary data corresponds to a data table. In practice, for each trimmed multi-source list data, the execution entity can perform the following steps: First, filter the "WHERE" predicates in the trimmed multi-source list data to obtain the total number of predicates in the trimmed multi-source list data as the target predicate total. Then, determine the summary data corresponding to the data table to be queried represented in the trimmed multi-source list data as the target summary data. Next, determine the value representing the total number of rows in the data table in the target summary data as the first target value. Then, determine the estimated input row number by multiplying the target predicate total number by the first target value. Finally, determine the sum of the estimated input row numbers as the predicted result row value.

[0062] The second step involves generating at least one multi-structured rewritten query language (MSL) data based on the distributed query instruction set and the various trimmed multi-source list data, in response to the determination that the predicted row value is greater than a preset row value. Each MSL data includes local aggregation information. This local aggregation information can be a signal indicating that information aggregation needs to be performed remotely. In practice, for each trimmed multi-source list data, the execution entity can perform the following steps: First, based on the distributed query instruction set, predicate pushdown processing is performed on the multi-source data query information of the trimmed multi-source list data to obtain a multi-source data pushdown instruction information set. Next, each pushdownable instruction in the pushdownable instruction information set of the multi-source data pushdown instruction information set is connected using a logical operator (e.g., "AND") and combined with the trimmed multi-source list data to obtain a multi-structured rewritten query language data. Then, local aggregation information is generated. Finally, the aforementioned local aggregation information is added to the aforementioned multi-structured rewritten query language data to obtain the added multi-structured rewritten query language data.

[0063] The third step is to send each of the at least one multi-structured rewritten query language data to a preset execution node, so that the preset execution node can perform data query aggregation processing on the multi-structured rewritten query language data to obtain the remote data aggregation query result.

[0064] The fourth step is to validate at least one remote data aggregation query result to obtain validation data. In practice, firstly, the aforementioned execution entity can verify whether the preset execution node returns information indicating completion (e.g., receiving a certain number of completion signals). In response to determining that all information has returned completion, it iterates through each data format in the at least one remote data aggregation query result to check for correctness, deletes data with incorrect formats, and obtains at least one remote data aggregation query result after format checking as validation data.

[0065] Fifth, based on the non-pushdown instruction information set included in the aforementioned multi-source data pushdown instruction information set and the aforementioned verification data, generate data query results. It should be noted that the method used to generate data query results based on the non-pushdown instruction information set included in the aforementioned multi-source data pushdown instruction information set and the aforementioned verification data is the same as the method used to generate data query results based on the non-pushdown instruction information set included in the aforementioned multi-source data pushdown instruction information set and at least one remote data pruning query result.

[0066] Step 6: Return the above data query results to the above-mentioned preset user terminal.

[0067] The above technical solution and its related content, as an inventive point of this disclosure, solve the technical problem of "causing query failure or interruption". Factors causing query failure or interruption often include: In information retrieval scenarios of mobile stations in remote areas (such as mobile medical stations), due to the low power consumption, low computing power, and small storage of the devices, each node directly returns all the large amount of information retrieved, which can easily lead to resource overload (such as memory overflow), resulting in query failure or interruption. Solving these factors can reduce the number of query failures or interruptions. To achieve this effect, firstly, based on the aforementioned trimmed multi-source list data and preset table-level summary information, a predicted result row value is generated. This allows us to obtain the predicted number of rows of data to be returned, and determine whether a preset execution node needs to aggregate information based on the number of rows. Next, in response to determining that the predicted result row value is greater than the preset row value, based on the aforementioned distributed query instruction set and the aforementioned trimmed multi-source list data, at least one multi-structured rewritten query language data is generated, wherein each of the at least one multi-structured rewritten query language data includes local aggregation information. Therefore, when the predicted result value is greater than a certain value, indicating a potentially large amount of data to be transmitted, at least one multi-structured rewritten query language data containing a prompt for the preset execution node to perform information aggregation is generated. Then, each of the at least one multi-structured rewritten query language data is sent to the preset execution node for data query aggregation processing, yielding a remote data aggregation query result. This results in a remote data aggregation query result with reduced data transmission after partial data aggregation by the preset execution node. Next, the at least one remote data aggregation query result is validated to obtain validation data. This yields grouped validation data that meets the requirements after validating the data in the at least one remote data aggregation query result. Then, based on the non-push-down instruction information set included in the multi-source data push-down instruction information set and the validation data, a data query result is generated. This yields a data query result generated after full aggregation at the execution entity, used to return to the preset user terminal. Finally, the data query result is returned to the preset user terminal. Because it first predicts the amount of data the user needs to query, and obtains the predicted result row values, when there are too many data rows to be returned (i.e., the values ​​represented by the predicted result row values ​​are too large), it chooses to perform a data aggregation and summary at the preset execution node in advance, merging information that can be merged under the same category, reducing the amount of data to be returned, thereby reducing the total amount of data transmitted for the large amount of information queried by each node, reducing the resource overload (such as memory overflow) caused by directly returning all the large amount of queried information, and thus reducing the number of query failures or interruptions.

[0068] In addressing the aforementioned technical problems in the application scenario—information retrieval by researchers in remote areas—the following technical challenges arise: the limited number and processing capabilities of data transmission devices for researchers, coupled with the high real-time requirements of most data, make it difficult for these devices to receive and process large volumes of query data returned by execution nodes simultaneously. Simply grouping and compressing all query data before transmitting it back to the data transmission devices for processing can lead to bottlenecks due to the limited processing capabilities of these devices in remote areas. This can result in high latency and long processing times for generating query results, ultimately reducing the real-time performance of multi-source remote data queries. Therefore, this application scenario requires minimizing data transmission and subsequent processing volume, and reducing latency caused by large-scale data processing, while still maintaining the available equipment. To address these technical challenges, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the aforementioned preset execution node can perform data query aggregation processing on the aforementioned multi-structured rewritten query language data through the following steps to obtain the remote data aggregation query results: The first step is to retrieve the target processing dataset from the preset database based on the above multi-structured rewritten query language data.

[0069] The second step is to read the target processing dataset from the preset database based on the preset batch sample values ​​to obtain the dataset to be processed.

[0070] Third, based on the dataset to be processed described above, perform the following processing steps: The first sub-step involves performing field retention processing on the dataset to be processed based on the aforementioned multi-structured rewritten query language data, resulting in a retained field dataset. In practice, the execution node can iterate through each row of data in the dataset to be processed, removing all fields that are not relevant to the target data, retaining only the relevant fields from the multi-structured rewritten query language data, and using the dataset after removing irrelevant fields as the retained field dataset. For example, each data point in the dataset to be processed may have six fields: "dept_id", "salary", "user_name", "hire_date", "address", and "phone". The multi-structured rewritten query language data represents the aggregation target: calculate the average salary (salary) for each department (dept_id) and the number of employees in that department (counted by user_name). Therefore, the fields to be retained are "dept_id", "salary", and "user_name", while the fields to be removed are "hire_date", "address", and "phone".

[0071] The second sub-step involves validating and filtering the aforementioned reserved field dataset to obtain a valid field dataset. In practice, the execution node can validate each reserved field in the reserved field dataset (checking for empty data and format errors), deleting reserved field data that does not meet the validation criteria, and obtaining the validated reserved field dataset as the filtered reserved field dataset.

[0072] The fourth sub-step involves reading the target processing dataset from the preset database again, based on the aforementioned preset batch sample values, to obtain the dataset to be processed.

[0073] The fifth sub-step is to execute the above processing steps again based on the above dataset to be processed, in response to determining that the target processing dataset meets the preset reading requirements.

[0074] The fourth step, in response to the determination that the target processing dataset does not meet the preset reading requirements, generates a dataset to be aggregated based on the obtained datasets of each valid field. Each piece of data in this dataset includes a key field. These key fields can be fields represented in the query requirements described in the multi-structured rewritten query language data. For example, in the multi-structured rewritten query language data, calculating the average salary (salary) and the number of employees in each department (dept_id) (counted by user_name) would correspond to the key fields "dept_id", "salary", and "user_name". In practice, the execution node can determine each valid field data in the obtained datasets of each valid field as the dataset to be aggregated.

[0075] Step 5: Obtain the historical aggregation information set. Each historical aggregation information set contains a corresponding key field. This historical aggregation information may include each key field, its corresponding cumulative value, and the aggregation count. For example, in the multi-structured rewritten query language data, each cumulative value corresponding to a key field can be the aggregated numerical data of the numerical data corresponding to the key field from various aggregated data to be aggregated. The aggregation count can be the cumulative number of aggregated data to be aggregated. For instance, a historical aggregation information set might be (user_name=10, sum_salary=56000, valid_row_count=3), indicating that there are 3 aggregated data entries where "user_name" is 10, and the cumulative sum of the "sum_salary" field is 56000.

[0076] Step 6: Perform the following aggregation steps on each piece of data in the dataset to be aggregated: The first sub-step is to determine the target aggregation information in response to the determination that the key fields in the data to be aggregated have corresponding historical aggregation information in the historical aggregation information set.

[0077] The second sub-step, in response to determining that the key fields in the data to be aggregated do not have corresponding historical aggregation information in the historical aggregation information set, involves creating target aggregation information and adding the target aggregation information to the historical aggregation information set to update the historical aggregation information set. The target aggregation information can have the same format as the historical aggregation information, where the sum of each field is the data in the data to be aggregated, and the aggregation count is 1. For example, the data to be aggregated could be (user_name=10, salary=6000), and the created target aggregation information could be (user_name=10, sum_salary=6000, valid_row_count=1).

[0078] The third sub-step involves updating the target aggregation information based on the data to be aggregated, resulting in the updated target aggregation information. In practice, the execution node can extract the values ​​from each field of the data to be aggregated, sum them with the values ​​in the corresponding fields of the target aggregation information, and increment the aggregation count by 1 to obtain the updated target aggregation information. For example, if the target aggregation information is (user_name=10, sum_salary=56000, valid_row_count=3), and the data to be aggregated is (user_name=10, sum_salary=4500), then the updated target aggregation information could be (user_name=10, sum_salary=10500, valid_row_count=4).

[0079] The fourth sub-step involves updating the historical aggregation information set based on the updated target aggregation information, resulting in an updated historical aggregation information set. In practice, the execution node can replace the corresponding historical aggregation information in the historical aggregation information set with the updated target aggregation information to obtain the updated historical aggregation information set.

[0080] The seventh sub-step generates remote data aggregation query results based on the updated historical aggregation information set. In practice, the execution node arranges each historical aggregation information in the updated historical aggregation information set according to a preset rule, and the resulting arranged updated historical aggregation information serves as the remote data aggregation query results. The preset rule can be an ascending or descending sort based on the information values ​​in a field; for example, the preset rule sorts the information values ​​in "dept_id" in ascending order.

[0081] The above-described technical solution and its related content, as an inventive point of this disclosure, solve the technical problem of "high latency and low real-time performance in generating query results." Factors leading to high latency and low real-time performance in generating query results are often as follows: the number of data transmission devices and data processing capabilities of researchers are limited, and most of the data researchers need to acquire requires high real-time performance. Researchers' data transmission devices often struggle to receive and process large amounts of query data returned by execution nodes at once. If all query data is simply grouped, compressed, and then uniformly transmitted back to the researchers' data transmission devices for processing and calculation, the limited operational capabilities of these devices in remote areas may cause them to lag when generating query results, leading to high latency and long processing times, thus resulting in low real-time performance for multi-source remote data queries. Solving these factors can shorten the latency in generating query results and improve real-time performance. To achieve this, firstly, based on the aforementioned multi-structured rewritten query language data, a preset database is retrieved to obtain the target processing dataset. Thus, the target query data set can be obtained. Next, based on the preset batch sample values, the target processing dataset is read from the preset database to obtain the dataset to be processed. This allows for the reading of a certain amount of data in batches, reducing the amount of data processed in a single session. Then, based on the dataset to be processed, the following processing steps are performed: First, based on the aforementioned multi-structured rewritten query language data, the dataset to be processed is processed by retaining fields, resulting in a retained field dataset. This allows for the retention of only the data containing the information needed for querying, reducing the amount of data that needs to be transmitted subsequently. Next, the retained field dataset is validated and filtered to obtain a valid field dataset. This removes erroneous data that does not meet the requirements, reducing query failures caused by data errors. Then, based on the preset batch sample values, the target processing dataset is read again from the preset database to obtain the dataset to be processed. This allows for the reading of the next batch of data for processing. Finally, in response to the determination that the target processing dataset meets the preset reading requirements, the above processing steps are executed again based on the dataset to be processed. This allows for the sequential reading and processing of each target processing data item in the target processing dataset. Next, in response to the determination that the target processing dataset does not meet the preset reading requirements, a dataset to be aggregated is generated based on the obtained datasets of each valid field. Each piece of data to be aggregated in this dataset includes a key field. Thus, the datasets of each valid field obtained from the initial reading of the target processing dataset are obtained, and a dataset to be aggregated for subsequent information aggregation is further generated. Then, a historical aggregation information set is obtained, where each piece of historical aggregation information has a corresponding key field.Next, the following aggregation steps are performed on each piece of data in the dataset to be aggregated: First, in response to determining that the key fields in the data to be aggregated have corresponding historical aggregation information in the historical aggregation information set, the corresponding historical aggregation information is identified as the target aggregation information. Thus, target aggregation information corresponding to the key fields is obtained. Next, in response to determining that the key fields in the data to be aggregated do not have corresponding historical aggregation information in the historical aggregation information set, target aggregation information is created and added to the historical aggregation information set to update the historical aggregation information set. Thus, target aggregation information can be created based on the data to be aggregated for subsequent aggregation of similar data. Then, based on the data to be aggregated, the target aggregation information is updated to obtain the updated target aggregation information. Thus, the updated target aggregation information is obtained. Then, based on the updated target aggregation information, the historical aggregation information set is updated to obtain the updated historical aggregation information set. Thus, an updated historical aggregation information set summarizing certain information is obtained. Finally, based on the updated historical aggregation information set, remote data aggregation query results are generated. Because by using multi-structured rewritten query language data, only an array of field contents contained within the multi-structured rewritten query language data is obtained as the target processing dataset. Data with errors or empty data in the target processing dataset is filtered out, reducing the transmission of useless information and the amount of data that needs to be transmitted. Furthermore, by combining the target query data format within the multi-structured rewritten query language data and summarizing the data at the aforementioned preset execution nodes, the amount of data returned and the amount of data that related devices need to process after return are reduced, thereby reducing the delay in generating query results and improving real-time performance.

[0082] The above-described embodiments of this disclosure have the following beneficial effects: the multi-source remote data query method of some embodiments of this disclosure reduces the amount of data transmitted and the waste of transmission resources. Specifically, the reason for the large amount of data transmitted and the waste of transmission resources is that: by querying the database to obtain a list of identifiers that meet the conditions, traversing each identifier in the list and sequentially initiating query requests to multiple other databases to obtain the corresponding detailed information, and finally integrating and splicing the scattered data to form the final complete query result, the SQL execution languages ​​of different data sources are different when executing the query action, which may cause some query information to be unable to be executed directly. Moreover, when initiating query requests to multiple other databases to obtain the corresponding detailed information, the corresponding whole row of information will be obtained, and some unnecessary column information will also be imported along with it (for example, if the target query is the cumulative performance of "10", the data row corresponding to "10" in the database may contain unnecessary information such as "contact information" and "join time" in addition to performance, which will also be imported and returned, resulting in data redundancy and a large amount of transmitted data, with some information redundancy, causing waste of transmission resources. Based on this, the multi-source remote data query method of some embodiments of this disclosure firstly, in response to receiving multi-source data query information containing user equipment configuration information sent by a preset user terminal, generates various trimmed multi-source list data based on the multi-source data query information. This allows for the decomposition and analysis of the user's multi-source data query information to obtain various trimmed multi-source list data representing each data table, the target query data column, and the filtering data conditions for different data tables. Next, a distributed query instruction set corresponding to the aforementioned user equipment configuration information is obtained. This yields a distributed query instruction set containing locally executable instruction functions and remotely executable instruction functions of the execution subject. Then, for each trimmed multi-source list data, the following steps are performed: First, based on the aforementioned distributed query instruction set, predicate pushdown processing is performed on the trimmed multi-source list data to obtain a multi-source data pushdown instruction information set, wherein the multi-source data pushdown instruction information set includes a pushdown instruction information set and a non-pushdown instruction information set. Therefore, the various filtering conditions representing the user's desired data acquisition in the trimmed multi-source list data can be divided into instructions that can be executed at a preset execution node (push-down instruction information set) and instructions that must be executed locally (non-push-down instruction information set), resulting in a multi-source data push-down instruction information set, which is used to subsequently generate multi-structured rewritten query language data. Next, based on the push-down instruction information set included in the multi-source data push-down instruction information set, the trimmed multi-source list data is rewritten using query language to obtain multi-structured rewritten query language data.Therefore, multi-source list data from different data sources can be rewritten using a single execution language, avoiding the possibility that different SQL execution languages ​​from different data sources might prevent some queries from being executed directly. This data is then sent to a preset execution node to retrieve the target data (satisfying the filtering conditions in the query information) from the target data list using multi-structured rewritten query language data. Each multi-structured rewritten query language data is then sent to the preset execution node for data pruning, resulting in remote data pruning query results. This yields remote data pruning query results returned by the preset execution node, containing only the data needed from each target query table, with useless data removed. Next, based on the non-pushdown instruction set included in the multi-source data pushdown instruction information set and at least one remote data pruning query result, a data query result is generated. This provides a data query result that meets the user's query requirements. Finally, the data query result is returned to the preset user terminal. Because the multi-source data query information sent by users is broken down, various tailored multi-source list data are obtained for different data tables and different target filtering conditions. At the same time, combined with the distributed query instruction set (representing whether some filtering conditions and operations are executed locally or can be executed by the execution end of the stored data table), the filtering conditions represented in each tailored multi-source list data are distinguished, and step instructions that can be operated on at the execution end of the stored data table (i.e., multi-structured rewritten query language data) are obtained. Furthermore, the fused perception and knowledge vector are sent to the preset execution node (i.e., the execution end of the stored data table). This allows the aforementioned preset execution nodes to filter target data using multi-structured rewritten query language data (for example, in the "10" data table, only the "ID" and "sign" columns with the "sign" value of "RED" are obtained; even if other data columns exist, they are not extracted or transmitted). The preset execution nodes return information containing only the target query content (i.e., remote data pruning query results), while data in the data table unrelated to the target query content is not transmitted or returned. Furthermore, based on the user device configuration information, the execution steps in the query data requirements are assessed to determine if local execution is suitable. This avoids directly loading large amounts of detailed information from multiple databases locally, preventing data redundancy and excessive data transmission, thus reducing wasted transmission resources.

[0083] Further reference Figure 2As an implementation of the methods shown in the figures, this disclosure provides some embodiments of a plate information recognition device, which are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0084] like Figure 2 As shown, a multi-source remote data query device 200 in some embodiments includes: a first generation unit 201, an acquisition unit 202, an execution unit 203, a sending unit 204, a second generation unit 205, and a return unit 206. The first generation unit 201 is configured to, in response to receiving multi-source data query information containing user equipment configuration information sent by a preset user terminal, generate various trimmed multi-source list data based on the multi-source data query information; the acquisition unit 202 is configured to acquire a distributed query instruction set corresponding to the user equipment configuration information; the execution unit 203 is configured to perform the following steps on each trimmed multi-source list data: based on the distributed query instruction set, perform predicate pushdown processing on the trimmed multi-source list data to obtain a multi-source data pushdown instruction information set, wherein the multi-source data pushdown instruction information set includes a pushable instruction information set and a non-pushable instruction information set; based on the multi-source data pushdown instruction information set... The information set includes a set of pushable instruction information. The multi-source list data is then processed using a query language rewriting technique to obtain multi-structured rewritten query language data. The sending unit 204 is configured to send each of the obtained multi-structured rewritten query language data to a preset execution node, whereby the preset execution node performs data query trimming processing on the multi-structured rewritten query language data to obtain remote data trimming query results. The second generation unit 205 is configured to generate data query results based on the non-pushable instruction information set included in the multi-source data push-down instruction information set and at least one obtained remote data trimming query result. The return unit 207 is configured to return the data query results to the preset user terminal.

[0085] It is understandable that the units described in the device 200 are related to the reference. Figure 1 The steps in the method described above correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units contained therein, and will not be repeated here.

[0086] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0087] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0088] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0089] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0090] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0091] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0092] The computer-readable medium may be contained within an electronic device or may exist independently, not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to respond to receiving multi-source data query information containing user device configuration information sent by a preset user terminal; generate various trimmed multi-source list data based on the multi-source data query information; obtain a distributed query instruction set corresponding to the user device configuration information; and perform the following steps for each trimmed multi-source list data: based on the distributed query instruction set, perform predicate pushdown processing on the trimmed multi-source list data to obtain a multi-source data pushdown instruction information set, wherein the multi-source data pushdown instruction information set includes a pushdown-enabled instruction information set and a non-pushdown-enabled instruction information set. The instruction information set; based on the pushable instruction information set included in the multi-source data pushdown instruction information set, the query language rewriting process is performed on the trimmed multi-source list data to obtain multi-structured rewritten query language data; each multi-structured rewritten query language data is sent to a preset execution node, so that the preset execution node can perform data query trimming processing on the multi-structured rewritten query language data to obtain remote data trimming query results; based on the non-pushable instruction information set included in the multi-source data pushdown instruction information set and at least one remote data trimming query result, a data query result is generated; the data query result is returned to the preset user terminal.

[0093] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0094] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0095] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a first generation unit, an acquisition unit, an execution unit, a sending unit, a second generation unit, and a return unit. The names of these units do not necessarily limit the specific unit itself; for example, the second generation unit may also be described as "a unit that generates a data query result based on the non-pushdown instruction information set included in the multi-source data pushdown instruction information set and at least one remote data trimming query result obtained."

[0096] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0097] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of technical features, but should also cover other technical solutions formed by arbitrary combinations of technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for multi-source remote data query, comprising: In response to receiving multi-source data query information containing user equipment configuration information sent by a preset user terminal, various trimmed multi-source list data are generated based on the multi-source data query information; Obtain the distributed query instruction set corresponding to the user equipment configuration information; For each piece of clipped multi-source list data in each of the aforementioned clipped multi-source list data, perform the following steps: Based on the distributed query instruction set, predicate pushdown processing is performed on the trimmed multi-source list data to obtain a multi-source data pushdown instruction information set, wherein the multi-source data pushdown instruction information set includes a pushdown instruction information set and a non-pushdown instruction information set; Based on the set of pushable instruction information included in the multi-source data pushdown instruction information set, the trimmed multi-source list data is rewritten using query language to obtain multi-structured rewritten query language data. Each of the obtained multi-structured rewritten query language data is sent to a preset execution node so that the preset execution node can perform data query and pruning processing on the multi-structured rewritten query language data to obtain remote data pruning query results. Based on the set of non-push-down instruction information included in the multi-source data push-down instruction information set and at least one remote data trimming query result obtained, a data query result is generated; The data query results are returned to the preset user terminal.

2. The method of claim 1, wherein, The process of generating various cropped multi-source list data based on the multi-source data query information includes: The multi-source data query information is parsed and processed to obtain various abstract syntax trees; Metadata trimming is performed on each of the abstract syntax trees to obtain trimmed multi-source list data.

3. The method according to claim 2, wherein, The metadata trimming process performed on each of the abstract syntax trees to obtain each trimmed multi-source list data includes: For each of the abstract syntax trees, perform the following steps: Based on the abstract syntax tree, a set of reference columns is generated; Based on the multi-source data query information, obtain the metadata information of the query target; Based on the target metadata information and the reference column set, a trimmed multi-source list data is generated.

4. The method according to claim 1, wherein, The predicate pushdown processing of the trimmed multi-source list data based on the distributed query instruction set yields a multi-source data pushdown instruction information set, including: Based on the trimmed multi-source list data, a predicate dataset is generated; Based on the distributed query instruction set, the predicate dataset is subjected to pushdown discrimination processing to obtain a pushdown instruction information set and a non-pushdown instruction information set; The push-down instruction information set and the non-push-down instruction information set are determined as a multi-source data push-down instruction information set.

5. The method according to claim 1, wherein, The data query results generated based on the non-pushdown instruction information set included in the multi-source data pushdown instruction information set and at least one remote data pruning query result include: Retrieve unpushed aggregate information; Based on the non-pushdown aggregated information, the non-pushdown instruction information set, and the at least one remote data pruning query result, the non-pushdown instruction information set included in the multi-source data pushdown instruction information set is subjected to legality discrimination processing to obtain discrimination information; In response to determining that the discrimination information does not meet the preset discrimination conditions, the generation of data query results is terminated, and an error message is returned to the preset user terminal; In response to determining that the discrimination information meets the preset discrimination conditions, a query result data set is generated based on the at least one remote data clipping query result; Information aggregation processing is performed on each query result data group in the query result data group set to obtain the aggregated information of each group; Based on the obtained aggregation information of each group, an aggregation result set is generated; The aggregation result set is validated to generate aggregation result validation information; In response to determining that the aggregation result verification information meets the preset verification conditions, a data query result is generated based on the query result data set.

6. A multi-source remote data query device, comprising: The first generation unit is configured to generate various trimmed multi-source list data based on the multi-source data query information received from a preset user terminal, in response to receiving multi-source data query information containing user equipment configuration information. The acquisition unit is configured to acquire a distributed query instruction set corresponding to the user equipment configuration information; The execution unit is configured to perform the following steps for each of the trimmed multi-source list data in each trimmed multi-source list data: performing predicate pushdown processing on the trimmed multi-source list data based on the distributed query instruction set to obtain a multi-source data pushdown instruction information set, wherein the multi-source data pushdown instruction information set includes a pushable instruction information set and a non-pushable instruction information set; and performing query language rewriting processing on the trimmed multi-source list data based on the pushable instruction information set included in the multi-source data pushdown instruction information set to obtain multi-structured rewritten query language data. The sending unit is configured to send each of the obtained multi-structured rewritten query language data to a preset execution node, so that the preset execution node can perform data query pruning processing on the multi-structured rewritten query language data to obtain remote data query pruning results. The second generation unit is configured to generate a data query result based on the non-pushdown instruction information set included in the multi-source data pushdown instruction information set and at least one remote data trimming query result obtained. The return unit is configured to return the data query results to the preset user terminal.

7. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 5.

8. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images