Semi-structured and non-structured data query method and system under lake-warehouse fusion architecture

Abstract

The invention belongs to the technical field of large-scale semi-structured and non-structured data access, and particularly relates to a method and a system for querying semi-structured and non-structured data under a lake-warehouse fusion architecture, and the method comprises the following specific processes: randomly sampling a data subset d on queried data D, extracting semantic embedding corresponding to the samples in the data subset d and a prediction result of the target sensing model M; according to data types in the data subset d, internal relevance of the data types is extracted to generate multi-dimensional coordinates, and discrete coordinates are converted into high-dimensional continuous relevance embedding through multi-resolution hash coding; semantic embedding and relevance embedding are spliced into total embedding, the total embedding serves as lightweight neural network input, and multi-resolution hash codes and the lightweight neural network are jointly optimized to regress a prediction result of the target sensing model; and applying the trained multi-resolution hash codes and the lightweight neural network to queried data to generate proxy scores corresponding to all the data.

Description

Technical Field

[0001] This invention belongs to the field of large-scale semi-structured and unstructured data access technology, and specifically relates to a method and system for querying semi-structured and unstructured data under a lake warehouse fusion architecture. Background Technology

[0002] In the real world, the forms and volumes of semi-structured and unstructured data are becoming increasingly diverse. To address the efficient access to massive amounts of semi-structured and unstructured data, a lake-warehouse converged architecture is a common solution. In this architecture, the raw data is stored in a data lake, while the relevant metadata is organized in a data warehouse.

[0003] Improving the query efficiency of massive semi-structured and unstructured data is a crucial issue. Most existing technologies are based on semantic vectorization, utilizing pre-trained models to extract semantic information from different modalities and employing clustering or ranking algorithms for relevant information mining. However, these methods support limited query formats and cannot cope with the ever-increasing diversity of query demands. Taking unstructured video data as an example, queries might focus on the number of targets or rare events in long videos. To address these queries, various surrogate score-based query methods (BlazeIt, SUPG, ExSample, etc.) have been proposed. Furthermore, methods such as TASTI utilize semantic embedding to implement a rule-based surrogate score estimation method. These methods randomly sample the most heterogeneous representative data from all data and use the target model for sampling to obtain the true results. For unrepresentative data, the distance to all representative data can be calculated, and the weighted estimate can be obtained using the ground truth values ​​of the k nearest representative data.

[0004] However, these methods ignore the inherent long-tail distribution characteristics of real-world data, and different queries focus on different data distribution segments; on the other hand, these methods ignore the inherent correlation between semi-structured and unstructured data. Summary of the Invention

[0005] In view of this, the present invention proposes a method and system for querying semi-structured and unstructured data under a lake-warehouse fusion architecture, which can gain insight into the inherent distribution characteristics of data and query data preferences, and use a lightweight neural network trained online to transform the materialized semantic embeddings in the lake-warehouse fusion architecture into surrogate scores.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows: Firstly, this invention provides a method for querying semi-structured and unstructured data under a lake warehouse fusion architecture, the specific process of which is as follows: Step 1, Semantic Embedding and Prediction Result Acquisition: Randomly sample a subset d from the queried data D, and extract the semantic embedding and prediction results of the target perception model M corresponding to the samples in the subset d; Step 2, Relationship Embedding Acquisition: Based on the data types in the data subset d, extract their inherent relationships to generate multidimensional coordinates; normalize each dimension of the coordinates, and use multi-resolution hash encoding to transform the discrete coordinates into high-dimensional continuous relationship embeddings; Step 3: Concatenate the semantic embedding and the relevance embedding into a total embedding, and use the total embedding as input to a lightweight neural network to jointly optimize the multi-resolution hash code and the lightweight neural network to regress the prediction results of the target perception model; during the training process, increase the data resampling ratio of the data segments of interest in the query according to the data distribution preference. Step four: Apply the trained multi-resolution hash code and lightweight neural network to the queried data to generate proxy scores for all data.

[0007] Optionally, the present invention describes the process of extracting the inherent correlations among the data types in the data subset d to generate multidimensional coordinates. Specifically, this involves: determining the data types in the data subset d; for audio and video data, extracting correlated time-series information as 1D coordinates; for document data, extracting block order information as 1D coordinates; for log data, extracting time-series patterns as 1D coordinates; and for graph data, extracting node graph embedding information as... Dimensional coordinates; for semi-structured JSON / XML data, nodes at the same level are numbered sequentially, and the numbering sequence obtained by depth-first search is used as... 3D coordinates; for other data, set the fixed coordinates to 0.

[0008] Optionally, the present invention normalizes each dimension of the coordinates and uses multi-resolution hash coding to transform discrete coordinates into high-dimensional continuous correlation embeddings. The specific process is as follows: set up Each resolution has a specified length. A hash table, where each hash table stores a learnable hash. dimensional vector; against Space usage The hypercube is divided into segments of a resolution, and all vertices are recorded and mapped into a hash table. For each normalized coordinate, calculate its Euclidean distance to all vertices of the hypercube, calculate the vector corresponding to the coordinate by weighting the vectors of all vertices, and use it as a high-dimensional continuous correlation embedding.

[0009] Optionally, the settings described in this invention The resolution, the first The step size for each resolution is Using formula Calculation, including growth factors .

[0010] Optionally, the vector corresponding to the coordinates described in this invention is obtained by weighting all vertex vectors, and is expressed as:

[0011] in, Indicates up to the Euclidean distance to each vertex; Indicates the first The vector corresponding to each vertex.

[0012] Optionally, in step one of the present invention, a data subset d is randomly sampled from the queried data D, and a perception operation is performed on all data in the data subset d using the target perception model M, and the data is checked one by one using the query rule R; if the query result in the data subset d has satisfied the query target or the termination condition T, then the data subset d is resampled.

[0013] Optionally, in the method of enhancing the data resampling ratio of the data segment of interest based on data distribution preferences described in this invention, the process of obtaining the data preferences is as follows: Using the current query requirements, query process information, and data distribution, a match is performed in rule table X to obtain the corresponding matching results and return the data distribution preference. The data preference of the current query is obtained by sampling from the distribution preference.

[0014] Optionally, the construction and query process of rule table X in this invention is as follows: (1) Initialize rule table X to be empty; (2) Determine the data preference of the query based on query Q, historical sampling process information, and data distribution; (3) Update the matching rules and corresponding priorities in the rule table X; the rule is in the form of: taking the requirements of a single query, query process information and data distribution as input, and taking the attention to different segments in the statistically obtained probability density distribution or frequency distribution as output; (4) When a new query is executed, the historical query process information and data distribution are used to match in the rule table X. The matching process is to match all rules in the rule table X in order of priority. When a rule is successfully matched, the output of that rule is returned directly. Each historical query will correspond to a matching result. If the query satisfies the rules in the rule table X, its data distribution preference is returned; otherwise, it falls back to the no preference assumption. Optionally, the attention level described in this invention is represented by a floating-point number between 0 and 1, and the sum of the attention levels of all segments is 1; the attention level will be used as the resampling ratio of the corresponding segment data.

[0015] Secondly, the present invention provides a semi-structured and unstructured data query system under a lake warehouse fusion architecture, comprising: The semantic embedding and prediction result acquisition module is used to randomly sample a data subset d from the queried data D, and extract the semantic embedding and prediction result of the target perception model M corresponding to the sample in the data subset d. The correlation embedding acquisition module is used to extract the inherent correlations of the data subset d based on the data type to generate multidimensional coordinates; normalize each dimension of the coordinates, and use multi-resolution hash encoding to transform the discrete coordinates into high-dimensional continuous correlation embeddings. A lightweight neural network is used to receive the total embedding formed by concatenating the semantic embedding and the relevance embedding, and output the proxy score corresponding to all data; During training, the multi-resolution hash encoding and the lightweight neural network are used to regress the prediction results of the target perception model, while simultaneously increasing the data resampling ratio of the data segments of interest in the query based on data distribution preferences.

[0016] Beneficial effects: This invention can perceive the data preferences of queries online, model the inherent correlation of data, improve the estimation accuracy of proxy scores, and use a lightweight neural network trained online to reduce the number of calls to expensive target models, thereby improving query efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the overall architecture of the present invention.

[0019] Figure 2 This is a flowchart of the proxy score generation process of the present invention.

[0020] Figure 3 This is a schematic diagram of the data preference perception process of the present invention. Detailed Implementation

[0021] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0022] It should be noted that, in the absence of conflict, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0023] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0024] like Figure 1 As shown, this invention includes two core technologies, namely, proxy score generation (e.g. Figure 2 (as shown) and query data preference awareness (such as...) Figure 3 As shown in the figure, the query data preference awareness is embedded in the process of generating agent scores.

[0025] The proxy score generation technique involves randomly sampling a representative subset of data from the full dataset in the data warehouse. A target-aware model is then used to sample this representative data to obtain true labels, and semantic embeddings are extracted from the lake warehouse fusion architecture. For data with inherent correlations, these correlations are extracted based on data type, and multi-dimensional coordinates are constructed to model the relationships between data. A multi-resolution hashing algorithm is used to map the normalized coordinates to high-dimensional continuous embeddings, which are then concatenated with the semantic embeddings to enhance prior correlations. These embeddings are input into a shallow neural network, and the multi-resolution hashing algorithm and the shallow neural network are jointly optimized to regress the prediction results of the target model. The query data preference awareness technique is embedded during training. The trained multi-resolution hashing algorithm and the shallow neural network are then applied to the full dataset to generate proxy scores for all data.

[0026] Query data preference awareness technology: The target-aware model is invoked on a very small subset of data for statistical modeling of data distribution. Cached past query results further enhance the accuracy of distribution modeling. During each query, the sampling method and results of the target-aware model are recorded as sampling process information and cached in the lakehouse fusion architecture. Based on the cached query results and sampling process information, a rule-based data preference matching algorithm is designed. When a new query arrives, its historical sampling process information and query semantics are fed into the matching algorithm for matching. The matching results of historical queries are used as weights to sample and select the preference for the current query. If the final selection for the current query is no preference, the query is marked after execution, and the rule table is further analyzed and expanded.

[0027] Firstly, this application provides a method for querying semi-structured and unstructured data under a lake warehouse fusion architecture, the specific process of which is as follows: Consider a query Q(D,M,R,T,F), where D is the queried data, M is the target-aware model used, R is the query rule, T is the query target or termination condition, and F is the sampling algorithm; a query data preference rule table X is also set (which can be based on domain expert experience); the following describes the specific implementation methods for these two operations: Step 1, Semantic Embedding and Prediction Result Acquisition: Randomly sample a subset d from the queried data D, and extract the semantic embedding and prediction results of the target perception model M corresponding to the samples in the subset d; Step 2, Relationship Embedding Acquisition: Based on the data types in the data subset d, extract their inherent relationships to generate multidimensional coordinates; normalize each dimension of the coordinates, and use multi-resolution hash encoding to transform the discrete coordinates into high-dimensional continuous relationship embeddings; Step 3: The semantic embedding and the relevance embedding are concatenated into a total embedding. The total embedding is used as the input of a lightweight neural network to jointly optimize the multi-resolution hash code and the lightweight neural network to regress the prediction results of the target perception model. During the training process, the data resampling ratio of the query-focused data segments is increased according to the data distribution preference to improve the surrogate score prediction accuracy of the corresponding segments. Step four: Apply the trained multi-resolution hash code and lightweight neural network to the queried data to generate proxy scores for all data.

[0028] Furthermore, in order to select a representative data subset d in this embodiment, the specific process of step one is as follows: based on semantic embedding, a random sampling of a data subset d is applied to the queried data D, the target perception model M is used to perform perception operations on all data in the data subset d, and the query rule R is used to check each data. If the query result in the data subset d has satisfied the query target or the termination condition T, the process is directly terminated; otherwise, the sampling of the data subset d is repeated.

[0029] Furthermore, the specific implementation process of generating the above multidimensional coordinates will be explained in detail below: Determine if the data types have inherent relationships, extract the inherent relationships, and generate multidimensional coordinates for the data. If no relationship exists, it is defined as having no inherent relationship and assigned a fixed number of 0. The specific determination and extraction process is as follows: If the data type is audio and video, then the frame numbers are related, so the frame numbers are extracted as 1-dimensional coordinates. If the data type is document data, then the block order information is related to each other, so the block order information is extracted as a 1-dimensional coordinate. If the data type is log data, then the time series recorded in the log are correlated, so the time series pattern recorded in the log is extracted as a 1-dimensional coordinate. If the data type is graph data, then the node graph embedding information is related. Therefore, random walk or graph neural network can be used to extract the node graph embedding information into n-dimensional coordinates. If the data type is semi-structured such as JSON / XML, it is parsed into a tree structure, and the nodes at the same level are numbered sequentially. The number sequence obtained by depth-first search is an n-dimensional coordinate.

[0030] Furthermore, the following describes a specific implementation of the method for multi-resolution hash encoding of correlation coordinates provided by the present invention: (1) Normalize each dimension of the data correlation coordinates; the maximum value for a certain dimension is The coordinates of this dimension are The normalized coordinates of the data are This fixes the coordinates of all data at In the hypercube; (2) Settings The resolution, the first The step size for each resolution is ,and To achieve uniform distribution across different resolutions, the formula can be used. Calculation, including growth factors ; (3) Space usage Hypercube segmentation at resolutions of 1, each resolution will Hypercube divided into Each subspace, total There are one vertex, and all vertices are recorded; if the data dimension is one, then... It equals one, that is to say, here Same as data dimensions.

[0031] (4) Set a length of for each resolution. A hash table, where each hash table stores a learnable hash. A dimensional vector; in implementation, it is equivalent to a learnable vector. dimensional tensor; (5) For the first Establish a level The hash mapping maps all vertices into a hash table; when the level is low, the resolution is coarse and no collisions will occur; when the level is high, multiple vertices may share a feature vector, but since the vector itself is learnable, it will learn to handle collisions during joint model optimization. (6) Given a normalized coordinate and a level The coordinates must fall within a hypercube subspace of that level; each hypercube contains Given vertices, extract the vector corresponding to each vertex. ; Calculate the Euclidean distance from the normalized coordinates to all vertices. ; (7) Hash the vertex coordinates and extract the vector corresponding to each vertex. ; (8) Based on the normalized coordinates, the vectors of all vertices are weighted and summed to obtain the target sample relevance embedding. .

[0032] Furthermore, the specific process of querying data preference awareness in this embodiment is as follows: 1. Use the target perception model M to perform perception operations on all data in the data subset d, and use the query rule R to check each data, and summarize and sort the check results to obtain the data distribution; specifically, if the target perception model M is a regression model, then convert all regression values ​​into probability distribution density; if the target perception model M is a classification model, then perform statistics on the frequency of classification results. 2. Input the historical query sampling process information and data distribution of query Q into the rule table X to obtain the data preference. The data preference indicates the different degrees of attention the current query pays to different segments in the data distribution. If the rule matching fails, it will fall back to the no-preference assumption, and the query Q and the process information will be recorded and the rules will be updated by the domain expert. 3. Based on the obtained data preferences, the sampling algorithm F is applied to the data subset D to sample the data. The sampling process information, including the sampling method and sampling results, is recorded and stored in the lake warehouse fusion architecture.

[0033] Furthermore, the construction and query process of rule table X in this embodiment is as follows: (1) Initialize rule table X to be empty; (2) Determine the data preference of the query based on query Q, historical sampling process information, and data distribution; (3) Update the matching rules and their corresponding priorities in the rule table X; the rule is in the form of: taking the requirements of a single query, query process information and data distribution as input, and taking the attention to different segments in the statistically obtained probability density distribution or frequency distribution as output; the attention is represented by a floating-point number between 0 and 1, and the sum of the attention of all segments is 1; the attention will be directly used as the resampling ratio of the corresponding segment data; the priority is represented by a positive integer, and the higher the value, the higher the priority of the rule; (4) When a new query is executed, the historical query process information and data distribution are used to match in the rule table X. The matching process is to match all rules in the rule table X in order of priority. When a rule is successfully matched, the output of that rule is returned directly. Each historical query will correspond to a matching result. If the query satisfies the rules in the rule table X, its data distribution preference is returned; otherwise, it falls back to the no preference assumption. (5) Sample from the preference distribution to obtain the data preferences of the current query.

[0034] The above is a preferred embodiment of the present invention, and those skilled in the art can implement or use the present invention in this manner. Meanwhile, those skilled in the art can modify the technical solutions or embodiments provided by the present invention. Therefore, the present invention is not limited to the embodiments described herein, and simple modifications to the embodiments without departing from the technical solutions of the present invention are all within the protection scope of the present invention.

[0035] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for querying semi-structured and unstructured data under a lakeware fusion architecture, characterized in that, The specific process is: Step one, semantic embedding and predicted result acquisition: randomly sampling a data subset d from the queried data D, taking out the semantic embedding corresponding to the sample in the data subset d and the predicted result of the target perception model M; Step two, correlation embedding acquisition: according to the data type in the data subset d, extracting the internal correlation to generate multi-dimensional coordinates; normalizing each dimension of the coordinates, and using multi-resolution hash coding to convert discrete coordinates into high-dimensional continuous correlation embedding; Step three, splicing the semantic embedding and the correlation embedding into a total embedding, taking the total embedding as the input of the lightweight neural network, and jointly optimizing the multi-resolution hash coding and the lightweight neural network to regress the predicted result of the target perception model; in the training process, according to the data distribution preference, the data resampling ratio of the query attention data section is strengthened; Step four, applying the trained multi-resolution hash coding and lightweight neural network to the queried data to generate the proxy score corresponding to all data.

2. The method as claimed in claim 1 under Lakehouse architecture for querying semi-structured and unstructured data, wherein, The multi-dimensional coordinates are generated according to the internal correlation of the data types in the data subset d, and the specific process is as follows: judging the data types in the data subset d, for audio and video data, extracting the time sequence information existing correlation as 1-dimensional coordinates; for document data, extracting the block order information as 1-dimensional coordinates; for log data, extracting the time sequence pattern as 1-dimensional coordinates; for graph data, extracting the node graph embedding information as dimensional coordinates; for JSON / XML semi-structured data, numbering the nodes in the same layer in order, and taking the numbering sequence obtained by depth-first search as dimensional coordinates; for other data, setting the fixed coordinate as 0.

3. The method of claim 2, wherein the method is implemented in a Lakehouse architecture. The specific process of normalizing each dimension of the coordinates and using multi-resolution hash coding to convert discrete coordinates into high-dimensional continuous correlation embedding is: Set a resolution, set a length of hash table for each resolution, each hash table stores a learnable dimensional vector inside; For Space usage hypercube partitioning of the individual resolutions, and record all vertices and map all vertices into a hash table; For each normalized coordinate, calculate the Euclidean distance to all vertices of the hypercube, calculate the vector corresponding to the coordinate by weighting all vertex vectors, and take it as the high-dimensional continuous correlation embedding.

4. The method of claim 3, wherein, The settings The step size for the second resolution is , using the formula where the growth factor .​ 5. The method for querying semi-structured and unstructured data under the lake warehouse fusion architecture according to claim 3, wherein, The calculation of the vector corresponding to the coordinate by weighting all vertex vectors is represented as: wherein, denotes the Euclidean distance to the th vertex; denotes the vector corresponding to the th vertex.

6. The method for querying semi-structured and unstructured data under the lake warehouse fusion architecture of claim 1, wherein, In step one, a data subset d is randomly sampled from the queried data D, the target perception model M is used to perform perception operation on all data in the data subset d, and the query rule R is used to check the data one by one; if the query result on the data subset d has met the query target or the termination condition T, the data subset d is resampled.

7. The method for querying semi-structured and unstructured data under the lake warehouse fusion architecture according to claim 1, wherein, In the process of strengthening the data resampling ratio of the query attention data section according to the data distribution preference, the process of obtaining the data preference is: Using the requirements of the current query, the query process information and the data distribution, matching in the rule table X, obtaining the corresponding matching result returning data distribution preference, and sampling the data preference of the current query from the distribution preference.

8. The method of claim 7, wherein the method is implemented in a lakehouse architecture. The construction and query process of the rule table X are: (1) initializing the rule table X to be empty; (2) judging the data preference of the query according to the query Q, the historical sampling process information and the data distribution; (3) updating the matching rules and the corresponding priority in the rule table X; the form of the rule is: taking the requirements of a single query, the query process information and the data distribution as the input, and taking the attention degree of different sections in the statistical probability density distribution or frequency distribution as the output; (4) when a new query is executed, the historical query process information and the data distribution are matched in the rule table X; the matching process is to match all rules in the rule table X in order of priority, and when a rule is matched successfully, the output of the rule is directly returned; each historical query corresponds to a matching result, if the query meets the rules in the rule table X, the data distribution preference is returned; otherwise, it is assumed to have no preference.

9. The method of claim 8, wherein the method is implemented in a lakehouse architecture. The attention degree is represented by a floating point number between 0 and 1, and the sum of the attention degrees of all sections is 1; the attention degree will be the resampling ratio of the corresponding section data.

10. A semi-structured and unstructured data query system under a lake-warehouse fusion architecture, characterized in that, Comprise: The semantic embedding and predicted result acquisition module is used for randomly sampling a data subset d on the queried data D, taking out the semantic embedding corresponding to the sample in the data subset d and the target perception model M predicted result; The relevance embedding acquisition module is used for extracting the internal relevance of the data subset d according to the data type to generate multi-dimensional coordinates; each dimension of the coordinates is normalized, and the discrete coordinates are converted into high-dimensional continuous relevance embedding by using multi-resolution hash coding; The lightweight neural network is used for receiving the total embedding spliced by the semantic embedding and the relevance embedding, and outputting the proxy score corresponding to all data pairs; Wherein, the multi-resolution hash coding and the lightweight neural network are trained to regress the predicted result of the target perception model, and at the same time, according to the data distribution preference, the data resampling ratio of the query attention data section is strengthened.

Images