Method, device and storage medium for analyzing unstructured data

Abstract

Disclosed are a method, device and storage medium for analyzing unstructured data. The method comprises: constructing a set of predefined standard semantic operators, each operator having a logical representation; receiving a natural language query input by a user as a query to be simplified; performing semantic analysis on the query to be simplified using a large language model to generate a logical representation thereof; screening candidate operators based on semantic similarity matching between the logical representation and the logical representations of the operators; reordering the candidate operators; applying the candidate operators to simplify the query in turn until the simplification is successful; and constructing and executing a plan according to the query simplification process when it is determined that the query after the current simplification is a minimum semantic unit that cannot be further decomposed. The application significantly reduces the technical threshold for unstructured data analysis and improves query efficiency and accuracy through automated plan generation and optimization.

Description

Technical Field

[0001] This application relates to the field of data processing and analysis technology, and in particular to a method, apparatus and storage medium for analyzing unstructured data. Background Technology

[0002] Data lake technology is widely used due to its ability to process large-scale heterogeneous data, but it still faces challenges in intelligent analysis, especially in processing unstructured data, where existing tools such as SQL provide insufficient support. The main purpose of unstructured data analysis is to quickly and accurately extract valuable information. How to efficiently support complex semantic analysis has become a current research focus, but existing solutions have high technical barriers and limited effectiveness for ordinary users.

[0003] Specifically, while some systems allow users to create analysis workflows through coding, they place excessive demands on users' technical skills. Retrieval Augmentation (RAG) methods have a narrow scope, handling only simple, small-scale data queries with low accuracy. Furthermore, existing large-scale language model applications, when processing complex queries, often rely on static, manually orchestrated execution pipelines. These pipelines are not only complex and costly to construct but also lack dynamic adaptability to intermediate errors, leading to distorted analysis results. Traditional RAG methods also fall short in multi-step reasoning and aggregation analysis.

[0004] Therefore, the main challenge in current unstructured data analysis lies in how to automate the processing of complex queries and build an analytical framework that can automate plan generation, dynamic execution, and real-time adjustment. Summary of the Invention

[0005] The purpose of this application is to provide a method for analyzing unstructured data, which can automatically generate and orchestrate plans for complex natural language queries, and can also perform dynamic and interactive execution and make dynamic adjustments based on real-time results.

[0006] According to one embodiment of this application, a method for analyzing unstructured data is proposed, comprising:

[0007] Receive natural language queries input by the user as queries to be simplified;

[0008] A large language model is used to perform semantic parsing on the query to be simplified, generating a logical representation of the query to be simplified;

[0009] Based on the semantic similarity between the logical representation of the query to be simplified and the logical representation of multiple predefined standard semantic operators, candidate operators are selected, and each of the standard semantic operators has at least one logical representation;

[0010] Evaluate the degree to which candidate operators solve the query to be simplified and their feasibility, and reorder the candidate operators;

[0011] The candidate operators are applied sequentially according to the sorting results to simplify the query until the current query is successfully simplified, resulting in the simplified query.

[0012] Once it is determined that the current simplified query is the smallest indivisible semantic unit, an execution plan is constructed based on the query simplification process, and the execution plan is executed using a topological sorting-based parallel execution strategy, outputting the analysis results.

[0013] In some implementations, the plurality of standard semantic operators include some or all of the following operators: Retrieve, Filter, Aggregate, Validate, OrderBy, Compare, Refine, Translate, Transform, Scan, Extract, Conceptualize, Generate, Explain, GroupBy, Cluster, Classify, Link, SetOP, Integrate, Compute, and Summarize.

[0014] In some implementations, the logical representation of the standard semantic operator is a structured template obtained by replacing the specific values ​​of semantic elements in a natural language segment with corresponding placeholders, which is used to represent the logical meaning of the natural language segment. The placeholders include entity placeholders, condition placeholders, and attribute placeholders.

[0015] In some implementations, each of the standard semantic operators has at least one corresponding physical operator, which is implemented by a pre-programmed algorithm or by processing semantic operations using a large language model. Some or all of the multiple standard semantic operators simultaneously have physical operators implemented by pre-programmed algorithms and physical operators implemented by processing semantic operations using a large language model.

[0016] In some implementations, a large language model is used to perform semantic parsing of the query to be simplified, generating a logical representation of the query, including:

[0017] Provide prompt words to a large language model to instruct it to parse semantic elements from the query to be simplified, including entities, conditions, and attributes;

[0018] Large-scale language models replace the specific values ​​corresponding to entities, conditions, and attributes in the query to be simplified with corresponding entity placeholders, condition placeholders, and attribute placeholders to generate a structured logical representation.

[0019] In some implementations, candidate operators are selected by matching the logical representation of the query to be simplified with the logical representations of a number of predefined standard semantic operators, including:

[0020] Convert the logical representation of the query to be simplified into a query semantic embedding vector;

[0021] The query semantic embedding vector is used as the query vector, and a nearest neighbor search is performed in a pre-built vector index to calculate the similarity between the query semantic embedding vector and the operator semantic embedding vector. The vector index stores the operator semantic embedding vectors obtained by converting the logical representation of the standard semantic operator.

[0022] The standard semantic operators corresponding to the semantic embedding vectors of the operators with the highest similarity are selected as candidate operators.

[0023] In some implementations, evaluating the degree to which candidate operators solve the query to be simplified and their executability, and reordering the candidate operators, includes:

[0024] Use a large language model to check whether the preconditions for each candidate operator have been met;

[0025] Using a large language model, the degree to which each candidate operator that meets the preconditions solves the query to be simplified is evaluated, wherein the degree is completely solved, partially solved, or not solved at all.

[0026] Candidate operators are ranked based on their ability to solve the query to be simplified and the semantic similarity between the logical representation of the candidate operator and the logical representation of the query to be simplified.

[0027] In some implementations, candidate operators are applied sequentially according to the sorting results to simplify the query until the current query is successfully simplified, resulting in a simplified query, including:

[0028] Using a large language model, the expected output of the reordered candidate operator is applied sequentially to replace the matching fragment of the candidate operator in the query to be simplified, and it is determined whether the replaced query meets the preset criteria.

[0029] The first candidate operator that yields a query that meets the preset criteria after replacement is taken as the operator that is successfully applied, and the corresponding query that meets the preset criteria is taken as the simplified query.

[0030] Generate intermediate result data and corresponding natural language descriptions representing the execution results of successfully applied operators, and add them to the set of available variables for use by subsequent operators.

[0031] In some implementations, the method further includes, after obtaining the simplified query, determining whether the current simplified query is an indivisible minimal semantic unit by:

[0032] Using a large language model, determine whether the simplified query is an indivisible minimal semantic unit, which includes a single entity, a number, or a Boolean condition.

[0033] In some embodiments, the method further includes:

[0034] When it is determined that the current simplified query is not the smallest indivisible semantic unit, the current simplified query is taken as the new query to be simplified, and the semantic similarity-based matching, candidate operator reordering, and simplification of the query to be simplified are re-executed until it is determined that the current simplified query is the smallest indivisible semantic unit.

[0035] In some implementations, the execution plan is executed using a topology-based parallel execution strategy, including:

[0036] Topological sorting is performed based on the dependencies between operators, and operators that have met the preconditions and have no dependencies on each other are executed in parallel.

[0037] In some implementations, the method further includes, during the execution of the execution plan, if the intermediate result data generated does not meet expectations, adjusting the current execution plan, regenerating the execution plan, or rewriting the original natural language query to re-identify the relevant data.

[0038] In some embodiments, the method further includes steps of semantic parsing and preprocessing the unstructured data, including:

[0039] The input unstructured data is segmented into words, identified as entities, and semantically labeled to generate an initial semantic feature vector.

[0040] By using a large language model to perform contextual analysis on the initial semantic feature vector, an enhanced context-relevant semantic representation is obtained;

[0041] Store the enhanced context-sensitive semantic representation as a machine-readable structured intermediate format.

[0042] According to one embodiment of this application, an electronic device is provided, the device including a memory and a processor, the memory being used to store computer instructions executable on the processor, and the processor being used to implement the method as described in any of the preceding claims when executing the computer instructions.

[0043] According to one embodiment of this application, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method as described in any of the preceding claims.

[0044] The scheme for analyzing unstructured data provided in this application accepts analysis requests input by users through a natural language query interface. Upon receiving a query, it first performs semantic parsing with the assistance of a large language model to generate a structured logical representation. Subsequently, combined with predefined semantic analysis operators, the query execution plan is automatically constructed and optimized through automated operator matching, candidate operator reordering, and recursive query simplification. This structured processing approach avoids reliance on advanced programming skills, enabling even those without specialized backgrounds to easily perform complex unstructured data analysis tasks. It significantly improves the overall efficiency and accuracy of extracting valuable information from large-scale unstructured data, effectively overcoming the inefficiencies and insufficient accuracy of traditional methods in this regard.

[0045] Furthermore, the scheme for analyzing unstructured data provided in this application can understand and process queries involving multi-step reasoning and complex logical analysis, surpassing the limitations of traditional methods such as RAG, which can only handle simple queries. According to this application, query recursion is simplified into the smallest semantic units to construct the logical plan, and then executed efficiently through a parallel execution strategy based on topological sorting. In addition, during execution, this application can verify intermediate results and, when discrepancies are found, self-correction and optimization are performed by rewriting queries, re-identifying data, and dynamically adjusting the execution plan. This solves the problem that traditional static analysis pipelines cannot handle intermediate operation failures or uncertainties, thus ensuring the reliability of the analysis results.

[0046] Other features and advantages of the technical solution proposed in this application are described below. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0048] Figure 1 A flowchart of a method for analyzing unstructured data according to an embodiment of this application is shown.

[0049] Figure 2A schematic diagram of a planned generation system architecture is shown according to an exemplary embodiment of this application.

[0050] Figure 3 An example diagram of an automated plan generation method according to an exemplary embodiment of this application is shown.

[0051] Figure 4 A schematic diagram illustrating semantic parsing of a natural language query according to an exemplary embodiment of this application is shown.

[0052] Figure 5 A schematic diagram is shown of an exemplary embodiment of the present application, illustrating a semantic matching-based operator for matching a query.

[0053] Figure 6 A schematic diagram illustrating the reordering of candidate operators according to an exemplary embodiment of this application is shown.

[0054] Figure 7 A schematic diagram illustrating the application of candidate operators to simplify queries according to an exemplary embodiment of this application is shown.

[0055] Figure 8 A schematic diagram illustrating a determination of whether a query has been completely decomposed according to an exemplary embodiment of this application is shown.

[0056] Figure 9 A schematic diagram illustrating a complete query decomposition and plan generation according to an exemplary embodiment of this application is shown.

[0057] Figure 10 A schematic diagram of a planned execution process according to an exemplary embodiment of this application is shown.

[0058] Figure 11 This is a schematic diagram of the structure of an electronic device shown in at least one embodiment of this application. Detailed Implementation

[0059] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0060] The embodiments of this application can be applied to computer systems / servers that can operate with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems, etc.

[0061] Computer systems / servers can be described in the general context of computer system executable instructions (such as program modules) executed by the computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through a communication network. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0062] This application provides a method for analyzing unstructured data, referring to... Figure 1 The method mainly includes the following steps 1 to 6.

[0063] Step 1: Receive the natural language query input by the user as the query to be simplified.

[0064] Users can input their analysis needs for unstructured data through the system's query interface. These needs are typically expressed in natural language. For example, a user could enter: "Among questions with over 500 views, which sport involving a ball has the highest ratio of injury-related questions to training-related questions?" Upon receiving this raw natural language query, the system treats it as an initial query to be simplified, preparing for subsequent parsing and processing.

[0065] Step 2: Use a large language model to perform semantic parsing on the query to be simplified, and generate a logical representation of the query to be simplified.

[0066] Step 2 leverages the capabilities of large language models to transform queries to be simplified (e.g., queries in natural language form or containing natural language fragments) into structured logical representations that are easier for machines to understand and process.

[0067] In some implementations, the semantic parsing process may include:

[0068] A prompt is provided to the large language model to instruct it to parse semantic elements from the query to be simplified, including entities, conditions, and attributes. The large language model then replaces the specific values ​​of entities, conditions, and attributes in the query to be simplified with corresponding entity placeholders, condition placeholders, and attribute placeholders to generate a structured logical representation.

[0069] In some exemplary embodiments, the prompts provided to a large language model can also instruct it to analyze and determine the type of final result expected by the user's query (i.e., the 'return type'), such as determining whether the user wants a list, a numerical statistical result, or a Boolean judgment. Predicting the return type helps assist in the selection of subsequent operators, further improving the user experience and the targeting of the analysis.

[0070] For example, Figure 4 The example shows a query to be simplified as "Among questions with over 500 views, which sport involving a ball has the highest ratio of injury-related questions to training-related questions?". A prompt provided to the large language model could be: "Please parse the following question to extract the entities, conditions, attributes, and the return type."

[0071] For example, Figure 4 In the parsing results shown, “questions” and “sport” in the original query are marked as entities ([Entity]), and “with over 500views” and “involving a ball” are marked as conditions ([Condition]). Figure 4It vividly illustrates the process of semantic parsing.

[0072] Step 3: Match the logical representation of the query to be simplified with the logical representations of multiple predefined standard semantic operators to filter out candidate operators. Each of the standard semantic operators has at least one logical representation.

[0073] According to this application, a series of standardized semantic operators can be predefined. These operators may include common data operations in traditional databases (such as filtering and aggregation), and may also include semantic-level operations on unstructured data. In some embodiments, the predefined standard semantic operators may include some or all of the following: Retrieve, Filter, Aggregate, Validate, OrderBy, Compare, Refine, Translate, Transform, Scan, Extract, Conceptualize, Generate, Explain, GroupBy, Cluster, Classify, Link, SetOP, Integrate, Compute, and Summarize. The following table describes several predefined standard semantic operators according to an exemplary embodiment of this application.

[0074] Table 1. Predefined standard semantic operators and their descriptions

[0075]

[0076]

[0077] Each standard semantic operator has at least one logical representation. The logical representation embodies the core functionality and matching pattern of the operator. In some implementations, the logical representation of a semantic operator is a structured template obtained by replacing the specific values ​​of semantic elements that typically represent a class of natural language fragments with similar logical structures with corresponding placeholders. Placeholders can be entity placeholders, condition placeholders, attribute placeholders, etc. This logical representation is structured and can be used to clearly represent the logical meaning of these natural language fragments. For example, a filter operator might have the logical representation "[Entity]that[Condition]", which can match natural language fragments like "Movies that were made in the 2000s". A standard semantic operator can have multiple logical representations to identify and process diverse user natural language expressions. For example, semantic operators (Filters) can be defined with a variety of different logical representations, such as "[Entity]that[Condition]", "[Entity]having[Condition]", or "[Entity]thatsatisfies[Condition]", thereby accurately identifying the essentially the same filtering intent expressed by the user using different wording.

[0078] The definition of a standard semantic operator may also include input / output definitions and execution procedures.

[0079] The input and output definitions of standard semantic operators specify what kind of input data or parameters are required when the operator is executed (e.g., specific types of intermediate result data, constant values, etc.), and what kind of output results will be produced after execution (e.g., new intermediate result data, Boolean values, text, etc.).

[0080] The execution process defines how a semantic operator performs its function; each standard semantic operator can be executed through a corresponding physical operator. According to this application, the physical operator can be implemented in two main ways:

[0081] Method 1 uses physical operators implemented through pre-programmed algorithms, similar to operators in traditional databases. It employs deterministic algorithms for execution, resulting in higher efficiency and suitability for structured operations or simple semantic judgments.

[0082] Method 2 utilizes physical operators implemented through semantic operations using large language models. These operators are used to handle tasks requiring deep semantic understanding, reasoning, generation, or complex judgments, such as filtering documents in the sports field based on whether the document content involves "teamwork".

[0083] Among the various standard semantic operators, some or all operators (e.g., most of the semantic operators in Table 1) can simultaneously have implementations of both types of physical operators to enhance the system's flexibility in handling tasks of varying complexity.

[0084] The constructed set of semantic analysis operators provides the foundation for query understanding and plan generation.

[0085] After obtaining a structured logical representation of the query to be simplified through step 2, step 3 can identify which predefined standard semantic operators may be applicable to solving the current query or a part thereof.

[0086] In some implementations, selecting candidate operators may include:

[0087] The logical representation of the query to be simplified is converted into a query semantic embedding vector. This process can be achieved through a semantic embedding model. The resulting high-dimensional vector can capture the deep semantic meaning of the logical representation.

[0088] The query semantic embedding vector is used as the query vector, and a nearest neighbor search is performed in the pre-built vector index to calculate the similarity between the query semantic embedding vector and the operator semantic embedding vector. The vector index stores the operator semantic embedding vectors obtained by converting the logical representation of the standard semantic operator. In some examples, cosine similarity or dot product similarity can be used to measure the similarity between two semantic embedding vectors.

[0089] The standard semantic operators corresponding to the semantic embedding vectors of the operators with the highest similarity are used as candidate operators. For example, by selecting multiple candidate operators instead of a single candidate operator, more choices and flexibility can be provided for subsequent reordering and optimization steps.

[0090] Figure 5 The operator matching process is illustrated exemplarily. The semantic parsing result is a query with placeholders. The system calculates the similarity between its embedding vector and the embedding vectors of pre-stored operator logical representations (such as "How many[Entity]are[Condition]", "[Entity]satisfy[Condition]", "[Entity]that[Condition]", etc.), selects the operator with the highest similarity as the candidate operator, and sorts them according to the similarity value.

[0091] Step 4: Evaluate the degree to which candidate operators solve the query to be simplified and their feasibility, and reorder the candidate operators.

[0092] The candidate operators are further screened through a more detailed evaluation and reordering process, so that the selected operators are more suitable for simplifying the current query.

[0093] In some implementations, evaluation and reordering may include:

[0094] Use a large language model to check whether the preconditions for each candidate operator have been met;

[0095] Using a large language model, the degree to which each candidate operator that meets the preconditions solves the query to be simplified is evaluated, wherein the degree is completely solved, partially solved, or not solved at all.

[0096] Candidate operators are ranked based on their ability to solve the query to be simplified and the semantic similarity between the logical representation of the candidate operator and the logical representation of the query to be simplified.

[0097] A large language model can be used to check whether the preconditions for each candidate operator are met. The preconditions of an operator typically refer to the availability of the input data required for its execution. The system can maintain a dynamic set of available variables, which are generally intermediate results generated by the execution of other operators. The large language model can determine whether all the input variables required by the current candidate operator are included in this set. Only operators whose preconditions are met are considered currently executable.

[0098] For candidate operators that meet the preconditions, a large language model can be used to evaluate the extent to which they can solve or partially solve the current query to be simplified. The large language model can make judgments based on predefined prompts, and the output degree may include fully solving, partially solving, or not solving at all.

[0099] Finally, according to this application, candidate operators can be comprehensively ranked based on the above evaluation results and semantic similarity, among other factors. The primary ranking criterion is the degree to which a candidate operator solves the query to be simplified; the higher the degree of solution, the higher the ranking. Semantic similarity between the logical representation of a candidate operator and the logical representation of the query to be simplified can also be considered. For example, similarity can be used to break the ranking of operators with similar degrees of simplification. Through this re-ranking, an ordered list of operators that is more likely to efficiently and successfully simplify the query can be obtained.

[0100] Figure 6 The reordering process is illustrated by example.

[0101] Step 5: Apply candidate operators to simplify the query in order of sorting results until the current query is successfully simplified, and obtain the simplified query.

[0102] Step 5 is the core step in query plan generation. It attempts to simplify the current query by sequentially applying the reordered candidate operators, thereby finding and applying an operator that can effectively advance the query decomposition, and obtaining a simplified query that is closer to the final solution.

[0103] In some implementations, it may include:

[0104] Using a large language model, the expected output of the reordered candidate operator is applied sequentially to replace the matching fragment of the candidate operator in the query to be simplified, and it is determined whether the replaced query meets the preset criteria.

[0105] The first candidate operator that yields a query that meets the preset criteria after replacement is taken as the operator that is successfully applied, and the corresponding query that meets the preset criteria is taken as the simplified query.

[0106] Generate intermediate result data and corresponding natural language descriptions representing the execution results of successfully applied operators, and add them to the set of available variables for use by subsequent operators.

[0107] According to this embodiment, the candidate operators are strictly ordered according to the reordered list, starting with the highest priority operator. Using a large language model and few-sample example prompts, the expected outputs of these candidate operators are sequentially applied to replace segments in the query to be simplified that can be matched by that candidate operator. After each replacement attempt, it can be determined whether the replaced query meets preset criteria. Those skilled in the art can set preset criteria as needed, such as whether the generated query has a reasonable structure, whether it semantically correctly reflects the operation of the operator, whether it is simpler or closer to a solvable state than the original query, and whether the large language model completed the transformation with high confidence.

[0108] The above-mentioned trial process will continue until a query obtained by the replacement meets the preset criteria. The corresponding candidate operator is then regarded as the operator successfully applied in this iteration, and the query obtained that meets the preset criteria is the simplified query in this iteration. Figure 7 The application process is illustrated: using a large language model and prompts, a query is rewritten (i.e. simplified) by sequentially selected operators (such as filters) and their expected outputs, resulting in a simplified query.

[0109] Once the successfully applied operators and simplified queries are identified, intermediate result data representing the execution results of those operators can be generated. A corresponding natural language description can be generated using a large language model and added to the set of available variables for use by subsequent operators, for example... Figure 7The simplified intermediate result data is "questions with over 500 views", and its data type is text document.

[0110] The simplification process described above can be performed recursively. According to some implementations, after obtaining the simplified query, a large language model can be used to determine whether the simplified query is an indivisible minimal semantic unit, which includes a single entity, a numerical value, or a Boolean condition. For example, Figure 9 Ultimately, it was simplified to "Group Name", which can be regarded as a single entity.

[0111] If it is determined that the current simplified query is not the smallest semantic unit, it means that the query needs further decomposition and simplification. At this point, the process can return to... Figure 1 Step 2 in the process involves treating the current simplified query as a new query to be simplified, performing semantic parsing (Step 2), matching based on semantic similarity (Step 3), reordering candidate operators (Step 4), and simplifying the query to be simplified (Step 5) until it is determined that the current simplified query is the smallest indivisible semantic unit. In other words, the query can be considered to have been completely decomposed, and then the recursive process ends.

[0112] According to this implementation, through recursive simplification, a complex initial natural language query can be gradually decomposed and transformed layer by layer until a set of indivisible semantic units that form the basis of the execution plan are obtained.

[0113] Step 6: When it is determined that the current simplified query is the smallest indivisible semantic unit, an execution plan is constructed based on the query simplification process, and the execution plan is executed using a parallel execution strategy based on topological sorting, and the analysis results are output.

[0114] During query simplification, whenever an operator is successfully applied, its input (which may come from the original data or previous intermediate results) and output (new intermediate results) are recorded, forming a node in the plan, and the dependencies between these nodes are naturally established. After the entire recursive simplification process is completed, the recorded sequence of operator applications and their dependencies constitute a complete, executable logical query plan.

[0115] The execution plan can be executed using a topology-based parallel execution strategy. For example, topology sorting can be performed based on the dependencies between operators in the execution plan. Operators whose preconditions are met and which have no direct dependencies on each other can be scheduled to be executed in parallel on different execution units.

[0116] During execution, the intermediate results generated by the operators can be verified to determine whether they meet expectations. If the intermediate results are found to be inconsistent with expectations, different adjustment strategies can be adopted based on the degree of deviation and the reasons. These strategies include adjusting the current execution plan, regenerating the execution plan, or rewriting the original natural language query to re-identify the relevant data.

[0117] Once all operators in the execution plan have been successfully executed, the system will obtain the final analysis results and output them to the user.

[0118] Furthermore, the method of this application may also include semantic parsing and preprocessing of the unstructured data itself before or during the actual processing of the unstructured data by the operators in the execution plan. This preprocessing can be performed in advance or when the operators need to access the original data. The preprocessing steps may include:

[0119] The input unstructured data (such as text, documents, etc.) is segmented, entity recognized, and semantically labeled to generate an initial semantic feature vector;

[0120] Contextual analysis of these initial semantic feature vectors is performed using a large language model to obtain enhanced, context-sensitive semantic representations;

[0121] Furthermore, these enhanced context-sensitive semantic representations are stored in a machine-readable structured intermediate format, such as structured JSON / XML documents, so that subsequent analysis operators can efficiently query and use them. This data preprocessing step helps improve the accuracy and efficiency of subsequent semantic analysis.

[0122] To further clarify the technical solution of this application, exemplary embodiments according to this application will be described below with reference to the accompanying drawings.

[0123] Figure 2 A system architecture diagram is generated according to an exemplary embodiment of this application. Figure 2 This demonstrates the core architecture from user-inputted query, through processing by the plan generation module, to the final execution plan. Within the plan generation module, the system identifies relevant data for the query (e.g., searching via an "index") and utilizes a predefined series of standard semantic operators (including extraction, grouping, filtering, and comparison) to repeatedly execute a recursive simplification process involving operator matching, operator reordering, and query simplification until the query is fully simplified, thus constructing a complete execution plan.

[0124] Figure 3 This is an example diagram of an automated plan generation method according to an exemplary embodiment of this application. Figure 3Taking the natural language query "Among questions with over 500 views, which sport involving a ball has the highest ratio of number of injury-related questions to number of training-related questions?" as an example, the main steps of query plan generation are illustrated. First, the query is semantically parsed, transforming it into a logical representation where specific values ​​are replaced by corresponding placeholders. Next, operator matching is performed to obtain the logical representations of matching candidate operators, such as "How many [Entity] are [Condition]" and "[Entity] satisfy [Condition]". Then, the operators are reordered, and the candidate operators are applied sequentially according to the sorting results to simplify the query, for example, simplifying "Among questions with over 500 views" to "Among questions". The simplification process is recursive. After each simplification, if the query is not yet completely decomposed (i.e., the simplified query is not yet the smallest indivisible semantic unit), the simplified query is treated as a new query to be simplified, and the steps of semantic parsing, operator matching, operator reordering, and query simplification are repeated until the query is completely decomposed. Then, based on this query simplification process, a logical plan, i.e., an execution plan, is formed.

[0125] Figure 4 This is a schematic diagram illustrating semantic parsing of a natural language query according to an exemplary embodiment of this application. The natural language query to be parsed is "Among questions with over 500 views, which sport involving a ball has the highest ratio of injury-related questions to training-related questions?". The query is parsed using a large model by providing the prompt "Please parse the following question to extract the entities, conditions, attributes, and thereturn type." Figure 4The rightmost figure shows the parsing results, where identified entity semantic elements are marked in red and identified conditional semantic elements are marked in green. The specific values ​​of these identified entities and conditions can then be replaced by the corresponding placeholders [Entity] and [Condition] to generate the logical representation of the query.

[0126] Figure 5 This is a schematic diagram illustrating a semantic matching-based operator for matching a query according to an exemplary embodiment of this application. For example... Figure 5 As shown, after obtaining the semantic parsing results, the logical representation of the query can be converted into the query embedding vector, i.e., the query semantic embedding vector, and the similarity between this vector and the pre-calculated and stored embedding vectors of the logical representations of operators (i.e., operator semantic embedding vectors) can be calculated. Based on the calculated similarity values, the standard semantic operators with the highest similarity are selected as candidate operators. Figure 5 In the example shown, the candidate operators selected according to the similarity value are "How many[Entity]are[Condition]", "[Entity]satisfy[Condition]", and "[Entity]that[Condition]", with corresponding similarity values ​​of 0.83, 0.77, and 0.65, respectively.

[0127] Figure 6 This is a schematic diagram illustrating the reordering of candidate operators according to an exemplary embodiment of this application. Figure 6The left side of the image shows a set of candidate operators, including phrases like "How many [Entity] are [Condition]", "[Entity] satisfy [Condition]", and "[Entity] that [Condition]". The first step in the reordering process is to check the preconditions of the candidate operators. This can be done by referring to existing intermediate data to check if the preconditions are met, i.e., whether all the inputs required for a candidate operator have been added to the current set of available variables. Then, a prompt is provided to the large language model: "Please check whether the operator can solve any part of the query; if so, output the degree of solution (fully solving, partially solving); otherwise, output not solving." This prompt evaluates the degree to which the operator solves the query. The large language model outputs an evaluation result that can be fully solving, partially solving, or not solving. Finally, based on the degree to which each candidate operator solves the query to be simplified, as output by the large language model, and combined with the semantic similarity between the candidate operator and the query to be simplified, the candidate operators are reordered. Figure 6 In the example shown, the candidate operator that ranked second in semantic similarity was reordered to first place.

[0128] Figure 7This is a schematic diagram illustrating the use of candidate operators to simplify a query according to an exemplary embodiment of this application. The query to be simplified is "Among questions with over 500 views, which sport involving a ball has the highest ratio of number of injury-related questions to number of training-related questions?". First, the Filter operator is attempted, which is logically represented as "[Entity]satisfy[Condition]", and the expected output is [Entity]. The large language model is provided with the prompt "Given the query and a matched logical representation (LR) of operator (OP) with expected output (OUTPUT), please rewrite the query with the operator by reducing the matched segment of the logical representation." The large language model successfully applies this operator to simplify the query, reducing "Among questions with over 500 views" to "Among questions." It also generates intermediate result data representing the execution result of the successfully applied operator Filter, namely, intermediate result data "questions with over 500 views" in text document format. This data will be added to the set of available variables for use by subsequent related operators, such as as input or as a basis for determining whether preconditions are met.

[0129] Figure 8 This is a schematic diagram illustrating the determination of whether the current simplified query is the smallest indivisible semantic unit according to an exemplary embodiment of this application. Figure 8In the example shown, the prompt provided to the large model is "Check whether the initial query [Original query] has been fully resolved given the [Variable set description] and the current reduced query [Reduced query]." Here, the Original query is "Among questions with over 500 views, which sport involving a ball has the highest ratio of injury-related questions to training-related questions?", and the Reduced query is "Among questions, which sport involving a ball has the highest ratio of injury-related questions to training-related questions?". The Variable set description is a description of the currently available set of variables. Figure 8 In the example shown, the large model determines that the current simplified query still needs to be further decomposed. Therefore, the current simplified query "Among questions, which sport involving a ball has the highest ratio of number of injury-related questions to number of training-related questions?" will be used as the original query for the next round of simplification. After semantic parsing, operator matching, operator reordering and query simplification, the large model determines that the query has been completely decomposed, that is, the current simplified query is the smallest semantic unit that cannot be further decomposed.

[0130] Figure 9 This is a schematic diagram illustrating the complete query decomposition and plan generation according to an exemplary embodiment of this application. Figure 9The process of decomposing the natural language query “Among questions with over 500 views, which sport involving a ball has the highest ratio of number of injury-related questions to number of training-related questions?” is illustrated using a tree-like process (or directed acyclic graph structure).

[0131] Figure 9 The process shown mainly includes the following steps:

[0132] Step 1: Apply the Filter operator to the original query "Among questions with over 500 views, which sport involving a ball has the highest ratio of number of injury-related questions to number of training-related questions?" and use the condition "with over 500 views" to perform preliminary filtering of the questions.

[0133] Step 2: Apply the Extract operator to the filtered results from the previous step to extract information related to "sport".

[0134] Step 3: Apply the GroupBy operator to group the data according to "sport".

[0135] Step 4: Apply the Filter operator to the grouped data and use the condition "about sports involving a ball" to further filter groups related to ball-driving sports.

[0136] Step 5: At this point, the query needs to identify which group has the highest proportion of "injury-related" and "training-related" problems. Therefore, these two types of problems will be processed in parallel subsequently.

[0137] Step 6 (Left Branch): For "injury-related" problems, apply the Filter operator to filter them.

[0138] Step ⑦ (Left Branch): Apply the Count operator to the "injury-related" questions selected in Step ⑥ to obtain the count result, denoted as "number of questions1".

[0139] Step 8 (Right Branch): Similar to step 6, apply the Filter operator to filter for "training-related" problems.

[0140] Step 9 (Right Branch): Apply the Count operator to the "training-related" questions selected in Step 8 to obtain the count result, denoted as "number of questions2".

[0141] Step 10: Using the two counting results ("number of questions1" and "number of questions2") obtained in steps 7 and 9 as input, apply the Compute operator to calculate the ratio between them ("ratio of number1 to number2").

[0142] step For all the proportion results calculated in step 10, apply the Max operator to find the group with the highest proportion value, and finally output the name of the group, "Group Name".

[0143] The accompanying diagram clearly illustrates the complete process of recursively simplifying a complex analysis task by combining different semantic operators, breaking it down into a series of simpler, executable subtasks, and ultimately obtaining the desired analysis results.

[0144] Figure 10 This is a schematic diagram illustrating the execution flow of a plan according to an exemplary embodiment of this application. First, the generated execution plan is executed from bottom to top according to the topology. Intermediate result data will be generated during execution. A large model can be used to determine whether this intermediate result data meets expectations, and different processing strategies at different levels can be adopted based on the determination results.

[0145] Continue execution: If the intermediate results are largely in line with expectations, then execution can continue;

[0146] Adjustment plan: As shown in the figure, the adjustment is relatively small. It is based on the current execution plan and is adjusted. For example, some operators, parameters or connection relationships in the current execution plan are adjusted.

[0147] Replanning: When a significant flaw is found in the current plan, but the problem lies primarily in the planning logic itself rather than in the initial data identification, more in-depth adjustments can be made, such as regenerating part or all of the execution plan.

[0148] Re-identify relevant data: This strategy can be enabled when it is determined that the problem may stem from inaccurate or incomplete input data initially identified. For example, the original natural language query can be rewritten into a semantically equivalent but differently expressed form, and the modified query can be used to re-find and locate relevant data from the data source, and the plan can be re-made based on the new data foundation.

[0149] The method for analyzing unstructured data provided in this application significantly lowers the technical threshold for unstructured data analysis through techniques such as natural language query understanding, application of predefined semantic operators, automated operator matching and reordering, and simplification of recursive queries, enabling users to perform deep data insights without complex programming. This method can automatically construct and optimize execution plans for complex queries, greatly improving analysis efficiency and ensuring the accuracy of information extraction from massive amounts of unstructured data. Furthermore, the system also considers monitoring and adjusting the execution process at the execution level, further enhancing the reliability of the analysis results.

[0150] Figure 11 An electronic device provided in at least one embodiment of this application includes a memory and a processor. The memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the method for analyzing unstructured data as described in any embodiment or implementation of this application when executing the computer instructions.

[0151] At least one embodiment of this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for analyzing unstructured data as described in any embodiment or implementation of this application.

[0152] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0153] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the data processing device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0154] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0155] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0156] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0157] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0158] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0159] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0160] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0161] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0162] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the scope of protection of one or more embodiments of this specification.

Claims

1. A method for analyzing unstructured data, characterized in that, include: Receive natural language queries input by the user as queries to be simplified; A large language model is used to perform semantic parsing on the query to be simplified, generating a logical representation of the query to be simplified; Based on the semantic similarity between the logical representation of the query to be simplified and the logical representation of multiple predefined standard semantic operators, candidate operators are selected, and each of the standard semantic operators has at least one logical representation; Evaluate the degree to which candidate operators solve the query to be simplified and their feasibility, and reorder the candidate operators; The candidate operators are applied sequentially according to the sorting results to simplify the query until the current query is successfully simplified, resulting in the simplified query. When it is determined that the current simplified query is an indivisible minimal semantic unit, an execution plan is constructed based on the query simplification process, and the execution plan is executed using a topological sorting-based parallel execution strategy. The analysis results are then output. During the execution of the execution plan, a large language model is used to determine whether the generated intermediate result data meets expectations. If the generated intermediate result data does not meet expectations, adjustments are made to the current execution plan, the execution plan is regenerated, or the original natural language query is rewritten to re-identify relevant data. When it is determined that the current simplified query is not an indivisible minimal semantic unit, the current simplified query is treated as a new query to be simplified. The query to be simplified is then subjected to semantic parsing, semantic similarity-based matching, candidate operator reordering, and simplification until it is determined that the current simplified query is an indivisible minimal semantic unit.

2. The method according to claim 1, characterized in that, The aforementioned standard semantic operators include some or all of the following operators: retrieval, filtering, aggregation, validation, sorting, comparison, optimization, translation, transformation, scanning, extraction, conceptualization, generation, interpretation, grouping, clustering, classification, linking, set operations, integration, computation, and summarization.

3. The method according to claim 1, characterized in that, The logical representation of the standard semantic operator is a structured template obtained by replacing the specific values ​​of semantic elements in a natural language segment with corresponding placeholders. This template is used to represent the logical meaning of the natural language segment. The placeholders include entity placeholders, condition placeholders, and attribute placeholders.

4. The method according to claim 1, characterized in that, Each of the standard semantic operators has at least one corresponding physical operator, which is implemented by a pre-programmed algorithm or by processing semantic operations using a large language model. Some or all of the multiple standard semantic operators simultaneously have physical operators implemented by pre-programmed algorithms and physical operators implemented by processing semantic operations using a large language model.

5. The method according to claim 1, characterized in that, A large language model is used to perform semantic parsing on the query to be simplified, generating a logical representation of the query, including: Provide prompt words to a large language model to instruct it to parse semantic elements from the query to be simplified, including entities, conditions, and attributes; Large-scale language models replace the specific values ​​corresponding to entities, conditions, and attributes in the query to be simplified with corresponding entity placeholders, condition placeholders, and attribute placeholders to generate a structured logical representation.

6. The method according to claim 1, characterized in that, Based on the semantic similarity between the logical representation of the query to be simplified and the logical representations of several predefined standard semantic operators, candidate operators are selected, including: Convert the logical representation of the query to be simplified into a query semantic embedding vector; The query semantic embedding vector is used as the query vector, and a nearest neighbor search is performed in a pre-built vector index to calculate the similarity between the query semantic embedding vector and the operator semantic embedding vector. The vector index stores the operator semantic embedding vectors obtained by converting the logical representation of the standard semantic operator. The standard semantic operators corresponding to the semantic embedding vectors of the operators with the highest similarity are selected as candidate operators.

7. The method according to claim 1, characterized in that, Evaluate the degree to which candidate operators solve the query to be simplified and their executability, and reorder the candidate operators, including: Use a large language model to check whether the preconditions for each candidate operator have been met; Using a large language model, the degree to which each candidate operator that meets the preconditions solves the query to be simplified is evaluated, wherein the degree is completely solved, partially solved, or not solved at all. Candidate operators are ranked based on their ability to solve the query to be simplified and the semantic similarity between the logical representation of the candidate operator and the logical representation of the query to be simplified.

8. The method according to claim 1, characterized in that, Candidate operators are applied sequentially according to the sorting results to simplify the query until the current query is successfully simplified, resulting in the simplified query, including: Using a large language model, the expected output of the reordered candidate operator is applied sequentially to replace the matching fragment of the candidate operator in the query to be simplified, and it is determined whether the replaced query meets the preset criteria. The first candidate operator that yields a query that meets the preset criteria after replacement is taken as the operator that is successfully applied, and the corresponding query that meets the preset criteria is taken as the simplified query. Generate intermediate result data and corresponding natural language descriptions representing the execution results of successfully applied operators, and add them to the set of available variables for use by subsequent operators.

9. The method according to claim 1, characterized in that, The method further includes, after obtaining the simplified query, determining whether the current simplified query is an indivisible minimal semantic unit by means of the following: Using a large language model, determine whether the simplified query is an indivisible minimal semantic unit, which includes a single entity, a number, or a Boolean condition.

10. The method according to claim 1, characterized in that, The execution plan is executed using a topology-based parallel execution strategy, including: Topological sorting is performed based on the dependencies between operators, and operators that have met the preconditions and have no dependencies on each other are executed in parallel.

11. The method according to claim 1, characterized in that, The method further includes steps for semantic parsing and preprocessing of unstructured data, including: The input unstructured data is segmented into words, identified as entities, and semantically labeled to generate an initial semantic feature vector. By using a large language model to perform contextual analysis on the initial semantic feature vector, an enhanced context-relevant semantic representation is obtained; Store the enhanced context-sensitive semantic representation as a machine-readable structured intermediate format.

12. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store computer instructions executable on the processor, and the processor being used to implement the method of any one of claims 1 to 11 when executing the computer instructions.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method described in any one of claims 1 to 11.

Images