AI large model intelligent question and answer cloud analysis method and system

Abstract

This invention discloses an AI large-scale intelligent question cloud analysis method and system, comprising: a question preprocessing module for acquiring and preprocessing natural language question requests, and simultaneously establishing a data asset set in the cloud; a semantic retrieval module for recalling and re-ranking candidate semantic elements and constructing a training sample set; a model fine-tuning module for performing multiple rounds of fine-tuning on the LLaMA model, and performing semantic parsing and multiple rounds of planning to form a question intent field; a path selection module for performing bidirectional shortest path search on the data asset set in the cloud and selecting the optimal execution path as the question execution path; an operator compilation module for cost optimization compilation to generate a cloud-executable task instruction sequence; and a feedback optimization module for collecting feedback data and performing incremental fine-tuning updates. This invention enables the understanding of natural language question intent and improves the accuracy of question intent parsing.

Description

Technical Field

[0001] This invention relates to the field of question analysis technology, and in particular to an AI large-scale intelligent question cloud analysis method and system. Background Technology

[0002] As enterprise data assets continue to grow, data tables, metrics, and dimensions exhibit characteristics of multi-source heterogeneity and continuous evolution. To meet the business side's need for rapid access to analytical results, existing systems typically provide a natural language-based query entry point, converting user questions into query statements and executing them on a cloud-based data platform, thereby lowering the analytical threshold and shortening the data retrieval process.

[0003] The mainstream implementation path of existing question-and-data solutions mostly adopts template rules or lightweight semantic matching to achieve the alignment of "question - field / metric". Due to the ambiguity of business expressions, the widespread existence of synonyms, and the significant differences in metric definitions and dimension levels between different departments, alignment methods based solely on keyword matching or static rules are difficult to reliably cover complex scenarios, and are prone to problems such as incomplete field recall, inaccurate candidate matching and sorting, and misselection of metric dimensions.

[0004] Furthermore, existing technologies lack a consistent, structured representation throughout the "intent resolution—execution path—query plan" chain. While some solutions can generate SQL or operator plans, they do not adequately characterize the connections between data assets, the availability of join keys, and permission constraints.

[0005] Meanwhile, existing question counting systems rely heavily on offline annotation or manual regression testing for model updates, making it difficult to fully utilize cloud-based execution feedback to form a continuous optimization loop. Even when logs of execution status, time consumption, and resource usage are collected, they often remain at the monitoring level, lacking a mechanism to transform feedback into preferred training signals and drive incremental fine-tuning of large models.

[0006] Therefore, how to provide AI large-scale model intelligent data cloud analysis methods and systems is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] One objective of this invention is to propose an AI-based large-scale intelligent question analysis method and system. This invention fully utilizes large language model fine-tuning technology, semantic vector retrieval, data asset association indexing, and a cloud-based execution feedback-driven direct preference optimization mechanism to construct a complete technical system from natural language question understanding, execution path search, operator-level query optimization to model incremental updates. It describes in detail the question semantic parsing method based on the large model, the data asset association path search method, and the connection order optimization based on genetic algorithms and the model continuous optimization method based on preference alignment. It has the advantages of high accuracy in question understanding, stable execution path selection, high query execution efficiency, and strong model adaptability.

[0008] The AI ​​large-scale intelligent data cloud analysis method and system according to embodiments of the present invention includes: The question preprocessing module is used to acquire natural language question requests and preprocess them to form a question element sequence, while simultaneously building a data asset collection in the cloud; The semantic retrieval module is used to recall and reorder candidate semantic elements in the question element sequence, and to construct a training sample set based on the reordering results according to semantic alignment rules. The model fine-tuning module is used to perform multiple rounds of fine-tuning on the LLaMA model based on the training sample set, and to use the fine-tuned model to perform semantic parsing on the question element sequence, and to perform multiple rounds of planning and reasoning based on the preset question task template to form the question intent field. The path selection module is used to expand the associated index of the data asset set in the cloud. Starting from the question intent field, it performs bidirectional shortest path search in the associated index structure to generate candidate execution paths, and calculates the path score using a semantic cost fusion scoring method. At the same time, it selects the optimal execution path as the question execution path based on the path score. The operator compilation module is used to map the query execution path to a sequence of query operators, and to perform cost optimization compilation on the sequence of query operators to generate a sequence of task instructions that can be executed in the cloud. The feedback optimization module is used to collect feedback data from the execution of task instruction sequences in the cloud and to perform incremental fine-tuning updates on the LLaMA model.

[0009] Optionally, modules can be integrated using the following methods: S1. Obtain natural language question requests and preprocess them to form a question element sequence, while simultaneously establishing a data asset collection in the cloud; S2. The vector semantic retrieval method is used to recall and reorder the candidate semantic elements of the question element sequence, and a training sample set is constructed based on the reordering results according to the semantic alignment rules. S3. Based on the training sample set, perform multiple rounds of fine-tuning on the LLaMA model, and use the fine-tuned model to perform semantic parsing on the question element sequence. Perform multiple rounds of planning and reasoning according to the preset question task template to form the question intent field. S4. Expand the data asset set with association index in the cloud. Starting from the question intent field, perform bidirectional shortest path search in the association index structure to generate candidate execution paths. Calculate the path score using the semantic cost fusion scoring method. Select the optimal execution path as the question execution path based on the path score. S5. An operator generation method based on intermediate representation is adopted to map the query execution path into a query operator sequence, and a connection order optimization method based on genetic algorithm is used to optimize the cost of the query operator sequence and generate a task instruction sequence that can be executed in the cloud. S6. Collect feedback data of the task instruction sequence executed in the cloud, and perform incremental fine-tuning updates on the LLaMA model through the Direct Preference Optimization (DPO) mechanism based on preference alignment.

[0010] Optionally, the preprocessing specifically includes: The system performs character normalization and noise removal on natural language query requests, performs word segmentation and word form merging on the cleaned text, performs entity recognition and terminology normalization on the word segmentation results, and labels the identified elements according to indicators, dimensions, time and conditions. Time window parsing and caliber conversion are performed on time elements, logical relationship parsing and predicate standardization are performed on condition elements, and the various elements that have been annotated and parsed are spliced ​​together according to preset serialization rules to obtain the question element sequence.

[0011] Optionally, the data asset set in the cloud consists of queryable data asset entries, each of which includes at least an asset type, an asset identifier, and a semantic name, and records association information for expanding the associated index for each data asset entry; The associated information includes at least the primary key field set, foreign key field set, join key field set, and partition field identifier of table-level assets, as well as the caliber identifier and aggregation rule identifier of indicator-level assets, the hierarchical identifier and value domain identifier of dimension-level assets, and the access control identifier of permission-level assets. Based on the aforementioned association information, a forward association index and a reverse association index are established in the cloud. The forward association index is used to locate connectable adjacent asset entries from the source asset entry, and the reverse association index is used to trace back to the connectable source asset entry from the adjacent asset entry.

[0012] Optionally, S2 specifically includes: S21. Vectorize the elements in the question element sequence to obtain element vectors, and vectorize the semantic names of data asset entries to obtain asset vectors. Use the element vectors as retrieval vectors to perform Top-K similarity recall on the asset vectors to generate a candidate data set. S22. Reorder the candidate data set, calculate the reordering score for each candidate data, the reordering score is obtained by weighting the similarity score between the element vector and the asset vector and the consistency score of the asset type, and sort the candidate data asset item set according to the reordering score to obtain the reordered candidate set. S23. Determine the alignment relationship between elements and data asset entries in the reordering candidate set according to semantic alignment rules. The semantic alignment rules include fusion score threshold constraints and one-to-one alignment constraints. The Hungarian matching algorithm is used to solve for the alignment results on candidate pairs that satisfy the fusion score threshold constraints. S24. Based on the alignment results, construct a training sample set, assemble the elements in the alignment results with the corresponding data asset entries into positive sample pairs, and select unaligned data asset entries from the reordering candidate set and assemble them with the same element into negative sample pairs. Merge the positive sample pairs and negative sample pairs to obtain the training sample set.

[0013] Optionally, S3 specifically includes: S31. The training sample set is templated and constructed. The question element sequence of each training sample is assembled into an instruction input segment, and the target intent field corresponding to the training sample is assembled into a structured output segment. The byte pair encoding BPE word segmentation algorithm is used to uniformly encode the input segment and the output segment to form an instruction sample. S32. The LLaMA model is updated in multiple rounds using a parameter-efficient fine-tuning method, and the stability of model parameters and consistency of output structure are jointly constrained in each round of fine-tuning. S33. Perform stepwise decoding on the question element sequence, expand the candidate sequence by branching according to the next word probability output by the fine-tuned LLaMA model and retain high-scoring candidates until the end marker of the structured output segment is generated, and obtain multiple sets of candidate intent output sequences, and parse them into a set of candidate question intent fields according to the structured field delimiter. S34. Based on the preset question task template, read the list of intent fields and the generation order. For the missing fields in the candidate question intent fields, form a list of fields to be completed. Based on the question element sequence and the list of fields to be completed, use the fine-tuned LLaMA model to generate field values ​​and backfill them to obtain the candidate question intent field set. S35. Perform consistency selection on the candidate question intent field set after multiple rounds of backfilling, and select the optimal candidate question intent field as the question intent field based on the number of consistent occurrences. Optionally, S34 specifically includes: S341. Read the preset question task template and parse it to obtain the list of intent fields and the order of field generation. The preset question task template records the list of fields and the order of field generation of indicator identifier, dimension identifier, time window identifier and condition identifier in the form of field configuration items. It also configures the value source identifier and field dependency identifier for each field. It traverses the candidate question intent fields according to the order of field generation and identifies missing fields to form a list of fields to be completed. S342. For each field in the list of fields to be completed, extract the assigned fields associated with the field based on the field dependency identifier, encode the assigned fields into key value sequences, and assemble the question element sequence and key value sequences into a generation input sequence. Locate the target field start mark in the generation input sequence and call the fine-tuned LLaMA model to generate up to the field end mark. Register the generated fragment between the start mark and the end mark as the field value set. S343. Bind the field values ​​to the set of field values, write the field values ​​into the candidate question intent field according to the field identifier and complete the backfilling, and update the missing field marker and conflict field marker of the candidate question intent field after backfilling; S344. When conflict field markers exist, locate conflict field pairs based on field mutual exclusion identifiers, assemble conflict field pairs and question element sequences into conflict resolution input sequences, input them into the fine-tuned LLaMA model to generate replacement field values, and use replacement field values ​​to cover conflict field values ​​in candidate question intent fields. S345. Repeat the field completion, backfilling, and conflict resolution steps until the list of fields to be completed is empty or the preset number of rounds is reached, to obtain a set of candidate question intent fields after multiple rounds of backfilling.

[0014] Optionally, S4 specifically includes: S41. On the cloud, based on the forward and reverse association indexes, adjacent entries are expanded to generate a set of adjacent entry pairs. For each adjacent entry pair, the corresponding set of connection key fields and access control identifiers are associated to obtain the association index structure. S42. Locate the starting point item set in the data asset items based on the question intent field. The starting point item set includes data asset items that match the indicator identifier, dimension identifier, time window identifier and condition identifier in the question intent field. Locate the data asset items that can be connected to the starting point item set in the associated index structure as the candidate endpoint item set. S43. In the associated index structure, a bidirectional shortest path search is performed with the starting entry set as the forward search boundary and the candidate ending entry set as the reverse search boundary to generate a candidate execution path set. The candidate execution path is composed of a sequence of data asset entries that can be connected along the forward and reverse associated indexes. S44. Calculate the path score for each candidate execution path in the candidate execution path set using a semantic cost fusion scoring method, specifically including: The semantic matching cost is calculated based on the semantic names of the question intent field and the data asset entries in the candidate execution path. The semantic matching cost is determined by the similarity value between the question intent field vector and the semantic name vector. The available cost of a connection is calculated based on the matching of the connection key field sets of adjacent entries in the candidate execution path. The available cost of a connection is determined by the number of overlaps and the number of missing connection key field sets. The permission risk cost is calculated based on the access control identifiers in the candidate execution path, and the permission risk cost is determined by the number of data asset entries with restricted access in the path. The semantic matching cost, connection availability cost, and permission risk cost are linearly weighted and fused according to preset weights to obtain the path score of the candidate execution path; S45. Sort the candidate execution path set according to the path score, specifically including: When there is a candidate execution path with the unique optimal path score in the candidate execution path set, select that candidate execution path as the question execution path; When multiple candidate execution paths in the candidate execution path set have the same path score or the difference in path score is less than a preset difference threshold, the candidate execution paths are sorted in a secondary order based on their path length, and the candidate execution path with the shortest path length is selected as the question execution path. If there are still parallel candidate execution paths after the second-level sorting, the third-level sorting is performed based on the available cost of connection and the risk cost of permissions of the parallel candidate execution paths, and the candidate execution path with the best score is selected as the question execution path.

[0015] Optionally, S5 specifically includes: S51. Perform operator structure mapping on the data asset item sequence in the query execution path, convert adjacent data asset item pairs into connection operators, map the condition identifier and time window identifier in the query intent field into filter operators, map the indicator identifier and dimension identifier into aggregation grouping operators, and construct a query operator sequence. S52. Encode the connection order of the connection operators in the query operator sequence, encode the data asset entries participating in the connection as a permutation sequence, and use the permutation sequence as the individual representation of the genetic algorithm to construct the initial population set; S53. Perform genetic iteration to update the connection permutation sequence of the initial population set. Calculate the fitness score of the permutation sequence in each iteration and generate a new generation population set accordingly. S54. Based on the optimal join permutation sequence, perform join rearrangement on the query operator sequence and complete cost optimization compilation, construct operator dependencies and use topological sorting algorithm to obtain linear execution sequence, encode the linear execution sequence into task instruction sequence according to preset instruction encoding rules, and obtain cloud-executable task instruction sequence.

[0016] Optionally, S6 specifically includes: S61. Collect feedback data of the task instruction sequence executed in the cloud, and associate and record the feedback data with the corresponding question element sequence and task instruction sequence. The feedback data includes execution status, execution time, resource consumption statistics and result verification conclusions. S62. Based on feedback data, under the condition of the same number of question element sequences, multiple task instruction sequences are grouped according to the result verification conclusion, and the Elo scoring algorithm is used to rank the task instruction sequences according to their merits. The task instruction sequences with higher ranking are determined as preference samples to form a preference training pair set. S63. Based on the set of preference training pairs, under the condition of the same question element sequence, the preference sample task instruction sequence and the non-preference sample task instruction sequence are respectively used as the target sequence to drive the LLaMA model to perform autoregressive forward propagation, extract the conditional log probability along the word position of the target sequence and accumulate to obtain the preference sample score and the non-preference sample score, and associate the score with the corresponding preference training pair identifier and record it as a sample score set. S64. Calculate the DPO preference alignment loss based on the sample rating set, and perform gradient backpropagation on the preference alignment loss to complete the incremental update of the trainable parameters of the LLaMA model.

[0017] The beneficial effects of this invention are: First, this invention preprocesses natural language into a sequence of question elements on the question request side, and constructs a training sample set by combining vector semantic retrieval with reordering and semantic alignment mechanisms. After driving the LLaMA model to fine-tune, it performs semantic parsing and task template planning reasoning on the question elements, and outputs a structured question intent field, reducing mismatches and intent drift caused by incomplete field recall and synonym ambiguity.

[0018] Secondly, this invention expands the relationship between adjacent entries on the cloud data asset side based on the forward and reverse association indexes, performs bidirectional shortest path search around the question intent field to form candidate execution paths, and adopts a semantic cost fusion scoring method to sort and filter based on semantic matching, connection availability and permission risk.

[0019] Finally, this invention maps the query execution path to a query operator sequence and optimizes the connection order using a genetic algorithm to generate a cloud-executable task instruction sequence. At the same time, it collects cloud execution feedback to construct preference training pairs and uses the DPO mechanism to perform incremental fine-tuning updates on the LLaMA model. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1This is a module structure diagram of the AI ​​large-scale intelligent data cloud analysis method and system proposed in this invention; Figure 2 This is a flowchart of the AI ​​large-scale intelligent data cloud analysis method and system proposed in this invention; Figure 3 This diagram illustrates the construction of the training sample set for the AI ​​large-scale intelligent question cloud analysis method and system proposed in this invention. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0022] refer to Figure 1 AI large-scale model intelligent data cloud analysis methods and systems, including: The question preprocessing module is used to acquire natural language question requests and preprocess them to form a question element sequence, while simultaneously building a data asset collection in the cloud; The semantic retrieval module is used to recall and reorder candidate semantic elements in the question element sequence, and to construct a training sample set based on the reordering results according to semantic alignment rules. The model fine-tuning module is used to perform multiple rounds of fine-tuning on the LLaMA model based on the training sample set, and to use the fine-tuned model to perform semantic parsing on the question element sequence, and to perform multiple rounds of planning and reasoning based on the preset question task template to form the question intent field. The path selection module is used to expand the associated index of the data asset set in the cloud. Starting from the question intent field, it performs bidirectional shortest path search in the associated index structure to generate candidate execution paths, and calculates the path score using a semantic cost fusion scoring method. At the same time, it selects the optimal execution path as the question execution path based on the path score. The operator compilation module is used to map the query execution path to a sequence of query operators, and to perform cost optimization compilation on the sequence of query operators to generate a sequence of task instructions that can be executed in the cloud. The feedback optimization module is used to collect feedback data from the execution of task instruction sequences in the cloud and to perform incremental fine-tuning updates on the LLaMA model.

[0023] refer to Figure 2-3 In this embodiment, the modules are interconnected using the following method: S1. Obtain natural language question requests and preprocess them to form a question element sequence, while simultaneously establishing a data asset collection in the cloud; S2. The vector semantic retrieval method is used to recall and reorder the candidate semantic elements of the question element sequence, and a training sample set is constructed based on the reordering results according to the semantic alignment rules. S3. Based on the training sample set, perform multiple rounds of fine-tuning on the LLaMA model, and use the fine-tuned model to perform semantic parsing on the question element sequence. Perform multiple rounds of planning and reasoning according to the preset question task template to form the question intent field. S4. Expand the data asset set with association index in the cloud. Starting from the question intent field, perform bidirectional shortest path search in the association index structure to generate candidate execution paths. Calculate the path score using the semantic cost fusion scoring method. Select the optimal execution path as the question execution path based on the path score. S5. An operator generation method based on intermediate representation is adopted to map the query execution path into a query operator sequence, and a connection order optimization method based on genetic algorithm is used to optimize the cost of the query operator sequence and generate a task instruction sequence that can be executed in the cloud. S6. Collect feedback data of the task instruction sequence executed in the cloud, and perform incremental fine-tuning updates on the LLaMA model through the Direct Preference Optimization (DPO) mechanism based on preference alignment.

[0024] In this embodiment, the preprocessing specifically includes: The system performs character normalization and noise removal on natural language query requests, performs word segmentation and word form merging on the cleaned text, performs entity recognition and terminology normalization on the word segmentation results, and labels the identified elements according to indicators, dimensions, time and conditions. Time window parsing and caliber conversion are performed on time elements, logical relationship parsing and predicate standardization are performed on condition elements, and the various elements that have been annotated and parsed are spliced ​​together according to preset serialization rules to obtain the question element sequence.

[0025] In this embodiment, the data asset set is composed of queryable data asset entries in the cloud. Each data asset entry includes at least an asset type, an asset identifier, and a semantic name, and records association information for expanding the associated index for each data asset entry. The associated information includes at least the primary key field set, foreign key field set, join key field set, and partition field identifier of table-level assets, as well as the caliber identifier and aggregation rule identifier of indicator-level assets, the hierarchical identifier and value domain identifier of dimension-level assets, and the access control identifier of permission-level assets. Based on the aforementioned association information, a forward association index and a reverse association index are established in the cloud. The forward association index is used to locate connectable adjacent asset entries from the source asset entry, and the reverse association index is used to trace back to the connectable source asset entry from the adjacent asset entry.

[0026] In this embodiment, S2 specifically includes: S21. Vectorize the elements in the question element sequence to obtain element vectors, and vectorize the semantic names of data asset entries to obtain asset vectors. Use the element vectors as retrieval vectors to perform Top-K similarity recall on the asset vectors to generate a candidate data set. S22. Reorder the candidate data set, calculate the reordering score for each candidate data, the reordering score is obtained by weighting the similarity score between the element vector and the asset vector and the consistency score of the asset type, and sort the candidate data asset item set according to the reordering score to obtain the reordered candidate set. S23. Determine the alignment relationship between elements and data asset entries in the reordering candidate set according to semantic alignment rules. The semantic alignment rules include fusion score threshold constraints and one-to-one alignment constraints. The Hungarian matching algorithm is used to solve for the alignment results on candidate pairs that satisfy the fusion score threshold constraints. S24. Based on the alignment results, construct a training sample set, assemble the elements in the alignment results with the corresponding data asset entries into positive sample pairs, and select unaligned data asset entries from the reordering candidate set and assemble them with the same element into negative sample pairs. Merge the positive sample pairs and negative sample pairs to obtain the training sample set.

[0027] In this embodiment, S3 specifically includes: S31. The training sample set is templated and constructed. The question element sequence of each training sample is assembled into an instruction input segment, and the target intent field corresponding to the training sample is assembled into a structured output segment. The byte pair encoding BPE word segmentation algorithm is used to uniformly encode the input segment and the output segment to form an instruction sample. S32. Employ a parameter-efficient fine-tuning method to perform multiple rounds of fine-tuning updates on the LLaMA model, and jointly constrain the stability of model parameters and the consistency of output structure during each round of fine-tuning, specifically including: In the LLaMA model, a preset linear transformation layer is selected, and low-rank adaptation parameters are injected into each selected linear transformation layer and the original model parameters are frozen. The low-rank adaptation parameters include a dimension reduction matrix and a dimension increase matrix. The instruction samples are trimmed and padded to generate training batches, and iterative training is performed on the training batches according to a preset number of rounds. In each training iteration, teacher-forced decoding is performed on the instruction input segment, autoregressive cross-entropy loss is calculated based on the structured output segment, and gradient backpropagation is performed on the loss to update the low-rank adaptation parameters. Before updating the gradient, perform norm clipping on the gradient and apply weight decay regularization to the low-rank adaptation parameters; After each round of updates, the structure validator is called to perform structure verification on the model output. The structure verification determines the consistency of field order constraints, field boundary mark constraints, and field value range constraints in the output segment. When the structure verification meets the preset number of times and the field coverage of the output segment is not lower than the preset threshold, the subsequent training rounds are terminated and the fine-tuned LLaMA model is output. S33. Perform stepwise decoding on the question element sequence, expand the candidate sequence by branching according to the next word probability output by the fine-tuned LLaMA model and retain high-scoring candidates until the end marker of the structured output segment is generated, and obtain multiple sets of candidate intent output sequences, and parse them into a set of candidate question intent fields according to the structured field delimiter. S34. Based on the preset question task template, read the list of intent fields and the generation order. For the missing fields in the candidate question intent fields, form a list of fields to be completed. Based on the question element sequence and the list of fields to be completed, use the fine-tuned LLaMA model to generate field values ​​and backfill them to obtain the candidate question intent field set. S35. Perform consistency selection on the candidate question intent field set after multiple rounds of backfilling, and select the optimal candidate question intent field as the question intent field based on the number of consistent occurrences. In this embodiment, S34 specifically includes: S341. Read the preset question task template and parse it to obtain the list of intent fields and the order of field generation. The preset question task template records the list of fields and the order of field generation of indicator identifier, dimension identifier, time window identifier and condition identifier in the form of field configuration items. It also configures the value source identifier and field dependency identifier for each field. It traverses the candidate question intent fields according to the order of field generation and identifies missing fields to form a list of fields to be completed. S342. For each field in the list of fields to be completed, extract the assigned fields associated with the field based on the field dependency identifier, encode the assigned fields into key value sequences, and assemble the question element sequence and key value sequences into a generation input sequence. Locate the target field start mark in the generation input sequence and call the fine-tuned LLaMA model to generate up to the field end mark. Register the generated fragment between the start mark and the end mark as the field value set. S343. Bind the field values ​​to the set of field values, write the field values ​​into the candidate question intent field according to the field identifier and complete the backfilling, and update the missing field marker and conflict field marker of the candidate question intent field after backfilling; S344. When conflict field markers exist, locate conflict field pairs based on field mutual exclusion identifiers, assemble conflict field pairs and question element sequences into conflict resolution input sequences, input them into the fine-tuned LLaMA model to generate replacement field values, and use replacement field values ​​to cover conflict field values ​​in candidate question intent fields. S345. Repeat the field completion, backfilling, and conflict resolution steps until the list of fields to be completed is empty or the preset number of rounds is reached, to obtain a set of candidate question intent fields after multiple rounds of backfilling.

[0028] In this embodiment, S4 specifically includes: S41. On the cloud, based on the forward and reverse association indexes, adjacent entries are expanded to generate a set of adjacent entry pairs. For each adjacent entry pair, the corresponding set of connection key fields and access control identifiers are associated to obtain the association index structure. S42. Locate the starting point item set in the data asset items based on the question intent field. The starting point item set includes data asset items that match the indicator identifier, dimension identifier, time window identifier and condition identifier in the question intent field. Locate the data asset items that can be connected to the starting point item set in the associated index structure as the candidate endpoint item set. S43. In the associated index structure, a bidirectional shortest path search is performed with the starting entry set as the forward search boundary and the candidate ending entry set as the reverse search boundary to generate a candidate execution path set. The candidate execution path is composed of a sequence of data asset entries that can be connected along the forward and reverse associated indexes. S44. Calculate the path score for each candidate execution path in the candidate execution path set using a semantic cost fusion scoring method, specifically including: The semantic matching cost is calculated based on the semantic names of the question intent field and the data asset entries in the candidate execution path. The semantic matching cost is determined by the similarity value between the question intent field vector and the semantic name vector. The available cost of a connection is calculated based on the matching of the connection key field sets of adjacent entries in the candidate execution path. The available cost of a connection is determined by the number of overlaps and the number of missing connection key field sets. The permission risk cost is calculated based on the access control identifiers in the candidate execution path, and the permission risk cost is determined by the number of data asset entries with restricted access in the path. The semantic matching cost, connection availability cost, and permission risk cost are linearly weighted and fused according to preset weights to obtain the path score of the candidate execution path; S45. Sort the candidate execution path set according to the path score, specifically including: When there is a candidate execution path with the unique optimal path score in the candidate execution path set, select that candidate execution path as the question execution path; When multiple candidate execution paths in the candidate execution path set have the same path score or the difference in path score is less than a preset difference threshold, the candidate execution paths are sorted in a secondary order based on their path length, and the candidate execution path with the shortest path length is selected as the question execution path. If there are still parallel candidate execution paths after the second-level sorting, the third-level sorting is performed based on the available cost of connection and the risk cost of permissions of the parallel candidate execution paths, and the candidate execution path with the best score is selected as the question execution path.

[0029] In this embodiment, S5 specifically includes: S51. Perform operator structure mapping on the data asset item sequence in the query execution path, convert adjacent data asset item pairs into connection operators, map the condition identifier and time window identifier in the query intent field into filter operators, map the indicator identifier and dimension identifier into aggregation grouping operators, and construct a query operator sequence. S52. Encode the connection order of the connection operators in the query operator sequence, encode the data asset entries participating in the connection as a permutation sequence, and use the permutation sequence as the individual representation of the genetic algorithm to construct the initial population set; S53. Perform genetic iteration to update the connection permutation sequence of the initial population set. In each iteration, calculate the fitness score of the permutation sequence and generate a new generation population set accordingly. Specifically, this includes: The fitness score of each permutation sequence is calculated based on the cost function, which is determined by the combination of the matching situation of the connection key field set, the selective estimate of the filter operator, and the number of connection steps. The fitness score is then written into the sequence scoring table. Based on the sequence scoring table, perform roulette wheel selection on the permuted sequence to obtain the parent sequence set, and perform deduplication and retention on the repeated permuted sequences in the parent sequence set; Perform partial mapping cross-PMX on the parent sequence set to generate the offspring permutation sequence, and perform exchange mutation on the generated offspring permutation sequence to change the arrangement position of adjacent entry identifiers; The parent sequence set and the offspring permutation sequence are merged to form a candidate update set. Based on the fitness score, the candidate update set is truncated to retain the high-scoring permutation sequence, and the new generation population set is obtained. S54. Based on the optimal join permutation sequence, perform join rearrangement on the query operator sequence and complete cost optimization compilation, construct operator dependencies and use topological sorting algorithm to obtain linear execution sequence, encode the linear execution sequence into task instruction sequence according to preset instruction encoding rules, and obtain cloud-executable task instruction sequence.

[0030] In this embodiment, S6 specifically includes: S61. Collect feedback data of the task instruction sequence executed in the cloud, and associate and record the feedback data with the corresponding question element sequence and task instruction sequence. The feedback data includes execution status, execution time, resource consumption statistics and result verification conclusions. S62. Based on feedback data, under the condition of the same number of question element sequences, multiple task instruction sequences are grouped according to the result verification conclusion, and the Elo scoring algorithm is used to rank the task instruction sequences according to their merits. The task instruction sequences with higher ranking are determined as preference samples to form a preference training pair set. S63. Based on the set of preference training pairs, under the condition of the same question element sequence, the preference sample task instruction sequence and the non-preference sample task instruction sequence are respectively used as the target sequence to drive the LLaMA model to perform autoregressive forward propagation, extract the conditional log probability along the word position of the target sequence and accumulate to obtain the preference sample score and the non-preference sample score, and associate the score with the corresponding preference training pair identifier and record it as a sample score set. S64. Calculate the DPO preference alignment loss based on the sample rating set, and perform gradient backpropagation on the preference alignment loss to complete the incremental update of the trainable parameters of the LLaMA model.

[0031] Example 1: To verify the feasibility of this invention in practice, it was applied to a cloud-based data analysis scenario. This scenario involves a large number of data asset entries, including table-level assets and their join key fields, metric-level assets with caliber identifiers and aggregation rule identifiers, dimension-level assets with hierarchical identifiers and value range identifiers, and access control identifiers related to permissions. Business personnel submit data queries in natural language, with common expressions including "trends of core indicators within a certain period, broken down by regional hierarchy, and filtering specific status records."

[0032] In existing solutions, when synonyms, differences in definitions, and the complexity of data asset relationships are combined, problems such as incomplete feature identification, field mapping offsets, unavailable connection paths, or restricted permissions can easily occur, leading to an increase in query failure rate, significant fluctuations in execution costs, and a tendency for effectiveness to diminish as data assets change.

[0033] In this scenario, the invention first standardizes the query requests and extracts metrics, dimensions, time, and conditional elements to form a query element sequence. Then, it uses vector semantic retrieval to complete candidate recall and re-ranking, and constructs a training sample set according to semantic alignment rules to fine-tune the LLaMA model, obtaining a structured query intent field. Subsequently, in the cloud, based on forward and reverse association indexes, it expands the adjacent item relationships of data asset items, constructs an association index structure, and performs bidirectional shortest path search to form candidate execution paths. Finally, it uses a semantic cost fusion scoring method to select the optimal query execution path.

[0034] The execution path for each question is mapped to a sequence of query operators, and a genetic algorithm is used to optimize the connection order and compile the result into a sequence of task instructions. Finally, cloud execution feedback is collected, preference training pairs are constructed, and the LLaMA model is incrementally fine-tuned and updated using the DPO mechanism, so that the model's preferences for subsequent questions are gradually aligned with the executable results.

[0035] During continuous use, this invention consistently improves in intent field accuracy, executable path hit rate, and query success rate, while exhibiting superior performance in execution cost and resource consumption. For example, as shown in Table 1, without enabling closed-loop updates, the invention achieves approximately 88.9% intent field accuracy and 91.5% query success rate. After introducing DPO closed-loop updates, these figures further increase to approximately 92.4% and 94.3%, respectively. Simultaneously, the average execution cost score and resource consumption index continue to decrease, and the user adoption rate also increases, reflecting the invention's more stable path selection and more sustainable adaptive capabilities in the cloud-based query chain.

[0036] Table 1 Comparison of DataCloud Analysis Results

[0037] As shown in Table 1, the present invention significantly outperforms the comparison scheme in all three core metrics: intent field accuracy, executable path hit rate, and query success rate. Furthermore, it achieves a lower average execution cost score, indicating that the present invention possesses more stable executability and better resource cost performance across the question intent parsing, associated path selection, and operator-level optimized compilation chain. Moreover, the introduction of DPO closed-loop updates further improves all metrics, demonstrating that incremental fine-tuning based on cloud feedback and preference alignment continuously enhances the model's ability to match business statements with executable results. The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An AI-powered large-scale intelligent data cloud analysis method and system, characterized in that, include: The question preprocessing module is used to acquire natural language question requests and preprocess them to form a question element sequence, while simultaneously building a data asset collection in the cloud; The semantic retrieval module is used to recall and reorder candidate semantic elements in the question element sequence, and to construct a training sample set based on the reordering results according to semantic alignment rules. The model fine-tuning module is used to perform multiple rounds of fine-tuning on the LLaMA model based on the training sample set, and to use the fine-tuned model to perform semantic parsing on the question element sequence, and to perform multiple rounds of planning and reasoning based on the preset question task template to form the question intent field. The path selection module is used to expand the associated index of the data asset set in the cloud. Starting from the question intent field, it performs bidirectional shortest path search in the associated index structure to generate candidate execution paths, and calculates the path score using a semantic cost fusion scoring method. At the same time, it selects the optimal execution path as the question execution path based on the path score. The operator compilation module is used to map the query execution path to a sequence of query operators, and to perform cost optimization compilation on the sequence of query operators to generate a sequence of task instructions that can be executed in the cloud. The feedback optimization module is used to collect feedback data from the execution of task instruction sequences in the cloud and to perform incremental fine-tuning updates on the LLaMA model.

2. The AI ​​large-scale model intelligent data cloud analysis method and system according to claim 1, characterized in that, The modules are connected in the following way: S1. Obtain natural language question requests and preprocess them to form a question element sequence, while simultaneously establishing a data asset collection in the cloud; S2. The vector semantic retrieval method is used to recall and reorder the candidate semantic elements of the question element sequence, and a training sample set is constructed based on the reordering results according to the semantic alignment rules. S3. Based on the training sample set, perform multiple rounds of fine-tuning on the LLaMA model, and use the fine-tuned model to perform semantic parsing on the question element sequence. Perform multiple rounds of planning and reasoning according to the preset question task template to form the question intent field. S4. Expand the data asset set with association index in the cloud. Starting from the question intent field, perform bidirectional shortest path search in the association index structure to generate candidate execution paths. Calculate the path score using the semantic cost fusion scoring method. Select the optimal execution path as the question execution path based on the path score. S5. An operator generation method based on intermediate representation is adopted to map the query execution path into a query operator sequence, and a connection order optimization method based on genetic algorithm is used to optimize the cost of the query operator sequence and generate a task instruction sequence that can be executed in the cloud. S6. Collect feedback data of the task instruction sequence executed in the cloud, and perform incremental fine-tuning updates on the LLaMA model through the Direct Preference Optimization (DPO) mechanism based on preference alignment.

3. The AI ​​large-scale intelligent data cloud analysis method and system according to claim 2, characterized in that, The preprocessing specifically includes: The system performs character normalization and noise removal on natural language query requests, performs word segmentation and word form merging on the cleaned text, performs entity recognition and terminology normalization on the word segmentation results, and labels the identified elements according to indicators, dimensions, time and conditions. Time window parsing and caliber conversion are performed on time elements, logical relationship parsing and predicate standardization are performed on condition elements, and the various elements that have been annotated and parsed are spliced ​​together according to preset serialization rules to obtain the question element sequence.

4. The AI ​​large-scale model intelligent data cloud analysis method and system according to claim 2, characterized in that, The data asset collection in the cloud consists of queryable data asset entries. Each data asset entry includes at least an asset type, an asset identifier, and a semantic name, and records association information for expanding the associated index for each data asset entry. The associated information includes at least the primary key field set, foreign key field set, join key field set, and partition field identifier of table-level assets, as well as the caliber identifier and aggregation rule identifier of indicator-level assets, the hierarchical identifier and value domain identifier of dimension-level assets, and the access control identifier of permission-level assets. Based on the aforementioned association information, a forward association index and a reverse association index are established in the cloud. The forward association index is used to locate connectable adjacent asset entries from the source asset entry, and the reverse association index is used to trace back to the connectable source asset entry from the adjacent asset entry.

5. The AI ​​large-scale model intelligent data cloud analysis method and system according to claim 2, characterized in that, S2 specifically includes: S21. Vectorize the elements in the question element sequence to obtain element vectors, and vectorize the semantic names of data asset entries to obtain asset vectors. Use the element vectors as retrieval vectors to perform Top-K similarity recall on the asset vectors to generate a candidate data set. S22. Reorder the candidate data set, calculate the reordering score for each candidate data, the reordering score is obtained by weighting the similarity score between the element vector and the asset vector and the consistency score of the asset type, and sort the candidate data asset item set according to the reordering score to obtain the reordered candidate set. S23. Determine the alignment relationship between elements and data asset entries in the reordering candidate set according to semantic alignment rules. The semantic alignment rules include fusion score threshold constraints and one-to-one alignment constraints. The Hungarian matching algorithm is used to solve for the alignment results on candidate pairs that satisfy the fusion score threshold constraints. S24. Based on the alignment results, construct a training sample set, assemble the elements in the alignment results with the corresponding data asset entries into positive sample pairs, and select unaligned data asset entries from the reordering candidate set and assemble them with the same element into negative sample pairs. Merge the positive sample pairs and negative sample pairs to obtain the training sample set.

6. The AI ​​large-scale model intelligent data cloud analysis method and system according to claim 2, characterized in that, S3 specifically includes: S31. The training sample set is templated and constructed. The question element sequence of each training sample is assembled into an instruction input segment, and the target intent field corresponding to the training sample is assembled into a structured output segment. The byte pair encoding BPE word segmentation algorithm is used to uniformly encode the input segment and the output segment to form an instruction sample. S32. The LLaMA model is updated in multiple rounds using a parameter-efficient fine-tuning method, and the stability of model parameters and consistency of output structure are jointly constrained in each round of fine-tuning. S33. Perform stepwise decoding on the question element sequence, expand the candidate sequence by branching according to the next word probability output by the fine-tuned LLaMA model and retain high-scoring candidates until the end marker of the structured output segment is generated, and obtain multiple sets of candidate intent output sequences, and parse them into a set of candidate question intent fields according to the structured field delimiter. S34. Based on the preset question task template, read the list of intent fields and the generation order. For the missing fields in the candidate question intent fields, form a list of fields to be completed. Based on the question element sequence and the list of fields to be completed, use the fine-tuned LLaMA model to generate field values ​​and backfill them to obtain the candidate question intent field set. S35. Perform consistency selection on the candidate question intention field set after multiple rounds of backfilling, and select the optimal candidate question intention field as the question intention field based on the number of consistent occurrences.

7. The AI ​​large-scale intelligent data cloud analysis method and system according to claim 6, characterized in that, S34 specifically includes: S341. Read the preset question task template and parse it to obtain the list of intent fields and the order of field generation. The preset question task template records the list of fields and the order of field generation of indicator identifier, dimension identifier, time window identifier and condition identifier in the form of field configuration items. It also configures the value source identifier and field dependency identifier for each field. It traverses the candidate question intent fields according to the order of field generation and identifies missing fields to form a list of fields to be completed. S342. For each field in the list of fields to be completed, extract the assigned fields associated with the field based on the field dependency identifier, encode the assigned fields into key value sequences, and assemble the question element sequence and key value sequences into a generation input sequence. Locate the target field start mark in the generation input sequence and call the fine-tuned LLaMA model to generate up to the field end mark. Register the generated fragment between the start mark and the end mark as the field value set. S343. Bind the field values ​​to the set of field values, write the field values ​​into the candidate question intent field according to the field identifier and complete the backfilling, and update the missing field marker and conflict field marker of the candidate question intent field after backfilling; S344. When conflict field markers exist, locate conflict field pairs based on field mutual exclusion identifiers, assemble conflict field pairs and question element sequences into conflict resolution input sequences, input them into the fine-tuned LLaMA model to generate replacement field values, and use replacement field values ​​to cover conflict field values ​​in candidate question intent fields. S345. Repeat the field completion, backfilling, and conflict resolution steps until the list of fields to be completed is empty or the preset number of rounds is reached, to obtain a set of candidate question intent fields after multiple rounds of backfilling.

8. The AI ​​large-scale intelligent data cloud analysis method and system according to claim 2, characterized in that, S4 specifically includes: S41. On the cloud, based on the forward and reverse association indexes, adjacent entries are expanded to generate a set of adjacent entry pairs. For each adjacent entry pair, the corresponding set of connection key fields and access control identifiers are associated to obtain the association index structure. S42. Locate the starting point item set in the data asset items based on the question intent field. The starting point item set includes data asset items that match the indicator identifier, dimension identifier, time window identifier and condition identifier in the question intent field. Locate the data asset items that can be connected to the starting point item set in the associated index structure as the candidate endpoint item set. S43. In the associated index structure, a bidirectional shortest path search is performed with the starting entry set as the forward search boundary and the candidate ending entry set as the reverse search boundary to generate a candidate execution path set. The candidate execution path is composed of a sequence of data asset entries that can be connected along the forward and reverse associated indexes. S44. Calculate the path score for each candidate execution path in the candidate execution path set using a semantic cost fusion scoring method, specifically including: The semantic matching cost is calculated based on the semantic names of the question intent field and the data asset entries in the candidate execution path. The semantic matching cost is determined by the similarity value between the question intent field vector and the semantic name vector. The available cost of a connection is calculated based on the matching of the connection key field sets of adjacent entries in the candidate execution path. The available cost of a connection is determined by the number of overlaps and the number of missing connection key field sets. The permission risk cost is calculated based on the access control identifiers in the candidate execution path, and the permission risk cost is determined by the number of data asset entries with restricted access in the path. The semantic matching cost, connection availability cost, and permission risk cost are linearly weighted and fused according to preset weights to obtain the path score of the candidate execution path; S45. Sort the candidate execution path set according to the path score, specifically including: When there is a candidate execution path with the unique optimal path score in the candidate execution path set, select that candidate execution path as the question execution path; When multiple candidate execution paths in the candidate execution path set have the same path score or the difference in path score is less than a preset difference threshold, the candidate execution paths are sorted in a secondary order based on their path length, and the candidate execution path with the shortest path length is selected as the question execution path. If there are still parallel candidate execution paths after the second-level sorting, the third-level sorting is performed based on the available cost of connection and the risk cost of permissions of the parallel candidate execution paths, and the candidate execution path with the best score is selected as the question execution path.

9. The AI ​​large-scale intelligent data cloud analysis method and system according to claim 2, characterized in that, S5 specifically includes: S51. Perform operator structure mapping on the data asset item sequence in the query execution path, convert adjacent data asset item pairs into connection operators, map the condition identifier and time window identifier in the query intent field into filter operators, map the indicator identifier and dimension identifier into aggregation grouping operators, and construct a query operator sequence. S52. Encode the connection order of the connection operators in the query operator sequence, encode the data asset entries participating in the connection as a permutation sequence, and use the permutation sequence as the individual representation of the genetic algorithm to construct the initial population set; S53. Perform genetic iteration to update the connection permutation sequence of the initial population set. Calculate the fitness score of the permutation sequence in each iteration and generate a new generation population set accordingly. S54. Based on the optimal join permutation sequence, perform join rearrangement on the query operator sequence and complete cost optimization compilation, construct operator dependencies and use topological sorting algorithm to obtain linear execution sequence, encode the linear execution sequence into task instruction sequence according to preset instruction encoding rules, and obtain cloud-executable task instruction sequence.

10. The AI ​​large-scale model intelligent data cloud analysis method and system according to claim 2, characterized in that, S6 specifically includes: S61. Collect feedback data of the task instruction sequence executed in the cloud, and associate and record the feedback data with the corresponding question element sequence and task instruction sequence. The feedback data includes execution status, execution time, resource consumption statistics and result verification conclusions. S62. Based on feedback data, under the condition of the same number of question element sequences, multiple task instruction sequences are grouped according to the result verification conclusion, and the Elo scoring algorithm is used to rank the task instruction sequences according to their merits. The task instruction sequences with higher ranking are determined as preference samples to form a preference training pair set. S63. Based on the set of preference training pairs, under the condition of the same question element sequence, the preference sample task instruction sequence and the non-preference sample task instruction sequence are respectively used as the target sequence to drive the LLaMA model to perform autoregressive forward propagation, extract the conditional log probability along the word position of the target sequence and accumulate to obtain the preference sample score and the non-preference sample score, and associate the score with the corresponding preference training pair identifier and record it as a sample score set. S64. Calculate the DPO preference alignment loss based on the sample rating set, and perform gradient backpropagation on the preference alignment loss to complete the incremental update of the trainable parameters of the LLaMA model.

Images