A conversational business intelligence platform process optimization method, system and device

Abstract

The present application belongs to the technical field of artificial intelligence, in order to solve the problems of random error and result adaptability defect existing in ChatBI process, the present application provides a dialogue business intelligence platform process optimization method, system and equipment, the output results of each link in the dialogue business intelligence platform process are obtained, the output results of each link are sequentially checked in multidimension according to the process execution sequence, the process link that may have problems can be found in time, and the error is corrected and optimized in time before it spreads to the next process link; moreover, the output results are checked from three aspects of adaptation to the original demand of user input problem, compliance with the business logic of the process link itself and adaptation to the use demand of the downstream process link, the adaptability of the result is guaranteed, and the problem that the output of each process link is disconnected with the downstream process link without obvious error is avoided.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field related to artificial intelligence, and particularly relates to a conversational business intelligence platform process optimization method, system and device. BACKGROUND

[0002] The statements in this section merely provide background information related to the application and do not necessarily constitute prior art.

[0003] With the deepening of digital transformation, the demand of enterprises for data value mining is increasingly urgent. As a core tool for supporting data-driven decision-making, the function evolution and technology upgrading of business intelligence BI always focus on "reducing the threshold of data use" and "improving analysis efficiency". In this context, conversational analysis such as ChatBI gradually becomes the mainstream way for enterprise users to obtain data insights, as users do not need to master complex SQL syntax or BI tool operation logic, but can initiate data queries and generate visual charts through daily language, greatly expanding the coverage of data applications.

[0004] The core function of the conversational business intelligence platform, ChatBI, is to use large language models and retrieval enhancement generation technology based on domain knowledge base to respond to users' data analysis needs. To achieve accurate conversion from natural language to data charts, ChatBI needs to complete multiple complex tasks, including natural language parsing, intent recognition, data retrieval, query statement generation, and visualization rendering. Since a single agent cannot balance the professionalism and efficiency of all aspects, a multi-agent collaborative architecture has gradually become the mainstream design idea: through a word segmentation agent to process language structure, a Rag service to call knowledge base information, a model selection agent to match the optimal processing model, a question understanding agent to analyze users' real needs, and an NL2SQL agent to generate data query statements, each module cooperates to form a complete processing link.

[0005] However, the multi-agent collaborative mode, while improving system flexibility, also introduces new technical challenges, specifically:

[0006] 1. There is a risk of "random error" in the processing results of each link. For example, large models may have hallucinations during intent understanding or SQL generation, i.e., generating content that does not match the facts, and the enhanced retrieval service may have low information similarity when retrieving relevant data due to limitations of the matching algorithm. These errors, if not discovered in time, will be passed down the processing chain, forming a "error accumulation effect", i.e., the deviation of one link may cause all subsequent steps to run based on incorrect premises, ultimately leading to a serious disconnection between the generated charts and user needs. Since the entire process is highly automated, real-time human monitoring cannot cover all details, greatly increasing the probability of losing control of the results.

[0007] 2. Even if each intelligent agent operates normally according to the preset logic, result adaptability defects may still occur. For example, when a user queries "quarterly revenue comparison of each department", the system may only return the original association data of department ID and revenue value, without considering the rationality of visual presentation such as lack of structured processing of time dimension, resulting in that the generated chart cannot intuitively reflect the trend and loses the actual analysis value.

[0008] The above problems directly restrict the reliability and user trust of the ChatBI system. Enterprise users may reduce the willingness to use or even abandon the reliance on the system decision due to frequent result deviations, thereby hindering the effective release of data value. Therefore, how to solve the problems of random errors and result adaptability defects in the multi-agent collaborative ChatBI process has become a key direction for current technical development to break through. SUMMARY

[0009] To overcome the above shortcomings of the prior art, the present application provides a conversational business intelligence platform process optimization method, system and device, which can timely find possible error process links, ensure the adaptability of the results, and avoid the problem that the output of each process link is not obviously wrong, but the output of the process link is disconnected with the downstream process link.

[0010] To achieve the above purpose, the present application adopts the following technical solutions:

[0011] In a first aspect, the present application provides a conversational business intelligence platform process optimization method, comprising:

[0012] Obtaining the output results of each link in the conversational business intelligence platform process, and sequentially performing multi-dimensional verification on the output results of each link according to the process execution order to determine whether the output results of the process link meet the output requirements; wherein the output requirements include adaptation to the original demand of the user input question, compliance with the business logic of the process link itself, and adaptation to the use demand of the downstream process link;

[0013] According to the verification results of the output results of each link, a standardized problem identifier is generated, and the corresponding error type and matching processing mode are determined by combining the pre-set keyword library with the standardized problem identifier, so as to realize the correction and optimization of the corresponding process link.

[0014] In a second aspect, the present application provides a conversational business intelligence platform process optimization system, comprising:

[0015] The check module is configured to: acquire output results of each link in a conversational business intelligence platform process, and sequentially check the output results of each link in multiple dimensions according to a process execution sequence, so as to determine whether the output results of the process link meet output requirements; wherein the output requirements include adaptation to original requirements of user input questions, compliance with business logic of the process link itself, and adaptation to use requirements of downstream process links.

[0016] The optimization module is configured to: generate a standardized question identifier according to a check result of the output results of each link, determine a corresponding error type and a matching processing mode by combining the standardized question identifier with a preset keyword library, and realize correction and optimization of the corresponding process link.

[0017] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the method of the first aspect is completed.

[0018] The above one or more technical solutions have the following beneficial effects:

[0019] In the present application, the output results of each link in the conversational business intelligence platform process are acquired, and the output results of each link are sequentially checked in multiple dimensions according to the process execution sequence, so that the process link that may have problems can be found in time, and the correction and optimization can be made in time before the error spreads to the next process link; and the output results are checked from three aspects of adaptation to original requirements of user input questions, compliance with business logic of the process link itself, and adaptation to use requirements of downstream process links, so that the adaptability of the results is ensured, and the problem that each process link has no obvious error, but the output of the process link is out of line with the downstream process link is avoided.

[0020] The advantages of the additional aspects of the present application will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0021] The drawings accompanying the specification of the present application serve to provide a further understanding of the present application, and the illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation on the present application.

[0022] Fig. 1 A flow chart of the conversational business intelligence platform process optimization method in the embodiment one of the present application;

[0023] Fig. 2 A block diagram of the conversational business intelligence platform process optimization method in the embodiment one of the present application. DETAILED DESCRIPTION

[0024] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0025] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit the exemplary embodiments according to the present application.

[0026] In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0027] Embodiment one

[0028] The embodiment discloses a conversational business intelligence platform process optimization method, comprising:

[0029] The output results of each link in the conversational business intelligence platform process are obtained, and the output results of each link are sequentially subjected to multi-dimensional verification according to the process execution sequence, so as to determine whether the output results of the process link meet the output requirements; wherein the output requirements include adaptation to the original requirement of the user input question, compliance with the business logic of the process link itself, and adaptation to the use requirement of the downstream process link;

[0030] According to the verification results of the output results of each link, a standardized question identifier is generated, and the corresponding error type and matching processing mode are determined by combining the preset keyword library with the standardized question identifier as an index, so as to realize the correction and optimization of the corresponding process link.

[0031] The embodiment obtains the output results of each link in the conversational business intelligence platform process, and sequentially subjects the output results of each link to multi-dimensional verification according to the process execution sequence, so as to timely find the process link that may have problems and timely correct and optimize before the error spreads to the next process link; and the output results are verified from three aspects of adaptation to the original requirement of the user input question, compliance with the business logic of the process link itself, and adaptation to the use requirement of the downstream process link, so as to ensure the adaptability of the results and avoid the problem that each process link has no obvious error, but the output of the process link is out of touch with the downstream process link.

[0032] The conversational business intelligence platform ChatBI mainly includes four core processes, which are entity extraction and enhanced retrieval service link, model selection link, question understanding link and execution SQL link.

[0033] The entity extraction stage mainly involves using the semantic understanding capabilities of intelligent agents to extract keywords from the natural language questions input by users. These keywords are then matched with structured information such as model fields and enumeration values ​​based on character similarity. Simultaneously, the enhanced retrieval service calls a vector library to recall key model description information and business knowledge, outputting a range of candidate models and supplementary knowledge systems corresponding to the user's question.

[0034] The model selection phase focuses on precise model matching. Model-related information is summarized from the recalled knowledge and extracted into prompt words. Leveraging the advanced language understanding capabilities of large-scale models, one or more models are selected that best match the current user's question. These selected models will serve as the core analytical vehicle for subsequent answering processes, laying the foundation for accurately answering user questions.

[0035] The problem understanding phase combines multi-dimensional information such as model field enumeration and model description to simulate the logical decomposition method of a professional data analyst, enabling a comprehensive and accurate understanding of the user's problem. Its core output includes several key details, specifically covering standardized query problem definitions, interpretation of query intent, selected models and fields, and whether data aggregation operations are needed, providing guidance for SQL generation.

[0036] After receiving the results from the intelligent problem-solving service, the SQL execution phase combines the supplementary knowledge obtained from word segmentation and Rag-enhanced retrieval processes with the rules provided by the prompt words to automatically generate the target SQL statement and execute the corresponding query operation. Finally, the query results are returned to the front end for display, thus providing a complete answer to the user's question.

[0037] This embodiment optimizes the process steps based on the existing conversational business intelligence platform, ChatBI. The following section combines... Figs. 1-2 This embodiment provides a detailed description of a conversational business intelligence platform process optimization method, which specifically includes:

[0038] Step 1: Obtain the output results of each stage in the conversational business intelligence platform process, and perform multi-dimensional verification on the output results of each stage in the order of process execution to determine whether the output results of the process stages meet the output requirements. The output requirements include adapting to the original requirements of the user input question, conforming to the business logic of the process stage itself, and adapting to the usage requirements of downstream process stages.

[0039] In this embodiment, the running status and output results of the entity extraction, model selection, problem understanding, NL2SQL, and SQL execution stages in the conversational business intelligence platform process are obtained. Based on the running status of the corresponding process stages, the output results of the entity extraction, model selection, problem understanding, NL2SQL, and SQL execution stages are checked in multiple dimensions according to the process execution order to determine whether the output results of the process stages meet the output requirements.

[0040] This embodiment starts with the user inputting a question into the conversational business intelligence platform. It then focuses on key steps such as entity extraction, model selection, question understanding, NL2SQL, and SQL execution. Monitoring, verification, decision intervention, and optimization feedback are carried out sequentially to form a closed loop of monitoring-verification-intervention-optimization, ensuring the reliability and accuracy of the system in handling user questions.

[0041] After the user completes the task of selecting a topic and entering a question, the process enters a stage where the operation status and output results of agents such as entity extraction, model selection, question understanding, NL2SQL, and SQL execution are captured in real time, establishing a data tracking link for each stage.

[0042] In this embodiment, the output results of the entity extraction stage are verified in multiple dimensions based on the running status of the entity extraction stage. Specifically, if the running status of the entity extraction stage is keyword matching success, the entity extraction results output by the current entity extraction stage are compared with the core elements through character similarity matching to determine whether they are semantically aligned with the user input question; it is determined whether the candidate models recalled by the enhanced retrieval service based on the extracted entities belong to the model type that is strongly associated with the extracted entities, and whether the business knowledge recalled by the enhanced retrieval service is semantically consistent with the extracted entities; based on semantic similarity, historical similar questions that match the user input question are determined, and the difference between the standard entities of the historical similar questions and the extracted entities of the user input question is calculated to verify the rationality of the entity extraction results.

[0043] In this embodiment, the entity extraction verification process includes comparing the user-input question with the entity extraction results (i.e., the keyword list) and the matching degree of the candidate models and business knowledge output by the RAG (Rich Search Group) service. This checks whether the model keywords are accurately identified and determines if there are any omissions or misidentifications of key information. For example, keywords include, but are not limited to, model fields, enumerated values, business metrics, time information, and dimensional information.

[0044] As a specific implementation method, the core element types in the user's original question are analyzed, and the element type coverage of the entity extraction results is compared. If any core element type is missing, it is marked as "incomplete entity". Core element types include, but are not limited to, business metrics, time dimensions, subject objects, and analysis dimensions. The extracted entity values ​​are compared with the model fields and enumeration values ​​in the user's input question to calculate character similarity. If the similarity is lower than a preset threshold, or if the entity value is not within the range of the model fields and enumeration values ​​in the user's input question, it is marked as "incorrect entity identification".

[0045] For example, if a query for a digital enterprise's revenue fails to retrieve the revenue or operating income, the entity is marked as "incomplete entity".

[0046] The similarity calculation can employ one of the following algorithms: cosine similarity, edit distance, etc.

[0047] As a specific implementation method, after verifying the entity extraction results for recall, the system checks whether the candidate models recalled by RAG based on the extracted entities belong to the model type strongly associated with the entity, thus achieving model matching verification; it also checks whether the business knowledge recalled by RAG is semantically consistent with the extracted entities, thus achieving business knowledge verification; if the proportion of knowledge unrelated to the extracted entities in the RAG recall results exceeds a preset threshold, such as extracting 3 core entities but recalling 10 pieces of irrelevant knowledge, then it is marked as "RAG Recall Redundancy"; if the number of recalled knowledge is lower than a preset threshold, such as extracting 3 core entities but only recalling 1 piece of relevant knowledge, then it is marked as "RAG Recall Insufficient".

[0048] For example, if the entity extraction result is "Operating Revenue", and RAG recalls "Profit and Loss Statement Model V3" and "Operating Revenue Calculation Rules", then the verification passes; if it recalls "Balance Sheet Model V2" and "Liability Field Definition", then it is marked as "RAG Model Recall Error, Business Knowledge Recall Error".

[0049] As a specific implementation method, using the user-input question as the search condition, historical similar questions with semantic similarity TOPX are recalled from the historical question database of the RAG knowledge base. Standard entities of these historical similar questions are extracted, and the difference between the current entity extraction result and the historical standard entities is calculated. If the difference exceeds a preset threshold (e.g., more than one core entity is missing), a secondary verification is triggered, and the entity extraction agent is called again. Based on the element type prompts of the historical standard entities, the missing entities are extracted. For the entity results extracted in the secondary step, semantic matching verification and RAG result verification are performed again. If the difference with the historical standard entities still cannot reach the preset threshold, it is finally marked as "entity extraction abnormal".

[0050] In this embodiment, semantic matching and RAG cross-validation are used in the entity extraction stage to avoid the transmission of missing / incorrect keywords and RAG recall bias knowledge to the downstream.

[0051] In this embodiment, the output results of the model selection stage are verified in multiple dimensions based on the running status of the model selection stage. Specifically, if the running status of the model selection stage is "model matching successful," it is determined whether the topic tags of the business description of the selected model output by the current model selection stage are consistent with the topic tags of the user input question; wherein, the topic tags are used to indicate the business scenario; based on the matching of the user input question and the corresponding topic tags, historical similar questions are obtained, and the model selection results of the historical similar questions are used to verify the model selection result of the current user input question; based on the entity extraction results of the entity extraction stage and the business knowledge of the enhanced retrieval service, the core field types required by the user input question are parsed, and it is verified whether the field list of the selected model covers the core field types required by the user input question.

[0052] In this embodiment, the suitability of the candidate model's business description with the current user problem scenario is verified, and the rationality of the model selection is evaluated by combining historical model call effect data.

[0053] As a specific implementation method, a dedicated intelligent agent is invoked to classify the business description of the selected model and the user input question into topic tags. If the topic tags of the two are inconsistent, they are directly marked as "model mismatch". If the topic tags are consistent, the matching degree between the model business description and the core requirements of the user input question is further calculated through a semantic similarity algorithm. If the matching degree is lower than a preset threshold, such as 80%, it is still marked as "insufficient scenario adaptability".

[0054] For example, the business description of the selected model is as follows: Inventory Model V4 in the model document is used for inventory turnover rate, inventory level, and inventory age analysis; the topic tags are as follows: sales analysis, inventory management, financial accounting, human resource statistics, etc.

[0055] For example, if a user enters a question on the topic of "sales analysis" and selects the model on the topic of "inventory management", it will be marked as "model mismatch".

[0056] As a specific implementation method, the core keywords of the question input by the user and the corresponding topic tags are used as search conditions to recall similar TOPX question records from the historical question database. The model selection results, solution accuracy rate and user feedback in the similar question records are extracted. If the average solution accuracy rate of the currently selected model in similar questions is lower than the preset standard, such as lower than 85%, or there are multiple records of invalid user feedback results, it is marked as "the historical performance of the model is not up to standard". At the same time, if there are better models in similar historical questions, but they are not selected at present, it is necessary to supplement and verify whether the reason for the current model selection is reasonable. If the reason is not sufficient, it is marked as "the reason for model selection is invalid".

[0057] For example, in similar historical "inventory turnover rate query" questions, "Inventory Model V4" had an average accuracy rate of 92%, and "Inventory Model V3" had an average accuracy rate of 75%. If "Inventory Model V3" is currently selected without a reasonable reason, it will be marked as "the model's historical performance is substandard and the reason for selection is invalid".

[0058] Based on the keywords and RAG business knowledge output from the preceding entity extraction step, the core field types required by the user's question are analyzed. These core field types include: time dimension fields, analysis dimension fields, core indicator fields, and filter condition fields. The field list of the selected model is queried to check whether it fully covers the user's required field types. If any core field is missing, it is marked as "Field Missing"; if the data types of the model fields and the user's required fields are incompatible, it is marked as "Field Type Mismatch".

[0059] For example, if the user input question requires the "department name" dimension for chart display, and the selected model only contains the "department ID" field, then mark it as "field missing"; if the model's "inventory turnover rate" field is a string type and cannot support numerical sorting, then mark it as "field type mismatch".

[0060] As an optional implementation, a structured entity list of user input questions is extracted. Based on the mapping rules between entity types and core field types, the field types required for user input questions are initially screened. The field types required for user input questions are then supplemented and corrected based on the business knowledge of the enhanced retrieval service to obtain the core field types required for user input questions. The business knowledge of the enhanced retrieval service includes the mapping relationship between entities and fields associated with user input questions, as well as the field types.

[0061] Specifically, the structured entity list output from the entity extraction stage is traversed, and the field types required by the user question are initially screened according to the mapping rules of entity type → core field type. If there are field types in the RAG business knowledge that are implicit in the user question but not covered by entity extraction, then the required time dimension field is added. If there is a mapping discrepancy between entity type and field type in the entity extraction results, such as the entity only mapping to "department ID", but the RAG knowledge shows that "department queries must also include ID". If the field name is used to support chart display, then the analysis subject field is corrected to Department ID + Department Name. Based on the entity-field mapping relationship output by RAG, the field type is associated with the actual field name in the model, and a structured list is generated, specifying the "field name, field type, and whether it is required" for each core field type. From the output of the model selection stage, the field list of the selected model is extracted, including "field name, field type, and field business meaning", forming the model field list. The core field requirement list of user issues is compared with the model field list item by item according to "field name + field type" to determine whether there is coverage. If all field names corresponding to a "required field type" do not appear in the model field list, then "required field missing" is determined. If the field name exists but the field type does not match, then "field type mismatch" is determined. If the field name corresponding to a "optional field type" does not appear in the model field list, then "optional field missing" is determined. If there are "irrelevant fields outside the core field requirement list" in the model field list, then "field redundancy" is marked.

[0062] In this embodiment, the model selection process ensures that the selected model matches the user's problem scenario through scenario adaptability and historical effect verification.

[0063] In this embodiment, the output of the problem understanding stage is verified in multiple dimensions based on its operational status. Specifically, if the problem understanding stage is in the state of element recognition completion, historical similar questions matching the user input question are determined based on semantic similarity, the standard understanding results of historical similar questions are extracted, and it is determined whether the understanding result output by the current problem understanding stage matches the standard understanding result. Based on a predefined list of core elements for problem understanding, it is verified whether the understanding result output by the current problem understanding stage covers all required core elements. It is also determined whether the understanding result output by the current problem understanding stage semantically conflicts with the entity extraction result output by the entity extraction stage, and whether it matches the model field list of the selected model output by the model selection stage.

[0064] This embodiment verifies the logical match between the understanding of the problem and the user's original needs, avoiding irrelevant answers.

[0065] As a specific implementation method, a decision-making intelligent agent is invoked to deconstruct the core query intent of the user's input question, and then the query intent interpretation output by the question understanding is compared. If there is a logical discrepancy between the two, "intention error" is marked.

[0066] Match historical similar questions with the current user's question semantic similarity TOPX from the historical question database, extract the standard understanding results of these similar questions, calculate the intent matching degree between the current understanding result and the standard understanding result, and mark "insufficient logical consistency" if the matching degree is lower than a preset threshold such as 85%.

[0067] As a specific implementation method, based on a predefined list of core elements for problem understanding, the system checks whether the understanding results cover all required elements to avoid deviations in downstream SQL generation due to missing elements. By verifying that the problem understanding output contains all the aforementioned required elements one by one, if any required element is missing, it is marked as "Element Missing" with the specific type of missing element noted; if the required element information is ambiguous, it is marked as "Element Information Unclear." The predefined list of core elements for problem understanding includes the query subject, core indicators, time dimension, calculation method, and filtering conditions.

[0068] For example, a predefined list of core elements for understanding user-input questions includes the query subject, core metrics, time dimension, calculation method, and filtering conditions.

[0069] As a specific implementation method, the usage indicators, filtering conditions, and time dimension output by the problem understanding are compared with the core keywords output by the preceding entity extraction stage. If the key information in the understanding result is not in the keyword list of entity extraction, or conflicts with the semantics of the keywords, it is marked as "mismatch with entity extraction results". The query model and usage fields output by the problem understanding are compared with the selected model ID and model field list output by the preceding model selection stage. If the query model in the understanding result is inconsistent with the selected model ID, or the usage fields are not in the field list of the selected model, it is marked as "mismatch with model selection results".

[0070] For example, if the core keywords output in the entity extraction stage include "East China region", and the filtering condition for the problem understanding result is "North China region", then it is marked as "not matching the entity extraction result"; if the model selection stage selects "Profit Statement Model V3", and "Profit Statement Model V3" includes the "Revenue Amount" field but does not have the "Profit Margin" field, then if the core indicator for the problem understanding result is "Profit Margin", then it is marked as "not matching the model selection result".

[0071] In this embodiment, the problem understanding process ensures that the understanding results align with user needs through logical consistency and element integrity checks.

[0072] In this embodiment, the output of the NL2SQL step is validated from multiple dimensions based on its operational status. Specifically, the SQL output by the NL2SQL step is parsed and analyzed to verify its syntax rules, and the table names and field names of the SQL are verified by combining the database metadata and the field list of the selected model. If the syntax parsing is successful and the metadata query is successful, the aggregate functions, dimension fields, and metric fields in the SQL are compared with the output of the problem understanding step to determine whether the business logic of the SQL is aligned with the user requirements output by the problem understanding step. Based on semantic similarity, historical similar questions matching the user input question are determined, the standard SQL of the historical similar questions is extracted, and the similarity between the SQL output by the current NL2SQL step and the standard SQL is calculated to verify the logical rationality of the current SQL.

[0073] As a specific implementation method, an SQL syntax parser is invoked to perform syntax analysis on the generated SQL, checking for issues such as misspelled keywords, missing syntax structures, unclosed parentheses, and unclosed quotation marks. If any such issues are found, an "SQL syntax error" is marked, and the location of the error is identified.

[0074] By combining the database metadata with the field list of the selected model, check whether the table name and field name in the SQL actually exist and whether the user has the necessary permissions: if the table specified in the SQL does not exist in the database, or the current user does not have query permissions, mark it as "Table does not exist / No permission"; if the field used in the SQL is not in the field list of the selected model, or the field name is misspelled, mark it as "Field does not exist / Spelling error"; check whether the way the field is used in the SQL is compatible with the field type, and if not, mark it as "Incorrect field type usage".

[0075] As a specific implementation method, the aggregate functions in the SQL are compared with the calculation methods output by the problem understanding stage. If the problem understanding requires summation but the SQL uses AVG (average value), or requires counting but the SQL uses SUM (summation), an "aggregate function error" is marked.

[0076] Extract the filtering conditions from the WHERE clause in the SQL and perform textual and semantic matching with the filtering conditions in the problem understanding stage: if the SQL is missing key filtering conditions, mark it as "Filtering conditions missing"; if the SQL filtering conditions conflict with the requirements, mark it as "Filtering conditions incorrect"; if the SQL has redundant filtering conditions and is not mentioned in the notes for problem understanding, mark it as "Filtering conditions redundant".

[0077] Check whether the dimension fields and measure fields in the SELECT clause of the SQL are consistent with the query dimensions and core indicators in the problem understanding stage: if a key dimension is missing, mark it as "dimensional missing"; if the measure fields are inconsistent with the core indicators, mark it as "measure field error".

[0078] For example, in the problem understanding stage, the core indicator is defined as total revenue, the filtering condition is East China region + 2024Q3, and the query dimension is department. If the generated SQL is SELECT department name, AVG(revenue amount) FROM bi.sale_2024WHERE region='East China', AVG≠SUM, and 2024Q3 is missing, then there will be "aggregate function error" and "filter condition omission" respectively.

[0079] In this embodiment, the NL2SQL step corrects SQL syntax errors and misuse of aggregate functions in advance through syntax compliance and business logic verification, thus avoiding invalid results after erroneous SQL execution.

[0080] In this embodiment, the output results of the SQL execution stage are verified in multiple dimensions based on the running status of the SQL execution stage. Specifically, if the running status of the SQL execution stage is an execution error, the key error information is extracted from the error log during the SQL execution process, and the key error information is matched with a preset SQL error type library to determine the root cause of the error based on the matching result. If the running status of the SQL execution stage is successful, the data dimensions of the SQL query results are matched and verified with the query dimensions output in the problem understanding stage, and the numeric fields of the SQL query results are verified based on the field data range obtained in the model selection stage.

[0081] As a specific implementation method, error logs during SQL execution are captured, key error information is extracted, a preset SQL error type library is called, the collected error information is matched with the categories and root causes in the library, and a clear error root cause is output based on the matching results.

[0082] The SQL error type library pre-stores a mapping relationship between error category, error scenario, corresponding root cause, and responsible link. For example, syntax errors - table does not exist / field error / syntax missing - SQL generated by the NL2SQL link does not conform to the database syntax specification - NL2SQL link; data errors - empty result set / data type conversion failure - filter conditions are too strict / no corresponding data / incorrect field type usage - SQL execution link.

[0083] As a specific implementation method, the data dimensions in the SQL query configuration information are compared with the query dimensions output in the preceding problem understanding stage. If a key dimension is missing, it is marked as "dimensional missing"; if there are redundant and irrelevant dimensions that are not specified in the query configuration information, it is marked as "dimensional redundancy". Based on the reasonable range of field data provided in the preceding model selection stage, the numerical fields in the query results are validated: if the value exceeds the reasonable range by ±50%, it is marked as "data abnormal (value exceeds the standard)"; if logically contradictory data appears, it is marked as "data abnormal (logical contradiction)". If the preceding problem understanding stage includes a TopN sorting requirement, targeted validation is performed: if the requirement is Top 5 and the result returns 10 data, it is marked as "quantity error (exceeds TopN requirement)"; if the requirement is "descending order sorting" and the result is sorted in ascending order by revenue, it is marked as "sorting error (opposite to requirement)". Determine whether the query results can be directly used for front-end chart rendering to avoid data that has no practical display meaning: If the results only contain a code field and no name field, the chart cannot display labels that users can understand, and mark it as "Insufficient adaptability (missing name field)"; if the result data format does not meet the front-end requirements, mark it as "Insufficient adaptability (incorrect format)".

[0084] For example, if the requirement for understanding the problem is "Top 5 departments in East China by revenue in Q3 2024 (sorted by revenue in descending order)", and the query results return "Department ID, Revenue Amount" (without "Department Name"), and return 8 data entries sorted by revenue in ascending order, then the query should be marked as "Dimension missing (missing department name)", "Incorrect quantity (8 entries ≠ 5 entries)", "Incorrect sorting (ascending order ≠ descending order)", or "Insufficient adaptability (missing name field)".

[0085] In this embodiment, the SQL execution process uses front-end compatibility validation to ensure that the results contain user-understandable name fields and that the data format meets the requirements for chart rendering, thus avoiding correct results that are unusable.

[0086] Step 2: Based on the verification results of the output of each step, generate standardized problem identifiers. Using the standardized problem identifiers as indexes and combined with a pre-set keyword library, determine the corresponding error types and matching methods, and correct and optimize the corresponding process steps based on the error types.

[0087] In this embodiment, the preset keyword library is a mapping association table of problem identifier, error type, processing method, and priority, with each problem identifier corresponding one-to-one with the standardized problem identifier.

[0088] Based on the validation results from ChatBI's full-process entity extraction, model selection, problem understanding, NL2SQL, SQL execution, and result verification, standardized problem identifiers are generated following the rules of process location and anomaly marking coding. This ensures that the identifiers are unique, traceable, and directly associated with the root cause of the error. Process location uses English abbreviations such as Entity Extraction = EE, Model Selection = MS, etc.; error types use two-digit codes such as Missing Information = 01, Matching Error = 02, etc.; and detailed scenarios are supplemented with Chinese keywords.

[0089] In this embodiment, after a verification report is generated at a certain process stage, process elements, error manifestation elements, and evidence supporting elements are extracted through keyword matching and semantic analysis. The extracted process elements are matched with a pre-set process-code mapping table to generate a process location code for the identifier. Based on the error manifestation elements, the problem is classified into the N major error types preset in the scheme and assigned a two-digit code to form the error type code for the identifier. Combining the specific details of the error manifestation elements, a Chinese sub-scenario description is added to form a sub-scenario code for the identifier, ensuring that the identifier can directly reflect the specific scenario of the problem. The rule is: sub-scenario = core error object + error behavior.

[0090] For example, the process code "EE" + error type code "01" + sub-scenario "Q3 keyword omission" → complete identifier "EE-01- Q3 keyword omission", which corresponds to the entity extraction process not extracting the keyword 'Q3'.

[0091] In this embodiment, after the entity extraction process is verified, a standardized verification conclusion is generated, which includes the verification status, anomaly markers, and anomaly details. Based on the standardized verification conclusion, a standardized problem identifier is generated, and then the decision intervention method is determined: if it is a minor deviation, such as RAG recall redundancy, the RAG screening agent is invoked to discard irrelevant knowledge; if it is a serious error, such as entity extraction anomaly, manual intervention is triggered, prompting the user to supplement key information or correct the entity extraction rules. At the same time, the verification result is recorded in the process optimization log for subsequent algorithm iteration of the entity extraction agent.

[0092] In this embodiment, after the model selection process is verified, a standardized verification conclusion is generated, including the verification status, anomaly markers, and anomaly details. Based on the standardized verification conclusion, a standardized problem identifier is generated, thereby determining the decision intervention method: if it is a minor deviation, such as "insufficient scenario adaptability but complete fields", the RAG is called to supplement the "model extended business description", and the model selection agent is re-triggered to optimize the selection; if it is a serious error, such as "model mismatch" or "missing core fields", manual intervention is triggered, and the "error link + recommended model list" is pushed. After manual confirmation, the model selection process is restarted; at the same time, the verification results are recorded in the process optimization log for subsequent reverse optimization of the model selection strategy.

[0093] In this embodiment, after the problem understanding process is verified, a standardized verification conclusion is generated, including the verification status, anomaly marker, and anomaly details. Based on the standardized verification conclusion, a standardized problem identifier is generated, and the decision intervention method is determined: if it is a minor deviation, such as "unclear element information", the RAG is called to supplement business knowledge, such as clarifying the specific quarter of the time dimension, and the problem understanding agent is re-triggered to correct the result; if it is a serious error, such as "incorrect intent", "missing core elements", or "serious mismatch with previous steps", manual intervention is triggered, and the user's original problem + the output of previous steps + the current understanding result is pushed. After the understanding logic is manually corrected, the process is restarted; at the same time, the verification result is recorded in the process optimization log for subsequent reverse optimization of the problem understanding agent algorithm.

[0094] In this embodiment, after the NL2SQL step verification is completed, a standardized verification conclusion is generated, which includes the verification status, anomaly markers, and anomaly details. Based on the standardized verification conclusion, a standardized problem identifier is generated, and then the decision intervention method is determined: if it is a minor deviation, such as "redundant filtering conditions" or "insufficient SQL optimization space", the SQL repair agent is invoked to regenerate the SQL; if it is a serious error, such as "SQL syntax error" or "aggregate function error", manual intervention is triggered, and "user requirements + problem understanding output + current SQL + anomaly marker" is pushed. After the SQL is manually corrected, the step is restarted. At the same time, the verification result is recorded in the process optimization log for subsequent reverse optimization of the NL2SQL agent.

[0095] In this embodiment, after the SQL execution verification is completed, a standardized verification conclusion is generated, including the verification status, anomaly marker, and anomaly details. Based on the standardized verification conclusion, a standardized problem identifier is generated, thereby determining the decision intervention method: If it is a minor deviation such as "dimensional redundancy" or "data format error," the data processing agent is invoked to correct the result; if it is a serious error such as "syntax error," manual intervention is triggered: if the responsible link is NL2SQL, the error information is pushed to the NL2SQL link to regenerate the SQL; if it is a data logic problem, manual adjustment of the filtering conditions or verification of the database data is prompted; at the same time, the verification result is recorded in the process optimization log for subsequent reverse optimization: if "empty result set" occurs frequently, the filtering condition generation rules of NL2SQL are optimized; if "data anomaly" occurs frequently, the reasonable range threshold of the model field is updated.

[0096] In this embodiment, if each step passes the verification and the intervention and repair are completed, the process will proceed normally to "result output" and the final result will be fed back to the user; if manual intervention still fails to repair the problem, the process will be marked as failed, the user will be notified of the error and prompted for manual assistance, and the failure case will be recorded for subsequent optimization.

[0097] This embodiment's solution centers on end-to-end monitoring, precise verification, layered intervention, and closed-loop optimization, covering key aspects of user problem handling while balancing automated repair with manual backup to ensure system reliability.

[0098] This embodiment also includes: adjusting the current problem processing chain based on log data. For example, if the RAG stage of a certain type of problem frequently has "insufficient recall", then the high recall vector model will be called first to achieve real-time optimization; optimizing the algorithm rules of each agent based on logs. For example, if the entity extraction stage frequently "omits the time dimension", then the keyword recognition library will be updated; if the NL2SQL stage frequently "omits conditions", then the filtering condition generation rules in the prompt word template will be optimized.

[0099] As an optional implementation, a larger and higher-performance intelligent agent is used to optimize the process steps. The designed prompts include inputs for each step, historical similar cases, error classification, and suggested corrective actions.

[0100] Here are some examples of prompt words: {

[0101] You are a professional database query evaluation expert, and you are now required to professionally evaluate the output results of a multi-agent collaborative system. Please verify according to the following process:

[0102] Output Requirements

[0103] Please provide suggested modifications. Please output the evaluation report in JSON format, including the following fields:

[0104] {

[0105] "vaild": true / false, / / true indicates no problem, false indicates a problem, one must be entered;

[0106] "severityLevel":"", / / Whether human intervention is required;

[0107] "stepName": "", / / Step name;

[0108] "suggestion": "" / should include: 1. Error flags, such as missing fields; 2. Specific error descriptions, such as: missing the 'business unit' field filter condition; 3. Supporting evidence chain, such as: the user's question requires '2023 data', but the SQL condition is 2022; 4. Optimization suggestions, such as: the ability to recognize numerical comparison operators needs to be enhanced.

[0109] }

[0110] [Evaluation Input]

[0111] 1. User's original natural language problem:<!--input!--> ;

[0112] 2. Entity extraction results obtained based on the question:<!--EntityExtractValue!--> ;

[0113] 3. Generated SQL statement:<!--showSql!--> ;

[0114] 4. Database query results:<!--dataQuery!--> ;

[0115] 5. NL2SQL prompt template reference:<!--nl2sqlPrompt!--> ;

[0116] 6. Model selection results:<!--modelSelect!--> ;

[0117] 7. Current system date:<!--KJRQ!--> ;

[0118] [Evaluation Process]

[0119] Please follow the steps below for verification and analysis:

[0120] Entity extraction and verification:

[0121] Check whether the entities extracted by the system from the user's question, such as table names, field names, and values, are complete.

[0122] Verify that the extracted entity values ​​are accurate and that there are no omissions or errors.

[0123] If any omissions or errors are found, please record the specific missing fields or incorrect values.

[0124] Model selection validation:

[0125] Evaluate whether the model selected by the system is suitable for handling the current user problem.

[0126] Review the model's performance on similar historical problems to confirm whether its selection was reasonable.

[0127] If the model selection is inappropriate, please explain the reasons and recommend a more suitable model.

[0128] SQL generation and verification:

[0129] Check if the generated SQL statement conforms to the syntax rules and if there are any syntax errors.

[0130] Verify that the WHERE clause in the SQL statement accurately reflects the filtering conditions in the user's question.

[0131] Verify that the SQL statement uses the correct aggregate functions such as SUM, AVG, etc., and the correct sorting method.

[0132] If the SQL statement contains logical errors or does not conform to the user's intent, please record the error type and location in detail.

[0133] Query result verification:

[0134] The generated SQL statement is executed in the database to obtain the query results.

[0135] Match the query results with the user's expected results and check whether the returned data dimensions are complete.

[0136] Verify that the data in numeric fields is within a reasonable range and that there are no outliers.

[0137] If the query results are incorrect or abnormal, please analyze the reasons and record them.

[0138] Overall logic verification: Evaluate whether the entire NL2SQL conversion process forms a closed loop, that is, whether the entire process from user questions to SQL statements to query results is logically coherent.

[0139] Check for errors or omissions during information transmission, such as inaccurate entity extraction leading to SQL statement errors.

[0140] If there is a problem with the overall logic, please point out the specific steps and the type of error.

[0141]

Precautions

[0142] When an error is identified, the specific steps in which the problem occurred must be clearly identified.

[0143] If the query result is correct but there are potential problems in the process, it should still be marked as an error.

[0144] For complex queries, it is recommended to verify the implementation of each filtering condition step by step.

[0145] The overall results are validated by combining information from previous questions.

[0146] }

[0147] Example 2

[0148] The purpose of this embodiment is to provide a conversational business intelligence platform process optimization system, including:

[0149] The verification module is configured to: obtain the output results of each step in the conversational business intelligence platform process, and perform multi-dimensional verification on the output results of each step in the order of process execution to determine whether the output results of the process steps meet the output requirements; wherein, the output requirements include adapting to the original requirements of the user input question, conforming to the business logic of the process step itself, and adapting to the usage requirements of downstream process steps.

[0150] The optimization module is configured to: generate standardized problem identifiers based on the verification results of the outputs of each stage; use these standardized problem identifiers as indexes in conjunction with a pre-defined keyword library to determine the corresponding error types and matching processing methods, thereby enabling the correction and optimization of the corresponding process stages.

[0151] In further embodiments, the following is also provided:

[0152] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When executed by the processor, the computer instructions perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0153] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0154] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0155] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0156] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0157] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0158] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for optimizing processes in a conversational business intelligence platform, characterized in that, include: The system acquires the output results of each stage in the conversational business intelligence platform process and performs multi-dimensional verification on the output results of each stage in the order of process execution to determine whether the output results of each stage meet the output requirements. Specifically, it acquires the running status and output results of the entity extraction stage, model selection stage, problem understanding stage, NL2SQL stage, and SQL execution stage in the conversational business intelligence platform process. Based on the operational status of the corresponding process steps, the output results of the entity extraction step, model selection step, problem understanding step, NL2SQL step, and SQL execution step are verified in multiple dimensions according to the process execution order to determine whether the output results of the process steps meet the output requirements. The output requirements include adapting to the original requirements of the user input problem, conforming to the business logic of the process step itself, and adapting to the usage requirements of downstream process steps. Based on the verification results of the output of each step, standardized problem identifiers are generated. Using the standardized problem identifiers as an index and combined with a pre-set keyword library, the corresponding error types and matching processing methods are determined, thereby realizing the correction and optimization of the corresponding process steps. Specifically, the output results of the entity extraction process are verified from multiple dimensions based on its operational status. If the entity extraction process is running in the state of keyword matching success, then the character similarity matching and core element comparison are used to determine whether the entity extraction result output by the current entity extraction process is semantically aligned with the user input question. Determine whether the candidate models for enhanced retrieval service based on entity recall belong to the model type that is strongly associated with the extracted entities, and determine whether the business knowledge recalled by enhanced retrieval service is semantically consistent with the extracted entities; Based on semantic similarity, historical similar questions matching the user input question are determined. The difference between the standard entities of the historical similar questions and the extracted entities of the user input question is calculated to verify the rationality of the entity extraction results. Specifically, the output results of the model selection process are validated from multiple dimensions based on the operational status of the model selection process, including: If the model selection process is in the "model matching successful" state, determine whether the topic tags of the business description of the selected model output by the current model selection process are consistent with the topic tags of the user input question; wherein, the topic tags are used to indicate the business scenario; Based on the matching of user input questions and corresponding topic tags, historical similar questions are obtained, and the model selection results of the current user input question are verified by using the model selection results of historical similar questions. Based on the entity extraction results from the entity extraction stage and the business knowledge of the enhanced retrieval service, we parse the core field types required by the user input question and verify whether the field list of the selected model covers the core field types required by the user input question. Specifically, the output of the problem understanding phase is verified from multiple dimensions, taking into account its operational status. If the problem understanding process is in the state of element recognition completion, then based on semantic similarity, determine the historical similar problems that match the user input problem, extract the standard understanding results of the historical similar problems, and determine whether the understanding result output by the current problem understanding process matches the standard understanding result. Based on a predefined list of core elements for problem understanding, verify whether the understanding results output in the current problem understanding stage cover all the required core elements; Determine whether the understanding result output by the current problem understanding stage conflicts semantically with the entity extraction result output by the entity extraction stage, and whether it matches the list of model fields of the selected model output by the model selection stage.

2. The process optimization method for a conversational business intelligence platform as described in claim 1, characterized in that, Based on the operational status of the NL2SQL step, the output results of the NL2SQL step are validated from multiple dimensions, specifically as follows: The SQL output from the NL2SQL stage is parsed and analyzed to verify the syntax rules. In addition, the table names and field names of the SQL are verified by combining the database metadata and the field list of the selected model. If the syntax parsing passes and the metadata query is successful, the aggregate functions, dimension fields, and metric fields in the SQL will be compared with the output results of the problem understanding stage to determine whether the business logic of the SQL is aligned with the user requirements output by the problem understanding stage. Based on semantic similarity, historical similar questions matching the user input question are determined, standard SQL of historical similar questions is extracted, the similarity between the SQL output of the current NL2SQL step and the standard SQL is calculated, and the logical rationality of the current SQL is verified.

3. The process optimization method for a conversational business intelligence platform as described in claim 1, characterized in that, Based on the execution status of the SQL execution phase, the output results of the SQL execution phase are validated from multiple dimensions, specifically: If the execution status of the SQL execution stage is an execution error, then extract the key error information from the error log during the SQL execution process, match the key error information with the preset SQL error type library, and determine the root cause of the error based on the matching result; If the execution status of the SQL step is successful, then the data dimensions of the SQL query results are matched and verified with the query dimensions output by the problem understanding step, and the numeric fields of the SQL query results are verified based on the field data range obtained by the model selection step.

4. The process optimization method for a conversational business intelligence platform as described in claim 1, characterized in that, The pre-set keyword library is a mapping association table of problem identifier, error type, processing method, and priority, and the problem identifier corresponds one-to-one with the standardized problem identifier.

5. A conversational business intelligence platform process optimization system, characterized in that, include: The verification module is configured to: acquire the output results of each stage in the conversational business intelligence platform process, and perform multi-dimensional verification on the output results of each stage in the order of process execution to determine whether the output results of the process stages meet the output requirements; specifically, to acquire the running status and output results of the entity extraction stage, model selection stage, problem understanding stage, NL2SQL stage and SQL execution stage in the conversational business intelligence platform process; Based on the operational status of the corresponding process steps, the output results of the entity extraction step, model selection step, problem understanding step, NL2SQL step, and SQL execution step are verified in multiple dimensions according to the process execution order to determine whether the output results of the process steps meet the output requirements. The output requirements include adapting to the original requirements of the user input question, conforming to the business logic of the process step itself, and adapting to the usage requirements of downstream process steps. Specifically, the output results of the entity extraction process are verified from multiple dimensions based on its operational status. If the entity extraction process is running in the state of keyword matching success, then the character similarity matching and core element comparison are used to determine whether the entity extraction result output by the current entity extraction process is semantically aligned with the user input question. Determine whether the candidate models for enhanced retrieval service based on entity recall belong to the model type that is strongly associated with the extracted entities, and determine whether the business knowledge recalled by enhanced retrieval service is semantically consistent with the extracted entities; Based on semantic similarity, historical similar questions matching the user input question are determined. The difference between the standard entities of the historical similar questions and the extracted entities of the user input question is calculated to verify the rationality of the entity extraction results. Specifically, the output results of the model selection process are validated from multiple dimensions based on the operational status of the model selection process, including: If the model selection process is in the "model matching successful" state, determine whether the topic tags of the business description of the selected model output by the current model selection process are consistent with the topic tags of the user input question; wherein, the topic tags are used to indicate the business scenario; Based on the matching of user input questions and corresponding topic tags, historical similar questions are obtained, and the model selection results of the current user input question are verified by using the model selection results of historical similar questions. Based on the entity extraction results from the entity extraction stage and the business knowledge of the enhanced retrieval service, we parse the core field types required by the user input question and verify whether the field list of the selected model covers the core field types required by the user input question. Specifically, the output of the problem understanding phase is verified from multiple dimensions, taking into account its operational status. If the problem understanding process is in the state of element recognition completion, then based on semantic similarity, determine the historical similar problems that match the user input problem, extract the standard understanding results of the historical similar problems, and determine whether the understanding result output by the current problem understanding process matches the standard understanding result. Based on a predefined list of core elements for problem understanding, verify whether the understanding results output in the current problem understanding stage cover all the required core elements; Determine whether the understanding result output by the current problem understanding stage conflicts semantically with the entity extraction result output by the entity extraction stage, and whether it matches the model field list of the selected model output by the model selection stage; The optimization module is configured to generate standardized problem identifiers based on the verification results of the outputs of each step, and use the standardized problem identifiers as an index in conjunction with a pre-set keyword library to determine the corresponding error types and matching processing methods, thereby achieving the correction and optimization of the corresponding process steps.

6. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-4.

Images