Method and system for dynamically processing complex business data based on large model

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a method and system for dynamically processing complex business data based on a large model, after data are input, standardized data are generated through cue word and code dual preprocessing; the dynamic path selection unit generates an optimal reasoning path according to an evaluation standard; the task execution module processes the data according to a selected path and flow, and if a result does not conform to expectation, path adjustment or prompt word modification is triggered; and the result evaluation module evaluates a result through the three-stage system and drives the system to iteratively optimize. According to the scheme, dynamic path selection and professional processing flow selection are carried out in a data driving mode in combination with the large model, the problems of illusion and indefinite reasoning path when the large model processes complex business data can be effectively solved, and the flexibility and efficiency of the large model in complex problem solving are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular, to a method and system for dynamically processing complex business data based on a large model. Background Art

[0002] Large models refer to artificial intelligence models such as GPT and BERT, which are applied in fields such as natural language processing, image recognition, and data analysis, and can process complex business data.

[0003] With the rapid development of artificial intelligence technology, large models have achieved remarkable results in fields such as natural language processing, image recognition, and data analysis. However, when processing complex business data, large models often face problems such as hallucination and unclear inference paths, resulting in inaccurate and inefficient output results. In addition, in the prior art, large models usually adopt fixed inference paths and static task execution processes, which are difficult to adapt to complex and changing business scenarios and lack flexibility and dynamic adjustment capabilities.

[0004] Therefore, there is an urgent need for a solution that can dynamically adjust the inference path and optimize the task execution process to enhance the flexibility and efficiency of large models in complex data processing. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned problems of the prior art, and provide a method and system for dynamically processing complex business data based on a large model, aiming to solve the technical problems of hallucination and unclear inference paths when large models process complex business data through data-driven dynamic path selection and professional processing flow matching, and improve the accuracy, flexibility, and resource utilization rate of processing.

[0006] The above purpose is achieved by the following technical solutions: A method for dynamically processing complex business data based on a large model, including: Step (1) preprocess the input complex business data, including data cleaning, format conversion, and feature extraction operations, to generate standardized data suitable for processing by the large model; Step (2) based on the standardized data, dynamically select the optimal inference path through the large model according to a preset evaluation criterion, where the inference path includes a task-oriented workflow path and a dynamic inference path embedded with a real-time decision maker; wherein, the dynamic inference path uses a non-uniform segmentation algorithm to parallelly segment the data, and the real-time decision maker adjusts the path according to the business rule matching degree and resource consumption trade-off, and uses a path memory bank to record the historical optimal path combination; Step (3) selects the most suitable processing flow from a predefined professional processing flow library according to the characteristics of business data and task requirements. Each of the processing flows represents a specific problem-solving strategy and can be optimized for different business scenarios; Step (4) executes the task according to the selected inference path and the processing flow, monitors the task progress in real time, and dynamically adjusts the path and flow according to the feedback; Step (5) evaluates the task results using a three-level evaluation system and iteratively optimizes the preset criteria and processing flow according to the evaluation results. The three-level evaluation system includes: immediate evaluation, path comparison, and manual feedback reinforcement learning.

[0007] Further, in step (1), the data cleaning includes noise filtering, missing value interpolation, and outlier replacement; the format conversion includes data type conversion, encoding conversion, and time format standardization.

[0008] Further, in the selection mechanism of the workflow path in step (2), a "task-subtask" decomposition map is established, and the user requirements are mapped to preset workflow nodes through semantic analysis; each of the preset workflow nodes has built-in dynamic routing logic and automatically selects the inference path according to the characteristics of the input data.

[0009] Further, in the dynamic inference path selection mechanism in step (2), through user question classification, task process selection decisions are made, and the output probability of the large model is adjusted by temperature scaling.

[0010] Further, the professional processing flow library in step (3) supports hot updates, can dynamically add industry-specific processes, and retrieves and adapts processes through a fuzzy matching algorithm.

[0011] A system for dynamically processing complex business data based on a large model, comprising: A data preprocessing module: integrating a data cleaning unit, a format conversion unit, and a feature extraction unit, supporting the generation of a semantic heat map for prompt words and code execution data conversion; A dynamic path selection unit: including a "task-subtask" decomposition map, a real-time decision maker, and a path memory library, supporting the dynamic generation of inference paths according to business rules, resource consumption, and context length; A professional processing flow library: storing extensible scenario-based process templates, supporting linked calls with the large model; A task execution module: including a model call engine and a real-time monitor component, supporting multi-model parallel computing and dynamic path adjustment; A result evaluation module: implementing a three-level evaluation system, outputting path optimization suggestions and feeding them back to the large model.

[0012] Further, the real-time decision maker adjusts the inference path based on three dimensions: business rule matching degree, resource consumption trade-off, and context length awareness.

[0013] Further, the path memory bank adopts a case-based reasoning algorithm, stores the historical optimal path combination, and automatically reuses it in similar tasks.

[0014] Further, the system supports docking with a graph database, generates a schema thinking chain containing subtask dependencies, and realizes the structured display and interpretability analysis of the inference path.

[0015] A method and system for dynamically processing complex business data based on a large model provided by the present invention, through multi-module collaborative innovation, data preprocessing fuses prompt words and algorithms to ensure data quality; dynamic path realizes intelligent routing and precise resource allocation through non-uniform segmentation and case-based reasoning; the professional process library supports multi-modal scenarios and hot updates; the three-level evaluation closed-loop continuously iterates and optimizes; the schema thinking chain visualizes the inference logic, effectively solving the problems of hallucination and unclear inference path when the large model processes complex business data, and improving the flexibility and efficiency of the large model in solving complex problems. This solution not only overcomes the pain points of large model hallucination, path rigidity, and low resource efficiency, improves the accuracy of complex business data processing and resource utilization rate, reduces the time consumption and error rate, but also enhances the interpretability and adapts to intelligent decision-making in multiple fields such as finance and healthcare. Description of the Drawings

[0016] Figure 1 It is a flowchart of a method for dynamically processing complex business data based on a large model according to the present invention. Detailed Description of the Embodiment

[0017] The present invention will be further described in detail below with reference to the drawings and embodiments. The described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0018] As Figure 1 shown, this solution provides a method for dynamically processing complex business data based on a large model, including: Step (1) Preprocess the input complex business data, including operations such as data cleaning, format conversion, and feature extraction, to generate standardized data suitable for processing by the large model; among them, feature extraction generates a semantic position scope through prompt words to identify high-information-density regions in business data, that is, extract useful information from the original data for the model to use, such as: numerical feature extraction, time feature extraction, text feature extraction, image feature extraction, feature combination; The purpose of this step is to convert complex business data into a format suitable for large model processing, ensure that the data can be effectively processed by the large model, and improve the model's performance.

[0019] This embodiment will be described from two aspects: prompt and code. Obtain business data from a certain platform through code, and at the same time convert some encodings of the data, such as converting numbers to language expressions, converting N / A to null, etc., and finally generate business data that is easy for the large model to understand. Extract features from business data through prompts to generate semantic position scopes and identify high-information-density regions; such as: business classification data, location of the block where it is located, business classification statistical data, location of the block where it is located, location of each business detail data block, etc.

[0020] In step (2), based on the standardized data, the optimal inference path is dynamically selected through the large model according to the preset evaluation criteria (path complexity, resource consumption). The inference path includes a task-oriented workflow path and a dynamic inference path embedded with a real-time decision maker; among them, the dynamic inference path uses a non-uniform segmentation algorithm (based on a semantic heat map, abandoning the method of dividing data by a fixed size during data extraction) to parallelly segment the data, and the real-time decision maker adjusts the path according to the business rule matching degree (such as classification and statistical services, each business detail data, etc.) and resource consumption trade-off (such as diverting high-complexity tasks to a dedicated fine-tuning model), and uses a path memory bank to record the historical optimal path combination to form case-driven adaptive optimization. In step (3), according to the characteristics of the business data and task requirements, the most suitable processing flow is selected from a predefined professional processing flow library. Each processing flow represents a specific problem-solving strategy and can be optimized for different business scenarios. In step (4), execute the task according to the selected inference path and the processing flow, monitor the progress of the task in real time, and dynamically adjust the path and flow according to the feedback to ensure the efficient completion of the task. In step (5), a three-level evaluation system is used to evaluate the task results, and the preset criteria and processing flow are iteratively optimized according to the evaluation results, and then fed back to the large model, and finally a reasonable inference path selection and processing flow selection are formed; the three-level evaluation system includes: immediate evaluation (automated scoring based on preset rules, such as fact consistency and logical completeness rules), path comparison (running multiple processing paths in parallel and outputting a difference report), and manual feedback reinforcement learning (annotators rate the results and reverse-optimize the routing strategy).

[0021] In step (1) of this method, the data cleaning includes noise filtering, missing value interpolation, and outlier replacement, aiming to process the noise, missing values, and outliers in the data to ensure data quality; the format conversion includes data type conversion, encoding conversion, and time format standardization, aiming to ensure that the data meets the requirements of the large model, such as: data type conversion, encoding conversion, time format conversion, standardization / normalization, etc.

[0022] In the selection mechanism of the workflow path in step (2) of this method, a "task - subtask" decomposition map is established, and the user requirements are mapped to preset workflow nodes through semantic analysis; each of the preset workflow nodes has built - in dynamic routing logic, which automatically selects the inference path according to the characteristics of the input data. For example: fixed classification and statistical tasks, detailed data task workflows, which are task - oriented and selected by the user.

[0023] In the dynamic inference path selection mechanism in step (2) of this method, through user question classification, task process selection decisions are made, and the output probability of the large model is adjusted through temperature scaling. Specifically: the code generation task is set to 0.0, the data extraction task is set to 1.0, the general dialogue task is set to 1.3, and the creative writing task is set to 1.5.

[0024] The professional processing flow library in step (3) of this method supports hot updates, can dynamically add industry - specific processes (such as financial risk control processes, medical diagnosis processes), and retrieves and adapts processes through fuzzy matching algorithms (such as TF - IDF cosine similarity).

[0025] This solution also provides a system for dynamically processing complex business data based on a large model to implement the above - mentioned solution, including: Data pre - processing module: Integrates a data cleaning unit, a format conversion unit, and a feature extraction unit, supporting the generation of semantic heat maps for prompt words and code execution data conversion (such as converting numbers to natural language, converting N / A to null values); Dynamic path selection unit: Includes a "task - subtask" decomposition map, a real - time decision - maker, and a path memory library, supporting the dynamic generation of inference paths according to business rules, resource consumption, and context length; Professional processing flow library: Stores extensible scenario - based process templates (such as multi - modal generation, inference creation, dialogue response), supporting linked calls with the large model; Task execution module: Includes a model call engine and a real - time monitor component, supporting multi - model parallel computing and dynamic path adjustment; Result evaluation module: Implements a three - level evaluation system, outputs path optimization suggestions, and feedbacks them to the large model.

[0026] In this system, the real-time decision maker adjusts the inference path based on three dimensions: business rule matching degree, resource consumption trade-off, and context length awareness.

[0027] Specifically, the business rule matching degree triggers a dedicated processing flow through regular expressions. For example, the financial blacklist rule triggers the risk warning process, etc. The resource consumption trade-off is used to automatically allocate a dedicated fine-tuning model for high-complexity tasks. The context length awareness dynamically adjusts the computing resource allocation according to the data segmentation density.

[0028] In this system, the path memory bank adopts the Case-Based Reasoning (CBR) algorithm, stores the historical optimal path combinations (such as "segmentation method + model selection + parameter setting"), and automatically reuses them in similar tasks.

[0029] This system supports docking with a graph database (such as Neo4j), generates a graphical thinking chain containing subtask dependencies, and realizes the structured display and interpretability analysis of the inference path.

[0030] Specifically, in order to enhance the ability to process complex and variable tasks, the graph structure can be defined and the task can be executed through a graph or a knowledge graph to form a complete graphical thinking chain, so as to reduce the probability of the large model having "hallucinations", and ultimately enhance the flexibility, efficiency, and accuracy of the large model in complex data processing.

[0031] This solution constructs a dynamic closed loop between the "path adjustment" of task execution and the "prompt word / task optimization" of result evaluation to solve the problems of "hallucinations" and path rigidity of the large model. It also deeply integrates prompt words (semantic guidance), code (engineering implementation), and dynamic paths (intelligent planning) to support the flexible processing of complex business scenarios.

[0032] As Figure 1 shown, for further illustration of this solution, the process is as follows: (1) Start: Initiate the dynamic processing flow of complex business data as the trigger point of the whole process.

[0033] (2) Data input: Enter two types of core information, including business parameters and original business data, to provide materials for subsequent processing.

[0034] (3) Data preprocessing: Use prompt words to guide the large model to focus on key data and generate a semantic heat map. Perform data cleaning, format conversion, and feature extraction through code, and output standardized data to lay a foundation for path selection.

[0035] (4) Dynamic path selection: Based on the preprocessed data, combined with evaluation criteria such as "path complexity" and "resource consumption", the optimal inference path is dynamically planned through the task decomposition graph and the real-time decision maker.

[0036] (5) Task execution: Input the prompt to clarify the execution instruction, and call the corresponding model / process to execute the task; if the result does not meet the expectation (such as the model confidence <0.6), then trigger the adjustment path process, roll back to the "dynamic path selection" link, and switch to the alternate path.

[0037] (6) Result evaluation: Automatically score from dimensions such as factual consistency and logical completeness; if the result does not meet the expectation, enter the link of modifying the prompt and refining the task; including optimizing the prompt and refining the task, and then feedback to the data input / data preprocessing / dynamic path selection stage (triggering the corresponding steps according to the adjusted content), forming a closed-loop optimization.

[0038] (7) Reply end: When the result meets the expectation, output the final processing result (such as the credit approval conclusion, etc.), and the process terminates.

[0039] The above is only to illustrate the implementation manner of the present invention and is not used to limit the present invention. For those skilled in the art, any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for dynamically processing complex business data based on a large model, characterized in that, include: Step (1) preprocessing the input complex business data, including data cleaning, format conversion and feature extraction operations, to generate standardized data suitable for large model processing; Step (2) based on the standardized data, dynamically select the optimal reasoning path through the large model and according to the preset evaluation criteria, the reasoning path includes a task-oriented workflow path and a dynamic reasoning path embedded with a real-time decision maker; wherein the dynamic reasoning path uses a non-uniform segmentation algorithm to parallelly segment the data, and the real-time decision maker adjusts the path according to the business rule matching degree and resource consumption trade-off, and uses the path memory library to record the historical optimal path combination; Step (3) selecting the most suitable processing flow from a predefined professional processing flow library according to business data characteristics and task requirements, each of which represents a specific problem-solving strategy and can be optimized for different business scenarios; Step (4) executing the task according to the selected reasoning path and processing flow, monitoring the task progress in real time and dynamically adjusting the path and flow according to feedback; Step (5) uses a three-level evaluation system to evaluate the task results, and iteratively optimizes the preset standards and processing procedures based on the evaluation results; the three-level evaluation system includes: real-time evaluation, path comparison, and manual feedback reinforcement learning.

2. The method for dynamically processing complex service data based on a large model according to claim 1, wherein, The data cleaning in step (1) includes noise filtering, missing value interpolation and outlier replacement; the format conversion includes data type conversion, encoding conversion and time format standardization.

3. A method for dynamically processing complex service data based on a large model according to claim 1, characterized in that In the workflow path selection mechanism described in step (2), a "task-subtask" decomposition graph is established, and user requirements are mapped to preset workflow nodes through semantic analysis; each preset workflow node has a built-in dynamic routing logic, which automatically selects an inference path based on the characteristics of the input data.

4. A method for dynamically processing complex service data based on a large model according to claim 1 or 3, characterized in that In the dynamic reasoning path selection mechanism described in step (2), task flow selection decisions are made by classifying user questions, and the output probability of the large model is adjusted by temperature scaling.

5. A method for dynamically processing complex business data based on a large model according to claim 1, characterized in that The professional processing flow library described in step (3) supports hot update, can dynamically add industry-specific processes, and retrieve adaptive processes through fuzzy matching algorithms.

6. A system for dynamically processing complex business data based on a large model, characterized in that, include: Data preprocessing module: integrates data cleaning unit, format conversion unit and feature extraction unit, supports prompt word generation of semantic heat map and code execution data conversion; Dynamic path selection unit: includes "task-subtask" decomposition graph, real-time decision maker and path memory, supporting dynamic generation of reasoning paths based on business rules, resource consumption and context length; Professional processing flow library: stores scalable scenario-based flow templates and supports linkage calls with large models; Task execution module: includes model calling engine and real-time monitoring instrument components, supports multi-model parallel computing and dynamic path adjustment; Result evaluation module: implements a three-level evaluation system, outputs path optimization suggestions and feeds them back to the big model.

7. A system for dynamically processing complex business data based on a large model according to claim 6, wherein, The real-time decision maker adjusts the reasoning path based on three dimensions: business rule matching degree, resource consumption trade-off, and context length perception.

8. A system for dynamically processing complex business data based on a large model according to claim 7, characterized in that, The path memory adopts a case-based reasoning algorithm to store historical optimal path combinations and automatically reuse them in similar tasks.

9. A system for dynamically processing complex service data based on a large model according to claim 6, characterized in that, The system supports docking with a graph database, generates a schema-based thinking chain containing subtask dependencies, and realizes the structured display and interpretability analysis of the reasoning path.

Images