Data processing method and device, nonvolatile storage medium and computer equipment

Abstract

The invention discloses a data processing method and device, a nonvolatile storage medium and computer equipment. The method comprises the steps that an enhancement instruction input based on a user side is acquired, the enhancement instruction is determined based on a basic analysis result of index data corresponding to a target product, and the enhancement instruction represents enhancement analysis on the index data; an enhancement instruction is input into the large language model, a corresponding enhancement sub-task is extracted, the enhancement sub-task comprises at least one of a prediction sub-task and a factor reduction task, the prediction sub-task is used for predicting the change trend of the index data in a preset time period, and the factor reduction task is used for determining factors enabling the index data to change; and based on the enhancement subtask, determining an enhancement analysis result of the index data. The technical problem that the accuracy and depth of data analysis are low due to the fact that an existing large model is only used for basic data processing is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data processing method, device, non-volatile storage medium and computer equipment. Background Art

[0002] In the modern business environment where data drives decision-making, data platforms and business intelligence (BI) systems, as the key to enterprise digital transformation, face the challenge of how to effectively integrate and utilize ultra-large-scale data. In recent years, with the rapid development of artificial intelligence technology, especially the widespread application of pre-trained large language models, the field of data analysis is exploring new paths to enhance data processing capabilities and improve decision-making efficiency through AI.

[0003] The big model has shown great potential in generating SQL statements, providing intelligent question-and-answer services, and generating preliminary data reports, simplifying the data usage process for non-professional users. However, although the big model has shown its potential in the field of data analysis, it still faces some significant problems in practical applications. For example, there are problems with the accuracy of SQL generation, and it is difficult to adapt to different database syntaxes; for another example, in the face of the needs of unstructured knowledge query and deep analysis report generation, the existing solutions are also incapable of fully integrating professional knowledge, which limits the depth and breadth of data analysis.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a data processing method, apparatus, non-volatile storage medium and computer device to at least solve the technical problem that currently large models are only used for basic data processing, resulting in low accuracy and depth of data analysis.

[0006] According to one aspect of an embodiment of the present invention, a data processing method is provided, comprising: obtaining an enhancement instruction based on a user-side input, wherein the enhancement instruction is determined based on a basic analysis result of indicator data corresponding to a target product, the indicator data includes quantity and amount, the basic analysis result includes at least one of the following: a query result, a calculation result, and a visualization result, and the enhancement instruction represents an enhanced analysis of the indicator data; inputting the enhancement instruction into a large language model, extracting a corresponding enhancement subtask, wherein the enhancement subtask includes at least one of the following: a prediction subtask and an attribution factor task, the prediction subtask is used to predict a change trend of the indicator data within a preset time period, and the attribution factor task is used to determine a factor that causes the indicator data to change; based on the enhancement subtask, determining an enhanced analysis result of the indicator data.

[0007] Optionally, the basic analysis result is determined based on the following method, including: obtaining basic instructions based on user-side input, wherein the basic instructions represent a basic analysis of the indicator data; inputting the basic instructions into a large language model, and extracting corresponding basic subtasks, wherein the basic subtasks include query subtasks, calculation subtasks, and visualization subtasks, the query subtask is used to query basic data corresponding to the indicator data, the calculation subtask is used to calculate the indicator data, and the visualization subtask is used to display the indicator data; based on the basic subtasks, determining the basic analysis result.

[0008] Optionally, the query subtask includes an unstructured data query subtask, wherein the unstructured data query subtask represents a query task without a unique standard answer, and the unstructured data query subtask is executed based on the RAG enhanced retrieval generation technology.

[0009] Optionally, the enhancement instruction is input into the large language model to extract the corresponding enhancement subtask, including: matching the keyword corresponding to the enhancement instruction based on a preset keyword database; and determining the enhancement subtask based on the DSL domain specific language and the keyword.

[0010] Optionally, when the enhancement subtask is a prediction subtask, based on the enhancement subtask, an enhanced analysis result of the indicator data is determined, including: based on multiple prediction small models preset in the large language model, the enhancement subtask is executed simultaneously to obtain initial results of multiple enhancement subtasks; and the average value of the initial results of multiple enhancement subtasks is calculated to obtain the enhanced analysis result.

[0011] Optionally, in the case where the enhanced subtask is an attribution factor task, based on the enhanced subtask, an enhanced analysis result of the indicator data is determined, including: based on multiple attribution small models preset in the large language model, the attribution factor tasks are respectively and simultaneously executed to obtain multiple factors; the Shapley values ​​corresponding to each of the multiple factors are calculated; and the factors whose corresponding Shapley values ​​among the multiple factors exceed a preset threshold are taken as the enhanced analysis results.

[0012] According to another aspect of an embodiment of the present invention, a data processing device is also provided, including: an acquisition module, used to acquire enhancement instructions based on user-side input, wherein the enhancement instructions are determined based on basic analysis results of indicator data corresponding to the target product, the indicator data include quantity and amount, the basic analysis results include at least one of the following: query results, calculation results and visualization results, and the enhancement instructions represent enhanced analysis of the indicator data; an extraction module, used to input the enhancement instructions into a large language model, and extract corresponding enhancement subtasks, wherein the enhancement subtasks include at least one of the following: a prediction subtask and an attribution factor task, the prediction subtask is used to predict the change trend of the indicator data within a preset time period, and the attribution factor task is used to determine the factors that cause the indicator data to change; a determination module, used to determine the enhanced analysis results of the indicator data based on the enhancement subtask.

[0013] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided, the non-volatile storage medium including a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any one of the above data processing methods.

[0014] According to yet another aspect of an embodiment of the present invention, a computer device is provided. The computer device includes a processor, and the processor is used to run a program. When the program is run, any one of the above data processing methods is executed.

[0015] According to yet another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, any one of the above data processing methods is implemented.

[0016] In an embodiment of the present invention, a data processing method is adopted, by obtaining an enhanced instruction based on a user-side input, wherein the enhanced instruction is determined based on a basic analysis result of the indicator data corresponding to the target product, the indicator data includes quantity and amount, the basic analysis result includes at least one of the following: query result, calculation result and visualization result, and the enhanced instruction represents an enhanced analysis of the indicator data; the enhanced instruction is input into a large language model, and the corresponding enhanced subtask is extracted, wherein the enhanced subtask includes at least one of the following: a prediction subtask and an attribution factor task, the prediction subtask is used to predict the change trend of the indicator data within a preset time period, and the attribution factor task is used to determine the factor that causes the indicator data to change; based on the enhanced subtask, the enhanced analysis result of the indicator data is determined, so as to achieve the purpose of providing a more in-depth change trend prediction and attribution analysis, thereby realizing the technical effect of improving the accuracy and comprehensiveness of data analysis, and further solving the technical problem that the current large model is only used for basic data processing, resulting in low accuracy and depth of data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a computer terminal for implementing a data processing method is shown;

[0019] Figure 2 is a flow chart of a data processing method provided according to an embodiment of the present invention;

[0020] Figure 3 is a flowchart of a large model enhanced analysis method based on Text2DSL according to an optional embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of a large model enhanced analysis system based on Text2DSL according to an optional embodiment of the present invention;

[0022] Figure 5 is a structural block diagram of a data processing device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0026] Augmented Analytics: It is a method that uses technologies such as machine learning and natural language processing to assist in data preparation and insight mining, supporting professionals and general users to quickly use data and make decisions. Augmented analytics includes three categories: diagnostic analysis - why data events occur, commonly seen in the attribution of abnormal changes in indicators, predictive analysis - future data trends, commonly seen in the classification or regression prediction of indicators, and prescriptive analysis - how to make decisions based on data, focusing on making specific recommendations.

[0027] Large Language Model: A large model is a deep model based on the Transformer structure, with more than billions of parameters and complex computing structures. It is trained on a variety of excessive data sources and used for multiple tasks in the fields of natural language processing and computer recognition. It emerges with intelligence and shows a level close to or even higher than that of humans. It belongs to the category of pre-trained language models, which have fewer parameters and training data than the former and do not have the ability to emerge.

[0028] DSL (Domain Specific Language): is a programming language designed for a specific domain. Compared with general programming languages, DSL is more focused on a certain professional field and provides a means to directly express the rules and operations in that field. Complex and repetitive business operations can be expressed by writing simple DSL.

[0029] RAG (Retrieval-Augmented Generation): is a technology that retrieves and matches relevant knowledge from the knowledge base, enhances the task understanding ability of the large model and improves the quality of answer generation. RAG technology ensures that the answers generated by the large model are based on real-time professional field information, thereby effectively avoiding the large model hallucination problem, and has great application value in scenarios such as professional knowledge question and answer and task-based dialogue.

[0030] According to an embodiment of the present invention, a data processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a data processing method. Figure 1As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0032] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0033] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the data processing method of the above-mentioned application program. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0034] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0035] Figure 2 is a flow chart of a data processing method according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0036] Step S201, obtaining an enhanced instruction based on user-side input, wherein the enhanced instruction is determined based on a basic analysis result of indicator data corresponding to the target product, the indicator data includes quantity and amount, the basic analysis result includes at least one of the following: query result, calculation result and visualization result, and the enhanced instruction represents an enhanced analysis of the indicator data.

[0037] In this step, when the user interacts with the system, he has received the basic analysis results on the quantity and amount indicators of the target product through preliminary query, calculation or visualization operations. These basic analysis results provide the current state of the data, but are often insufficient to meet the user's needs for in-depth business insights. Based on the review and understanding of the basic analysis results, the user may make further analysis requests, which are intended to go beyond simple data retrieval and display, explore the deep meaning behind the data or predict the future trend of the data. For example, the user may request to analyze the month-on-month growth of the transaction volume of product p in May, and further inquire about the forecast trend in the next 7 days and the main factors leading to the change in transaction volume. This type of request is an "enhanced instruction", which requires the system to not only provide data, but also perform enhanced analysis tasks such as predictive analysis and causal attribution to provide richer and more decision-making insights. The generation of enhanced instructions is based on the user's interpretation and feedback on the basic analysis results. The user can make deeper analysis requirements by observing charts, reading calculation results or based on personal experience.

[0038] Step S202, input the enhancement instruction into the large language model, and extract the corresponding enhancement subtask, wherein the enhancement subtask includes at least one of the following: a prediction subtask and an attribution task, the prediction subtask is used to predict the change trend of the indicator data within a preset time period, and the attribution task is used to determine the factors that cause the indicator data to change.

[0039] In this step, when the user issues an enhanced instruction, these instructions are usually expressed in the form of natural language, involving the need for in-depth analysis of existing data, such as predicting the future trend of product sales or analyzing the main driving factors of sales changes. The large language model can parse the semantics of user instructions through its excellent natural language understanding and generation capabilities, and identify the prediction or attribution analysis requirements contained therein. Specifically, a prompt template for subtask planning can be constructed, and the large model can extract complex tasks into simple subtasks and generate them in json format. At the same time, subtask planning examples corresponding to common problems can be pre-written and common problems can be encoded into vector storage through a pre-trained language model, so that the input enhanced instruction of any user can extract the closest subtask planning example through vector matching and fill in as part of the prompt word. Since the basic analysis of the indicator data has been performed before the enhanced analysis, after interacting with the user and obtaining the instruction to continue the enhanced analysis, the large model can switch the prompt template different from the basic analysis stage and perform task planning again. Different prompt templates effectively isolate basic data analysis and enhanced analysis, feed different expertise to the big model to perform tasks at different stages, and avoid putting all prompts into a unified template and reducing the planning performance of the big model.

[0040] The prediction subtask generates a specific data prediction plan based on the prediction requirements in the user's instructions. For example, it can include: selecting or recommending a prediction model (such as a time series analysis model, a machine learning regression model, etc.), setting the model's training parameters (such as the length of the time window, the length of the prediction window, etc.), and determining the prediction time range or period. For example, if the user wants to predict the development of the product p transaction volume in the next 7 days, the big model will generate a prediction subtask, set the training window to the past 32 days of data, and the prediction window to the next 7 days, so as to predict the future trend of the target indicator for the user.

[0041] The attribution task focuses on determining which factors have caused changes in specific indicator data, which is crucial to understanding the reasons behind the data changes. For example, when a user asks to analyze the main reasons for the increase in the transaction volume of product p, the big model will generate an attribution task, summarize the potential combination of influencing factors based on the relevant knowledge retrieved from the knowledge base, and use a specific attribution analysis model (such as the attribution model based on Shapley Value) to evaluate the contribution of each factor. The system selects the top few factors with the highest contribution as the main influencing factors, and feeds these results back to the big model in a structured form for in-depth interpretation and suggestion generation in the final report.

[0042] Step S203: Determine the enhanced analysis result of the indicator data based on the enhanced subtask.

[0043] In this step, the determination of the enhanced analysis results combines the outputs of the prediction subtask and the attribution subtask. The prediction results reveal the future trend of the indicator data, while the attribution analysis analyzes the reasons behind the indicator changes. These results will be returned to the large language model in a structured form, summarized through a fixed template, and the basic data collection and analysis, indicator prediction results and attribution results will be displayed in turn. Finally, the large model will generate an analysis report with expert insights based on professional knowledge and analysis results, and provide practical guidance for business strategy formulation. Deep analysis of predictive, diagnostic and prescriptive nature in enhanced analysis is achieved. For example, based on the results of prediction and attribution analysis, the large model will generate a report that not only shows the predicted trend of the future transaction volume of product p, but also explains the reasons for the possible increase in transaction volume, such as the positive impact of promotional activities, and puts forward corresponding business suggestions, such as increasing promotion efforts or optimizing promotion strategies, in order to achieve better business results.

[0044] Through the above steps, the purpose of providing more in-depth trend prediction and attribution analysis is achieved, thereby realizing the technical effect of improving the accuracy and comprehensiveness of data analysis, and further solving the technical problem that the current large model is only used for basic data processing, resulting in low accuracy and depth of data analysis.

[0045] As an optional embodiment, the basic analysis result is determined based on the following method, including: obtaining basic instructions based on user-side input, wherein the basic instructions represent a basic analysis of the indicator data; inputting the basic instructions into a large language model, and extracting corresponding basic subtasks, wherein the basic subtasks include query subtasks, calculation subtasks, and visualization subtasks, the query subtask is used to query the basic data corresponding to the indicator data, the calculation subtask is used to calculate the indicator data, and the visualization subtask is used to display the indicator data; based on the basic subtasks, determining the basic analysis result.

[0046] Optionally, before performing enhanced analysis of the data, basic analysis can be performed first. Specifically, the system receives basic instructions input by the user, which are expressed in natural language and cover the user's query, calculation or display needs for specific indicator data. For example, a user may ask, "What is the transaction volume of product p of department A in May?" Such a demand is a basic instruction, which points to a preliminary analysis of specific data. The system inputs the basic instructions into the large language model. Based on its powerful natural language understanding ability, the large model parses the semantics of the instructions and generates corresponding basic subtask descriptions, including query subtasks, calculation subtasks, and visualization subtasks.

[0047] The query subtasks may include structured data query subtasks and unstructured data query subtasks. For the query subtasks of structured data, a parser may be set to convert the DSL language into the query language corresponding to the data source. For example, for the read-in csv file, the parser converts the query subtasks described in the DSL language into the Pandas query language, and for data sources such as Hive and Mysql, it can be converted into corresponding query statements. Compared with the traditional Text2SQL method, converting the problem into DSL first and then converting it into SQL or other query statements according to the actual data source can effectively constrain and standardize the query statements generated by the large model, reduce the illusion problem, and improve the query accuracy in professional fields.

[0048] At the same time, in order to further improve the accuracy of the query conditions generated by the large model, it is necessary to perform vector matching of the generated query departments and query indicators with all department and indicator names respectively, and select the department and indicator names with the highest vector similarity as the constraints finally sent to the query engine. Specifically, all department names and indicator names are concatenated in advance, encoded into vectors through a pre-trained language model, and a vector library of department names + indicator names is constructed. The query department and indicator conditions generated by the large model are also encoded as vectors, and the cosine similarity is calculated in the vector library. The department name + indicator name with the highest similarity is taken out, which is the final condition to be queried:

[0049]

[0050] The above is the cosine similarity calculation formula, i and j are the two vectors whose similarity needs to be calculated.

[0051] The calculation subtask can input the query results of structured data into a calculator developed based on Python's math package to complete the calculation. The actual role of the calculation subtask is to reduce the difficulty of obtaining derived indicator data and improve the accuracy of acquisition. For example, if the user needs "the average monthly transaction amount of product p", in the task planning generated by the large model, the entire task is split into first collecting the daily transaction amount through the query, and then handing it over to the calculation tool to calculate the average of the collected data to complete the entire analysis process. In addition to the mean peak value, the year-on-year and year-on-year comparison of indicators in different statistical periods has always been a commonly used calculation method in data analysis. The task planning described by the DSL language can decompose these processes into two steps: collecting basic data and data processing. Compared with the existing large model that directly generates a set of SQL methods including reading and calculation, it reduces the task difficulty of the large model and avoids hallucinations, and improves the reliability of the entire analysis process.

[0052] The visualization subtask can input the query results or calculation results of structured data into the visualization tool developed based on the Python matplotlib package to complete the visualization execution. Other parameters of the visualization tool, such as chart types, have been pre-set when the large model is used for task planning, so there is no need to call the large model to generate repeatedly. The actual function of visualization is to achieve the role of intelligently generating charts and converting complex data arrangements into intuitive and easy-to-understand charts.

[0053] After completing each subtask, the user's original question, subtask plan, and subtask execution results can be sent to the big model, allowing the big model to summarize the results and describe the visual results. This process completes the basic data collection and processing of the user's question, which belongs to the intermediate results in the field of enhanced analysis and is returned to the user first. The basic data collection and visual analysis only need to call the big model once to generate, so the overall response time is short. It usually takes 3-5 seconds from the user inputting the question to returning the result. Then, based on the above intermediate results, two interactive goals can be completed in sequence on the user side. Specifically, the first interactive goal is "Does the user need to collect other data for the current question?" Based on the intermediate results, the user may need other supporting data. For example, if the user wants to compare the transaction volume of another product q in May, based on the multi-round conversation capability of the big model, the task plan can be issued again. For the above task plan, it is a derivative of the big model based on the user's requirements, so the subtask number depends on the previous task plan, which means that it is the post-placement of the previous task plan. When the user has no other data collection requirements, the big model interacts with the second goal. The second interactive goal is "Does the user need to perform enhanced analysis for the current problem?", which can guide the user to give an affirmative or negative answer. If the user does not need to perform enhanced data analysis, the question is answered. If the user gives an affirmative answer, the subsequent enhanced analysis phase will be entered.

[0054] As an optional embodiment, a large model enhanced analysis method based on Text2DSL is also provided. Figure 3 is a flowchart of a large model enhanced analysis method based on Text2DSL according to an optional embodiment of the present invention. Figure 3As shown in the figure, the system first receives the data analysis requirements proposed by the user in natural language, and then parses these requirements through the large language model and converts them into basic or enhanced analysis subtasks based on DSL (domain-specific language) descriptions, including data query, calculation, visualization, prediction and attribution analysis. Subsequently, the system executes the subtasks described by these DSLs, using professional tools to retrieve, process, predict and attribution analysis of data, and enhances the retrieval of unstructured knowledge through RAG technology. The results of the subtask execution are summarized and a detailed analysis report is generated through the large model, covering basic data, prediction results, attribution analysis and intuitive chart display, and outputting the final result report.

[0055] As an optional embodiment, the query subtask includes an unstructured data query subtask, wherein the unstructured data query subtask represents a query task without a unique standard answer, and the unstructured data query subtask is executed based on the RAG enhanced retrieval generation technology.

[0056] Alternatively, structured data sources often cannot meet the user's question requirements. For example, users may ask for detailed information about product p, such as the product's design plan, person in charge, and other knowledge. These "chatty" questions do not have a single standard answer, and often require a broad knowledge base to provide background information. Therefore, RAG technology can be used as a submodule to query knowledge carried by unstructured natural language. RAG technology ensures that the answers generated by the big model are based on immediate and professional field information, thereby effectively avoiding the "hallucination" problem caused by the limitations of the training data of the big model and enhancing the accuracy and reliability of the answers. Specifically, RAG will quickly locate the relevant text in the knowledge base based on the keywords involved in the user's instructions, such as department name, indicator type, time range, etc., and extract the most relevant knowledge fragments with high precision by calculating vector similarity. These knowledge fragments may include the definition of specific indicators, description of business processes, industry trend analysis, etc., providing rich background information for the big model, so that it can combine professional knowledge when generating answers and provide more detailed and valuable explanations.

[0057] The execution of the unstructured data query subtask is based on the multi-indexed RAG vector library. This indexing mechanism pre-classifies and encodes the text in the knowledge base according to dimensions such as time, department, and indicators, and builds a vector library for efficient retrieval. When a user asks a question, the system can quickly locate the corresponding index based on the keywords of the question and retrieve highly matched knowledge fragments from the vector library of the relevant text, thereby greatly improving the efficiency and accuracy of the retrieval. This process not only helps the large model better understand the key points of the problem and avoid semantic interference, but also provides important information supplements for generating result reports, enhancing the comprehensiveness and professionalism of the report.

[0058] As an optional embodiment, the enhancement instruction is input into the large language model to extract the corresponding enhancement subtask, including: matching the keyword corresponding to the enhancement instruction based on a preset keyword database; and determining the enhancement subtask based on the DSL domain specific language and keywords.

[0059] Optionally, when users issue enhanced instructions, these instructions may contain a series of keywords, such as "forecast", "trend", "attribution", "growth", etc. These keywords express the user's specific needs for in-depth data analysis. The system first uses the preset keyword database to match the user's instructions with keywords. This database contains common keywords and their related meanings in the field of data analysis, which can help the system quickly understand the intent and type of the instruction. For example, when the user's instruction contains the keyword "forecast", the system will recognize that the user's need is to predict the trend of future data. This step is the basis for subsequent task planning.

[0060] Based on the matched keywords, the system further uses DSL (Domain Specific Language) to determine the specific enhanced subtasks. DSL is a programming language designed for a specific domain, which provides a direct and concise way to express the operations and rules in the domain. In this embodiment, DSL is designed to plan and describe data analysis tasks. With the guidance of keywords, the system can generate DSL descriptions of enhanced subtasks such as prediction subtasks or attribution tasks, including task parameters, objectives, and execution methods. For example, for instructions to predict the future trend of the transaction amount of product p, the DSL description generated by the system may include: the selected time series model, the time window for training and prediction, the number of days for prediction, and other key information, ensuring the accuracy of the enhanced analysis task and the efficiency of execution.

[0061] For query subtasks, the big model can generate query restrictions, such as limited department names, indicator names, time ranges, etc. For calculation subtasks, the big model can generate calculation methods, which can only be selected from the following eight options: sum, mean, maximum, minimum, difference, month-on-month, year-on-year, and proportion. Other parameters that can be generated include: x, y, and result name. x and y are operators involved in the calculation, and null values ​​can be accepted. For example, when you need to calculate the sum of the elements contained in x, you can specify y as a null value. For visualization subtasks, the big model can generate visualization methods, which can only be selected from the following three options: bar chart, line chart, and pie chart. Other parameters that can be generated include: x, y, x_label, y_label, and result name. x and y are the horizontal and vertical coordinates, respectively, and x_label and y_label are the coordinate axis scales, respectively.

[0062] Through the above method, a DSL language dedicated to the field of data analysis can be constructed, and the large model can directly understand the semantic relationship between the subtask format and the original problem, thereby generating a stable data analysis task plan that is decoupled from the business logic of the data source.

[0063] As an optional embodiment, when the enhancement subtask is a prediction subtask, based on the enhancement subtask, an enhanced analysis result of the indicator data is determined, including: based on multiple prediction small models preset in the large language model, the enhancement subtask is executed simultaneously to obtain initial results of multiple enhancement subtasks; and the average value of the initial results of multiple enhancement subtasks is calculated to obtain the enhanced analysis result.

[0064] Optionally, in the prediction subtask, the large model can set one or more small regression models to predict the target indicator and the length of time for the prediction. For example, model a is a time series prediction model based on bidirectional LSTM, which has completed the training and weight preservation of the predictions of various indicators in advance, while model b is a small MLP network that predicts independently from model a. When this subtask is executed, the corresponding model weights can be called and the indicator values ​​can be predicted; when encountering indicators and weights that have not been pre-trained, long-term data will be automatically taken out for training, and then predictions will be made. In addition, while predicting accurate indicator values, interval predictions can also be introduced, such as 95% and 90% interval ranges. The function of the prediction interval is to explain the probability that the actual value of the indicator is located in different ranges, and to improve the interpretability of the indicator prediction function. Its training method uses Quantile Loss to complete:

[0065]

[0066] The prediction subtask will call each small model planned by the large model to perform prediction. The final prediction result is obtained by averaging the prediction results of the small models and displaying them visually.

[0067] As an optional embodiment, when the enhanced subtask is an attribution task, based on the enhanced subtask, an enhanced analysis result of the indicator data is determined, including: based on multiple attribution small models preset in the large language model, the attribution task is executed respectively and simultaneously to obtain multiple factors; the Shapley values ​​corresponding to each of the multiple factors are calculated; and the factors whose corresponding Shapley values ​​exceed a preset threshold are taken as the enhanced analysis results.

[0068] Optionally, similar to the prediction subtask, the big model calls different attribution models to perform the attribution task. The big model will summarize the potential factor combinations based on the knowledge retrieved from the knowledge base and send them to the attribution model as features. This embodiment uses Shapley Value to measure the contribution of each factor feature, and takes the top three factors as the main factors for the indicator changes, and submits them to the big model in json format, so that the big model can generate the next report based on professional domain knowledge and actual factor contribution.

[0069] As an optional embodiment, a large model enhanced analysis system based on Text2DSL is also provided. Figure 4 Schematic diagram of a large model enhanced analysis system based on Text2DSL provided according to an optional embodiment of the present invention. Figure 4 As shown in the figure, the system can include four main modules: 1) Task planning for basic data collection and analysis; 2) Subtask execution and summary of intermediate results; 3) Enhanced analysis task planning; 4) Enhanced analysis task execution and report generation. In basic data collection, DSL language is used to describe tasks, and it is aligned to different data source query languages ​​for structured data query and unstructured knowledge extraction. The data processing and visualization are then completed by the calculation subtask and visualization subtask, and the intermediate results are returned to the user first. When the user determines that in-depth analysis is required, the large model plans the prediction subtask and attribution task again, and creates or calls a small model to complete it. The final report summarizes the execution results of each step and the insights generated by the expert knowledge of the large model, realizing the process from basic data collection, visualization analysis, to enhanced analysis completed by the machine learning model. Among them, the pre-trained language model can use chinese-roberta-wwm-ext; the large language model can use MiniCPM-2B-sft-bf16; the vector library for unstructured knowledge storage can use FAISS; the indicator prediction and change attribution model can use bidirectional LSTM and MLP, and the training hyperparameters include learning rate = 1e -3 , training step size = 32, number of hidden layers = 2, hidden layer dimension = 32, training epoch = 200, training batchsize = 128.

[0070] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0071] Through the description of the above implementation methods, those skilled in the art can clearly understand that the data processing method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0072] According to an embodiment of the present invention, a data processing device for implementing the above data processing method is also provided. Figure 5 is a structural block diagram of a data processing device provided according to an embodiment of the present invention, such as Figure 5 As shown, the data processing device includes: an acquisition module 51, an extraction module 52 and a determination module 53. The data processing device is described below.

[0073] The acquisition module 51 is used to obtain the enhanced instruction based on the user-side input, wherein the enhanced instruction is determined based on the basic analysis result of the indicator data corresponding to the target product, the indicator data includes quantity and amount, the basic analysis result includes at least one of the following: query result, calculation result and visualization result, and the enhanced instruction represents the enhanced analysis of the indicator data.

[0074] The extraction module 52 is connected to the acquisition module 51, and is used to input the enhancement instruction into the large language model and extract the corresponding enhancement subtask, wherein the enhancement subtask includes at least one of the following: a prediction subtask and an attribution task. The prediction subtask is used to predict the change trend of the indicator data within a preset time period, and the attribution task is used to determine the factors that cause the indicator data to change.

[0075] The determination module 53 is connected to the extraction module 52 and is used to determine the enhanced analysis result of the indicator data based on the enhanced subtask.

[0076] It should be noted that the acquisition module 51, extraction module 52 and determination module 53 correspond to steps S201 to S203 in the embodiment, and the examples and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in the embodiment.

[0077] An embodiment of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one network device among multiple network devices of a computer network. The computer device includes a memory and a processor.

[0078] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the data processing method and device in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned data processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0079] The processor can call the information and application programs stored in the memory through the transmission device to execute the following steps: obtain an enhancement instruction based on the user-side input, wherein the enhancement instruction is determined based on the basic analysis results of the indicator data corresponding to the target product, the indicator data includes quantity and amount, the basic analysis results include at least one of the following: query results, calculation results and visualization results, and the enhancement instruction represents the enhanced analysis of the indicator data; input the enhancement instruction into the large language model, and extract the corresponding enhancement subtask, wherein the enhancement subtask includes at least one of the following: a prediction subtask and an attribution factor task, the prediction subtask is used to predict the change trend of the indicator data within a preset time period, and the attribution factor task is used to determine the factors that cause the indicator data to change; based on the enhancement subtask, determine the enhanced analysis results of the indicator data.

[0080] According to an embodiment of the present invention, a data processing method is provided. By obtaining an enhanced instruction based on user-side input, wherein the enhanced instruction is determined based on the basic analysis result of the indicator data corresponding to the target product, the indicator data includes quantity and amount, and the basic analysis result includes at least one of the following: query result, calculation result and visualization result, and the enhanced instruction represents the enhanced analysis of the indicator data; the enhanced instruction is input into a large language model, and the corresponding enhanced subtask is extracted, wherein the enhanced subtask includes at least one of the following: a prediction subtask and an attribution factor task, the prediction subtask is used to predict the change trend of the indicator data within a preset time period, and the attribution factor task is used to determine the factor that causes the indicator data to change; based on the enhanced subtask, the enhanced analysis result of the indicator data is determined, so as to achieve the purpose of providing a more in-depth change trend prediction and attribution analysis, thereby achieving the technical effect of improving the accuracy and comprehensiveness of data analysis, and thus solving the technical problem that the current large model is only used for basic data processing, resulting in low accuracy and depth of data analysis.

[0081] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a non-volatile storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0082] The embodiment of the present invention further provides a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the data processing method provided by the above embodiment.

[0083] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0084] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: obtaining an enhancement instruction based on user-side input, wherein the enhancement instruction is determined based on a basic analysis result of the indicator data corresponding to the target product, the indicator data includes quantity and amount, the basic analysis result includes at least one of the following: query results, calculation results and visualization results, and the enhancement instruction represents an enhanced analysis of the indicator data; inputting the enhancement instruction into the large language model, extracting the corresponding enhancement subtask, wherein the enhancement subtask includes at least one of the following: a prediction subtask and an attribution task, the prediction subtask is used to predict the changing trend of the indicator data within a preset time period, and the attribution task is used to determine the factors that cause the indicator data to change; based on the enhancement subtask, determining the enhanced analysis result of the indicator data.

[0085] An embodiment of the present invention also provides a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it can achieve: obtaining an enhancement instruction based on a user-side input, wherein the enhancement instruction is determined based on a basic analysis result of the indicator data corresponding to the target product, the indicator data includes quantity and amount, and the basic analysis result includes at least one of the following: query results, calculation results and visualization results, and the enhancement instruction represents an enhanced analysis of the indicator data; inputting the enhancement instruction into a large language model, extracting the corresponding enhancement subtask, wherein the enhancement subtask includes at least one of the following: a prediction subtask and an attribution factor task, the prediction subtask is used to predict the change trend of the indicator data within a preset time period, and the attribution factor task is used to determine the factors that cause the indicator data to change; based on the enhancement subtask, determining the enhanced analysis result of the indicator data.

[0086] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0087] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0088] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0089] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0090] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0091] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0092] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Acquire an enhancement instruction based on a user terminal input, wherein the enhancement instruction is determined based on a basic analysis result of indicator data corresponding to the target product, the indicator data includes quantity and amount, the basic analysis result includes at least one of the following: a query result, a calculation result, and a visualization result, and the enhancement instruction represents an enhanced analysis of the indicator data; Input the enhancement instruction into the large language model, and extract the corresponding enhancement subtask, wherein the enhancement subtask includes at least one of the following: a prediction subtask and an attribution task, wherein the prediction subtask is used to predict the change trend of the indicator data within a preset time period, and the attribution task is used to determine the factor that causes the indicator data to change; Based on the enhanced subtask, an enhanced analysis result of the indicator data is determined.

2. The method according to claim 1, characterized in that: The basic analysis results are determined based on the following methods, including: Acquire a basic instruction based on a user terminal input, wherein the basic instruction represents a basic analysis of the indicator data; Inputting the basic instruction into the large language model, extracting corresponding basic subtasks, wherein the basic subtasks include query subtasks, calculation subtasks, and visualization subtasks, the query subtask is used to query basic data corresponding to the indicator data, the calculation subtask is used to calculate the indicator data, and the visualization subtask is used to display the indicator data; Based on the basic subtasks, the basic analysis results are determined.

3. The method according to claim 2, characterized in that The query subtask includes an unstructured data query subtask, wherein the unstructured data query subtask represents a query task without a unique standard answer, and the unstructured data query subtask is executed based on the RAG enhanced retrieval generation technology.

4. The method according to claim 1, characterized in that: The step of inputting the enhancement instruction into the large language model and extracting the corresponding enhancement subtask comprises: Based on a preset keyword database, matching the keyword corresponding to the enhancement instruction; The enhancement subtask is determined based on the DSL domain specific language and the keyword.

5. The method according to claim 1, characterized in that In the case where the enhancement subtask is the prediction subtask, determining the enhancement analysis result of the indicator data based on the enhancement subtask includes: Based on a plurality of small prediction models preset in the large language model, the enhancement subtasks are respectively and simultaneously executed to obtain initial results of a plurality of the enhancement subtasks; The average value of the initial results of the plurality of enhancement subtasks is calculated to obtain the enhancement analysis result.

6. The method according to claim 1, characterized in that In the case where the enhanced subtask is the attribution task, determining the enhanced analysis result of the indicator data based on the enhanced subtask includes: Based on a plurality of attribution small models preset in the large language model, the attribution factor tasks are respectively and simultaneously executed to obtain a plurality of factors; Calculating the Shapley values ​​corresponding to each of the multiple factors; The factors among the multiple factors whose corresponding Shapley values ​​exceed a preset threshold are taken as the enhanced analysis results.

7. A data processing device, characterized in that: include: an acquisition module, configured to acquire an enhancement instruction based on an input from a user terminal, wherein the enhancement instruction is determined based on a basic analysis result of indicator data corresponding to a target product, the indicator data includes quantity and amount, the basic analysis result includes at least one of the following: a query result, a calculation result, and a visualization result, and the enhancement instruction represents an enhanced analysis of the indicator data; An extraction module, used to input the enhancement instruction into the large language model, and extract the corresponding enhancement subtask, wherein the enhancement subtask includes at least one of the following: a prediction subtask and an attribution task, wherein the prediction subtask is used to predict the change trend of the indicator data within a preset time period, and the attribution task is used to determine the factor that causes the indicator data to change; A determination module is used to determine the enhanced analysis result of the indicator data based on the enhanced subtask.

8. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 6.

9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor executes the data processing method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 6 is implemented.

Images