Data processing method and device based on large model
Through the large-model-based data processing method, automated processing and analysis of enterprise operation data, the problem of low manual analysis efficiency is solved and efficient data processing and analysis is achieved.
Patent Information
- Application Number
- CN202411079146.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-08-02
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
AI Technical Summary
In the prior art, data analysts have low efficiency in manually processing and analyzing enterprise operation data, resulting in low analysis efficiency.
The data processing method based on the big model is adopted, and the data to be analyzed is obtained by receiving analysis requests, and the data statistical analysis logic is obtained, and the results are input to the data analysis model to obtain abnormal and trend analysis results, and finally sent to the terminal device.
Data processing and analysis can be completed without the participation of data analysts, which significantly improves the efficiency of data processing and analysis.
Smart Images

Figure CN119005513A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and device based on a large model. Background Art
[0002] With the advent of the big data era, data processing and analysis technologies are widely used in finance, banking, e-commerce and other fields. For enterprises in the fields of finance, banking, e-commerce and other fields, the data information brought by enterprise development is growing exponentially. In order to ensure better development of enterprises, it is necessary to process and analyze the large amount of data generated in the operation process of enterprises based on data processing and analysis technologies, so as to provide effective basis for enterprise leaders to make decisions.
[0003] At present, it is usually the data analysts of the target enterprise who manually process and analyze the data generated in the operation process of the target enterprise. That is, after the data analysts obtain the data to be analyzed according to the problem to be analyzed, they first perform data statistical analysis on the data to be analyzed based on the analysis rules to obtain the data statistical analysis results corresponding to the data to be analyzed, and then perform abnormal analysis and trend analysis on the data statistical analysis results corresponding to the data to be analyzed based on personal work experience and analysis rules to obtain abnormal analysis results and trend analysis results corresponding to the data to be analyzed. Finally, the data statistical analysis results, abnormal analysis results and trend analysis results corresponding to the data to be analyzed are provided to the enterprise leaders as decision results, so that the enterprise leaders can make decisions based on the decision results. However, it takes a lot of time for the data analysts to manually process and analyze the data generated in the operation process of the target enterprise, which leads to low analysis efficiency in data processing and analysis of the data generated in the operation process of the target enterprise. Summary of the invention
[0004] The embodiments of the present application provide a data processing method and device based on a large model, the main purpose of which is to improve the analysis efficiency of data processing and analysis of data generated during the operation of a target enterprise.
[0005] In order to solve the above technical problems, the embodiments of the present application provide the following technical solutions:
[0006] In a first aspect, the present application provides a data processing method based on a large model, the method comprising:
[0007] Receiving an analysis request sent by a target terminal device, wherein the analysis request includes a problem to be analyzed;
[0008] Acquire the data to be analyzed according to the problem to be analyzed, and determine the statistical analysis logic of the target data corresponding to the problem to be analyzed;
[0009] Determine the data statistical analysis result corresponding to the problem to be analyzed according to the data to be analyzed and the statistical analysis logic of the target data;
[0010] Inputting the data statistical analysis results into a data analysis model, the data analysis model outputs the abnormality analysis results and trend analysis results corresponding to the problem to be analyzed, wherein the data analysis model is obtained after iterative training of a preset large model;
[0011] The data statistical analysis results, abnormality analysis results and trend analysis results corresponding to the problem to be analyzed are sent to the target terminal device.
[0012] In a second aspect, the present application further provides a data processing device based on a large model, the device comprising:
[0013] A first receiving unit, configured to receive an analysis request sent by a target terminal device, wherein the analysis request includes a problem to be analyzed;
[0014] A first determining unit, configured to obtain data to be analyzed according to the problem to be analyzed, and determine a target data statistical analysis logic corresponding to the problem to be analyzed;
[0015] A second determining unit, configured to determine a data statistical analysis result corresponding to the problem to be analyzed according to the data to be analyzed and the target data statistical analysis logic;
[0016] An input unit, used to input the data statistical analysis results into a data analysis model, and the data analysis model outputs an abnormality analysis result and a trend analysis result corresponding to the problem to be analyzed, wherein the data analysis model is obtained after iterative training of a preset large model;
[0017] The sending unit is used to send the data statistical analysis results, abnormality analysis results and trend analysis results corresponding to the problem to be analyzed to the target terminal device.
[0018] In a third aspect, an embodiment of the present application provides a storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the large model-based data processing method described in the first aspect.
[0019] In a fourth aspect, an embodiment of the present application provides a data processing device based on a big model, the device comprising a storage medium; and one or more processors, the storage medium is coupled to the processor, and the processor is configured to execute program instructions stored in the storage medium; when the program instructions are executed, the data processing method based on the big model described in the first aspect is executed.
[0020] By means of the above technical solution, the technical solution provided by this application has at least the following advantages:
[0021] The present application provides a data processing method and device based on a large model. The present application can, after a data processing application receives an analysis request sent by a target terminal device and carries a problem to be analyzed, first obtain the data to be analyzed according to the problem to be analyzed by the data processing application, and determine the target data statistical analysis logic corresponding to the problem to be analyzed; then determine the data statistical analysis result corresponding to the problem to be analyzed according to the data to be analyzed and the target data statistical analysis logic; then, input the data statistical analysis result corresponding to the problem to be analyzed into the data analysis model, so that the data analysis model outputs the abnormal analysis result and trend analysis result corresponding to the problem to be analyzed; finally, send the data statistical analysis result, abnormal analysis result and trend analysis result corresponding to the problem to be analyzed to the target terminal device, so that after the target terminal device outputs and displays the data statistical analysis result, abnormal analysis result and trend analysis result corresponding to the problem to be analyzed, the user on the target terminal device side can make a decision according to the data statistical analysis result, abnormal analysis result and trend analysis result corresponding to the problem to be analyzed. Since the data processing and analysis of the data generated in the operation process of the target enterprise can be completed without the participation of the data analyst of the target enterprise, the analysis efficiency of the data processing and analysis of the data generated in the operation process of the target enterprise can be effectively improved.
[0022] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:
[0024] Figure 1 A flow chart of a data processing method based on a large model provided in an embodiment of the present application is shown;
[0025] Figure 2 Another flow chart of a data processing method based on a large model provided in an embodiment of the present application is shown;
[0026] Figure 3 A block diagram of a data processing device based on a large model provided in an embodiment of the present application is shown;
[0027] Figure 4 A block diagram showing the composition of another large model-based data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0028] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.
[0029] In addition, the words “first”, “second” and the like used in the present application do not indicate any order, quantity or importance, but are only used to distinguish different parts.
[0030] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.
[0031] At present, data analysts of target enterprises usually manually process and analyze the data generated during the operation of the target enterprises. However, it takes a lot of time for data analysts to manually process and analyze the data generated during the operation of the target enterprises, which leads to low analysis efficiency of data processing and analysis of the data generated during the operation of the target enterprises.
[0032] Therefore, in order to improve the efficiency of data processing and analysis of data generated during the operation of the target enterprise, the embodiments of the present application provide a data processing method based on a large model. In some examples, the embodiments of the present application are applied to electronic devices, which are equipped with a data processing device based on a large model, such as a data processing application. Figure 1 As shown, the method at least includes 101-105.
[0033] 101. Receive an analysis request sent by a target terminal device.
[0034] The analysis request includes a question to be analyzed, and the question to be analyzed records relevant information of the data to be processed and analyzed (i.e., the data to be analyzed), such as the target department, target time period, target indicator, etc. corresponding to the data to be analyzed. That is, according to the question to be analyzed, it can be determined which department and which indicator in which time period of the target enterprise need to be processed and analyzed. For example, when the target enterprise is an e-commerce company, the question to be analyzed can be, but is not limited to, question a, analyzing the sales situation of sales department A in July 2024. According to question a, it can be determined that the target department corresponding to the data to be analyzed is sales department A, and the target department is sales department A. The target indicators are monthly total sales, average daily sales, highest daily sales, lowest daily sales, etc., and the target time period is from July 1, 2024 to July 31, 2024; when the target enterprise is a bank, the problem to be analyzed can be, but is not limited to, problem b, analyzing the credit card processing of department B in May 2024. According to problem b, it can be determined that the target department corresponding to the data to be analyzed is department B, and the target indicators are the monthly total number of credit cards processed, the average number of credit cards processed per day, the highest number of credit cards processed per day, the lowest number of credit cards processed per day, etc., and the target time period is from May 1, 2024 to May 31, 2024. It should be noted that the above is only an illustrative example, and the embodiments of this application do not specifically limit the content of the problem to be analyzed.
[0035] In the embodiment of the present application, the execution subject in each step is a data processing application running in an electronic device, wherein the electronic device may be, but is not limited to, a computer, a server, etc.
[0036] When the user on the target terminal device side has the need to process and analyze the data generated during the operation of the target enterprise, he will write the questions to be analyzed according to the actual needs, and send an analysis request carrying the questions to be analyzed to the electronic device with the data processing application installed through the target terminal device. At this time, the data processing application can receive the analysis request sent by the target terminal device.
[0037] 102. Obtain the data to be analyzed according to the problem to be analyzed, and determine the statistical analysis logic of the target data corresponding to the problem to be analyzed.
[0038] In some embodiments, the data generated during the operation of the target enterprise are all stored in the target database; wherein, the target data statistical analysis logic corresponding to the problem to be analyzed is used to perform data statistical analysis on the data to be analyzed, and the target data statistical analysis logic can specifically be a data statistical analysis function, through which the target data statistical analysis logic can perform sum calculations, average calculations, maximum values, minimum values, etc. on the data to be analyzed.
[0039] After receiving the analysis request sent by the target terminal device, the data processing application can obtain the data to be analyzed in the target database according to the problem to be analyzed contained in the analysis request, and after obtaining the data to be analyzed, determine the target data statistical analysis logic corresponding to the problem to be analyzed.
[0040] Specifically, in this step, the data processing application can obtain the data to be analyzed according to the problem to be analyzed in the following manner: first generate an SQL statement for searching the data to be analyzed according to the problem to be analyzed, and then use the generated SQL statement to query the target database to obtain the data to be analyzed.
[0041] Specifically, in this step, the data processing application can determine the target data statistical analysis logic corresponding to the problem to be analyzed in the following manner: pre-store multiple sample problems and the data statistical analysis logic corresponding to each sample problem in a local storage space or a target database, match the problem to be analyzed with the multiple sample problems, and determine the data statistical analysis logic corresponding to the successfully matched sample problem as the target data statistical analysis logic corresponding to the problem to be analyzed.
[0042] 103. Determine the data statistical analysis result corresponding to the problem to be analyzed based on the statistical analysis logic of the data to be analyzed and the target data.
[0043] After the data processing application obtains the data to be analyzed and determines the target data statistical analysis logic corresponding to the problem to be analyzed, it can determine the data statistical analysis results corresponding to the problem to be analyzed based on the data to be analyzed and the target data statistical analysis logic, that is, perform data statistical analysis on the data to be analyzed based on the target data statistical analysis logic to obtain the data statistical analysis results corresponding to the problem to be analyzed.
[0044] 104. Input the data statistical analysis results into the data analysis model so that the data analysis model can output the abnormal analysis results and trend analysis results corresponding to the problem to be analyzed.
[0045] Among them, the data analysis model is obtained after iterative training of the preset large model based on the training sample set. The preset large model can be any large model (Large Language Models) known in the field of data processing technology. The embodiment of the present application does not specifically limit the specific type of the preset large model. The training sample set includes multiple training samples. For any training sample, the training sample includes a model input part and a model output part. The model input part includes the sample data statistical analysis result, and the model output part includes the sample abnormality analysis result and the sample trend analysis result corresponding to the sample data statistical analysis result. Since multiple model input parts (i.e., sample data statistical analysis results) are used as the input of the preset large model, and the model output part corresponding to each model input part (i.e., the sample abnormality analysis result and the sample trend analysis result corresponding to the sample data statistical analysis result) is used as the output of the preset large model, the preset large model is iteratively trained to obtain the data analysis model. Therefore, after a certain data statistical analysis result is input into the data analysis model, the data analysis model can accurately output the abnormality analysis result and the trend analysis result corresponding to the data statistical analysis result. Among them, the abnormal analysis result corresponding to the problem to be analyzed records relevant information of the abnormal data in the data to be analyzed, that is, the date corresponding to the abnormal data, the abnormal situation corresponding to the abnormal data, the cause analysis corresponding to the abnormal data, etc. For example, the problem to be analyzed is the sales situation of sales department A in July 2024, and the data to be analyzed is the daily sales of each employee in sales department A in July 2024. The abnormal analysis result corresponding to the problem to be analyzed records: a. The sales of employee X on July 23, 2024 is significantly higher than other dates, and the cause of the abnormality is xxx; b. The sales of sales department A on July 3, 2024 is significantly lower than other dates, and the cause of the abnormality is xxx, etc.; the trend analysis result corresponding to the problem to be analyzed records the change trend and change law corresponding to the data to be analyzed.
[0046] After the data processing application determines the data statistical analysis results corresponding to the problem to be analyzed, the data statistical analysis results can be input into the data analysis model so that the data analysis model can output the abnormal analysis results and trend analysis results corresponding to the data statistical analysis results, that is, the abnormal analysis results and trend analysis results corresponding to the problem to be analyzed.
[0047] 105. Send the data statistical analysis results, abnormal analysis results and trend analysis results corresponding to the problem to be analyzed to the target terminal device.
[0048] After completing the above steps 101-104, the data processing application can send the data statistical analysis results, anomaly analysis results and trend analysis results corresponding to the problem to be analyzed to the target terminal device, so that the target terminal device can output and display the data statistical analysis results, anomaly analysis results and trend analysis results corresponding to the problem to be analyzed. The user on the target terminal device side can make decisions based on the data statistical analysis results, anomaly analysis results and trend analysis results corresponding to the problem to be analyzed.
[0049] The embodiment of the present application provides a data processing method based on a large model. The embodiment of the present application can, after the data processing application receives the analysis request sent by the target terminal device and carries the problem to be analyzed, first obtain the data to be analyzed according to the problem to be analyzed by the data processing application, and determine the target data statistical analysis logic corresponding to the problem to be analyzed; then determine the data statistical analysis result corresponding to the problem to be analyzed according to the data to be analyzed and the target data statistical analysis logic; then, input the data statistical analysis result corresponding to the problem to be analyzed into the data analysis model, so that the data analysis model outputs the abnormal analysis result and trend analysis result corresponding to the problem to be analyzed; finally, send the data statistical analysis result, abnormal analysis result and trend analysis result corresponding to the problem to be analyzed to the target terminal device, so that after the target terminal device outputs and displays the data statistical analysis result, abnormal analysis result and trend analysis result corresponding to the problem to be analyzed, the user on the target terminal device side can make a decision according to the data statistical analysis result, abnormal analysis result and trend analysis result corresponding to the problem to be analyzed. Since the data processing and analysis of the data generated in the operation process of the target enterprise can be completed without the participation of the data analyst of the target enterprise, the analysis efficiency of the data processing and analysis of the data generated in the operation process of the target enterprise can be effectively improved.
[0050] In certain embodiments of the present application, the data analysis model is: a preset large model is used as the initial model, a fine-tuned large model obtained by training the preset large model with a first training sample set is used as the intermediate model, and the fine-tuned large model is trained with a second training sample set. That is to say, the data analysis model is not obtained directly by training the initial model only as in the traditional method, but is obtained based on the initial model training to obtain a fine-tuned large model, and then the data analysis model is obtained based on the fine-tuned large model training. By analyzing data using this data analysis model, not only can the abnormal analysis results and trend analysis results of the data to be analyzed be obtained, but also the intermediate features for data statistical analysis of the data to be analyzed (such as the SQL statements and data statistical analysis logic corresponding to the problem to be analyzed) can be obtained. In some embodiments, the first training sample includes a first model input part (i.e., a sample problem, a first preset instruction, and professional domain knowledge corresponding to the sample problem) and a first model output part (i.e., a sample SQL statement corresponding to the sample problem and a sample data statistical analysis logic), and the second training sample includes a second model input part (i.e., a sample data statistical analysis result and a second preset instruction) and a second model output part (i.e., a sample abnormality analysis result and a sample trend analysis result corresponding to the sample data statistical analysis result). The data analysis model is obtained by first iteratively training a preset large model based on multiple first training samples to obtain a fine-tuned large model, and then iteratively training the fine-tuned large model based on multiple second training samples. Therefore, the data analysis model obtained by training has two functions: Function 1: After the problem to be analyzed is input into the data analysis model, the data analysis model can accurately output the SQL statement and data statistical analysis logic corresponding to the problem to be analyzed; Function 2: After the data statistical analysis result is input into the data analysis model, the data analysis model can accurately output the abnormality analysis result and trend analysis result corresponding to the data statistical analysis result. Since the structure of the large model is complex and contains ultra-large-scale parameters, the size of the large model is very large. Therefore, the data analysis model (1) with two functions obtained by training a large model will occupy far fewer resources and use fewer parameters than the two large models obtained based on the first training sample and the second training sample respectively. That is: if the first large model is iteratively trained based on multiple first training samples to obtain model A with the above-mentioned function 1, and the second large model is iteratively trained based on multiple second training samples to obtain model B with the above-mentioned function 2, then the size of model A plus model B is much larger than the above-mentioned data analysis model, and the resources occupied by model A plus model B of the electronic device (such as video memory, GPU and other computing resources, power resources, etc.) will also be much larger than the above-mentioned data analysis model.In some methods, a small-scale model with a simpler structure and fewer parameters is used for training. However, the accuracy of the output results of the trained small model is much lower than the accuracy of the output results of the trained large model. Therefore, the first small model is iteratively trained based on multiple first training samples to obtain model C with the above-mentioned function 1, and the second small model is iteratively trained based on multiple second training samples to obtain model D with the above-mentioned function 2. The accuracy of the output results of model C and model D is much lower than the data analysis model mentioned above. Therefore, in summary, training a large model to obtain a data analysis model with two functions takes into account multiple aspects such as model size, resource utilization, and data analysis accuracy, and outputs more accurate results while occupying fewer resources.
[0051] To explain in more detail below, the present application embodiment provides another data processing method based on a large model, as follows Figure 2 As shown, the method includes at least 201-207.
[0052] 201. Perform iterative training on a preset large model based on a first training sample set and a second training sample set to obtain a data analysis model.
[0053] In order to ensure that the data processing application can accurately obtain the data to be analyzed according to the problem to be analyzed carried in the analysis request after receiving the analysis request sent by the target terminal device, and accurately determine the target data statistical analysis logic corresponding to the problem to be analyzed, and after performing data statistical analysis on the data to be analyzed based on the target data statistical analysis logic to obtain the data statistical analysis results corresponding to the problem to be analyzed, accurately determine the abnormal analysis results and trend analysis results corresponding to the problem to be analyzed according to the data statistical analysis results, it is necessary to pre-train the data analysis model. The following will explain in detail how to train the data analysis model.
[0054] (1) Obtain a first training sample set and a second training sample set.
[0055] Among them, the first training sample set includes multiple first training samples. For any first training sample, the first training sample includes a first model input part and a first model output part. The first model input part includes a sample problem, a first preset instruction and professional field knowledge corresponding to the sample problem. The first model output part includes a sample SQL statement and sample data statistical analysis logic corresponding to the sample problem. The first preset instruction is used to indicate that the model needs to output SQL statements and data statistical analysis logic, and the professional field knowledge corresponding to the sample problem is the background knowledge of the professional field involved in the sample problem; the second training sample set includes multiple second training samples. For any second training sample, the second training sample includes a second model input part and a second model output part. The second model input part includes a sample data statistical analysis result and a second preset instruction. The second model output part includes a sample anomaly analysis result and a sample trend analysis result corresponding to the sample data statistical analysis result. The second preset instruction is used to indicate that the model needs to output anomaly analysis results and trend analysis results.
[0056] (2) Iteratively train the preset large model based on the first training sample set to obtain a fine-tuned large model.
[0057] Among them, the preset large model can be any large model (LargeLanguage Models) known in the field of data processing technology. The embodiment of the present application does not specifically limit the specific type of the preset large model.
[0058] The preset large model is iteratively trained based on multiple first training samples, that is, multiple first model input parts are used as the input of the preset large model, and the first model output part corresponding to each first model input part is used as the output of the preset large model, and the preset large model is iteratively trained; wherein, after each round of training, it is determined whether the loss function of the preset large model converges; if the loss function does not converge, the model parameters of the preset large model are optimized and adjusted according to the loss function, and the next round of training is performed based on the preset large model after optimization and adjustment; if the loss function converges, the preset large model obtained after this round of training is determined as the fine-tuning large model.
[0059] That is, after performing multiple rounds of iterative training on the preset large model based on the first training sample set, when the loss function of the preset large model converges, it is determined that the preset large model at this time is the fine-tuning large model.
[0060] It should be noted that since multiple first model input parts (i.e., sample problems, first preset instructions, and professional domain knowledge corresponding to sample problems) are used as inputs of the preset big model, and the first model output part corresponding to each first model input part (i.e., the sample SQL statement corresponding to the sample problem and the sample data statistical analysis logic) is used as the output of the preset big model, the preset big model is iteratively trained to obtain the fine-tuned big model. Therefore, after a problem to be analyzed, the first preset instruction, and the professional domain knowledge corresponding to the problem to be analyzed are input into the fine-tuned big model, the fine-tuned big model can accurately output the SQL statement and data statistical analysis logic corresponding to the problem to be analyzed.
[0061] In some specific cases, even after a large number of iterative trainings, the loss function of the preset large model will not converge. Therefore, in order to avoid the endless iterative training of the preset large model, when it is determined that the loss function of the preset large model obtained after the current round of training has not converged, the following three methods can be used for processing, but are not limited to:
[0062] 1. If the loss function of the preset large model has not converged, determine whether the current cumulative iterative training time of the iterative training of the preset large model based on the first training sample set reaches the preset time threshold.
[0063] If the current cumulative iterative training time reaches the preset time threshold, it means that the iterative training time has met the requirement. At this time, the iterative training can be stopped, and the preset large model obtained after this round of training is determined as the fine-tuned large model.
[0064] If the current cumulative iterative training time does not reach the preset time threshold, the step of optimizing and adjusting the model parameters of the preset large model according to the loss function of the preset large model and performing the next round of training based on the optimized and adjusted preset large model can be entered.
[0065] 2. If the loss function of the preset large model has not converged, determine whether the current cumulative number of iterative training of the preset large model based on the first training sample set has reached a preset number threshold.
[0066] If the current cumulative number of iterative training times reaches the preset threshold, it means that the number of iterative training times has reached the requirement. At this time, the iterative training can be stopped, and the preset large model obtained after this round of training is determined as the fine-tuned large model.
[0067] If the current cumulative number of iterative training times does not reach the preset threshold, the steps of optimizing and adjusting the model parameters of the preset large model according to the loss function of the preset large model and performing the next round of training based on the optimized and adjusted preset large model can be entered.
[0068] 3. If the loss function of the preset large model has not converged, the preset large model is verified based on the first verification sample set to obtain the accuracy, precision and recall rate corresponding to the preset large model.
[0069] If the accuracy, precision, and recall rates of the preset large model meet the preset requirements, the iterative training can be stopped, and the preset large model obtained after this round of training can be determined as the fine-tuned large model.
[0070] If any of the accuracy, precision and recall rates corresponding to the preset large model does not meet the preset requirements, the step of optimizing and adjusting the model parameters of the preset large model according to the loss function of the preset large model and performing the next round of training based on the optimized and adjusted preset large model can be performed.
[0071] Among them, the first verification sample set includes multiple first verification samples. For any first verification sample, the first verification sample includes a fifth model input part and a fifth model output part. The fifth model input part includes sample questions, first preset instructions and professional field knowledge corresponding to the sample questions. The fifth model output part includes sample SQL statements corresponding to the sample questions and sample data statistical analysis logic.
[0072] (3) Iteratively train the fine-tuned large model based on the second training sample set to obtain a data analysis model.
[0073] The fine-tuning large model is iteratively trained based on multiple second training samples, that is, multiple second model input parts are used as inputs of the fine-tuning large model, and the second model output part corresponding to each second model input part is used as the output of the fine-tuning large model, and the fine-tuning large model is iteratively trained to obtain a data analysis model; wherein, after each round of training, it is determined whether the loss function of the fine-tuning large model converges; if the loss function does not converge, the model parameters of the fine-tuning large model are optimized and adjusted according to the loss function, and the next round of training is performed based on the optimized and adjusted fine-tuning large model; if the loss function converges, the fine-tuning large model obtained after this round of training is determined as the data analysis model.
[0074] That is, after performing multiple rounds of iterative training on the fine-tuned large model based on the second training sample set, when the loss function of the fine-tuned large model converges, the fine-tuned large model at this time is determined to be a data analysis model.
[0075] It should be noted that since multiple second model input parts (i.e., sample data statistical analysis results and second preset instructions) are used as inputs of the fine-tuning large model, and the second model output parts corresponding to each second model input part (i.e., the sample anomaly analysis results and sample trend analysis results corresponding to the sample data statistical analysis results) are used as outputs of the fine-tuning large model, the fine-tuning large model is iteratively trained to obtain the data analysis model. Therefore, after a certain data statistical analysis result and the second preset instruction are input into the data analysis model, the data analysis model can accurately output the anomaly analysis result and trend analysis result corresponding to the data statistical analysis result.
[0076] In some specific cases, even after a large number of iterative training, the loss function of the fine-tuned large model will not converge. Therefore, in order to avoid the endless iterative training of the fine-tuned large model, when it is determined that the loss function of the fine-tuned large model obtained after the current round of training has not converged, the following three methods can be used to deal with it, but are not limited to:
[0077] 1. If the loss function of the fine-tuned large model has not converged, determine whether the current cumulative iterative training time of the iterative training of the fine-tuned large model based on the second training sample set reaches a preset time threshold.
[0078] If the current cumulative iterative training time reaches the preset time threshold, it means that the iterative training time has met the requirement. At this time, the iterative training can be stopped, and the fine-tuned large model obtained after this round of training can be determined as the data analysis model.
[0079] If the current cumulative iterative training time does not reach the preset time threshold, the steps of optimizing and adjusting the model parameters of the fine-tuned large model according to the loss function of the fine-tuned large model and performing the next round of training based on the optimized and adjusted fine-tuned large model can be entered.
[0080] 2. If the loss function of the fine-tuned large model has not converged, determine whether the current cumulative number of iterative training of the fine-tuned large model based on the second training sample set has reached a preset number threshold.
[0081] If the current cumulative number of iterative training times reaches the preset threshold, it means that the number of iterative training times has reached the requirement. At this time, the iterative training can be stopped, and the fine-tuned large model obtained after this round of training can be determined as the data analysis model.
[0082] If the current cumulative number of iterative training times does not reach the preset threshold, the steps of optimizing and adjusting the model parameters of the fine-tuned large model according to the loss function of the fine-tuned large model and performing the next round of training based on the optimized and adjusted fine-tuned large model can be entered.
[0083] 3. If the loss function of the fine-tuned large model has not converged, the fine-tuned large model is verified based on the second verification sample set to obtain the accuracy, precision and recall rate corresponding to the fine-tuned large model.
[0084] If the accuracy, precision, and recall rates of the fine-tuned large model meet the preset requirements, the iterative training can be stopped, and the fine-tuned large model obtained after this round of training can be determined as the data analysis model.
[0085] If any of the accuracy, precision and recall rates corresponding to the fine-tuned large model does not meet the preset requirements, the steps of optimizing the model parameters of the fine-tuned large model according to the loss function of the fine-tuned large model and performing the next round of training based on the optimized fine-tuned large model can be performed.
[0086] Among them, the second verification sample set includes multiple second verification samples. For any second verification sample, the second verification sample includes a sixth model input part and a sixth model output part. The sixth model input part includes the statistical analysis results of the sample data and the second preset instructions, and the sixth model output part includes the sample abnormality analysis results and the sample trend analysis results corresponding to the statistical analysis results of the sample data.
[0087] It should be noted that, in an embodiment of the present application, the preset large model is iteratively trained based on the first training sample set and the second training sample set, and the data analysis model obtained by training has two functions: 1. After the problem to be analyzed is input into the data analysis model, the data analysis model can accurately output the SQL statement and data statistical analysis logic corresponding to the problem to be analyzed. Compared with the data processing application generating the SQL statement corresponding to the problem to be analyzed according to the problem to be analyzed, and selecting the data statistical analysis logic corresponding to the problem to be analyzed from the multiple data statistical analysis logics pre-stored according to the problem to be analyzed, the SQL statement and data statistical analysis logic corresponding to the problem to be analyzed can be quickly determined based on the data analysis model; 2. After the data statistical analysis results are input into the data analysis model, the data analysis model can accurately output the abnormal analysis results and trend analysis results corresponding to the data statistical analysis results, so that the data to be analyzed data can be quickly and accurately processed and analyzed based on the data analysis model. Since the structure of the large model is complex and contains ultra-large-scale parameters, the volume of the large model is very large. Therefore, the first large model is iteratively trained based on multiple first training samples to obtain Model A with the above-mentioned function 1, and the second large model is iteratively trained based on multiple second training samples to obtain Model B with the above-mentioned function 2. The volume of Model A plus Model B is much larger than the above-mentioned data analysis model, and the video memory resources occupied by Model A plus Model B of the electronic device will also be much larger than the above-mentioned data analysis model. In addition, although the structure of the small model is simple and contains fewer parameters, the accuracy of the output results of the trained small model is much lower than the accuracy of the output results of the trained large model. Therefore, the first small model is iteratively trained based on multiple first training samples to obtain Model C with the above-mentioned function 1, and the second small model is iteratively trained based on multiple second training samples to obtain Model D with the above-mentioned function 2. The accuracy of the output results of Model C and Model D is much lower than the above-mentioned data analysis model.
[0088] 202. Receive an analysis request sent by a target terminal device.
[0089] Regarding step 202, receiving the analysis request sent by the target terminal device, please refer to Figure 1 The description of the corresponding parts will not be repeated here in the embodiments of the present application.
[0090] 203. Obtain the data to be analyzed according to the problem to be analyzed, and determine the statistical analysis logic of the target data corresponding to the problem to be analyzed.
[0091] After receiving the analysis request sent by the target terminal device, the data processing application can obtain the data to be analyzed in the target database according to the problem to be analyzed contained in the analysis request, and after obtaining the data to be analyzed, determine the target data statistical analysis logic corresponding to the problem to be analyzed. The following will explain in detail how the data processing application obtains the data to be analyzed according to the problem to be analyzed, and how to determine the target data statistical analysis logic corresponding to the problem to be analyzed.
[0092] (1) Determine the professional domain knowledge corresponding to the problem to be analyzed.
[0093] The target knowledge base stores a plurality of analyzed problems, professional domain knowledge corresponding to each analyzed problem, and a plurality of keywords.
[0094] Specifically, in this step, any one of the following two methods can be used to determine the professional field knowledge corresponding to the problem to be analyzed: 1. First, calculate the similarity between each analyzed problem and the problem to be analyzed, and then, determine the professional field knowledge corresponding to the analyzed problem whose similarity to the problem to be analyzed is higher than a preset threshold as the professional field knowledge corresponding to the problem to be analyzed; 2. First, determine multiple keywords corresponding to the problem to be analyzed, and then, match the multiple keywords corresponding to each analyzed problem with the multiple keywords corresponding to the problem to be analyzed respectively, and then, determine the analyzed problem with the highest matching degree with the problem to be analyzed as the successfully matched analyzed problem. For example, the analyzed problem a has 5 keywords, 3 of which are the same as the 3 keywords of the problem to be analyzed, and the analyzed problem b has 5 keywords, 4 of which are the same as the 4 keywords of the problem to be analyzed, then it is considered that the analyzed problem b has a higher matching degree with the problem to be analyzed, and finally, determine the professional field knowledge corresponding to the successfully matched analyzed problem as the professional field knowledge corresponding to the problem to be analyzed.
[0095] (2) Inputting the problem to be analyzed, professional domain knowledge and the first preset instruction into the data analysis model, so that the data analysis model outputs the target SQL statement and target data statistical analysis logic corresponding to the problem to be analyzed.
[0096] (3) Use the target SQL statement to query the target database to obtain the data to be analyzed.
[0097] Specifically, in this step, the target database may be queried using a target SQL statement through the Agent module to obtain the data to be analyzed, but the present invention is not limited thereto.
[0098] 204. Determine the data statistical analysis result corresponding to the problem to be analyzed according to the statistical analysis logic of the data to be analyzed and the target data.
[0099] After obtaining the data to be analyzed and determining the target data statistical analysis logic corresponding to the problem to be analyzed, the data processing application can determine the data statistical analysis results corresponding to the problem to be analyzed based on the data to be analyzed and the target data statistical analysis logic.
[0100] Specifically, in this step, the specific process of the data processing application determining the data statistical analysis results corresponding to the problem to be analyzed based on the data to be analyzed and the target data statistical analysis logic is: first, the data to be analyzed is substituted into the target data statistical analysis logic; second, the target data statistical analysis logic is executed so that the target data statistical analysis logic outputs the data statistical analysis results corresponding to the problem to be analyzed.
[0101] 205. Input the data statistical analysis result and the second preset instruction into the data analysis model so that the data analysis model outputs the abnormality analysis result and the trend analysis result corresponding to the problem to be analyzed.
[0102] After the data processing application determines the data statistical analysis results corresponding to the problem to be analyzed, the data statistical analysis results and the second preset instruction can be input into the data analysis model so that the data analysis model can output the abnormal analysis results and trend analysis results corresponding to the data statistical analysis results, that is, the abnormal analysis results and trend analysis results corresponding to the problem to be analyzed.
[0103] 206. Send the data statistical analysis results, abnormal analysis results and trend analysis results corresponding to the problem to be analyzed to the target terminal device.
[0104] Regarding step 206, sending the data statistical analysis results, abnormal analysis results and trend analysis results corresponding to the problem to be analyzed to the target terminal device, please refer to Figure 1 The description of the corresponding parts will not be repeated here in the embodiments of the present application.
[0105] In some embodiments, after determining the abnormal analysis results and trend analysis results corresponding to the problem to be analyzed, the data processing application can also control the electronic device to output and display the problem to be analyzed, the data to be analyzed, the target SQL statement corresponding to the problem to be analyzed, the target data statistical analysis logic, the data statistical analysis results, the abnormal analysis results and the trend analysis results, so that the data analyst can analyze whether there are problems with the target SQL statement, the target data statistical analysis logic, the abnormal analysis results and the trend analysis results output by the data analysis model; when the data analyst determines that there are problems with the target SQL statement, the target data statistical analysis logic, the abnormal analysis results and the trend analysis results output by the data analysis model, the data analyst will send a correction request carrying the corrected SQL statement corresponding to the problem to be analyzed, the corrected data statistical analysis logic, the corrected data statistical analysis results, the corrected abnormal analysis results and the corrected trend analysis results to the data processing application through the input device of the electronic device, and at this time, the data processing application can receive the correction request. After receiving the correction request, the data processing application can determine the problem to be analyzed, the first preset instruction, the professional field knowledge corresponding to the problem to be analyzed, the corrected SQL statement and the corrected data statistical analysis logic as the first training sample, and determine the second preset instruction, the corrected data statistical analysis result, the corrected anomaly analysis result and the corrected trend analysis result corresponding to the corrected data statistical analysis result as the second training sample, and store the first training sample and the second training sample in the target database, so as to subsequently use multiple first training samples and multiple second training samples to retrain the data analysis model to improve the accuracy of the output results of the data analysis model.
[0106] It should be noted that at this time, the corrected data statistical analysis results, corrected anomaly analysis results and corrected trend analysis results corresponding to the problem to be analyzed need to be sent to the target terminal device to ensure that the analysis results obtained by the user on the target terminal device side are accurate.
[0107] 207. Iteratively retrain the data analysis model based on the multiple first training samples and the multiple second training samples.
[0108] In order to improve the accuracy of the output results of the data analysis model, when the number of first retraining samples and the number of second retraining samples stored in the target database reach a preset number threshold, the data processing application needs to iteratively retrain the data analysis model based on multiple first retraining samples and multiple second retraining samples. The following will describe in detail how the data processing application iteratively retrains the data analysis model based on multiple first retraining samples and multiple second training samples.
[0109] First, a plurality of first training samples and a plurality of second training samples are obtained, wherein the first training sample comprises a third model input part and a third model output part, the third model input part comprises the analyzed problem, the first preset instruction and the professional field knowledge corresponding to the analyzed problem, the third model output part comprises the modified SQL statement and the modified data statistical analysis logic corresponding to the analyzed problem, and the second training sample comprises a fourth model input part and a fourth model output part, the fourth model input part comprises the second preset instruction and the modified data statistical analysis result, and the fourth model output part comprises the modified abnormality analysis result and the modified trend analysis result corresponding to the modified data statistical analysis result; secondly, the data analysis model is iteratively retrained based on the plurality of first training samples, that is, the plurality of third model input parts are taken as the input of the data analysis model, the third model output part corresponding to each third model input part is taken as the output of the data analysis model, and the The data analysis model is iteratively retrained; wherein, after each round of training, it is determined whether the loss function of the data analysis model has converged; if the loss function has not converged, the model parameters of the data analysis model are optimized and adjusted according to the loss function, and the next round of retraining is performed based on the optimized and adjusted data analysis model; if the loss function has converged, the data analysis model is iteratively retrained based on multiple second training samples, that is, multiple fourth model input parts are used as inputs of the data analysis model, and the fourth model output parts corresponding to each fourth model input part are used as outputs of the data analysis model, and the data analysis model is iteratively retrained; wherein, after each round of training, it is determined whether the loss function of the data analysis model has converged; if the loss function has not converged, the model parameters of the data analysis model are optimized and adjusted according to the loss function, and the next round of retraining is performed based on the optimized and adjusted data analysis model; if the loss function has converged, the iterative retraining is stopped.
[0110] Because, in certain specific cases, the loss function of the data analysis model will not converge even after a large number of iterative retraining, therefore, in order to avoid the iterative retraining of the data analysis model from going on endlessly, when it is determined that the loss function of the data analysis model obtained after this round of training has not converged, you can refer to the description of the relevant part in the above step 201 and perform corresponding processing to complete the iterative retraining task of the data analysis model, which will not be elaborated in the embodiments of the present application.
[0111] Furthermore, as a response to the above Figure 1 and Figure 2In order to realize the method shown in the figure, another embodiment of the present application also provides a data processing device based on a large model. This device embodiment corresponds to the aforementioned method embodiment. For ease of reading, this device embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. This device is used to improve the analysis efficiency of data processing and analysis of data generated during the operation of the target enterprise, specifically as follows Figure 3 As shown, the device comprises:
[0112] The first receiving unit 301 is used to receive an analysis request sent by a target terminal device, wherein the analysis request includes a problem to be analyzed;
[0113] The first determining unit 302 is used to obtain the data to be analyzed according to the problem to be analyzed received by the first receiving unit 301, and determine the target data statistical analysis logic corresponding to the problem to be analyzed;
[0114] A second determining unit 303 is used to determine the data statistical analysis result corresponding to the problem to be analyzed according to the data to be analyzed and the target data statistical analysis logic acquired by the second determining unit 303;
[0115] An input unit 304 is used to input the data statistical analysis result determined by the second determination unit 303 into a data analysis model, and the data analysis model outputs an abnormality analysis result and a trend analysis result corresponding to the problem to be analyzed, wherein the data analysis model is obtained after iterative training of a preset large model;
[0116] The sending unit 305 is used to send the data statistical analysis results, abnormality analysis results and trend analysis results corresponding to the problem to be analyzed to the target terminal device.
[0117] Further, such as Figure 4 As shown, the device also includes:
[0118] The training unit 306 is used to obtain a first training sample set and a second training sample set before the first receiving unit 301 receives the analysis request sent by the target terminal device, wherein the first training sample set includes multiple first training samples, the first training sample includes a first model input part and a first model output part, the first model input part includes a sample problem, a first preset instruction and professional field knowledge corresponding to the sample problem, the first model output part includes a sample SQL statement corresponding to the sample problem and a sample data statistical analysis logic, the second training sample set includes multiple second training samples, the second training sample includes a second model input part and a second model output part, the second model input part includes a sample data statistical analysis result and a second preset instruction, and the second model output part includes a sample abnormality analysis result and a sample trend analysis result corresponding to the sample data statistical analysis result;
[0119] Using the plurality of first model input parts as inputs of a preset large model, using the first model output part corresponding to each of the first model input parts as outputs of the preset large model, and iteratively training the preset large model to obtain a fine-tuned large model;
[0120] Use multiple second model input parts as inputs of the fine-tuning large model, use the second model output part corresponding to each second model input part as output of the fine-tuning large model, and iteratively train the fine-tuning large model to obtain the data analysis model.
[0121] Further, such as Figure 4 As shown, the first determining unit 302 includes:
[0122] Determination module 3021, used to determine the professional field knowledge corresponding to the problem to be analyzed;
[0123] An input module 3022, used to input the problem to be analyzed, the professional domain knowledge and the first preset instruction into the data analysis model, and the data analysis model outputs a target SQL statement and a target data statistical analysis logic corresponding to the problem to be analyzed;
[0124] The query module 3023 is used to query the target database using the target SQL statement to obtain the data to be analyzed.
[0125] Further, such as Figure 4As shown, the target knowledge base stores a plurality of analyzed problems, professional domain knowledge corresponding to each of the analyzed problems, and a plurality of keywords; the determination module 3021 is specifically used to calculate the similarity between each of the analyzed problems and the problem to be analyzed, and determine the professional domain knowledge corresponding to the analyzed problems whose similarity to the problem to be analyzed is higher than a preset threshold as the professional domain knowledge corresponding to the problem to be analyzed; or
[0126] Determine multiple keywords corresponding to the problem to be analyzed, and match the multiple keywords corresponding to each of the analyzed problems with the multiple keywords corresponding to the problem to be analyzed respectively, and determine the professional field knowledge corresponding to the successfully matched analyzed problems as the professional field knowledge corresponding to the problem to be analyzed.
[0127] Further, such as Figure 4 As shown, the device also includes:
[0128] The second receiving unit 307 is used to receive a correction request, wherein the correction request includes a correction SQL statement corresponding to the problem to be analyzed, a correction data statistical analysis logic, a correction data statistical analysis result, a correction abnormality analysis result, and a correction trend analysis result;
[0129] A third determining unit 308 is used to determine the problem to be analyzed, the first preset instruction, the professional field knowledge corresponding to the problem to be analyzed, the modified SQL statement and the modified data statistical analysis logic as a first training sample;
[0130] A fourth determining unit 309, configured to determine the second preset instruction, the modified data statistical analysis result, the modified abnormality analysis result and the modified trend analysis result corresponding to the modified data statistical analysis result as a second training sample;
[0131] The storage unit 310 is configured to store the first retraining sample and the second retraining sample in the target database.
[0132] Further, such as Figure 4 As shown, the device also includes:
[0133] The retraining unit 311 is used to obtain a plurality of the first retraining samples and a plurality of the second retraining samples, wherein the first retraining sample comprises a third model input part and a third model output part, the third model input part comprises the analyzed problem, the first preset instruction and the professional field knowledge corresponding to the analyzed problem, the third model output part comprises the modified SQL statement and the modified data statistical analysis logic corresponding to the analyzed problem, the second retraining sample comprises a fourth model input part and a fourth model output part, the fourth model input part comprises the second preset instruction and the modified data statistical analysis result, and the fourth model output part comprises the modified abnormality analysis result and the modified trend analysis result corresponding to the modified data statistical analysis result;
[0134] The data analysis model is iteratively retrained by using multiple third model input parts as inputs of the data analysis model, and the third model output part corresponding to each third model input part is used as output of the data analysis model. After the retraining is completed, the data analysis model is iteratively retrained by using multiple fourth model input parts as inputs of the data analysis model, and the fourth model output part corresponding to each fourth model input part is used as output of the data analysis model.
[0135] Further, such as Figure 4 As shown, the second determination unit 303 is specifically used to substitute the data to be analyzed into the target data statistical analysis logic; and execute the target data statistical analysis logic so that the target data statistical analysis logic outputs the data statistical analysis result corresponding to the problem to be analyzed.
[0136] The embodiment of the present application provides a data processing method and device based on a large model. The embodiment of the present application can, after the data processing application receives the analysis request sent by the target terminal device and carries the problem to be analyzed, the data processing application first obtains the data to be analyzed according to the problem to be analyzed, and determines the target data statistical analysis logic corresponding to the problem to be analyzed; then determines the data statistical analysis result corresponding to the problem to be analyzed according to the data to be analyzed and the target data statistical analysis logic; then, inputs the data statistical analysis result corresponding to the problem to be analyzed into the data analysis model, so that the data analysis model outputs the abnormal analysis result and trend analysis result corresponding to the problem to be analyzed; finally, sends the data statistical analysis result, abnormal analysis result and trend analysis result corresponding to the problem to be analyzed to the target terminal device, so that after the target terminal device outputs and displays the data statistical analysis result, abnormal analysis result and trend analysis result corresponding to the problem to be analyzed, the user on the target terminal device side can make a decision according to the data statistical analysis result, abnormal analysis result and trend analysis result corresponding to the problem to be analyzed. Since the data processing and analysis of the data generated in the operation process of the target enterprise can be completed without the participation of the data analyst of the target enterprise, the analysis efficiency of the data processing and analysis of the data generated in the operation process of the target enterprise can be effectively improved.
[0137] An embodiment of the present application provides a storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above-mentioned large model-based data processing method.
[0138] The storage medium may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one storage chip.
[0139] An embodiment of the present application also provides a data processing device based on a large model, the device comprising a storage medium; and one or more processors, the storage medium is coupled to the processor, the processor is configured to execute program instructions stored in the storage medium; when the program instructions are executed, the above-mentioned data processing method based on the large model is executed.
[0140] The embodiment of the present application provides a device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented:
[0141] Receiving an analysis request sent by a target terminal device, wherein the analysis request includes a problem to be analyzed;
[0142] Acquire the data to be analyzed according to the problem to be analyzed, and determine the statistical analysis logic of the target data corresponding to the problem to be analyzed;
[0143] Determine the data statistical analysis result corresponding to the problem to be analyzed according to the data to be analyzed and the statistical analysis logic of the target data;
[0144] Inputting the data statistical analysis results into a data analysis model, the data analysis model outputs the abnormality analysis results and trend analysis results corresponding to the problem to be analyzed, wherein the data analysis model is obtained after iterative training of a preset large model;
[0145] The data statistical analysis results, abnormality analysis results and trend analysis results corresponding to the problem to be analyzed are sent to the target terminal device.
[0146] Furthermore, before receiving the analysis request sent by the target terminal device, the method further includes:
[0147] Obtain a first training sample set and a second training sample set, wherein the first training sample set includes a plurality of first training samples, the first training sample includes a first model input part and a first model output part, the first model input part includes a sample problem, a first preset instruction, and professional domain knowledge corresponding to the sample problem, the first model output part includes a sample SQL statement corresponding to the sample problem and a sample data statistical analysis logic, the second training sample set includes a plurality of second training samples, the second training sample includes a second model input part and a second model output part, the second model input part includes a sample data statistical analysis result and a second preset instruction, and the second model output part includes a sample abnormality analysis result and a sample trend analysis result corresponding to the sample data statistical analysis result;
[0148] Using the plurality of first model input parts as inputs of a preset large model, using the first model output part corresponding to each of the first model input parts as outputs of the preset large model, and iteratively training the preset large model to obtain a fine-tuned large model;
[0149] Use multiple second model input parts as inputs of the fine-tuning large model, use the second model output part corresponding to each second model input part as output of the fine-tuning large model, and iteratively train the fine-tuning large model to obtain the data analysis model.
[0150] Furthermore, the obtaining of data to be analyzed according to the problem to be analyzed and determining the target data statistical analysis logic corresponding to the problem to be analyzed includes:
[0151] Determine the professional domain knowledge corresponding to the problem to be analyzed;
[0152] The problem to be analyzed, the professional domain knowledge and the first preset instruction are input into the data analysis model, and the data analysis model outputs the target SQL statement and target data statistical analysis logic corresponding to the problem to be analyzed;
[0153] The target SQL statement is used to query the target database to obtain the data to be analyzed.
[0154] Furthermore, the target knowledge base stores a plurality of analyzed problems, professional domain knowledge corresponding to each of the analyzed problems, and a plurality of keywords; the determining of the professional domain knowledge corresponding to the problem to be analyzed includes:
[0155] Calculate the similarity between each of the analyzed problems and the problem to be analyzed, and determine the professional field knowledge corresponding to the analyzed problems whose similarity to the problem to be analyzed is higher than a preset threshold as the professional field knowledge corresponding to the problem to be analyzed; or
[0156] Determine multiple keywords corresponding to the problem to be analyzed, and match the multiple keywords corresponding to each of the analyzed problems with the multiple keywords corresponding to the problem to be analyzed respectively, and determine the professional field knowledge corresponding to the successfully matched analyzed problems as the professional field knowledge corresponding to the problem to be analyzed.
[0157] Furthermore, the method further comprises:
[0158] Receive a correction request, wherein the correction request includes a correction SQL statement corresponding to the problem to be analyzed, a correction data statistical analysis logic, a correction data statistical analysis result, a correction abnormality analysis result, and a correction trend analysis result;
[0159] Determine the problem to be analyzed, the first preset instruction, the professional field knowledge corresponding to the problem to be analyzed, the modified SQL statement and the modified data statistical analysis logic as a first training sample;
[0160] Determine the second preset instruction, the modified data statistical analysis result, the modified abnormality analysis result and the modified trend analysis result corresponding to the modified data statistical analysis result as a second training sample;
[0161] The first retraining sample and the second retraining sample are stored in the target database.
[0162] Furthermore, the method further comprises:
[0163] Acquire a plurality of the first retraining samples and a plurality of the second retraining samples, wherein the first retraining samples include a third model input part and a third model output part, the third model input part includes the analyzed problem, the first preset instruction and the professional field knowledge corresponding to the analyzed problem, the third model output part includes the modified SQL statement and the modified data statistical analysis logic corresponding to the analyzed problem, and the second retraining samples include a fourth model input part and a fourth model output part, the fourth model input part includes the second preset instruction and the modified data statistical analysis result, and the fourth model output part includes the modified abnormality analysis result and the modified trend analysis result corresponding to the modified data statistical analysis result;
[0164] The data analysis model is iteratively retrained by using multiple third model input parts as inputs of the data analysis model, and the third model output part corresponding to each third model input part is used as output of the data analysis model. After the retraining is completed, the data analysis model is iteratively retrained by using multiple fourth model input parts as inputs of the data analysis model, and the fourth model output part corresponding to each fourth model input part is used as output of the data analysis model.
[0165] Further, the determining of the data statistical analysis result corresponding to the problem to be analyzed according to the data to be analyzed and the target data statistical analysis logic includes:
[0166] Substituting the data to be analyzed into the target data statistical analysis logic;
[0167] The target data statistical analysis logic is executed so that the target data statistical analysis logic outputs the data statistical analysis result corresponding to the problem to be analyzed.
[0168] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program code that initializes the following method steps: receiving an analysis request sent by a target terminal device, wherein the analysis request includes a problem to be analyzed; obtaining data to be analyzed based on the problem to be analyzed, and determining a target data statistical analysis logic corresponding to the problem to be analyzed; determining a data statistical analysis result corresponding to the problem to be analyzed based on the data to be analyzed and the target data statistical analysis logic; inputting the data statistical analysis result into a data analysis model, and the data analysis model outputs an abnormal analysis result and a trend analysis result corresponding to the problem to be analyzed, wherein the data analysis model is obtained after iterative training of a preset large model; and sending the data statistical analysis result, the abnormal analysis result and the trend analysis result corresponding to the problem to be analyzed to the target terminal device.
[0169] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0170] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0171] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0173] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0174] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0175] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0176] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0177] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0178] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
Claims
1. A data processing method based on a large model, It is characterized in that The method comprises: Receiving an analysis request sent by a target terminal device, wherein the analysis request includes a problem to be analyzed; Acquire the data to be analyzed according to the problem to be analyzed, and determine the statistical analysis logic of the target data corresponding to the problem to be analyzed; Determine the data statistical analysis result corresponding to the problem to be analyzed according to the data to be analyzed and the statistical analysis logic of the target data; Inputting the data statistical analysis results into a data analysis model, the data analysis model outputs the abnormality analysis results and trend analysis results corresponding to the problem to be analyzed, wherein the data analysis model is obtained after iterative training of a preset large model; The data statistical analysis results, abnormality analysis results and trend analysis results corresponding to the problem to be analyzed are sent to the target terminal device.
2. The method according to claim 1, It is characterized in that Before receiving the analysis request sent by the target terminal device, the method further includes: Obtain a first training sample set and a second training sample set, wherein the first training sample set includes a plurality of first training samples, the first training sample includes a first model input part and a first model output part, the first model input part includes a sample problem, a first preset instruction, and professional domain knowledge corresponding to the sample problem, the first model output part includes a sample SQL statement corresponding to the sample problem and a sample data statistical analysis logic, the second training sample set includes a plurality of second training samples, the second training sample includes a second model input part and a second model output part, the second model input part includes a sample data statistical analysis result and a second preset instruction, and the second model output part includes a sample abnormality analysis result and a sample trend analysis result corresponding to the sample data statistical analysis result; Using the plurality of first model input parts as inputs of a preset large model, using the first model output part corresponding to each of the first model input parts as outputs of the preset large model, and iteratively training the preset large model to obtain a fine-tuned large model; Use multiple second model input parts as inputs of the fine-tuning large model, use the second model output part corresponding to each second model input part as output of the fine-tuning large model, and iteratively train the fine-tuning large model to obtain the data analysis model.
3. The method according to claim 2, It is characterized in that The step of obtaining the data to be analyzed according to the problem to be analyzed and determining the target data statistical analysis logic corresponding to the problem to be analyzed includes: Determine the professional domain knowledge corresponding to the problem to be analyzed; The problem to be analyzed, the professional domain knowledge and the first preset instruction are input into the data analysis model, and the data analysis model outputs the target SQL statement and target data statistical analysis logic corresponding to the problem to be analyzed; The target SQL statement is used to query the target database to obtain the data to be analyzed.
4. The method according to claim 3, It is characterized in that The target knowledge base stores a plurality of analyzed problems, professional domain knowledge corresponding to each of the analyzed problems, and a plurality of keywords; The determining of the professional domain knowledge corresponding to the problem to be analyzed includes: Calculating the similarity between each of the analyzed problems and the problem to be analyzed, and determining the professional field knowledge corresponding to the analyzed problems whose similarity to the problem to be analyzed is higher than a preset threshold as the professional field knowledge corresponding to the problem to be analyzed; or Determine multiple keywords corresponding to the problem to be analyzed, and match the multiple keywords corresponding to each of the analyzed problems with the multiple keywords corresponding to the problem to be analyzed respectively, and determine the professional field knowledge corresponding to the successfully matched analyzed problems as the professional field knowledge corresponding to the problem to be analyzed.
5. The method according to claim 3, It is characterized in that The method further comprises: Receive a correction request, wherein the correction request includes a correction SQL statement corresponding to the problem to be analyzed, a correction data statistical analysis logic, a correction data statistical analysis result, a correction abnormality analysis result, and a correction trend analysis result; Determine the problem to be analyzed, the first preset instruction, the professional field knowledge corresponding to the problem to be analyzed, the modified SQL statement and the modified data statistical analysis logic as a first training sample; Determine the second preset instruction, the modified data statistical analysis result, the modified abnormality analysis result and the modified trend analysis result corresponding to the modified data statistical analysis result as a second training sample; The first retraining sample and the second retraining sample are stored in the target database.
6. The method according to claim 5, It is characterized in that The method further comprises: Acquire a plurality of the first retraining samples and a plurality of the second retraining samples, wherein the first retraining samples include a third model input part and a third model output part, the third model input part includes the analyzed problem, the first preset instruction and the professional field knowledge corresponding to the analyzed problem, the third model output part includes the modified SQL statement and the modified data statistical analysis logic corresponding to the analyzed problem, and the second retraining samples include a fourth model input part and a fourth model output part, the fourth model input part includes the second preset instruction and the modified data statistical analysis result, and the fourth model output part includes the modified abnormality analysis result and the modified trend analysis result corresponding to the modified data statistical analysis result; The data analysis model is iteratively retrained by using multiple third model input parts as inputs of the data analysis model, and the third model output part corresponding to each third model input part is used as output of the data analysis model. After the retraining is completed, the data analysis model is iteratively retrained by using multiple fourth model input parts as inputs of the data analysis model, and the fourth model output part corresponding to each fourth model input part is used as output of the data analysis model.
7. The method according to any one of claims 1 to 6, It is characterized in that The determining of the data statistical analysis result corresponding to the problem to be analyzed according to the data to be analyzed and the target data statistical analysis logic includes: Substituting the data to be analyzed into the target data statistical analysis logic; The target data statistical analysis logic is executed so that the target data statistical analysis logic outputs the data statistical analysis result corresponding to the problem to be analyzed.
8. A data processing device based on a large model, It is characterized in that The device comprises: A first receiving unit, configured to receive an analysis request sent by a target terminal device, wherein the analysis request includes a problem to be analyzed; A first determining unit, configured to obtain data to be analyzed according to the problem to be analyzed, and determine a target data statistical analysis logic corresponding to the problem to be analyzed; A second determining unit, configured to determine a data statistical analysis result corresponding to the problem to be analyzed according to the data to be analyzed and the target data statistical analysis logic; An input unit, used to input the data statistical analysis results into a data analysis model, and the data analysis model outputs an abnormality analysis result and a trend analysis result corresponding to the problem to be analyzed, wherein the data analysis model is obtained after iterative training of a preset large model; The sending unit is used to send the data statistical analysis results, abnormality analysis results and trend analysis results corresponding to the problem to be analyzed to the target terminal device.
9. A storage medium, It is characterized in that The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the large model-based data processing method described in any one of claims 1 to 7.
10. A data processing device based on a large model, It is characterized in that The device includes a storage medium; and one or more processors, the storage medium is coupled to the processor, and the processor is configured to execute program instructions stored in the storage medium; when the program instructions are executed, the large model-based data processing method described in any one of claims 1 to 7 is executed.
