Data analysis method and device based on large model and nonvolatile storage medium

Through the large model, the data sets are automatically divided and the model is selected for analysis, the problem of low automation in the existing technology is solved, efficient and accurate data analysis is achieved, manual intervention is reduced and analysis efficiency is improved.

CN119004026APending Publication Date: 2024-11-22北京大学长沙计算与数字经济研究院 +1

Patent Information

Application Number
CN202411066468.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The low degree of automation of existing data analysis methods leads to low efficiency when processing complex data sets, inability to accurately capture the inherent characteristics of the data, and requires manual intervention to adapt to data changes.

Method used

Through the large model, structured data is divided into different task type data sets based on the target scenario information, and the appropriate model is automatically selected for analysis, including data preprocessing and model training until the analysis results are generated.

Benefits of technology

Fully automatic data analysis is realized, analysis efficiency is improved, manual intervention is reduced, model can be quickly adjusted to adapt to data changes, and model accuracy and efficiency is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004026A_ABST
    Figure CN119004026A_ABST
Patent Text Reader

Abstract

The invention discloses a data analysis method and device based on a large model and a nonvolatile storage medium. The method comprises the following steps: acquiring structured data; the task type to which the structured data belongs is determined according to the target scene information through the large model, the structured data is divided into different data sets according to the task type, each data set corresponds to one task type, different task types correspond to different model sets, each model set comprises a plurality of different models, and the different models correspond to the different data sets; each model comprises capability description information and parameter setting information, the capability description information is used for describing task functions realized by the model, and the parameter setting information is used for setting initial parameters of the model and adjusting parameters of the model; and analyzing a data set corresponding to the task type through a model set corresponding to the large model and the task type, and generating and displaying an analysis result. The technical problem of low analysis efficiency caused by low automation degree of a data analysis method in related technologies is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of machine learning, and more specifically, to a large model-based data analysis method, device, and non-volatile storage medium. Background Art

[0002] Various data analysis methods in the relevant technologies on the market are based on automated machine learning solutions, which usually include the use of statistical techniques, Bayesian optimization or genetic algorithms to select the best machine learning models and parameters. Although these technologies provide a certain degree of automation, they usually require manual setting of search scopes and constraints and rely on repeated trial and error processes to gradually approach the optimal solution. These methods are less efficient when processing complex data sets and often fail to accurately capture the inherent characteristics of all data.

[0003] Traditional automatic machine learning technology still requires a certain degree of data science and machine learning knowledge, especially in terms of model and parameter selection. Human intervention is still required in some links, such as in the data preprocessing or feature selection stage. Faced with the rapid changes or diversity of data, the automated machine learning tools in related technologies have a low degree of automation and may not be able to quickly adjust and optimize the model to adapt to the new data environment.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a data analysis method, device and non-volatile storage medium based on a large model, so as to at least solve the technical problem of low analysis efficiency caused by the low degree of automation of the data analysis method in the related art.

[0006] According to one aspect of an embodiment of the present application, a data analysis method based on a big model is provided, including: obtaining structured data, wherein each structured data includes a target scenario information, and the target scenario information is used to indicate the data analysis result that the user expects to obtain from the structured data; determining the task type to which the structured data belongs according to the target scenario information through the big model, and dividing the structured data into different data sets according to the task type, each data set corresponding to a task type, wherein different task types correspond to different model sets, each model set contains multiple different models, each model contains capability description information and parameter setting information, the capability description information is used to describe the task function implemented by the model, and the parameter setting information is used to set the initial parameters of the model and adjust the parameters of the model; analyzing the data set corresponding to the task type by selecting the model set corresponding to the big model and the task type, generating and displaying the analysis results.

[0007] In some embodiments of the present application, a data set corresponding to a task type is analyzed based on a model set corresponding to the task type selected through a large model and the task type, including: selecting a first model from the model set corresponding to the task type based on the task type through the large model to train the data set corresponding to the task type to obtain a first training result; evaluating the first training result using pre-divided test data through the large model to obtain an evaluation index, and when the evaluation index is greater than a first preset index, the evaluation passes, and the first training result is used as the analysis result; when the evaluation index is less than or equal to the first preset index, the evaluation fails, and in the case of evaluation failure: when the evaluation index is greater than the second preset index, the first model is improved and retrained through the large model until the evaluation passes; when the evaluation index is less than or equal to the second preset index, the model is reselected from the model set corresponding to the task type through the large model for training until the evaluation passes.

[0008] In some embodiments of the present application, task types include regression tasks, classification tasks, and clustering tasks; determining the task type to which structured data belongs includes: automatically identifying user intent information in target scene information through a large model, and determining the task type to which the structured data belongs based on the user intent information.

[0009] In some embodiments of the present application, before analyzing the data set corresponding to the task type by selecting a model set corresponding to the large model and the task type, the method further includes:

[0010] Automatically perform data standardization processing on the data set corresponding to the task type using a preset data processing tool, wherein automatically performing data standardization processing on the data set corresponding to the task type using a preset data processing tool includes at least one of the following: performing target column selection, missing value filling, data normalization, data encoding, feature relationship calculation, principal component analysis (PAC) dimensionality reduction, and data segmentation on the data set corresponding to the task type.

[0011] In some embodiments of the present application, a first model is selected from a model set corresponding to the task type based on the task type through a large model to train a data set corresponding to the task type to obtain a first training result, including: determining a first model from multiple models based on the data set corresponding to the task type and capability description information and parameter setting information of each model in the model set corresponding to the task type; and using the first model to train the data set corresponding to the task type to obtain a first training result.

[0012] In some embodiments of the present application, a first model is determined from multiple models, including: automatically identifying target scenario information of structured data in a data set corresponding to a task type through a large model; matching capability description information of each model in the model set corresponding to the task type with the target scenario information, selecting a model with the highest degree of match between the capability description information and the target scenario information, and setting initial parameters for the model with the highest degree of match through parameter setting information corresponding to the model with the highest degree of match, and using the model with the set initial parameters as the first model.

[0013] In some embodiments of the present application, the target scenario information can be used to indicate at least the following scenarios: optimization of renewable energy distribution in power systems, trend forecasting of carbon emission data from multiple sources, energy usage in construction and manufacturing industries, carbon price trends in carbon trading markets, and energy distribution optimization for various types of factories.

[0014] According to another aspect of an embodiment of the present application, a data analysis device is also provided, including: an acquisition module, used to acquire structured data, wherein each structured data includes a target scene information, and the target scene information is used to indicate the data analysis results that the user expects to obtain from the structured data; a division module, used to determine the task type to which the structured data belongs according to the target scene information through a large model, and divide the structured data into different data sets according to the task type, each data set corresponding to a task type, wherein different task types correspond to different model sets, each model set contains multiple different models, each model contains capability description information and parameter setting information, the capability description information is used to describe the task function implemented by the model, and the parameter setting information is used to set the initial parameters of the model and adjust the parameters of the model; an analysis module, used to analyze the data set corresponding to the task type based on the model set corresponding to the task type selected through the large model and the task type, generate analysis results and display them.

[0015] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above-mentioned large model-based data analysis method.

[0016] According to another aspect of an embodiment of the present application, there is also provided an electronic device comprising: a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the above-mentioned large model-based data analysis method when running.

[0017] According to another aspect of an embodiment of the present application, a computer program product is also provided, including computer instructions, wherein the computer instructions are executed by a processor to perform the above-mentioned large model-based data analysis method.

[0018] In an embodiment of the present application, structured data is obtained, wherein each structured data includes a target scenario information, and the target scenario information is used to indicate the data analysis results that the user expects to obtain from the structured data; the task type to which the structured data belongs is determined according to the target scenario information through a large model, and the structured data is divided into different data sets according to the task type, each data set corresponds to a task type, wherein different task types correspond to different model sets, each model set contains multiple different models, each model contains capability description information and parameter setting information, the capability description information is used to describe the task function implemented by the model, and the parameter setting information is used to set the initial parameters of the model and adjust the parameters of the model; the data set corresponding to the task type is analyzed by selecting a model set corresponding to the large model and the task type, and the analysis results are generated and displayed. By dividing the structured data according to different task types, and automatically selecting models according to different task types for model training and predictive analysis, the purpose of fully automatic data analysis is achieved, thereby solving the technical problem of low analysis efficiency caused by the low degree of automation of data analysis methods in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a schematic diagram of the hardware structure of a data analysis method based on a large model according to an embodiment of the present application;

[0021] Figure 2 is a flow chart of a data analysis method based on a large model according to an embodiment of the present application;

[0022] Figure 3 is a flowchart of an optional large model-based data analysis method according to an embodiment of the present application;

[0023] Figure 4 It is a structural schematic diagram of a data analysis device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0027] Agent: In the field of machine learning, an agent is an entity that can receive information from the environment, make decisions through a learning process, and influence the environment. Agents can be applied to a variety of tasks, including but not limited to automation, prediction, optimization, classification, and recommendation. The concept of agents is widely used in artificial intelligence, control theory, machine learning, and other related fields. Agents have the following key characteristics: agents can independently control their own behavior and internal state without the need for continuous guidance from external systems or users, agents can communicate and collaborate with other agents or humans to achieve common goals, agents can perceive their environment and respond quickly to perceived changes, agents have the ability to achieve specific goals or tasks, which are usually defined by optimizing some performance metric, and agents can learn through experience, that is, improve their decision-making and behavior strategies through interaction with the environment.

[0028] In the related art, with the development of large pre-trained models, intelligent analysis solutions based on these large models have been applied in various fields, especially in processing and analyzing tabular data. These solutions mainly rely on the excellent natural language understanding ability of large models to parse the user's query intention, and generate execution code based on this, and finally obtain the required analysis results, but the prior art provides a certain degree of automation, but they usually require manual setting of search ranges and constraints, and rely on repeated trial and error processes to gradually approach the optimal solution. These methods are less efficient when processing complex data sets, and often cannot accurately capture the inherent characteristics of all data. Therefore, there is a problem of low analysis efficiency caused by the low degree of automation of data analysis methods in related technologies. In order to solve this problem, a relevant solution is provided in the embodiments of the present application, which is described in detail below.

[0029] According to an embodiment of the present application, a method embodiment of a data analysis method based on a large model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure diagram of a computer terminal for implementing a data analysis method based on a large model. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0031] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0032] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data analysis method based on the large model in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned data analysis method based on the large model. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0033] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0034] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0035] In the above operating environment, the embodiment of the present application provides a flowchart of a data analysis method based on a large model, such as Figure 2 As shown, the method comprises the following steps:

[0036] Step S202, obtaining structured data;

[0037] In the technical solution provided in step S202, each piece of structured data includes a piece of target scene information, and the target scene information is used to indicate the data analysis result that the user expects to obtain from the structured data.

[0038] Step S204, determining the task type to which the structured data belongs according to the target scene information through the large model, and dividing the structured data into different data sets according to the task type, each data set corresponding to a task type;

[0039] In the technical solution provided in step S204, the task types include regression tasks, classification tasks, and clustering tasks. There are many ways to determine the task type to which the structured data belongs. For example, the task type to which the structured data belongs can be determined in the following ways: automatically identifying user intention information in the target scene information through a large model (for example, an Agent intelligent body built based on the large model), and determining the task type to which the structured data belongs based on the user intention information; different task types correspond to different model sets, each model set contains multiple different models, each model contains capability description information and parameter setting information, the capability description information is used to describe the task function implemented by the model, and the parameter setting information is used to set the initial parameters of the model and adjust the parameters of the model.

[0040] The following is a specific embodiment:

[0041] The classification tasks correspond to different model sets including: Logistic Regression, Support Vector Machine, Naive Bayes, Random Forest, AdaptiveBoosting, Gradient Boosting Trees;

[0042] Different model sets corresponding to regression tasks include: Linear Regression, Ridge Regression, Lasso Regression, Random Forest Regressor, Gradient Boosting Regressor, and Elastic Net Regression;

[0043] The clustering tasks correspond to different model sets including: K-Means Clustering, Density-Based Spatial Clustering of Applications with Noise, Gaussian Mixture Model, Hierarchical Clustering, and Spectral Clustering. Each model has a capability description and parameter setting description. The large model will select a model suitable for the current training data based on these descriptions and instructions.

[0044] The technical solution provided in step S204 also includes a large model (for example, an Agent intelligent body built based on the large model) using a preset data processing tool to automatically perform data standardization processing on the data set corresponding to the task type. There are many ways to automatically perform data standardization processing on the data set corresponding to the task type using the preset data processing tool, such as: target column selection, missing value filling (for example, using the mean, median or mode to fill numerical data; using the mode or the most common label to fill categorical data, etc.), data normalization, data encoding, feature relationship calculation, principal component analysis (PCA for short), also known as PCA dimensionality reduction, and data segmentation for the data set corresponding to the task type.

[0045] It should be noted that there are many ways to select target columns, fill missing values, normalize data, encode data, calculate feature relationships, perform PCA dimensionality reduction, and split data for the data set corresponding to the task type. For example, use the target column identification tool to select the target column: first identify all columns of the data set, then identify the task type of the data set corresponding to the target scene information based on the target scene information, and identify the target column based on the task type; use the missing value filling tool to fill missing values: use the mean, median or mode to fill missing values; use the data normalization tool to normalize data: use the minimum-maximum (MinMaxSc aler normalization (scaling the data to between 0 and 1 by subtracting the minimum value of the feature from each value and then dividing by the difference between the maximum and minimum values ​​of the feature) and z-score normalization (converting the data to a distribution with a mean of 0 and a standard deviation of 1, scaling the features of the data so that their mean is 0 and the standard deviation is 1, which is achieved by subtracting the mean of the feature from each value and then dividing by the standard deviation of the feature); use data encoding tools for data encoding: convert categorical variables (i.e. the above dataset) into numerical form, for example, use one-hot encoding (One-Hot Encoding), convert each categorical variable into multiple binary variables or label encoding (Label Encoding) converts categorical variables into integers; use the feature relationship calculation tool to calculate the feature relationship: analyze the relationship between features, which is achieved by calculating the correlation coefficient matrix, to identify the linear relationship between features, so as to find the redundancy or importance between features; use the PCA dimensionality reduction tool to reduce the dimension of the data using principal component analysis PCA. PCA transforms the data into a new coordinate system through linear transformation, so that the variance on the new coordinate axis is maximized, which helps to remove noise in the data and reduce the complexity of model training; use the data splitting tool to split the data: divide the data set into a training set and a test set. Usually, the training set is used for model training, and the test set is used to evaluate the performance of the model. Choose an appropriate ratio for splitting (such as 80% training set and 20% test set).

[0046] Step S206, analyzing the data set corresponding to the task type by selecting a model set corresponding to the large model and the task type, generating and displaying the analysis results;

[0047] In the technical solution provided in step S206, there are many ways to analyze the data set corresponding to the task type by selecting the model set corresponding to the large model and the task type. For example, the following methods can be used: the large model (for example, the Agent intelligent body constructed by the large model) selects a first model from the model set corresponding to the task type based on the task type to train the data set corresponding to the task type to obtain a first training result; the large model (for example, the Agent intelligent body constructed by the large model) uses the pre-divided test data to evaluate the first training result (accuracy evaluation of the result) to obtain an evaluation index. When the evaluation index is greater than a first preset index (for example, 90%), the evaluation passes, and the first training result is used as the analysis result; when the evaluation index is less than or equal to the first preset index, the evaluation fails. In the case of evaluation failure: when the evaluation index is greater than a second preset index (for example, 80%), the first model is improved and retrained until the evaluation passes; when the evaluation index is less than or equal to the second preset index, a model is reselected from the model set corresponding to the task type for training until the evaluation passes.

[0048] It should be noted that there are multiple ways to implement the above-mentioned method of selecting a first model from a set of models corresponding to the task type based on the task type to train the data set corresponding to the task type and obtain the first training result. For example: the big model (for example, the Agent intelligent body constructed by the big model) selects a first model from a set of models corresponding to the task type based on the task type to train the data set corresponding to the task type and obtain the first training result: based on the data set corresponding to the task type and the capability description information and parameter setting information of each model in the set of models corresponding to the task type, the first model is determined from multiple models; the Agent uses the first model to train the data set corresponding to the task type and obtains the first training result; there are multiple ways to determine the first model from multiple models, for example, The first model is determined from multiple models in the following manner: the target scenario information of the structured data in the data set corresponding to the task type is automatically identified through a large model; the capability description information of each model in the model set corresponding to the task type is matched with the target scenario information (for example, similarity is calculated), the model with the highest matching degree between the capability description information and the target scenario information is selected, and initial parameters are set for the model with the highest matching degree through the parameter setting information corresponding to the model with the highest matching degree, and the model with the set initial parameters is used as the first model; the target scenario information can be used to indicate at least the following scenarios: optimization of renewable energy distribution in power systems, trend prediction of carbon emission data from multiple sources, energy usage in construction and manufacturing industries, carbon price trends in carbon trading markets, and energy distribution optimization of various types of factories.

[0049] The following are several specific application scenarios:

[0050] When the Agent (intelligent body) built based on the large model recognizes that the target scenario information indicated in the structured data input by the user is the optimization of renewable energy allocation in the power system, the Agent will obtain historical power grid data, including power consumption, power grid load, power generation data, etc., as well as historical output data of renewable energy, such as wind power, solar energy, etc.; collect weather forecast data, including wind speed, cloud cover, temperature, etc.; and perform data cleaning on the above data, including at least one of the following: target column selection, missing value filling (for example, using the mean, median or mode to fill numerical data; using the mode or the most common label to fill categorical data, etc.), data normalization, data encoding, feature relationship calculation, PCA dimensionality reduction, data segmentation, and divide the cleaned data into a data set of structured data corresponding to the user intent information in the target scenario information and test data (for example, the division ratio is 80%:20%, 80% is used for training the model, and 20% is used as test data for evaluating the model training results);

[0051] Based on the user intention information, the Agent divides the data (i.e., the structured data) into a data set corresponding to the regression task, and uses a model set corresponding to the regression task (for example, different model sets corresponding to the regression task include: linear regression, ridge regression, lasso regression, random forest regressor, gradient boosting regressor, elastic net regression) to select a first model to train the data set corresponding to the task type, and obtain a first training result, specifically: obtaining linear regression, ridge regression, lasso regression, random forest regressor, gradient boosting regressor, elastic net regression. Regression), calculate the similarity between the acquired capability description information and the above user intent information (i.e., the above matching degree), select the model with the highest similarity as the first model, and set the initial parameters of the model for the first model through the parameter setting information to train the data, and obtain the first training result;

[0052] The agent evaluates the first training result: the first training result is evaluated using the above test data to obtain an evaluation index (for example, the evaluation index can be a percentage). The evaluation index is greater than the first preset evaluation index (for example, 90%). When the evaluation is passed, the first training result (i.e., the renewable energy distribution result in the power system) is used as the analysis result, and the analysis result and the intermediate process of obtaining the analysis result are organized into a document (or a chart form) for analysis result display. When the evaluation fails, in the case of evaluation failure: when the evaluation index is greater than the second preset index (for example, 80%), the first model is improved (for example, the parameters of the model are reset through parameter setting information) and then retrained until the evaluation is passed; when the evaluation index is less than or equal to the second preset index, a model is reselected from the model set corresponding to the task type (for example, the model is selected in the order of sorting from large to small according to the above similarity, and the model selected each time is not repeated), and the training is selected until the evaluation is passed, and the training result when the evaluation is passed is used as the analysis result, and the analysis result and the intermediate process of obtaining the analysis result are organized into a document (or a chart form) for analysis result display. Through the above process, energy waste can be reduced, production costs can be reduced, and carbon emissions can be reduced.

[0053] The method of the present application can also be used to analyze different data of various scenarios to obtain analysis results. For example, it can also be applied to the target scenario information of renewable energy distribution optimization in the power system. The agent can automatically select the best machine learning model to predict the output of renewable energy and optimize its distribution in the power grid based on weather forecasts, historical power grid data and renewable energy output, thereby improving the utilization rate of renewable energy, reducing dependence on fossil fuels, and supporting carbon neutrality goals.

[0054] In the target scenario information of trend prediction of carbon emission data from various sources, such as factory emissions, vehicle emissions, etc., it automatically selects appropriate machine learning models to monitor and predict future carbon emission trends in real time, providing data support for policymakers to formulate more effective emission reduction measures and monitor the implementation effect of carbon emission policies;

[0055] In the target scenario information of energy usage in the construction and manufacturing industries, the method of the present application can automatically select appropriate machine learning tools to analyze the energy usage of enterprises or buildings, determine whether it complies with national or regional energy efficiency standards, and help enterprises adjust operations in a timely manner to ensure compliance with regulatory requirements while promoting energy efficiency improvements.

[0056] In the target scenario information of carbon price trends in the carbon trading market, the method of this application can automatically select and train models based on market dynamics, historical trading data and policy changes, predict carbon price trends, help companies formulate optimal strategies for purchasing or selling carbon quotas, optimize carbon trading decisions, reduce costs, and increase the competitiveness of companies in the carbon market.

[0057] The method of the present application fully automates the process of machine learning through an Agent (intelligent agent) built based on a large model. For the structured data and problems uploaded by users, automatic machine learning is realized for three types of tasks such as classification, regression or clustering. In the process of machine learning, the Agent built by the large model performs the full automatic implementation of the processes such as target column selection, data processing, model selection, model training, and model evaluation according to the intermediate results. This automated process is based on the input data and the prompt words related to the data (i.e., the above-mentioned target scene information). The Agent analyzes the data characteristics and automatically determines which data processing method, machine learning model and its parameter settings to use. In addition, the Agent can adjust its decision-making process in real time and optimize the model training parameters to adapt to data changes and achieve the best training results, thereby obtaining data analysis results and reducing the time from data to insight. The automated process reduces the need for manual intervention, allowing machine learning projects to be started and executed faster, thereby accelerating the decision-making process and the development of new products; the automated decision-making process includes data processing, model selection and parameter adjustment, etc., which are all based on the data characteristics and real-time feedback analyzed by the system. This precise automatic adjustment helps reduce the possibility of human error, while improving the performance and accuracy of the model by optimizing algorithm parameters; it lowers the threshold for machine learning, allowing users without professional data science backgrounds to take advantage of advanced machine learning techniques.

[0058] The present application embodiment also provides a flowchart of an optional data analysis method based on a large model, such as Figure 3 As shown, the method comprises the following steps:

[0059] First, the agent obtains the tabular data (i.e., the structured data) proposed and uploaded by the user, calls the tool set, and executes the pipeline pipeline to determine the task type (i.e., the corresponding subtask: classification, regression, clustering) to which the tabular data proposed and uploaded by the user belongs based on the target scene information in the data, and divides the tabular data (i.e., the structured data) proposed and uploaded by the user into different data sets according to the task type. Each data set corresponds to a task type (implementation method and Figure 2The classification results are fed back to the agent in the same way as described in the previous section, and will not be repeated here. The pipeline is a process that connects multiple processing steps in series, so that the data is processed from the original state and finally used for model training or prediction. It can improve the maintainability, reusability and efficiency of the code. The steps of data preprocessing, feature extraction, model training, model evaluation, etc. are encapsulated into independent modules. The pipeline can automatically execute each step;

[0060] The agent calls the tool set and executes the pipeline pipeline to perform data processing such as target column selection, default value filling, data normalization, data encoding, feature relationship calculation, principal component analysis dimensionality reduction, data segmentation, etc. on the data set of the divided task type (implementation method and Figure 2 The method described in is the same as that described in , which will not be repeated here), and is fed back to the intelligent agent;

[0061] The intelligent agent calls the tool set to process the data set after data processing and executes the pipeline pipeline to perform model selection, model training, and model evaluation: the first model is selected from the model set corresponding to the task type based on the task type through the large model to train the data set corresponding to the task type, and a first training result is obtained: based on the data set corresponding to the task type and the capability description information and parameter setting information of each model in the model set corresponding to the task type, a first model is determined from multiple models; the data set corresponding to the task type is trained using the first model to obtain a first training result; the first training result is evaluated by the large model using pre-divided test data to obtain an evaluation index, and when the evaluation index is greater than the first preset index, the evaluation passes, and the first training result is used as the analysis result; when the evaluation index is less than or equal to the first preset index, the evaluation fails. In the case of evaluation failure: when the evaluation index is greater than the second preset index, the first model is improved and retrained through the large model until the evaluation passes; when the evaluation index is less than or equal to the second preset index, the model is reselected from the model set corresponding to the task type through the large model for training until the evaluation passes, and the final output (i.e., the above-mentioned analysis result) is obtained.

[0062] The present application also provides an optional data analysis device, a schematic diagram of the structure of the device, as shown in FIG. Figure 4 As shown, it includes an acquisition module 40 for acquiring structured data, wherein each structured data includes a target scene information, and the target scene information is used to indicate the data analysis result that the user expects to obtain from the structured data.

[0063] The division module 42 is used to determine the task type to which the structured data belongs based on the target scenario information through the large model, and divide the structured data into different data sets according to the task type. Each data set corresponds to a task type, wherein different task types correspond to different model sets, each model set contains multiple different models, and each model contains capability description information and parameter setting information. The capability description information is used to describe the task function implemented by the model, and the parameter setting information is used to set the initial parameters of the model and adjust the parameters of the model.

[0064] The partitioning module 42 is also used to perform target column selection, missing value filling, data normalization, data encoding, feature relationship calculation, PCA dimensionality reduction, and data segmentation on the data set corresponding to the task type.

[0065] The analysis module 44 is used to analyze the data set corresponding to the task type by selecting a model set corresponding to the large model and the task type, generate and display the analysis results.

[0066] The analysis module 44 is also used to select a first model from a model set corresponding to the task type based on the task type through the large model to train the data set corresponding to the task type to obtain a first training result; evaluate the first training result using pre-divided test data through the large model to obtain an evaluation index, and when the evaluation index is greater than the first preset index, the evaluation passes, and the first training result is used as the analysis result; when the evaluation index is less than or equal to the first preset index, the evaluation fails, and in the case of evaluation failure: when the evaluation index is greater than the second preset index, the first model is improved and retrained through the large model until the evaluation passes; when the evaluation index is less than or equal to the second preset index, the model is reselected from the model set corresponding to the task type through the large model for training until the evaluation passes.

[0067] The analysis module 44 is also used to select a first model from a model set corresponding to the task type based on the task type through a large model to train the data set corresponding to the task type to obtain a first training result: determine the first model from multiple models based on the data set corresponding to the task type and the capability description information and parameter setting information of each model in the model set corresponding to the task type; use the first model to train the data set corresponding to the task type to obtain a first training result; determine the first model from multiple models: automatically identify the target scenario information of the structured data in the data set corresponding to the task type through the large model; match the capability description information of each model in the model set corresponding to the task type with the target scenario information, select the model with the highest matching degree between the capability description information and the target scenario information, and set initial parameters for the model with the highest matching degree through the parameter setting information corresponding to the model with the highest matching degree, and use the model with the set initial parameters as the first model; the target scenario information can be used to indicate at least the following scenarios: optimization of renewable energy distribution in power systems, trend prediction of carbon emission data from multiple sources, energy usage in construction and manufacturing industries, carbon price trends in the carbon trading market, and energy distribution optimization of various factories.

[0068] It should be noted that the various modules in the above-mentioned big model-based data analysis method can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0069] The embodiment of the present application also provides a non-volatile storage medium, the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above data analysis method based on the big model. For example, structured data is obtained, wherein each structured data contains a target scene information, and the target scene information is used to indicate the data analysis results that the user expects to obtain from the structured data; the task type to which the structured data belongs is determined according to the target scene information through the big model, and the structured data is divided into different data sets according to the task type, and each data set corresponds to a task type, wherein different task types correspond to different model sets, and each model set contains multiple different models, and each model contains capability description information and parameter setting information, the capability description information is used to describe the task function implemented by the model, and the parameter setting information is used to set the initial parameters of the model and adjust the parameters of the model; based on the model set corresponding to the task type selected by the big model and the task type, the data set corresponding to the task type is analyzed, and the analysis results are generated and displayed.

[0070] The embodiment of the present application also provides an electronic device, the electronic device includes a processor, the processor is used to run a program, wherein the above data analysis method based on a large model is executed when the program is running. For example, structured data is obtained, wherein each structured data contains a target scene information, and the target scene information is used to indicate the data analysis results that the user expects to obtain from the structured data; the task type to which the structured data belongs is determined by the large model according to the target scene information, and the structured data is divided into different data sets according to the task type, each data set corresponds to a task type, wherein different task types correspond to different model sets, each model set contains multiple different models, each model contains capability description information and parameter setting information, the capability description information is used to describe the task function implemented by the model, and the parameter setting information is used to set the initial parameters of the model and adjust the parameters of the model; the data set corresponding to the task type is analyzed by selecting the corresponding model set through the large model and the task type, and the analysis results are generated and displayed.

[0071] According to another aspect of the embodiment of the present application, a computer program product is also provided, including a computer program, which implements the above data analysis method based on a large model when executed by a processor. For example, structured data is obtained, wherein each structured data contains a target scene information, and the target scene information is used to indicate the data analysis results that the user expects to obtain from the structured data; the task type to which the structured data belongs is determined by the large model according to the target scene information, and the structured data is divided into different data sets according to the task type, and each data set corresponds to a task type, wherein different task types correspond to different model sets, and each model set contains multiple different models, and each model contains capability description information and parameter setting information, and the capability description information is used to describe the task function implemented by the model, and the parameter setting information is used to set the initial parameters of the model and adjust the parameters of the model; the data set corresponding to the task type is analyzed by selecting the corresponding model set through the large model and the task type, and the analysis results are generated and displayed.

[0072] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0073] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0074] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0075] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0076] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0077] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data analysis method based on a large model, characterized in that: include: Acquire structured data, wherein each piece of the structured data includes a piece of target scene information, and the target scene information is used to indicate a data analysis result that a user expects to obtain from the structured data; Determine the task type to which the structured data belongs according to the target scenario information through a large model, and divide the structured data into different data sets according to the task type, each of the data sets corresponds to one task type, wherein the different task types correspond to different model sets, each of the model sets contains a plurality of different models, each model contains at least capability description information and parameter setting information, the capability description information is used to describe the task function implemented by the model, and the parameter setting information is used to set the initial parameters of the model and adjust the parameters of the model; The data set corresponding to the task type is analyzed by selecting a model set corresponding to the large model and the task type, and the analysis results are generated and displayed.

2. The method according to claim 1, characterized in that Analyzing the data set corresponding to the task type by selecting a model set corresponding to the large model and the task type includes: Selecting a first model from the model set corresponding to the task type based on the task type by the large model to train the data set corresponding to the task type, and obtaining a first training result; The first training result is evaluated by the large model using the pre-divided test data to obtain an evaluation index. When the evaluation index is greater than the first preset index, the evaluation is passed, and the first training result is used as the analysis result. When the evaluation index is less than or equal to the first preset index, the evaluation fails. In the case of evaluation failure: When the evaluation index is greater than the second preset index, the first model is improved through the large model and then retrained until the evaluation passes; when the evaluation index is less than or equal to the second preset index, the model is reselected from the model set corresponding to the task type through the large model for training until the evaluation passes.

3. The method according to claim 1, characterized in that: The task types include regression tasks, classification tasks, and clustering tasks; Determining the task type to which the structured data belongs includes: automatically identifying user intent information in the target scene information through the large model, and determining the task type to which the structured data belongs according to the user intent information.

4. The method according to claim 1, characterized in that: Before analyzing the data set corresponding to the task type by selecting a model set corresponding to the large model and the task type, the method further includes: Automatically perform data standardization processing on the data set corresponding to the task type using a preset data processing tool, wherein automatically performing data standardization processing on the data set corresponding to the task type using a preset data processing tool includes at least one of the following: performing target column selection, missing value filling, data normalization, data encoding, feature relationship calculation, principal component analysis method dimensionality reduction, and data segmentation on the data set corresponding to the task type.

5. The method according to claim 2, characterized in that: The first model is selected from the model set corresponding to the task type by the large model based on the task type to train the data set corresponding to the task type, so as to obtain a first training result, including: Determine the first model from a plurality of models based on the data set corresponding to the task type and the capability description information and the parameter setting information of each model in the model set corresponding to the task type; The first model is used to train the data set corresponding to the task type through the large model to obtain the first training result.

6. The method according to claim 5, characterized in that Determining the first model from a plurality of the models comprises: Automatically identifying the target scene information of the structured data in the data set corresponding to the task type through the large model; Match the capability description information of each model in the model set corresponding to the task type with the target scene information, select the model with the highest matching degree between the capability description information and the target scene information, and set initial parameters for the model with the highest matching degree through the parameter setting information corresponding to the model with the highest matching degree, and use the model with the set initial parameters as the first model.

7. The method according to claim 1, characterized in that: The target scenario information can be used to indicate at least the following scenarios: optimization of renewable energy distribution in power systems, trend forecasting of carbon emission data from multiple sources, energy usage in construction and manufacturing industries, carbon price trends in carbon trading markets, and energy distribution optimization for various types of factories.

8. A data analysis device, characterized in that: include: An acquisition module, used for acquiring structured data, wherein each piece of the structured data includes a piece of target scene information, and the target scene information is used for indicating a data analysis result that a user expects to obtain from the structured data; A division module, used to determine the task type to which the structured data belongs according to the target scene information through a large model, and divide the structured data into different data sets according to the task type, each of which corresponds to one task type, wherein the different task types correspond to different model sets, each of which contains a plurality of different models, each of which contains capability description information and parameter setting information, wherein the capability description information is used to describe the task function implemented by the model, and the parameter setting information is used to set the initial parameters of the model and adjust the parameters of the model; The analysis module is used to analyze the data set corresponding to the task type by selecting a model set corresponding to the large model and the task type, generate and display the analysis results.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the large model-based data analysis method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program, when running, executes the large model-based data analysis method described in any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the large model-based data analysis method described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Data analysis method and device, model training method and device, equipment and storage medium

    CN110163380A

  • Big data analysis model algorithm type selection method and device, electronic equipment and medium

    CN110990461A

  • Data analysis method and device, electronic equipment and storage medium

    CN116777113A

  • Multi-task processing method and system based on large model and computing equipment

    CN116976467A

  • Multi-agent game decision-making system and method based on cloud native

    CN117076993A

Cited By

  • Artificial intelligence auxiliary judgment method, system and device, storage medium and product

    CN119477610A

  • Artificial intelligence-assisted trial methods, systems, equipment, storage media and products

    CN119477610B

  • Method and device for providing multi-subject personalized power data service

    CN119918856A

  • Intelligent agent evolution method and device, processing system, equipment and medium

    CN120874888A

  • Intelligent code analysis system and method based on graph database

    CN120951325A