Method for completing big data analysis by using big model

Abstract

The invention relates to the technical field of data analysis, and discloses a method for completing big data analysis by using a big model, which comprises the following steps of: performing data preprocessing on original analysis data to obtain a smooth data chain and an irregular data chain; judging whether a transmission association mark is generated for the smooth data chain and the irregular data chain or not; when it is judged that the transmission association mark is generated, the smooth data chain and the irregular data chain are connected, and a to-be-transmitted data chain is obtained; dividing a to-be-transmitted data chain into a plurality of transmission data blocks, determining a data transmission deviation value of each transmission data block and a data temporary storage area of the data analysis model, setting a transmission speed based on the data transmission deviation values, and transmitting the to-be-transmitted data chain to the large data analysis model; data analysis and analysis result display are carried out on the to-be-transmitted data chain based on the data analysis large model, correlation analysis can be carried out on mass data, the data with correlation can be rapidly input into the data analysis large model, and the data analysis efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a method capable of completing big data analysis using a large model. Background Art

[0002] With the rapid development of information technology, the amount of data generated, stored and processed has shown explosive growth. Big data technology has become essential because of its ability to process massive data sets and provide valuable insights. Enterprises and research institutions use big data analysis to discover trends, patterns and associations, which has significant value in decision support, market analysis, customer relationship management, etc.

[0003] Traditional big data analysis methods rely on statistics and machine learning algorithms to process and analyze data. However, as the amount of data continues to increase, these methods have encountered challenges in computing resources, processing speed, and model complexity. Especially when faced with large-scale data sets with mixed data, traditional methods often have difficulty in efficiently determining whether all data can be transmitted collaboratively, reducing data analysis efficiency. Summary of the invention

[0004] The embodiment of the present invention provides a method for completing big data analysis using a large model, so as to solve the technical problems in the prior art that massive data cannot be efficiently analyzed and data cannot be quickly input into the model, resulting in low data analysis efficiency.

[0005] In order to achieve the above object, the present invention provides a method for using a large model to complete big data analysis, comprising:

[0006] Acquire original analysis data, and perform data preprocessing on the original analysis data to obtain smooth data chains and irregular data chains;

[0007] Analyzing the smooth data link and the irregular data link, and determining whether to generate a transmission association mark for the smooth data link and the irregular data link based on the analysis result;

[0008] When it is determined that a transmission association mark is generated for the smooth data link and the irregular data link, the smooth data link and the irregular data link are connected to obtain a data link to be transmitted;

[0009] Divide the data chain to be transmitted into a plurality of transmission data blocks, determine a data transmission deviation value between each transmission data block and a data analysis model data temporary storage area, set a transmission speed of the data chain to be transmitted based on all data transmission deviation values, and transmit the data chain to be transmitted to the data analysis model based on the transmission speed;

[0010] Based on the data analysis big model, data analysis is performed on the data link to be transmitted, and the data analysis results are displayed, wherein the data analysis includes data recursion, data growth rate, and data transition rate.

[0011] Furthermore, when the original analysis data is obtained and data preprocessing is performed on the original analysis data to obtain a smooth data chain and an irregular data chain, it includes:

[0012] Output the importance value of each raw analysis data based on the pre-trained data importance evaluation model;

[0013] All importance values ​​are deployed, and curve fitting is performed on all importance values ​​according to a preset curve fitting method to obtain a smooth importance value curve;

[0014] Determine whether there are unfitted importance values, and if so, calculate the distances between all unfitted importance values ​​and the importance value curve respectively;

[0015] Compare all distances with preset distances, and associate all distances less than or equal to the preset distance with the importance value curve;

[0016] Establishing non-association marks between all distances greater than the preset distance and the importance value curve;

[0017] Generate the smoothed data chain according to the original analysis data corresponding to the importance value curve and the original analysis data corresponding to the association identifier;

[0018] The irregular data chain is generated according to the original analysis data corresponding to the non-association identifier.

[0019] Further, when analyzing the smooth data link and the irregular data link, and judging whether to generate a transmission association mark for the smooth data link and the irregular data link based on the analysis result, it includes:

[0020] The smooth data chain and the irregular data chain are sorted respectively based on the importance value relationship, and the tail analysis data of the smooth data chain and the head analysis data of the irregular data chain are extracted, and the importance value of the tail analysis data is the smallest, and the importance value of the head analysis data is the largest;

[0021] Determining a first data value of the tail analysis data for the smoothed data chain;

[0022]

[0023] Wherein, a1 is the first data value, n1 is the number of original analysis data of the smoothed data chain, a is the importance value corresponding to the tail analysis data, ei is the importance value corresponding to the i-th original analysis data of the smoothed data chain, and fi is the weight corresponding to the i-th original analysis data;

[0024] determining a second data value of the header analysis data for the irregular data chain;

[0025]

[0026] Wherein, a2 is the value of the second data, n2 is the number of original analysis data of the irregular data chain, g is the importance value corresponding to the head analysis data, hj is the importance value corresponding to the jth original analysis data of the irregular data chain, and kj is the weight corresponding to the jth original analysis data;

[0027] Calculating a data value difference between the first data value and the second data value;

[0028] Obtaining a preset data value difference, and if the data value difference is greater than or equal to the preset data value difference, generating an inter-chain association identifier for the tail analysis data and the head analysis data;

[0029] Analyze all remaining original analysis data of the smooth data chain and the irregular data chain, and generate corresponding inter-chain association identifiers based on the analysis results;

[0030] The number of identifiers of all inter-link association identifiers is counted, and it is determined whether to generate a transmission association mark for the smooth data link and the irregular data link according to the number of identifiers.

[0031] Further, when counting the number of identifiers of all inter-link association identifiers and judging whether to generate a transmission association mark for the smooth data link and the irregular data link according to the number of identifiers, it includes:

[0032] Obtaining a preset number of preset identifiers, and determining whether to generate a transmission association mark for the smooth data link and the irregular data link according to a relationship between the number of identifiers and the preset number of identifiers;

[0033] When the number of identifiers is less than the preset number of identifiers, it is determined that no transmission association mark is generated for the smooth data link and the irregular data link;

[0034] When the number of identifiers is greater than or equal to the preset number of identifiers, it is determined whether to generate transmission association marks for the smooth data link and the irregular data link.

[0035] Further, when the transmission speed of the data link to be transmitted is set based on all the data transmission deviation values, it includes:

[0036] Extracting the same data transmission deviation value from all data transmission deviation values ​​and constructing multiple data transmission deviation value sets;

[0037] Counting a first set number of statistical data transmission deviation value sets;

[0038] Extracting a data transmission deviation value from all data transmission deviation value sets respectively, and calculating a first data transmission deviation value and a value;

[0039] Obtaining a preset data transmission deviation value, eliminating all data transmission deviation value sets that are smaller than the preset data transmission deviation value, and counting the number of second sets of remaining data transmission deviation value sets;

[0040] extracting a data transmission deviation value from each of the remaining data transmission deviation value sets, and calculating a second data transmission deviation value and a value;

[0041] Calculate the overall deviation factor of the data link to be transmitted according to the first set quantity, the second set quantity, the first data transmission deviation value and the second data transmission deviation value and the value;

[0042] The transmission speed of the data link to be transmitted is set based on the overall deviation factor.

[0043] Further, when calculating the overall deviation factor of the data link to be transmitted according to the first set quantity, the second set quantity, the first data transmission deviation value and the second data transmission deviation value and the value, it includes:

[0044] The overall deviation factor of the data link to be transmitted is calculated according to the following formula:

[0045]

[0046] Among them, R is the overall deviation factor of the data link to be transmitted, S2 is the second data transmission deviation value and value, S1 is the first data transmission deviation value and value, W1 is the first set number, W2 is the second set number, and r is the preset data transmission deviation value.

[0047] Further, when the transmission speed of the data link to be transmitted is set based on the overall deviation factor, it includes:

[0048] Setting a plurality of overall deviation factor intervals, wherein each overall deviation factor interval includes a first preset overall deviation factor, a second preset overall deviation factor, and a preset transmission speed;

[0049] Traversing the overall deviation factor of the data link to be transmitted in all overall deviation factor intervals, determining the overall deviation factor interval corresponding to the overall deviation factor, and extracting the corresponding preset transmission speed;

[0050] The extracted preset transmission speed is used as the transmission speed of the data link to be transmitted.

[0051] Furthermore, the training method of the data analysis large model is as follows:

[0052] Acquire historical analysis data, clean and standardize the historical analysis data, and generate a preprocessed data set;

[0053] Extract relevant features from the preprocessed data set and use a feature selection algorithm to determine a feature subset that has a significant impact on the prediction target;

[0054] Selecting an initial machine learning model and initializing parameters of the initial machine learning model;

[0055] Using the feature subset to train the initial machine learning model, adjusting model parameters by an iterative optimization method until a predetermined convergence condition is met;

[0056] Use cross-validation to evaluate the performance of the trained model, and adjust the model's hyperparameters based on the evaluation results until the optimal model performance is achieved;

[0057] The optimal model that has been trained and verified is used as the data analysis model.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] The present invention discloses a method for completing big data analysis using a large model. The method comprises the following steps: performing data preprocessing on original analysis data to obtain a smooth data chain and an irregular data chain; judging whether to generate a transmission association mark for the smooth data chain and the irregular data chain; when judging to generate a transmission association mark, connecting the smooth data chain and the irregular data chain to obtain a data chain to be transmitted; dividing the data chain to be transmitted into a plurality of transmission data blocks, determining a data transmission deviation value of each transmission data block and a data analysis model data temporary storage area, setting a transmission speed based on the data transmission deviation value, and transmitting the data chain to be transmitted to a large data analysis model; performing data analysis on the data chain to be transmitted based on the large data analysis model and displaying the analysis results, performing correlation analysis on a large amount of data, and quickly inputting the correlated data into the large data analysis model, thereby improving the data analysis efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0061] Figure 1 A flow chart of a method for completing big data analysis using a large model in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0062] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0063] In the description of the present application, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0064] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0065] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0066] The following is a description of preferred embodiments of the present invention with reference to the accompanying drawings.

[0067] like Figure 1 As shown, an embodiment of the present invention discloses a method for completing big data analysis using a big model, including:

[0068] S110: acquiring original analysis data, and performing data preprocessing on the original analysis data to obtain a smooth data chain and an irregular data chain;

[0069] In some embodiments of the present application, when obtaining original analysis data and performing data preprocessing on the original analysis data to obtain smooth data chains and irregular data chains, the method includes:

[0070] Output the importance value of each raw analysis data based on the pre-trained data importance evaluation model;

[0071] All importance values ​​are deployed, and curve fitting is performed on all importance values ​​according to a preset curve fitting method to obtain a smooth importance value curve;

[0072] Determine whether there are unfitted importance values, and if so, calculate the distances between all unfitted importance values ​​and the importance value curve respectively;

[0073] Compare all distances with preset distances, and associate all distances less than or equal to the preset distance with the importance value curve;

[0074] Establishing non-association marks between all distances greater than the preset distance and the importance value curve;

[0075] Generate the smoothed data chain according to the original analysis data corresponding to the importance value curve and the original analysis data corresponding to the association identifier;

[0076] The irregular data chain is generated according to the original analysis data corresponding to the non-association identifier.

[0077] In this embodiment, the original analysis data refers to initial data that has not been processed.

[0078] In this embodiment, the data importance evaluation model is pre-trained, and is obtained through training based on different data samples and importance values ​​matched by the data samples.

[0079] In this embodiment, linear fitting, polynomial fitting and other methods may be used to perform curve fitting on all importance values.

[0080] In this embodiment, the error function method, Lagrangian function method, etc. can be used to calculate the distance between all unfitted importance values ​​and the importance value curve. The specific calculation process is complicated and mature, and is not specifically limited here.

[0081] The beneficial effect of the above technical solution is: the present invention generates a smooth data chain based on the original analysis data corresponding to the importance value curve and the original analysis data corresponding to the associated identifier, and generates an irregular data chain based on the original analysis data corresponding to the non-associated identifier, which can realize the preliminary classification of data and avoid instability when the data is transmitted to the large data analysis model.

[0082] In some embodiments of the present application, when analyzing the smooth data link and the irregular data link, and judging whether to generate a transmission association mark for the smooth data link and the irregular data link based on the analysis result, the method includes:

[0083] The smooth data chain and the irregular data chain are sorted respectively based on the importance value relationship, and the tail analysis data of the smooth data chain and the head analysis data of the irregular data chain are extracted, and the importance value of the tail analysis data is the smallest, and the importance value of the head analysis data is the largest;

[0084] Determining a first data value of the tail analysis data for the smoothed data chain;

[0085]

[0086] Wherein, a1 is the first data value, n1 is the number of original analysis data of the smoothed data chain, a is the importance value corresponding to the tail analysis data, ei is the importance value corresponding to the i-th original analysis data of the smoothed data chain, and fi is the weight corresponding to the i-th original analysis data;

[0087] determining a second data value of the header analysis data for the irregular data chain;

[0088]

[0089] Wherein, a2 is the value of the second data, n2 is the number of original analysis data of the irregular data chain, g is the importance value corresponding to the head analysis data, hj is the importance value corresponding to the jth original analysis data of the irregular data chain, and kj is the weight corresponding to the jth original analysis data;

[0090] Calculating a data value difference between the first data value and the second data value;

[0091] Obtaining a preset data value difference, and if the data value difference is greater than or equal to the preset data value difference, generating an inter-chain association identifier for the tail analysis data and the head analysis data;

[0092] Analyze all remaining original analysis data of the smooth data chain and the irregular data chain, and generate corresponding inter-chain association identifiers based on the analysis results;

[0093] The number of identifiers of all inter-link association identifiers is counted, and it is determined whether to generate a transmission association mark for the smooth data link and the irregular data link according to the number of identifiers.

[0094] In this embodiment, the head analysis data of the smooth data chain has the largest importance value, and the tail analysis data has the smallest importance value. The head analysis data of the irregular data chain has the largest importance value, and the tail analysis data has the smallest importance value.

[0095] In this embodiment, the preset data value difference is obtained based on historical data, and is preferably 0.75.

[0096] In this embodiment, when analyzing all remaining original analysis data of the smooth data chain and the irregular data chain, the adjacent analysis data of the tail analysis data on the smooth data chain are first selected, and then the adjacent analysis data of the head analysis data on the irregular data chain are selected. These two adjacent analysis data are calculated and analyzed according to the above method, and all remaining original analysis data are analyzed in turn. If a single original analysis data exists, it is deleted.

[0097] The beneficial effect of the above technical solution is: the present invention analyzes all original analysis data of smooth data links and irregular data links, and generates corresponding inter-chain association identifiers based on the analysis results, which can realize the coordinated transmission of smooth data links and irregular data links, shorten data transmission time, and improve data analysis efficiency.

[0098] S120: Analyze the smooth data link and the irregular data link, and determine whether to generate a transmission association mark for the smooth data link and the irregular data link based on the analysis result;

[0099] In some embodiments of the present application, when counting the number of identifiers of all inter-link association identifiers and determining whether to generate a transmission association mark for the smooth data link and the irregular data link according to the number of identifiers, it includes:

[0100] Obtaining a preset number of preset identifiers, and determining whether to generate a transmission association mark for the smooth data link and the irregular data link according to a relationship between the number of identifiers and the preset number of identifiers;

[0101] When the number of identifiers is less than the preset number of identifiers, it is determined that no transmission association mark is generated for the smooth data link and the irregular data link;

[0102] When the number of identifiers is greater than or equal to the preset number of identifiers, it is determined whether to generate transmission association marks for the smooth data link and the irregular data link.

[0103] In this embodiment, the number of preset identifiers is set according to actual conditions, and 12 is preferably set here.

[0104] The beneficial effect of the above technical solution is: the present invention determines whether to generate transmission association marks for smooth data links and irregular data links based on the relationship between the number of identifiers and the preset number of identifiers, thereby achieving accurate judgment, avoiding large errors caused by manual participation, and achieving intelligent judgment.

[0105] S130: When it is determined that a transmission association mark is generated for the smooth data link and the irregular data link, the smooth data link and the irregular data link are connected to obtain a data link to be transmitted;

[0106] S140: Divide the data chain to be transmitted into a plurality of transmission data blocks, determine a data transmission deviation value between each transmission data block and the data temporary storage area of ​​the data analysis model, set a transmission speed of the data chain to be transmitted based on all the data transmission deviation values, and transmit the data chain to be transmitted to the data analysis large model based on the transmission speed;

[0107] In some embodiments of the present application, when the transmission speed of the data link to be transmitted is set based on all data transmission deviation values, it includes:

[0108] Extracting the same data transmission deviation value from all data transmission deviation values ​​and constructing multiple data transmission deviation value sets;

[0109] Counting a first set number of statistical data transmission deviation value sets;

[0110] Extracting a data transmission deviation value from all data transmission deviation value sets respectively, and calculating a first data transmission deviation value and a value;

[0111] Obtaining a preset data transmission deviation value, eliminating all data transmission deviation value sets that are smaller than the preset data transmission deviation value, and counting the number of second sets of remaining data transmission deviation value sets;

[0112] extracting a data transmission deviation value from each of the remaining data transmission deviation value sets, and calculating a second data transmission deviation value and a value;

[0113] Calculate the overall deviation factor of the data link to be transmitted according to the first set quantity, the second set quantity, the first data transmission deviation value and the second data transmission deviation value and the value;

[0114] The transmission speed of the data link to be transmitted is set based on the overall deviation factor.

[0115] In this embodiment, the data transmission deviation value refers to the difference between data and its standard or reference position during the data transmission process.

[0116] In this embodiment, if all data transmission deviation values ​​are {3,3,4,6,7,7,8,8}, then the data transmission deviation value sets are {3,3}, {7,7}, {8,8}, the number of first sets is 3, and one data transmission deviation value is extracted from each of all data transmission deviation value sets, that is, 3,7,8, and the sum of the first data transmission deviation values ​​= 3+7+8. If the preset data transmission deviation value is 5, the remaining data transmission deviation value sets are {7,7}, {8,8}, and the number of second sets of the remaining data transmission deviation value sets is 2, and one data transmission deviation value is extracted from each of the remaining data transmission deviation value sets, that is, 7,8, and the sum of the second data transmission deviation values ​​= 7+8. It should be understood that the above is shown by way of example and is not specifically limited.

[0117] The beneficial effect of the above technical solution is: the present invention calculates the overall deviation factor of the data link to be transmitted based on the first set quantity, the second set quantity, the first data transmission deviation value and the second data transmission deviation value and value, which can lay the foundation for setting the transmission speed of the data link to be transmitted and ensure the setting accuracy and flexibility.

[0118] In some embodiments of the present application, when calculating the overall deviation factor of the data link to be transmitted according to the first set quantity, the second set quantity, the first data transmission deviation value and the second data transmission deviation value and the value, it includes:

[0119] The overall deviation factor of the data link to be transmitted is calculated according to the following formula:

[0120]

[0121] Among them, R is the overall deviation factor of the data link to be transmitted, S2 is the second data transmission deviation value and value, S1 is the first data transmission deviation value and value, W1 is the first set number, W2 is the second set number, and r is the preset data transmission deviation value.

[0122] In some embodiments of the present application, when the transmission speed of the data link to be transmitted is set based on the overall deviation factor, it includes:

[0123] Setting a plurality of overall deviation factor intervals, wherein each overall deviation factor interval includes a first preset overall deviation factor, a second preset overall deviation factor, and a preset transmission speed;

[0124] Traversing the overall deviation factor of the data link to be transmitted in all overall deviation factor intervals, determining the overall deviation factor interval corresponding to the overall deviation factor, and extracting the corresponding preset transmission speed;

[0125] The extracted preset transmission speed is used as the transmission speed of the data link to be transmitted.

[0126] In this embodiment, the first preset overall deviation factor is smaller than the second preset overall deviation factor. The first preset overall deviation factor, the second preset overall deviation factor and the preset transmission speed are obtained based on historical data, which can put the data link to be transmitted in the best transmission state.

[0127] The beneficial effect of the above technical solution is: the present invention traverses the overall deviation factor of the data link to be transmitted in all overall deviation factor intervals, determines the overall deviation factor interval corresponding to the overall deviation factor, and extracts the corresponding preset transmission speed, and uses the extracted preset transmission speed as the transmission speed of the data link to be transmitted, which simplifies the calculation process and improves the accuracy of the calculation process, avoids subjective errors caused by manual participation in the setting, and ensures stable and reliable transmission of data.

[0128] S150: Perform data analysis on the data link to be transmitted based on the data analysis big model, and display the data analysis results, wherein the data analysis includes data recursion, data growth rate, and data transition rate.

[0129] In some embodiments of the present application, the training method of the data analysis large model is as follows:

[0130] Acquire historical analysis data, clean and standardize the historical analysis data, and generate a preprocessed data set;

[0131] Extract relevant features from the preprocessed data set and use a feature selection algorithm to determine a feature subset that has a significant impact on the prediction target;

[0132] Selecting an initial machine learning model and initializing parameters of the initial machine learning model;

[0133] Using the feature subset to train the initial machine learning model, adjusting model parameters by an iterative optimization method until a predetermined convergence condition is met;

[0134] Use cross-validation to evaluate the performance of the trained model, and adjust the model's hyperparameters based on the evaluation results until the optimal model performance is achieved;

[0135] The optimal model that has been trained and verified is used as the data analysis model.

[0136] In this embodiment, historical data analysis: First, historical analysis data needs to be collected. These data come from different sources, such as databases, log files, online trading platforms, etc. The diversity and quality of the data directly affect the accuracy of subsequent analysis. Cleaning and standardization: Data cleaning includes steps such as removing outliers, filling missing values, and format unification. Standardization processing aims to convert data into a common format, such as adjusting the data to a standard normal distribution, which helps to improve the convergence speed and performance of the model.

[0137] In this embodiment, feature extraction: extract relevant features from the preprocessed data set. Feature selection algorithm: apply feature selection algorithm (such as recursive feature elimination, model-based feature selection, etc.) to determine the feature subset that has a significant impact on the prediction target. This can effectively reduce the complexity of the model and improve the explanatory power and performance of the model.

[0138] In this embodiment, the initial model is selected: one or more initial machine learning models (such as decision tree, random forest, support vector machine, etc.) are selected according to the nature of the problem (such as classification, regression, etc.). Parameter initialization: the parameters of the selected model are initialized. This includes setting the learning rate, number of iterations, number of layers, etc. The initial values ​​of these parameters will affect the efficiency of model training and the final performance.

[0139] In this embodiment, model training: the initial model is trained using a feature subset. During the training process, the model parameters are adjusted by iterative optimization methods (such as gradient descent, stochastic gradient descent, etc.) to minimize the loss function. Convergence conditions: training will continue until a predetermined convergence condition is met, such as reaching a preset number of iterations, the error is below a certain threshold, or the performance on the validation set stops improving.

[0140] In this embodiment, cross validation: a cross validation method is used to evaluate the performance of the trained model. This method can provide an estimate of the performance of the model on an independent data set and reduce the risk of overfitting the model. Hyperparameter adjustment: Based on the evaluation results of cross validation, the hyperparameters of the model, such as the learning rate, regularization coefficient, etc., are adjusted to obtain better model performance.

[0141] In this embodiment, model deployment: deploy the trained and verified optimal model as a large data analysis model. Before deployment, it is necessary to ensure the stability and reliability of the model, as well as its adaptability in the production environment. Continuous monitoring and maintenance: After the model is deployed, its performance is continuously monitored, and adjustments and maintenance are made based on changes in actual data. This may include regular retraining of the model to adapt to new data distributions, or updating feature sets to capture the latest business trends.

[0142] The beneficial effects of the above technical solution are: through strict data processing, reasonable feature selection, appropriate model training and evaluation, and continuous model maintenance, an accurate and reliable data analysis model can be constructed to achieve real-time display of data delivery, data growth rate, and data transition rate.

[0143] In the description of the above embodiments, specific features, structures, materials or characteristics may be combined in a suitable manner in any one or more embodiments or examples.

[0144] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed by the present invention may be used in combination with each other in any manner, and the fact that these combinations are not fully described in this specification is only for the sake of omitting space and saving resources.

[0145] Those skilled in the art can understand that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions recorded in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for performing big data analysis using a big model, characterized in that: include: Acquire original analysis data, and perform data preprocessing on the original analysis data to obtain smooth data chains and irregular data chains; Analyzing the smooth data link and the irregular data link, and determining whether to generate a transmission association mark for the smooth data link and the irregular data link based on the analysis result; When it is determined that a transmission association mark is generated for the smooth data link and the irregular data link, the smooth data link and the irregular data link are connected to obtain a data link to be transmitted; Divide the data chain to be transmitted into a plurality of transmission data blocks, determine a data transmission deviation value between each transmission data block and a data analysis model data temporary storage area, set a transmission speed of the data chain to be transmitted based on all data transmission deviation values, and transmit the data chain to be transmitted to the data analysis model based on the transmission speed; Based on the data analysis big model, data analysis is performed on the data link to be transmitted, and the data analysis results are displayed, wherein the data analysis includes data recursion, data growth rate, and data transition rate.

2. The method for performing big data analysis using a big model according to claim 1, characterized in that: When obtaining the original analysis data and performing data preprocessing on the original analysis data to obtain a smooth data chain and an irregular data chain, it includes: Output the importance value of each raw analysis data based on the pre-trained data importance evaluation model; All importance values ​​are deployed, and curve fitting is performed on all importance values ​​according to a preset curve fitting method to obtain a smooth importance value curve; Determine whether there are unfitted importance values, and if so, calculate the distances between all unfitted importance values ​​and the importance value curve respectively; Compare all distances with preset distances, and associate all distances less than or equal to the preset distance with the importance value curve; Establishing non-association marks between all distances greater than the preset distance and the importance value curve; Generate the smoothed data chain according to the original analysis data corresponding to the importance value curve and the original analysis data corresponding to the association identifier; The irregular data chain is generated according to the original analysis data corresponding to the non-association identifier.

3. The method for performing big data analysis using a big model according to claim 2, characterized in that: When analyzing the smooth data link and the irregular data link, and judging whether to generate a transmission association mark for the smooth data link and the irregular data link based on the analysis result, the method includes: The smooth data chain and the irregular data chain are sorted respectively based on the importance value relationship, and the tail analysis data of the smooth data chain and the head analysis data of the irregular data chain are extracted, and the importance value of the tail analysis data is the smallest, and the importance value of the head analysis data is the largest; Determining a first data value of the tail analysis data for the smoothed data chain; Wherein, a1 is the first data value, n1 is the number of original analysis data of the smoothed data chain, a is the importance value corresponding to the tail analysis data, ei is the importance value corresponding to the i-th original analysis data of the smoothed data chain, and fi is the weight corresponding to the i-th original analysis data; determining a second data value of the header analysis data for the irregular data chain; Wherein, a2 is the value of the second data, n2 is the number of original analysis data of the irregular data chain, g is the importance value corresponding to the head analysis data, hj is the importance value corresponding to the jth original analysis data of the irregular data chain, and kj is the weight corresponding to the jth original analysis data; Calculating a data value difference between the first data value and the second data value; Obtaining a preset data value difference, and if the data value difference is greater than or equal to the preset data value difference, generating an inter-chain association identifier for the tail analysis data and the head analysis data; Analyze all remaining original analysis data of the smooth data chain and the irregular data chain, and generate corresponding inter-chain association identifiers based on the analysis results; The number of identifiers of all inter-link association identifiers is counted, and it is determined whether to generate a transmission association mark for the smooth data link and the irregular data link according to the number of identifiers.

4. The method for performing big data analysis using a big model according to claim 3, characterized in that: When counting the number of identifiers of all inter-link association identifiers and judging whether to generate a transmission association mark for the smooth data link and the irregular data link according to the number of identifiers, the method includes: Obtaining a preset number of preset identifiers, and determining whether to generate a transmission association mark for the smooth data link and the irregular data link according to a relationship between the number of identifiers and the preset number of identifiers; When the number of identifiers is less than the preset number of identifiers, it is determined that no transmission association mark is generated for the smooth data link and the irregular data link; When the number of identifiers is greater than or equal to the preset number of identifiers, it is determined whether to generate transmission association marks for the smooth data link and the irregular data link.

5. The method for performing big data analysis using a big model according to claim 1, characterized in that: When the transmission speed of the data link to be transmitted is set based on all the data transmission deviation values, it includes: Extracting the same data transmission deviation value from all data transmission deviation values ​​and constructing multiple data transmission deviation value sets; Counting a first set number of statistical data transmission deviation value sets; Extracting a data transmission deviation value from all data transmission deviation value sets respectively, and calculating a first data transmission deviation value and a value; Obtaining a preset data transmission deviation value, eliminating all data transmission deviation value sets that are smaller than the preset data transmission deviation value, and counting the number of second sets of remaining data transmission deviation value sets; extracting a data transmission deviation value from each of the remaining data transmission deviation value sets, and calculating a second data transmission deviation value and a value; Calculate the overall deviation factor of the data link to be transmitted according to the first set quantity, the second set quantity, the first data transmission deviation value and the second data transmission deviation value and the value; The transmission speed of the data link to be transmitted is set based on the overall deviation factor.

6. The method for performing big data analysis using a big model according to claim 5, characterized in that: When calculating the overall deviation factor of the data link to be transmitted according to the first set quantity, the second set quantity, the first data transmission deviation value and the second data transmission deviation value and the value, comprising: The overall deviation factor of the data link to be transmitted is calculated according to the following formula: Among them, R is the overall deviation factor of the data link to be transmitted, S2 is the second data transmission deviation value and value, S1 is the first data transmission deviation value and value, W1 is the first set number, W2 is the second set number, and r is the preset data transmission deviation value.

7. The method for performing big data analysis using a big model according to claim 6, characterized in that: When the transmission speed of the data link to be transmitted is set based on the overall deviation factor, it includes: Setting a plurality of overall deviation factor intervals, wherein each overall deviation factor interval includes a first preset overall deviation factor, a second preset overall deviation factor, and a preset transmission speed; Traversing the overall deviation factor of the data link to be transmitted in all overall deviation factor intervals, determining the overall deviation factor interval corresponding to the overall deviation factor, and extracting the corresponding preset transmission speed; The extracted preset transmission speed is used as the transmission speed of the data link to be transmitted.

8. The method for performing big data analysis using a big model according to claim 1, characterized in that: The training method of the data analysis large model is as follows: Acquire historical analysis data, clean and standardize the historical analysis data, and generate a preprocessed data set; Extract relevant features from the preprocessed data set and use a feature selection algorithm to determine a feature subset that has a significant impact on the prediction target; Selecting an initial machine learning model and initializing parameters of the initial machine learning model; Using the feature subset to train the initial machine learning model, adjusting model parameters by an iterative optimization method until a predetermined convergence condition is met; Use cross-validation to evaluate the performance of the trained model, and adjust the model's hyperparameters based on the evaluation results until the optimal model performance is achieved; The optimal model that has been trained and verified is used as the data analysis model.

Images