Big data unified analysis processing method and system based on cloud computing

Abstract

The invention discloses a big data unified analysis processing method and system based on cloud computing, and relates to the technical field of big data intelligent analysis, and the method comprises the steps: calculating the optimal score of each data source, selecting the data source with the highest optimal score, collecting original data, and preprocessing the original data; respectively extracting global and local features through a model combination mode based on the preprocessed data to form a feature data set, calculating feature importance scores based on the feature data set to generate a new feature data set, removing abnormal features in the new feature data set, and generating a frequent item group; and performing joint prediction and generating a task decision target based on the frequent item group in combination with fuzzy reasoning and adaptive regression, generating task decision data based on the task decision target, and transmitting the original data and the task decision data to a database for storage and backup. According to the method, by adopting frequent item set mining and association rule screening, potential modes in the data are automatically identified, low-efficiency modes are eliminated, and the interpretability of data analysis is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data intelligent analysis, and particularly to a unified big data analysis and processing method and system based on cloud computing. Background Art

[0002] With the rapid development of information technology, cloud computing and big data analysis have become key technologies to promote intelligent decision-making and optimize resource management. The high concurrent processing ability and elastic expansion characteristics of cloud computing make it possible to store and calculate large-scale data. Big data analysis uses a variety of statistical methods, machine learning algorithms, and data mining techniques to extract valuable information from massive data. In practical applications, big data analysis is widely used in many fields such as smart cities, industrial Internet, financial risk control, and intelligent scheduling.

[0003] However, existing big data analysis methods still have many limitations in aspects such as data source selection, feature extraction, pattern recognition, and prediction optimization, which restrict their application effects in complex task scheduling and decision-making scenarios. For example, in the face of heterogeneous data sources, how to dynamically evaluate the quality and relevance of data sources to ensure the accuracy and reliability of data analysis is still a challenging problem. In addition, during the feature extraction process, existing methods often rely on a single model and lack joint modeling of global and local features, resulting in insufficiently comprehensive feature representation and affecting the accuracy of downstream analysis. At the same time, in the pattern recognition stage, most existing technologies use static rules or traditional clustering methods, which are difficult to effectively mine the implicit associations between data, thus affecting the accuracy and generalization ability of pattern recognition. In terms of prediction and decision-making generation, existing methods usually rely on fixed regression or classification models and lack effective modeling of complex data distributions and uncertain information, resulting in low accuracy and stability of prediction results. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a unified big data analysis and processing method based on cloud computing, which solves the reliability problem of data quality, improves the integrity and distinctiveness of data feature representation, enhances the effectiveness of pattern recognition, improves the accuracy of prediction, breaks through the limitations of existing big data analysis technologies, realizes full-link improvement from data source selection, feature extraction, pattern recognition to prediction optimization, has higher analysis accuracy and stronger adaptability, and is applicable to many fields such as intelligent scheduling, resource allocation, and complex decision-making support.

[0006] To solve the above technical problems, the present invention provides the following technical solutions:

[0007] In the first aspect, the present invention provides a unified big data analysis and processing method based on cloud computing, which includes,

[0008] Calculate the optimal score for each data source, select the data source with the highest optimal score to collect the original data and preprocess it;

[0009] Based on the preprocessed data, extract global and local features respectively through the model combination method to form a feature dataset;

[0010] Based on the feature dataset, calculate the feature importance score to generate a new feature dataset, identify the potential patterns in the new feature dataset and generate frequent item sets;

[0011] Based on the frequent item sets, combine fuzzy inference and adaptive regression for joint prediction and generate task decision targets, and generate task decision data based on the task decision targets;

[0012] Transmit the original data and task decision data to the database for storage and backup.

[0013] As a preferred solution of the big data unified analysis and processing method based on cloud computing according to the present invention, wherein: the calculation of the optimal score for each data source, selecting the data source with the highest optimal score to collect the original data and preprocess it refers to calculating the optimal score for each data source, selecting the data source with the highest optimal score to collect the original data and preprocess it, which means collecting image data and text data from the database, API interface and Kafka data stream through the cloud service tool AWSGlue, and defining the original variables according to the task-related data Including task ID, task priority, task planned completion time, task start time and task end time, calculate the task execution time based on the task start time and end time, calculate the task success rate and on-time completion rate in combination with the task planned completion time, and use the calculated task success rate and on-time completion rate as the intermediate variables of the task success rate and the intermediate variable of the task completion rate. According to The intermediate variables, respectively define the target variables of the highest task success rate and on-time completion rate;

[0014] Calculate the Pearson correlation coefficient and grey relational degree between the intermediate variables and the target variables, use z-score normalization to map the values of the Pearson correlation coefficient and grey relational degree to the interval [0, 1], assign weights to the Pearson correlation coefficient and grey relational degree, and calculate the optimal score of each data source through weighted fusion, and select the data source with the highest score as the final data source for collecting the original data;

[0015] The preprocessing refers to cleaning and standardizing the collected final original data.

[0016] As a preferred solution of the big data unified analysis and processing method based on cloud computing according to the present invention, wherein: the step of extracting global and local features respectively by a model combination method from the preprocessed data to form a feature dataset refers to using the BoW model to construct a vocabulary for the text data in the preprocessed data, converting all terms in the vocabulary into term frequency vectors, assigning a fixed initial weight to each term in the vocabulary in combination with the Hybrid IDF (inverse document frequency), generating self-attention weights through the Transformer model, and multiplying the initial weight of each term by the self-attention weight to generate the final weight;

[0017] In the Transformer model, the pre-normalization Pre-LayerNorm structure is adopted. After combining the term frequency vector with the position encoding, the combined result is input into the pre-normalized Transformer model in combination with the final weight. After being processed by multiple layers of self-attention and feed-forward neural networks, global features are generated;

[0018] For the image data, the Retinex algorithm is used for illumination correction, and a CNN model is constructed to extract CNN features from the image data;

[0019] Using the Generalized Image Statistical Feature (GIST), each image in the image data is divided into fixed 8×8 grid blocks. The mean gradient response of each grid block is calculated using a Gabor filter with 4 directions × 4 scales to obtain the local feature vector of each grid block. The local feature vectors of all grid blocks are concatenated as the final GIST descriptor. Based on the spatial block structure of each image represented by the GIST descriptor, a GCN (Graph Convolutional Network) is constructed, and the constructed GCN is used to extract GCN features from the image data;

[0020] The CNN features and the GCN features are concatenated into an image feature vector. The initial weights of the CNN features and the GCN features are respectively set and input into a Multi-Layer Perceptron (MLP). The loss gradients of the initial weights of the CNN and GCN features are calculated through the backpropagation algorithm. The initial weights are updated based on the loss gradients. According to the image feature vector and the updated initial weights, the local features are output after weighted fusion;

[0021] The global features and the local features are combined to generate a feature dataset.

[0022] As a preferred solution of the big data unified analysis and processing method based on cloud computing according to the present invention, wherein: the step of calculating the feature importance score based on the feature dataset to generate a new feature dataset and removing the abnormal features in the new feature dataset and generating frequent item sets refers to calculating the information gain and information entropy of each feature in the feature dataset, using the information gain as the weight of each feature, and calculating the feature importance score based on the weight of each feature and the corresponding information entropy;

[0023] Sort all the feature importance scores, set a score threshold according to the distribution characteristics of the feature information gain, and retain the features whose feature importance scores meet the score threshold to form a new feature dataset;

[0024] Use the DBSCAN clustering algorithm to calculate the number of features around each feature in the new feature dataset as the number of neighboring points, set the minimum number of neighboring points, when the number of neighboring points of a feature is greater than or equal to the minimum number of neighboring points, take this feature as a core point, take the neighboring features of the core point as boundary points, identify the features that do not belong to the neighborhood of the core point, mark them as outliers and remove them;

[0025] Perform normalization processing on the new feature dataset processed by the DBSCAN clustering algorithm, use One-Class SVM to map the features in the new feature dataset to a high-dimensional space through the radial basis function RBF kernel, and solve the optimal normal vector, bias term, and slack variables by minimizing the objective function;

[0026] Use One-Class SVM to identify and remove abnormal features in the new feature dataset based on the optimal normal vector, bias term, and slack variables;

[0027] Convert the features in the new feature dataset processed by One-Class SVM into binary vectors to form a candidate item list, use the Apriori algorithm to calculate the support degree of each candidate item, take the candidate items whose support degree is greater than or equal to the set threshold as frequent items, each frequent item generates new candidate items according to the k-item set expansion principle, calculate the support degree of the new candidate items again and compare it with the set threshold, and perform multiple iterative operations until no new frequent items are generated;

[0028] Generate association rules based on frequent items through conditional probability, and calculate the confidence and lift values of each association rule Retain the frequent items that simultaneously meet the conditions that the confidence is greater than the set threshold and the lift is positively correlated, and merge the frequent items that meet the conditions into frequent item groups.

[0029] As a preferred solution of the big data unified analysis and processing method based on cloud computing according to the present invention, wherein: the combined fuzzy inference and adaptive regression based on the frequent item group for joint prediction and generation of task decision objectives refers to through the statistical distribution analysis of the frequent item group, using the triangular membership function to construct a fuzzy set, each frequent item is divided into different fuzzy sets, using the FURIA algorithm to calculate the similarity between fuzzy rules to remove redundant rules in the fuzzy set, using the fuzzy extension mechanism of FURIA to adjust the boundary of the fuzzy rules, and calculating the membership degree of each feature in the frequent item in the fuzzy set;

[0030] For the features in the frequent items, use the minimum value method to calculate the joint fuzzy membership degree of each frequent item under different fuzzy rules. Based on the joint fuzzy membership degree, use the product method to calculate the activation degree of each fuzzy rule. Based on the activation degree of each fuzzy rule and the predefined output categories of each fuzzy rule, calculate the support degrees of all categories. Normalize the support degrees of all categories to calculate and output the confidence degrees of all categories. Select the prediction category with the highest confidence degree as the final prediction category, and Use one-hot encoding to convert it into binary form and add it as a new feature to the frequent item to form a new frequent item. Input the updated frequent item into the trained ridge regression model to output the regression prediction value;

[0031] Define the task estimated completion time and task priority parameters based on the task-related data, and adjust the task estimated completion time and task priority parameters through the regression prediction value;

[0032] Take the corrected task completion time and task priority as the task decision objective.

[0033] As a preferred solution of the big data unified analysis and processing method based on cloud computing according to the present invention, wherein: generating task decision data based on the task decision objective means parsing out the specific parameters in the task decision objective as the optimal parameters based on the task decision objective, calculating the deviation values between the parameters by comparing the operation parameters of the current decision-making scheme, substituting the deviation values into a machine learning model for simulation prediction through the reinforcement learning method, and outputting the final optimized parameters of each item. Use the final optimized parameters of each item as task decision data for the current task decision-making scheme.

[0034] As a preferred solution of the big data unified analysis and processing method based on cloud computing according to the present invention, wherein: transmitting the original data and task decision data to the database for storage and backup means implementing data encryption during the transmission process through the TLS protocol based on the distributed data storage technology and end-to-end encryption transmission mechanism, using the AES-256 algorithm to encrypt the storage of the original data and task decision data, recording the data storage and backup operation logs in combination with the blockchain technology, improving the scalability and reliability of data storage through the distributed storage system HDFS, and at the same time enabling the off-site backup mechanism to perform multi-node storage and verification of data regularly.

[0035] In a second aspect, the present invention provides a big data unified analysis and processing system based on cloud computing, including

[0036] A data collection and storage module, responsible for collecting the original data and using the blockchain technology to store and backup the original data and task decision data;

[0037] A data source optimization and data processing module that filters multiple data sources and preprocesses the raw data;

[0038] A feature extraction and feature dataset generation module that extracts features from the preprocessed data and generates a feature dataset;

[0039] A feature importance scoring and frequent item set generation module that calculates the importance score for each feature based on the feature dataset and generates frequent item sets based on the feature dataset;

[0040] A joint prediction and task decision generation module that performs joint prediction by combining fuzzy inference and adaptive regression and generates final task decision data based on the prediction results.

[0041] In a third aspect, the present invention provides a computer device, including a memory and a processor, where the memory stores a computer program, and: when the computer program is executed by the processor, any step of the big data unified analysis and processing method based on cloud computing as described in the first aspect of the present invention is implemented.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and: when the computer program is executed by the processor, any step of the big data unified analysis and processing method based on cloud computing as described in the first aspect of the present invention is implemented.

[0043] The beneficial effects of the present invention are as follows: Through mutual information value, grey relational degree and weight calculation, the optimal selection of data sources is realized to ensure data quality and analysis stability. The Transformer, CNN, GCN and other models are used to jointly extract the features of text and image data, and information gain, DBSCAN and One-Class SVM are combined for feature screening to improve the representativeness of the feature set. Frequent item set mining and association rule screening are adopted, and inefficient patterns are eliminated to improve the interpretability of data analysis. By combining fuzzy inference and adaptive regression, accurate prediction of task objectives is realized, and decision-making parameters are continuously optimized through reinforcement learning to improve the efficiency and accuracy of the final task decision. Description of the Drawings

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0045] Figure 1 It is a flowchart of the big data unified analysis and processing method based on cloud computing in Embodiment 1.

[0046] Figure 2 This is a structural diagram of the big data unified analysis and processing system based on cloud computing in Example 1. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0048] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0049] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0050] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides a big data unified analysis and processing method based on cloud computing, comprising the following steps:

[0051] S1. Calculate the preferred score of each data source, select the data source with the highest preferred score to collect and preprocess the original data;

[0052] Specifically, the preferred score of each data source is calculated, and the data source with the highest preferred score is selected to collect the original data and preprocess it, which means collecting image data and text data from the database, API interface and Kafka data stream through the cloud service tool AWSGlue, defining the original variables according to the task-related data, including task ID, task priority, task planned completion time, task start time and task end time, calculating the task execution time based on the task start time and end time, calculating the task success rate and on-time completion rate in combination with the task planned completion time, and using the calculated task success rate and on-time completion rate as the task success rate intermediate variable and the task completion rate intermediate variable, and defining the highest task success rate and on-time completion rate target variables according to the two intermediate variables respectively;

[0053] Calculate the Pearson correlation coefficient between the intermediate variable and the target variable , It is expressed as:

[0054]

[0055] Among them, and respectively represent the intermediate variable and the target variable. If represents the intermediate variable of the task success rate, then represents the target variable of the task success rate. If represents the intermediate variable of the task completion rate, then represents the target variable of the on-time completion rate. represents and the covariance between and represents and the standard deviations of

[0056] Calculate the grey relational degree between the intermediate variable and the target variable , which is expressed as:

[0057]

[0058] Among them, represents the intermediate variable of the th data source, represents the target variable, represents the distinguishing coefficient, usually taking a value of 0.5;

[0059] The closer the value of the grey relational degree is to 1, the higher the correlation degree between and ;

[0060] Use z-score standardization to map the values of the Pearson correlation coefficient and the grey relational degree to the interval [0, 1], assign weights to the Pearson correlation coefficient and the grey relational degree according to expert opinions, and calculate the optimal score of each data source through weighted fusion , which is expressed as:

[0061]

[0062] Among them, represents the optimal score of a single data source, and are the weights of the Pearson correlation coefficient and the grey relational degree respectively, and respectively represent the normalized Pearson correlation coefficient and grey relational degree;

[0063] Select the data source with the highest score as the data source finally used to collect the original data;

[0064] The said preprocessing refers to cleaning and standardizing the finally collected original data.

[0065] By comprehensively considering multiple task-related factors (such as task priority, execution time, etc.), the present invention realizes automated data source evaluation, which not only reduces human intervention, improves the scientific nature of selection, but also can dynamically adjust the optimal score according to the actual needs of the task, thereby improving data collection efficiency. By quantitatively evaluating the advantages and disadvantages of each data source, it provides a clear selection criterion for final data collection and preprocessing. By combining the Pearson correlation coefficient and the grey relational degree, it not only considers the linear relationship between the task and the data source (Pearson correlation coefficient), but also introduces the grey system theory, which can handle incomplete information and uncertainty. Through weighted fusion, these characteristics are comprehensively considered, and the advantages and disadvantages of each data source can be accurately evaluated, avoiding the deviation that may be brought by a single evaluation criterion.

[0066] S2. Based on the preprocessed data, global and local features are respectively extracted through a model combination method to form a feature data set;

[0067] Specifically, extracting global and local features through a model combination method based on the preprocessed data to form a feature data set means using the BoW model to construct a vocabulary for the text data in the preprocessed data, converting all terms in the vocabulary into term frequency vectors, and combining the Hybrid IDF (inverse document frequency) to assign a fixed initial weight to each term in the vocabulary. The term weight is expressed as:

[0068]

[0069] where, represents the term, represents the term the number of occurrences of in the vocabulary d, represents the term with the most occurrences in the vocabulary d the number of occurrences of;

[0070] This vocabulary contains all unique terms that appear in the text data;

[0071] The dynamic adjustment of each initial weight is carried out through the self-attention mechanism of the Transformer model. The self-attention mechanism is expressed by the formula:

[0072]

[0073] where, represents the query matrix, represents the key matrix, represents the value matrix, represents the dimension of the key, represents the transpose of the matrix, It means operating to calculate the weights of each value and performing a weighted sum on the value matrix according to each weight;

[0074] The key and value matrices of the Transformer model are divided into multiple heads. Each head uses the self-attention mechanism formula for calculation, and the outputs of all heads are linearly transformed to obtain the final attention output;

[0075] Through the feed-forward neural network layer of the Transformer model, the final attention output is input into the feed-forward neural network layer. After the first linear layer, the input is mapped to a higher-dimensional space, and the nonlinearity is increased through the activation function ReLu. After the second linear layer, the final attention output is mapped back to the original dimension to generate self-attention weights, and the initial weight of each term is multiplied by the self-attention weights to generate the final weights;

[0076] In the Transformer model, the Pre-LayerNorm structure is adopted. Low-dimensional feature vectors are generated by random sampling as the training data of the Transformer model. The training data is input into the Transformer model for training. The trained model generates high-dimensional features similar to the original data from the low-dimensional features. The pixel loss and adversarial loss functions are used to evaluate the difference between the features generated by the trained model and the original data. Based on the evaluation results, the parameters of the Transformer model are optimized. After multiple training iterations until the loss function converges, the word frequency vector and position encoding are combined, and then combined with the final weights and input into the pre-normalized Transformer model. After being processed by multiple layers of self-attention and feed-forward neural networks, the PCA dimensionality reduction is performed on the Transformer features of each layer, the dimensionality-reduced features are concatenated across layers, and the global features are generated through average pooling;

[0077] The Retinex algorithm is used to correct the illumination of the image data, enhance the local contrast, and reduce the influence of illumination changes on feature extraction. A CNN model is constructed, and the illumination-corrected image data is input into the trained CNN model. Based on the convolution operation and pooling operation of the CNN model, CNN features are extracted from the image data;

[0078] Each image in the image data is divided into fixed 8×8 grid blocks using the Generalized Image Statistical Feature (GIST). The mean of the gradient responses of each grid block is calculated using 4 directions × 4 scales of Gabor filters to obtain the local feature vector of each grid block. The local feature vectors of all grid blocks are concatenated as the final GIST descriptor. A Graph Convolutional Network (GCN) is constructed based on the spatial block structure of each image represented by the GIST descriptor, and the constructed GCN is used to extract GCN features from the image data;

[0079] Concatenate the CNN features and the GCN features into an image feature vector, respectively set the initial weights of the CNN features and the GCN features, input them into the multi-layer perceptron MLP, calculate the loss gradients of the initial weights of the CNN and GCN features through the backpropagation algorithm, update the initial weights based on the loss gradients, and output the local features after weighted fusion according to the image feature vector and the updated initial weights;

[0080] Combine the global features and the local features to generate a feature dataset containing global information and local information.

[0081] In the present invention, the initial weight of each term is calculated by using Hybrid IDF and dynamically adjusted under the self-attention mechanism of the Transformer model to generate the final weight. It has the ability of global modeling and can dynamically adjust the weight to make the feature expression more accurate. The weight is further optimized through the Transformer feed-forward neural network to keep the weight stable in the semantic information of different levels, reduce the redundancy of text features, and improve the effectiveness of global feature extraction. By normalizing at the input of the Transformer layer instead of normalizing after the traditional residual connection, the gradient stability is enhanced, the training efficiency is improved, the problem of gradient disappearance or gradient explosion that may occur during the training of the Transformer is solved, the training stability of the model is improved, and the Transformer can more effectively learn the global features of text data;

[0082] Enhance the local contrast of the image by adopting the Retinex algorithm, reduce the influence of illumination change on feature extraction, improve the stability of the CNN and GIST features, use GIST to calculate the local structure information of the image, construct a GCN based on the spatial block structure calculated by GIST, extract more spatially consistent local features, calculate the weights of the CNN and GCN features through MLP, and optimize the weight allocation through backpropagation to improve the rationality of feature fusion, ensure that the fusion of the CNN and GCN features is adaptively adjusted according to the data distribution instead of a fixed weight set artificially, and improve the dynamics and robustness of feature fusion.

[0083] S3. Calculate the feature importance scores based on the feature dataset to generate a new feature dataset, remove the abnormal features in the new feature dataset and generate frequent item sets;

[0084] Specifically, calculating the feature importance scores based on the feature dataset to generate a new feature dataset, removing the abnormal features in the new feature dataset and generating frequent item sets means calculating the information gain and information entropy of each feature in the feature dataset, taking the information gain as the weight of each feature, and calculating the feature importance scores based on the weight of each feature and the corresponding information entropy, expressed as:

[0085]

[0086] Among them, represents the importance score of a feature ; represents the weight of a feature ; represents the information entropy of

[0087] Sort all the feature importance scores, set a threshold range according to the distribution characteristics of the feature information gain. The distribution characteristics of the feature information gain refer to that a small number of features have a high information gain and have a great impact on the target variable, while most features have a low information gain, small contribution or even close to noise. Retain the features with the feature importance scores within the set threshold range to form a new feature dataset, and only retain the features that are most valuable for prediction and analysis.

[0088] Use the DBSCAN clustering algorithm to calculate the number of features around each feature in the new feature dataset as the number of neighboring points, set the minimum number of neighboring points. When the number of neighboring points of a feature is greater than or equal to the minimum number of neighboring points, regard this feature as a core point, regard the neighboring features of the core point as boundary points, identify the features that do not belong to the neighborhood of the core point, mark them as outlier points and remove them.

[0089] Perform normalization processing on the new feature dataset processed by the DBSCAN clustering algorithm. Use One-Class SVM to construct a hyperplane in a high-dimensional space to distinguish normal data and abnormal data in the new feature dataset. The normal data and abnormal data respectively refer to the data points with the largest similar distribution ratio and the data points with a non-largest similar distribution ratio in the hyperplane. Map the features in the new feature dataset to a high-dimensional space through the radial basis function (RBF) kernel, solve the optimal normal vector, bias term and slack variables that allow violating the constraints by minimizing the objective function, and use One-Class SVM to identify and remove the abnormal features in the new feature dataset based on the optimal normal vector, bias term and slack variables.

[0090] Convert the features in the new feature dataset processed by One-Class SVM into binary vectors to form a list of candidate items. Each candidate item represents the combined relationship of one or more features in the new feature dataset. Among them, all single features are used as the initial candidate items, and the Apriori algorithm is used to calculate the support degree of each candidate item. The support degree is divided into single-feature support degree and multi-feature combination support degree. The single-feature support degree represents the frequency of a single feature of a certain candidate item appearing in the new feature dataset. The calculation method is to calculate by the number of times this feature appears in the new feature dataset divided by the total number of features. The multi-feature combination support degree represents the frequency of multiple features of a certain candidate item appearing simultaneously in the new feature dataset. The calculation method is the number of times this feature combination (composed of multiple single features) appears in the new feature dataset divided by the number of all possible feature combinations in the new feature dataset. For example, in a certain candidate item, a certain feature represents that the task execution time is long, and a certain feature represents that the task success rate is high. These two features are regarded as a feature combination. The candidate items with a support degree greater than or equal to the set threshold are used as frequent items. Each frequent item generates new candidate items according to the k-itemset expansion principle. The example of the k-itemset expansion principle starts from a frequent item of a single feature, generates candidate items composed of two features. If the support degree of the candidate items composed of two features meets the set threshold, generate candidate items composed of three features. Then calculate the support degree of the new candidate items again and compare it with the set threshold. After multiple iterative operations until no new frequent items are generated. Only the candidate items with a support degree higher than or equal to the set threshold can be called frequent items. Frequent items refer to the set of features that often appear together in the data, and there may be some potential associations between these features;

[0091] Generate association rules based on frequent items. For example, given a frequent item , ,the association rules that can be generated , G, etc. Calculate the confidence of each association rule. The confidence is expressed as:

[0092]

[0093] Among them, represents the association rule. If occurs, then may also occur. represents the association rule 's confidence. and represent the features in the frequent item and feature , represents feature and feature appearing simultaneously's support degree. Indicates a feature Support for individual occurrences;

[0094] Calculate the lift of each association rule The value is expressed as:

[0095]

[0096] where Indicates a feature , A value greater than 1 indicates a positive correlation between item sets, less than 1 indicates a negative correlation, and equal to 1 indicates independence between two item sets;

[0097] Retain the frequent items that simultaneously satisfy the confidence greater than the set threshold and the lift is positively correlated, and merge the frequent items that meet the conditions into frequent item groups.

[0098] The present invention calculates the information gain and information entropy, combines the feature weights to calculate the feature importance score, sets a threshold based on the distribution characteristics of the feature information gain, screens out the most valuable features for prediction and analysis, measures the contribution of the features through the information gain, and combines the information entropy to eliminate the low-contribution features, ensuring that the selected features have both predictive value and avoid interference from redundant features, improving the data quality and calculation efficiency. Through the division of core points and boundary points, this method can effectively identify isolated abnormal features and avoid the influence of noise features on subsequent analysis, thereby improving the reliability of the data;

[0099] By combining the DBSCAN and One-Class SVM dual screening mechanisms, it is ensured that the final new feature dataset retains as many features with predictive value as possible, further improving the data expression ability. By combining the confidence and lift to screen the association rules, it is possible to more accurately mine the feature relationships hidden in the data and avoid the problem that valuable rules are ignored by relying solely on the support for screening.

[0100] S4. Based on the frequent item groups, combine fuzzy inference and adaptive regression for joint prediction and generate task decision objectives, and generate task decision data based on the task decision objectives;

[0101] Specifically, joint prediction based on frequent item sets combined with fuzzy reasoning and adaptive regression to generate task decision goals means that through the statistical distribution analysis of frequent item sets, a triangular membership function is used to construct a fuzzy set. Each frequent item is divided into different fuzzy sets. For example, a task success rate fuzzy set is constructed through a triangular membership function, representing low, medium, and high task success rates. The FURIA algorithm is used to calculate the similarity between fuzzy rules to remove redundant rules in the fuzzy set. The fuzzy extension mechanism of FURIA is used to adjust the boundaries of fuzzy rules. By analyzing the redundancy and inconsistency in the fuzzy rule set, the fuzzy rules are optimized to ensure that each rule has better representativeness and prediction accuracy. The membership degree of each feature in the frequent item in the fuzzy set is calculated. , denoted as:

[0102]

[0103] Wherein, represents the membership degree of the feature in the fuzzy set, represents the th feature in the frequent item;

[0104] The features in the frequent item are weighted, and the minimum value method is used to calculate the joint fuzzy membership degree of each frequent item under different fuzzy rules. The joint fuzzy membership degree represents the matching degree of the features in the frequent item under different fuzzy rules. The fuzzy rule represents the mapping relationship between the fuzzy set of input features and the output category or value. The output category means that in fuzzy reasoning, each fuzzy rule has a predefined output value, and this output value represents the category or value that should be returned when the rule is activated. In fuzzy reasoning, rules are usually described in the form of if-then. For example, if the task execution time exceeds the planned time, then the task status is delayed. In this fuzzy rule, the output category is that the task status is delayed. The joint fuzzy membership degree is denoted as:

[0105]

[0106] Wherein, represents the joint fuzzy membership degree of the frequent item , represents the feature weight, represents the number of features in the frequent item;

[0107] The membership degree of each feature affects the calculation of the joint membership degree through the weight of the feature, reflecting that some features have a greater impact on the matching degree of the frequent item, and thus higher weights are assigned;

[0108] The activation degree of each fuzzy rule is calculated using the product method based on the combined fuzzy membership degree. The closer the activation degree value is to 1, the better the rule matches in the current situation and the greater its contribution to the final prediction result. Conversely, when the activation degree value is small and close to 0, it indicates that the matching degree of the rule is low and its impact on the prediction result is small. Based on the activation degree of each fuzzy rule and the predefined output category of each fuzzy rule, the total support degree of all categories is calculated, which is expressed as:

[0109]

[0110] Among them, represents the total support degree of category , that is, the cumulative support degree of this category in all rules, represents the activation degree of fuzzy rule , represents fuzzy rule corresponding output category;

[0111] The support degrees of all categories are normalized and calculated, and the confidence degrees of all categories are output, which are expressed as:

[0112]

[0113] Among them, represents the confidence degree of category , indicating the relative credibility of this category among all possible categories. The normalized value ranges between [0,1], represents the sum of the total support degrees of all categories;

[0114] The confidence degree represents the credibility of this category and reflects the support degree of the fuzzy rule for this category during the reasoning process. The category with the maximum confidence degree is selected as the final prediction category, and the final prediction category is converted into a binary form using one-hot encoding and added as a new feature to the frequent item to form a new frequent item. This frequent item contains the fuzzified feature values and the prediction category, and the updated frequent item is input into the trained ridge regression model to output the regression prediction value , represents the regression estimated value of the target variable;

[0115] Based on the task-related data, the expected task completion time and task priority parameters are defined, and the expected task completion time and task priority parameters are adjusted through the regression prediction value. The adjustment of the expected task completion time is expressed as:

[0116]

[0117] Among them, represents the corrected task completion time, Indicates the historical task completion time, obtained from the historical task scheduling scheme, Indicates the weight;

[0118] The task priority adjustment is expressed as:

[0119]

[0120] Among them, Indicates the corrected task priority, Indicates the historical task priority, obtained from the historical task scheduling scheme, Indicates the weight;

[0121] Take the corrected task completion time and task priority as the task decision goals.

[0122] The present invention defines a fuzzy set through a triangular membership function, enabling task characteristics (such as task success rate) to be smoothly mapped to different fuzzy categories, rather than using a hard classification method, improving the flexibility and adaptability of the fuzzy set. By calculating the similarity between fuzzy rules through the FURIA algorithm, redundant rules are removed, and the fuzzy extension mechanism of FURIA is used to adjust the rule boundaries, improving the representativeness of the rules;

[0123] By using the joint fuzzy membership degree to measure the overall matching degree of multiple features, ensuring the rationality of the predicted category, screening the most relevant fuzzy rules, reducing the influence of irrelevant rules on the predicted category, and improving the reasoning efficiency. Using the product method to calculate the activation degree can more accurately reflect the applicability of the rules and ensure the accuracy of the reasoning results;

[0124] Through One-hot encoding, the predicted category can be converted into a numerical feature, enabling it to be better used in the ridge regression model, improving the final prediction accuracy. By adjusting the task completion time according to the regression prediction value, ensuring more accurate task scheduling, and the task priority adjustment mechanism enables the system to dynamically adapt to changes in the importance of tasks, improving the scheduling efficiency.

[0125] Furthermore, generating task decision data based on the task decision goal means that based on the task decision goal, parsing out the specific parameters in the task decision goal as the optimal parameters, comparing the operating parameters of the current decision scheme to calculate the deviation values between the parameters, through a machine learning model, substituting the deviation values for simulation prediction, and adjusting the deviation values through the reinforcement learning method, outputting the final optimized parameters for each item, and using the final optimized parameters for each item as task decision data in the current task decision scheme.

[0126] By clarifying the parameters in the task decision-making objective, the present invention effectively clarifies the potential objectives of task scheduling, reduces ambiguity, and through simulation and prediction, realizes flexible adjustment of task scheduling and resource allocation. By generating optimized parameters, the system can dynamically adjust according to the actual task requirements and resource status, reducing manual intervention and improving the automation and intelligence level of decision-making.

[0127] S5. Transmit the original data and task decision data to the database for storage and backup;

[0128] Specifically, based on the distributed data storage technology and end-to-end encryption transmission mechanism, data encryption during transmission is realized through the TLS protocol, the AES-256 algorithm is used to encrypt the storage of the original data and task decision data, the blockchain technology is combined to record the data storage and backup operation logs, and the distributed storage system HDFS is used to improve the scalability and reliability of data storage. At the same time, the off-site backup mechanism is enabled to perform multi-node storage and verification of data regularly.

[0129] By combining the distributed data storage technology, end-to-end encryption transmission mechanism and blockchain technology, the present invention significantly improves the data security, storage reliability and system scalability.

[0130] This embodiment also provides a big data unified analysis and processing system based on cloud computing, including:

[0131] The data collection and storage module is responsible for collecting the original data and using the blockchain technology to store and backup the original data and task decision data;

[0132] The data source optimization and data processing module screens multiple data sources and preprocesses the original data;

[0133] The feature extraction and feature dataset generation module extracts features from the preprocessed data and generates a feature dataset;

[0134] The feature importance scoring and frequent item set generation module calculates the importance score of each feature based on the feature dataset and generates frequent item sets based on the feature dataset;

[0135] The joint prediction and task decision generation module performs joint prediction by combining fuzzy inference and adaptive regression and generates the final task decision data based on the prediction results.

[0136] This embodiment also provides a computer device applicable to the situation of the big data unified analysis and processing method based on cloud computing, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the big data unified analysis and processing method based on cloud computing as proposed in the above embodiment.

[0137] The computer device may be a terminal, which includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a carrier network, NFC (Near Field Communication), or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or may be a button, a trackball, or a touchpad provided on the casing of the computer device, or may also be an external keyboard, touchpad, or mouse, etc.

[0138] This embodiment also provides a storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the method for unified analysis and processing of big data based on cloud computing as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc.

[0139] In summary, the present invention calculates the mutual information and gray correlation degree to ensure that the selection of data sources takes into account both the prediction ability of data and the similarity of data structures. By normalizing at the input of the Transformer layer instead of after the traditional residual connection, the gradient stability is enhanced and the training efficiency is improved.

[0140] By using GIST to calculate the local structure information of the image, constructing a GCN based on the spatial block structure calculated by GIST, extracting more spatially consistent local features, calculating the weights of CNN and GCN features through MLP, and optimizing the weight allocation through backpropagation, the rationality of feature fusion is improved, ensuring that the fusion of CNN and GCN features is adaptively adjusted according to the data distribution.

[0141] By calculating the information gain and information entropy, and combining the feature weights to calculate the feature importance score, it is ensured that the selected features not only have predictive value but also avoid the interference of redundant features, improving the data quality and calculation efficiency. By combining the DBSCAN and One-Class SVM double screening mechanisms, it is ensured that the final new feature dataset retains as many predictive value features as possible, further improving the data representation ability. By combining the confidence and lift to screen the association rules, the feature relationships hidden in the data can be mined more accurately.

[0142] Through the construction of the fuzzy set, not only can the generalization ability of the model be improved, but also the uncertainty can be effectively processed, avoiding the overconstraint of the features by hard rules. By jointly calculating the fuzzy membership degree, the understanding and application of the feature relationships under multiple fuzzy rules can be improved, avoiding the deviation under a single fuzzy rule, and enhancing the adaptability of the system to complex task scheduling scenarios. By calculating the activation degree, it is ensured that the contributions of different fuzzy rules to the final prediction result are reasonably evaluated. Using the weighted activation degree for prediction can effectively avoid the excessive influence of certain rules on the result. By combining the ridge regression model and the weighted prediction category, the advantages of fuzzy inference and regression model can be combined to improve the stability and accuracy of prediction.

[0143] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A unified big data analysis and processing method based on cloud computing, characterized in that: including, calculating the preferred score of each data source, selecting the data source with the highest preferred score to collect the original data and preprocess it; extracting global and local features respectively through a model combination method based on the preprocessed data to form a feature dataset; calculating the feature importance score based on the feature dataset to generate a new feature dataset, removing the abnormal features in the new feature dataset and generating frequent item sets; conducting joint prediction based on the frequent item sets by combining fuzzy inference and adaptive regression to generate a task decision target, and generating task decision data based on the task decision target; transmitting the original data and the task decision data to the database for storage and backup.

2. The big data unified analysis and processing method based on cloud computing according to claim 1, characterized in that: Calculating the preferred score of each data source, selecting the data source with the highest preferred score for collecting raw data and preprocessing it means calculating the preferred score of each data source, selecting the data source with the highest preferred score for collecting raw data and preprocessing it, which means collecting image data and text data from databases, API interfaces, and Kafka data streams through the cloud service tool AWS Glue, and defining original variables according to task-related data Including task ID, task priority, task planned completion time, task start time, and task end time, calculating the task execution time based on the task start time and end time, calculating the task success rate and on-time completion rate in combination with the task planned completion time, and using the calculated task success rate and on-time completion rate as the intermediate variables of the task success rate and the intermediate variable of the task completion rate. According to The intermediate variables, respectively define the target variables of the highest task success rate and on-time completion rate; Calculating the Pearson correlation coefficient and the grey relational degree between the intermediate variable and the target variable, using z-score normalization to map the values of the Pearson correlation coefficient and the grey relational degree to the interval [0, 1], assigning weights to the Pearson correlation coefficient and the grey relational degree, and calculating the preferred score of each data source through weighted fusion, and selecting the data source with the highest score as the final data source for collecting the original data; The preprocessing refers to cleaning and standardizing the finally collected original data.

3. The big data unified analysis and processing method based on cloud computing according to claim 2, characterized in that: The step of extracting global and local features respectively through a model combination method based on the preprocessed data to form a feature dataset means constructing a vocabulary for the text data in the preprocessed data using the BoW model, converting all terms in the vocabulary into a term frequency vector, assigning a fixed initial weight to each term in the vocabulary by combining HybridIDF (Hybrid Inverse Document Frequency), generating self-attention weights through the Transformer model, and multiplying the initial weight of each term by the self-attention weight to generate the final weight; Adopting the Pre-LayerNorm structure of pre-normalization in the Transformer model, after combining the term frequency vector with the position encoding, inputting it into the pre-normalized Transformer model in combination with the final weight, and generating global features after being processed by multiple layers of self-attention and feed-forward neural networks; Performing illumination correction on the image data using the Retinex algorithm, constructing a CNN model, and extracting CNN features from the image data; Using the Generalized Image Statistical Feature (GIST) to divide each image in the image data into fixed 8×8 grid blocks, calculating the mean of the gradient responses of each grid block using 4 directions × 4 scales of Gabor filters to obtain the local feature vector of each grid block, splicing the local feature vectors of all grid blocks as the final GIST descriptor, constructing a Graph Convolutional Network (GCN) based on the spatial block structure of each image represented by the GIST descriptor, and extracting GCN features from the image data using the constructed GCN; Splicing the CNN features and the GCN features into an image feature vector, respectively setting the initial weights of the CNN features and the GCN features, inputting them into a Multi-Layer Perceptron (MLP), calculating the loss gradients of the initial weights of the CNN and GCN features through the backpropagation algorithm, updating the initial weights based on the loss gradients, and outputting local features after weighted fusion according to the image feature vector and the updated initial weights; Combining the global features and the local features to generate a feature dataset.

4. The big data unified analysis and processing method based on cloud computing according to claim 3, characterized in that: Calculating the feature importance score based on the feature dataset to generate a new feature dataset, removing the abnormal features in the new feature dataset and generating frequent item sets means calculating the information gain and information entropy of each feature in the feature dataset, taking the information gain as the weight of each feature, and calculating the feature importance score based on the weight of each feature and the corresponding information entropy; Sorting all the feature importance scores, setting a score threshold according to the distribution characteristics of the feature information gain, and retaining the features whose feature importance scores meet the score threshold to form a new feature dataset; Using the DBSCAN clustering algorithm to calculate the number of features around each feature in the new feature dataset as the number of neighboring points, setting the minimum number of neighboring points. When the number of neighboring points of a feature is greater than or equal to the minimum number of neighboring points, taking this feature as a core point, taking the neighboring features of the core point as boundary points, identifying the features that do not belong to the neighborhood of the core point, marking them as abnormal points and removing them; Performing normalization processing on the new feature dataset processed by the DBSCAN clustering algorithm, using One-Class SVM to map the features in the new feature dataset to a high-dimensional space through the radial basis function RBF kernel, and solving the optimal normal vector, bias term and slack variables by minimizing the objective function; Using One-Class SVM to identify and remove abnormal features in the new feature dataset based on the optimal normal vector, bias term and slack variables; Converting the features in the new feature dataset processed by One-Class SVM into binary vectors to form a list of candidate items, using the Apriori algorithm to calculate the support degree of each candidate item, taking the candidate items whose support degree is greater than or equal to the set threshold as frequent items, and generating new candidate items for each frequent item according to the k-item set expansion principle, calculating the support degree of the new candidate items again and comparing it with the set threshold, and performing multiple iterative operations until no new frequent items are generated; Generating association rules based on frequent items, calculating the confidence and Lift value of each association rule, retaining the frequent items that simultaneously meet the condition that the confidence is greater than the set threshold and the Lift is positively correlated, and combining the frequent items that meet the conditions into frequent item sets.

5. The method for unified analysis and processing of big data based on cloud computing according to claim 4, characterized in that: The joint prediction based on the frequent item set combined with fuzzy inference and adaptive regression and generating the task decision target means analyzing the statistical distribution of the frequent item set, constructing a fuzzy set using the triangular membership function, dividing each frequent item into different fuzzy sets, using the FURIA algorithm to calculate the similarity between fuzzy rules to remove redundant rules in the fuzzy set, using the fuzzy extension mechanism of FURIA to adjust the boundary of the fuzzy rules, and calculating the membership degree of each feature in the frequent item in the fuzzy set; Weight the features in the frequent items. Use the minimum value method to calculate the joint fuzzy membership degree of each frequent item under different fuzzy rules. Based on the joint fuzzy membership degree, use the product method to calculate the activation degree of each fuzzy rule. Based on the activation degree of each fuzzy rule and the predefined output categories of each fuzzy rule, calculate the support degrees of all categories. Normalize the support degrees of all categories to calculate and output the confidence degrees of all categories. Select the prediction category with the highest confidence degree as the final prediction category, and use the final prediction category One-hot encoding is used to convert it into binary form and add it as a new feature to the frequent items to form new frequent items Input the updated frequent items into the trained ridge regression model and output the regression prediction values; Defining the task expected completion time and task priority parameters based on the task-related data, and adjusting the task expected completion time and task priority parameters through the regression prediction value; Taking the corrected task completion time and task priority as the task decision target.

6. The big data unified analysis and processing method based on cloud computing according to claim 5, characterized in that: The generation of task decision data based on task decision objectives means that based on task decision objectives, specific parameters in the task decision objectives are parsed as optimal parameters, the deviation values between various parameters are calculated by comparing the operating parameters of the current decision-making scheme, and through a machine learning model, the deviation values are substituted for simulation prediction, and the deviation values are adjusted by the reinforcement learning method, and the final optimized parameters of each item are output, and the final optimized parameters of each item are used as task decision data in the current task decision-making scheme.

7. The big data unified analysis and processing method based on cloud computing according to claim 6, characterized in that: The transmission of the original data and task decision data to the database for storage and backup means that based on the distributed data storage technology and the end-to-end encryption transmission mechanism, data encryption during the transmission process is realized through the TLS protocol, the original data and task decision data are stored and encrypted by using the AES-256 algorithm, the blockchain technology is combined to record the data storage and backup operation logs, and the data storage scalability and reliability are improved through the distributed storage system HDFS. At the same time, the off-site backup mechanism is enabled to perform multi-node storage and verification of the data regularly.

8. A big data unified analysis and processing system based on cloud computing, based on the big data unified analysis and processing method based on cloud computing according to any one of claims 1 to 7, characterized in that: including, The data collection and storage module is responsible for collecting the original data and using the blockchain technology to store and backup the original data and task decision data; The data source optimization and data processing module screens multiple data sources and preprocesses the original data; The feature extraction and feature dataset generation module extracts features from the preprocessed data and generates a feature dataset; The feature importance scoring and frequent item set generation module calculates the importance score of each feature based on the feature dataset and generates frequent item sets based on the feature dataset; The joint prediction and task decision generation module performs joint prediction by combining fuzzy inference and adaptive regression and generates final task decision data based on the prediction results.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that: When the processor executes the computer program, the steps of the big data unified analysis and processing method based on cloud computing according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of the big data unified analysis and processing method based on cloud computing according to any one of claims 1 to 7 are implemented.

Images