Data analysis service recommendation method and system for constructing data analysis process

Abstract

The invention relates to the technical field of data analysis, in particular to a data analysis service recommendation method and system for constructing a data analysis process, and the method comprises the steps: receiving target and data features inputted by a user; constructing a service use relation graph with weights based on a historical process; extracting a context feature vector; based on the relation graph and the context, generating a plurality of candidate service combination sequences through an improved multi-objective optimization algorithm; scoring the candidate sequences by using a pre-training model; outputting a high-score recommendation sequence and supporting interactive adjustment; dynamically updating the relation graph and the model according to user feedback. The system correspondingly comprises seven functional modules. According to the method, process-level intelligent recommendation which learns from history, combines context sensing, balances multi-target performance and can be evolved continuously is realized, the construction difficulty of a data analysis process is effectively reduced, and the analysis efficiency and the result quality are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, and in particular to a data analysis service recommendation method and system for constructing a data analysis process. BACKGROUND

[0002] With the development of big data technology, the construction of data analysis process becomes increasingly complex. Generally, a complete data analysis process involves multiple steps such as data cleaning, feature extraction, model training, and result visualization. Each step may have multiple available data analysis services. For non-professional users or users with expertise in a specific field but without technical background, it is a challenge to select appropriate services from a large number of services and combine them into an effective process.

[0003] In the prior art, there are some service recommendation methods, such as service recommendation based on collaborative filtering and recommendation based on content similarity. However, these methods mostly focus on recommending a single service and lack consideration of end-to-end process combination. The method based on fixed templates has poor flexibility and is difficult to adapt to diverse data and task requirements. A few studies involving service combination optimization only consider a single objective, such as execution time, and fail to balance efficiency, resources, and accuracy. Moreover, these existing solutions generally lack a closed-loop learning mechanism that can evolve based on actual user feedback, making it difficult to continuously optimize the recommendation effect.

[0004] Therefore, there is a need for a data analysis service recommendation method and system that can consider service compatibility, context features, and multi-objective optimization to reduce the process construction threshold and improve analysis efficiency and result quality. SUMMARY

[0005] The present application aims to solve the problem of data analysis service recommendation in the prior art, which only focuses on single-point services and ignores process combination and context awareness. The present application provides a data analysis service recommendation method and system for constructing a data analysis process.

[0006] A data analysis service recommendation method for constructing a data analysis process includes the following steps: Step S1: receiving a user-input data analysis target and original data features; Step S2: based on historical data analysis process data, constructing a service usage relationship graph, wherein nodes represent data analysis services and edges represent the order of service usage and weights between services; Step S3: extracting a context feature vector of the current task according to the original data features and user target; Step S4: Based on the service usage relationship graph and context feature vector, generate multiple candidate data analysis service combination sequences that meet preset constraints using a multi-objective optimization algorithm; Step S5: Use a pre-trained scoring model to perform a comprehensive score on each candidate service combination sequence; Step S6: Output the highest-scoring candidate service combination sequence as the recommendation result, and support interactive user adjustment; Step S7: Based on user feedback on the recommendation results, dynamically update the edge weights in the service usage relationship graph, and add the final service combination sequence adopted by the user to the historical process data for incremental learning of the subsequent model.

[0007] Preferably, in step S2, the construction of the service relationship graph includes: Extract the input and output data patterns of each data analysis service from historical process logs; Establish connectable relationships between services based on data pattern compatibility; Edge weights are calculated based on the frequency of service co-occurrence and the degree of sequential closeness in the historical process. Among them, edge weight The calculation formula is: , in For service To Service edge weights, Here are the weighting coefficients, and F represents the service... Follow The frequency of service A and service B is S, where S is the order of service A and service B. , For the degree of order tightness .

[0008] Preferably, in step S4, the multi-objective optimization algorithm is an improved algorithm based on NSGA-II, and its optimization objectives include at least minimizing process execution time, minimizing computational resource consumption, and maximizing the accuracy of prediction results; The improvement includes employing a fitness function based on context feature vectors, wherein the fitness function is: , in, , , These are the estimated execution time, resource consumption, and result accuracy of the candidate sequences, respectively. , , This corresponds to the historical maximum value. , , These are the weighting coefficients, and their sum is 1.

[0009] Preferably, in step S5, the scoring model is a model trained based on gradient boosting decision trees. Its input features include the context feature vector, the topological features of the service combination sequence, and the service category distribution features. The output is a comprehensive score. , in, , , These are the scores for normalized execution efficiency, resource consumption, and result accuracy, respectively. , , These are the weighting coefficients, and their sum is 1.

[0010] Preferably, in step S7, the formula for dynamically updating edge weights is: , in, For the updated weights, The weights before the update are γ, and the update coefficient is γ. The feedback coefficient is based on user feedback behavior. This is the preset minimum weight value.

[0011] Preferably, in step S1, the data analysis objective includes at least one of classification, regression, clustering, and association rule mining; The original data characteristics include at least three of the following: data volume, feature dimension, proportion of missing values, data type distribution, and data sparsity.

[0012] Preferably, in step S6, the interactive adjustment includes adding, deleting, replacing, and modifying parameters of service nodes in the recommended service combination sequence, and the system verifies the input-output compatibility between services when performing the replacement operation.

[0013] A data analysis service recommendation system for building data analysis processes includes: The input receiving module is used to receive the data analysis objectives and raw data characteristics input by the user; The historical modeling module is used to build a service usage relationship diagram based on historical data analysis process data; The context extraction module is used to extract the context feature vector of the current task; The combination generation module is used to generate multiple candidate service combination sequences based on the service usage relationship graph and context feature vectors through a multi-objective optimization algorithm. The scoring module is used to score candidate sequences using a pre-trained scoring model. The recommended output module outputs at least one candidate sequence with the highest score and supports user interaction for adjustment. The feedback learning module is used to dynamically update the service usage relationship diagram and scoring model based on user feedback.

[0014] Preferably, the historical modeling module includes: Pattern extraction unit is used to analyze historical service input and output data patterns; Relationship building unit, used to establish directed connections between services based on data schema compatibility; The weight calculation unit is used to calculate and update the weights of edges based on historical co-occurrence frequency and order tightness.

[0015] Preferably, the feedback learning module includes: The feedback collection unit is used to record users' actions regarding the adoption, modification, or rejection of the recommendation results; The weight update unit is used to update the weights based on feedback behavior and a formula. Update the service using the edge weights in the relationship graph; The model increment unit is used to add the final service combination sequence and execution data adopted by the user to the training set for incremental training of the scoring model.

[0016] The technical effects and advantages of this invention are as follows: 1. Process-level intelligent recommendation: Directly outputs complete and executable service sequences, solving the problem that single-service recommendations cannot guarantee combination compatibility and overall performance.

[0017] 2. Deep Context Awareness: By co-encoding the analysis target and data features, the recommendation results are highly adapted to the current task, thus improving the accuracy of the recommendation.

[0018] 3. Multidimensional performance balance: Through an improved multi-objective optimization algorithm, it simultaneously balances execution efficiency, resource consumption, and result accuracy, providing a set of Pareto optimal solutions for users to choose from.

[0019] 4. Multidimensional performance balancing: Introducing a feedback-based dynamic update mechanism enables the system to learn from users' actual usage behavior, continuously optimize recommendation strategies, and achieve personalized recommendations.

[0020] 5. User-friendly interface: Provides visual process displays and intuitive adjustment tools, greatly reducing the difficulty of operation for non-professional users. Attached Figure Description

[0021] Fig. 1 This is a flowchart illustrating a data analysis service recommendation method for constructing a data analysis workflow, provided in an embodiment of this application. Fig. 2 This is a structural block diagram of a data analysis system for constructing a data analysis process, provided in an embodiment of this application. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and design various embodiments with various modifications suitable for a particular purpose. Example 1

[0023] Please see Figs. 1-2 This embodiment provides a data analysis service recommendation method for constructing a data analysis process; including the following steps: Step S1: Receive the user's input data analysis objective and raw data features; the data analysis objective includes at least one of classification, regression, clustering, and association rule mining; the raw data features include at least three of data volume, feature dimension, missing value ratio, data type distribution, and data sparsity; Step S2: Based on historical data analysis process data, construct a service usage relationship graph, where nodes represent data analysis services and edges represent the sequential usage relationship and weight between services; In step S2, the construction of the service relationship graph includes: Extract the input and output data patterns of each data analysis service from historical process logs; Establish connectable relationships between services based on data pattern compatibility; Edge weights are calculated based on the frequency of service co-occurrence and the degree of sequential closeness in the historical process. Among them, edge weight The calculation formula is: , in For service To Service edge weights, Here are the weighting coefficients, and F represents the service... Follow The frequency of service A and service B is S, where S is the order of service A and service B. , For the degree of order tightness ; Step S3: Extract the context feature vector of the current task based on the original data features and the user's goal; Step S4: Based on the service usage relationship graph and context feature vector, generate multiple candidate data analysis service combination sequences that meet preset constraints using a multi-objective optimization algorithm; The multi-objective optimization algorithm is an improved algorithm based on NSGA-II, and its optimization objectives include at least minimizing process execution time, minimizing computational resource consumption, and maximizing the accuracy of prediction results. The improvement includes employing a fitness function based on context feature vectors, wherein the fitness function is: , in, , , These are the estimated execution time, resource consumption, and result accuracy of the candidate sequences, respectively. , , This corresponds to the historical maximum value. , , These are the weighting coefficients, and their sum is 1; Step S5: Utilize a pre-trained scoring model to comprehensively score each candidate service combination sequence; the scoring model is a gradient boosting decision tree-trained model, whose input features include the context feature vector, the topological features of the service combination sequence, and the service category distribution features, and whose output is a comprehensive score. , in, , , These are the scores for normalized execution efficiency, resource consumption, and result accuracy, respectively. , , These are the weighting coefficients, and their sum is 1.

[0024] Preferably, in step S7, the formula for dynamically updating edge weights is: , in, For the updated weights, The weights before the update are γ, and the update coefficient is γ. The feedback coefficient is based on user feedback behavior. This is the preset minimum weight value; Step S6: Output the at least one candidate service combination sequence with the highest score as the recommendation result, and support interactive adjustment by the user; in step S6, the interactive adjustment includes adding, deleting, replacing and modifying parameters of service nodes in the recommended service combination sequence, and the system verifies the input and output compatibility between services when performing replacement operations; Step S7: Based on user feedback on the recommendation results, dynamically update the edge weights in the service usage relationship graph, and add the final service combination sequence adopted by the user to the historical process data for incremental learning of the subsequent model. Example 2

[0025] A data analysis service recommendation system for building data analysis processes includes: The input receiving module is used to receive the data analysis objectives and raw data characteristics input by the user; The historical modeling module is used to construct a service usage relationship diagram based on historical data analysis process data; the historical modeling module includes: Pattern extraction unit is used to analyze historical service input and output data patterns; Relationship building unit, used to establish directed connections between services based on data schema compatibility; The weight calculation unit is used to calculate and update the weights of edges based on historical co-occurrence frequency and order density. The context extraction module is used to extract the context feature vector of the current task; The combination generation module is used to generate multiple candidate service combination sequences based on the service usage relationship graph and context feature vectors through a multi-objective optimization algorithm. The scoring module is used to score candidate sequences using a pre-trained scoring model. The recommended output module outputs at least one candidate sequence with the highest score and supports user interaction for adjustment. The feedback learning module is used to dynamically update the service usage relationship graph and rating model based on user feedback; the feedback learning module includes: The feedback collection unit is used to record users' actions regarding the adoption, modification, or rejection of the recommendation results; The weight update unit is used to update the weights based on feedback behavior and a formula. Update the service using the edge weights in the relationship graph; The model increment unit is used to add the final service combination sequence and execution data adopted by the user to the training set for incremental training of the scoring model.

[0026] When using this invention: Example: Process for non-professional users to build a customer classification model A salesperson needs to categorize and analyze a set of customer data to predict the risk of customer churn.

[0027] Step S101: Receive user input.

[0028] Users input through the system form: Data analysis objective: "Classification".

[0029] Original data characteristics: 100,000 data entries, 50 feature dimensions, 10% missing values, numeric data type, and 0.8 data sparsity.

[0030] The system validates the input. If the data volume is a positive integer and the proportion of missing values ​​is between 0 and 1, the system will proceed to the next step after the validation is passed.

[0031] Step S102: Construct a service usage relationship graph.

[0032] The historical modeling module parses historical tasks from the process library; for example, it discovers that the "missing value imputation - mean" service is often followed by the "standardization" service, and calculates the weight of this edge: Frequency F: "Missing value imputation - mean" appeared 200 times, of which "standardization" appeared 100 times. F = 100 / 200 = 0.5.

[0033] Density S: In the historical process, these two services are usually adjacent to each other with no other services in between, S=1 / (0+1)=1.

[0034] If we take α=0.5, then the weight W=0.5×0.5 + 0.5×1 = 0.75.

[0035] Similarly, a directed weighted graph is constructed, where nodes represent various services and the numbers on the edges represent weights.

[0036] Step S103: Extract the context feature vector.

[0037] The context extraction module works as follows: Target encoding: The "classification" target is encoded using One-Hot encoding. Assuming that it supports four categories: [classification, regression, clustering, association], the resulting vector is [1, 0, 0, 0].

[0038] Feature normalization: Based on the historical data range, the original features are normalized to the [0,1] interval. For example, if the data volume range is [10,000, 1,000,000], then 100,000 normalized features are (10-1) / (1,000-1)≈0.091.

[0039] Vector concatenation: The encoded target vector is concatenated with the normalized feature vector to obtain the final context feature vector, for example [1, 0, 0, 0, 0.091, 0.444, 0.2, 0.8].

[0040] Step S104: Multi-objective optimization to generate candidate sequences.

[0041] The combined generation module searches the space, setting constraints such as sequence length 3-8, starting with the preprocessing service and ending with the evaluation service, using an improved NSGA-II algorithm: Fitness calculation: using the function defined in claim 3, assuming a historical baseline =100s, =10GB, =0.95. For a candidate sequence, if its estimated T=30s, R=2GB, A=0.92, and we take β1=β2=β3=1 / 3, then its fitness = (1-30 / 100) / 3 + (1-2 / 10) / 3 + (0.92 / 0.95) / 3 ≈ 0.233 +0.267 + 0.323 = 0.823.

[0042] Through iterative search using specific crossover and mutation operations, a set of Pareto optimal solutions is eventually obtained, for example: Sequence 1: Missing value imputation → Standardization → Feature selection → Random forest → Accuracy calculation; Sequence 2: Missing value imputation → Standardization → PCA dimensionality reduction → Logistic regression → Confusion matrix.

[0043] Step S105: Scoring by the scoring model.

[0044] The scoring module loads the trained gradient boosting decision tree model, extracts the features of each candidate sequence (such as context features, sequence length, service type ratio, etc.) and inputs them into the model. The model outputs a comprehensive score (0-100 points) based on the learned pattern.

[0045] For example, Sequence 1 scored 92 and Sequence 2 scored 85.

[0046] Step S106: Output and interactive recommendations.

[0047] The recommended output module presents the highest-scoring sequence, such as sequence 1, to the user in the form of a flowchart. If the user thinks that the parameters of "random forest" need to be adjusted, they can adjust the "number of trees" from 100 to 150 through the interface slider and replace the "feature selection" method from "variance threshold" to "mutual information". The system updates the process after verifying compatibility in real time.

[0048] Step S107: Feedback Learning.

[0049] The user executes the adjusted process, and the feedback learning module records this action - then adopts and modifies it: Update edge weights: For the edge “Standardization → Feature Selection “Mutual Information Method””, the original weight W=0.6, set γ=0.2, and the feedback coefficient F=0.5 (modified). =0.001, then the new weight W' = max(0.001, 0.6×(1+0.2×0.5)) = 0.66.

[0050] Incremental learning: Add the process and execution results to the historical dataset to trigger incremental training of the scoring model, making it more accurate in recommending similar tasks in the future.

[0051] Each module of the system implementing the above method can be implemented by a general-purpose computing device (server, computer) executing a specific program. The functions of each module are as described in claims 8-10, and they work together to complete the entire process from input, calculation, recommendation to learning.

[0052] Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art and related fields based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described and explained in the present invention, unless otherwise specified or limited, shall be implemented according to conventional means in the art.

Claims

1. A data analysis service recommendation method for constructing a data analysis workflow, characterized in that, Includes the following steps: Step S1: Receive the user's input of the data analysis target and raw data characteristics; Step S2: Based on historical data analysis process data, construct a service usage relationship graph, where nodes represent data analysis services and edges represent the sequential usage relationship and weight between services; Step S3: Extract the context feature vector of the current task based on the original data features and the user's goal; Step S4: Based on the service usage relationship graph and context feature vector, generate multiple candidate data analysis service combination sequences that meet preset constraints using a multi-objective optimization algorithm; Step S5: Use a pre-trained scoring model to perform a comprehensive score on each candidate service combination sequence; Step S6: Output the highest-scoring candidate service combination sequence as the recommendation result, and support interactive user adjustment; Step S7: Based on user feedback on the recommendation results, dynamically update the edge weights in the service usage relationship graph, and add the final service combination sequence adopted by the user to the historical process data for incremental learning of the subsequent model.

2. The data analysis service recommendation method for constructing a data analysis workflow according to claim 1, characterized in that, In step S2, the construction of the service relationship graph includes: Extract the input and output data patterns of each data analysis service from historical process logs; Establish connectable relationships between services based on data pattern compatibility; Calculate edge weights based on the frequency of service co-occurrence and the degree of sequential closeness in the historical process; Among them, edge weight The calculation formula is: , in For service To Service edge weights, Here are the weighting coefficients, and F represents the service... Follow The frequency of service A and service B is S, where S is the order of service A and service B. ， For the degree of order tightness 。 3. The data analysis service recommendation method for constructing a data analysis workflow according to claim 1, characterized in that, In step S4, the multi-objective optimization algorithm is an improved algorithm based on NSGA-II, and its optimization objectives include at least minimizing process execution time, minimizing computational resource consumption, and maximizing the accuracy of prediction results. The improvement includes employing a fitness function based on context feature vectors, wherein the fitness function is: , in, , , These are the estimated execution time, resource consumption, and result accuracy of the candidate sequences, respectively. , , This corresponds to the historical maximum value. , , These are the weighting coefficients, and their sum is 1.

4. The data analysis service recommendation method for constructing a data analysis process according to claim 1, characterized in that, In step S5, the scoring model is a model trained based on gradient boosting decision trees. Its input features include the context feature vector, the topological features of the service combination sequence, and the service category distribution features. The output is a comprehensive score. , in, , , These are the scores for normalized execution efficiency, resource consumption, and result accuracy, respectively. , , These are the weighting coefficients, and their sum is 1.

5. The data analysis service recommendation method for constructing a data analysis workflow according to claim 1, characterized in that, In step S7, the formula for dynamically updating the edge weights is: , in, For the updated weights, The weights before the update are γ, and the update coefficient is γ. The feedback coefficient is based on user feedback behavior. This is the preset minimum weight value.

6. The data analysis service recommendation method for constructing a data analysis workflow according to claim 1, characterized in that, In step S1, the data analysis objective includes at least one of classification, regression, clustering, and association rule mining; The original data characteristics include at least three of the following: data volume, feature dimension, proportion of missing values, data type distribution, and data sparsity.

7. The data analysis service recommendation method for constructing a data analysis workflow according to claim 1, characterized in that, In step S6, the interactive adjustment includes adding, deleting, replacing, and modifying parameters of service nodes in the recommended service combination sequence, and the system verifies the input-output compatibility between services when performing the replacement operation.

8. A data analysis service recommendation system for constructing data analysis workflows, characterized in that, include: The input receiving module is used to receive the data analysis objectives and raw data characteristics input by the user; The historical modeling module is used to build a service usage relationship diagram based on historical data analysis process data; The context extraction module is used to extract the context feature vector of the current task; The combination generation module is used to generate multiple candidate service combination sequences based on the service usage relationship graph and context feature vectors through a multi-objective optimization algorithm. The scoring module is used to score candidate sequences using a pre-trained scoring model. The recommended output module outputs at least one candidate sequence with the highest score and supports user interaction for adjustment. The feedback learning module is used to dynamically update the service usage relationship diagram and scoring model based on user feedback.

9. A data analysis service recommendation system for constructing a data analysis process according to claim 8, characterized in that, The historical modeling module includes: Pattern extraction unit is used to analyze historical service input and output data patterns; Relationship building unit, used to establish directed connections between services based on data schema compatibility; The weight calculation unit is used to calculate and update the weights of edges based on historical co-occurrence frequency and order tightness.

10. A data analysis service recommendation system for constructing a data analysis process according to claim 8, characterized in that, The feedback learning module includes: The feedback collection unit is used to record users' actions regarding the adoption, modification, or rejection of the recommendation results; The weight update unit is used to update the weights based on feedback behavior and a formula. Update the service using the edge weights in the relationship graph; The model increment unit is used to add the final service combination sequence and execution data adopted by the user to the training set for incremental training of the scoring model.

Images