Multi-data source plug-in computing resource scheduling method and system supporting algorithm iteration

Abstract

The invention provides a multi-data source plug-in computing resource scheduling method and system supporting algorithm iteration, relates to the technical field of data sources, and comprises the steps of extracting multi-dimensional features by constructing a data source feature analysis engine, generating a feature fingerprint identifier, designing an intelligent matching mechanism based on the feature fingerprint identifier, and optimizing algorithm plug-in selection. The method has the beneficial effects that the utilization efficiency of computing resources is improved, resource allocation is adjusted in real time, stable system performance is ensured, and closed-loop optimization is formed.

Description

Technical Field

[0001] The present invention relates to data source technology, and in particular to a multi-data source plug-in computing resource scheduling method and system supporting algorithm iteration. Background Art

[0002] In the era of big data, the diversity and complexity of data sources make the scheduling of computing resources increasingly important. Existing computing resource scheduling methods usually rely on static algorithm selection and fixed resource allocation strategies, which are difficult to adapt to dynamically changing data characteristics and computing needs. With the continuous increase in data volume, traditional scheduling methods face many challenges when dealing with multiple data sources.

[0003] First, existing technologies often lack a comprehensive analysis of the characteristics of multiple data sources, resulting in the inability to fully consider the structure, timing, and quality characteristics of the data when selecting the algorithm, thus affecting the scheduling effect. Second, the existing algorithm matching mechanism is usually based on static parameters and fails to update and adjust in real time, resulting in a decrease in matching when data characteristics change, affecting the utilization efficiency of computing resources. Finally, the existing dynamic resource allocation strategy lacks intelligence and cannot be effectively adjusted according to the real-time monitoring system performance indicators, resulting in resource waste or performance degradation, and unable to form a closed-loop optimization. Summary of the invention

[0004] The embodiments of the present invention provide a multi-data source plug-in computing resource scheduling method and system that supports algorithm iteration, which can solve the problems in the prior art.

[0005] According to a first aspect of the embodiments of the present invention, Provides a multi-data source plug-in computing resource scheduling method that supports algorithm iteration, including: Construct a data source feature analysis engine, extract features from multiple data sources, extract data structure features to generate a structure feature matrix, extract data time series features to generate a time series feature matrix, and extract data quality features to generate a quality feature matrix; input the structure feature matrix, the time series feature matrix, and the quality feature matrix into a feature fusion model, the feature fusion model uses an adaptive weight mechanism to calculate the importance coefficients of the three types of features, and implements correlation analysis between matrices through a feature interaction network, and finally fuses to generate a feature fingerprint identifier of the data source, the feature fingerprint identifier contains full-dimensional feature information and feature importance weights of the data source; Based on the feature fingerprint identification, an algorithm plug-in intelligent matching mechanism is designed. The algorithm plug-in intelligent matching mechanism first extracts algorithm feature parameters from the algorithm plug-in library and establishes an improved two-way matching scoring model. The improved two-way matching scoring model calculates the matching score between each algorithm plug-in and the data source according to the feature information and weight in the feature fingerprint identification; an algorithm optimization decision tree is constructed. The algorithm optimization decision tree generates a priority-ranked candidate algorithm sequence according to the matching score, algorithm complexity and historical performance records; and a sequence dynamic update mechanism is established. The sequence dynamic update mechanism adjusts the ranking weight of the candidate algorithm sequence in real time according to the change trend of the feature fingerprint identification; The ranking results of the candidate algorithm sequence are used to realize intelligent scheduling of computing resources, select the algorithm plug-in with the highest priority in the candidate algorithm sequence, and build a resource demand prediction model based on the feature fingerprint identification. The resource demand prediction model outputs the resource consumption curve of the algorithm plug-in at different execution stages; design a resource dynamic allocation strategy according to the resource consumption curve, and the resource dynamic allocation strategy monitors system performance indicators in real time, including computing efficiency indicators, resource utilization indicators and task completion quality indicators. When the system performance indicators are monitored to be lower than the preset performance threshold, an optimization response mechanism is triggered; the optimization response mechanism attempts to adjust the resource allocation parameters. If the performance continues to be abnormal after adjustment, an alternative algorithm plug-in is selected from the candidate algorithm sequence for replacement, and the performance optimization result is fed back to the feature fingerprint identification, and the feature weight and algorithm ranking are updated to form a closed-loop optimization.

[0006] Based on the feature fingerprint identification, an algorithm plug-in intelligent matching mechanism is designed. The algorithm plug-in intelligent matching mechanism first extracts algorithm feature parameters from the algorithm plug-in library and establishes an improved two-way matching scoring model. The improved two-way matching scoring model calculates the matching score between each algorithm plug-in and the data source according to the feature information and weight in the feature fingerprint identification, including: The feature fingerprint identification includes data structure features, data timing features and data quality features. The data structure features, data timing features and data quality features are mapped into a feature vector group by a feature vectorization processing unit, and an initial importance weight of each feature is generated based on information entropy calculation; Based on the feature vector group and the initial importance weight, an algorithm feature extraction model is constructed, algorithm feature parameters are extracted from the algorithm plug-in library, time complexity parameters, space complexity parameters and communication complexity parameters are extracted to generate a complexity feature matrix, data processing parameters, calculation accuracy parameters and parallel degree parameters are extracted to generate a functional feature matrix, data sensitivity parameters, resource dependency parameters and expansion capability parameters are extracted to generate an adaptability feature matrix; the complexity feature matrix, the functional feature matrix and the adaptability feature matrix are combined to form an algorithm feature parameter set; Constructing a feature space mapping model, wherein the feature space mapping model converts the feature vector group and the algorithm feature parameter set into a unified feature space, uses an adaptive feature completion unit to process feature missing, uses a nonlinear feature transformation unit to unify feature dimensions, and generates a standardized feature representation matrix; Based on the standardized feature representation matrix, an improved two-way matching scoring model is established; based on the improved two-way matching scoring model, a multi-core similarity calculation network is constructed; for different feature types, the cosine similarity kernel function, the Jaccard coefficient kernel function and the edit distance kernel function are used to calculate the similarity to form a multi-dimensional similarity matrix; using a dynamic weight optimizer, according to the initial importance weight and the historical matching effect, the feature fusion weight coefficient is calculated; the multi-dimensional similarity matrix and the feature fusion weight coefficient are weightedly combined to output the matching score between the algorithm plug-in and the data source.

[0007] Based on the standardized feature representation matrix, an improved two-way matching scoring model is established, and a multi-core similarity calculation network is constructed based on the improved two-way matching scoring model. The cosine similarity kernel function, the Jaccard coefficient kernel function and the edit distance kernel function are used to calculate the similarity for different feature types, and a multi-dimensional similarity matrix is ​​formed, including: Establishing a standardized feature representation matrix, wherein the standardized feature representation matrix includes continuous numerical features, discrete category features, and string type features; Inputting the standardized feature representation matrix into the improved two-way matching scoring model; extracting feature vectors from the standardized feature representation matrix, and calculating the angular similarity between feature vectors using a cosine similarity kernel function for the continuous numerical features, wherein the cosine similarity kernel function divides the inner product of the feature vector by the product of the Euclidean norm of the feature vector to generate a first similarity matrix; Extracting feature sets from the standardized feature representation matrix, and calculating the overlap between feature sets using a Jaccard coefficient kernel function for the discrete category features, wherein the Jaccard coefficient kernel function divides the intersection cardinality of the feature sets by the union cardinality of the feature sets to generate a second similarity matrix; Extracting a string sequence from the standardized feature representation matrix, and using a weighted edit distance kernel function to calculate the edit operation cost between the strings for the string sequence, wherein the weighted edit distance kernel function sets weight coefficients for replacement operation, insertion operation and deletion operation respectively, to generate a third similarity matrix; An adaptive weight optimization model is established, a weight coefficient is determined by minimizing the matching error, and the first similarity matrix, the second similarity matrix and the third similarity matrix are weightedly combined using the weight coefficient to generate a multidimensional similarity matrix.

[0008] Constructing an algorithm optimization decision tree, the algorithm optimization decision tree generates a priority-ranked candidate algorithm sequence based on the matching score, algorithm complexity and historical performance records; at the same time, establishing a sequence dynamic update mechanism, the sequence dynamic update mechanism adjusts the ranking weight of the candidate algorithm sequence in real time according to the change trend of the feature fingerprint identification, including: Collecting algorithm evaluation data, the algorithm evaluation data including matching scores, algorithm complexity and historical performance records, and performing standardization on the algorithm evaluation data; Constructing an algorithm optimization decision tree, wherein the algorithm optimization decision tree adopts a three-layer structure, wherein the first layer sets a preferred weight coefficient according to the matching score, the second layer sets a computing resource weight coefficient according to the algorithm complexity, and the third layer sets a stability weight coefficient according to the historical performance record, and the preferred weight coefficient, the computing resource weight coefficient and the stability weight coefficient are weighted and combined into three-layer weight coefficients, and the algorithm optimization decision tree performs hierarchical evaluation on candidate algorithms through the three-layer weight coefficients; Calculating an evaluation score for each candidate algorithm based on the algorithm optimization decision tree, wherein the evaluation score is a weighted sum of weight coefficients of each layer, and prioritizing the candidate algorithms according to the evaluation score to generate a candidate algorithm sequence; Collect characteristic fingerprint identification data, and calculate the change trend of the characteristic fingerprint identification data within a preset time window, wherein the change trend includes the characteristic value change amplitude and change rate, wherein the characteristic value change amplitude represents the change direction and intensity of the characteristic value, and the change rate represents the change speed of the characteristic value; Establishing a sequence dynamic update mechanism, calculating a ranking weight adjustment coefficient according to the change trend of the characteristic fingerprint identification data, wherein the ranking weight adjustment coefficient is positively correlated with the characteristic value change amplitude and the change rate, and the ranking weight adjustment coefficient is used to adjust the priority weight of each algorithm in the candidate algorithm sequence; The characteristic value change amplitude and the change rate are monitored in real time. When the characteristic value change amplitude exceeds a first preset characteristic threshold or the change rate exceeds a second preset characteristic threshold, a sequence update operation is triggered, and the algorithm priority in the candidate algorithm sequence is dynamically adjusted using the sorting weight adjustment coefficient to generate an updated candidate algorithm sequence.

[0009] Construct an algorithm optimization decision tree, wherein the algorithm optimization decision tree adopts a three-layer structure, wherein the first layer sets the optimization weight coefficient according to the matching score, the second layer sets the computing resource weight coefficient according to the algorithm complexity, and the third layer sets the stability weight coefficient according to the historical performance record, including: Construct an algorithm optimization decision tree, and establish a three-layer structural framework of a matching evaluation layer, a complexity evaluation layer, and a stability evaluation layer based on the algorithm optimization decision tree; collect feature vectors of the data to be processed, wherein the feature vectors include data scale, distribution characteristics, and sparsity; Establish an algorithm feature matching model, input the feature vector into the algorithm feature matching model, calculate the similarity matching degree between the candidate algorithm and the feature vector; multiply the similarity matching degree by a preset feature weight to generate a matching score, and set the matching score as a first-layer optimization weight coefficient in the matching evaluation layer of the algorithm optimization decision tree; The matching score calculation formula is as follows: ; Among them, M ij is the matching score between the i-th feature vector and the j-th candidate algorithm, S ij is the similarity matching degree between the i-th feature vector and the j-th candidate algorithm, W f is the preset feature weight; Extracting the time complexity and space complexity information of the candidate algorithm and constructing a complexity quantification model; inputting the time complexity and space complexity information into the complexity quantification model to generate an evaluation score; calculating the ratio of the evaluation score to the system resource capacity, and setting the ratio as a second-layer computing resource weight coefficient in the complexity evaluation layer of the algorithm optimization decision tree; Collecting historical operation performance records of the candidate algorithm, wherein the historical operation performance records include response delay, throughput and resource occupancy rate; dividing the historical operation performance records into intervals according to time series, and calculating the stability score of the performance indicator in each interval; setting the stability score as the third-layer stability weight coefficient in the stability evaluation layer of the algorithm optimization decision tree; The stability score is calculated as follows: ; Among them, S jis the stability score of the jth algorithm, σ j is the performance index variance of the jth algorithm, δ j is the maximum fluctuation range of the performance index, δ max is the maximum acceptable fluctuation threshold.

[0010] The ranking result of the candidate algorithm sequence is used to realize intelligent scheduling of computing resources, and the algorithm plug-in with the highest priority in the candidate algorithm sequence is selected. A resource demand prediction model is constructed based on the feature fingerprint identification. The resource demand prediction model outputs the resource consumption curve of the algorithm plug-in at different execution stages, including: Obtaining a ranking result of the candidate algorithm sequence, the ranking result including a priority score of each candidate algorithm, and selecting an algorithm plug-in with the highest priority score in the candidate algorithm sequence; Collecting multi-dimensional resource consumption characteristics of the algorithm plug-in, the multi-dimensional resource consumption characteristics include computing density characteristics, data access characteristics and parallelism characteristics, and generating a resource feature vector; performing time series analysis on the feature fingerprint identifier based on the resource feature vector, extracting dynamic change characteristics of the feature fingerprint identifier within a preset time window, the dynamic change characteristics include periodic change rules and mutation characteristics, and generating a fingerprint time series vector; The dynamic fusion mechanism is used to calculate the correlation matrix between the resource feature vector and the fingerprint time series vector, and a feature fusion weight is constructed based on the correlation matrix; the resource feature vector and the fingerprint time series vector are input into the adaptive feature selection module, and the adaptive feature selection module selects prediction features based on the information gain criterion; the prediction features are ranked by importance, and the important features are combined according to the feature fusion weight to form an enhanced feature matrix; Constructing a multi-layer perception prediction model, the multi-layer perception prediction model includes a feature extraction layer, a hierarchical attention layer and a prediction output layer; setting a temporal attention unit and a feature dimension attention unit in the hierarchical attention layer, the temporal attention unit captures the dependency relationship of the time dimension, and the feature dimension attention unit captures the correlation relationship between features; The enhanced feature matrix is ​​input into the multi-layer perception prediction model to generate an initial prediction result; a residual learning structure is established to perform residual compensation on the initial prediction result and the historical prediction deviation to obtain a compensated prediction result; the compensated prediction result is divided according to the execution stage to generate a segmented resource consumption prediction curve.

[0011] The optimization response mechanism attempts to adjust the resource allocation parameters. If the performance continues to be abnormal after the adjustment, an alternative algorithm plug-in is selected from the candidate algorithm sequence for replacement, and the performance optimization result is fed back to the feature fingerprint identifier, and the feature weight and algorithm ranking are updated to form a closed-loop optimization including: The optimization response mechanism includes a resource parameter adjustment module and an algorithm replacement module; collects system operation status data, obtains processor utilization, memory occupancy, response time and throughput, and generates system performance indicators; A dynamic threshold detection model is established based on the system performance indicator, and when the system performance indicator deviates from the dynamic threshold, the resource parameter adjustment module is activated; the resource parameter adjustment module calculates the influence of resource parameters on performance, generates a parameter sensitivity matrix, and determines the optimal parameter adjustment scheme based on the parameter sensitivity matrix; Execute the optimal parameter adjustment scheme, record the adjusted system performance index, and calculate the performance improvement value; when the performance improvement value is lower than the preset improvement threshold for multiple times in a row, determine that the parameter adjustment is invalid, and trigger the algorithm replacement module; the algorithm replacement module selects a set of candidate algorithms from a preset candidate algorithm sequence based on the current system performance index; Calculating the expected performance score for each algorithm in the candidate algorithm set, and selecting the algorithm with the highest expected performance score as the replacement target; performing the algorithm replacement operation, collecting the system performance indicators after the replacement, and generating an optimization result including the parameter adjustment effect and the algorithm replacement effect; Based on the optimization results, the algorithm feature weights are updated, and the candidate algorithm sequence is reordered according to the updated algorithm feature weights; the parameter adjustment plan, algorithm replacement record and performance improvement data in this round of optimization are stored in the optimization history database; based on the optimization history database, the timing of the next performance bottleneck is predicted, and a new round of optimization response is triggered to form a closed-loop optimization.

[0012] According to a second aspect of the embodiments of the present invention, Provides a multi-data source plug-in computing resource scheduling system that supports algorithm iteration, including: The first unit is used to build a data source feature analysis engine, extract features from multiple data sources, extract data structure features to generate a structure feature matrix, extract data time series features to generate a time series feature matrix, and extract data quality features to generate a quality feature matrix; the structure feature matrix, the time series feature matrix, and the quality feature matrix are input into a feature fusion model, the feature fusion model uses an adaptive weight mechanism to calculate the importance coefficients of three types of features, and realizes correlation analysis between matrices through a feature interaction network, and finally fuses to generate a feature fingerprint identifier of the data source, the feature fingerprint identifier contains full-dimensional feature information and feature importance weights of the data source; The second unit is used to design an algorithm plug-in intelligent matching mechanism based on the feature fingerprint identification, wherein the algorithm plug-in intelligent matching mechanism first extracts algorithm feature parameters from the algorithm plug-in library, establishes an improved two-way matching scoring model, and calculates the matching score between each algorithm plug-in and the data source according to the feature information and weight in the feature fingerprint identification; constructs an algorithm optimization decision tree, wherein the algorithm optimization decision tree generates a priority-ranked candidate algorithm sequence according to the matching score, algorithm complexity and historical performance records; and simultaneously establishes a sequence dynamic update mechanism, wherein the sequence dynamic update mechanism adjusts the ranking weight of the candidate algorithm sequence in real time according to the changing trend of the feature fingerprint identification; The third unit is used to use the sorting results of the candidate algorithm sequence to realize intelligent scheduling of computing resources, select the algorithm plug-in with the highest priority in the candidate algorithm sequence, and build a resource demand prediction model based on the feature fingerprint identification. The resource demand prediction model outputs the resource consumption curve of the algorithm plug-in at different execution stages; the resource dynamic allocation strategy is designed according to the resource consumption curve. The resource dynamic allocation strategy monitors system performance indicators in real time, including computing efficiency indicators, resource utilization indicators and task completion quality indicators. When the system performance indicators are monitored to be lower than the preset performance threshold, an optimization response mechanism is triggered; the optimization response mechanism attempts to adjust the resource allocation parameters. If the performance continues to be abnormal after adjustment, an alternative algorithm plug-in is selected from the candidate algorithm sequence for replacement, and the performance optimization result is fed back to the feature fingerprint identification, the feature weight and algorithm sorting are updated, and a closed-loop optimization is formed.

[0013] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0014] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0015] The beneficial effects of this application are as follows: 1. By building a data source feature analysis engine, we can fully extract the structure, timing and quality features of multiple data sources, generate feature fingerprints, and thus achieve in-depth understanding and analysis of data sources. This feature fusion method improves the accuracy and comprehensiveness of data source features, and provides a reliable basis for subsequent algorithm matching and resource scheduling.

[0016] 2. The design of the algorithm plug-in intelligent matching mechanism makes the matching score between the algorithm and the data source more accurate, and can dynamically adjust the priority of the candidate algorithm sequence according to the feature fingerprint. This intelligent matching mechanism improves the efficiency and accuracy of algorithm selection and ensures the optimal use of computing resources.

[0017] 3. By real-time monitoring of system performance indicators and implementing dynamic resource allocation strategies, it is possible to respond to changes in system performance in a timely manner and optimize resource allocation. This closed-loop optimization mechanism not only improves the system's computing efficiency and resource utilization, but also ensures the quality of task completion and enhances the stability and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a multi-data source plug-in computing resource scheduling method supporting algorithm iteration according to an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a multi-data source plug-in computing resource scheduling system that supports algorithm iteration according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0021] Figure 1 FIG. 1 is a flow chart of a multi-data source plug-in computing resource scheduling method supporting algorithm iteration according to an embodiment of the present invention. Figure 1 As shown, the method includes: S11. Build a data source feature analysis engine, extract features from multiple data sources, extract data structure features to generate a structure feature matrix, extract data time series features to generate a time series feature matrix, and extract data quality features to generate a quality feature matrix; input the structure feature matrix, the time series feature matrix, and the quality feature matrix into a feature fusion model, the feature fusion model uses an adaptive weight mechanism to calculate the importance coefficients of the three types of features, and implements correlation analysis between matrices through a feature interaction network, and finally fuses to generate a feature fingerprint identifier of the data source, the feature fingerprint identifier contains full-dimensional feature information and feature importance weights of the data source; S12. Based on the feature fingerprint identification, an algorithm plug-in intelligent matching mechanism is designed. The algorithm plug-in intelligent matching mechanism first extracts algorithm feature parameters from the algorithm plug-in library and establishes an improved two-way matching scoring model. The improved two-way matching scoring model calculates the matching score between each algorithm plug-in and the data source according to the feature information and weight in the feature fingerprint identification; constructs an algorithm optimization decision tree. The algorithm optimization decision tree generates a priority-ranked candidate algorithm sequence according to the matching score, algorithm complexity and historical performance records; and establishes a sequence dynamic update mechanism. The sequence dynamic update mechanism adjusts the ranking weight of the candidate algorithm sequence in real time according to the change trend of the feature fingerprint identification; S13. Utilize the ranking results of the candidate algorithm sequence to realize intelligent scheduling of computing resources, select the algorithm plug-in with the highest priority in the candidate algorithm sequence, and build a resource demand prediction model based on the feature fingerprint identifier. The resource demand prediction model outputs the resource consumption curve of the algorithm plug-in at different execution stages; design a dynamic resource allocation strategy based on the resource consumption curve, and the dynamic resource allocation strategy monitors system performance indicators in real time, including computing efficiency indicators, resource utilization indicators and task completion quality indicators. When the system performance indicators are monitored to be lower than the preset performance threshold, an optimization response mechanism is triggered; the optimization response mechanism attempts to adjust the resource allocation parameters. If the performance continues to be abnormal after adjustment, an alternative algorithm plug-in is selected from the candidate algorithm sequence for replacement, and the performance optimization result is fed back to the feature fingerprint identifier, and the feature weight and algorithm ranking are updated to form a closed-loop optimization.

[0022] In an optional implementation, based on the feature fingerprint identification, an algorithm plug-in intelligent matching mechanism is designed, the algorithm plug-in intelligent matching mechanism first extracts algorithm feature parameters from the algorithm plug-in library, establishes an improved two-way matching scoring model, and the improved two-way matching scoring model calculates the matching score between each algorithm plug-in and the data source according to the feature information and weight in the feature fingerprint identification, including: The feature fingerprint identification includes data structure features, data timing features and data quality features. The data structure features, data timing features and data quality features are mapped into a feature vector group by a feature vectorization processing unit, and an initial importance weight of each feature is generated based on information entropy calculation; Based on the feature vector group and the initial importance weight, an algorithm feature extraction model is constructed, algorithm feature parameters are extracted from the algorithm plug-in library, time complexity parameters, space complexity parameters and communication complexity parameters are extracted to generate a complexity feature matrix, data processing parameters, calculation accuracy parameters and parallel degree parameters are extracted to generate a functional feature matrix, data sensitivity parameters, resource dependency parameters and expansion capability parameters are extracted to generate an adaptability feature matrix; the complexity feature matrix, the functional feature matrix and the adaptability feature matrix are combined to form an algorithm feature parameter set; Constructing a feature space mapping model, wherein the feature space mapping model converts the feature vector group and the algorithm feature parameter set into a unified feature space, uses an adaptive feature completion unit to process feature missing, uses a nonlinear feature transformation unit to unify feature dimensions, and generates a standardized feature representation matrix; Based on the standardized feature representation matrix, an improved two-way matching scoring model is established; based on the improved two-way matching scoring model, a multi-core similarity calculation network is constructed; for different feature types, the cosine similarity kernel function, the Jaccard coefficient kernel function and the edit distance kernel function are used to calculate the similarity to form a multi-dimensional similarity matrix; using a dynamic weight optimizer, according to the initial importance weight and the historical matching effect, the feature fusion weight coefficient is calculated; the multi-dimensional similarity matrix and the feature fusion weight coefficient are weightedly combined to output the matching score between the algorithm plug-in and the data source.

[0023] Based on the feature fingerprint identification, an algorithm plug-in intelligent matching mechanism is designed. First, the algorithm feature parameters are extracted from the algorithm plug-in library, and an improved two-way matching scoring model is established. The model calculates the matching score between each algorithm plug-in and the data source according to the feature information and weight in the feature fingerprint identification.

[0024] Feature fingerprint identification includes data structure features, data timing features and data quality features. The feature vectorization processing unit is used to map the data structure features, data timing features and data quality features into feature vector groups, and the initial importance weight of each feature is generated based on information entropy calculation. The calculation process of information entropy involves analyzing the distribution of features to determine their importance in the overall data.

[0025] Based on the feature vector group and the initial importance weight, an algorithm feature extraction model is constructed. The model extracts algorithm feature parameters from the algorithm plug-in library, extracts time complexity, space complexity and communication complexity parameters, and generates a complexity feature matrix.

[0026] At the same time, data processing parameters, calculation accuracy parameters and parallel degree parameters are extracted to generate a functional feature matrix; data sensitivity parameters, resource dependency parameters and expansion capability parameters are extracted to generate an adaptive feature matrix.

[0027] Finally, the complexity feature matrix, the functional feature matrix and the adaptability feature matrix are combined to form the algorithm feature parameter set. A feature space mapping model is constructed to transform the feature vector group and the algorithm feature parameter set into a unified feature space.

[0028] The adaptive feature completion unit is used to handle feature missing, and the nonlinear feature transformation unit is used to unify feature dimensions and generate a standardized feature representation matrix. The standardization process ensures that different features are compared at the same scale to avoid deviations caused by different feature value ranges. Based on the standardized feature representation matrix, an improved two-way matching scoring model is established.

[0029] The model constructs a multi-core similarity calculation network, and uses cosine similarity, Jaccard coefficient, and edit distance to calculate similarity for different feature types, forming a multi-dimensional similarity matrix. The dynamic weight optimizer calculates the feature fusion weight coefficient based on the initial importance weight and historical matching effect. The multi-dimensional similarity matrix and the feature fusion weight coefficient are weighted and combined to output the matching score between the algorithm plug-in and the data source.

[0030] The solution of this application can: Improve the matching accuracy between algorithm plug-ins and data sources, and ensure that the optimal algorithm is selected to meet specific data processing requirements. Through feature vectorization and standardization, the comparability of features is enhanced, and matching errors caused by feature differences are reduced. The dynamic weight optimization mechanism makes the matching process more flexible, and can continuously adjust the importance of features based on historical data, improving the adaptability and intelligence of the system.

[0031] In an optional implementation, based on the standardized feature representation matrix, an improved two-way matching scoring model is established, and based on the improved two-way matching scoring model, a multi-core similarity calculation network is constructed, and the cosine similarity kernel function, the Jaccard coefficient kernel function, and the edit distance kernel function are used to calculate the similarity for different feature types, respectively, to form a multi-dimensional similarity matrix including: Establishing a standardized feature representation matrix, wherein the standardized feature representation matrix includes continuous numerical features, discrete category features, and string type features; Inputting the standardized feature representation matrix into the improved two-way matching scoring model; extracting feature vectors from the standardized feature representation matrix, and calculating the angular similarity between feature vectors using a cosine similarity kernel function for the continuous numerical features, wherein the cosine similarity kernel function divides the inner product of the feature vector by the product of the Euclidean norm of the feature vector to generate a first similarity matrix; Extracting feature sets from the standardized feature representation matrix, and calculating the overlap between feature sets using a Jaccard coefficient kernel function for the discrete category features, wherein the Jaccard coefficient kernel function divides the intersection cardinality of the feature sets by the union cardinality of the feature sets to generate a second similarity matrix; Extracting a string sequence from the standardized feature representation matrix, and using a weighted edit distance kernel function to calculate the edit operation cost between the strings for the string sequence, wherein the weighted edit distance kernel function sets weight coefficients for replacement operation, insertion operation and deletion operation respectively, to generate a third similarity matrix; An adaptive weight optimization model is established, a weight coefficient is determined by minimizing the matching error, and the first similarity matrix, the second similarity matrix and the third similarity matrix are weightedly combined using the weight coefficient to generate a multidimensional similarity matrix.

[0032] Based on the standardized feature representation matrix, an improved two-way matching scoring model is established, a multi-core similarity calculation network is constructed, and the cosine similarity kernel function, Jaccard coefficient kernel function and edit distance kernel function are used to calculate the similarity for different feature types.

[0033] Establish a standardized feature representation matrix. This matrix contains continuous numerical features, discrete categorical features, and string type features. Continuous numerical features can be the user's age, income, etc., discrete categorical features can be the user's gender, occupation, etc., and string type features can be the user's hobbies, etc. By standardizing these features, ensure that different features are compared on the same scale.

[0034] The standardized feature representation matrix is ​​input into the improved two-way matching scoring model. The model calculates similarity by extracting feature vectors. For continuous numerical features, the cosine similarity kernel function is used to calculate the angular similarity between feature vectors. Specifically, the cosine similarity kernel function calculates the inner product of the feature vector and compares it with the product of the Euclidean norm of the feature vector to generate the first similarity matrix.

[0035] Extract feature sets from the standardized feature representation matrix. For discrete category features, the Jaccard coefficient kernel function is used to calculate the overlap between feature sets. The Jaccard coefficient kernel function generates a second similarity matrix by calculating the ratio of the intersection cardinality and the union cardinality of the feature sets.

[0036] Extract the string sequence from the standardized feature representation matrix. For the string sequence, use the weighted edit distance kernel function to calculate the edit operation cost between the strings. The weighted edit distance kernel function sets weight coefficients for replacement, insertion and deletion operations respectively, thereby generating a third similarity matrix.

[0037] An adaptive weight optimization model is established. The weight coefficients are determined by minimizing the matching error, and the weight coefficients are used to perform a weighted combination of the first similarity matrix, the second similarity matrix, and the third similarity matrix to generate a multidimensional similarity matrix.

[0038] The solution of this application can: The accuracy of feature similarity calculation is improved, and the relationship between different features can be better reflected. Through the adaptive weight optimization model, the weight of each feature type can be dynamically adjusted to enhance the flexibility and adaptability of the model. The formed multi-dimensional similarity matrix provides a richer information basis for subsequent data analysis and decision-making, and supports more accurate recommendations and matching.

[0039] In an optional implementation, an algorithm optimization decision tree is constructed, wherein the algorithm optimization decision tree generates a priority-ranked candidate algorithm sequence according to the matching score, algorithm complexity, and historical performance records; and a sequence dynamic update mechanism is established, wherein the sequence dynamic update mechanism adjusts the ranking weight of the candidate algorithm sequence in real time according to the change trend of the feature fingerprint identifier, including: Collecting algorithm evaluation data, the algorithm evaluation data including matching scores, algorithm complexity and historical performance records, and performing standardization on the algorithm evaluation data; Constructing an algorithm optimization decision tree, wherein the algorithm optimization decision tree adopts a three-layer structure, wherein the first layer sets a preferred weight coefficient according to the matching score, the second layer sets a computing resource weight coefficient according to the algorithm complexity, and the third layer sets a stability weight coefficient according to the historical performance record, and the preferred weight coefficient, the computing resource weight coefficient and the stability weight coefficient are weighted and combined into three-layer weight coefficients, and the algorithm optimization decision tree performs hierarchical evaluation on candidate algorithms through the three-layer weight coefficients; Calculating an evaluation score for each candidate algorithm based on the algorithm optimization decision tree, wherein the evaluation score is a weighted sum of weight coefficients of each layer, and prioritizing the candidate algorithms according to the evaluation score to generate a candidate algorithm sequence; Collect characteristic fingerprint identification data, and calculate the change trend of the characteristic fingerprint identification data within a preset time window, wherein the change trend includes the characteristic value change amplitude and change rate, wherein the characteristic value change amplitude represents the change direction and intensity of the characteristic value, and the change rate represents the change speed of the characteristic value; Establishing a sequence dynamic update mechanism, calculating a ranking weight adjustment coefficient according to the change trend of the characteristic fingerprint identification data, wherein the ranking weight adjustment coefficient is positively correlated with the characteristic value change amplitude and the change rate, and the ranking weight adjustment coefficient is used to adjust the priority weight of each algorithm in the candidate algorithm sequence; The characteristic value change amplitude and the change rate are monitored in real time. When the characteristic value change amplitude exceeds a first preset characteristic threshold or the change rate exceeds a second preset characteristic threshold, a sequence update operation is triggered, and the algorithm priority in the candidate algorithm sequence is dynamically adjusted using the sorting weight adjustment coefficient to generate an updated candidate algorithm sequence.

[0040] Collect algorithm evaluation data, including matching scores, algorithm complexity, and historical performance records. The matching scores reflect the performance of the algorithm on a specific task, the algorithm complexity indicates the computing resources required by the algorithm, and the historical performance records provide the stability and efficiency of the algorithm in past executions. These data need to be standardized to ensure that data of different dimensions can be effectively compared.

[0041] The algorithm optimization decision tree is constructed with a three-layer structure. The first layer sets the optimization weight coefficient according to the matching score to ensure that the best performing algorithm is given priority among the candidate algorithms. The second layer sets the computing resource weight coefficient according to the algorithm complexity to ensure that the appropriate algorithm is selected when resources are limited. The third layer sets the stability weight coefficient according to the historical performance record to ensure that the selected algorithm has shown good stability in history. The weight coefficients of these three layers are weighted and combined to form a comprehensive three-layer weight coefficient, which is used to grade the candidate algorithms.

[0042] Based on the constructed algorithm optimization decision tree, the evaluation score of each candidate algorithm is calculated. The evaluation score is the weighted sum of the weight coefficients of each layer, reflecting the comprehensive evaluation of the algorithm in terms of matching degree, complexity and historical performance. The candidate algorithms are prioritized according to the evaluation scores to generate a prioritized candidate algorithm sequence.

[0043] Collect characteristic fingerprint identification data and calculate the change trend of characteristic fingerprint identification data. The change trend includes the amplitude and rate of change of characteristic value. The amplitude of characteristic value change represents the direction and intensity of characteristic value change, while the rate of change represents the speed of characteristic value change. These data will be used for the subsequent dynamic update mechanism. Establish a sequence dynamic update mechanism and calculate the sorting weight adjustment coefficient according to the change trend of characteristic fingerprint identification data. The sorting weight adjustment coefficient is positively correlated with the amplitude and rate of change of characteristic value, which means that when the characteristic value changes significantly, the priority weight of the algorithm will be adjusted accordingly.

[0044] The adjustment coefficient is used to dynamically adjust the priority weight of each algorithm in the candidate algorithm sequence. The characteristic value change amplitude and change rate are monitored in real time. When the characteristic value change amplitude exceeds the first preset characteristic threshold or the change rate exceeds the second preset characteristic threshold, the sequence update operation is triggered. The algorithm priority in the candidate algorithm sequence is dynamically adjusted using the sorting weight adjustment coefficient to generate an updated candidate algorithm sequence.

[0045] The solution of this application can: Improve the accuracy of algorithm selection by comprehensively considering matching, complexity, and historical performance to ensure that the selected algorithm performs best in a specific task. Implement dynamic adjustment of algorithm priority to respond to changes in feature fingerprint identification data in a timely manner to ensure that the algorithm selection always meets the current environment and needs. Enhance the stability and efficiency of the system by introducing historical performance records to ensure that the selected algorithm has performed well in past executions, thereby reducing system risks.

[0046] In an optional implementation, an algorithm optimization decision tree is constructed, wherein the algorithm optimization decision tree adopts a three-layer structure, wherein the first layer sets the optimization weight coefficient according to the matching score, the second layer sets the computing resource weight coefficient according to the algorithm complexity, and the third layer sets the stability weight coefficient according to the historical performance record, including: Construct an algorithm optimization decision tree, and establish a three-layer structural framework of a matching evaluation layer, a complexity evaluation layer, and a stability evaluation layer based on the algorithm optimization decision tree; collect feature vectors of the data to be processed, wherein the feature vectors include data scale, distribution characteristics, and sparsity; Establish an algorithm feature matching model, input the feature vector into the algorithm feature matching model, calculate the similarity matching degree between the candidate algorithm and the feature vector; multiply the similarity matching degree by a preset feature weight to generate a matching score, and set the matching score as a first-layer optimization weight coefficient in the matching evaluation layer of the algorithm optimization decision tree; The matching score calculation formula is as follows: ; Among them, M ijis the matching score between the i-th feature vector and the j-th candidate algorithm, S ij is the similarity matching degree between the i-th feature vector and the j-th candidate algorithm, W f is the preset feature weight; Extracting the time complexity and space complexity information of the candidate algorithm and constructing a complexity quantification model; inputting the time complexity and space complexity information into the complexity quantification model to generate an evaluation score; calculating the ratio of the evaluation score to the system resource capacity, and setting the ratio as a second-layer computing resource weight coefficient in the complexity evaluation layer of the algorithm optimization decision tree; Collecting historical operation performance records of the candidate algorithm, wherein the historical operation performance records include response delay, throughput and resource occupancy rate; dividing the historical operation performance records into intervals according to time series, and calculating the stability score of the performance indicator in each interval; setting the stability score as the third-layer stability weight coefficient in the stability evaluation layer of the algorithm optimization decision tree; The stability score is calculated as follows: ; Among them, S j is the stability score of the jth algorithm, σ j is the performance index variance of the jth algorithm, δ j is the maximum fluctuation range of the performance index, δ max is the maximum acceptable fluctuation threshold.

[0047] A three-layer algorithm optimization decision tree is established. The first layer is the matching evaluation layer, and the optimization weight coefficient is set according to the matching score of the feature vector. The feature vector includes information such as data scale, distribution characteristics, and sparsity. By collecting the features of the data to be processed, the feature vector is formed, which provides a basis for subsequent algorithm matching.

[0048] Construct an algorithm feature matching model. Input the collected feature vector into the model and calculate the similarity between the candidate algorithm and the feature vector. The similarity can be calculated by comparing the various dimensions of the feature vector to obtain the performance of the candidate algorithm on the feature vector. Multiply the calculated similarity with the preset feature weight to generate a matching score, and use the score as the preferred weight coefficient of the first layer.

[0049] Extract the time complexity and space complexity information of the candidate algorithm and build a complexity quantification model. Input the extracted time complexity and space complexity information into the model to generate an evaluation score. The ratio of the evaluation score to the system resource capacity will be used as the computing resource weight coefficient of the second layer and set in the complexity evaluation layer.

[0050] Collect the historical performance records of candidate algorithms, including response delay, throughput, and resource utilization. Divide these historical records into time series and calculate the stability score of the performance indicators in each interval. The stability score will be used as the stability weight coefficient of the third layer and set in the stability assessment layer. Finally, the evaluation results of the three layers are combined to form the final optimal decision tree. The decision tree will select the candidate algorithms based on the weight coefficients of the three dimensions of matching, complexity, and stability to ensure that the optimal algorithm is selected.

[0051] The solution of this application can: Improve the accuracy of algorithm selection, and ensure that the algorithm that best meets the needs is selected through multi-dimensional evaluation. Optimize resource allocation, and reasonably allocate computing resources based on the ratio of algorithm complexity and system resource capacity to improve system efficiency. Enhance the stability of the algorithm, and ensure the reliability and stability of the selected algorithm in actual applications through the analysis of historical performance records.

[0052] In an optional implementation, the ranking result of the candidate algorithm sequence is used to realize intelligent scheduling of computing resources, the algorithm plug-in with the highest priority in the candidate algorithm sequence is selected, and a resource demand prediction model is constructed based on the feature fingerprint identification. The resource demand prediction model outputs the resource consumption curve of the algorithm plug-in at different execution stages, including: Obtaining a ranking result of the candidate algorithm sequence, the ranking result including a priority score of each candidate algorithm, and selecting an algorithm plug-in with the highest priority score in the candidate algorithm sequence; Collecting multi-dimensional resource consumption characteristics of the algorithm plug-in, the multi-dimensional resource consumption characteristics include computing density characteristics, data access characteristics and parallelism characteristics, and generating a resource feature vector; performing time series analysis on the feature fingerprint identifier based on the resource feature vector, extracting dynamic change characteristics of the feature fingerprint identifier within a preset time window, the dynamic change characteristics include periodic change rules and mutation characteristics, and generating a fingerprint time series vector; The dynamic fusion mechanism is used to calculate the correlation matrix between the resource feature vector and the fingerprint time series vector, and a feature fusion weight is constructed based on the correlation matrix; the resource feature vector and the fingerprint time series vector are input into the adaptive feature selection module, and the adaptive feature selection module selects prediction features based on the information gain criterion; the prediction features are ranked by importance, and the important features are combined according to the feature fusion weight to form an enhanced feature matrix; Constructing a multi-layer perception prediction model, the multi-layer perception prediction model includes a feature extraction layer, a hierarchical attention layer and a prediction output layer; setting a temporal attention unit and a feature dimension attention unit in the hierarchical attention layer, the temporal attention unit captures the dependency relationship of the time dimension, and the feature dimension attention unit captures the correlation relationship between features; The enhanced feature matrix is ​​input into the multi-layer perception prediction model to generate an initial prediction result; a residual learning structure is established to perform residual compensation on the initial prediction result and the historical prediction deviation to obtain a compensated prediction result; the compensated prediction result is divided according to the execution stage to generate a segmented resource consumption prediction curve.

[0053] In an optional implementation, the optimization response mechanism attempts to adjust the resource allocation parameters. If the performance continues to be abnormal after the adjustment, an alternative algorithm plug-in is selected from the candidate algorithm sequence for replacement, and the performance optimization result is fed back to the feature fingerprint identifier, and the feature weight and algorithm ranking are updated to form a closed-loop optimization including: The optimization response mechanism includes a resource parameter adjustment module and an algorithm replacement module; collects system operation status data, obtains processor utilization, memory occupancy, response time and throughput, and generates system performance indicators; A dynamic threshold detection model is established based on the system performance indicator, and when the system performance indicator deviates from the dynamic threshold, the resource parameter adjustment module is activated; the resource parameter adjustment module calculates the influence of resource parameters on performance, generates a parameter sensitivity matrix, and determines the optimal parameter adjustment scheme based on the parameter sensitivity matrix; Execute the optimal parameter adjustment scheme, record the adjusted system performance index, and calculate the performance improvement value; when the performance improvement value is lower than the preset improvement threshold for multiple times in a row, determine that the parameter adjustment is invalid, and trigger the algorithm replacement module; the algorithm replacement module selects a set of candidate algorithms from a preset candidate algorithm sequence based on the current system performance index; Calculating the expected performance score for each algorithm in the candidate algorithm set, and selecting the algorithm with the highest expected performance score as the replacement target; performing the algorithm replacement operation, collecting the system performance indicators after the replacement, and generating an optimization result including the parameter adjustment effect and the algorithm replacement effect; Based on the optimization results, the algorithm feature weights are updated, and the candidate algorithm sequence is reordered according to the updated algorithm feature weights; the parameter adjustment plan, algorithm replacement record and performance improvement data in this round of optimization are stored in the optimization history database; based on the optimization history database, the timing of the next performance bottleneck is predicted, and a new round of optimization response is triggered to form a closed-loop optimization.

[0054] First, the core of the optimization response mechanism is the resource parameter adjustment module and the algorithm replacement module. The system collects operating status data to obtain performance indicators such as processor utilization, memory occupancy, response time, and throughput. These indicators provide basic data for the subsequent dynamic threshold detection model. Next, based on the collected system performance indicators, a dynamic threshold detection model is established. This model can monitor the changes in system performance indicators in real time and activate the resource parameter adjustment module when the indicators deviate from the set dynamic threshold.

[0055] The task of the resource parameter adjustment module is to calculate the impact of different resource parameters on system performance and generate a parameter sensitivity matrix. By analyzing the matrix, the optimal parameter adjustment scheme is determined. After executing the optimal parameter adjustment scheme, the system will record the adjusted performance indicators and calculate the performance improvement value. If the performance improvement value is lower than the preset improvement threshold for multiple consecutive times, the system will determine that the parameter adjustment is invalid and trigger the algorithm replacement module. The algorithm replacement module selects a set of alternative algorithms from the preset candidate algorithm sequence based on the current system performance indicators.

[0056] For each algorithm in the set of candidate algorithms, the system calculates its expected performance score and selects the algorithm with the highest score as the replacement target. After executing the algorithm replacement operation, the system collects the performance indicators after replacement and generates an optimization result that includes the parameter adjustment effect and the algorithm replacement effect. Finally, based on the optimization results, the system updates the algorithm feature weights and reorders the candidate algorithm sequence according to the updated weights. At the same time, the parameter adjustment plan, algorithm replacement record and performance improvement data in this round of optimization are stored in the optimization history database. By analyzing the optimization history database, the system can predict the next time a performance bottleneck will occur, thereby triggering a new round of optimization response and forming a closed-loop optimization.

[0057] The solution of this application can: Improve system performance: By dynamically adjusting resource parameters and replacing algorithms, the system can respond quickly when performance degrades, ensuring continuous and efficient operation. Reduce manual intervention: The optimization response mechanism has a high degree of automation, which reduces dependence on manual intervention and improves the system's adaptive ability. Form a continuous optimization closed loop: Through the accumulation and analysis of historical data, the system can continuously optimize its own performance, form a virtuous cycle, and improve overall efficiency.

[0058] Figure 2 FIG. 1 is a schematic diagram of a multi-data source plug-in computing resource scheduling system supporting algorithm iteration according to an embodiment of the present invention. Figure 2 As shown, the system comprises: The first unit is used to build a data source feature analysis engine, extract features from multiple data sources, extract data structure features to generate a structure feature matrix, extract data time series features to generate a time series feature matrix, and extract data quality features to generate a quality feature matrix; the structure feature matrix, the time series feature matrix, and the quality feature matrix are input into a feature fusion model, the feature fusion model uses an adaptive weight mechanism to calculate the importance coefficients of three types of features, and realizes correlation analysis between matrices through a feature interaction network, and finally fuses to generate a feature fingerprint identifier of the data source, the feature fingerprint identifier contains full-dimensional feature information and feature importance weights of the data source; The second unit is used to design an algorithm plug-in intelligent matching mechanism based on the feature fingerprint identification, wherein the algorithm plug-in intelligent matching mechanism first extracts algorithm feature parameters from the algorithm plug-in library, establishes an improved two-way matching scoring model, and calculates the matching score between each algorithm plug-in and the data source according to the feature information and weight in the feature fingerprint identification; constructs an algorithm optimization decision tree, wherein the algorithm optimization decision tree generates a priority-ranked candidate algorithm sequence according to the matching score, algorithm complexity and historical performance records; and simultaneously establishes a sequence dynamic update mechanism, wherein the sequence dynamic update mechanism adjusts the ranking weight of the candidate algorithm sequence in real time according to the changing trend of the feature fingerprint identification; The third unit is used to use the sorting results of the candidate algorithm sequence to realize intelligent scheduling of computing resources, select the algorithm plug-in with the highest priority in the candidate algorithm sequence, and build a resource demand prediction model based on the feature fingerprint identification. The resource demand prediction model outputs the resource consumption curve of the algorithm plug-in at different execution stages; the resource dynamic allocation strategy is designed according to the resource consumption curve. The resource dynamic allocation strategy monitors system performance indicators in real time, including computing efficiency indicators, resource utilization indicators and task completion quality indicators. When the system performance indicators are monitored to be lower than the preset performance threshold, an optimization response mechanism is triggered; the optimization response mechanism attempts to adjust the resource allocation parameters. If the performance continues to be abnormal after adjustment, an alternative algorithm plug-in is selected from the candidate algorithm sequence for replacement, and the performance optimization result is fed back to the feature fingerprint identification, the feature weight and algorithm sorting are updated, and a closed-loop optimization is formed.

[0059] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0060] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0061] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-data source plug-in computing resource scheduling method supporting algorithm iteration, characterized in that: include: Construct a data source feature analysis engine, extract features from multiple data sources, extract data structure features to generate a structure feature matrix, extract data time series features to generate a time series feature matrix, and extract data quality features to generate a quality feature matrix; input the structure feature matrix, the time series feature matrix, and the quality feature matrix into a feature fusion model, the feature fusion model uses an adaptive weight mechanism to calculate the importance coefficients of the three types of features, and implements correlation analysis between matrices through a feature interaction network, and finally fuses to generate a feature fingerprint identifier of the data source, the feature fingerprint identifier contains full-dimensional feature information and feature importance weights of the data source; Based on the feature fingerprint identification, an algorithm plug-in intelligent matching mechanism is designed. The algorithm plug-in intelligent matching mechanism first extracts algorithm feature parameters from the algorithm plug-in library and establishes an improved two-way matching scoring model. The improved two-way matching scoring model calculates the matching score between each algorithm plug-in and the data source according to the feature information and weight in the feature fingerprint identification; Constructing an algorithm optimization decision tree, wherein the algorithm optimization decision tree generates a priority-ranked candidate algorithm sequence according to matching scores, algorithm complexity, and historical performance records; and establishing a sequence dynamic update mechanism, wherein the sequence dynamic update mechanism adjusts the ranking weight of the candidate algorithm sequence in real time according to the changing trend of the feature fingerprint identification; The ranking results of the candidate algorithm sequence are used to realize intelligent scheduling of computing resources, select the algorithm plug-in with the highest priority in the candidate algorithm sequence, and build a resource demand prediction model based on the feature fingerprint identification. The resource demand prediction model outputs the resource consumption curve of the algorithm plug-in at different execution stages; design a resource dynamic allocation strategy according to the resource consumption curve, and the resource dynamic allocation strategy monitors system performance indicators in real time, including computing efficiency indicators, resource utilization indicators and task completion quality indicators. When the system performance indicators are monitored to be lower than the preset performance threshold, an optimization response mechanism is triggered; the optimization response mechanism attempts to adjust the resource allocation parameters. If the performance continues to be abnormal after adjustment, an alternative algorithm plug-in is selected from the candidate algorithm sequence for replacement, and the performance optimization result is fed back to the feature fingerprint identification, and the feature weight and algorithm ranking are updated to form a closed-loop optimization.

2. The method according to claim 1, characterized in that Based on the feature fingerprint identification, an algorithm plug-in intelligent matching mechanism is designed. The algorithm plug-in intelligent matching mechanism first extracts algorithm feature parameters from the algorithm plug-in library and establishes an improved two-way matching scoring model. The improved two-way matching scoring model calculates the matching score between each algorithm plug-in and the data source according to the feature information and weight in the feature fingerprint identification, including: The feature fingerprint identification includes data structure features, data timing features and data quality features. The data structure features, data timing features and data quality features are mapped into a feature vector group by a feature vectorization processing unit, and an initial importance weight of each feature is generated based on information entropy calculation; Based on the feature vector group and the initial importance weight, an algorithm feature extraction model is constructed, algorithm feature parameters are extracted from the algorithm plug-in library, time complexity parameters, space complexity parameters and communication complexity parameters are extracted to generate a complexity feature matrix, data processing parameters, calculation accuracy parameters and parallel degree parameters are extracted to generate a functional feature matrix, data sensitivity parameters, resource dependency parameters and expansion capability parameters are extracted to generate an adaptability feature matrix; the complexity feature matrix, the functional feature matrix and the adaptability feature matrix are combined to form an algorithm feature parameter set; Constructing a feature space mapping model, wherein the feature space mapping model converts the feature vector group and the algorithm feature parameter set into a unified feature space, uses an adaptive feature completion unit to process feature missing, uses a nonlinear feature transformation unit to unify feature dimensions, and generates a standardized feature representation matrix; Based on the standardized feature representation matrix, an improved two-way matching scoring model is established; based on the improved two-way matching scoring model, a multi-core similarity calculation network is constructed; for different feature types, the cosine similarity kernel function, the Jaccard coefficient kernel function and the edit distance kernel function are used to calculate the similarity to form a multi-dimensional similarity matrix; using a dynamic weight optimizer, according to the initial importance weight and the historical matching effect, the feature fusion weight coefficient is calculated; the multi-dimensional similarity matrix and the feature fusion weight coefficient are weightedly combined to output the matching score between the algorithm plug-in and the data source.

3. The method according to claim 2, characterized in that Based on the standardized feature representation matrix, an improved two-way matching scoring model is established, and a multi-core similarity calculation network is constructed based on the improved two-way matching scoring model. The cosine similarity kernel function, the Jaccard coefficient kernel function and the edit distance kernel function are used to calculate the similarity for different feature types, and a multi-dimensional similarity matrix is ​​formed, including: Establishing a standardized feature representation matrix, wherein the standardized feature representation matrix includes continuous numerical features, discrete category features, and string type features; Inputting the standardized feature representation matrix into the improved two-way matching scoring model; extracting feature vectors from the standardized feature representation matrix, and calculating the angular similarity between feature vectors using a cosine similarity kernel function for the continuous numerical features, wherein the cosine similarity kernel function divides the inner product of the feature vector by the product of the Euclidean norm of the feature vector to generate a first similarity matrix; Extracting feature sets from the standardized feature representation matrix, and calculating the overlap between feature sets using a Jaccard coefficient kernel function for the discrete category features, wherein the Jaccard coefficient kernel function divides the intersection cardinality of the feature sets by the union cardinality of the feature sets to generate a second similarity matrix; Extracting a string sequence from the standardized feature representation matrix, and using a weighted edit distance kernel function to calculate the edit operation cost between the strings for the string sequence, wherein the weighted edit distance kernel function sets weight coefficients for replacement operation, insertion operation and deletion operation respectively, to generate a third similarity matrix; An adaptive weight optimization model is established, a weight coefficient is determined by minimizing the matching error, and the first similarity matrix, the second similarity matrix and the third similarity matrix are weightedly combined using the weight coefficient to generate a multidimensional similarity matrix.

4. The method according to claim 1, characterized in that Constructing an algorithm optimization decision tree, the algorithm optimization decision tree generates a priority-ranked candidate algorithm sequence based on the matching score, algorithm complexity and historical performance records; at the same time, establishing a sequence dynamic update mechanism, the sequence dynamic update mechanism adjusts the ranking weight of the candidate algorithm sequence in real time according to the change trend of the feature fingerprint identification, including: Collecting algorithm evaluation data, the algorithm evaluation data including matching scores, algorithm complexity and historical performance records, and performing standardization on the algorithm evaluation data; Constructing an algorithm optimization decision tree, wherein the algorithm optimization decision tree adopts a three-layer structure, wherein the first layer sets a preferred weight coefficient according to the matching score, the second layer sets a computing resource weight coefficient according to the algorithm complexity, and the third layer sets a stability weight coefficient according to the historical performance record, and the preferred weight coefficient, the computing resource weight coefficient and the stability weight coefficient are weighted and combined into three-layer weight coefficients, and the algorithm optimization decision tree performs hierarchical evaluation on candidate algorithms through the three-layer weight coefficients; Calculating an evaluation score for each candidate algorithm based on the algorithm optimization decision tree, wherein the evaluation score is a weighted sum of weight coefficients of each layer, and prioritizing the candidate algorithms according to the evaluation score to generate a candidate algorithm sequence; Collect characteristic fingerprint identification data, and calculate the change trend of the characteristic fingerprint identification data within a preset time window, wherein the change trend includes the characteristic value change amplitude and change rate, wherein the characteristic value change amplitude represents the change direction and intensity of the characteristic value, and the change rate represents the change speed of the characteristic value; Establishing a sequence dynamic update mechanism, calculating a ranking weight adjustment coefficient according to the change trend of the characteristic fingerprint identification data, wherein the ranking weight adjustment coefficient is positively correlated with the characteristic value change amplitude and the change rate, and the ranking weight adjustment coefficient is used to adjust the priority weight of each algorithm in the candidate algorithm sequence; The characteristic value change amplitude and the change rate are monitored in real time. When the characteristic value change amplitude exceeds a first preset characteristic threshold or the change rate exceeds a second preset characteristic threshold, a sequence update operation is triggered, and the algorithm priority in the candidate algorithm sequence is dynamically adjusted using the sorting weight adjustment coefficient to generate an updated candidate algorithm sequence.

5. The method according to claim 4, characterized in that Construct an algorithm optimization decision tree, wherein the algorithm optimization decision tree adopts a three-layer structure, wherein the first layer sets the optimization weight coefficient according to the matching score, the second layer sets the computing resource weight coefficient according to the algorithm complexity, and the third layer sets the stability weight coefficient according to the historical performance record, including: Construct an algorithm optimization decision tree, and establish a three-layer structural framework of a matching evaluation layer, a complexity evaluation layer, and a stability evaluation layer based on the algorithm optimization decision tree; collect feature vectors of the data to be processed, wherein the feature vectors include data scale, distribution characteristics, and sparsity; Establish an algorithm feature matching model, input the feature vector into the algorithm feature matching model, calculate the similarity matching degree between the candidate algorithm and the feature vector; multiply the similarity matching degree by a preset feature weight to generate a matching score, and set the matching score as a first-layer optimization weight coefficient in the matching evaluation layer of the algorithm optimization decision tree; The matching score calculation formula is as follows: ； Among them, M ij is the matching score between the i-th feature vector and the j-th candidate algorithm, S ij is the similarity matching degree between the i-th feature vector and the j-th candidate algorithm, W f is the preset feature weight; Extracting the time complexity and space complexity information of the candidate algorithm and constructing a complexity quantification model; inputting the time complexity and space complexity information into the complexity quantification model to generate an evaluation score; calculating the ratio of the evaluation score to the system resource capacity, and setting the ratio as a second-layer computing resource weight coefficient in the complexity evaluation layer of the algorithm optimization decision tree; Collecting historical operation performance records of the candidate algorithm, wherein the historical operation performance records include response delay, throughput and resource occupancy rate; dividing the historical operation performance records into intervals according to time series, and calculating the stability score of the performance indicator in each interval; setting the stability score as the third-layer stability weight coefficient in the stability evaluation layer of the algorithm optimization decision tree; The stability score is calculated as follows: ； Among them, S j is the stability score of the jth algorithm, σ j is the performance index variance of the jth algorithm, δ j is the maximum fluctuation range of the performance index, δ max is the maximum acceptable fluctuation threshold.

6. The method according to claim 1, characterized in that The ranking result of the candidate algorithm sequence is used to realize intelligent scheduling of computing resources, and the algorithm plug-in with the highest priority in the candidate algorithm sequence is selected. A resource demand prediction model is constructed based on the feature fingerprint identification. The resource demand prediction model outputs the resource consumption curve of the algorithm plug-in at different execution stages, including: Obtaining a ranking result of the candidate algorithm sequence, the ranking result including a priority score of each candidate algorithm, and selecting an algorithm plug-in with the highest priority score in the candidate algorithm sequence; Collecting multi-dimensional resource consumption characteristics of the algorithm plug-in, the multi-dimensional resource consumption characteristics include computing density characteristics, data access characteristics and parallelism characteristics, and generating a resource feature vector; performing time series analysis on the feature fingerprint identifier based on the resource feature vector, extracting dynamic change characteristics of the feature fingerprint identifier within a preset time window, the dynamic change characteristics include periodic change rules and mutation characteristics, and generating a fingerprint time series vector; The dynamic fusion mechanism is used to calculate the correlation matrix between the resource feature vector and the fingerprint time series vector, and a feature fusion weight is constructed based on the correlation matrix; the resource feature vector and the fingerprint time series vector are input into the adaptive feature selection module, and the adaptive feature selection module selects prediction features based on the information gain criterion; the prediction features are ranked by importance, and the important features are combined according to the feature fusion weight to form an enhanced feature matrix; Constructing a multi-layer perception prediction model, the multi-layer perception prediction model includes a feature extraction layer, a hierarchical attention layer and a prediction output layer; setting a temporal attention unit and a feature dimension attention unit in the hierarchical attention layer, the temporal attention unit captures the dependency relationship of the time dimension, and the feature dimension attention unit captures the correlation relationship between features; The enhanced feature matrix is ​​input into the multi-layer perception prediction model to generate an initial prediction result; a residual learning structure is established to perform residual compensation on the initial prediction result and the historical prediction deviation to obtain a compensated prediction result; the compensated prediction result is divided according to the execution stage to generate a segmented resource consumption prediction curve.

7. The method according to claim 1, characterized in that The optimization response mechanism attempts to adjust the resource allocation parameters. If the performance continues to be abnormal after the adjustment, an alternative algorithm plug-in is selected from the candidate algorithm sequence for replacement, and the performance optimization result is fed back to the feature fingerprint identifier, and the feature weight and algorithm ranking are updated to form a closed-loop optimization including: The optimization response mechanism includes a resource parameter adjustment module and an algorithm replacement module; collects system operation status data, obtains processor utilization, memory occupancy, response time and throughput, and generates system performance indicators; A dynamic threshold detection model is established based on the system performance indicator, and when the system performance indicator deviates from the dynamic threshold, the resource parameter adjustment module is activated; the resource parameter adjustment module calculates the influence of resource parameters on performance, generates a parameter sensitivity matrix, and determines the optimal parameter adjustment scheme based on the parameter sensitivity matrix; Execute the optimal parameter adjustment scheme, record the adjusted system performance index, and calculate the performance improvement value; when the performance improvement value is lower than the preset improvement threshold for multiple times in a row, determine that the parameter adjustment is invalid, and trigger the algorithm replacement module; the algorithm replacement module selects a set of candidate algorithms from a preset candidate algorithm sequence based on the current system performance index; Calculating the expected performance score for each algorithm in the candidate algorithm set, and selecting the algorithm with the highest expected performance score as the replacement target; performing the algorithm replacement operation, collecting the system performance indicators after the replacement, and generating an optimization result including the parameter adjustment effect and the algorithm replacement effect; Based on the optimization results, the algorithm feature weights are updated, and the candidate algorithm sequence is reordered according to the updated algorithm feature weights; the parameter adjustment plan, algorithm replacement record and performance improvement data in this round of optimization are stored in the optimization history database; based on the optimization history database, the timing of the next performance bottleneck is predicted, and a new round of optimization response is triggered to form a closed-loop optimization.

8. A multi-data source plug-in computing resource scheduling system supporting algorithm iteration, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to build a data source feature analysis engine, extract features from multiple data sources, extract data structure features to generate a structure feature matrix, extract data time series features to generate a time series feature matrix, and extract data quality features to generate a quality feature matrix; the structure feature matrix, the time series feature matrix, and the quality feature matrix are input into a feature fusion model, the feature fusion model uses an adaptive weight mechanism to calculate the importance coefficients of three types of features, and realizes correlation analysis between matrices through a feature interaction network, and finally fuses to generate a feature fingerprint identifier of the data source, the feature fingerprint identifier contains full-dimensional feature information and feature importance weights of the data source; The second unit is used to design an algorithm plug-in intelligent matching mechanism based on the feature fingerprint identification, wherein the algorithm plug-in intelligent matching mechanism first extracts algorithm feature parameters from the algorithm plug-in library and establishes an improved two-way matching scoring model, wherein the improved two-way matching scoring model calculates the matching score between each algorithm plug-in and the data source according to the feature information and weight in the feature fingerprint identification; Constructing an algorithm optimization decision tree, wherein the algorithm optimization decision tree generates a priority-ranked candidate algorithm sequence according to matching scores, algorithm complexity, and historical performance records; and establishing a sequence dynamic update mechanism, wherein the sequence dynamic update mechanism adjusts the ranking weight of the candidate algorithm sequence in real time according to the changing trend of the feature fingerprint identification; The third unit is used to use the sorting results of the candidate algorithm sequence to realize intelligent scheduling of computing resources, select the algorithm plug-in with the highest priority in the candidate algorithm sequence, and build a resource demand prediction model based on the feature fingerprint identification. The resource demand prediction model outputs the resource consumption curve of the algorithm plug-in at different execution stages; the resource dynamic allocation strategy is designed according to the resource consumption curve. The resource dynamic allocation strategy monitors system performance indicators in real time, including computing efficiency indicators, resource utilization indicators and task completion quality indicators. When the system performance indicators are monitored to be lower than the preset performance threshold, an optimization response mechanism is triggered; the optimization response mechanism attempts to adjust the resource allocation parameters. If the performance continues to be abnormal after adjustment, an alternative algorithm plug-in is selected from the candidate algorithm sequence for replacement, and the performance optimization result is fed back to the feature fingerprint identification, the feature weight and algorithm sorting are updated, and a closed-loop optimization is formed.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images