Metadata acquisition method based on user-defined strategy optimization algorithm

Abstract

The invention discloses a metadata acquisition method based on a user-defined strategy optimization algorithm. The metadata acquisition method comprises the following steps: S1, extracting metadata from a data source; s2, the extracted metadata is processed; s3, performing quality evaluation on the processed metadata; s4, applying an algorithm combining a graph convolutional network and a self-attention mechanism to separately propose a preliminary acquisition strategy; s5, optimizing the preliminary acquisition strategy by adopting a genetic algorithm to generate an optimized acquisition strategy; s7, constructing a self-adaptive strategy optimization model; s8, performing global optimization on the strategy by applying a Bayesian optimization algorithm; s9, implementing an adaptive data segmentation technology; s10, monitoring an acquisition result in real time through a feedback cycle mechanism; and S11, providing a user interface, and allowing a user to configure an initial acquisition strategy, check an acquisition result and adjust parameters. According to the method, optimization algorithms such as the graph convolutional network and the deep Q network are utilized, a metadata acquisition strategy is dynamically adjusted, and the method has the advantages of high efficiency, flexibility and improvement of data quality.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing and deep learning, and in particular to a metadata collection method based on a custom strategy optimization algorithm. Background Art

[0002] In today's big data era, metadata plays a vital role in data management and data analysis. Metadata not only describes the structure, content and source of data, but also provides necessary support for data storage, retrieval, sharing and analysis. Efficient and accurate collection and management of metadata is crucial to improving data utilization efficiency and ensuring data quality. Existing metadata collection technologies mainly include SQL query, API call and file reading, combined with processing technologies such as data parsing, data cleaning and data standardization, which can achieve a certain degree of metadata collection. However, with the diversification of data sources and the continuous expansion of data scale, traditional metadata collection methods face many challenges and shortcomings.

[0003] Traditional metadata collection methods usually use fixed collection strategies, which are difficult to cope with changes in data sources and data characteristics. Fixed strategies may be applicable during initial setup, but as time goes by and data sources change, fixed strategies often cannot flexibly respond to new data requirements, resulting in low collection efficiency and reduced data quality. For example, for metadata collection in SQL databases, fixed SQL query statements may not be able to adapt to dynamically changing database structures and data content, resulting in incomplete or inaccurate collected metadata. For API calls, fixed call parameters and frequencies may not be able to cope with changes in API response time and the frequency of data updates, resulting in collection delays or omissions.

[0004] In addition, existing metadata collection methods lack intelligent optimization capabilities and are unable to adjust strategies based on collection results and real-time feedback. Traditional methods usually rely on manual experience and predefined rules to set strategies, and it is difficult to dynamically optimize according to actual collection conditions. For example, when faced with large-scale data sources, manually adjusting collection strategies is not only time-consuming and labor-intensive, but also prone to errors, making it difficult to ensure the efficiency and quality of collection. As data sources continue to expand and become more complex, the limitations of traditional methods become increasingly apparent.

[0005] The rise of machine learning and artificial intelligence technologies has provided new ideas and methods for metadata collection. By introducing machine learning algorithms, we can analyze the collection results and propose optimization strategies. However, a single machine learning algorithm is still limited in effect when facing a complex and changing data environment.

[0006] Therefore, how to provide a metadata collection method based on a custom strategy optimization algorithm is a problem that those skilled in the art urgently need to solve. Summary of the invention

[0007] One object of the present invention is to propose a metadata collection method based on a custom strategy optimization algorithm. The present invention utilizes an algorithm combining a graph convolutional network and a self-attention mechanism, combined with a genetic algorithm, a deep Q network combined with a variational autoencoder, and a Bayesian optimization algorithm to dynamically adjust and optimize the metadata collection strategy. It can not only quickly generate and adjust the collection strategy, but also ensure that the collected metadata has high integrity, accuracy and consistency. At the same time, the adaptive data segmentation technology is used to further improve the accuracy and efficiency of data collection.

[0008] A metadata collection method based on a custom strategy optimization algorithm according to an embodiment of the present invention includes the following steps:

[0009] S1, extract metadata from data sources;

[0010] S2, perform data analysis, data cleaning and data standardization on the extracted metadata;

[0011] S3. Perform quality assessment on the processed metadata, including completeness, accuracy, and consistency indicators;

[0012] S4. Apply the algorithm combining graph convolutional network and self-attention mechanism to analyze the metadata collection results and propose a preliminary collection strategy;

[0013] S5, using genetic algorithm to optimize the preliminary acquisition strategy, and generating an optimized acquisition strategy through selection, crossover and mutation operations;

[0014] S6. Build an adaptive strategy optimization model, using a deep Q-network combined with a variational autoencoder to learn the optimal acquisition strategy in a dynamic environment;

[0015] S7. Apply the Bayesian optimization algorithm to globally optimize the acquisition strategy by constructing a probability model of the acquisition strategy and using the Gaussian process to estimate the distribution of the objective function;

[0016] S8. Implement adaptive data segmentation technology to dynamically adjust the scope and depth of data collection based on data source characteristics and metadata distribution;

[0017] S9, monitor the collection results in real time through the feedback loop mechanism to evaluate the collection efficiency and quality;

[0018] S10. Provide a user interface to allow the user to configure the initial collection strategy, view the collection results, and adjust parameters.

[0019] Furthermore, the S4 specifically includes:

[0020] S41. Use graph convolutional networks to structure metadata representation by embedding metadata nodes into high-dimensional space to capture the relationships and features between nodes.

[0021] S42. In the graph convolutional network, the input data is set to the feature matrix X∈R of the metadata node N×F , where N represents the number of nodes and F represents the feature dimension; the adjacency matrix of the graph is set to A∈R N×N , represents the connection relationship between nodes;

[0022] S43. Calculate node feature updates through graph convolutional layer:

[0023]

[0024] in, I represents the identity matrix, which is used to add self-connection to the adjacency matrix. express The degree matrix, H (l) represents the node feature matrix of the lth layer, H (l+1) represents the node feature matrix of the l+1th layer, W (l) represents the weight matrix of the lth layer, σ represents the activation function;

[0025] S44. In the self-attention mechanism, the feature matrix of the metadata node is set as input to generate the query matrix Q, key matrix K and value matrix V:

[0026] Q=XW Q ,K=XW K ,V=XW V ,Q′=XW Q′ ,K′=XW K′ ,V′=XW V′ ;

[0027] Among them, W Q , W K , W V , W Q′ , W K′ and W V′ represents the trainable weight matrix, Q′ represents the enhanced query matrix, K′ represents the enhanced key matrix, and V′ represents the enhanced value matrix;

[0028] S45. Through the multi-head attention mechanism, the self-attention mechanism is repeated many times to obtain the output matrix MultiHead(Q,K,V,Q′,K′,V′) of the multi-head attention mechanism:

[0029] MultiHead(Q,K,V,Q′,K′,V′)=Concat(head1,head2,…,head h)W O ;

[0030]

[0031] Among them, head i represents the i-th enhanced attention head, represents the trainable weight matrix of the i-th query matrix, represents the trainable weight matrix of the i-th key matrix, represents the trainable weight matrix of the i-th value matrix, represents the trainable weight matrix of the i-th augmented query matrix, represents the trainable weight matrix of the i-th augmented bond matrix, Represents the trainable weight matrix of the i-th enhancement value matrix, W O Represents the output trainable weight matrix of the multi-head attention mechanism.

[0032] S46. Perform residual connection and layer normalization on the output results of the multi-head attention mechanism:

[0033] Output=LayerNorm(X+MultiHead(Q,K,V,Q′,K′,V′));

[0034] Among them, LayerNorm represents the layer normalization operation;

[0035] S47. Propose a preliminary collection strategy.

[0036] Furthermore, the S5 specifically includes:

[0037] S51, initialize the genetic algorithm parameters, and use the preliminary acquisition strategy as the initial population, each individual in the population represents a acquisition strategy, and each individual is composed of a set of strategy parameters;

[0038] S52. Evaluate each individual in the initial population. The evaluation criteria include collection efficiency, collection quality, and resource consumption. Use the fitness function f(x) to calculate the fitness of each individual. The fitness function is defined as:

[0039]

[0040] Wherein, E(x) represents the collection efficiency, Q(x) represents the collection quality, C(x) represents the resource consumption, w1, w2 and w3 represent the weight parameters, and satisfy w1+w2+w3=1;

[0041] S53, selecting individuals according to fitness values, using roulette wheel selection method, and selecting individuals with higher fitness values ​​from the current population as parent individuals;

[0042] S54, randomly select a crossover point, combine the part of the parent individual before the point with the part of another parent individual after the point to form a child individual, the specific formula is: for the parent individual P1 = (p 1,1 ,p 1,2 ,...,p 1,n ) and P2=(p 2,1 ,p 2,2 ,...,p 2,n ), generate offspring individuals C = (c1, c2, ..., c n ),

[0043]

[0044] Where k represents a randomly selected intersection point;

[0045] S55, for the offspring individual C=(c1,c2,...,c n ) Randomly select a mutation point and randomly change the gene value of the point to obtain the mutated offspring individual C′=(c′1,c′2,...,c′ n ):

[0046]

[0047] Among them, k′ represents a randomly selected mutation point;

[0048] S56, evaluating the newly generated offspring individuals, and using the fitness function f(x) to calculate the fitness value of each offspring individual;

[0049] S57, merge the offspring individuals with the individuals in the current population, sort them from high to low according to the fitness value, and select the first N individuals as the new population;

[0050] S58, judging whether the maximum number of iterations has been reached or whether the individual fitness values ​​in the population have reached a preset threshold; if the termination condition is met, outputting the final population as the optimized acquisition strategy; otherwise, returning to S53 to continue iteration.

[0051] Furthermore, the S6 specifically includes:

[0052] S61, initializing the parameters of the deep Q network combined with the variational autoencoder, defining the state space S, the action space A and the reward function R, wherein the state space includes the current acquisition strategy and metadata distribution characteristics, the action space includes the adjustment operation of the acquisition strategy, and the reward function is defined according to the efficiency and quality of metadata acquisition;

[0053] S62. Through the experience replay mechanism, store the state-action-reward-next state (s, a, r, s′) quadruple collected from the environment and build an experience replay pool D;

[0054] S63, construct a target Q network for generating a target Q value. The parameters of the target Q network are synchronized from the deep Q network combined with the variational autoencoder every certain number of steps. The calculation formula of the target Q value is:

[0055]

[0056] Among them, r represents the current reward value, γ represents the discount factor, represents the maximum Q value of all possible actions a′ in the next state s′, θ - represents the parameters of the target Q grid;

[0057] S64. Use the deep Q network combined with the variational autoencoder to update parameters and introduce latent variables to learn state representation, including:

[0058] Encode the input state and generate the mean μ and variance σ of the latent variable 2 :

[0059] μ=f μ (s;θ enc );

[0060] log2σ 2 =f σ (s;θ enc );

[0061] Among them, f μ and f σ represents the encoder network, θ enc Represents encoder parameters;

[0062] Sampling from the distribution of latent variables generates latent variables z:

[0063]

[0064] Where z represents the latent variable sampled, ∈ represents the noise sampled from the standard normal distribution,

[0065] represents a standard normal distribution with a mean of 0 and a variance of 1;

[0066] Calculate the reconstruction error and KL divergence, and define the loss function of the variational autoencoder. The formula is as follows:

[0067] L VAE =E q(z|s) [log2p(s|z)]-DKL (q(z|s)||p(z));

[0068] Among them, L VAE represents the loss function of the variational autoencoder, F q(z|s) [log p(s|z)] represents the reconstruction error, D KL represents the KL divergence, which represents the difference between the approximate posterior distribution q(z|s) and the prior distribution p(z);

[0069] The loss function is used to update the parameters of the deep Q network combined with the variational autoencoder. The loss function is:

[0070] L(θ)=E (s,a,r,s′)～D [(Q(s,a;θ)-Q target ) 2 ]+L VAE ;

[0071] Where L(θ) represents the total loss function, θ represents the parameters of the deep Q network combined with the variational autoencoder, and Q(s,a;θ) represents the actual Q value under the current state s and action a;

[0072] S65, select actions through greedy strategy, select random actions with probability ∈, select actions with maximum Q value in the current state with probability 1-∈, and gradually reduce the ∈ value;

[0073] S66. Execute the selected action in the dynamic environment, update the state and calculate the reward, and repeat steps S62-S65 until the deep Q network combined with the variational autoencoder converges to obtain the local optimal acquisition strategy.

[0074] By combining a Deep Q Network (DQN-VAE) with a Variational Autoencoder (VAE), the following benefits can be achieved:

[0075] Improve the generalization ability of the strategy: By introducing VAE, it is possible to better capture the potential structural information in the environment, thereby improving the generalization ability of the strategy and making it applicable to more different data collection scenarios.

[0076] Reduce the risk of overfitting of the model: VAE can effectively reduce the overfitting phenomenon of the model and improve the stability and reliability of the acquisition strategy by regularizing the latent space.

[0077] Improved data efficiency: The deep Q network combined with VAE can use data more efficiently, extract more useful information from the limited experience replay pool, and speed up strategy optimization.

[0078] Furthermore, the S7 specifically includes:

[0079] S71, constructing an initial probability model of the acquisition strategy, the objective function is L(x), and the input is the parameter x of the acquisition strategy;

[0080] S72. Use a Gaussian process as a prior model of the objective function L(x), where the mean function of the Gaussian process is μ(x) and the covariance function is k(x, x′);

[0081] S73. In each collection strategy evaluation, record the corresponding collection efficiency and quality evaluation results, update the posterior distribution of the Gaussian process, and calculate the posterior mean μ using the Bayesian theorem. n (x) and covariance k n (x,x′):

[0082] μ n (x) = μ(x) + K(x, X n )[K(X n ,X n )+σ 2 I] -1 (Y n -μ(X n ));

[0083] k n (x,x′)=k(x,x′)-K(x,X n )[K(X n ,X n )+σ 2 I] -1 K(X n ,x′);

[0084] Among them, K(x,X n ) represents the acquisition strategy parameter x and the existing strategy parameter set X n The covariance matrix between n represents the corresponding evaluation result, σ 2 represents the noise variance, I represents the identity matrix;

[0085] S74. Calculate the expected improvement of the acquisition strategy parameter x according to the posterior distribution of the Gaussian process:

[0086]

[0087] Among them, L(x + ) represents the objective function value of the current optimal acquisition strategy, represents the indicator function. When L(x)>L(x + ) takes 1, otherwise takes 0;

[0088] S75. Select the acquisition strategy parameter with the maximum expected improvement value as the candidate parameter for the next evaluation:

[0089]

[0090] Among them, x next represents the candidate parameters for the next evaluation;

[0091] S76, repeating steps S72-S75 until a predetermined optimization termination condition is met;

[0092] S77, determine the final optimal acquisition strategy parameter x through the Bayesian optimization process * .

[0093] Furthermore, the S8 specifically includes:

[0094] S81, performing initial segmentation on the data source, setting initial segmentation parameters according to the type and data volume of the data source, and dividing the data source into several sub-areas;

[0095] S82, perform characteristic analysis on the data of each sub-region, extract key features of the data, and generate a characteristic description vector V = {v1, v2, ..., v n}, where v i Represents the data feature value of the i-th sub-region;

[0096] S83, based on the characteristic description vector V and the metadata distribution, formulate an adaptive segmentation strategy, use a recursive segmentation algorithm to further segment the data source, and dynamically adjust the segmentation depth by setting a segmentation threshold θ to ensure that the segmented data area has good homogeneity;

[0097] S84. According to the change of data characteristics during the collection process, the data segmentation strategy is dynamically adjusted. For the area where the data characteristics change greatly, the segmentation depth d is increased, and for the area where the data characteristics change less, the segmentation depth is reduced, so as to ensure the flexible adjustment of the data collection range and depth;

[0098] S85. During the data collection process, real-time monitoring of the collection effect of each data area is performed to evaluate the efficiency and quality of the collection. Each data area is evaluated according to the set evaluation index to generate an evaluation report R = {r1, r2, ..., r m}, where r i represents the evaluation result of the i-th data region;

[0099] S86. According to the real-time monitoring and evaluation results, the data segmentation strategy is optimized and adjusted. The feedback information in the evaluation report R is input into the adaptive segmentation module. The segmentation parameters and thresholds are adjusted through the optimization algorithm to achieve dynamic optimization of the segmentation strategy and ensure the efficiency and high quality of data collection.

[0100] The beneficial effects of the present invention are:

[0101] First, the present invention can dynamically adjust the collection strategy. By applying the algorithm combining graph convolutional network and self-attention mechanism to analyze the collection results of metadata, and combining genetic algorithm and deep Q network combined with variational autoencoder and other optimization algorithms, the collection strategy can be dynamically adjusted. Compared with the traditional fixed strategy, this method can better adapt to the changes of data source and data characteristics, and improve the flexibility and adaptability of collection.

[0102] Secondly, the present invention significantly improves the collection efficiency. Through the optimization operation of the genetic algorithm, the collection strategy can be quickly generated and adjusted, which significantly improves the efficiency of metadata collection. Especially in the scenario of large-scale data sources, it can effectively reduce the collection time and resource consumption, and improve the overall collection performance.

[0103] In addition, the present invention also improves data quality. By performing data analysis, cleaning and standardization on metadata and performing quality assessment during the collection process, it is ensured that the collected metadata has high integrity, accuracy and consistency. At the same time, through the application of deep reinforcement learning algorithms and Bayesian optimization algorithms, the collection strategy is further optimized and the data quality is improved.

[0104] The present invention also uses adaptive data segmentation technology. The data collection scope and depth are dynamically adjusted according to the data source characteristics and metadata distribution, which can flexibly deal with the heterogeneity and complexity of the data source and improve the accuracy and efficiency of data collection. It has significant advantages especially when dealing with large-scale heterogeneous data sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0106] Figure 1 A flowchart of a metadata collection method based on a custom strategy optimization algorithm proposed by the present invention;

[0107] Figure 2 A flowchart of a metadata collection method based on a custom strategy optimization algorithm proposed in the present invention, which combines a graph convolutional network and a self-attention mechanism with an algorithm to analyze metadata collection results;

[0108] Figure 3 A schematic diagram of a deep Q network combined with a variational autoencoder for a metadata collection method based on a custom strategy optimization algorithm proposed in the present invention. DETAILED DESCRIPTION

[0109] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0110] refer to Figure 1-3 , a metadata collection method based on a custom strategy optimization algorithm, comprising the following steps:

[0111] S1, extract metadata from data sources;

[0112] S2, perform data analysis, data cleaning and data standardization on the extracted metadata;

[0113] S3. Perform quality assessment on the processed metadata, including completeness, accuracy, and consistency indicators;

[0114] S4. Apply the algorithm combining graph convolutional network and self-attention mechanism to analyze the metadata collection results and propose a preliminary collection strategy;

[0115] S5, using genetic algorithm to optimize the preliminary acquisition strategy, and generating an optimized acquisition strategy through selection, crossover and mutation operations;

[0116] S6. Build an adaptive strategy optimization model, using a deep Q-network combined with a variational autoencoder to learn the optimal acquisition strategy in a dynamic environment;

[0117] S7. Apply the Bayesian optimization algorithm to globally optimize the acquisition strategy by constructing a probability model of the acquisition strategy and using the Gaussian process to estimate the distribution of the objective function;

[0118] S8. Implement adaptive data segmentation technology to dynamically adjust the scope and depth of data collection based on data source characteristics and metadata distribution;

[0119] S9, monitor the collection results in real time through the feedback loop mechanism to evaluate the collection efficiency and quality;

[0120] S10. Provide a user interface to allow the user to configure the initial collection strategy, view the collection results, and adjust parameters.

[0121] In this implementation, S4 specifically includes:

[0122] S41. Use graph convolutional networks to structure metadata representation by embedding metadata nodes into high-dimensional space to capture the relationships and features between nodes.

[0123] S42. In the graph convolutional network, the input data is set to the feature matrix X∈R of the metadata node N×F , where N represents the number of nodes and F represents the feature dimension; the adjacency matrix of the graph is set to A∈R N×N, represents the connection relationship between nodes;

[0124] S43. Calculate node feature updates through graph convolutional layer:

[0125]

[0126] in, I represents the identity matrix, which is used to add self-connection to the adjacency matrix. express The degree matrix, H (l) represents the node feature matrix of the lth layer, H (l+1) represents the node feature matrix of the l+1th layer, W (l) represents the weight matrix of the lth layer, σ represents the activation function;

[0127] S44. In the self-attention mechanism, the feature matrix of the metadata node is set as input to generate the query matrix Q, key matrix K and value matrix V:

[0128] Q=XW Q ,K=XW K ,V=XW V ,Q′=XW Q′ ,K′=XW K′ ,V′=XW V′ ;

[0129] Among them, W Q , W K , W V , W Q′ , W K′ and W V′ represents the trainable weight matrix, Q′ represents the enhanced query matrix, K′ represents the enhanced key matrix, and V′ represents the enhanced value matrix;

[0130] S45. Through the multi-head attention mechanism, the self-attention mechanism is repeated many times to obtain the output matrix MultiHead(Q,K,V,Q′,K′,V′) of the multi-head attention mechanism:

[0131] MultiHead(Q,K,V,Q′,K′,V′)=Concat(head1,head2,…,head h )W O ;

[0132]

[0133] Among them, head i represents the i-th enhanced attention head, represents the trainable weight matrix of the i-th query matrix, represents the trainable weight matrix of the i-th key matrix, represents the trainable weight matrix of the i-th value matrix, represents the trainable weight matrix of the i-th augmented query matrix, represents the trainable weight matrix of the i-th augmented bond matrix, Represents the trainable weight matrix of the i-th enhancement value matrix, W O Represents the output trainable weight matrix of the multi-head attention mechanism.

[0134] S46. Perform residual connection and layer normalization on the output results of the multi-head attention mechanism:

[0135] Output=LayerNorm(X+MultiHead(Q,K,V,Q′,K′,V′));

[0136] Among them, LayerNorm represents the layer normalization operation;

[0137] S47. Propose a preliminary collection strategy.

[0138] In this implementation manner, S5 specifically includes:

[0139] S51, initialize the genetic algorithm parameters, and use the preliminary acquisition strategy as the initial population, each individual in the population represents a acquisition strategy, and each individual is composed of a set of strategy parameters;

[0140] S52. Evaluate each individual in the initial population. The evaluation criteria include collection efficiency, collection quality, and resource consumption. Use the fitness function f(x) to calculate the fitness of each individual. The fitness function is defined as:

[0141]

[0142] Wherein, E(x) represents the collection efficiency, Q(x) represents the collection quality, C(x) represents the resource consumption, w1, w2 and w3 represent the weight parameters, and satisfy w1+w2+w3=1;

[0143] S53, selecting individuals according to fitness values, using roulette wheel selection method, and selecting individuals with higher fitness values ​​from the current population as parent individuals;

[0144] S54, randomly select a crossover point, combine the part of the parent individual before the point with the part of another parent individual after the point to form a child individual, the specific formula is: for the parent individual P1 = (p 1,1 ,p 1,2 ,...,p 1,n ) and P2=(p 2,1 ,p2,2 ,...,p 2,n ), generate offspring individuals C = (c1, c2, ..., c n ),

[0145]

[0146] Where k represents a randomly selected intersection point;

[0147] S55, for the offspring individual C=(c1,c2,...,c n ) Randomly select a mutation point and randomly change the gene value of the point to obtain the mutated offspring individual C′=(c′1,c′2,...,c′ n ):

[0148]

[0149] Among them, k′ represents a randomly selected mutation point;

[0150] S56, evaluating the newly generated offspring individuals, and using the fitness function f(x) to calculate the fitness value of each offspring individual;

[0151] S57, merge the offspring individuals with the individuals in the current population, sort them from high to low according to the fitness value, and select the first N individuals as the new population;

[0152] S58, judging whether the maximum number of iterations has been reached or whether the individual fitness values ​​in the population have reached a preset threshold; if the termination condition is met, outputting the final population as the optimized acquisition strategy; otherwise, returning to S53 to continue iteration.

[0153] In this implementation manner, S6 specifically includes:

[0154] S61, initializing the parameters of the deep Q network combined with the variational autoencoder, defining the state space S, the action space A and the reward function R, wherein the state space includes the current acquisition strategy and metadata distribution characteristics, the action space includes the adjustment operation of the acquisition strategy, and the reward function is defined according to the efficiency and quality of metadata acquisition;

[0155] S62. Through the experience replay mechanism, store the state-action-reward-next state (s, a, r, s′) quadruple collected from the environment and build an experience replay pool D;

[0156] S63, construct a target Q network for generating a target Q value. The parameters of the target Q network are synchronized from the deep Q network combined with the variational autoencoder every certain number of steps. The calculation formula of the target Q value is:

[0157]

[0158] Among them, r represents the current reward value, γ represents the discount factor, represents the maximum Q value of all possible actions a′ in the next state s′, θ - represents the parameters of the target Q grid;

[0159] S64. Use the deep Q network combined with the variational autoencoder to update parameters and introduce latent variables to learn state representation, including:

[0160] Encode the input state and generate the mean μ and variance σ of the latent variable 2 :

[0161] μ=f μ (s;θ enc );

[0162] log2σ 2 =f σ (s;θ enc );

[0163] Among them, f μ and f σ represents the encoder network, θ enc Represents encoder parameters;

[0164] Sampling from the distribution of latent variables generates latent variables z:

[0165]

[0166] Where z represents the latent variable sampled, ∈ represents the noise sampled from the standard normal distribution, represents a standard normal distribution with a mean of 0 and a variance of 1;

[0167] Calculate the reconstruction error and KL divergence, and define the loss function of the variational autoencoder. The formula is as follows:

[0168] L VAE =E q(z|s) [log2p(s|z)]-D KL (q(z|s)||p(z));

[0169] Among them, L VAE represents the loss function of the variational autoencoder, E q(z|s) [log p(s|z)] represents the reconstruction error, D KL represents the KL divergence, which represents the difference between the approximate posterior distribution q(z|s) and the prior distribution p(z);

[0170] The loss function is used to update the parameters of the deep Q network combined with the variational autoencoder. The loss function is:

[0171] L(θ)=E (s,a,r,s′)～ D[(Q(s,a;θ)-Q target ) 2 ]+L VAE ;

[0172] Where L(θ) represents the total loss function, θ represents the parameters of the deep Q network combined with the variational autoencoder, and Q(s,a;θ) represents the actual Q value under the current state s and action a;

[0173] S65, select actions through greedy strategy, select random actions with probability ∈, select actions with maximum Q value in the current state with probability 1-∈, and gradually reduce the ∈ value;

[0174] S66. Execute the selected action in the dynamic environment, update the state and calculate the reward, and repeat steps S62-S65 until the deep Q network combined with the variational autoencoder converges to obtain the local optimal acquisition strategy.

[0175] Enhanced modeling capabilities for complex environments: VAE can learn the distribution of complex environments through latent variables, allowing deep Q networks to perform better when processing high-dimensional and nonlinear data.

[0176] By combining a Deep Q Network (DQN-VAE) with a Variational Autoencoder (VAE), the following benefits can be achieved:

[0177] Improve the generalization ability of the strategy: By introducing VAE, it is possible to better capture the potential structural information in the environment, thereby improving the generalization ability of the strategy and making it applicable to more different data collection scenarios.

[0178] Reduce the risk of overfitting of the model: VAE can effectively reduce the overfitting phenomenon of the model and improve the stability and reliability of the acquisition strategy by regularizing the latent space.

[0179] Improved data efficiency: The deep Q network combined with VAE can use data more efficiently, extract more useful information from the limited experience replay pool, and speed up strategy optimization.

[0180] In this implementation manner, the S7 specifically includes:

[0181] S71, constructing an initial probability model of the acquisition strategy, the objective function is L(x), and the input is the parameter x of the acquisition strategy;

[0182] S72. Use a Gaussian process as a prior model of the objective function L(x), where the mean function of the Gaussian process is μ(x) and the covariance function is k(x, x′);

[0183] S73. In each collection strategy evaluation, record the corresponding collection efficiency and quality evaluation results, update the posterior distribution of the Gaussian process, and calculate the posterior mean μ using the Bayesian theorem. n (x) and covariance k n (x,x′):

[0184] μ n (x) = μ(x) + K(x, X n )[K(X n ,X n )+σ 2 I] -1 (Y n -μ(X n ));

[0185] k n (x,x′)=k(x,x′)-K(x,X n )[K(X n ,X n )+σ 2 I] -1 K(X n ,x′);

[0186] Among them, K(x,X n ) represents the acquisition strategy parameter x and the existing strategy parameter set X n The covariance matrix between n represents the corresponding evaluation result, σ 2 represents the noise variance, I represents the identity matrix;

[0187] S74. Calculate the expected improvement of the acquisition strategy parameter x according to the posterior distribution of the Gaussian process:

[0188]

[0189] Among them, L(x + ) represents the objective function value of the current optimal acquisition strategy, represents the indicator function. When L(x)>L(x + ) takes 1, otherwise takes 0;

[0190] S75. Select the acquisition strategy parameter with the maximum expected improvement value as the candidate parameter for the next evaluation:

[0191]

[0192] Among them, x next represents the candidate parameters for the next evaluation;

[0193] S76, repeating steps S72-S75 until a predetermined optimization termination condition is met;

[0194] S77, determine the final optimal acquisition strategy parameter x through the Bayesian optimization process * .

[0195] In this implementation manner, S8 specifically includes:

[0196] S81, performing initial segmentation on the data source, setting initial segmentation parameters according to the type and data volume of the data source, and dividing the data source into several sub-areas;

[0197] S82, perform characteristic analysis on the data of each sub-region, extract key features of the data, and generate a characteristic description vector V = {v1, v2, ..., v n}, where v i Represents the data feature value of the i-th sub-region;

[0198] S83, based on the characteristic description vector V and the metadata distribution, formulate an adaptive segmentation strategy, use a recursive segmentation algorithm to further segment the data source, and dynamically adjust the segmentation depth by setting a segmentation threshold θ to ensure that the segmented data area has good homogeneity;

[0199] S84. According to the change of data characteristics during the collection process, the data segmentation strategy is dynamically adjusted. For the area where the data characteristics change greatly, the segmentation depth d is increased, and for the area where the data characteristics change less, the segmentation depth is reduced, so as to ensure the flexible adjustment of the data collection range and depth;

[0200] S85. During the data collection process, real-time monitoring of the collection effect of each data area is performed to evaluate the efficiency and quality of the collection. Each data area is evaluated according to the set evaluation indicators to generate an evaluation report R = {r1, r2, ..., r m}, where r i represents the evaluation result of the i-th data region;

[0201] S86. According to the real-time monitoring and evaluation results, the data segmentation strategy is optimized and adjusted. The feedback information in the evaluation report R is input into the adaptive segmentation module. The segmentation parameters and thresholds are adjusted through the optimization algorithm to achieve dynamic optimization of the segmentation strategy and ensure the efficiency and high quality of data collection.

[0202] Embodiment 1:

[0203] In order to verify the feasibility of the present invention in implementation, the present invention is applied to the data management system of a large enterprise. The system has multiple heterogeneous data sources, including SQL databases, RESTful API interfaces and file systems. These data sources generate and update a large amount of data every day, and metadata needs to be efficiently extracted and processed to support data analysis and decision-making. However, the traditional fixed metadata collection strategy is difficult to cope with the dynamic changes of data sources and data characteristics, resulting in low collection efficiency and low data quality, which cannot meet the needs of enterprises for real-time data analysis and decision support.

[0204] To solve the above problems, the company decided to introduce a metadata collection method based on a custom strategy optimization algorithm. First, metadata is extracted from the data source, including SQL queries, API calls, and file reading. The extracted metadata is parsed, cleaned, and standardized to ensure data consistency and availability. The processed metadata is analyzed by an algorithm that combines a graph convolutional network and a self-attention mechanism to capture data features and the relationship between nodes, and propose a preliminary collection strategy.

[0205] Based on the preliminary collection strategy, a genetic algorithm was used for optimization. The genetic algorithm continuously generates new collection strategies by simulating the selection, crossover and mutation operations of biological evolution, and evaluates and selects these strategies. After several generations of iterative optimization, the generated collection strategy has significantly improved in terms of collection efficiency and data quality.

[0206] In order to further optimize the acquisition strategy, a deep Q-network (DQN) combined with a variational autoencoder is introduced. The deep Q-network combined with a variational autoencoder stores the state-action-reward-next-state quadruple collected from the environment through the experience replay mechanism and builds an experience replay pool. Actions are selected through a greedy strategy and executed in a dynamic environment to update the state and calculate the reward. The deep Q-network combined with the variational autoencoder continuously adjusts the acquisition strategy to gradually improve the acquisition efficiency and data quality.

[0207] In the process of optimizing the acquisition strategy, the Bayesian optimization algorithm is applied to globally optimize the strategy by constructing a probability model of the acquisition strategy and using the Gaussian process to estimate the distribution of the objective function. Bayesian optimization can find the optimal acquisition strategy with a smaller number of trials, greatly improving the optimization efficiency and effect.

[0208] The company's data management system implemented the above optimization methods from January to June 2023.

[0209] Table 1. Comparison of overall performance improvement of traditional method and the method of the present invention

[0210]

[0211] From Table 1, we can clearly see that the method of the present invention has significantly improved in all aspects compared with the traditional method. First, the average collection time is shortened from 8 hours in the traditional method to 2 hours, an increase of 75%. This significant improvement shows that through the custom strategy optimization algorithm, the metadata collection process has been greatly accelerated, improving the overall work efficiency, allowing enterprises to obtain the required data support faster.

[0212] Secondly, the quality of metadata has been significantly improved. Specifically, the completeness of metadata has been improved from 85% of the traditional method to 95%, the accuracy has been improved from 80% to 92%, and the consistency has been improved from 82% to 90%. These data show that through the optimized collection strategy, the completeness, accuracy and consistency of the collected metadata have been greatly improved. This is of great significance to data analysis and decision support, ensuring the reliability and availability of data.

[0213] Finally, resource consumption and user experience have also been significantly improved. The CPU usage of the optimized method has dropped from 85% to 65%, and the memory usage has dropped from 75% to 60%, which shows that the optimized method is more efficient and reduces the occupation of system resources. In addition, user satisfaction has increased from 70% to 90%, indicating that the optimized system has not only improved technical performance, but also improved the user experience. These improvements further verify the effectiveness and advantages of the method of the present invention in practical applications.

[0214] Through the above embodiments, the present invention fully demonstrates the beneficial effects in practical applications. The metadata collection method based on the custom strategy optimization algorithm not only significantly improves the collection efficiency and data quality, but also enhances the adaptability and flexibility of the system, reduces resource consumption, and improves user satisfaction, which has important practical application value and innovation.

[0215] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A metadata collection method based on a custom strategy optimization algorithm, characterized in that: The steps include: S1, extract metadata from data sources; S2, perform data analysis, data cleaning and data standardization on the extracted metadata; S3. Perform quality assessment on the processed metadata, including completeness, accuracy, and consistency indicators; S4. Apply the algorithm combining graph convolutional network and self-attention mechanism to analyze the metadata collection results and propose a preliminary collection strategy; S5, using genetic algorithm to optimize the preliminary acquisition strategy, and generating an optimized acquisition strategy through selection, crossover and mutation operations; S6. Build an adaptive strategy optimization model, using a deep Q-network combined with a variational autoencoder to learn the optimal acquisition strategy in a dynamic environment; S7. Apply the Bayesian optimization algorithm to globally optimize the acquisition strategy by constructing a probability model of the acquisition strategy and using the Gaussian process to estimate the distribution of the objective function; S8. Implement adaptive data segmentation technology to dynamically adjust the scope and depth of data collection based on data source characteristics and metadata distribution; S9, monitor the collection results in real time through the feedback loop mechanism to evaluate the collection efficiency and quality; S10. Provide a user interface to allow the user to configure the initial collection strategy, view the collection results, and adjust parameters.

2. According to claim 1, a metadata collection method based on a custom strategy optimization algorithm is characterized in that: The S4 specifically includes: S41. Use graph convolutional networks to structure metadata representation by embedding metadata nodes into high-dimensional space to capture the relationships and features between nodes. S42. In the graph convolutional network, the input data is set to the feature matrix X∈R of the metadata node N×F , where N represents the number of nodes and F represents the feature dimension; the adjacency matrix of the graph is set to A∈R N×N , represents the connection relationship between nodes; S43. Calculate node feature updates through graph convolutional layer: in, I represents the identity matrix, which is used to add self-connection to the adjacency matrix. express The degree matrix, H (l) represents the node feature matrix of the lth layer, H (l+1) represents the node feature matrix of the l+1th layer, W (l) represents the weight matrix of the lth layer, σ represents the activation function; S44. In the self-attention mechanism, the feature matrix of the metadata node is set as input to generate the query matrix Q, key matrix K and value matrix V: Q=XW Q ,K=XW K ,V=XW V ,Q′=XW Q′ ,K′=XW K′ ,V′=XW V′ ; Among them, W Q , W K , W V , W Q′ , W K′ and W V′ represents the trainable weight matrix, Q′ represents the enhanced query matrix, K′ represents the enhanced key matrix, and V′ represents the enhanced value matrix; S45. Through the multi-head attention mechanism, the self-attention mechanism is repeated many times to obtain the output matrix MultiHead(Q,K,V,Q′,K′,V′) of the multi-head attention mechanism: MultiHead(Q,K,V,Q′,K′,V′)=Concat(head1,head2,…,head h )W O ; Among them, head i represents the i-th enhanced attention head, represents the trainable weight matrix of the i-th query matrix, represents the trainable weight matrix of the i-th key matrix, represents the trainable weight matrix of the i-th value matrix, represents the trainable weight matrix of the i-th augmented query matrix, represents the trainable weight matrix of the i-th augmented bond matrix, Represents the trainable weight matrix of the i-th enhancement value matrix, W O Represents the output trainable weight matrix of the multi-head attention mechanism. S46. Perform residual connection and layer normalization on the output results of the multi-head attention mechanism: Output=LayerNorm(X+MultiHead(Q,K,V,Q′,K′,V′)); Among them, LayerNorm represents the layer normalization operation; S47. Propose a preliminary collection strategy.

3. The metadata collection method based on a custom strategy optimization algorithm according to claim 1, characterized in that: The S5 specifically includes: S51, initialize the genetic algorithm parameters, and use the preliminary acquisition strategy as the initial population, each individual in the population represents a acquisition strategy, and each individual is composed of a set of strategy parameters; S52. Evaluate each individual in the initial population. The evaluation criteria include collection efficiency, collection quality, and resource consumption. Use the fitness function f(x) to calculate the fitness of each individual. The fitness function is defined as: Wherein, E(x) represents the collection efficiency, Q(x) represents the collection quality, C(x) represents the resource consumption, w1, w2 and w3 represent the weight parameters, and satisfy w1+w2+w3=1; S53, selecting individuals according to fitness values, using roulette wheel selection method, and selecting individuals with higher fitness values ​​from the current population as parent individuals; S54, randomly select a crossover point, combine the part of the parent individual before the point with the part of another parent individual after the point to form a child individual, the specific formula is: for the parent individual P1 = (p 1,1 ,p 1,2 ,...,p 1,n ) and P2=(p 2,1 ,p 2,2 ,...,p 2,n ), generate offspring individuals C = (c1, c2, ..., c n ), Where k represents a randomly selected intersection point; S55, for the offspring individual C=(c1,c2,...,c n ) Randomly select a mutation point and randomly change the gene value of the point to obtain the mutated offspring individual C′=(c′1,c′2,...,c′ n ): Among them, k′ represents a randomly selected mutation point; S56, evaluating the newly generated offspring individuals, and using the fitness function f(x) to calculate the fitness value of each offspring individual; S57, merge the offspring individuals with the individuals in the current population, sort them from high to low according to the fitness value, and select the first N individuals as the new population; S58, judging whether the maximum number of iterations has been reached or whether the individual fitness values ​​in the population have reached a preset threshold; if the termination condition is met, outputting the final population as the optimized acquisition strategy; otherwise, returning to S53 to continue iteration.

4. The metadata collection method based on a custom strategy optimization algorithm according to claim 1 is characterized in that: The S6 specifically includes: S61, initializing the parameters of the deep Q network combined with the variational autoencoder, defining the state space S, the action space A and the reward function R, wherein the state space includes the current acquisition strategy and metadata distribution characteristics, the action space includes the adjustment operation of the acquisition strategy, and the reward function is defined according to the efficiency and quality of metadata acquisition; S62. Through the experience replay mechanism, store the state-action-reward-next state (s, a, r, s′) quadruple collected from the environment and build an experience replay pool D; S63, construct a target Q network for generating a target Q value. The parameters of the target Q network are synchronized from the deep Q network combined with the variational autoencoder every certain number of steps. The calculation formula of the target Q value is: Among them, r represents the current reward value, γ represents the discount factor, represents the maximum Q value of all possible actions a′ in the next state s′, θ - represents the parameters of the target Q grid; S64. Use the deep Q network combined with the variational autoencoder to update parameters and introduce latent variables to learn state representation, including: Encode the input state and generate the mean μ and variance σ of the latent variable 2 : μ=f μ (s;θ enc ); log2σ 2 =f σ (s;θ enc ); Among them, f μ and f σ represents the encoder network, θ enc Represents encoder parameters; Sampling from the distribution of latent variables generates latent variables z: Where z represents the sampled latent variable, ∈ represents the noise sampled from the standard normal distribution, represents a standard normal distribution with a mean of 0 and a variance of 1; Calculate the reconstruction error and KL divergence, and define the loss function of the variational autoencoder. The formula is as follows: L VAE =E q(z|s) [log2 p(s|z)]-D KL (q(z|s)||p(z)); Among them, L VAE represents the loss function of the variational autoencoder, E q(z|s) [log p(s|z)] represents the reconstruction error, D KL represents the KL divergence, which represents the difference between the approximate posterior distribution q(z|s) and the prior distribution p(z); The loss function is used to update the parameters of the deep Q network combined with the variational autoencoder. The loss function is: L(θ)=E (s,a,r,s′)～ D[(Q(s,a;θ)-Q target ) 2 ]+L VAE ; Where L(θ) represents the total loss function, θ represents the parameters of the deep Q network combined with the variational autoencoder, and Q(s,a;θ) represents the actual Q value under the current state s and action a; S65, select actions through greedy strategy, select random actions with probability ∈, select actions with maximum Q value in the current state with probability 1-∈, and gradually reduce the ∈ value; S66. Execute the selected action in the dynamic environment, update the state and calculate the reward, and repeat steps S62-S65 until the deep Q network combined with the variational autoencoder converges to obtain the local optimal acquisition strategy.

5. The metadata collection method based on a custom strategy optimization algorithm according to claim 1 is characterized in that: The S7 specifically includes: S71, constructing an initial probability model of the acquisition strategy, the objective function is L(x), and the input is the parameter x of the acquisition strategy; S72. Use a Gaussian process as a prior model of the objective function L(x), where the mean function of the Gaussian process is μ(x) and the covariance function is k(x, x′); S73. In each collection strategy evaluation, record the corresponding collection efficiency and quality evaluation results, update the posterior distribution of the Gaussian process, and use Bayesian theorem to calculate the posterior mean μ n (x) and covariance k n (x,x′): m n (x)=μ(x)+K(x,X n )[K(X n ,X n )+s 2 I] -1 (Y n -μ(X n )); k n (x,x′)=k(x,x′)-K(x,X n )[K(X n ,X n )+σ 2 I] -1 K(X n ,x′); Among them, K(x,X n ) represents the acquisition strategy parameter x and the existing strategy parameter set X n The covariance matrix between n represents the corresponding evaluation result, σ 2 represents the noise variance, I represents the identity matrix; S74. Calculate the expected improvement of the acquisition strategy parameter x according to the posterior distribution of the Gaussian process: Among them, L(x + ) represents the objective function value of the current optimal acquisition strategy, represents the indicator function. When L(x)>L(x + ) takes 1, otherwise takes 0; S75. Select the acquisition strategy parameter with the maximum expected improvement value as the candidate parameter for the next evaluation: Among them, x next represents the candidate parameters for the next evaluation; S76, repeating steps S72-S75 until a predetermined optimization termination condition is met; S77, determine the final optimal acquisition strategy parameter x through the Bayesian optimization process * .

6. The metadata collection method based on a custom strategy optimization algorithm according to claim 1, characterized in that: The S8 specifically includes: S81, performing initial segmentation on the data source, setting initial segmentation parameters according to the type and data volume of the data source, and dividing the data source into several sub-areas; S82, perform characteristic analysis on the data of each sub-region, extract key features of the data, and generate a characteristic description vector V = {v1, v2, ..., v n }, where v i Represents the data feature value of the i-th sub-region; S83, based on the feature description vector V and the metadata distribution, formulate an adaptive segmentation strategy, use a recursive segmentation algorithm to further segment the data source, and dynamically adjust the segmentation depth by setting a segmentation threshold θ; S84, dynamically adjusting the data segmentation strategy according to changes in data characteristics during the acquisition process, increasing the segmentation depth d for areas with large changes in data characteristics, and reducing the segmentation depth d for areas with small changes in data characteristics; S85. During the data collection process, real-time monitoring of the collection effect of each data area is performed to evaluate the efficiency and quality of the collection. Each data area is evaluated according to the set evaluation index to generate an evaluation report R = {r1, r2, ..., r m }, where r i represents the evaluation result of the i-th data region; S86. According to the real-time monitoring and evaluation results, the data segmentation strategy is optimized and adjusted, the feedback information in the evaluation report R is input into the adaptive segmentation module, and the segmentation parameters and threshold θ are adjusted through the optimization algorithm to achieve dynamic optimization of the segmentation strategy.

Images