Dataset construction method and system for training large models in professional fields

Abstract

The present invention discloses a data set construction method and system for training large models in professional fields, the method includes collecting multi-source heterogeneous data, fusing and preprocessing the data through a dynamic time series tensor decomposition method, and generating a standardized three-dimensional data tensor; using a hierarchical graph structure processing method to reorganize and optimize the data, generating an optimized feature matrix, a feature importance vector, and a data reliability matrix; using a probability matrix decomposition method to enhance the data and fuse knowledge to generate an enhanced data set; using a polyhedral projection method to verify and dynamically optimize the data to generate a final training data set. The present invention improves the quality of professional field data and solves technical problems such as insufficient data time series correlation analysis, inaccurate quality assessment, fixed enhancement strategy, and insufficient verification constraints.

Description

Technical Field

[0001] The present invention belongs to the field of large model data construction, and in particular to a data set construction method and system for training large models in professional fields. Background Art

[0002] With the rapid development of artificial intelligence technology, large models are increasingly used in various professional fields. By introducing domain-specific data in the training phase, large models in professional fields can significantly improve their task-solving capabilities and knowledge understanding depth in specific fields. High-quality professional field data sets can not only help the model accurately understand domain terminology, master the professional knowledge system, and grasp the development trends of the field, but also improve the model's performance in tasks such as professional question-answering, decision analysis, and knowledge reasoning. Therefore, building a high-quality data set suitable for training large models in professional fields has important theoretical significance and practical value for improving the application level of artificial intelligence in professional fields.

[0003] At present, the construction of professional field data sets mainly adopts a combination of manual annotation and automatic crawling. In terms of manual annotation, experts in the field are hired to organize and annotate literature and establish training data such as question-answer pairs and knowledge graphs; in terms of automatic crawling, structured and unstructured data are obtained from professional websites and databases, and preliminary processing is performed through methods such as rule matching and pattern recognition. In terms of data quality control, it mainly relies on manual review and simple statistical screening methods, such as keyword matching, length filtering, and integrity checking. For the timeliness maintenance of data sets, regular updates are generally adopted to expand data sets by incrementally adding new data.

[0004] However, the existing dataset construction methods have the following technical problems: First, in terms of data organization, the temporal correlation between data is not fully considered, which makes it difficult for the model to accurately grasp the evolution law and development trend of knowledge; second, data quality assessment relies too much on simple statistical features and lacks in-depth analysis of the internal structure and semantic association of data, which affects the overall quality of the dataset; third, in the data enhancement link, the existing methods often adopt fixed enhancement strategies and fail to dynamically adjust the enhancement parameters according to the data characteristics, which limits the diversity of data; fourth, in the data verification process, the collaborative constraints of multi-dimensional features are not considered enough, which makes it difficult to ensure the rationality of enhanced data; fifth, there is a lack of precise quantification and dynamic optimization mechanism for data consistency, which affects the internal consistency of the dataset; sixth, in the process of knowledge fusion, the structural differences and semantic conflicts between data from different sources are not effectively handled, which reduces the fusion effect; finally, the existing methods do not pay enough attention to the timeliness characteristics of data and are difficult to respond to the rapid update of professional knowledge in a timely manner. These technical problems seriously restrict the training effect and application efficiency of large models in professional fields. Summary of the invention

[0005] The purpose of the invention is to provide a method and system for constructing a data set for training large models in professional fields, so as to solve the above-mentioned problems existing in the prior art.

[0006] The technical solution is a method for constructing a data set for training a large model in a professional field, including the following steps:

[0007] S1. Collect pre-stored multi-source heterogeneous data, fuse and pre-process the multi-source heterogeneous data through the dynamic time series tensor decomposition method, and generate a standardized three-dimensional data tensor;

[0008] S2, based on the standardized three-dimensional data tensor, a hierarchical graph structure processing method is used for reorganization and optimization to generate an optimized feature matrix, feature importance vector and data reliability matrix;

[0009] S3, based on the optimized feature matrix, feature importance vector and data reliability matrix, the probability matrix decomposition method is used for enhancement and knowledge fusion to generate an enhanced data set;

[0010] S4. Based on the enhanced dataset, the polyhedron projection method is used for verification and dynamic optimization to generate the final training dataset.

[0011] A dataset construction system for training large models in professional fields, including:

[0012] at least one processor; and,

[0013] a memory communicatively connected to at least one of the processors; wherein,

[0014] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the data set construction method for training a large model in a professional field.

[0015] Beneficial effects: The present invention realizes efficient fusion and quality optimization of heterogeneous data, improves the efficiency of data organization and the quality of feature extraction; realizes reliable expansion of data and deep integration of knowledge, and ensures all-round quality assurance of data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the present invention.

[0017] Figure 2 This is a flow chart of step S1 of the present invention.

[0018] Figure 3 This is a flow chart of step S2 of the present invention.

[0019] Figure 4 This is a flow chart of step S3 of the present invention.

[0020] Figure 5 This is a flow chart of step S4 of the present invention. DETAILED DESCRIPTION

[0021] like Figure 1 As shown, this application proposes a method for constructing a data set for training a large model in a professional field, comprising the following steps:

[0022] S1. Collect pre-stored multi-source heterogeneous data, fuse and pre-process the multi-source heterogeneous data through the dynamic time series tensor decomposition method, and generate a standardized three-dimensional data tensor;

[0023] S2, based on the standardized three-dimensional data tensor, a hierarchical graph structure processing method is used for reorganization and optimization to generate an optimized feature matrix, feature importance vector and data reliability matrix;

[0024] S3, based on the optimized feature matrix, feature importance vector and data reliability matrix, the probability matrix decomposition method is used for enhancement and knowledge fusion to generate an enhanced data set;

[0025] S4. Based on the enhanced dataset, the polyhedron projection method is used for verification and dynamic optimization to generate the final training dataset.

[0026] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:

[0027] S11. Based on preset data dimension parameters, obtain pre-stored multi-source heterogeneous data; map the multi-source heterogeneous data to a four-dimensional data space to construct an initial four-dimensional data tensor; wherein the first dimension represents the data source type, the second dimension represents the data time information, the third dimension represents the data feature attribute, and the fourth dimension represents the data quality indicator; each element value in the initial four-dimensional data tensor represents a specific feature value of a specific data source at a specific time point and its corresponding quality measurement value;

[0028] S12. Based on the time dimension data of the initial four-dimensional data tensor, construct a time dimension matrix, calculate the time interval between any two time points; calculate the time decay value based on the time interval and the preset time decay coefficient; determine the time correlation coefficient based on the preset time threshold parameter; use the product of the time decay value and the time correlation coefficient as the corresponding element value to generate a time series weight matrix;

[0029] S13, extracting feature dimension data from the initial four-dimensional data tensor, and calculating the cosine similarity between feature vectors based on the feature dimension data; obtaining feature co-occurrence frequency, taking the sum of the cosine similarity and the logarithm of the feature co-occurrence frequency as the feature association, and generating a feature association matrix;

[0030] S14, constructing a tensor decomposition optimization function based on the initial four-dimensional data tensor, the time series weight matrix and the feature association matrix; solving the tensor decomposition optimization function through iterative optimization to obtain a reconstructed data tensor; wherein the tensor decomposition optimization function includes a tensor reconstruction term, a time series constraint term and a feature association constraint term;

[0031] S15, extracting quality dimension data of the reconstructed data tensor, and performing weighted median filtering within a time window of a preset size based on the quality dimension data, wherein the weight value is determined by the credibility of the data source, to generate a quality score vector;

[0032] S16. Based on the quality score vector, weighted sum is performed on the quality dimensions of the reconstructed data tensor to obtain a four-dimensional data tensor; and the four-dimensional data tensor is compressed into a standardized three-dimensional data tensor, wherein information on the data source dimension, the time dimension, and the feature dimension is retained.

[0033] In one embodiment of the present application, pre-stored multi-source heterogeneous data is obtained to construct a fourth-order basic data tensor T∈R M×N×P×Q , where M is the data source dimension (text, table, image, etc.), N is the time dimension (data generation time), P is the feature dimension (data attribute), and Q is the quality dimension (data reliability); construct the tensor element t_mnpq, which represents the data item with feature p from the mth data source at time point n, and its quality metric is q. Construct the time series decay matrix W∈R N×N , W_ij=exp(-α|t_i-t_j|)*β_ij, where t_i and t_j represent different time points, α is the time attenuation coefficient, β_ij is the time series correlation coefficient, and is constructed by piecewise function: β_ij={1.0, if |t_i - t_j|≤δ; 0.8, if δ< |t_i-t_j|≤2δ; 0.5, if 2δ<|t_i-t_j|≤3δ; 0.2, other cases}, δ is the time threshold parameter. Construct the feature association matrix C∈R P ×P , c_ij=cos(v_i, v_j)*(1+log(1+f_ij)), where v_i, v_j are feature vectors, f_ij is the feature co-occurrence frequency, and cos(v_i, v_j) is the vector cosine similarity. Construct an improved tensor decomposition objective function: min_X||TX|| F 2 +λ_1tr(X T WX)+λ_2||C⊕X||1, where X is the reconstruction tensor, || || F is the Frobenius norm, tr( ) is the trace of the matrix, ⊕ is the Hadamard product, λ_1 and λ_2 are the balance parameters. Construct the quality score vector q∈R Q, apply weighted median filtering: q'i = median(q(ik):q_(i+k))*γ_i, where k is the filter window size and γ_i is the data source credibility weight. Compress the four-dimensional tensor T into a three-dimensional tensor T'∈R M×N×P , t'_mnp =∑(q=1 to Q) t_mnpq * q'_q / ∑(q=1 to Q) q'_q.

[0034] In one embodiment of the present application, the sources of multi-source heterogeneous data include professional domain shared knowledge and simulation platforms. Professional domain shared knowledge data is collected to build a large model secondary training data set; specific scenarios are simulated through the simulation platform, and simulation data is collected to build an instruction fine-tuning data set.

[0035] Acquire shared knowledge data in professional fields, including game data in forums, professional field regulations and rules, Wiki Chinese and other matching content. The network can automatically crawl web page content. Its working principle is to communicate through the HTTP protocol and download corresponding resources from various websites or servers. The network starts with a pre-defined seed URL, which is the starting point for accessing the Internet. Set the seed URL to the web page we want to crawl; send an HTTP request to the seed URL, and the server will return the corresponding web page content after receiving the request; store, process and analyze the web page content obtained after parsing, extract useful information and save it as a text file.

[0036] Collecting and analyzing simulation data through the simulation platform can further enrich the diversity and practicality of the data set, and generate real-time tasks and evaluate the effects. The simulation platform can create models and tasks, adjust parameters to run simulations, and conduct real-time interactions. Based on the simulation platform, specific scenarios are simulated, and these simulation data are automatically collected by writing scripts to form typical scenario question-answer data. In the simulation platform, each scenario corresponds to a configuration file, which includes background information, task description, task time, task list and other information. For the configuration file, scripts are written to interact with the simulation platform, and simulation process data and simulation results are automatically extracted. After this part of the data is sorted, a relatively complete and accurate question-answer data is formed. The question contains two parts: "instruction" and "input", and the answer contains "output". "Instruction" is a description of the scenario, "input" is a description of the task and goal, and "output" is a decision plan for the task, aiming to improve the dialogue and decision-making capabilities of large models in professional fields.

[0037] This embodiment fuses and preprocesses multi-source heterogeneous data, and realizes efficient unified representation and quality optimization of data. By constructing a four-dimensional data tensor, the data source type, time information, feature attributes and quality indicators are uniformly encoded, so that data from different sources and in different forms can be expressed and processed in a unified mathematical space. By introducing the time series decay matrix and the time series correlation coefficient, a dynamic weight mechanism of the data time dimension is established, so that data with similar time have stronger correlation, thereby more accurately describing the time series evolution characteristics of the data. Through the construction of the feature association matrix, the cosine similarity of the feature vector is combined with the co-occurrence frequency, which not only considers the numerical similarity of the features, but also introduces statistical laws to improve the reliability of feature association. By constructing a tensor decomposition optimization function containing reconstruction terms, time series constraints and feature association constraints, data noise reduction and structural optimization are achieved, and data quality is further improved through weighted median filtering. This embodiment improves the fusion effect of heterogeneous data and lays a good foundation for subsequent data processing.

[0038] According to one aspect of the present application, step S14 is further:

[0039] S141, constructing a rank parameter set for tensor decomposition based on the initial four-dimensional data tensor, the time series weight matrix and the feature association matrix; performing singular value decomposition on the initial four-dimensional data tensor based on the rank parameter set to obtain a decomposition basis matrix; selecting a main component from the decomposition basis matrix based on a preset cutoff threshold to generate an initial decomposition tensor;

[0040] S142, based on the initial decomposition tensor, construct a gradient calculation function, calculate the gradient value of the tensor reconstruction item, and obtain a reconstruction gradient matrix; based on the timing weight matrix, calculate the gradient value of the timing constraint item, and obtain the timing gradient matrix; based on the feature association matrix, calculate the gradient value of the feature association constraint item, and obtain the association gradient matrix; weightedly combine the reconstruction gradient matrix, the timing gradient matrix, and the association gradient matrix to form a comprehensive gradient matrix;

[0041] S143, constructing an adaptive learning rate function based on the comprehensive gradient matrix; obtaining the current iteration round, and calculating the basic learning rate based on the current iteration round and the adaptive learning rate function; calculating the momentum term based on the pre-stored historical gradient information; combining the basic learning rate with the momentum term to obtain an optimized step size matrix;

[0042] S144, based on the initial decomposition tensor and the optimized step size matrix, perform a parameter update operation to obtain an updated decomposition tensor; calculate a reconstruction error between the updated decomposition tensor and the initial four-dimensional data tensor to obtain an error metric value;

[0043] S145. Collect error metric values ​​of consecutive predetermined rounds and construct a convergence judgment criterion; based on the convergence judgment criterion, calculate the rate of change of the error metric value to obtain a convergence index value; when the convergence index value is less than a preset threshold, determine the corresponding updated decomposition tensor as the reconstructed data tensor.

[0044] This embodiment achieves high-quality reconstruction and optimization of data. Through the set of rank parameters of tensor decomposition, multi-dimensional joint optimization of data is achieved. During the optimization process, the initial decomposition tensor is obtained through singular value decomposition and principal component selection, and the reconstruction gradient, timing gradient and association gradient are weighted and combined by constructing a comprehensive gradient matrix to achieve precise control of the optimization direction. By introducing an adaptive learning rate mechanism, combined with the basic learning rate and momentum term, dynamic adjustment of the optimization step size is achieved, and the stability and convergence of the optimization process are improved. By constructing a convergence judgment criterion based on the error metric, the reliability of the optimization results is ensured. This embodiment not only improves the quality of data reconstruction, but also ensures the effective maintenance of timing features and association features, providing a high-quality foundation for subsequent data processing.

[0045] According to one aspect of the present application, step S15 is further:

[0046] S151, extracting the quality dimension data of the reconstructed data tensor, segmenting the quality dimension data based on a preset time window size to obtain a predetermined time window data; calculating statistical features based on each time window data to obtain a window feature matrix; calculating the data distribution difference between adjacent windows based on the window feature matrix to obtain a window difference vector; determining the optimal window division based on the window feature matrix and the window difference vector, and generating a time window sequence;

[0047] S152, based on the time window sequence, extract the credibility information of the data source in each time window, and construct a hierarchical weight calculation model; based on the hierarchical weight calculation model, calculate the historical performance score of the data source to obtain a historical weight vector; based on the historical weight vector, evaluate the current state of the data source to obtain a state weight vector; predict the future reliability of the data source to obtain a predicted weight vector; and obtain a comprehensive weight vector by nonlinearly combining the historical weight vector, the state weight vector, and the predicted weight vector;

[0048] S153. Collect the quality dimension data and comprehensive weight vector in each time window to construct an adaptive filter; based on the adaptive filter, calculate the local correlation of the data to obtain a correlation matrix; based on the correlation matrix, construct filter kernel parameters; based on the filter kernel parameters, perform weighted median filtering operations to generate a quality score vector.

[0049] This embodiment realizes accurate assessment and optimization of data quality. Through segmented processing based on time windows, the window feature matrix and window difference vector are calculated, and dynamic assessment of data quality is realized. By constructing a hierarchical weight calculation model, a comprehensive weight vector is generated by comprehensively considering the historical performance, current status and future reliability of the data source, thereby improving the accuracy and comprehensiveness of the quality assessment. In the filtering processing link, by constructing an adaptive filter and combining the local correlation of the data, accurate control of the weighted median filter is achieved, effectively reducing the impact of noise. This embodiment not only improves the accuracy of data quality assessment, but also realizes dynamic optimization of quality control through an adaptive weight mechanism, providing reliable quality assurance for the construction of a data set.

[0050] like Figure 3 As shown, according to one aspect of the present application, step S2 is further:

[0051] S21, extracting data source dimension information and feature dimension information from the standardized three-dimensional data tensor, and constructing a three-layer data node set; based on the association relationship between the nodes in the three-layer data node set, constructing an edge set from source data to relationship and an edge set from relationship to target feature, to form an initial hierarchical graph structure; wherein the first layer of the three-layer data node set is a source data node set, the second layer is a relationship node set, and the third layer is a target feature node set;

[0052] S22, based on the node pairs in the initial hierarchical graph structure, calculate the graph distance between the nodes; based on the graph distance and the preset distance scale parameter, calculate the distance attenuation factor; collect the observation frequency of the edge, extract the maximum observation frequency therein, and calculate the frequency normalization factor based on the maximum observation frequency; based on the node pairs in the initial hierarchical graph structure, calculate the node feature vector angle and the time series correlation coefficient, and determine the correlation factor; use the product of the distance attenuation factor, the frequency normalization factor and the correlation factor as the weight value of the corresponding edge to generate a weighted hierarchical graph;

[0053] S23. Based on the structural characteristics of the weighted hierarchical graph, a graph optimization objective function is constructed, including the product term of edge weight and edge length, the node quantity constraint term, the edge quantity constraint term and the transition redundancy constraint term; the graph optimization objective function is solved by iterative optimization to obtain an optimized hierarchical graph; wherein the transition redundancy is calculated according to the sum of squares of the difference between the actual degree of the node and the ideal degree;

[0054] S24, extracting the target feature node in the optimization hierarchical graph, and calculating the local structure feature, global position feature and time series evolution feature of each node based on the target feature node; combining the local structure feature, global position feature and time series evolution feature to form a node feature vector; wherein the local structure feature is calculated by the weighted sum of neighboring nodes, the global position feature is calculated by the cumulative product of path weights, and the time series evolution feature is calculated by combining preset time series feature functions;

[0055] S25. Based on the node feature vector, construct a feature fusion matrix, calculate the Frobenius norm between the feature fusion matrix and the preconfigured graph structure reconstruction feature matrix; based on the Frobenius norm, verify the feature fusion matrix to obtain a verification result; based on the verification result, calculate a feature score based on the local structure and a feature score based on the global consistency; based on the feature score based on the local structure, the feature score based on the global consistency and a preset balance factor, calculate the feature reliability score and generate a feature importance vector;

[0056] S26. Based on the dimension of the node feature vector, a sample-feature matrix is ​​constructed as an optimized feature matrix; based on the feature reliability scores, a reliability relationship matrix between samples is constructed as a data reliability matrix.

[0057] In one embodiment of the present application, a three-layer graph structure G=(V, E) is constructed based on a three-dimensional tensor T', V={V_s∪V_r∪V_t}, where V_s is a set of source data nodes from the M dimension of T'; V_r is a set of relationship nodes, representing the association relationship between data; V_t is a set of target feature nodes from the P dimension of T'; E={E_sr∪E_rt}, where E_sr is an edge set from source data to relationship, and E_rt is an edge set from relationship to target feature. Construct the edge weight update function: w(e_ij)=ρ_ij*η_ij*μ_ij, where ρ_ij=exp(-d_ij / σ), represents the distance decay factor, d_ij is the graph distance between nodes i and j, and σ is the distance scale parameter; η_ij=(1+log(f_ij)) / (1+log(f_max)), represents the frequency normalization factor, f_ij is the observation frequency of edge e_ij, and f_max is the maximum observation frequency; μ_ij=cos(θ_ij)*(1+|corr_ij|), represents the correlation factor, θ_ij is the node feature vector angle, and corr_ij is the time series correlation coefficient.

[0058] Construct the graph structure optimization objective function: min_GΩ(G)=∑(e∈E)w(e)*len(e)+λ1|V|+λ2|E|+λ3TR(G), where len(e) is the length of the edge (feature space distance), |V|, |E| are the cardinalities of the node set and edge set, and TR(G) is the transition redundancy of the graph, defined as: TR(G)=∑(v∈V)(deg(v)-deg_opt(v)) 2 , deg(v) is the degree of node v, deg_opt(v) is the ideal degree of node v (estimated by data distribution), λ1, λ2 and λ3 are balance parameters. For each target feature node v_t, construct a feature vector: F(v_t)=[φ1(v_t), φ2(v_t), φ3(v_t)], where φ1 is the local structural feature, φ1(v_t) =∑(u∈N(v_t)) w(e_ut) * h(u), N(v_t) is the neighbor set of node v_t, h(u) is the implicit representation of node u, w(e_ut) represents the weight of edge e_ut; φ2 is the global position feature, φ2(v_t)=∑(p∈P(v_t))∏(e∈p) w(e), where ∏ represents the product operator, w(e) represents the weight of edge e, P(v_t) represents the set of all paths to the target feature node v_t; φ3 is the temporal evolution feature, φ3(v_t)=[ψ_1(t), ψ_2(t), ..., ψ_k(t)], where ψ_ i(t) is the i-th temporal feature function.

[0059] Construct feature fusion matrix M∈R |V_t|×d , where d is the fused feature dimension, and the consistency constraint is applied: ||M-M'|| F ≤ε, where M' is the feature matrix reconstructed by the graph structure, ε is the consistency threshold; construct the feature reliability score: s(f)= (1-α)s_local(f)+αs_global(f), where s_local is the score based on the local structure, s_global is the score based on the global consistency, and α is the balance factor. Generate a standardized feature matrix X∈R n×d , n is the number of samples, d is the optimized feature dimension; generate feature importance vector w∈R d , generate data reliability matrix R∈R n×n .

[0060] This embodiment realizes deep-level organization optimization and feature extraction of data. A three-layer graph structure including source data nodes, relationship nodes and target feature nodes is constructed, and the multi-dimensional association between data is accurately portrayed through the dynamic update mechanism of edge weights. In particular, in the calculation of edge weights, the distance attenuation factor, frequency normalization factor and correlation factor are comprehensively considered, so that the graph structure can more accurately reflect the intrinsic connection of the data. By introducing the graph structure optimization objective function, not only the edge weight and length are considered, but also the node number constraint and transition redundancy constraint are introduced, thereby realizing the adaptive optimization of the graph structure. In the feature extraction link, by jointly considering local structural features, global position features and time series evolution features, a multi-dimensional node feature vector is constructed, which improves the integrity and accuracy of feature expression. Through feature fusion and consistency verification mechanism, the reliability and stability of extracted features are ensured. This embodiment not only improves the efficiency of data organization, but also enhances the quality of feature extraction, providing a reliable foundation for subsequent data enhancement and optimization.

[0061] According to one aspect of the present application, step S21 is further:

[0062] S211, based on the standardized three-dimensional data tensor, extract the data source dimension information and construct the source node feature matrix; based on the source node feature matrix, calculate the information entropy for each data source to obtain the entropy value vector; based on the entropy value vector, perform hierarchical clustering to obtain the source node hierarchy tree; based on the preset hierarchical threshold, extract the node set from the source node hierarchy tree to generate the source data node set;

[0063] S212, based on the standardized three-dimensional data tensor, extracting feature dimension data, constructing a feature node mapping function; based on the feature node mapping function, calculating the mutual information between features to obtain an information matrix; performing spectral clustering on the information matrix to obtain a feature clustering matrix; based on the feature clustering matrix, extracting representative features to form a target feature node set;

[0064] S213, extracting node attributes of the source data node set and the target feature node set, and constructing a relational reasoning model; based on the relational reasoning model, calculating the association strength between node pairs, and obtaining an association strength matrix; performing threshold segmentation on the association strength matrix, and obtaining a relation candidate set; based on a preset rule, screening valid relations from the relation candidate set, and generating a relation node set;

[0065] S214, constructing a source-relationship edge generation function based on the source data node set and the relationship node set; calculating the connection probability between node pairs based on the source-relationship edge generation function to obtain a connection probability matrix; determining the existence of edges based on the connection probability matrix and a preset probability threshold to obtain a source relationship edge set;

[0066] S215, constructing a relationship-feature edge generation function based on the relationship node set and the target feature node set; calculating the mapping strength between node pairs based on the relationship-feature edge generation function to obtain a mapping strength matrix; determining the directionality of the edge based on the mapping strength matrix and a preset strength threshold to obtain a relationship feature edge set;

[0067] S216. Construct a graph structure merging function based on the source data node set, the relationship node set, the target feature node set, the source relationship edge set, and the relationship feature edge set; based on the graph structure merging function, assemble according to the topological relationship to generate an initial hierarchical graph structure.

[0068] This embodiment realizes the hierarchical organization and optimization of data structure. The source node feature matrix is ​​constructed based on the standardized three-dimensional tensor, and the scientific division of source data nodes is realized by calculating the information entropy and hierarchical clustering. In the feature node construction link, by calculating the mutual information and spectral clustering between features, representative target feature nodes are extracted, which improves the efficiency of feature expression. By constructing a relationship reasoning model, the association strength between node pairs is calculated, and a reliable relationship node set is established through threshold segmentation and rule screening. In the edge generation link, by constructing the generation function of source-relationship edge and relationship-feature edge, the accurate construction of the edge is realized based on the connection probability and mapping strength. Through the graph structure merging function, the node set and the edge set are assembled according to the topological relationship to form a complete initial hierarchical graph structure. This embodiment not only improves the efficiency of data organization, but also enhances the expression ability of the data structure through the establishment of multiple relationships, laying a solid foundation for subsequent feature extraction and optimization.

[0069] According to one aspect of the present application, step S23 is further:

[0070] S231, extracting node degree information in the weighted hierarchical graph and constructing a degree distribution statistical model; based on the degree distribution statistical model, calculating the actual degree distribution and obtaining the actual degree sequence; obtaining data characteristics, estimating the ideal degree distribution and obtaining the ideal degree sequence; calculating the deviation between the actual degree sequence and the ideal degree sequence and generating a degree deviation vector;

[0071] S232, extracting structural information of the weighted hierarchical graph and constructing a path analysis model; based on the path analysis model, counting the shortest paths between nodes to obtain a path length matrix; based on the path length matrix, calculating the clustering coefficient of the path to obtain a clustering coefficient vector; obtaining path features, and based on the clustering coefficient vector and the path features, evaluating structural redundancy to generate a redundancy matrix;

[0072] S233, constructing a graph optimization objective function based on the degree deviation vector, the path length matrix and the redundancy matrix; based on the graph optimization objective function, calculating the gradient direction of the structure optimization to obtain an optimization direction matrix; based on the optimization direction matrix, constructing a step adjustment strategy to obtain a step sequence; based on the step sequence, performing an iterative optimization operation to generate an optimization parameter set;

[0073] S234. Based on the optimization parameter set, a graph structure reorganization function is constructed; based on the graph structure reorganization function, the connection relationship between nodes is adjusted to obtain a reorganized edge set; the weight information of the edges in the reorganized edge set is updated to obtain an updated weight matrix; based on the updated weight matrix, an optimized hierarchical graph is generated.

[0074] This embodiment achieves efficient optimization and simplification of the data structure. By constructing a degree distribution statistical model, calculating the actual degree distribution and the ideal degree distribution, the degree of deviation of the structure is accurately quantified. In the path analysis stage, by counting the shortest path and calculating the clustering coefficient, the structural redundancy is evaluated in combination with the path characteristics, providing a reliable basis for optimization. By constructing an optimization mechanism that includes the structural optimization gradient direction and step size adjustment strategy, dynamic optimization of the graph structure is achieved. Through the graph structure reorganization function, the connection relationship between nodes is adjusted and the edge weights are updated to generate an optimized hierarchical graph structure. This embodiment not only improves the efficiency of data organization, but also improves the accuracy of data expression through redundancy control and structural reorganization, providing a high-quality data foundation for subsequent feature extraction.

[0075] According to one aspect of the present application, step S24 is further:

[0076] S241. Obtain node adjacency information in the optimized hierarchical graph and construct a local feature extraction model; calculate the attribute statistics of the first-order neighbor nodes to obtain a neighbor feature matrix; statistically distribute edge weights to obtain a weight distribution vector; construct a local subgraph for each node to obtain a local subgraph set; calculate isomorphism based on subgraph structural features to obtain an isomorphism metric matrix; combine the above features to form a local structural feature set.

[0077] S242. Collect the global connection information in the optimized hierarchical graph and construct a centrality calculation model; calculate the degree centrality of the node to obtain the degree centrality vector; calculate the closeness centrality of the node to obtain the closeness centrality vector; calculate the betweenness centrality of the node to obtain the betweenness centrality vector; calculate the eigenvector centrality of the node to obtain the eigenvector centrality vector; combine various centrality indicators to obtain the centrality characteristic matrix.

[0078] S243. Extract the evolution sequence of the optimized hierarchical graph and construct a time series feature extraction model; calculate the time autocorrelation of node attributes to obtain the autocorrelation matrix; analyze the evolution pattern of node relationships to obtain the evolution pattern vector; extract mutation point features to obtain a mutation feature sequence; evaluate trend stability to obtain a stability indicator matrix; combine the time series features to form a time series feature set.

[0079] S244, obtain the local structure feature set, the centrality feature matrix and the time series feature set, and construct a feature fusion function; calculate the complementarity between features to obtain a complementary measurement matrix; construct a feature weight scheme to obtain a fusion weight vector; perform a weighted combination operation to generate a node feature vector.

[0080] This embodiment realizes the multi-dimensional expression and optimization of data features. By constructing a local feature extraction model, calculating the attribute statistics and edge weight distribution of neighbor nodes, and through the isomorphism analysis of local subgraphs, a complete set of local structural features is obtained. In the global feature extraction link, a comprehensive centrality feature matrix is ​​constructed by calculating various centrality indicators such as degree centrality, closeness centrality, betweenness centrality and eigenvector centrality. Through the time series feature extraction model, the time autocorrelation and relationship evolution pattern of node attributes are analyzed, and combined with mutation feature detection and trend stability assessment, a rich time series feature set is formed. Through the feature fusion function, the fusion weight is constructed based on the complementarity between features, and the effective combination of multi-dimensional features is realized. This embodiment not only improves the integrity of feature expression, but also enhances the expressiveness of features through the fusion of multi-dimensional features, providing a high-quality feature foundation for subsequent data processing.

[0081] like Figure 4 As shown, according to one aspect of the present application, step S3 is further:

[0082] S31, based on the optimized feature matrix, feature importance vector and data reliability matrix, set the number parameters of potential features and construct a basic mapping function group; based on the basic mapping function group, calculate the mapping value between each data sample and each potential feature; perform softmax operation on the mapping value to obtain the sample-feature probability distribution matrix; wherein the mapping value is obtained by weighted combination of the basic mapping function group;

[0083] S32, based on the quantity parameters of the potential features, calculating the Euclidean distance between the potential feature vectors, and based on the Euclidean distance and the preset scale parameter, determining the concept similarity; counting the number of co-occurrences of the potential features, and calculating the co-occurrence probability; extracting the time series data of the potential features, and calculating the time series correlation coefficient; taking the product of the concept similarity, the co-occurrence probability and the time series correlation coefficient as the corresponding element value, and generating a knowledge consistency matrix;

[0084] S33. Based on the probability distribution matrix and the knowledge consistency matrix, a data enhancement objective function is constructed, including a probability distribution difference term, a knowledge consistency constraint term, and a regularization term; based on the data enhancement objective function, a feature extraction function is constructed to calculate the feature distance between samples; based on the feature distance and a preset scale parameter, a reliability function value is determined; based on the probability distribution matrix and the reliability function value, an enhancement strategy function is constructed to generate an enhancement strategy matrix;

[0085] S34, performing matrix decomposition on the enhancement strategy matrix to obtain a sample potential feature matrix and a concept potential feature matrix; constructing an optimization objective function based on the sample potential feature matrix and the concept potential feature matrix, including a reconstruction error term, a potential feature regularization term, and a sample similarity constraint term; obtaining similarity relationships between samples, constructing a degree matrix and a similarity matrix, and calculating a Laplace matrix; based on the Laplace matrix, solving the optimization objective function through iterative optimization to obtain a reconstruction probability matrix;

[0086] S35, calculating the KL divergence between the reconstruction probability matrix and the preset target distribution matrix, constructing a knowledge distillation function; calculating the entropy value of the target distribution matrix as an entropy regularization term; calculating the structural difference between the reconstruction probability matrix and the target distribution matrix as a structure preservation term; combining the knowledge distillation function, the entropy regularization term and the structure preservation term to form a fusion objective function, and obtaining a fusion probability matrix through optimization;

[0087] S36. Based on the fusion probability matrix and the preset basic quality metric function group, calculate various quality indicators for each data sample; use dynamic weight coefficients to weighted sum each quality indicator to obtain the quality score of the sample; combine the quality score, the data sample in the fusion probability matrix and the corresponding label to form an enhanced data set.

[0088] In one embodiment of the present application, based on the feature matrix X and the reliability matrix R, a probability distribution matrix P∈R is constructed. n×k , n is the number of samples, k is the number of potential features, and the element p_ij in the probability distribution matrix is ​​p(z_j|x_i)=softmax(Φ(x_i, z_j)), where Φ(x_i, z_j) represents the feature-concept mapping function, and the mapping function is: Φ(x_i, z_j) =∑(t=1 to T)ω_t*φ_t(x_i, z_j), ω_t is the mapping weight, φ_t is the basic mapping function family, and T is the number of mapping functions. Construct the knowledge consistency matrix K∈R k×k , the element k_ij in the knowledge consistency matrix is ​​k_ij=ξ_ij*π_ij*τ_ij, where ξ_ij is the concept similarity, ξ_ij=exp(-||z_i-z_j|| 2 / 2σ 2);π_ij is the co-occurrence probability, π_ij=(z_i, z_j) / total, where z_i, z_j represent the i-th and j-th features of the potential feature z, and total represents the total number of all feature pairs. τ_ij is the temporal correlation, τ_ij=corr(z_i(t), z_j(t)); Construct consistency constraints: tr(P T KP)≥θ, where θ is the consistency threshold.

[0089] Construct data enhancement objective function: L(Q)=||P - Q|| F 2 + λ1tr(Q T KQ) + λ2Ω(Q), where Q is the enhanced probability distribution matrix, Ω(Q) is the regularization term, Ω(Q)=∑(i, j) Ξ(q_ij), Ξ(x)=log(1+exp(x)) represents the smoothing function; construct the enhanced strategy function: E(x)=argmax_y{P(y|x)*R(x, y)}, where R(x, y) is the reliability function, R(x, y)=exp(-||f(x) - f(y)|| 2 / γ), where f( ) is the feature extraction function and γ is the scale parameter; decompose the probability distribution matrix P into: P≈UV T , where U∈R n×r Represents the sample potential feature matrix, V∈R k×r Represents the concept potential feature matrix, r is the potential feature dimension; construct the optimization objective function: min_(U, V)||P-UV T || F 2 +λ1(||U|| F 2 +||V|| F 2 ) + λ2tr(U T LU), where L is the sample similarity Laplace matrix, L = D – S, D is the degree matrix, and S is the similarity matrix.

[0090] Construct the knowledge distillation function: D(P||Q)=∑(i,j) p_ij log(p_ij / q_ij), construct the fusion target: min_QD(P||Q)+ αH(Q) +βJ(Q), where H(Q) represents the entropy regularization term, H(Q)=-∑(i,j)q_ij log(q_ij), J(Q) represents the structure preservation term, J(Q) =||QQ T -PP T || F 2. Construct quality evaluation indicators: ν(x) =∑(i=1 toM)ω_i*μ_i(x), where μ_i is the basic quality measurement function, ω_i is the weight coefficient, and output the enhanced data set: D'={(x_i, y_i, q_i)}, x_i is the enhanced sample, y_i is the corresponding label, and q_i is the quality score.

[0091] The above steps of this embodiment are the specific implementation process of the probability matrix decomposition method, which realizes the high-quality expansion of data and the deep integration of knowledge. By constructing a multimodal probability distribution matrix, the sample features are mapped to the potential feature space, and the richness of feature expression is improved through the weighted combination of multiple basic mapping functions. By introducing the knowledge consistency matrix, the concept similarity, co-occurrence probability and temporal correlation are comprehensively considered, and a strict knowledge consistency constraint mechanism is established. In the data enhancement link, the adaptive optimization of the enhancement strategy is realized by constructing an objective function including probability distribution difference terms, knowledge consistency constraints and regularization terms. The potential feature extraction of samples and concepts is realized by probability matrix decomposition, and the sample similarity constraint is introduced by the Laplace matrix to improve the accuracy of feature expression. Through the knowledge distillation and fusion mechanism, while maintaining the original knowledge structure, the effective expansion of knowledge is achieved. This embodiment improves the quality and effect of data enhancement and ensures the reliability and diversity of enhanced data.

[0092] According to one aspect of the present application, step S31 is further:

[0093] S311. Obtain the optimized feature matrix and construct a basic mapping function family; construct a linear mapping function to obtain a linear mapping matrix; construct a nonlinear mapping function to obtain a nonlinear mapping matrix; construct a kernel mapping function to obtain a kernel mapping matrix; combine the three types of mapping matrices to form a mapping function set.

[0094] S312, collect a set of mapping functions and construct an adaptive selection model; evaluate the expressiveness of the mapping function to obtain an expressiveness vector; calculate the stability of the mapping result to obtain a stability vector; analyze the computational complexity to obtain a complexity vector; and construct a selection strategy by combining the three vectors to generate a function weight vector.

[0095] S313, extracting feature data of each sample, applying a mapping function set and a function weight vector; performing feature mapping operations to obtain a mapping result matrix; calculating a mapping confidence to obtain a confidence vector; normalizing the mapping results to generate a normalized mapping matrix.

[0096] S314, obtaining a normalized mapping matrix and constructing a probability conversion model; calculating the similarity between samples to obtain a similarity matrix; constructing a transition probability based on local connectivity to obtain a transition probability matrix; performing a probability smoothing operation to generate a probability distribution matrix.

[0097] This embodiment realizes the probabilistic expression and optimization of data features. By constructing a diverse family of basic mapping functions, including linear mapping, nonlinear mapping and kernel mapping functions, a rich set of mapping functions is formed. By constructing an adaptive selection model, the expression ability, stability and complexity of the mapping function are evaluated, and the dynamic selection and weight distribution of the mapping function are realized. During the mapping process, the reliability of the feature mapping is guaranteed by calculating the confidence and normalization of the mapping results. By constructing a probability conversion model, the accurate calculation of the probability distribution is realized based on sample similarity and local connectivity. This embodiment not only improves the accuracy of feature expression, but also enhances the adaptability of feature mapping through an adaptive selection mechanism, providing a reliable probabilistic basis for subsequent data enhancement.

[0098] According to one aspect of the present application, step S33 is further:

[0099] S331. Obtain the probability distribution matrix and build a data enhancement strategy selection model; calculate the distribution characteristics of the samples to obtain the distribution feature vector; analyze the sparsity of the samples to obtain the sparsity matrix; evaluate the uncertainty of the samples to obtain the uncertainty vector; build a decision tree model based on these three characteristics and generate a strategy selection matrix.

[0100] S332. Collect strategy selection matrix and build parameter optimization model; analyze strategy sensitivity to obtain sensitivity vector; calculate parameter influence range to obtain influence range matrix; evaluate parameter combination effect to obtain combination effect vector; optimize parameter configuration through game theory method to generate optimized parameter set.

[0101] S333, extract the basic operation operators of each enhancement strategy and construct a combined enhancement model; construct a permutation operator to obtain a permutation operator set; construct a transformation operator to obtain a transformation operator set; construct a hybrid operator to obtain a hybrid operator set; configure operator parameters based on the optimization parameter set to generate an enhancement operator library.

[0102] S334, obtain the sample data to be enhanced, apply the operators in the enhancement operator library; perform data transformation to obtain a transformed sample set; calculate the effectiveness after the transformation to obtain an effectiveness score vector; screen effective enhancement results and generate an enhancement strategy matrix.

[0103] This embodiment achieves high-quality expansion and optimization of data. By constructing a data enhancement strategy selection model, comprehensively analyzing the distribution characteristics, sparsity and uncertainty of the samples, a decision tree model for strategy selection is established, and the intelligent selection of enhancement strategies is achieved. In the parameter optimization link, by analyzing the sensitivity and scope of influence of the strategy and evaluating the effect of parameter combinations, the game theory method is used to achieve precise optimization of parameter configuration. By constructing a variety of enhancement operators, including permutation, transformation and hybrid operators, a rich library of enhancement operators is formed. In the data transformation process, the quality control of the enhancement results is achieved by calculating the effectiveness score of the transformed samples. This embodiment not only improves the quality of data expansion, but also improves the diversity of data through a variety of enhancement strategies, providing richer training samples for model training.

[0104] According to one aspect of the present application, step S35 is further:

[0105] S351. Obtain the probability distribution matrix and the fusion probability matrix, and construct a distribution alignment model; calculate the Wasserstein distance between distributions to obtain a distance matrix; construct an optimal transmission mapping to obtain a transmission mapping matrix; construct an alignment strategy based on the transmission mapping and generate an alignment parameter set.

[0106] S352. Collect the structural information of the original distribution and the target distribution, and construct a structure-preserving model; extract local structural features to obtain a local structural matrix; analyze the global topological relationship to obtain a topological relationship graph; calculate the structural difference measure to obtain a difference measure vector; construct a structural mapping function to generate a structural mapping set.

[0107] S353. Extract distribution data and structural information, build a consistency constraint model; construct Hard constraint conditions to obtain a hard constraint set; construct Soft constraint conditions to obtain a soft constraint set; define the constraint violation degree to obtain a violation measurement vector; construct a constraint satisfaction evaluation function to generate a constraint evaluation matrix.

[0108] S354, obtain the alignment parameter set, structure mapping set and constraint evaluation matrix, and construct a comprehensive optimization model; calculate the alignment loss to obtain the alignment loss vector; calculate the structure loss to obtain the structure loss vector; calculate the constraint loss to obtain the constraint loss vector; solve through the multi-objective optimization method to generate a fusion probability matrix.

[0109] This embodiment realizes efficient transfer and integration of knowledge. By constructing a distribution alignment model, calculating the Wasserstein distance and optimal transmission mapping between distributions, accurate alignment between different knowledge representations is achieved. In the structure preservation link, by analyzing local structural features and global topological relationships, a structural mapping function is constructed to ensure the effective preservation of the knowledge structure. By constructing a consistency constraint model and combining hard constraints and soft constraints, strict control of the knowledge fusion process is achieved. By constructing a multi-objective optimization method, the alignment loss, structural loss and constraint loss are comprehensively considered to achieve effective fusion of knowledge. This embodiment not only improves the accuracy of knowledge transfer, but also ensures the integrity of knowledge through a structure preservation mechanism, providing a high-quality knowledge foundation for model training.

[0110] like Figure 5 As shown, according to one aspect of the present application, step S4 is further:

[0111] S41. Based on the data samples in the enhanced data set, a basis vector group of the multidimensional verification space is constructed, including an integrity basis vector, a consistency basis vector, a timeliness basis vector, a relevance basis vector, and an accuracy basis vector; based on the basis vector group, a projection operation is performed on each data sample in the direction of each basis vector, and the projection value is used as the verification value of the corresponding dimension to generate a verification matrix;

[0112] S42, obtaining sample verification values ​​in the verification matrix, constructing an inequality constraint matrix and an equality constraint matrix; calculating constraint function values ​​based on a preset basis function group and constraint weights; combining the inequality constraint matrix, the equality constraint matrix and the constraint function values ​​to form a convex polyhedron constraint condition, and generating a constraint space;

[0113] S43, extracting the verification value of each dimension in the verification matrix, calculating the standardized difference between the verification value of each dimension and the preset expected value for each data sample, and obtaining a single-dimensional consistency value; taking the cumulative product of all single-dimensional consistency values ​​as the overall consistency measurement value of the sample; based on the overall consistency measurement value and the preset consistency threshold, screening the data samples to form a consistency data set;

[0114] S44, based on the consistency data set, construct a first optimization objective function; construct an adaptive learning rate calculation formula, based on the adaptive learning rate calculation formula, solve the first optimization objective function through iterative optimization to obtain an optimization parameter set; wherein the first optimization objective function includes a sample loss term and a regularization term; the sample loss term is composed of a negative log-likelihood value and a parameter regularization value, and the regularization term is composed of a weighted combination of a preset basis function group;

[0115] S45, based on the optimized parameter set, calculating the distance value between the sample and the preset center point in each dimension; using a preset metric matrix to perform weighted combination on the distance value to obtain anomaly detection scores; based on the anomaly detection scores, obtaining abnormal samples; performing correction operations on the abnormal samples in the constraint space to generate a corrected data set;

[0116] S46. Extract samples from the revised data set, and use a preset scoring sub-function group to calculate the basic score; obtain the time factor and timeliness parameter, and calculate the dynamic weight coefficient; use the weighted sum of the basic score and the dynamic weight coefficient as the updated score of the sample; calculate the mean and standard deviation of the updated score, and construct a dynamic screening threshold; based on the dynamic screening threshold, screen the samples in the revised data set to obtain the final training data set.

[0117] In one embodiment of the present application, a verification space V∈R is constructed based on the enhanced dataset D′. m , construct the basis vector group {e_1, e_2, ..., e_m}, where e_1 is the integrity dimension, e_2 is the consistency dimension, e_3 is the timeliness dimension, e_4 is the relevance dimension, e_5 is the accuracy dimension, .... Construct the verification matrix H∈R n×m , verify that the element value in the matrix h_ij = proj_j(x_i), where proj_j represents the projection function of the jth dimension, and x_i represents the i-th data sample. Construct convex polyhedron constraints: P = {x | Ax≤b, Cx = d}, where A is the inequality constraint matrix, b is the inequality constraint vector, C is the equality constraint matrix, and d is the equality constraint vector; construct constraint generation function: g(x) = [g_1(x), g_2(x), ..., g_k(x)] T , where g_i(x) is the i-th constraint function, g_i(x)=∑(j=1 to m)α_ijφ_j(x)+β_i, α_ij is the constraint weight, φ_j is the basis function, and β_i is the bias term.

[0118] Construct a consistency metric function: C(x)=∏(i=1 to m)c_i(x), where c_i(x) is a single-dimensional consistency function, c_i(x)=1-|v_i(x)-μ_i| / σ_i, vi_i(x) is the verification value of the i-th dimension, μ_i is the expected value, and σ_i is the allowable deviation; Construct a consistency constraint: for all x∈D': C(x)≥η, where η is the consistency threshold. Construct an optimization objective function: min_θL(θ)=∑(x∈D')Λ(x,θ)+Ω(θ), where Λ(x,θ) is a sample loss function, Λ(x,θ)=-log(p_θ(x))+λ1||θ||1 + λ2||θ||2 2; Ω(θ) is the regularization term, Ω(θ)=∑(i=1 to k)ω_iψ_i(θ), where ψ_i is the regularization basis function and ω_i is the weight coefficient. Construct an iterative update rule: θ_t+1 = θ_t-η_t ▽L(θ_t), where η_t is the adaptive learning rate, η_t=η_0 / (1 + γt) δ , η_0 is the initial learning rate, γ and δ are the decay parameters.

[0119] Construct anomaly detection function: ε(x)=∑(i=1 to m)κ_i * d_i(x), where d_i(x) is the distance function, d_i(x)=||x -μ_i|| M , μ_i is the center point, M is the metric matrix, and κ_i is the weight coefficient. Construct a correction strategy: x'=argmin_y{||y - x|| 2 +λε(y)}, subject to: y∈P. Construct an updated scoring function: s(x) =∑(i=1 tom)w_i * f_i(x), where f_i is the scoring subfunction, w_i is the dynamic weight, w_i = exp(ρ_i * t) / ∑(j) exp(ρ_j * t), ρ_i is the timeliness parameter, and t is the time factor; generate the final data set: D*={x∈D'|s(x)≥τ}, where τ is the dynamic threshold, τ=μ_s+k*σ_s, μ_s is the mean score, σ_s is the score standard deviation, and k is the adjustment parameter.

[0120] In another embodiment of the present application, data filtering and screening can also be performed by a classifier, specifically: training a text quality judgment model, and using the model to identify and filter low-quality data. The classifier is trained using a set of high-quality question-answer pair data, with the goal of giving higher scores to question-answer pair data similar to the training data. This classifier can be used to evaluate the content quality of the question-answer pair data and filter low-quality data.

[0121] Data format processing and conversion are performed based on script files. Specifically, the dataset is converted into a professional field training dataset in the format proposed by the large model by writing script code, which is used for secondary pre-training of the large model and instruction fine-tuning.

[0122] This embodiment achieves all-round guarantee of data quality by verifying and dynamically optimizing the data. A multidimensional verification space including dimensions such as integrity, consistency, timeliness, relevance and accuracy is constructed, and multidimensional evaluation of data is achieved through projection operations. A strict data verification framework is established by constructing convex polyhedron constraints, combining inequality constraint matrices and equality constraint matrices. In the consistency verification link, a comprehensive consistency measurement function is constructed by considering the expected value and allowable deviation of each dimension to ensure the overall consistency of the data. By constructing an optimization objective function including sample loss terms and regularization terms, and introducing an adaptive learning rate mechanism, dynamic optimization of parameters is achieved. In the anomaly detection and correction link, accurate identification and effective correction of abnormal data are achieved through comprehensive distance measurement and correction strategies. Through the dynamic update mechanism, timeliness factors are taken into account to ensure real-time optimization of the data set. This embodiment improves the comprehensiveness and accuracy of data verification and ensures the high quality of the final training data set.

[0123] According to one aspect of the present application, step S41 is further:

[0124] S411. Obtain the construction parameters of the multidimensional verification space and construct a basis vector generation model; construct an integrity basis vector to obtain an integrity basis matrix; construct a consistency basis vector to obtain a consistency basis matrix; construct a timeliness basis vector to obtain a timeliness basis matrix; construct a correlation basis vector to obtain a correlation basis matrix; and combine each basis matrix to form a basis vector group.

[0125] S412, collect sample data and basis vector groups, build a projection calculation model; calculate the orthogonal projection components to obtain the orthogonal projection matrix; calculate the oblique projection components to obtain the oblique projection matrix; evaluate the projection accuracy to obtain the accuracy evaluation vector; select the optimal projection method to generate the projection result matrix.

[0126] S413, extract the projection result matrix and construct a dimension mapping model; analyze the dimension correlation to obtain the correlation matrix; calculate the dimension weight to obtain the weight vector; construct the dimension transformation function to obtain the transformation function set; perform the dimension reconstruction operation to generate the verification matrix.

[0127] This embodiment realizes a comprehensive assessment of data quality. By constructing a multi-dimensional basis vector group, including dimensions such as completeness, consistency, timeliness, relevance and accuracy, a comprehensive verification space is established. In the projection calculation link, by constructing calculation models for orthogonal projection and oblique projection, and evaluating the projection accuracy, accurate mapping of samples in each dimension is achieved. Through the dimensional mapping model, the dimensional correlation is analyzed and the transformation function is constructed, and the optimization reconstruction of the verification results is achieved. This embodiment not only improves the comprehensiveness of data quality assessment, but also improves the accuracy of the assessment results through precise projection calculations, providing a reliable evaluation basis for the quality assurance of the data set.

[0128] According to one aspect of the present application, step S44 is further:

[0129] S441. Obtain samples from the consistent data set and construct a parameter initialization model; calculate the statistical moment distribution of the samples to obtain the moment distribution matrix; perform parameter distribution estimation to obtain the distribution parameter vector; calculate the initial value based on the maximum likelihood method to obtain the initial parameter set; evaluate the stability of the initial parameters and generate a parameter stability matrix.

[0130] S442. Collect the initial parameter set and build a gradient evaluation model; calculate the first-order gradient to obtain the first-order gradient matrix; calculate the second-order gradient to obtain the second-order gradient matrix; evaluate the stability of the gradient to obtain the gradient stability vector; analyze the sparsity of the gradient to obtain the gradient sparse matrix; combine various indicators to build a gradient clipping strategy and generate a gradient optimization matrix.

[0131] S443. Extract the gradient optimization matrix and construct a dynamic step size calculation model; analyze the historical gradient information to obtain the historical gradient sequence; calculate the momentum accumulation value to obtain the momentum matrix; evaluate the parameter sensitivity to obtain the sensitivity vector; construct an adaptive step size strategy and generate a step size adjustment matrix.

[0132] S444, obtain the current parameters and step size adjustment matrix, and construct a parameter update model; calculate the parameter update direction to obtain the update direction vector; evaluate the update amplitude to obtain the update amplitude matrix; test the update effectiveness to obtain the effectiveness score vector; perform parameter update operation to generate an optimized parameter set.

[0133] This embodiment realizes accurate optimization and adjustment of data parameters. By constructing a parameter initialization model, reliable initialization of parameters is realized based on the statistical moment distribution and maximum likelihood estimation of samples. In the gradient evaluation link, an accurate gradient clipping strategy is constructed by calculating the first-order gradient and the second-order gradient, and analyzing the stability and sparsity of the gradient. In terms of learning rate adjustment, adaptive adjustment of the learning rate is realized by analyzing historical gradient information and momentum accumulation, combined with parameter sensitivity analysis. Through the parameter update model, the update direction and amplitude are comprehensively considered to realize accurate optimization of parameters. This embodiment not only improves the efficiency of parameter optimization, but also improves the stability of the optimization process through an adaptive mechanism, providing reliable parameter support for data processing.

[0134] According to one aspect of the present application, step S45 is further:

[0135] S451. Obtain sample data and construct a distance calculation model; calculate the Euclidean distance component to obtain the Euclidean distance matrix; calculate the Mahalanobis distance component to obtain the Mahalanobis distance matrix; calculate the cosine distance component to obtain the cosine distance matrix; construct a distance fusion strategy to generate a comprehensive distance matrix.

[0136] S452. Collect the comprehensive distance matrix and build a local density estimation model; calculate the K nearest neighbor density to obtain the nearest neighbor density vector; estimate the kernel density distribution to obtain the kernel density matrix; analyze the density gradient to obtain the density gradient vector; build the cluster boundary based on the density feature and generate the density distribution map.

[0137] S453. Extract the density distribution map and construct a threshold calculation model; analyze the data distribution to obtain the distribution feature vector; calculate the statistical significance to obtain the significance matrix; evaluate the abnormal sensitivity to obtain the sensitivity vector; construct an adaptive threshold strategy and generate a threshold parameter set.

[0138] S454, obtain a threshold parameter set and construct an anomaly detection model; calculate local anomaly scores to obtain a local anomaly matrix; calculate global anomaly scores to obtain a global anomaly matrix; evaluate the reliability of the anomaly to obtain a reliability vector; perform an anomaly determination operation to generate an anomaly labeling matrix.

[0139] S455. Collect detected abnormal samples and construct a correction strategy model; analyze the causes of the abnormalities to obtain a cause analysis matrix; construct correction operators to obtain a correction operator set; evaluate the correction effect to obtain an effect evaluation vector; select the optimal correction plan and generate a correction parameter set.

[0140] S456, extract the samples to be corrected and the correction parameter set, and build an execution correction model; apply the correction operator to obtain the correction result matrix; verify the constraint satisfaction to obtain the constraint test vector; evaluate the rationality after correction to obtain the rationality score vector; screen the effective correction results and generate the correction data set.

[0141] This embodiment achieves precise control of data quality. By constructing a variety of distance measurement models, including Euclidean distance, Mahalanobis distance and cosine distance, a multi-dimensional evaluation of the degree of sample abnormality is achieved. In the density estimation stage, by calculating the K nearest neighbor density and kernel density distribution and analyzing the density gradient, accurate identification of abnormal samples is achieved. By constructing an adaptive threshold strategy, combining data distribution characteristics and statistical significance, dynamic optimization of abnormal judgment is achieved. In the correction stage, by analyzing the causes of anomalies and constructing targeted correction operators, combined with constraint checking and rationality evaluation, effective correction of abnormal samples is achieved. This embodiment not only improves the accuracy of data quality control, but also improves the efficiency of exception handling through adaptive mechanisms, providing reliable technical support for quality optimization of data sets.

[0142] According to one aspect of the present application, a data set construction system for training a large model in a professional field includes:

[0143] at least one processor; and,

[0144] a memory communicatively connected to at least one of the processors; wherein,

[0145] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the data set construction method for training a large model in a professional field as described in any of the above embodiments.

[0146] The present invention realizes the high-quality construction of large-model training data sets in professional fields by systematically integrating methods such as dynamic time series tensor decomposition, hierarchical graph structure processing, probability matrix decomposition and polyhedron projection. Through multi-dimensional data representation and dynamic time series processing, efficient fusion and quality optimization of heterogeneous data are achieved; through the construction and optimization of hierarchical graph structure, the efficiency of data organization and the quality of feature extraction are improved; through data enhancement and knowledge fusion driven by probability matrix decomposition, reliable expansion of data and deep integration of knowledge are achieved; through the verification and optimization mechanism of polyhedron projection, all-round quality assurance of data sets is ensured. The present invention not only solves the key technical problems in data processing in professional fields, but also improves the overall quality of data sets, provides high-quality data support for the training of large models, and has important application value in the training of large models in professional fields.

[0147] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. A method for constructing a data set for training a large model in a professional field, characterized in that: The steps include: S1. Collect pre-stored multi-source heterogeneous data, fuse and pre-process the multi-source heterogeneous data through the dynamic time series tensor decomposition method, and generate a standardized three-dimensional data tensor; S2, based on the standardized three-dimensional data tensor, a hierarchical graph structure processing method is used for reorganization and optimization to generate an optimized feature matrix, feature importance vector and data reliability matrix; S3, based on the optimized feature matrix, feature importance vector and data reliability matrix, the probability matrix decomposition method is used for enhancement and knowledge fusion to generate an enhanced data set; S4. Based on the enhanced data set, the polyhedron projection method is used for verification and dynamic optimization to generate the final training data set; Step S1 is further as follows: S11. Based on preset data dimension parameters, obtain pre-stored multi-source heterogeneous data; map the multi-source heterogeneous data to a four-dimensional data space to construct an initial four-dimensional data tensor; wherein the initial four-dimensional data tensor includes a data source dimension, a time dimension, a feature dimension, and a quality dimension; and the data source dimension includes text and images; S12. Based on the time dimension data of the initial four-dimensional data tensor, construct a time dimension matrix, calculate the time interval between any two time points; based on the time interval and the preset time decay coefficient, calculate the time decay value; Based on the preset time threshold parameter, determine the time correlation coefficient; use the product of the time attenuation value and the time correlation coefficient as the corresponding element value to generate a time series weight matrix; S13, extracting feature dimension data from the initial four-dimensional data tensor, and calculating the cosine similarity between feature vectors based on the feature dimension data; obtaining feature co-occurrence frequency, taking the sum of the cosine similarity and the logarithm of the feature co-occurrence frequency as the feature association, and generating a feature association matrix; S14, constructing a tensor decomposition optimization function based on the initial four-dimensional data tensor, the time series weight matrix and the feature association matrix; Solving the tensor decomposition optimization function through iterative optimization, obtaining the reconstructed data tensor; wherein the tensor decomposition optimization function includes tensor reconstruction terms, timing constraint terms and feature association constraint terms; S15, extracting quality dimension data of the reconstructed data tensor, and performing weighted median filtering within a time window of a preset size based on the quality dimension data to generate a quality score vector; S16. Based on the quality score vector, weighted sum is performed on the quality dimensions of the reconstructed data tensor to obtain a four-dimensional data tensor; the four-dimensional data tensor is compressed into a standardized three-dimensional data tensor; wherein the standardized three-dimensional data tensor includes a data source dimension, a time dimension, and a feature dimension; Step S2 is further as follows: S21, extracting data source dimension information and feature dimension information from the standardized three-dimensional data tensor, and constructing a three-layer data node set; based on the association relationship between the nodes in the three-layer data node set, constructing an edge set from source data to relationship and an edge set from relationship to target feature, to form an initial hierarchical graph structure; wherein the first layer of the three-layer data node set is a source data node set, the second layer is a relationship node set, and the third layer is a target feature node set; S22, based on the node pairs in the initial hierarchical graph structure, calculate the graph distance between the nodes; based on the graph distance and the preset distance scale parameter, calculate the distance attenuation factor; collect the observation frequency of the edge, extract the maximum observation frequency therein, and calculate the frequency normalization factor based on the maximum observation frequency; based on the node pairs in the initial hierarchical graph structure, calculate the node feature vector angle and the time series correlation coefficient, and determine the correlation factor; use the product of the distance attenuation factor, the frequency normalization factor and the correlation factor as the weight value of the corresponding edge to generate a weighted hierarchical graph; S23. Based on the structural characteristics of the weighted hierarchical graph, a graph optimization objective function is constructed, including a product term of edge weight and edge length, a node quantity constraint term, an edge quantity constraint term, and a transition redundancy constraint term; the graph optimization objective function is solved by iterative optimization to obtain an optimized hierarchical graph; S24, extracting target feature nodes in the optimization hierarchical graph, and calculating local structural features, global position features, and temporal evolution features of each node based on the target feature nodes; combining the local structural features, global position features, and temporal evolution features to form a node feature vector; S25. Based on the node feature vector, construct a feature fusion matrix, calculate the Frobenius norm between the feature fusion matrix and the pre-configured graph structure reconstruction feature matrix; based on the Frobenius norm, verify the feature fusion matrix to obtain a verification result; Based on the verification results, the feature scores based on the local structure and the feature scores based on the global consistency are calculated; based on the feature scores based on the local structure, the feature scores based on the global consistency and the preset balance factors, the feature reliability scores are calculated to generate the feature importance vector; S26. Based on the dimension of the node feature vector, construct a sample-feature matrix as an optimized feature matrix; Based on the feature reliability scores, the reliability relationship matrix between samples is constructed as the data reliability matrix; Step S14 is further as follows: S141, constructing a rank parameter set for tensor decomposition based on the initial four-dimensional data tensor, the time series weight matrix and the feature association matrix; performing singular value decomposition on the initial four-dimensional data tensor based on the rank parameter set to obtain a decomposition basis matrix; selecting a main component from the decomposition basis matrix based on a preset cutoff threshold to generate an initial decomposition tensor; S142, constructing a gradient calculation function based on the initial decomposition tensor, calculating the gradient value of the tensor reconstruction item, and obtaining a reconstructed gradient matrix; Based on the timing weight matrix, the gradient value of the timing constraint item is calculated to obtain the timing gradient matrix; Based on the feature association matrix, the gradient value of the feature association constraint item is calculated to obtain the association gradient matrix; the reconstruction gradient matrix, the temporal gradient matrix and the association gradient matrix are weighted and combined to form a comprehensive gradient matrix; S143, constructing an adaptive learning rate function based on the comprehensive gradient matrix; Get the current iteration round, and calculate the basic learning rate based on the current iteration round and the adaptive learning rate function; Calculate the momentum term based on the pre-stored historical gradient information; Combine the basic learning rate with the momentum term to get the optimized step size matrix; S144, based on the initial decomposition tensor and the optimized step size matrix, performing a parameter update operation to obtain an updated decomposition tensor; Calculate the reconstruction error between the updated decomposition tensor and the initial four-dimensional data tensor to obtain an error metric value; S145, collecting error measurement values ​​of consecutive predetermined rounds and constructing a convergence judgment criterion; Based on the convergence judgment criterion, the rate of change of the error metric value is calculated to obtain the convergence index value; when the convergence index value is less than a preset threshold, the corresponding updated decomposition tensor is determined as the reconstructed data tensor.

2. The method for constructing a data set for training a large model in a professional field according to claim 1, characterized in that: Step S3 is further as follows: S31, based on the optimized feature matrix, feature importance vector and data reliability matrix, set the quantity parameters of the potential features and construct a basic mapping function group; based on the basic mapping function group, calculate the mapping value between each data sample and each potential feature; perform softmax operation on the mapping value to obtain the sample-feature probability distribution matrix; S32, based on the quantity parameters of the potential features, calculating the Euclidean distance between the potential feature vectors, and determining the concept similarity based on the Euclidean distance and the preset scale parameter; counting the number of co-occurrences of the potential features and calculating the co-occurrence probability; extracting the time series data of the potential features and calculating the time series correlation coefficient; The product of concept similarity, co-occurrence probability and temporal correlation coefficient is used as the corresponding element value to generate the knowledge consistency matrix; S33. Based on the probability distribution matrix and the knowledge consistency matrix, a data enhancement objective function is constructed, including a probability distribution difference term, a knowledge consistency constraint term, and a regularization term; Based on the data enhancement objective function, a feature extraction function is constructed to calculate the feature distance between samples; Determine the reliability function value based on the characteristic distance and the preset scale parameter; Based on the probability distribution matrix and the reliability function value, an enhancement strategy function is constructed to generate an enhancement strategy matrix; S34, performing matrix decomposition on the enhancement strategy matrix to obtain a sample latent feature matrix and a concept latent feature matrix; Based on the sample latent feature matrix and the concept latent feature matrix, an optimization objective function is constructed, including a reconstruction error term, a latent feature regularization term, and a sample similarity constraint term; similarity relationships between samples are obtained, a degree matrix and a similarity matrix are constructed, and the Laplace matrix is ​​calculated; based on the Laplace matrix, the optimization objective function is solved through iterative optimization to obtain a reconstruction probability matrix; S35, calculating the KL divergence between the reconstruction probability matrix and the preset target distribution matrix, constructing a knowledge distillation function; calculating the entropy value of the target distribution matrix as an entropy regularization term; calculating the structural difference between the reconstruction probability matrix and the target distribution matrix as a structure preservation term; combining the knowledge distillation function, the entropy regularization term and the structure preservation term to form a fusion objective function, and obtaining a fusion probability matrix through optimization; S36. Based on the fusion probability matrix and the preset basic quality metric function group, calculate various quality indicators for each data sample; use dynamic weight coefficients to weighted sum each quality indicator to obtain the quality score of the sample; combine the quality score, the data sample in the fusion probability matrix and the corresponding label to form an enhanced data set.

3. The method for constructing a data set for training a large model in a professional field according to claim 2, characterized in that: Step S4 is further as follows: S41. Based on the data samples in the enhanced data set, a basis vector group of the multidimensional verification space is constructed, including an integrity basis vector, a consistency basis vector, a timeliness basis vector, a relevance basis vector, and an accuracy basis vector; based on the basis vector group, a projection operation is performed on each data sample in the direction of each basis vector to generate a verification matrix; S42, obtaining sample verification values ​​in the verification matrix, constructing an inequality constraint matrix and an equality constraint matrix; and calculating constraint function values ​​based on a preset basis function group and constraint weights; The inequality constraint matrix, the equality constraint matrix and the constraint function value are combined to form a convex polyhedron constraint condition and generate a constraint space; S43, extracting the verification value of each dimension in the verification matrix, calculating the standardized difference between the verification value of each dimension and the preset expected value for each data sample, and obtaining a single-dimensional consistency value; taking the cumulative product of all single-dimensional consistency values ​​as the overall consistency measurement value of the sample; based on the overall consistency measurement value and the preset consistency threshold, screening the data samples to form a consistency data set; S44, constructing a first optimization objective function based on the consistent data set; Construct an adaptive learning rate calculation formula, and based on the adaptive learning rate calculation formula, solve the first optimization objective function through iterative optimization to obtain an optimization parameter set; S45, based on the optimized parameter set, calculating the distance value between the sample and the preset center point in each dimension; The distance values ​​are weighted and combined using a preset metric matrix to obtain anomaly detection scores; Based on the anomaly detection score, obtain abnormal samples; In the constraint space, the abnormal samples are corrected and a corrected data set is generated; S46, extracting samples from the modified data set, and using a preset scoring sub-function group to calculate a basic score; obtaining a time factor and a timeliness parameter, and calculating a dynamic weight coefficient; and taking a weighted sum of the basic score and the dynamic weight coefficient as an updated score of the sample; Calculate the mean and standard deviation of the update score and construct a dynamic screening threshold; Based on the dynamic screening threshold, the samples in the revised data set are screened to obtain the final training data set.

4. The method for constructing a data set for training a large model in a professional field according to claim 3, characterized in that: Step S15 is further as follows: S151, extracting quality dimension data of the reconstructed data tensor, segmenting the quality dimension data based on a preset time window size, and obtaining predetermined time window data; Based on each time window data, calculate the statistical features and obtain the window feature matrix; Based on the window feature matrix, the data distribution difference between adjacent windows is calculated to obtain the window difference vector; based on the window feature matrix and the window difference vector, the optimal window division is determined to generate a time window sequence; S152, based on the time window sequence, extracting the data source credibility information in each time window, and constructing a hierarchical weight calculation model; Based on the hierarchical weight calculation model, the historical performance score of the data source is calculated to obtain the historical weight vector; based on the historical weight vector, the current state of the data source is evaluated to obtain the state weight vector; the future reliability of the data source is predicted to obtain the prediction weight vector; the historical weight vector, the state weight vector and the prediction weight vector are nonlinearly combined to obtain the comprehensive weight vector; S153, collecting quality dimension data and comprehensive weight vectors in each time window, and constructing an adaptive filter; Based on the adaptive filter, the local correlation of the data is calculated to obtain the correlation matrix; based on the correlation matrix, the filter kernel parameters are constructed; Based on the filter kernel parameters, a weighted median filter operation is performed to generate a quality score vector.

5. The method for constructing a data set for training a large model in a professional field according to claim 3, characterized in that: Step S21 is further as follows: S211. Based on the standardized three-dimensional data tensor, extract the data source dimension information and construct a source node feature matrix; based on the source node feature matrix, calculate the information entropy for each data source to obtain an entropy value vector; Based on the entropy value vector, hierarchical clustering is performed to obtain a hierarchical tree of source nodes; Based on the preset hierarchical threshold, a node set is extracted from the source node hierarchy tree to generate a source data node set; S212. Extracting feature dimension data based on the standardized three-dimensional data tensor and constructing a feature node mapping function; Based on the feature node mapping function, the mutual information between features is calculated to obtain the information matrix; Perform spectral clustering on the information matrix to obtain the feature clustering matrix; Based on the feature clustering matrix, representative features are extracted to form a target feature node set; S213, extracting node attributes of the source data node set and the target feature node set, and constructing a relational reasoning model; Based on the relational reasoning model, the association strength between node pairs is calculated to obtain the association strength matrix; Perform threshold segmentation on the association strength matrix to obtain a relationship candidate set; Based on preset rules, filter valid relationships from the relationship candidate set and generate a relationship node set; S214, constructing a source-relationship edge generation function based on the source data node set and the relationship node set; Based on the source-relation edge generation function, the connection probability between node pairs is calculated to obtain the connection probability matrix; Based on the connection probability matrix and the preset probability threshold, the existence of the edge is determined to obtain the source relationship edge set; S215, constructing a relationship-feature edge generation function based on the relationship node set and the target feature node set; Based on the relationship-feature edge generation function, the mapping strength between node pairs is calculated to obtain the mapping strength matrix; Based on the mapping intensity matrix and the preset intensity threshold, the directionality of the edge is determined to obtain the relationship feature edge set; S216, constructing a graph structure merging function based on the source data node set, the relationship node set, the target feature node set, the source relationship edge set, and the relationship feature edge set; Based on the graph structure merging function, the graph is assembled according to the topological relationship to generate the initial hierarchical graph structure.

6. The method for constructing a data set for training a large model in a professional field according to claim 3, characterized in that: Step S23 is further as follows: S231, extracting node degree information in the weighted hierarchical graph and constructing a degree distribution statistical model; based on the degree distribution statistical model, calculating the actual degree distribution and obtaining the actual degree sequence; obtaining data characteristics, estimating the ideal degree distribution and obtaining the ideal degree sequence; calculating the deviation between the actual degree sequence and the ideal degree sequence and generating a degree deviation vector; S232, extracting structural information of the weighted hierarchical graph and constructing a path analysis model; Based on the path analysis model, the shortest paths between nodes are counted to obtain the path length matrix; Based on the path length matrix, the clustering coefficient of the path is calculated to obtain the clustering coefficient vector; Obtain path features, evaluate structural redundancy based on clustering coefficient vector and path features, and generate a redundancy matrix; S233, constructing a graph optimization objective function based on the degree deviation vector, the path length matrix and the redundancy matrix; Based on the graph optimization objective function, the gradient direction of the structural optimization is calculated to obtain the optimization direction matrix; based on the optimization direction matrix, the step size adjustment strategy is constructed to obtain the step size sequence; Based on the step sequence, iterative optimization operations are performed to generate a set of optimization parameters; S234, constructing a graph structure reorganization function based on the optimization parameter set; adjusting the connection relationship between nodes based on the graph structure reorganization function to obtain a reorganized edge set; updating the weight information of the edges in the reorganized edge set to obtain an updated weight matrix; Based on the updated weight matrix, an optimized hierarchy graph is generated.

7. A data set construction system for training large models in professional fields, characterized by: include: at least one processor; as well as, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the data set construction method for training a large model in a professional field as described in any one of claims 1 to 6.

Images