Automatic processing method and system of unstructured data based on deep learning

Abstract

The present invention provides a method and system for automatically processing unstructured data based on deep learning, which relates to the field of data processing technology, including obtaining unstructured data through an intelligent data collector and establishing a data feature fingerprint library, using an incremental learning algorithm to build a data quality assessment model for data standardization processing; using a multi-layer cascade network structure and a self-attention mechanism to extract explicit features and implicit features, construct a feature association graph and a two-way reasoning channel; performing rule matching and causal reasoning based on a domain knowledge base, and finally realizing automatic structuring of unstructured data. The present invention improves the automation and accuracy of data processing, reduces the cost of manual intervention, and has strong versatility and scalability.

Description

Technical Field

[0001] The present invention relates to data processing technology, and in particular to a method and system for automatically processing unstructured data based on deep learning. Background Art

[0002] With the rapid development of information technology, the generation and storage of unstructured data has exploded, and unstructured data in the form of text, images, audio, video, etc. has been widely used in all walks of life. However, unstructured data lacks a fixed storage structure, has diverse data formats and complex content, and traditional data processing methods are difficult to meet the needs of efficient classification, labeling and utilization. Especially in scenarios where the data volume is huge and updated frequently, it is a huge challenge to quickly realize the mining and utilization of data value.

[0003] Existing technologies mostly rely on manual or semi-automated methods to process unstructured data, but these methods are usually inefficient and costly, especially in data quality assessment and deep feature extraction, which are difficult to achieve automated and accurate processing. At the same time, there is still a lack of effective model architecture for the comprehensive mining of implicit features (such as association rules, changing trends) and explicit features (such as time series, numerical distribution) of data. In addition, in the process of constructing inference rules based on feature association relationships, current technologies cannot fully utilize domain knowledge and causal reasoning methods, resulting in limited effects of data optimization and structured transformation.

[0004] In order to solve the above problems, there is an urgent need for an automatic processing method for unstructured data based on deep learning, which can automate the entire process from data collection, classification and labeling to deep feature mining, rule reasoning and optimization processing, so as to improve the efficiency and quality of data processing. Summary of the invention

[0005] The embodiments of the present invention provide a method and system for automatic processing of unstructured data based on deep learning, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provided is a method for automatic processing of unstructured data based on deep learning, comprising:

[0008] Using an intelligent data collector to acquire unstructured data, building a data feature fingerprint library, classifying and labeling the unstructured data based on the data feature fingerprint library to obtain classified labeled data, using an incremental learning algorithm to build a data quality assessment model, the data quality assessment model scores the classified labeled data from three dimensions of data integrity, timeliness and value density to obtain a scoring result, building a priority processing queue based on the scoring result, and performing data cleaning and normalization on the classified labeled data in the priority processing queue to obtain a standardized training data set;

[0009] A multi-layer cascade network structure is used to perform deep learning processing on the standardized training data set. The multi-layer cascade network structure identifies target features in the standardized training data set based on the category information of the classified annotation data and through a self-attention mechanism, constructs a feature association graph based on the target features, and constructs a bidirectional reasoning channel on the feature association graph. The bidirectional reasoning channel includes a forward channel and a reverse channel, wherein the forward channel extracts numerical distribution features, time series features, and attribute features in the standardized training data set through a convolutional neural network to obtain explicit feature data, and the reverse channel extracts data item association rules, data change trends, and data interaction patterns in the standardized training data set through a recurrent neural network to obtain implicit feature data, and a dynamic weight allocation strategy is used to perform feature combination on the explicit feature data and the implicit feature data to obtain a multidimensional feature tensor;

[0010] Based on a preset domain knowledge base, rule matching is performed on the multidimensional feature tensor to obtain an inference rule set, the multidimensional feature tensor is hierarchically processed using the inference rule set to obtain initial structured data, the initial structured data is analyzed by a causal reasoning method to obtain a decision basis chain, optimization parameters are generated based on the decision basis chain, and the initial structured data is optimized according to the optimization parameters and category information of the classified annotation data to obtain final structured data.

[0011] In an optional embodiment,

[0012] Using an intelligent data collector to acquire unstructured data, building a data feature fingerprint library, and classifying and labeling the unstructured data based on the data feature fingerprint library to obtain classified and labeled data includes:

[0013] Acquire unstructured data through an intelligent data collector, the intelligent data collector includes a local feature channel and a global feature channel, wherein the local feature channel uses a hierarchical attention network to extract word-level features and sentence-level features to obtain local correlation features, the global feature channel uses a graph neural network to construct a data relationship graph and extracts global relationship features through message passing, and the local correlation features are fused with the global relationship features to obtain fused feature data;

[0014] The feature importance weights are obtained by performing multi-head attention calculation on the fused feature data, the fused feature data are screened according to the feature importance weights, a dynamic local sensitive hashing algorithm is used to map the screened features to generate feature fingerprints, and the feature fingerprints are constructed as a data feature fingerprint library that supports nearest neighbor queries;

[0015] Performing data enhancement on the fused feature data to obtain an enhanced feature set, calculating the contrast loss of sample pairs based on the enhanced feature set to obtain a semantic similarity matrix, constructing a data association graph using the neighbor query results of the data feature fingerprint library and the semantic similarity matrix, propagating known labels on the data association graph through graph attention to obtain an initial classification annotation result, and extracting entity relationships according to the data association graph to construct a domain knowledge graph;

[0016] Construct a meta-learner, use the feature vector and confidence of the initial classification and labeling results as input, evaluate and model the classification results of each sample, obtain a classification reliability score, mark the samples that are lower than a preset reliability threshold and verified by the data feature fingerprint library as the first type of data to be reviewed, use the domain knowledge graph to perform logical verification on the initial classification and labeling results, and mark the samples that have not passed the verification as the second type of data to be reviewed;

[0017] The selection process of the first category of data to be reviewed and the second category of data to be reviewed is modeled as a Markov decision process, a hybrid reward function is constructed according to the classification reliability score and the sample information gain, the sample selection strategy is optimized through deep reinforcement learning, and the optimal sample to be reviewed is selected for manual labeling to obtain a manual labeling result;

[0018] The manual annotation results are weightedly fused with the initial classification annotation results, and the consistency of the weighted fused annotation results is verified using the similarity index of the data feature fingerprint library to generate classification annotation data including the fused feature data, semantic similarity matrix, classification reliability score and domain knowledge graph verification results.

[0019] In an optional embodiment,

[0020] An incremental learning algorithm is used to construct a data quality assessment model, which scores the classified and labeled data from three dimensions: data integrity, timeliness, and value density to obtain a scoring result, and a priority processing queue is constructed according to the scoring result. The classified and labeled data in the priority processing queue are cleaned and normalized to obtain a standardized training data set including:

[0021] Obtaining timestamp information of the classified and labeled data, using an exponential decay function to calculate a data time decay coefficient, and performing weighted processing on the classified and labeled data based on the data time decay coefficient to obtain a sample weight;

[0022] Constructing a data quality assessment model, using the sample weight as a weight term to construct a loss function, using an online stochastic gradient descent algorithm to optimize the loss function until the model converges, obtaining a trained data quality assessment model, and using the trained data quality assessment model to perform quality assessment on the classified and labeled data;

[0023] The effective feature ratio is calculated based on the quality assessment result to obtain a data integrity score, the data timeliness score is obtained according to the data time decay coefficient, and the data value density score is calculated in combination with the sample weight. The data integrity score, data timeliness score and data value density score are weighted and fused to obtain a data quality score;

[0024] Constructing a priority processing queue according to the data quality score, marking the classified labeled data whose data quality score is lower than a first preset threshold as data to be eliminated, marking the classified labeled data whose data quality score is higher than a second preset threshold as data for priority processing, calculating the data processing priority based on the data quality score and the sample weight, and generating a priority data queue with processing marks;

[0025] According to the priority order of the priority data queue, the field name and data format of the classified and labeled data are uniformly converted to obtain standardized features, a time series completion model is constructed based on the data timeliness score and the data integrity score, and the missing fields are completed using the time series completion model to obtain structured features;

[0026] Constructing feature fingerprints according to the structural features and the data value density scores, identifying duplicate data based on the feature fingerprints, performing anomaly detection and denoising processing in combination with the data quality scores, and generating quality processing records and cleaned feature data;

[0027] The numerical features, category features and text features in the cleaned feature data are respectively standardized and converted, and the feature fingerprint is used to guide the word vector mapping of the text features, and the converted standardized features are combined with the data quality score, data processing priority and quality processing record to generate a standardized training data set.

[0028] In an optional embodiment,

[0029] The standardized training data set is subjected to deep learning processing using a multi-layer cascade network structure, wherein the multi-layer cascade network structure identifies target features in the standardized training data set according to category information of the classified annotation data and through a self-attention mechanism, including:

[0030] Constructing a multi-layer cascade network structure, the multi-layer cascade network structure includes a feature extraction layer, an attention layer and a feature fusion layer, wherein the feature extraction layer is composed of multiple convolution blocks, the attention layer adopts a self-attention mechanism, and the feature fusion layer is used for feature dimensionality reduction and combination;

[0031] Extracting category information from the classified labeled data in the standardized training data set, constructing a category embedding matrix, and calculating a category mapping matrix based on the category embedding matrix and the standardized training data set;

[0032] The standardized training data set is processed using a feature extraction layer of a multi-layer cascade network structure, and the category mapping matrix is ​​multiplied by the extracted features to obtain category-aware features, and the category-aware features extracted at different levels are spliced ​​in the channel dimension to obtain multi-scale cascade features;

[0033] In the attention layer, a query matrix, a key matrix, and a value matrix are generated respectively based on the multi-scale cascade features and the category embedding matrix, and the similarity between the query matrix and the key matrix is ​​calculated through the self-attention mechanism to obtain the attention weight, and the attention weight is multiplied by the value matrix to obtain the attention feature;

[0034] In the feature fusion layer, the attention features are residually connected and normalized to obtain normalized features, gating weights are constructed based on the normalized features and the category embedding matrix, and the gating weights are weighted with the normalized features to obtain enhanced features;

[0035] The enhanced features of each layer are transferred layer by layer in the multi-layer cascade network structure and feature cascade is performed. Through cross-layer feature aggregation, the target features in the standardized training data set are obtained.

[0036] In an optional embodiment,

[0037] The inference rule set obtained by performing rule matching on the multi-dimensional feature tensor based on the preset domain knowledge base includes:

[0038] Using a feature dimensionality reduction network to perform dimensionality reduction processing on a multidimensional feature tensor to obtain a feature sequence, constructing a feature index tree on the feature sequence, setting a sliding window of a dynamic size based on the feature index tree, the size of the sliding window being adaptively adjusted according to the feature sequence, and performing sliding scanning on the feature sequence to extract feature patterns;

[0039] Input the feature pattern into a pre-trained text encoder to obtain a feature encoding vector, and input the rule pattern in a preset domain knowledge base into a pre-trained text encoder to obtain a rule encoding vector; calculate the cosine similarity between the feature encoding vector and the rule encoding vector as the matching degree;

[0040] Calculating the mean and standard deviation of the matching degree, determining an adaptive threshold coefficient according to a preset confidence interval, weighting the mean and standard deviation based on the adaptive threshold coefficient to obtain an adaptive threshold, screening rule patterns with a matching degree higher than the adaptive threshold, and forming a candidate rule set with the screened rule patterns and their corresponding structured templates;

[0041] Dependency analysis is performed on the rules in the candidate rule set, and dependency relationships between the rules are established based on the input features and output features of the rules. The dependency relationships are represented as a directed graph, in which nodes represent rules and directed edges represent dependency relationships. The in-degree and out-degree of each rule node are calculated, and the dependency level of the rules is determined. An improved topological sorting algorithm is used to assign rules with direct dependency relationships to adjacent levels to obtain an inference rule set.

[0042] In an optional embodiment,

[0043] The multidimensional feature tensor is processed hierarchically using the inference rule set to obtain initial structured data including:

[0044] For each rule layer in the inference rule set, feature vectors of rule-related regions are extracted from feature sequences, and the feature vectors are input into a rule matching network, which includes a feature conversion layer and a matching degree calculation layer, and outputs a rule applicability score, which is normalized as the rule confidence;

[0045] Based on the rule confidence, the rules in each rule layer are sorted in descending order to obtain a sorted rule set, and the rules are sequentially applied to the corresponding structured templates in the sorted order to generate structured fragments through feature mapping, wherein the structured template defines a mapping relationship between features and structured data;

[0046] Detecting the effective areas of the rules in the sorting rule set, when the effective areas of multiple rules overlap, extracting the range of the overlapping area, calculating the coverage of each rule in the overlapping area, taking the weighted sum of the rule confidence and coverage as the rule priority, selecting the rule with the highest priority and applying it to the overlapping area, and ensuring the generation of a unique structured fragment;

[0047] Construct a multi-layer verification network including attribute verification layer, relationship verification layer and value range verification layer. In the attribute verification layer, the integrity of the structured fragment is verified based on the attribute graph. In the relationship verification layer, the rationality of the relationship between the structured fragments is verified based on the knowledge graph. In the value range verification layer, the validity of the attribute values ​​in the structured fragment is verified based on the statistical model. The verification results of each layer are integrated to obtain the verified structured fragment.

[0048] Extract semantic features from the verified structured fragments, calculate the semantic similarity between the fragments, build a hierarchical organizational structure of the fragments based on the semantic similarity, integrate semantically related structured fragments according to the hierarchical organizational structure, and generate initial structured data that meets preset specifications.

[0049] In an optional embodiment,

[0050] Analyzing the initial structured data by a causal reasoning method to obtain a decision basis chain, generating optimization parameters based on the decision basis chain, and optimizing the initial structured data according to the optimization parameters and the category information of the classified annotation data to obtain the final structured data includes:

[0051] Calculating partial correlation coefficients for variable pairs in the initial structured data, performing conditional independence tests based on the partial correlation coefficients to obtain conditional independence relationships between variables, and constructing a causal graph based on the conditional independence relationships;

[0052] A linear structural equation model is constructed based on the causal diagram, and the parameters of the linear structural equation model are optimized by maximum likelihood estimation to obtain a structural causal model, wherein the structural causal model is used to quantify the strength of causal effects between variables in the causal diagram;

[0053] Calculate the number of associated edges of each node in the causal graph, determine the node whose number of associated edges exceeds a preset threshold of the number of associated edges as a decision node, trace the associated nodes from the decision node along the causal edges in the causal graph step by step to obtain a factor chain, calculate the cumulative causal effect of the factor chain based on the structural causal model, and determine the factor chain whose cumulative causal effect exceeds a preset effect threshold as a candidate decision basis chain;

[0054] In the structural causal model, the value of each node in the candidate decision basis chain is changed and substituted into the structural causal model to obtain the downstream node prediction value corresponding to the node value change, and the stability of the candidate decision basis chain is determined based on the change range of the prediction value, and the candidate decision basis chain whose stability meets the preset conditions is determined as the decision basis chain;

[0055] Based on the structural causal model, the influence degree of each node in the decision basis chain on the target variable is calculated, and the nodes whose influence degree exceeds a preset influence threshold are selected as a feature set according to the size of the influence degree, and the influence degree of each node in the feature set is used as a feature importance score, and the feature importance score is combined with the category discrimination degree of the classified annotation data to obtain a feature optimization parameter;

[0056] Based on the feature optimization parameters, weighted adjustment is performed on the values ​​of corresponding features in the initial structured data to obtain optimized data, wherein the feature optimization parameters are used to determine the weight coefficient of the feature adjustment;

[0057] The difference in feature distribution between the optimized data and the initial structured data is calculated to obtain an optimization effect index, the optimized data is input into the structural causal model to verify the causal relationship constraints to obtain a verification result, the optimized data is iteratively optimized according to the optimization effect index and the verification result, and when the optimization index reaches a preset target and the verification result satisfies the causal consistency, the current optimized data is determined as the final structured data.

[0058] According to a second aspect of the embodiments of the present invention,

[0059] Provided is a deep learning-based unstructured data automatic processing system, comprising:

[0060] The first unit is used to acquire unstructured data using an intelligent data collector, build a data feature fingerprint library, classify and annotate the unstructured data based on the data feature fingerprint library to obtain classified annotated data, use an incremental learning algorithm to build a data quality assessment model, the data quality assessment model scores the classified annotated data from three dimensions of data integrity, timeliness and value density to obtain a scoring result, build a priority processing queue according to the scoring result, and perform data cleaning and normalization on the classified annotated data in the priority processing queue to obtain a standardized training data set;

[0061] The second unit is used to perform deep learning processing on the standardized training data set using a multi-layer cascade network structure, wherein the multi-layer cascade network structure identifies target features in the standardized training data set based on the category information of the classified annotation data and through a self-attention mechanism, constructs a feature association graph based on the target features, and constructs a bidirectional reasoning channel on the feature association graph, wherein the bidirectional reasoning channel includes a forward channel and a reverse channel, wherein the forward channel extracts numerical distribution features, time series features, and attribute features in the standardized training data set through a convolutional neural network to obtain explicit feature data, and the reverse channel extracts data item association rules, data change trends, and data interaction patterns in the standardized training data set through a recurrent neural network to obtain implicit feature data, and a dynamic weight allocation strategy is used to perform feature combination on the explicit feature data and the implicit feature data to obtain a multidimensional feature tensor;

[0062] The third unit is used to perform rule matching on the multidimensional feature tensor based on a preset domain knowledge base to obtain an inference rule set, use the inference rule set to perform hierarchical processing on the multidimensional feature tensor to obtain initial structured data, analyze the initial structured data through a causal reasoning method to obtain a decision basis chain, generate optimization parameters based on the decision basis chain, and optimize the initial structured data according to the optimization parameters and category information of the classified annotation data to obtain final structured data.

[0063] According to a third aspect of the embodiments of the present invention,

[0064] An electronic device is provided, comprising:

[0065] processor;

[0066] a memory for storing processor-executable instructions;

[0067] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0068] According to a fourth aspect of the embodiments of the present invention,

[0069] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0070] In this embodiment, by constructing a data feature fingerprint library and combining an incremental learning algorithm to establish a data quality assessment model, the data is scored from three dimensions: data integrity, timeliness, and value density, and a priority processing queue is established, which can effectively improve the efficiency and accuracy of data processing, ensure that important data is processed first, and ensure the quality of training data through data cleaning and normalization. A multi-layer cascade network structure is used in combination with a self-attention mechanism for deep learning processing, and explicit and implicit features are extracted through a bidirectional reasoning channel, and a dynamic weight allocation strategy is used for feature combination, which can fully capture various types of feature information in the data, improve the integrity and accuracy of feature extraction, and enhance the model's ability to understand complex unstructured data. Based on a preset domain knowledge base, rule matching and causal reasoning are performed, and the optimization process is guided by a decision-making basis chain, realizing the intelligent conversion from unstructured data to structured data, improving the interpretability and reliability of data processing, making the processing results more in line with actual application needs, and providing high-quality data support for subsequent data analysis and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A flowchart of a method for automatically processing unstructured data based on deep learning according to an embodiment of the present invention;

[0072] Figure 2 Schematic diagram of the structure of an automatic unstructured data processing system based on deep learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0075] Figure 1 Schematic diagram of the process of the method for automatically processing unstructured data based on deep learning according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0076] S101. Using an intelligent data collector to acquire unstructured data, construct a data feature fingerprint library, classify and annotate the unstructured data based on the data feature fingerprint library to obtain classified annotated data, and use an incremental learning algorithm to construct a data quality assessment model. The data quality assessment model scores the classified annotated data from three dimensions of data integrity, timeliness, and value density to obtain a scoring result. A priority processing queue is constructed according to the scoring result, and the classified annotated data in the priority processing queue is cleaned and normalized to obtain a standardized training data set;

[0077] S102. A multi-layer cascade network structure is used to perform deep learning processing on the standardized training data set. The multi-layer cascade network structure identifies the target features in the standardized training data set according to the category information of the classified annotation data and through a self-attention mechanism, and constructs a feature association graph based on the target features. A bidirectional reasoning channel is constructed on the feature association graph, and the bidirectional reasoning channel includes a forward channel and a reverse channel, wherein the forward channel extracts the numerical distribution features, time series features and attribute features in the standardized training data set through a convolutional neural network to obtain explicit feature data, and the reverse channel extracts the data item association rules, data change trends and data interaction patterns in the standardized training data set through a recurrent neural network to obtain implicit feature data, and a dynamic weight allocation strategy is used to perform feature combination on the explicit feature data and the implicit feature data to obtain a multidimensional feature tensor;

[0078] S103. Based on a preset domain knowledge base, rule matching is performed on the multidimensional feature tensor to obtain an inference rule set, and the multidimensional feature tensor is hierarchically processed using the inference rule set to obtain initial structured data. The initial structured data is analyzed by a causal reasoning method to obtain a decision basis chain, and optimization parameters are generated based on the decision basis chain. The initial structured data is optimized according to the optimization parameters and the category information of the classified annotation data to obtain final structured data.

[0079] Among them, intelligent data collector refers to a technical tool or device that can automatically acquire, process and store unstructured data. It collects unstructured data such as text, images, audio, video, etc. by identifying and parsing data from different sources, while ensuring the integrity and accuracy of the data. Data feature fingerprint library is a database specifically used to store and manage data features. It extracts and fingerprints the multi-dimensional features of the data content (such as shape features, structural features, statistical features, etc.) to form a unique and identifiable set of data features for data classification, retrieval and annotation. Feature association graph is a data expression form based on graph structure, which is used to describe the association relationship between features. Its nodes represent features, and edges represent the dependencies or interactions between features. Through this graph, the intrinsic structure of the data can be better captured.

[0080] The bidirectional reasoning channel is a computing framework that combines forward reasoning and reverse reasoning. The forward channel uses convolutional neural networks to extract numerical distribution features, time series features, and attribute features, while the reverse channel uses recurrent neural networks to capture the complex association rules and change trends between data items. The dynamic weight allocation strategy is a technology that dynamically adjusts the weights of different features according to task requirements. By optimizing the feature combination process, the comprehensive analysis capabilities of explicit and implicit features can be improved.

[0081] Multidimensional feature tensor is a tensor used to represent high-dimensional data structures in deep learning. It integrates explicit feature data and implicit feature data, and can provide rich contextual information for subsequent reasoning and decision-making.

[0082] In an optional implementation, using an intelligent data collector to acquire unstructured data, constructing a data feature fingerprint library, and classifying and labeling the unstructured data based on the data feature fingerprint library to obtain classified labeling data includes:

[0083] Acquire unstructured data through an intelligent data collector, the intelligent data collector includes a local feature channel and a global feature channel, wherein the local feature channel uses a hierarchical attention network to extract word-level features and sentence-level features to obtain local correlation features, the global feature channel uses a graph neural network to construct a data relationship graph and extracts global relationship features through message passing, and the local correlation features are fused with the global relationship features to obtain fused feature data;

[0084] The feature importance weights are obtained by performing multi-head attention calculation on the fused feature data, the fused feature data are screened according to the feature importance weights, a dynamic local sensitive hashing algorithm is used to map the screened features to generate feature fingerprints, and the feature fingerprints are constructed as a data feature fingerprint library that supports nearest neighbor queries;

[0085] Performing data enhancement on the fused feature data to obtain an enhanced feature set, calculating the contrast loss of sample pairs based on the enhanced feature set to obtain a semantic similarity matrix, constructing a data association graph using the neighbor query results of the data feature fingerprint library and the semantic similarity matrix, propagating known labels on the data association graph through graph attention to obtain an initial classification annotation result, and extracting entity relationships according to the data association graph to construct a domain knowledge graph;

[0086] Construct a meta-learner, use the feature vector and confidence of the initial classification and labeling results as input, evaluate and model the classification results of each sample, obtain a classification reliability score, mark the samples that are lower than a preset reliability threshold and verified by the data feature fingerprint library as the first type of data to be reviewed, use the domain knowledge graph to perform logical verification on the initial classification and labeling results, and mark the samples that have not passed the verification as the second type of data to be reviewed;

[0087] The selection process of the first category of data to be reviewed and the second category of data to be reviewed is modeled as a Markov decision process, a hybrid reward function is constructed according to the classification reliability score and the sample information gain, the sample selection strategy is optimized through deep reinforcement learning, and the optimal sample to be reviewed is selected for manual labeling to obtain a manual labeling result;

[0088] The manual annotation results are weightedly fused with the initial classification annotation results, and the consistency of the weighted fused annotation results is verified using the similarity index of the data feature fingerprint library to generate classification annotation data including the fused feature data, semantic similarity matrix, classification reliability score and domain knowledge graph verification results.

[0089] Exemplarily, when using an intelligent data collector to acquire unstructured data, feature extraction is first performed through local feature channels and global feature channels. In the local feature channel, the hierarchical attention network includes a word-level attention layer and a sentence-level attention layer. The word-level attention layer uses a bidirectional long short-term memory network to encode the input text to obtain a word vector sequence, and calculates the importance weight of each word through the attention mechanism, and weights the word-level feature representation. Based on the word-level features, the sentence-level attention layer calculates the degree of correlation between sentences, generates a sentence-level feature representation, and finally obtains a local correlation feature.

[0090] In the global feature channel, the graph neural network first builds a graph structure based on the association between data. Nodes represent data instances, and edges represent the relationship between instances. Through the message passing mechanism, each node obtains information from neighboring nodes and updates its own representation. After multiple iterations, node features containing global relationship information are obtained. The local association features and global relationship features are fused through the multi-head attention mechanism to obtain fused feature data.

[0091] When performing feature screening on fused feature data, the multi-head attention is calculated to obtain the importance weight of the feature dimension, and the feature dimension with higher weight is retained. The dynamic local sensitive hashing algorithm is used to reduce the dimension of the screened features to generate a compact feature fingerprint. The feature fingerprint is stored in an index structure that supports KNN nearest neighbor query to build a data feature fingerprint library.

[0092] In the data enhancement stage, the fused feature data is randomly cropped, rotated, and transformed to generate enhanced samples. The contrast loss between the original sample and the enhanced sample is calculated to obtain the semantic similarity matrix between the samples. The data association graph is constructed by combining the neighbor query results and the semantic similarity matrix of the feature fingerprint library. The known labels are propagated on the association graph through the graph attention network to obtain the initial classification results of the unlabeled samples. At the same time, the entity relationship triples are extracted from the association graph to construct the domain knowledge graph.

[0093] The meta-learner uses a gradient boosting tree model. The input features include the sample's feature vector and classification confidence, and the output is a classification reliability score. Samples with scores below the threshold and verified by the feature fingerprint library are marked as the first type of data to be reviewed. The relationship rules in the domain knowledge graph are used to logically verify the initial classification results, and samples that do not meet the rules are marked as the second type of data to be reviewed.

[0094] The selection process of the first and second types of data to be reviewed is modeled based on the Markov decision process. In this modeling process, each sample is regarded as a state, and the decision of the system to move from the current state to the next state is based on a series of constraints and objective functions. By modeling the selection process of the data to be reviewed, dynamic optimization can be achieved to ensure that the selected samples are the most valuable for manual annotation.

[0095] The classification reliability score reflects the credibility of the classification result of the sample. It is calculated based on multiple factors such as the consistency and confidence of the initial classification annotation result and the sample features, as well as the similarity with related samples in the data feature fingerprint library. Sample information gain refers to the potential contribution of the annotation results of each sample in improving the performance of the overall data classification model. By combining the classification reliability score and the sample information gain, a hybrid reward function is constructed. The goal of this function is to balance the selection of samples with high-value information while avoiding the selection of repetitive or inefficient samples.

[0096] Deep reinforcement learning is introduced to optimize sample selection strategies. In this process, the reinforcement learning agent continuously updates its strategy through interaction with the environment to maximize the mixed reward function. Specifically, the agent selects the best sample to be reviewed in each round of decision-making based on the current sample characteristics, classification reliability score, and information gain calculation. As training progresses, the agent gradually learns to more effectively select samples that are most valuable for improving data quality and model performance.

[0097] After obtaining the manual annotation results, they are weighted fused with the initial classification annotation results. Weighted fusion achieves priority trust in manual annotation results by assigning dynamic weights, while retaining the reliable parts of the initial classification annotation results. This process ensures that the fused annotation results can reflect the high accuracy of manual annotation while maintaining the robustness of the model.

[0098] The consistency of the weighted fusion annotation results is verified through the similarity index of the data feature fingerprint library. The data feature fingerprint library provides high-dimensional feature fingerprints of known samples. Through fast retrieval and similarity matching, it can be verified whether the fusion annotation results are consistent with the known samples. This step is to troubleshoot potential annotation errors or inconsistencies and further improve the reliability of the annotation results.

[0099] The final generated classification annotation data includes not only fusion feature data, but also semantic similarity matrix, classification reliability score and domain knowledge graph verification results. This information constitutes a complete data annotation system, which provides a strong foundation for subsequent data analysis, model training and decision support.

[0100] In this embodiment, through the dual-channel feature extraction method of hierarchical attention network and graph neural network, the local semantic features of the data are captured, and the global relationship between the data is modeled, which improves the comprehensiveness and accuracy of the feature representation. Dynamic local sensitive hashing and neighbor indexing are used to build a feature fingerprint library, and contrastive learning and graph attention propagation are combined to achieve efficient label propagation, which greatly reduces the cost of manual labeling. At the same time, logical verification is performed through the knowledge graph to ensure the accuracy of the labeling results. Based on the active learning strategy of deep reinforcement learning, the most valuable samples are adaptively selected for manual labeling, which significantly improves the labeling efficiency. The consistency and reliability of the labeling results are ensured through multiple verification mechanisms, which improves the quality of classification labeling.

[0101] In an optional implementation, an incremental learning algorithm is used to construct a data quality assessment model, the data quality assessment model scores the classified and labeled data from three dimensions of data integrity, timeliness, and value density to obtain a scoring result, a priority processing queue is constructed according to the scoring result, and the classified and labeled data in the priority processing queue are cleaned and normalized to obtain a standardized training data set, including:

[0102] Obtaining timestamp information of the classified and labeled data, using an exponential decay function to calculate a data time decay coefficient, and performing weighted processing on the classified and labeled data based on the data time decay coefficient to obtain a sample weight;

[0103] Constructing a data quality assessment model, using the sample weight as a weight term to construct a loss function, using an online stochastic gradient descent algorithm to optimize the loss function until the model converges, obtaining a trained data quality assessment model, and using the trained data quality assessment model to perform quality assessment on the classified and labeled data;

[0104] The effective feature ratio is calculated based on the quality assessment result to obtain a data integrity score, the data timeliness score is obtained according to the data time decay coefficient, and the data value density score is calculated in combination with the sample weight. The data integrity score, data timeliness score and data value density score are weighted and fused to obtain a data quality score;

[0105] Constructing a priority processing queue according to the data quality score, marking the classified labeled data whose data quality score is lower than a first preset threshold as data to be eliminated, marking the classified labeled data whose data quality score is higher than a second preset threshold as data for priority processing, calculating the data processing priority based on the data quality score and the sample weight, and generating a priority data queue with processing marks;

[0106] According to the priority order of the priority data queue, the field name and data format of the classified and labeled data are uniformly converted to obtain standardized features, a time series completion model is constructed based on the data timeliness score and the data integrity score, and the missing fields are completed using the time series completion model to obtain structured features;

[0107] Constructing feature fingerprints according to the structural features and the data value density scores, identifying duplicate data based on the feature fingerprints, performing anomaly detection and denoising processing in combination with the data quality scores, and generating quality processing records and cleaned feature data;

[0108] The numerical features, category features and text features in the cleaned feature data are respectively standardized and converted, and the feature fingerprint is used to guide the word vector mapping of the text features, and the converted standardized features are combined with the data quality score, data processing priority and quality processing record to generate a standardized training data set.

[0109] Exemplarily, first obtain the classified annotated data, including data content and timestamp information. For each piece of data, calculate the time decay coefficient based on its generation time. In specific implementation, you can set a reference time point, such as the current time, calculate the time difference between each piece of data and the reference time, and calculate the decay coefficient by the time difference. For example, for a piece of data generated 30 days ago, its time decay coefficient may be 0.85, while the decay coefficient of data 60 days ago may drop to 0.72.

[0110] Based on the calculated time decay coefficient, the data is weighted to obtain the sample weight. The weight value reflects the timeliness of the data, and newer data has a higher weight. For example, if the time decay coefficient of a piece of data is 0.85, its initial sample weight may be set to 0.85.

[0111] When constructing a data quality assessment model, the sample weight is used as the weight term of the loss function. The model is trained using online stochastic gradient descent. A batch of data is randomly selected for training each time, and the model parameters are continuously adjusted until convergence. The trained model can be used to assess data quality.

[0112] For each piece of data, we score it based on three dimensions: completeness, timeliness, and value density:

[0113] The completeness score is calculated based on the proportion of valid features. For example, if a piece of data contains 10 feature fields, 8 of which are valid, its completeness score may be 0.8.

[0114] The timeliness score directly uses the time decay coefficient calculated previously. For example, if the decay coefficient is 0.85, the timeliness score is 0.85.

[0115] The value density score is calculated in combination with the sample weight. For example, if the sample weight of a data is 0.85 and its feature information richness is high, its value density score may be 0.88.

[0116] The scores of the three dimensions are weighted and integrated to obtain the final data quality score. For example, the completeness score, timeliness score, and value density score can be weighted 0.3, 0.3, and 0.4 respectively, and the weighted average is calculated as the final score.

[0117] Construct a priority processing queue based on data quality scores. Set two thresholds, such as 0.6 and 0.8. Data with a score lower than 0.6 is marked as to be eliminated, and data with a score higher than 0.8 is marked as priority processing. Combine the quality score and sample weight to calculate the processing priority and generate a priority queue.

[0118] Process the data in order of priority. First, standardize the fields, unify dates in different formats into a standard format, and convert values ​​in different units into a unified unit. For missing fields, fill them in based on the time series completion model. For example, for missing sales data, fill them in based on historical data forecasts.

[0119] Construct feature fingerprints for deduplication and anomaly detection. Feature fingerprints can contain hash value combinations of key fields. For example, for commodity data, fields such as category, specification, and production date can be combined to generate fingerprints. Duplicate data is identified based on fingerprints, and anomaly detection is performed in combination with quality scores.

[0120] Finally, the cleaned data is standardized. The numerical features are normalized, the category features are converted into encoding forms, and the text features are converted into word vectors. The standardized features are combined with information such as quality scores to generate the final training data set.

[0121] In this embodiment, through the time decay mechanism and multi-dimensional quality assessment, the accurate quantification of data value is achieved, the accuracy and efficiency of data processing are improved, and the data quality assessment is more objective and reasonable. The priority queue and feature fingerprint mechanism are adopted to realize the intelligent scheduling and abnormal identification of data processing, reduce the resource consumption of data processing, and improve the efficiency and accuracy of data cleaning. Through standardized processing and feature conversion, the uniformity of data format and the standardization of feature expression are ensured, the effect of subsequent model training is improved, and the reliability and stability of data application are enhanced.

[0122] In an optional implementation, a multi-layer cascade network structure is used to perform deep learning processing on the standardized training data set, wherein the multi-layer cascade network structure identifies target features in the standardized training data set according to category information of the classified annotation data and through a self-attention mechanism, including:

[0123] Constructing a multi-layer cascade network structure, the multi-layer cascade network structure includes a feature extraction layer, an attention layer and a feature fusion layer, wherein the feature extraction layer is composed of multiple convolution blocks, the attention layer adopts a self-attention mechanism, and the feature fusion layer is used for feature dimensionality reduction and combination;

[0124] Extracting category information from the classified labeled data in the standardized training data set, constructing a category embedding matrix, and calculating a category mapping matrix based on the category embedding matrix and the standardized training data set;

[0125] The standardized training data set is processed using a feature extraction layer of a multi-layer cascade network structure, and the category mapping matrix is ​​multiplied by the extracted features to obtain category-aware features, and the category-aware features extracted at different levels are spliced ​​in the channel dimension to obtain multi-scale cascade features;

[0126] In the attention layer, a query matrix, a key matrix, and a value matrix are generated respectively based on the multi-scale cascade features and the category embedding matrix, and the similarity between the query matrix and the key matrix is ​​calculated through the self-attention mechanism to obtain the attention weight, and the attention weight is multiplied by the value matrix to obtain the attention feature;

[0127] In the feature fusion layer, the attention features are residually connected and normalized to obtain normalized features, gating weights are constructed based on the normalized features and the category embedding matrix, and the gating weights are weighted with the normalized features to obtain enhanced features;

[0128] The enhanced features of each layer are transferred layer by layer in the multi-layer cascade network structure and feature cascade is performed. Through cross-layer feature aggregation, the target features in the standardized training data set are obtained.

[0129] For example, the construction process of the multi-layer cascade network structure first needs to design the feature extraction layer. The feature extraction layer is composed of multiple convolution blocks, each of which contains three convolution layers, with convolution kernel sizes of 3×3, 5×5, and 7×7, respectively, a step size of 1, and the padding method using same padding. Each convolution layer is followed by a batch normalization layer and a ReLU activation function to extract feature information of different scales.

[0130] For the classified labeled data in the standardized training dataset, the category information is converted into a vector form by one-hot encoding. Assuming that the dataset contains 10 categories, each category label is converted into a 10-dimensional vector. These vectors are combined into a category embedding matrix with the matrix dimension of the number of categories × embedding dimension. Taking the image classification task as an example, the input image size is 224×224×3, and the feature map size obtained by the feature extraction layer is 28×28×256.

[0131] In the feature extraction process, the category mapping matrix is ​​dot-producted with the extracted features to obtain features with category perception. The features extracted by different convolution blocks are concatenated in the channel dimension to form multi-scale cascade features. Taking the above image classification task as an example, the feature dimensions of the three different scales are 28×28×256, 14×14×512 and 7×7×1024, respectively. After concatenation, a multi-scale feature of 28×28×1792 is obtained.

[0132] During the processing of the attention layer, the query matrix is ​​generated based on the multi-scale cascade features, and the key matrix and value matrix are generated based on the category embedding matrix. The attention weight is obtained by calculating the similarity between the query matrix and the key matrix, and the weight is normalized using the softmax function. The normalized attention weight is multiplied by the value matrix to obtain the attention feature that integrates the category information.

[0133] In the feature fusion layer, the attention features are first residually connected, the input features are added to the attention features, and then standardized by layer normalization. Based on the normalized features and the category embedding matrix, a gating weight mechanism is constructed using a fully connected layer. The weight is used to control the importance of different features. The gating weight is element-wise multiplied with the normalized features to obtain the enhanced feature representation.

[0134] Finally, the enhanced features of each layer are passed layer by layer in the network, and feature aggregation is performed using cross-layer connections. Specifically, the low-level features are aligned with the high-level feature size through upsampling, and then spliced ​​in the channel dimension, finally obtaining the target features that integrate multi-level information.

[0135] In this embodiment, through the combination of multi-layer cascade structure and self-attention mechanism, it is possible to effectively capture multi-scale feature information in the data, improve the comprehensiveness and accuracy of feature extraction, and enhance the model's adaptability to targets of different scales. The introduction of category information to guide the feature extraction process, through the construction of category-aware features and the application of attention mechanism, improves the model's ability to recognize category-related features and reduces noise interference in the feature extraction process. By combining the gating mechanism with the residual connection, the adaptive fusion and optimization of features are achieved, the discriminability of feature representation is improved, while maintaining the original feature information and enhancing the generalization ability of the model.

[0136] In an optional implementation, performing rule matching on the multidimensional feature tensor based on a preset domain knowledge base to obtain an inference rule set includes:

[0137] Using a feature dimensionality reduction network to perform dimensionality reduction processing on a multidimensional feature tensor to obtain a feature sequence, constructing a feature index tree on the feature sequence, setting a sliding window of a dynamic size based on the feature index tree, the size of the sliding window being adaptively adjusted according to the feature sequence, and performing sliding scanning on the feature sequence to extract feature patterns;

[0138] Input the feature pattern into a pre-trained text encoder to obtain a feature encoding vector, and input the rule pattern in a preset domain knowledge base into a pre-trained text encoder to obtain a rule encoding vector; calculate the cosine similarity between the feature encoding vector and the rule encoding vector as the matching degree;

[0139] Calculating the mean and standard deviation of the matching degree, determining an adaptive threshold coefficient according to a preset confidence interval, weighting the mean and standard deviation based on the adaptive threshold coefficient to obtain an adaptive threshold, screening rule patterns with a matching degree higher than the adaptive threshold, and forming a candidate rule set with the screened rule patterns and their corresponding structured templates;

[0140] Dependency analysis is performed on the rules in the candidate rule set, and dependency relationships between the rules are established based on the input features and output features of the rules. The dependency relationships are represented as a directed graph, in which nodes represent rules and directed edges represent dependency relationships. The in-degree and out-degree of each rule node are calculated, and the dependency level of the rules is determined. An improved topological sorting algorithm is used to assign rules with direct dependency relationships to adjacent levels to obtain an inference rule set.

[0141] Exemplarily, first, the input multidimensional feature tensor is subjected to feature dimensionality reduction processing. A deep neural network is used as the feature dimensionality reduction network, which includes multiple convolutional layers and pooling layers. Taking the medical diagnosis scenario as an example, the input multidimensional feature tensor contains various examination indicators of the patient, with a dimension of 128x64x32. It is converted into a one-dimensional feature sequence with a length of 512 through the feature dimensionality reduction network.

[0142] After obtaining the feature sequence, a feature index tree is constructed. The feature sequence is stored in a B+ tree structure, with leaf nodes storing feature values ​​and non-leaf nodes storing index information. A sliding window of dynamic size is set on the feature sequence, with the initial window size set to 32 and dynamically adjusted according to the local changes in the feature sequence. When the feature sequence fluctuates greatly, the window size is reduced to capture fine-grained features; when the feature sequence is stable, the window size is increased to obtain macro features.

[0143] The sliding window scans the feature sequence to extract feature patterns. Each feature pattern contains a sequence of feature values ​​within the window. The extracted feature pattern is input into the pre-trained BERT text encoder to obtain a 768-dimensional feature encoding vector. At the same time, the rule pattern in the preset domain knowledge base is also input into the same encoder to obtain a rule encoding vector.

[0144] The cosine similarity between the feature encoding vector and the rule encoding vector is calculated to obtain the matching degree. Taking the medical diagnosis rule as an example, a rule pattern is "blood pressure continues to rise and is accompanied by headaches", and the cosine similarity between its rule encoding vector and the encoding vector of a feature pattern is 0.85.

[0145] Count the mean and standard deviation of all matching degrees. Assume the mean is 0.7 and the standard deviation is 0.15. Set a 95% confidence interval, and the corresponding adaptive threshold coefficient is 1.96. Weight the mean and standard deviation to get the adaptive threshold: 0.7+1.96×0.15=0.994. Filter the rule patterns with matching degrees higher than the threshold to form a candidate rule set.

[0146] Perform dependency analysis on the rules in the candidate rule set. Build a dependency network between rules by analyzing the input and output features of the rules. Dependencies are represented by directed graphs, where nodes represent rules and directed edges represent dependencies between rules. Calculate the in-degree and out-degree of each node to identify the relative position and importance of the rules.

[0147] An improved topological sorting algorithm is used to hierarchically divide the rules according to the dependencies between the rules. The result of topological sorting is an ordered list, which ensures that the execution order of the rules conforms to the logical dependencies and avoids the occurrence of circular dependencies.

[0148] Finally, the sorted rule sets are organized into inference rule sets. The rules in the inference rule set are executed step by step according to the dependency hierarchy to ensure the orderliness and efficiency of the inference process.

[0149] In this embodiment, by extracting feature patterns through a dynamically adjusted sliding window, feature information of different scales can be adaptively captured, thereby improving the flexibility and accuracy of feature extraction. A pre-trained text encoder is used to calculate the similarity between feature patterns and rule patterns, making full use of the semantic understanding ability of the deep learning model to improve the accuracy and generalization ability of rule matching. A hierarchical reasoning rule set is constructed based on rule dependencies to make the logical relationship between rules clearer and improve the interpretability and efficiency of the reasoning process.

[0150] In an optional implementation, the multidimensional feature tensor is hierarchically processed using the inference rule set to obtain initial structured data including:

[0151] For each rule layer in the inference rule set, feature vectors of rule-related regions are extracted from feature sequences, and the feature vectors are input into a rule matching network, which includes a feature conversion layer and a matching degree calculation layer, and outputs a rule applicability score, which is normalized as the rule confidence;

[0152] Based on the rule confidence, the rules in each rule layer are sorted in descending order to obtain a sorted rule set, and the rules are sequentially applied to the corresponding structured templates in the sorted order to generate structured fragments through feature mapping, wherein the structured template defines a mapping relationship between features and structured data;

[0153] Detecting the effective areas of the rules in the sorting rule set, when the effective areas of multiple rules overlap, extracting the range of the overlapping area, calculating the coverage of each rule in the overlapping area, taking the weighted sum of the rule confidence and coverage as the rule priority, selecting the rule with the highest priority and applying it to the overlapping area, and ensuring the generation of a unique structured fragment;

[0154] Construct a multi-layer verification network including attribute verification layer, relationship verification layer and value range verification layer. In the attribute verification layer, the integrity of the structured fragment is verified based on the attribute graph. In the relationship verification layer, the rationality of the relationship between the structured fragments is verified based on the knowledge graph. In the value range verification layer, the validity of the attribute values ​​in the structured fragment is verified based on the statistical model. The verification results of each layer are integrated to obtain the verified structured fragment.

[0155] Extract semantic features from the verified structured fragments, calculate the semantic similarity between the fragments, build a hierarchical organizational structure of the fragments based on the semantic similarity, integrate semantically related structured fragments according to the hierarchical organizational structure, and generate initial structured data that meets preset specifications.

[0156] Exemplarily, the multidimensional feature tensor is first processed hierarchically using an inference rule set to generate initial structured data, and the feature sequence is first extracted by region according to the rule level in the inference rule set. In the extraction process, for each rule in the rule level, the feature region related to the rule condition is identified, and its feature vector is extracted as input.

[0157] The extracted feature vector is input into the rule matching network. The rule matching network consists of a feature conversion layer and a matching degree calculation layer. The feature conversion layer normalizes and enhances the input feature vector. The matching degree calculation layer calculates the degree of matching between the feature and the rule conditions and outputs the applicability score of each rule. The applicability score reflects the degree of fit between the rule and the feature. Subsequently, the applicability score is standardized to generate the rule confidence, which is a key indicator for measuring the reliability of the rule.

[0158] Based on the rule confidence, the rules in each rule layer are sorted in descending order to generate a sorted rule set. The sorted rule set is arranged in order according to the importance and applicability of the rules to ensure that high-confidence rules are applied first. Subsequently, the rules in the sorted rule set are applied to the corresponding structured template in turn. The structured template defines the mapping rules from features to structured data, and generates structured fragments by combining the mapping relationship between feature vectors and templates. Structured fragments are the basic units of initial structured data.

[0159] When the rule action areas overlap, detect the range of the overlapping area, extract all the rules in the overlapping area, and calculate the coverage of each rule in the overlapping area. Coverage represents the proportion of the scope of the rule in the overlapping area. Weight the rule confidence and coverage to get the rule priority. The rule with the highest priority is applied to the overlapping area to ensure that only unique structured fragments are generated in each area, thus avoiding data conflicts and redundancy.

[0160] A multi-layer verification network is constructed to perform multi-dimensional verification on the generated structured fragments. The multi-layer verification network includes an attribute verification layer, a relationship verification layer, and a value range verification layer. The attribute verification layer checks the integrity of the structured fragment based on the attribute graph, for example, confirming whether the fragment contains all necessary attribute values. The relationship verification layer verifies whether the relationship between structured fragments is reasonable based on the knowledge graph, for example, checking whether different fragments conform to predefined logical or semantic associations. The value range verification layer checks the validity of the fragment attribute values ​​based on the statistical model, for example, confirming whether the value is within a reasonable range. Integrate these verification results, correct or remove the fragments that do not meet the specifications, and obtain the final verified structured fragment.

[0161] Semantic features are extracted from the verified structured fragments, and the relevance between fragments is calculated through semantic similarity. Semantic features capture the deep semantic information of the fragment content, and semantic similarity reflects the degree of semantic relevance between fragments. A hierarchical organizational structure of fragments is constructed based on semantic similarity, and semantically related fragments are aggregated into larger units. By integrating related fragments, initial structured data that meets the preset specifications is generated.

[0162] In this embodiment, the accuracy of feature extraction is improved through hierarchical processing and rule matching. The rule matching network can adapt to the feature representation of different scenarios, improving the flexibility and robustness of rule application. The multi-layer verification network realizes multi-dimensional data quality control, ensures the accuracy of structured results from aspects such as attribute integrity, relationship rationality and value validity, and significantly improves data reliability. The hierarchical organization based on semantic similarity realizes the intelligent integration of structured fragments, solves the problems of data redundancy and inconsistency, and improves the standardization and availability of structured data.

[0163] In an optional implementation, analyzing the initial structured data by a causal reasoning method to obtain a decision basis chain, generating optimization parameters based on the decision basis chain, and optimizing the initial structured data according to the optimization parameters and the category information of the classified annotation data to obtain the final structured data includes:

[0164] Calculating partial correlation coefficients for variable pairs in the initial structured data, performing conditional independence tests based on the partial correlation coefficients to obtain conditional independence relationships between variables, and constructing a causal graph based on the conditional independence relationships;

[0165] A linear structural equation model is constructed based on the causal diagram, and the parameters of the linear structural equation model are optimized by maximum likelihood estimation to obtain a structural causal model, wherein the structural causal model is used to quantify the strength of causal effects between variables in the causal diagram;

[0166] Calculate the number of associated edges of each node in the causal graph, determine the node whose number of associated edges exceeds a preset threshold of the number of associated edges as a decision node, trace the associated nodes from the decision node along the causal edges in the causal graph step by step to obtain a factor chain, calculate the cumulative causal effect of the factor chain based on the structural causal model, and determine the factor chain whose cumulative causal effect exceeds a preset effect threshold as a candidate decision basis chain;

[0167] In the structural causal model, the value of each node in the candidate decision basis chain is changed and substituted into the structural causal model to obtain the downstream node prediction value corresponding to the node value change, and the stability of the candidate decision basis chain is determined based on the change range of the prediction value, and the candidate decision basis chain whose stability meets the preset conditions is determined as the decision basis chain;

[0168] Based on the structural causal model, the influence degree of each node in the decision basis chain on the target variable is calculated, and the nodes whose influence degree exceeds a preset influence threshold are selected as a feature set according to the size of the influence degree, and the influence degree of each node in the feature set is used as a feature importance score, and the feature importance score is combined with the category discrimination degree of the classified annotation data to obtain a feature optimization parameter;

[0169] Based on the feature optimization parameters, weighted adjustment is performed on the values ​​of corresponding features in the initial structured data to obtain optimized data, wherein the feature optimization parameters are used to determine the weight coefficient of the feature adjustment;

[0170] The difference in feature distribution between the optimized data and the initial structured data is calculated to obtain an optimization effect index, the optimized data is input into the structural causal model to verify the causal relationship constraints to obtain a verification result, the optimized data is iteratively optimized according to the optimization effect index and the verification result, and when the optimization index reaches a preset target and the verification result satisfies the causal consistency, the current optimized data is determined as the final structured data.

[0171] For example, all variables in the initial structured data are first paired, and the partial correlation coefficient between each pair of variables is calculated. The partial correlation coefficient reflects the degree of linear relationship between two variables, while excluding the interference of other variables. By analyzing the partial correlation coefficients between variables, it is possible to preliminarily determine which variables may have direct or indirect causal relationships.

[0172] Based on the calculated partial correlation coefficient, a conditional independence test is further performed to determine whether the variables are independent after controlling other variables. If two variables are conditionally independent, there is no direct causal relationship between them. Based on these conditional independence relationships, a causal graph is constructed to represent the causal relationship between variables. The nodes of the causal graph represent variables, and the directed edges represent the direct causal effects between variables. During the construction process, it is necessary to ensure that the structure of the graph is an acyclic graph and conforms to prior knowledge or domain constraints.

[0173] Based on the causal diagram, a structural equation model is constructed to quantify the strength of the causal relationship between variables. In the structural equation model, the value of each variable can be expressed as a linear combination of its causal precursor variables. The maximum likelihood estimation method is used to optimize the model parameters, and finally a structural causal model that describes the quantitative relationship of the causal diagram is generated. This model is used to evaluate the size of the causal effect between variables and provide a quantitative basis for subsequent analysis.

[0174] Analyze the nodes in the causal graph and mark the nodes with more than a preset threshold of associated edges as decision nodes. Decision nodes are usually in a key position in the causal graph and have a greater impact on the overall causal structure. From each decision node, trace back to its other associated nodes along the causal path to form a series of causal chains. The cumulative causal effect of these causal chains is quantified using the structural causal model. The cumulative causal effect represents the overall causal influence strength from the starting point to the end point of the chain. By setting the threshold of the cumulative causal effect, the chains with stronger causal effects are screened out as candidate decision basis chains.

[0175] Verify the stability of the candidate decision basis chain. Change the values ​​of the nodes in the chain one by one in the structural causal model, and observe the change in the predicted values ​​of the downstream nodes in the chain. If a candidate decision basis chain can maintain a high prediction stability when the node values ​​change, it is more reliable. The candidate decision basis chain that meets the stability condition is confirmed as the final decision basis chain.

[0176] The structural causal model is used to calculate the direct or indirect impact of each node in the decision-making basis chain on the target variable and quantify it as the degree of impact. The nodes whose impact exceeds the threshold are filtered as feature sets, and the impact of each node is recorded as the feature importance score. Combined with the statistical characteristics of the category discrimination in the classified annotation data, the feature importance score and the category discrimination are calculated comprehensively to generate the feature optimization parameters used in the optimization process.

[0177] Based on the feature optimization parameters, the corresponding features in the initial structured data are weighted and adjusted. The weighted adjustment process takes into account the size of the feature optimization parameters to ensure that the key features are given higher weights, thereby optimizing their values ​​more significantly. The adjusted data is used as optimized data and needs to be further verified for effectiveness.

[0178] Compare the optimized data with the initial data, analyze the changes in the feature distribution of the optimized data, and generate optimization effect indicators. At the same time, re-enter the optimized data into the structural causal model to verify whether it meets the causal constraint conditions. If the causal constraint is violated during the optimization process, it is necessary to readjust the feature optimization parameters and data weight allocation strategy.

[0179] Through multiple rounds of iteration, the consistency between the feature distribution of the optimized data and the target distribution is gradually improved, while ensuring that the optimization results maintain causal consistency. When the optimization effect index reaches the preset target range and the causal verification result meets the constraint requirements, the iteration is stopped and the current optimized data is confirmed as the final structured data.

[0180] The structured data finally generated meets the preset requirements in terms of causal consistency, rationality of feature distribution and optimization effect, and can be directly used for subsequent data analysis or decision support tasks.

[0181] In this embodiment, by constructing a causal graph and a structural causal model, the causal relationship in the data is deeply excavated, and the accuracy and interpretability of data analysis are improved. Compared with traditional correlation analysis, causal reasoning can better reflect the true relationship between variables and avoid the misleading caused by false correlation. The concept of decision-making basis chain is introduced, and a reliable theoretical basis is provided for data optimization by tracing the causal chain and evaluating its stability. This causal-based feature selection and optimization method can more effectively identify key features and improve the generalization ability and robustness of the model. Causal consistency is taken into account in the optimization process, and iterative verification is used to ensure that the optimized data still meets the original causal relationship constraints. This approach not only improves data quality, but also ensures the interpretability and credibility of the optimization results, providing a more reliable data basis for subsequent decision analysis and model application.

[0182] Figure 2 FIG. 1 is a schematic diagram of a structure of an automatic processing system for unstructured data based on deep learning according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0183] The first unit is used to acquire unstructured data using an intelligent data collector, build a data feature fingerprint library, classify and annotate the unstructured data based on the data feature fingerprint library to obtain classified annotated data, use an incremental learning algorithm to build a data quality assessment model, the data quality assessment model scores the classified annotated data from three dimensions of data integrity, timeliness and value density to obtain a scoring result, build a priority processing queue according to the scoring result, and perform data cleaning and normalization on the classified annotated data in the priority processing queue to obtain a standardized training data set;

[0184] The second unit is used to perform deep learning processing on the standardized training data set using a multi-layer cascade network structure, wherein the multi-layer cascade network structure identifies target features in the standardized training data set based on the category information of the classified annotation data and through a self-attention mechanism, constructs a feature association graph based on the target features, and constructs a bidirectional reasoning channel on the feature association graph, wherein the bidirectional reasoning channel includes a forward channel and a reverse channel, wherein the forward channel extracts numerical distribution features, time series features, and attribute features in the standardized training data set through a convolutional neural network to obtain explicit feature data, and the reverse channel extracts data item association rules, data change trends, and data interaction patterns in the standardized training data set through a recurrent neural network to obtain implicit feature data, and a dynamic weight allocation strategy is used to perform feature combination on the explicit feature data and the implicit feature data to obtain a multidimensional feature tensor;

[0185] The third unit is used to perform rule matching on the multidimensional feature tensor based on a preset domain knowledge base to obtain an inference rule set, use the inference rule set to perform hierarchical processing on the multidimensional feature tensor to obtain initial structured data, analyze the initial structured data through a causal reasoning method to obtain a decision basis chain, generate optimization parameters based on the decision basis chain, and optimize the initial structured data according to the optimization parameters and category information of the classified annotation data to obtain final structured data.

[0186] According to a third aspect of the embodiments of the present invention,

[0187] An electronic device is provided, comprising:

[0188] processor;

[0189] a memory for storing processor-executable instructions;

[0190] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0191] According to a fourth aspect of the embodiments of the present invention,

[0192] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0193] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatic processing of unstructured data based on deep learning, characterized in that: include: Using an intelligent data collector to acquire unstructured data, building a data feature fingerprint library, classifying and labeling the unstructured data based on the data feature fingerprint library to obtain classified labeled data, using an incremental learning algorithm to build a data quality assessment model, the data quality assessment model scores the classified labeled data from three dimensions of data integrity, timeliness and value density to obtain a scoring result, building a priority processing queue based on the scoring result, and performing data cleaning and normalization on the classified labeled data in the priority processing queue to obtain a standardized training data set; A multi-layer cascade network structure is used to perform deep learning processing on the standardized training data set. The multi-layer cascade network structure identifies target features in the standardized training data set based on the category information of the classified annotation data and through a self-attention mechanism, constructs a feature association graph based on the target features, and constructs a bidirectional reasoning channel on the feature association graph. The bidirectional reasoning channel includes a forward channel and a reverse channel, wherein the forward channel extracts numerical distribution features, time series features, and attribute features in the standardized training data set through a convolutional neural network to obtain explicit feature data, and the reverse channel extracts data item association rules, data change trends, and data interaction patterns in the standardized training data set through a recurrent neural network to obtain implicit feature data, and a dynamic weight allocation strategy is used to perform feature combination on the explicit feature data and the implicit feature data to obtain a multidimensional feature tensor; Based on a preset domain knowledge base, rule matching is performed on the multidimensional feature tensor to obtain an inference rule set, the multidimensional feature tensor is hierarchically processed using the inference rule set to obtain initial structured data, the initial structured data is analyzed by a causal reasoning method to obtain a decision basis chain, optimization parameters are generated based on the decision basis chain, and the initial structured data is optimized according to the optimization parameters and category information of the classified annotation data to obtain final structured data.

2. The method according to claim 1, characterized in that: Using an intelligent data collector to acquire unstructured data, building a data feature fingerprint library, and classifying and labeling the unstructured data based on the data feature fingerprint library to obtain classified and labeled data includes: Acquire unstructured data through an intelligent data collector, the intelligent data collector includes a local feature channel and a global feature channel, wherein the local feature channel uses a hierarchical attention network to extract word-level features and sentence-level features to obtain local correlation features, the global feature channel uses a graph neural network to construct a data relationship graph and extracts global relationship features through message passing, and the local correlation features are fused with the global relationship features to obtain fused feature data; The feature importance weights are obtained by performing multi-head attention calculation on the fused feature data, the fused feature data are screened according to the feature importance weights, a dynamic local sensitive hashing algorithm is used to map the screened features to generate feature fingerprints, and the feature fingerprints are constructed as a data feature fingerprint library that supports nearest neighbor queries; Performing data enhancement on the fused feature data to obtain an enhanced feature set, calculating the contrast loss of sample pairs based on the enhanced feature set to obtain a semantic similarity matrix, constructing a data association graph using the neighbor query results of the data feature fingerprint library and the semantic similarity matrix, propagating known labels on the data association graph through graph attention to obtain an initial classification annotation result, and extracting entity relationships according to the data association graph to construct a domain knowledge graph; Construct a meta-learner, use the feature vector and confidence of the initial classification and labeling results as input, evaluate and model the classification results of each sample, obtain a classification reliability score, mark the samples that are lower than a preset reliability threshold and verified by the data feature fingerprint library as the first type of data to be reviewed, use the domain knowledge graph to perform logical verification on the initial classification and labeling results, and mark the samples that have not passed the verification as the second type of data to be reviewed; The selection process of the first category of data to be reviewed and the second category of data to be reviewed is modeled as a Markov decision process, a hybrid reward function is constructed according to the classification reliability score and the sample information gain, the sample selection strategy is optimized through deep reinforcement learning, and the optimal sample to be reviewed is selected for manual labeling to obtain a manual labeling result; The manual annotation results are weightedly fused with the initial classification annotation results, and the consistency of the weighted fused annotation results is verified using the similarity index of the data feature fingerprint library to generate classification annotation data including the fused feature data, semantic similarity matrix, classification reliability score and domain knowledge graph verification results.

3. The method according to claim 1, characterized in that An incremental learning algorithm is used to construct a data quality assessment model, which scores the classified and labeled data from three dimensions: data integrity, timeliness, and value density to obtain a scoring result, and a priority processing queue is constructed according to the scoring result. The classified and labeled data in the priority processing queue are cleaned and normalized to obtain a standardized training data set including: Obtaining timestamp information of the classified and labeled data, using an exponential decay function to calculate a data time decay coefficient, and performing weighted processing on the classified and labeled data based on the data time decay coefficient to obtain a sample weight; Constructing a data quality assessment model, using the sample weight as a weight term to construct a loss function, using an online stochastic gradient descent algorithm to optimize the loss function until the model converges, obtaining a trained data quality assessment model, and using the trained data quality assessment model to perform quality assessment on the classified and labeled data; The effective feature ratio is calculated based on the quality assessment result to obtain a data integrity score, the data timeliness score is obtained according to the data time decay coefficient, and the data value density score is calculated in combination with the sample weight. The data integrity score, data timeliness score and data value density score are weighted and fused to obtain a data quality score; Constructing a priority processing queue according to the data quality score, marking the classified labeled data whose data quality score is lower than a first preset threshold as data to be eliminated, marking the classified labeled data whose data quality score is higher than a second preset threshold as data for priority processing, calculating the data processing priority based on the data quality score and the sample weight, and generating a priority data queue with processing marks; According to the priority order of the priority data queue, the field name and data format of the classified and labeled data are uniformly converted to obtain standardized features, a time series completion model is constructed based on the data timeliness score and the data integrity score, and the missing fields are completed using the time series completion model to obtain structured features; Constructing feature fingerprints according to the structural features and the data value density scores, identifying duplicate data based on the feature fingerprints, performing anomaly detection and denoising processing in combination with the data quality scores, and generating quality processing records and cleaned feature data; The numerical features, category features and text features in the cleaned feature data are respectively standardized and converted, and the feature fingerprint is used to guide the word vector mapping of the text features, and the converted standardized features are combined with the data quality score, data processing priority and quality processing record to generate a standardized training data set.

4. The method according to claim 1, characterized in that: The standardized training data set is subjected to deep learning processing using a multi-layer cascade network structure, wherein the multi-layer cascade network structure identifies target features in the standardized training data set according to category information of the classified annotation data and through a self-attention mechanism, including: Constructing a multi-layer cascade network structure, the multi-layer cascade network structure includes a feature extraction layer, an attention layer and a feature fusion layer, wherein the feature extraction layer is composed of multiple convolution blocks, the attention layer adopts a self-attention mechanism, and the feature fusion layer is used for feature dimensionality reduction and combination; Extracting category information from the classified labeled data in the standardized training data set, constructing a category embedding matrix, and calculating a category mapping matrix based on the category embedding matrix and the standardized training data set; The standardized training data set is processed using a feature extraction layer of a multi-layer cascade network structure, and the category mapping matrix is ​​multiplied by the extracted features to obtain category-aware features, and the category-aware features extracted at different levels are spliced ​​in the channel dimension to obtain multi-scale cascade features; In the attention layer, a query matrix, a key matrix, and a value matrix are generated respectively based on the multi-scale cascade features and the category embedding matrix, and the similarity between the query matrix and the key matrix is ​​calculated through the self-attention mechanism to obtain the attention weight, and the attention weight is multiplied by the value matrix to obtain the attention feature; In the feature fusion layer, the attention features are residually connected and normalized to obtain normalized features, gating weights are constructed based on the normalized features and the category embedding matrix, and the gating weights are weighted with the normalized features to obtain enhanced features; The enhanced features of each layer are transferred layer by layer in the multi-layer cascade network structure and feature cascade is performed. Through cross-layer feature aggregation, the target features in the standardized training data set are obtained.

5. The method according to claim 1, characterized in that The inference rule set obtained by performing rule matching on the multi-dimensional feature tensor based on the preset domain knowledge base includes: Using a feature dimensionality reduction network to perform dimensionality reduction processing on a multidimensional feature tensor to obtain a feature sequence, constructing a feature index tree on the feature sequence, setting a sliding window of a dynamic size based on the feature index tree, the size of the sliding window being adaptively adjusted according to the feature sequence, and performing sliding scanning on the feature sequence to extract feature patterns; Input the feature pattern into a pre-trained text encoder to obtain a feature encoding vector, and input the rule pattern in a preset domain knowledge base into a pre-trained text encoder to obtain a rule encoding vector; calculate the cosine similarity between the feature encoding vector and the rule encoding vector as the matching degree; Calculating the mean and standard deviation of the matching degree, determining an adaptive threshold coefficient according to a preset confidence interval, weighting the mean and standard deviation based on the adaptive threshold coefficient to obtain an adaptive threshold, screening rule patterns with a matching degree higher than the adaptive threshold, and forming a candidate rule set with the screened rule patterns and their corresponding structured templates; Dependency analysis is performed on the rules in the candidate rule set, and dependency relationships between the rules are established based on the input features and output features of the rules. The dependency relationships are represented as a directed graph, in which nodes represent rules and directed edges represent dependency relationships. The in-degree and out-degree of each rule node are calculated, and the dependency level of the rules is determined. An improved topological sorting algorithm is used to assign rules with direct dependency relationships to adjacent levels to obtain an inference rule set.

6. The method according to claim 1, characterized in that The multidimensional feature tensor is processed hierarchically using the inference rule set to obtain initial structured data including: For each rule layer in the inference rule set, feature vectors of rule-related regions are extracted from feature sequences, and the feature vectors are input into a rule matching network, which includes a feature conversion layer and a matching degree calculation layer, and outputs a rule applicability score, which is normalized as the rule confidence; Based on the rule confidence, the rules in each rule layer are sorted in descending order to obtain a sorted rule set, and the rules are sequentially applied to the corresponding structured templates in the sorted order to generate structured fragments through feature mapping, wherein the structured template defines a mapping relationship between features and structured data; Detecting the effective areas of the rules in the sorting rule set, when the effective areas of multiple rules overlap, extracting the range of the overlapping area, calculating the coverage of each rule in the overlapping area, taking the weighted sum of the rule confidence and coverage as the rule priority, selecting the rule with the highest priority and applying it to the overlapping area, and ensuring the generation of a unique structured fragment; Construct a multi-layer verification network including attribute verification layer, relationship verification layer and value range verification layer. In the attribute verification layer, the integrity of the structured fragment is verified based on the attribute graph. In the relationship verification layer, the rationality of the relationship between the structured fragments is verified based on the knowledge graph. In the value range verification layer, the validity of the attribute values ​​in the structured fragment is verified based on the statistical model. The verification results of each layer are integrated to obtain the verified structured fragment. Extract semantic features from the verified structured fragments, calculate the semantic similarity between the fragments, build a hierarchical organizational structure of the fragments based on the semantic similarity, integrate semantically related structured fragments according to the hierarchical organizational structure, and generate initial structured data that meets preset specifications.

7. The method according to claim 1, characterized in that Analyzing the initial structured data by a causal reasoning method to obtain a decision basis chain, generating optimization parameters based on the decision basis chain, and optimizing the initial structured data according to the optimization parameters and the category information of the classified annotation data to obtain the final structured data includes: Calculating partial correlation coefficients for variable pairs in the initial structured data, performing conditional independence tests based on the partial correlation coefficients to obtain conditional independence relationships between variables, and constructing a causal graph based on the conditional independence relationships; A linear structural equation model is constructed based on the causal diagram, and the parameters of the linear structural equation model are optimized by maximum likelihood estimation to obtain a structural causal model, wherein the structural causal model is used to quantify the strength of causal effects between variables in the causal diagram; Calculate the number of associated edges of each node in the causal graph, determine the node whose number of associated edges exceeds a preset threshold of the number of associated edges as a decision node, trace the associated nodes from the decision node along the causal edges in the causal graph step by step to obtain a factor chain, calculate the cumulative causal effect of the factor chain based on the structural causal model, and determine the factor chain whose cumulative causal effect exceeds a preset effect threshold as a candidate decision basis chain; In the structural causal model, the value of each node in the candidate decision basis chain is changed and substituted into the structural causal model to obtain the downstream node prediction value corresponding to the node value change, and the stability of the candidate decision basis chain is determined based on the change range of the prediction value, and the candidate decision basis chain whose stability meets the preset conditions is determined as the decision basis chain; Based on the structural causal model, the influence degree of each node in the decision basis chain on the target variable is calculated, and the nodes whose influence degree exceeds a preset influence threshold are selected as a feature set according to the size of the influence degree, and the influence degree of each node in the feature set is used as a feature importance score, and the feature importance score is combined with the category discrimination degree of the classified annotation data to obtain a feature optimization parameter; Based on the feature optimization parameters, weighted adjustment is performed on the values ​​of corresponding features in the initial structured data to obtain optimized data, wherein the feature optimization parameters are used to determine the weight coefficient of the feature adjustment; The difference in feature distribution between the optimized data and the initial structured data is calculated to obtain an optimization effect index, the optimized data is input into the structural causal model to verify the causal relationship constraints to obtain a verification result, the optimized data is iteratively optimized according to the optimization effect index and the verification result, and when the optimization index reaches a preset target and the verification result satisfies the causal consistency, the current optimized data is determined as the final structured data.

8. A deep learning-based automatic processing system for unstructured data, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to acquire unstructured data using an intelligent data collector, build a data feature fingerprint library, classify and annotate the unstructured data based on the data feature fingerprint library to obtain classified annotated data, use an incremental learning algorithm to build a data quality assessment model, the data quality assessment model scores the classified annotated data from three dimensions of data integrity, timeliness and value density to obtain a scoring result, build a priority processing queue according to the scoring result, and perform data cleaning and normalization on the classified annotated data in the priority processing queue to obtain a standardized training data set; The second unit is used to perform deep learning processing on the standardized training data set using a multi-layer cascade network structure, wherein the multi-layer cascade network structure identifies target features in the standardized training data set based on the category information of the classified annotation data and through a self-attention mechanism, constructs a feature association graph based on the target features, and constructs a bidirectional reasoning channel on the feature association graph, wherein the bidirectional reasoning channel includes a forward channel and a reverse channel, wherein the forward channel extracts numerical distribution features, time series features, and attribute features in the standardized training data set through a convolutional neural network to obtain explicit feature data, and the reverse channel extracts data item association rules, data change trends, and data interaction patterns in the standardized training data set through a recurrent neural network to obtain implicit feature data, and a dynamic weight allocation strategy is used to perform feature combination on the explicit feature data and the implicit feature data to obtain a multidimensional feature tensor; The third unit is used to perform rule matching on the multidimensional feature tensor based on a preset domain knowledge base to obtain an inference rule set, use the inference rule set to perform hierarchical processing on the multidimensional feature tensor to obtain initial structured data, analyze the initial structured data through a causal reasoning method to obtain a decision basis chain, generate optimization parameters based on the decision basis chain, and optimize the initial structured data according to the optimization parameters and category information of the classified annotation data to obtain final structured data.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images