Method for adapting file processing of structured and unstructured data
By classifying, preprocessing, extracting features, optimizing mutual information, and dynamically fusing multimodal data, the heterogeneity and uncertainty issues in multimodal data processing are resolved, achieving efficient and robust data fusion and improving the accuracy and reliability of data processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING HANXINSHENG TECH CO LTD
- Filing Date
- 2025-04-22
- Publication Date
- 2026-08-04
AI Technical Summary
Traditional data processing tools struggle to operate multimodal data uniformly, resulting in inconsistencies in information content and difficulty in quantifying the correlation between modalities. This leads to insufficient accuracy in weight allocation and feature representation during data fusion, and a lack of universality, making it difficult to meet the needs of complex application scenarios.
By identifying and classifying structured, semi-structured and unstructured data, preprocessing and feature extraction are performed. Multimodal mutual information is used to optimize the quantification correlation, dynamically adjust the fusion weights, and model uncertainty through variational inference to output unified data results.
It achieves efficient fusion of multimodal data, significantly improves the accuracy and robustness of data processing, provides an efficient solution for multimodal data fusion in complex scenarios, and improves the accuracy of intermodal correlation measurement and the reliability of fusion results.
Smart Images
Figure CN120030193B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of document processing technology, specifically to a document processing method adapted to both structured and unstructured data. Background Technology
[0002] With the rapid development of information technology, the application of structured and unstructured data is becoming increasingly widespread across various fields. Structured data typically exists in the form of tables, databases, etc., and has a clear row and column relationship and data structure. Unstructured data, on the other hand, includes forms such as images, text, audio, and video, and lacks a unified structured format. Meanwhile, semi-structured data lies between the two, storing information in a hierarchical structure. Traditional methods often face the following technical bottlenecks in addressing the processing needs of different types of data.
[0003] Structured, semi-structured, and unstructured data differ significantly in storage format, feature representation, and distribution, making it difficult for traditional data processing tools to operate uniformly when handling multimodal data. Inconsistencies in information content and difficulties in quantifying the correlation between modalities among multimodal data lead to insufficient accuracy in weight allocation and feature representation during data fusion. Unstructured data, such as images and text, may contain noise or missing values, resulting in information loss or error accumulation during multimodal fusion, thereby reducing the credibility and robustness of the results. Traditional data processing methods are mostly designed for single data types or specific domains, lacking versatility, and have limited capabilities in dynamic weighted fusion and uncertainty modeling, making it difficult to meet the needs of complex application scenarios. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention provides a file processing method adapted to both structured and unstructured data, solving the problems of heterogeneity, uncertainty, and insufficient robustness in quantification and fusion of intermodal correlations in multimodal data processing.
[0005] To achieve the above objectives, the present invention provides the following technical solution: a file processing method adapted to structured and unstructured data, comprising the following steps; S1. Identify and classify the type of input files, including structured data, semi-structured data, and unstructured data; S2. Perform preprocessing and feature extraction for different types of data; S3. Optimize the correlation between different data modes based on multimodal mutual information; S4. Verify the consistency of interactions between data modalities and dynamically adjust the fusion weights; S5. Use variational inference to model and optimize the uncertainty of data; S6. Output the processed unified data results and perform performance evaluation.
[0006] Preferably, step S1 specifically includes the following steps; S1.1. Initially identify the file type by using the file extension and header information; S1.2 Further analyze the file content. For table-structured data with clear row and column relationships, classify it as structured data. S1.3 For data files without a clear structure, their type is identified by the file content pattern. If it is an image file, the grayscale distribution of the pixel matrix is extracted. S1.4 Record the identification and classification results in a preset data type classification table, where structured data includes tabular data, semi-structured data includes hierarchical files, and unstructured data includes image data, audio data, video data, and natural language text data.
[0007] Preferably, step S2, the preprocessing and feature extraction steps for structured data, includes: ; in, For time points interpolation, and (a) Time points adjacent to known values; (b) Identification and removal of outliers by calculating Z-scores. ,like The data is considered abnormal. The mean, Standard deviation The threshold value is used.
[0008] Preferably, step S2, the preprocessing and feature extraction steps for unstructured data, includes: For image data, normalization and noise reduction are performed, and feature vectors are extracted using a convolutional neural network (CNN). For text data, word segmentation, word frequency statistics, and embedding models are used to convert the text into vector representations, where the formula for word frequency statistics is: ; Among them, TF For words In the document word frequency in Total number of documents For containing words (c) For audio data, the short-time Fourier transform is used to extract spectral features, and the formula is: ; in, For time and frequency The time-frequency characteristics, This is a window function.
[0009] Preferably, the method based on multimodal mutual information optimization in step S3 includes: estimating mutual information by optimizing the following variational lower bound. ; in, For conditional distribution, For edge distributions, a neural network is used to parameterize them.
[0010] Preferably, the method for consistency verification in step S4 includes: calculating the interaction consistency score between modalities; ; in, Representing modes and mutual information, and Modal and Information entropy.
[0011] Preferably, the dynamic weighting method based on consistency score in step S4 includes: Calculate the weights of each mode. , ; The result after fusion is: ; in, For the first Data for each modality, These are the corresponding dynamic weighting coefficients.
[0012] Preferably, the uncertainty modeling based on variational inference in step S5 includes: The following evidence lower bound (ELBO) is used to examine latent variables. posterior distribution Approximation; ; in, For variational distribution, KL represents the KL divergence.
[0013] Preferably, the method based on uncertainty quantification in step S5 includes: ; Uncertainty values are used to weight data points with high noise or low confidence.
[0014] Preferably, the performance verification in step S6 includes; Mutual information gain evaluation verifies the improvement in mutual information before and after data fusion; Perception performance verification involves evaluating the accuracy of the output results based on IoU and mAP metrics.
[0015] This invention provides a file processing method adapted to both structured and unstructured data. It has the following beneficial effects: 1. This invention classifies and extracts features from structured, semi-structured, and unstructured data, and utilizes mutual information optimization and dynamic weighting strategies to achieve efficient fusion of multimodal data. Compared with traditional methods, this invention can significantly improve the accuracy and robustness of data processing, providing an efficient solution for multimodal data fusion in complex scenarios.
[0016] 2. By introducing a mutual information optimization method based on variational lower bounds, this invention accurately quantifies the correlation between modes. Compared with existing correlation analysis techniques, mutual information optimization can better capture the complex correlation between modes, providing a solid theoretical foundation for dynamic weighted fusion.
[0017] 3. This invention uses a dynamic weighting strategy to dynamically adjust the modal weights based on the interaction consistency scores between modalities, so that highly correlated modalities contribute more to the fusion result, while the influence of low-correlation modalities is weakened. This method effectively solves the robustness problem caused by fixed weights in traditional methods. Attached Figure Description
[0018] Figure 1 This is a perspective view of the present invention. Detailed Implementation
[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] Please see the appendix Figure 1 This invention provides a file processing method adapted to both structured and unstructured data, comprising the following steps; S1. Identify and classify the type of input files, including structured data, semi-structured data, and unstructured data; S2. Perform preprocessing and feature extraction for different types of data; S3. Optimize the correlation between different data modes based on multimodal mutual information; S4. Verify the consistency of interactions between data modalities and dynamically adjust the fusion weights; S5. Use variational inference to model and optimize the uncertainty of data; S6. Output the processed unified data results and perform performance evaluation; Specific; In this embodiment, the input file is first identified in terms of its type and classified, as detailed in the steps below. As an alternative, this embodiment makes a preliminary judgment based on the file extension and file header information. For some files with non-standard file extensions, further judgment can be made based on the file header information. Specifically, after determining the file extension, the file content needs to be further parsed. If the file has a clear row and column relationship, its row and column data structure can be extracted by a table parsing tool and classified as structured data. For semi-structured data, this embodiment classifies data by parsing the hierarchical structure of files. Files typically store information in the form of nested key-value pairs, and the hierarchical content is represented by nested tags. As one implementation method, these hierarchical structures can be traversed using a depth-first search (DFS) algorithm to record their nesting relationships and hierarchical depth. It is important to emphasize that, for the complexity of hierarchical data, this embodiment further employs information entropy for quantification. The specific formula is as follows; ; in, Entropy represents hierarchical information. For nodes The probability distribution, Represents a node The frequency of occurrence in a hierarchical structure. Information entropy can reflect the complexity of data hierarchy, thus providing a reference for classification; For data files without a clear structure, such as images and audio, this embodiment classifies them through content pattern analysis. In one exemplary implementation, image files can be analyzed using grayscale histograms to determine their pixel distribution characteristics, and edge detection algorithms can be used to further verify that they are image data. Text files can be analyzed using word segmentation and keyword extraction methods to determine their text attributes.
[0021] It should be noted that, to ensure the accuracy of classification, this embodiment records all identification results in a unified data type classification table, which exemplarily includes the following: Structured data: such as tabular files, data exported from relational databases; Semi-structured data: such as JSON files and XML files; Unstructured data: such as image data, audio files, video files, and free text.
[0022] In this embodiment, for input files of various types, the system identifies and classifies their types using multi-level rules and algorithms. Specifically, this includes the following: In this embodiment, step S2 mainly involves targeted preprocessing and feature extraction for structured, semi-structured, and unstructured data to achieve a unified representation of multimodal data. It should be noted that this step aims to ensure that different modalities, after specific processing, can be input into the subsequent fusion stage in a standardized high-dimensional feature form, providing a foundation for the final dynamic weighting and uncertainty modeling.
[0023] Specifically, structured data is normalized through outlier detection, interpolation completion, and statistical feature extraction; semi-structured data is optimized through hierarchical structure parsing and information entropy calculation; and unstructured data is extracted using deep learning and signal processing methods to extract efficient feature representations. In this embodiment, the processing of structured data mainly involves data cleaning, outlier handling, and statistical feature extraction. This embodiment first addresses the missing values in the structured data. Specifically, imputation can be performed using mean imputation, interpolation, or regression prediction. For example, for time series data, interpolation calculates the value of missing points using the following formula; ; in, For time points interpolated values, and These are the time points adjacent to known values; To detect and remove outliers in the data, this embodiment uses the Z-score method, and the specific calculation formula is as follows; ; in, For data points Standardized scores, and These are the mean and standard deviation of the data, respectively. When (generally If we take 3), then we can determine that the data point is an outlier. It should be noted that, for the cleaned data, this embodiment further extracts its statistical characteristics. For example, these characteristics include mean, variance, skewness, kurtosis, and correlation matrix, used to characterize the statistical properties of the data and the relationships between variables; Preprocessing and feature extraction of semi-structured data In this embodiment, for semi-structured data, feature representation is mainly achieved through hierarchical structure parsing and information entropy calculation.
[0024] Specifically, this embodiment uses a depth-first search algorithm to recursively parse the nested key-value pairs in the JSON file and records the structural information of each level of nodes. In one possible implementation, the nested tags in an XML file can also be parsed into a tree structure using a similar method. To quantify the complexity of the hierarchical structure, this embodiment calculates the hierarchical information entropy. The formula is as follows; ; in, For hierarchical information entropy, For the first The type or label of the layer node, This represents the probability of a node appearing. It should be noted that a larger hierarchical information entropy value indicates a more complex data structure. This embodiment can also represent the parsed hierarchical structure as a high-dimensional tensor for subsequent fusion processing; Preprocessing and feature extraction of unstructured data In this embodiment, image processing technology, natural language processing method and signal processing technology are used to extract features from unstructured data.
[0025] For image data, this embodiment employs preprocessing operations such as image normalization and noise reduction. Alternatively, a high-dimensional feature vector of the image can be extracted using a convolutional neural network.
[0026] For text data, this embodiment employs word segmentation, stop word filtering, and embedding model transformation methods. For example, TF-IDF (Term Frequency-Inverse Document Frequency) can be used to statistically analyze the importance of each word, and its calculation formula is as follows; ; Among them, TF For words In the document Frequency of occurrence in Total number of documents For containing words The number of documents; For audio data, this embodiment uses Short-Time Fourier Transform (STFT) to extract spectral features. The specific formula is as follows: ; in, For time and frequency The time-frequency characteristics, For window functions; The preprocessed unstructured data described above are all converted into standardized high-dimensional feature representations for use in the subsequent data fusion stage; The preprocessing of different types of data may need to be adjusted according to data distribution or business needs. For example: Structured data cleaning strategies can be combined with specific business rules, such as constraints on the fluctuation range of financial data; The hierarchical information entropy calculation of semi-structured data can introduce label importance weights to highlight key fields; Feature extraction of unstructured data can be performed using different embedding models or signal processing methods depending on the application scenario; It should be noted that the preprocessing and feature extraction processes described above provide standardized inputs for subsequent data fusion, consistency verification, and dynamic weighting, ensuring the robustness and consistency of multimodal data processing. Step S3 aims to quantify the correlation between different data modalities using a multimodal mutual information optimization method. It should be noted that multimodal data are typically difficult to fuse directly due to differences in modal characteristics, distribution patterns, and feature dimensions. This embodiment calculates the mutual information between modalities to accurately measure the correlation between different modal data, providing a theoretical basis for subsequent consistency verification and dynamic weighted fusion. In this embodiment, mutual information is used to describe modal data. and The degree of correlation; This embodiment uses joint distribution. and marginal distribution , The mutual information is calculated using the logarithmic ratio, and its mathematical expression is: ; It should be noted that in this formula It is modal and The joint probability distribution of ; and They are modal and In one possible implementation of the marginal probability distribution, it is difficult to directly calculate the above integral for high-dimensional feature data. Therefore, this embodiment uses a variational inference method to optimize the solution of mutual information. In this invention, an optimizable distribution is constructed. ,right An approximate model is performed, and mutual information is optimized using a lower bound. Specifically, the mathematical expression for the variational lower bound is: ; in: It is a conditional distribution, used for approximation. ; It is a marginal distribution, by and The derivation yields These are the parameters of the distributed parameterized model; It should be noted that the above lower bound optimization problem is solved using the gradient descent method, and the optimization objective is: ; In one possible implementation, this embodiment utilizes a deep neural network to... and Parametric modeling is performed, and the input modal features are processed using a multilayer perceptron (MLP). and Project onto the joint space and learn the characteristics of its joint distribution; Normalized Consistency Score In this embodiment, in order to further measure the correlation between modal data In this embodiment, to further measure the correlation between modal data, the present invention calculates a consistency score between modalities. The formula is as follows; ; It should be noted that in this formula; Representing modes Information entropy; Representing modes Information entropy; This is used to normalize the mutual information value, ensuring the result is between 0 and 1. Through the calculation of the consistency score, this embodiment can quantify the strength of the correlation between different modalities, providing a weighting basis for subsequent dynamic weighted fusion. Multimodal correlation matrix In one possible implementation, this embodiment stores the mutual information values and consistency scores between all modalities as a multimodal correlation matrix, exemplarily represented as follows; ; in Representing modes and The correlation between them.
[0027] It should be noted that the generation of this matrix can provide a direct reference for weight allocation and consistency adjustment in the multimodal fusion stage.
[0028] In this embodiment, step S4 verifies the interaction consistency of different data modalities, quantifies the collaborative relationship between modalities, and thus provides a basis for dynamic weighting for multimodal data fusion. It should be noted that the interaction consistency between data modalities is a crucial guarantee for the accuracy and robustness of multimodal fusion. This embodiment achieves adaptive adjustment of the fusion strategy through the calculation of consistency scores and dynamic weight allocation. In this embodiment, in order to quantify the interaction consistency between modal data, a normalized consistency score calculation method based on mutual information is proposed. Alternatively, this embodiment utilizes the modal mutual information value calculated in step S3. Combined with modal information entropy and Calculate the consistency score Its formula is: ; It should be noted that: Representing modes and Mutual information between them; and Modal and Information entropy; Normalization factor Used to assign consistency scores Limited to the range [0,1]; Specifically, the closer the consistency score is to 1, the higher the degree of interaction between modes and the stronger the data consistency; when the score is close to 0, it indicates that there is a large deviation or noise influence between modes. Calculation of dynamic weights In this embodiment, in order to achieve dynamic adjustment of the fusion weight, it is based on the consensus score. Assign weights to each mode The weight calculation formula is: ; in the formula; Indicates the first Data fusion weights for each modality; Representing modes and Consistency score; denominator Used to normalize the weights so that the sum of the weights is 1; This embodiment dynamically adjusts the weights to make the high-consistency mode contribute more to the fusion result, while the influence of the low-consistency mode is weakened. Dynamic weighted calculation of fusion results In this embodiment, the fusion result Through modal data and weight The weighted summation yields the mathematical expression: ; Specifically; This represents the fusion result of multimodal data; Indicates the first Data features of each modality; Represents the dynamic weights of the corresponding modes; The dynamic weighting formula can adaptively adjust the contribution of each mode to the fusion result based on the interaction consistency score between modes, thereby improving the robustness and accuracy of the fusion result; Storage of the consistency matrix In one possible implementation, this embodiment stores the consistency scores between all modes as a consistency matrix. Its expression is; ; in Representing modes and The consistency score is used to determine the consistency matrix. It should be noted that this consistency matrix provides a global reference for subsequent modality selection and dynamic weighting. The modality with the highest relevance is selected for fusion based on the consistency score in the matrix, while low-relevance modalities are ignored. In this embodiment, step S5 uses variational inference methods to model and optimize the uncertainty of multimodal data, thereby improving the reliability of the fusion result. It should be noted that during multimodal data processing, noise, missing values, and intermodal differences can lead to data uncertainty. Without modeling and optimization, this may negatively impact the reliability of the final result. This embodiment addresses this by constructing latent variables. The probability distribution model is used to quantify and optimize the uncertainty of the fusion results, so as to further enhance the robustness of data processing. Modeling latent variables In this embodiment, latent variables are introduced to describe the uncertainty of the data. And assume It is multimodal data The potential representation of , whose joint distribution can be expressed as; ; in the formula; Representing latent variables The prior distribution of the is usually assumed to be the standard normal distribution; Representing modes Given latent variables Likelihood distribution under the given conditions surface This represents the collection of all modal data; This embodiment utilizes a deep neural network to... Perform parametric modeling to ensure it can adapt to complex modal distributions; Optimization of variational inference In this embodiment, in order to determine the posterior distribution To approximate this, we introduce a variational distribution. Parameter learning is achieved through Optimized Evidence Lower Bound (ELBO). The specific optimization objective is: ; It should be noted that; First item Let log-likelihood be the term representing the data. Given latent variables The generation probability under certain conditions; Second item KL divergence is used to measure variational distributions. With prior distribution The differences between them; Alternatively, this embodiment utilizes reparameterization techniques to... Optimize. For example, by... Represented as; ; middle, and Latent variables The mean and standard deviation parameters, in one possible implementation, optimize the objective. Solving using the gradient descent algorithm, thereby learning... The optimal parameters; Quantification of uncertainty In this embodiment, variational distribution is used. The variance quantifies the uncertainty of the fusion result, and the specific formula is as follows; ; It should be noted that: The latent variable representation of the fusion result; Var The variance of the variational distribution is used to measure the uncertainty of data fusion. When the uncertainty is large, this embodiment can reduce the weight of the corresponding data, thereby reducing its negative impact on the fusion result. This embodiment further optimizes the uncertainty quantification results, and the specific steps include: The weights of data points with high uncertainty are adjusted to reduce their contribution to the fusion process; Increase the weight of low-uncertainty data points to enhance the influence of high-confidence data; Specifically, the optimized weight calculation formula is as follows: ; Among them, Uncertainty Representing modes Regarding the quantification results of uncertainty, it should be noted that this optimization strategy can significantly improve the robustness of the fusion results, especially when the data quality is uneven. In this embodiment, step S6 outputs the processed multimodal fusion data and verifies the effectiveness of the fusion result through a series of performance evaluation indicators. It should be noted that the unified data result not only includes the feature representation after fusion, but also has a description of the quantification of uncertainty, so as to provide reliable decision support in subsequent application scenarios. This embodiment ensures the integrity and practical applicability of the data processing scheme by verifying the multi-dimensional performance of the fusion result. Output of data results In this embodiment, the processed unified data result The final feature representation, which includes multimodal data fusion, is calculated in this embodiment using a dynamic weighted fusion method. Its formula is: ; in: Indicates the first Data features of each modality; Representing modes Dynamic weights; It should be noted that, This represents the fused high-dimensional feature vector, which can be used for subsequent classification, regression, or clustering tasks. As one possible implementation, this embodiment further... The uncertainty quantification results for each mode are output together to enhance the interpretability of the data results. The uncertainty quantification results can be calculated using the following formula; ; in, Var This represents the variance of the latent variable distribution in the fusion result; Mutual information gain evaluation In this embodiment, to verify the change in information content before and after data fusion, mutual information gain is calculated. Specifically, mutual information gain is defined as: ; in: Indicates the fusion result and the target variable Mutual information between them; Representing modes With target variable Mutual information between them; It should be noted that a positive value of mutual information gain indicates that data fusion has increased the amount of information, while a negative value may indicate that the fusion weights of certain modalities need further optimization. In one possible implementation, this embodiment minimizes the following objective function on the weights. Make dynamic adjustments; ; The above optimization process can further improve the information effectiveness of the fusion results; Perception performance verification In this embodiment, in order to further verify the practicality of the data processing scheme, the accuracy of the fusion result is measured using the perception performance evaluation index. This embodiment is based on the following common indexes for verification. IoU (Intersection over Union): Applicable to object detection or segmentation tasks, the formula is: ; Where the numerator is the area of overlap between the predicted region and the actual region, and the denominator is the area of the union of the two regions; mAP (Mean Average Precision): Applicable to multi-class classification tasks, it represents the average precision score across all classes, and the formula is: ; in, For category average accuracy, Total number of categories; This embodiment comprehensively measures the performance of the fusion results in a specific task by calculating the above indicators, thereby verifying the actual effect of the data processing solution. Summarize By classifying, preprocessing, extracting features, optimizing mutual information, modeling uncertainty, and dynamically fusing multimodal data, this invention addresses the issues of heterogeneity, uncertainty, and insufficient consistency in multimodal data processing. Through the introduction of mutual information optimization and variational inference modeling, this invention effectively improves the accuracy of correlation measurement between modalities and optimizes the reliability of data fusion results through a dynamic weighting strategy.
[0029] The technical advantages of this invention are that it enables efficient fusion and optimization of multiple types of data, resulting in high levels of information content, robustness, and practical application effectiveness in the data processing results. Furthermore, through uncertainty quantification and performance evaluation, this invention further ensures the reliability of the processing results, providing reliable technical support for a wide range of multimodal data application scenarios.
[0030] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A file processing method adapted to both structured and unstructured data, characterized in that, Includes the following steps: S1. Identify and classify the type of input files, including structured data, semi-structured data, and unstructured data; S2. Perform preprocessing and feature extraction for different types of data; S3. Optimize the correlation between different data modes based on multimodal mutual information; The method based on multimodal mutual information optimization in step S3 includes: estimating mutual information by optimizing the following variational lower bound; ； in; It is a conditional distribution, used for approximation. ; It is a marginal distribution, by and It is derived that, It is modal The marginal probability distribution; These are the parameters of the distributed parameterized model; It is modal and The joint probability distribution of ; S4. Verify the consistency of interactions between data modalities and dynamically adjust the fusion weights; The method for consistency verification in step S4 includes: calculating the interaction consistency score between modalities; ； in; Representing modes and Mutual information between them; Representing modes Information entropy It is modal The marginal probability distribution; Representing modes Information entropy; Used to normalize mutual information values, ensuring the result is between 0 and 1; The dynamic weighting method based on consistency scores in step S4 includes: Calculate the weights of each mode. : ； in the formula; Indicates the first Data fusion weights for each modality; Representing modes and Consistency score; denominator Used to normalize the weights so that the sum of the weights is 1; The result after fusion is: ； in; This represents the fusion result of multimodal data; Indicates the first Data features of each modality; S5. Use variational inference to model and optimize the uncertainty of data; The uncertainty modeling based on variational inference in step S5 includes: The following evidence outlines the lower bound for latent variables. posterior distribution Approximation; ； in; It is a likelihood distribution; , Divergence, a measure of variational distribution With prior distribution Differences; The uncertainty quantification-based method in step S5 includes: ； in; Variational distribution The variance; S6. Output the processed unified data results and perform performance evaluation.
2. The file processing method for adapting structured and unstructured data according to claim 1, characterized in that, Step S1 specifically includes the following steps; S1.
1. Initially identify the file type by using the file extension and header information; S1.2 Further analyze the file content. For table-structured data with clear row and column relationships, classify it as structured data. S1.3 For data files without a clear structure, their type is identified by the file content pattern. If it is an image file, the grayscale distribution of the pixel matrix is extracted. S1.4 Record the identification and classification results in a preset data type classification table, where structured data includes tabular data, semi-structured data includes hierarchical files, and unstructured data includes image data, audio data, video data, and natural language text data.
3. The file processing method for adapting structured and unstructured data according to claim 1, characterized in that, The preprocessing and feature extraction steps of the structured data in step S2 include: ； in; For time points interpolation, and These are the time points adjacent to known values; Outlier identification and removal are achieved by calculating Z-scores. ,like Then the data is considered abnormal; in; For data points; The mean of the data; The standard deviation of the data; The threshold value is used.
4. The file processing method for adapting structured and unstructured data according to claim 1, characterized in that, The preprocessing and feature extraction steps for unstructured data in step S2 include: For image data, normalization and noise reduction are performed, and feature vectors are extracted using convolutional neural networks. For text data, word segmentation, word frequency statistics, and embedding models are used to convert the text into vector representations.
5. The file processing method for adapting structured and unstructured data according to claim 1, characterized in that, The performance evaluation in step S6 includes: Mutual information gain evaluation verifies the improvement in mutual information before and after data fusion; Perception performance verification involves evaluating the accuracy of the output results based on IoU and mAP metrics.