A deep learning-based multi-modal data quality evaluation method

Abstract

The application discloses a kind of based on deep learning multi-modal data quality evaluation method, comprising obtaining database in first modality record data and second modality record data labeled as alignment;First modality record data and second modality record data are respectively carried out semantic feature extraction to obtain text structured data semantic encoding feature tensor and metadata description semantic encoding feature vector;Text structured data semantic encoding feature tensor and metadata description semantic encoding feature vector cross-modal joint coding to obtain structured-metadata cross-modal quality evaluation joint coding feature tensor;Based on structured-metadata cross-modal quality evaluation joint coding feature tensor, reconstruct record data is generated, and the offset feature of reconstruct record data is calculated;Offset feature is evaluated by quality by first deep learning model;The application improves the accuracy of data quality evaluation by semantic encoding feature extraction, fine-grained hash cross-modal coding and bidirectional reconstruction offset evaluation.

Description

Technical Field

[0001] This invention relates to the field of multimodal data quality assessment technology in artificial intelligence, and more specifically, to a multimodal data quality assessment method based on deep learning. Background Technology

[0002] In the modern information age, databases, as core tools for data storage and management, carry massive amounts of multimodal data. This multimodal data includes various forms such as text, images, audio, and video, and is widely used in fields such as healthcare, finance, transportation, and education. For example, a medical database may contain patient medical records, X-ray images, electrocardiogram signals, and other data in multiple modalities; a transportation database may contain vehicle driving records, surveillance videos, and road condition sensor data.

[0003] However, with the continuous accumulation of multimodal data, data quality issues in databases are becoming increasingly prominent. These problems not only affect the usability and value of the data, but may also lead to erroneous decisions and analytical results.

[0004] Traditional database data quality assessment methods mainly rely on statistical analysis, rule matching, and expert experience. These methods are effective to some extent when dealing with single-modal data, but they have significant limitations when faced with multimodal data.

[0005] Single-modal data quality assessment methods:

[0006] While assessing data quality by calculating statistical metrics (such as mean, variance, and percentage of missing values) is simple, it cannot handle the complex relationships between multimodal data.

[0007] Checking the accuracy of data based on predefined rules (such as data format, range, etc.) relies on expert experience and the completeness of the rules, making it difficult to adapt to the diversity of multimodal data.

[0008] Relying on the experience and knowledge of domain experts to assess data quality is a highly subjective approach that is difficult to standardize and automate.

[0009] The complexity of multimodal data quality assessment:

[0010] Data from different modalities have different characteristics and quality indicators. Traditional single-modality evaluation methods are difficult to apply directly to multimodal data and lack fine-grained hash cross-modal coding.

[0011] The quality of multimodal data may change over time and with the environment. Traditional static evaluation methods are difficult to adapt to dynamically changing data and lack bidirectional reconstruction offset evaluation.

[0012] In summary, both single-modal and multi-modal data quality assessment methods have drawbacks, including a lack of accuracy. Summary of the Invention

[0013] The purpose of this invention is to provide a multimodal data quality assessment method based on deep learning to solve the above-mentioned problems in the prior art.

[0014] The application is as follows:

[0015] A deep learning-based method for multimodal data quality assessment includes the following steps:

[0016] The acquisition engine is used to obtain first modality record data and second modality record data marked as aligned in the database, wherein the first modality record data is a structured relational table and the second modality record data is an associated metadata description;

[0017] Semantic features are extracted from the first modality record data and the second modality record data respectively to obtain the text structured data semantic encoding feature tensor and the metadata description semantic encoding feature vector;

[0018] The semantic coding feature tensor of the text structured data and the semantic coding feature vector of the metadata description are jointly encoded across modalities to obtain the structured-metadata cross-modal quality assessment joint encoded feature tensor;

[0019] Data reconstruction is performed based on the structured-metadata cross-modal quality assessment joint encoded feature tensor to obtain reconstructed record data;

[0020] Calculate the data consistency offset between the reconstructed record data and the first modality record data to generate a first offset feature; calculate the data consistency offset between the reconstructed record data and the second modality record data to generate a second offset feature; evaluate the quality of the first offset feature and the second offset feature using a preset first deep learning model, and output the multimodal data quality evaluation result;

[0021] The cross-modal joint coding includes:

[0022] The text structure data semantic encoding feature tensor and the metadata description semantic encoding feature vector are queried and encoded using a fine-grained database attribute hash algorithm. The encoded result is a set of structured-metadata text description local hash query encoded vectors.

[0023] Using the set as descriptive information, feature extraction is performed on the semantic coding feature tensor of the text structured data to generate the joint coding feature tensor of cross-modal quality assessment of structured-metadata.

[0024] Furthermore, using the acquisition engine to obtain the first modality record data and the second modality record data marked as aligned in the database includes:

[0025] The system first uses the acquisition engine to receive input containing first modality record data and second modality record data. The system then receives instructions provided by the user in natural language, which represent the needs for database name processing and data form processing under the database name.

[0026] The first modality recording data is subjected to text semantic feature extraction based on the MobileNet convolutional neural network model to obtain the text structured data semantic encoding feature tensor:

[0027] P(x,y)= ,

[0028] Where P(x,y) represents the output of the semantic encoding feature tensor of the text structured data at the two-dimensional convolution position (x,y) of the first modality record data, K represents the convolution kernel, I represents the input first modality record data, i and j are the index positions in the convolution kernel, and n is the size of the convolution kernel;

[0029] The second modality recording data is semantically analyzed using a Long Short-Term Memory (LSTM) network to generate the first semantics. The first semantics is then encoded to obtain the metadata description semantic encoding feature vector.

[0030] ,

[0031] in, It is the hidden state at time step t. It is the input vector at time step t. and These are the weight matrices from the input state to the hidden state and from the current hidden state to the next hidden state, respectively. and It is the bias term, and Sigmoid is the activation function.

[0032] Further, the semantic encoding feature tensor of the text structured data and the semantic encoding feature vector of the metadata description are subjected to fine-grained database attribute hashing algorithm query encoding. After encoding, a set of structured-metadata text description local hash query encoded vectors is obtained, including:

[0033] The semantic encoding feature tensor of the text structured data is subjected to local feature decomposition along the channel dimension to obtain a set of local semantic feature matrices of the text.

[0034] Define the metadata description semantic encoding feature vector as a unified query vector, and treat each matrix in the set of local semantic feature matrices of the text as an independent database key-value matrix;

[0035] For each database key-value matrix, a fine-grained database attribute hash algorithm query encoding operation is performed. The unified query vector is matched and calculated with the current key-value matrix to generate the corresponding structured-metadata text description local hash query encoding vector.

[0036] The processing results of all database key-value matrices are aggregated to form a complete set of structured metadata text description local hash query encoded vectors.

[0037] Furthermore, for each database key-value matrix, a fine-grained database attribute hash algorithm query encoding operation is performed. The unified query vector is matched and calculated with the current key-value matrix to generate the corresponding structured-metadata text description local hash query encoding vector, including:

[0038] Calculate the cosine similarity between the query vector and each key vector in the database key matrix to obtain a set of local semantic similarities of the structured-metadata text description;

[0039] The set of local semantic similarities of the structured-metadata text description is input into the Sigmoid function for normalization to obtain a normalized set of local semantic similarities of the structured-metadata text description.

[0040] Furthermore, the processing results of all database key-value matrices are aggregated to form a complete set of structured-metadata text description local hash query encoded vectors, including:

[0041] Using the set of normalized structured-metadata text description local semantic similarities as the weight distribution, the position-weighted sum of each key vector in the database key-value matrix is ​​calculated to obtain the set of structured-metadata text description local hash query encoding vectors.

[0042] Further, using the set as descriptive information, feature extraction is performed on the semantic coding feature tensor of the text structured data to generate the joint coding feature tensor of structured-metadata cross-modal quality assessment, including:

[0043] Each local hash query encoding vector in the set of the metadata description semantic encoding feature vector and the structured-metadata text description local hash query encoding vector is input into a cross-modal mask weaving network based on description information;

[0044] Through the processing of the cross-modal mask weaving network, a set of text local semantic feature weight mask matrices based on descriptive information is generated;

[0045] Based on the set of weighted masks of local semantic features of the text, feature clustering is performed on the set of local semantic feature matrices of the text.

[0046] After feature clustering and partitioning, based on the set of text local semantic feature weight mask matrices, feature extraction and aggregation are performed on the set of text local semantic feature matrices to obtain the structured-metadata cross-modal quality assessment joint encoded feature tensor.

[0047] Further, each local hash query encoding vector in the set of metadata description semantic encoding feature vectors and structured-metadata text description local hash query encoding vectors is input into a cross-modal mask weaving network based on description information; through the processing of the cross-modal mask weaving network, a set of text local semantic feature weight mask matrices based on description information is generated, including:

[0048] Calculate the vector product between the semantic encoded feature vector of the metadata description and the transpose of the local hash query encoded vector of the structured-metadata text description to obtain the cross-modal query association matrix of the structured-metadata text description;

[0049] Divide the cross-modal query association matrix of the structured-metadata text description by the square root of the feature scale of the local hash query encoding vector of the structured-metadata text description, and then input it into the softmax function for normalization to obtain the text local semantic feature weight mask matrix based on the description information.

[0050] Further, after feature clustering and partitioning, based on the set of text local semantic feature weight mask matrices, feature extraction and aggregation are performed on the set of text local semantic feature matrices to obtain the structured-metadata cross-modal quality assessment joint encoded feature tensor, including:

[0051] The set of weighted masks of local semantic features based on descriptive information and the set of local semantic feature matrices of text are multiplied by position to obtain the set of cross-modal local fine-grained significant transformation matrices of structured-metadata text description.

[0052] The set of cross-modal local fine-grained significant transformation matrices of the structured-metadata text description is subjected to feature concatenation aggregation to obtain the joint encoded feature tensor of the structured-metadata cross-modal quality assessment.

[0053] Furthermore, the quality of the first and second offset features is evaluated using a pre-defined first deep learning model, and the multimodal data quality evaluation results are output, including:

[0054] Calculate the first and second offset features for the current evaluation period, and then fuse the first and second offset features to generate a fused feature;

[0055] Calculate the Euclidean similarity between the fusion features of the current evaluation period and the fusion features of interest in the previous evaluation period, and determine whether the fusion features of the current evaluation period are the fusion features of interest based on the Euclidean similarity.

[0056] If the fusion feature of the current evaluation period is the fusion feature of interest, the fusion feature of the current evaluation period is processed by the first model that has been pre-trained to obtain the deep feature of the current evaluation period. Then, multiple two-dimensional convolutional kernels are used to fuse the deep feature of the time-aligned cached fusion feature of interest with the deep feature of the current evaluation period to obtain the first quality feature of the current evaluation period. The first model adopts the ShuffleNet network.

[0057] If the fusion feature of the current evaluation period is a non-interesting fusion feature, the fusion feature of the current evaluation period is processed by the pre-trained second model to obtain the shallow feature of the current evaluation period. The shallow feature is transformed to obtain the deep feature. Then, multiple two-dimensional convolution kernels are used to fuse the deep feature of the time-aligned cached interest fusion feature with the deep feature of the current evaluation period to obtain the second quality feature of the current evaluation period. The second model adopts the EfficientNet network.

[0058] The first deep learning model is used to evaluate the quality of the first and second quality features, and the multimodal data quality evaluation results are output.

[0059] Furthermore, multiple two-dimensional convolutional kernels are used to fuse the deep features of the time-aligned cached interest fusion features with the deep features of the current evaluation period to obtain the first quality feature of the current evaluation period, including:

[0060] Obtain the deep features of the current evaluation period from the output of the first model. ;

[0061] Calculate the deep features at position b1 in the current evaluation period. Deep features of interest fused with cached features at position b2 The correlation matrix W(b1,b2):

[0062] W(b1,b2) = ,

[0063] in, The deep features at position b1 in the current evaluation period Transform into a two-dimensional matrix. Deep features of the fusion features of interest cached at position b2 in the current evaluation period. Transform into a two-dimensional matrix. For a two-dimensional matrix, the deep features of the cached fused features of interest are the previous deep features of interest;

[0064] Calculate the fusion features of interest for the cache aligned to position b1. :

[0065] = ,

[0066] Interested fusion features for caching with deep features The fusion was performed to obtain the evaluation features at position b1. :

[0067] =R(concat( , )),

[0068] in, (.) represents a 2*2 convolution operation, concat(.) represents a connection operation based on the channel dimension, and R(.) represents a convolution operation with three consecutive two-dimensional convolution kernels, each of which is 2×2 in size.

[0069] Evaluation features of all b1 positions The first quality characteristic that constitutes the current evaluation period .

[0070] Compared with the prior art, the present invention achieves the following beneficial effects:

[0071] This invention provides a method for acquiring first-modal and second-modal record data marked as aligned in a database using a data acquisition engine; extracting semantic features from the first-modal and second-modal record data to obtain a text structured data semantic coding feature tensor and a metadata description semantic coding feature vector; performing cross-modal joint coding on the text structured data semantic coding feature tensor and the metadata description semantic coding feature vector to obtain a structured-metadata cross-modal quality assessment joint coding feature tensor; reconstructing the data based on the structured-metadata cross-modal quality assessment joint coding feature tensor to obtain reconstructed record data; calculating the data consistency offset between the reconstructed record data and the first-modal record data to generate a first offset feature; calculating the data consistency offset between the reconstructed record data and the second-modal record data to generate a second offset feature; and performing quality assessment on the first and second offset features using a preset first deep learning model to output a multimodal data quality assessment result. This invention improves the accuracy of data quality assessment through semantic coding feature extraction, fine-grained hash cross-modal coding, and bidirectional reconstruction offset assessment. Attached Figure Description

[0072] Figure 1 This is a flowchart illustrating a deep learning-based multimodal data quality assessment method provided in an embodiment of the present invention. Detailed Implementation

[0073] The present invention will now be described in detail with reference to the accompanying drawings.

[0074] Example 1

[0075] This invention provides a method for multimodal data quality assessment based on deep learning, such as... Figure 1 This includes the following steps:

[0076] S1. Use the acquisition engine to obtain the first modality record data and the second modality record data marked as aligned in the database, wherein the first modality record data is a structured relational table and the second modality record data is an associated metadata description;

[0077] S2. Semantic feature extraction is performed on the first modality recording data and the second modality recording data respectively to obtain the text structured data semantic encoding feature tensor and the metadata description semantic encoding feature vector;

[0078] S3. Perform cross-modal joint encoding on the semantic coding feature tensor of the text structured data and the semantic coding feature vector of the metadata description to obtain the structured-metadata cross-modal quality assessment joint encoded feature tensor;

[0079] S4. Based on the structured-metadata cross-modal quality assessment joint encoded feature tensor, data reconstruction is performed to obtain reconstructed record data;

[0080] S5. Calculate the data consistency offset between the reconstructed record data and the first modality record data to generate a first offset feature; calculate the data consistency offset between the reconstructed record data and the second modality record data to generate a second offset feature; evaluate the quality of the first offset feature and the second offset feature using a preset first deep learning model, and output the multimodal data quality evaluation result.

[0081] The cross-modal joint coding includes:

[0082] The text structure data semantic encoding feature tensor and the metadata description semantic encoding feature vector are queried and encoded using a fine-grained database attribute hash algorithm. The encoded result is a set of structured-metadata text description local hash query encoded vectors.

[0083] Using the set as descriptive information, feature extraction is performed on the semantic coding feature tensor of the text structured data to generate the joint coding feature tensor of cross-modal quality assessment of structured-metadata.

[0084] It should be noted that the first modality record data (structured relational table) is the main data carrier, while the second modality record data (related metadata description) is essentially a descriptive label, just like attaching category labels (related metadata description) to book content (structured relational table). The core of quality assessment is to verify whether the content conforms to the label description.

[0085] The reason for using the set as descriptive information to extract features from the semantic encoding feature tensor of the text structured data without needing to extract tensors from the semantic encoding feature vector of the metadata description is as follows:

[0086] The associated metadata description has already undergone knowledge distillation in the early hash query. When the semantically encoded feature vector of the metadata description is used as a query vector to retrieve structured features, the key information has already been injected into the local hash query encoded vector set. This is similar to highlighting key points in a book using keywords (associated metadata description) (local hash query encoding). Subsequently, only these key points need to be analyzed.

[0087] Structured relational tables need to be deconstructed to reduce dimensionality. The high-dimensionality of tabular data (e.g., 100 columns x 1 million rows) must be reduced in computational complexity through feature extraction. However, the metadata description of the association is already a low-dimensional vector (e.g., 512 dimensions), and further extraction will result in information loss.

[0088] The reconstruction task requires information integrity. Experimental data shows that if feature extraction is also performed on the feature vector of the metadata description semantic encoding, the small perturbations of the feature vector of the metadata description semantic encoding will be amplified. For example, "data row field" will be incorrectly abstracted as "data column field".

[0089] It should be further explained that "using a set as descriptive information" limits the role of the associated metadata description to the guidance level, avoiding treating the associated metadata description itself as a processing object, which is also one of the technical highlights of this invention.

[0090] Specifically, the invention utilizes a data acquisition engine to obtain first-modal and second-modal record data marked as aligned in the database; semantic features are extracted from the first-modal and second-modal record data respectively to obtain a text structured data semantic coding feature tensor and a metadata description semantic coding feature vector; cross-modal joint coding is performed on the text structured data semantic coding feature tensor and the metadata description semantic coding feature vector to obtain a structured-metadata cross-modal quality assessment joint coding feature tensor; data reconstruction is performed based on the structured-metadata cross-modal quality assessment joint coding feature tensor to obtain reconstructed record data; the data consistency offset between the reconstructed record data and the first-modal record data is calculated to generate a first offset feature; the data consistency offset between the reconstructed record data and the second-modal record data is calculated to generate a second offset feature; the first and second offset features are quality-assessed using a preset first deep learning model, and the multimodal data quality assessment result is output; this invention improves the accuracy of data quality assessment through semantic coding feature extraction, fine-grained hash cross-modal coding, and bidirectional reconstruction offset assessment.

[0091] In the above embodiments, specifically, acquiring the first modality record data and the second modality record data marked as aligned in the database using the acquisition engine includes:

[0092] The system first uses the acquisition engine to receive input containing first modality record data and second modality record data. The system then receives instructions provided by the user in natural language, which represent the needs for database name processing and data form processing under the database name.

[0093] It should be noted that when a user inputs a natural language command, such as "Evaluate the data quality of the 'Process_Records' table in the database 'Oil_Refinery_DB'", the pre-trained BERT model is used to extract the semantic vector of the command, and the database name (Oil_Refinery_DB) and table name (Process_Records) are extracted through Named Entity Recognition (NER).

[0094] The data acquisition engine's process includes extracting aligned first-modal data (structured relational tables) and second-modal data (metadata descriptions) from the database:

[0095] Example: Set SELECT * FROM Oil_Refinery_DB.Process_Records to retrieve structured table data;

[0096] Set SELECT column_name, data_type, description;

[0097] FROM information_schema.columns;

[0098] WHERE table_name = 'Process_Records' to retrieve metadata description;

[0099] The first modality recording data is subjected to text semantic feature extraction based on the MobileNet convolutional neural network model to obtain the text structured data semantic encoding feature tensor:

[0100] P(x,y)= ,

[0101] Where P(x,y) represents the output of the semantic encoding feature tensor of the text structured data at the two-dimensional convolution position (x,y) of the first modality record data, K represents the convolution kernel, I represents the input first modality record data, i and j are the index positions in the convolution kernel, and n is the size of the convolution kernel;

[0102] It should be noted that, firstly, the structured relation table is converted into an image format based on the MobileNet convolutional neural network model. Each row of the table in the structured relation table is converted into a grayscale image (the values ​​are normalized to 0-255), with the row height being the image height and the column width being the image width.

[0103] The second modality recording data is semantically analyzed using a Long Short-Term Memory (LSTM) network to generate the first semantics. The first semantics is then encoded to obtain the metadata description semantic encoding feature vector.

[0104] ,

[0105] in, It is the hidden state at time step t. It is the input vector at time step t. and These are the weight matrices from input to the hidden state and from hidden state, respectively. and It is the bias term, and Sigmoid is the activation function.

[0106] It should be noted that the metadata description of the second modality record data is used to construct a text sequence, and the constructed text sequence is then converted into a word sequence:

[0107] Input: [("temperature", "float", "reactor temperature"), ("pressure", "int", "filtration pressure")];

[0108] Output sequence: ["column:temperature", "type:float", "desc:reactor temperature",...];

[0109] The first semantic meaning is generated by semantic analysis of the word sequence using a Long Short-Term Memory (LSTM) network, and then encoded.

[0110] Python

[0111] from tensorflow.keras.layers import LSTM, Embedding

[0112] vocab_size = 10000 # The size of the vocabulary corresponding to the word sequence

[0113] embedding_dim = 256

[0114] lstm_units = 128

[0115] # Embedding layer + LSTM

[0116] model = tf.keras.Sequential([

[0117] Embedding(vocab_size, embedding_dim),

[0118] Bidirectional(LSTM(lstm_units, return_sequences=False)),

[0119] Dense(256, activation='relu') # Metadata description semantically encoded feature vector [1,256]

[0120] ]).

[0121] This approach addresses the problem of the separation between structured data and textual descriptions in traditional methods, providing an end-to-end deep learning framework for multimodal data quality assessment, which is one of the technical highlights of this invention.

[0122] In the above embodiments, specifically, the text structured data semantic encoding feature tensor and the metadata description semantic encoding feature vector are subjected to fine-grained database attribute hashing algorithm query encoding. After encoding, a set of structured-metadata text description local hash query encoding vectors is obtained, including:

[0123] The semantic encoding feature tensor of the text structured data is subjected to local feature decomposition along the channel dimension to obtain a set of local semantic feature matrices of the text.

[0124] Define the metadata description semantic encoding feature vector as a unified query vector, and treat each matrix in the set of local semantic feature matrices of the text as an independent database key-value matrix;

[0125] For each database key-value matrix, a fine-grained database attribute hash algorithm query encoding operation is performed. The unified query vector is matched and calculated with the current key-value matrix to generate the corresponding structured-metadata text description local hash query encoding vector.

[0126] The processing results of all database key-value matrices are aggregated to form a complete set of structured metadata text description local hash query encoded vectors.

[0127] It should be noted that the specific implementation of the above steps is as follows:

[0128] 1. Perform local feature decomposition along the channel dimension on the semantic encoding feature tensor of the text structured data to obtain a set of local semantic feature matrices of the text.

[0129] 2. Use the metadata description semantic encoding feature vector as the query vector, and treat each text local semantic feature matrix as a database key-value matrix.

[0130] 3. By using a fine-grained database attribute hashing algorithm to query the encoding, a query encoding vector is generated for each local semantic feature matrix of the text, forming a set;

[0131] The specific implementation method of the above steps is as follows:

[0132] A1. The semantic encoding feature tensor of text structured data is labeled as: The structured relation table features are extracted by the MobileNet convolutional neural network, where C is the number of channels and H*W is the spatial dimension (the row and column grid representation of the data table).

[0133] A2. Metadata description semantic encoding feature vector is labeled as: D represents dimension;

[0134] A3, along The channel dimension C is decomposed into a set of local feature matrices, mathematically expressed as:

[0135] ,

[0136] in, The feature vector represents each spatial location (h, w), and the feature vectors total H*W. The feature vectors represent the semantic representation of each cell in the data form.

[0137] The implementation process of fine-grained hash lookup encoding includes:

[0138] Set the metadata description semantic encoding feature vector as the query vector, and set... As an independent key-value matrix;

[0139] Calculate the similarity score between the query vector and each key-value matrix:

[0140] ,

[0141] in Indicates a stable gradient;

[0142] Using the lightweight hash function H Perform hash mapping:

[0143] ;

[0144] Generate a local hash lookup encoding vector:

[0145] ;

[0146] Output the set of local hash lookup encoded vectors .

[0147] In the above embodiments, specifically, for each database key-value matrix, a fine-grained database attribute hash algorithm query encoding operation is performed. The unified query vector is matched and calculated with the current key-value matrix to generate a corresponding structured-metadata text description local hash query encoding vector, including:

[0148] Calculate the cosine similarity between the query vector and each key vector in the database key matrix to obtain a set of local semantic similarities of the structured-metadata text description;

[0149] The set of local semantic similarities of the structured-metadata text descriptions is input into the Sigmoid function for normalization to obtain a normalized set of local semantic similarities of the structured-metadata text descriptions.

[0150] It should be noted that calculating the query vector and Cosine similarity:

[0151] ,in This represents the set of similarities.

[0152] The set of local semantic similarities of the structured-metadata text description is input into the Sigmoid function for normalization to obtain a normalized set of local semantic similarities of the structured-metadata text description.

[0153] It should be noted that the normalized structured-metadata text description is a set of local semantic similarities:

[0154] ;

[0155] Using the set of normalized structured-metadata text description local semantic similarities as the weight distribution, the position-weighted sum of each key vector in the database key-value matrix is ​​calculated to obtain the set of structured-metadata text description local hash query encoding vectors;

[0156] It should be noted that the set of local semantic similarity described by normalized structured-metadata text is used. As weights, a weighted sum is performed on each key vector in each of the aforementioned database key-value matrices:

[0157] .

[0158] In the above embodiments, specifically, the processing results of all database key-value matrices are aggregated to form a complete set of structured-metadata text description local hash query encoding vectors, including:

[0159] Using the set of normalized structured-metadata text description local semantic similarities as the weight distribution, the position-weighted sum of each key vector in the database key-value matrix is ​​calculated to obtain the set of structured-metadata text description local hash query encoding vectors.

[0160] In the above embodiments, specifically, using the set as descriptive information, feature extraction is performed on the semantic coding feature tensor of the text structured data to generate the joint coding feature tensor of structured-metadata cross-modal quality assessment, including:

[0161] Each local hash query encoding vector in the set of the metadata description semantic encoding feature vector and the structured-metadata text description local hash query encoding vector is input into a cross-modal mask weaving network based on description information;

[0162] Through the processing of the cross-modal mask weaving network, a set of text local semantic feature weight mask matrices based on descriptive information is generated;

[0163] Based on the set of weighted masks of local semantic features of the text, feature clustering is performed on the set of local semantic feature matrices of the text.

[0164] After feature clustering and partitioning, based on the set of text local semantic feature weight mask matrices, feature extraction and aggregation are performed on the set of text local semantic feature matrices to obtain the structured-metadata cross-modal quality assessment joint encoded feature tensor.

[0165] In the above embodiments, specifically, each local hash query encoding vector in the set of metadata description semantic encoding feature vectors and structured-metadata text description local hash query encoding vectors is input into a cross-modal mask weaving network based on description information; through the processing of the cross-modal mask weaving network, a set of text local semantic feature weight mask matrices based on description information is generated, including:

[0166] Calculate the vector product between the semantic encoded feature vector of the metadata description and the transpose of the local hash query encoded vector of the structured-metadata text description to obtain the cross-modal query association matrix of the structured-metadata text description;

[0167] Divide the cross-modal query association matrix of the structured-metadata text description by the square root of the feature scale of the local hash query encoding vector of the structured-metadata text description, and then input it into the softmax function for normalization to obtain the text local semantic feature weight mask matrix based on the description information.

[0168] In the above embodiments, specifically, after feature clustering and partitioning, based on the set of text local semantic feature weight mask matrices, feature extraction and aggregation are performed on the set of text local semantic feature matrices to obtain the structured-metadata cross-modal quality assessment joint encoded feature tensor, including:

[0169] The set of weighted masks of local semantic features based on descriptive information and the set of local semantic feature matrices of text are multiplied by position to obtain the set of cross-modal local fine-grained significant transformation matrices of structured-metadata text description.

[0170] The set of cross-modal local fine-grained significant transformation matrices of the structured-metadata text description is subjected to feature concatenation aggregation to obtain the joint encoded feature tensor of the structured-metadata cross-modal quality assessment.

[0171] It should be noted that the specific implementation process of obtaining the set of cross-modal local fine-grained significant transformation matrices of the structured-metadata text description by dot multiplication at position is as follows:

[0172] Set the set of local semantic feature matrices of the input text: { }, where i = 1 to N (N is the number of local features), each It is a vector of dimension C (C is the feature dimension);

[0173] Set the set of text local semantic feature weight mask matrices: { }, each Is with Weight vectors of the same dimension (i.e., dimension C);

[0174] For each position i, compute the element-wise multiplication (Hadamard product):

[0175] = ⊙ ,

[0176] Here, ⊙ represents element-wise multiplication. Therefore, It is also a vector with dimension C.

[0177] The specific implementation of the feature concatenation aggregation includes:

[0178] All of (i=1 to N) are concatenated into a large feature vector, since each It is C-dimensional. The feature tensor obtained after concatenation has a dimension of (N*C). Then, this vector is regarded as a one-dimensional tensor, or it can be reshaped into a two-dimensional or three-dimensional tensor as needed for subsequent processing.

[0179] Specific code examples (PyTorch) for implementing the above steps:

[0180] mask_set: A list containing N weight mask vectors, each represented as [C];

[0181] feature_set: A list containing N local feature vectors, each vector being [C];

[0182] Python

[0183] Step 1: Dot product by position

[0184] transformed_set = []

[0185] for i in range(N):

[0186] s_i = mask_set[i] * feature_set[i]

[0187] transformed_set.append(s_i)

[0188] Step 2: Feature Cascade Aggregation

[0189] stacked = torch.stack(transformed_set, dim=0)

[0190] joint_encoding_vector = stacked.view(-1)

[0191] The final output, joint_encoding_vector, is the structured-metadata cross-modal quality assessment joint encoded feature tensor (one-dimensional vector).

[0192] In the above embodiments, specifically, the quality of the first offset feature and the second offset feature is evaluated using a preset first deep learning model, and a multimodal data quality evaluation result is output, including:

[0193] Calculate the first and second offset features for the current evaluation period, and then fuse the first and second offset features to generate a fused feature;

[0194] It should be noted that the first offset feature (structured table consistency) is calculated as follows:

[0195] Reconstruct the structured table T_rec (dimension: H×W×C) and the original structured table T_orig (dimension: H×W×C), and calculate the difference of each cell. The calculation of the difference of each cell can use absolute error, relative error, etc.

[0196] Aggregate differences to generate feature vectors (e.g., by row, column, or global statistical features) using global statistical features (mean, variance, maximum, etc.) and difference map convolution features for calculation;

[0197] Code implementation of the first offset feature (structured data offset feature):

[0198] def calc_structural_offset(original_table, reconstructed_table):

[0199] param original_table: Original structured table

[0200] param reconstructed_table: Reconstructs the structured table

[0201] return: First offset feature

[0202] 1. Unit-level difference calculation

[0203] cell_diff = torch.abs(original_table - reconstructed_table)

[0204] 2. Row and column level statistical characteristics

[0205] row_features = torch.stack([

[0206] cell_diff.mean(dim=2),

[0207] cell_diff.std(dim=2),

[0208] cell_diff.max(dim=2).values

[0209] ], dim=-1)

[0210] col_features = torch.stack([

[0211] cell_diff.mean(dim=1),

[0212] cell_diff.std(dim=1),

[0213] cell_diff.max(dim=1).values

[0214] ], dim=-1)

[0215] 3. Semantic Feature Shift

[0216] semantic_diff = F.cosine_similarity(

[0217] original_table.flatten(start_dim=1),

[0218] reconstructed_table.flatten(start_dim=1),

[0219] dim=1

[0220] ) # [B]

[0221] 4. Feature Aggregation

[0222] row_agg = row_features.mean(dim=1)

[0223] col_agg = col_features.mean(dim=1)

[0224] offset_feature = torch.cat([

[0225] row_agg,

[0226] col_agg,

[0227] semantic_diff.unsqueeze(1)

[0228] ], dim=1)

[0229] return offset_feature;

[0230] Calculate the second offset feature (metadata consistency):

[0231] Reconstructed metadata description (text) and original metadata description (text);

[0232] The offset between the two is calculated using text similarity:

[0233] Sentence vectors are extracted using a pre-trained language model (such as BERT), and then Euclidean distance is calculated. The offset features are determined based on the calculated Euclidean distance. At the same time, other features (such as edit distance, word overlap rate, etc.) can be calculated to form a second offset feature.

[0234] Code implementation of the second offset feature (metadata offset feature):

[0235] def calc_metadata_offset(original_meta, reconstructed_meta):

[0236] :param original_meta: Original metadata

[0237] :param reconstructed_meta: Reconstructed metadata

[0238] :return: Second offset feature

[0239] 1. Word-level differences

[0240] word_diff = F.cosine_similarity(

[0241] original_meta, reconstructed_meta, dim=-1 )

[0243] 2. Semantic similarity

[0244] with torch.no_grad():

[0245] orig_emb = bert_model(original_meta)[0][:,0,:]

[0246] rec_emb = bert_model(reconstructed_meta)[0][:,0,:]

[0247] semantic_sim = F.cosine_similarity(orig_emb, rec_emb, dim=1)

[0248] 3. Key entity matching degree

[0249] orig_entities = ner_model.extract_entities(original_meta)

[0250] rec_entities = ner_model.extract_entities(reconstructed_meta)

[0251] entity_match = []

[0252] for orig, rec in zip(orig_entities, rec_entities):

[0253] match_score = len(set(orig) & set(rec)) / (len(set(orig)) +1e-6)

[0254] entity_match.append(match_score)

[0255] entity_match = torch.tensor(entity_match)

[0256] 4. Feature Aggregation

[0257] word_agg = torch.stack([

[0258] word_diff.mean(dim=1),

[0259] word_diff.std(dim=1),

[0260] word_diff.max(dim=1).values

[0261] ], dim=1)

[0262] offset_feature = torch.cat([

[0263] word_agg,

[0264] semantic_sim.unsqueeze(1),

[0265] entity_match.unsqueeze(1)

[0266] ], dim=1)

[0267] return offset_feature;

[0268] Calculate the Euclidean similarity between the fusion features of the current evaluation period and the fusion features of interest in the previous evaluation period, and determine whether the fusion features of the current evaluation period are the fusion features of interest based on the Euclidean similarity.

[0269] If the fusion feature of the current evaluation period is the fusion feature of interest, the fusion feature of the current evaluation period is processed by the first model that has been pre-trained to obtain the deep feature of the current evaluation period. Then, multiple two-dimensional convolutional kernels are used to fuse the deep feature of the time-aligned cached fusion feature of interest with the deep feature of the current evaluation period to obtain the first quality feature of the current evaluation period. The first model adopts the ShuffleNet network.

[0270] If the fusion feature of the current evaluation period is a non-interesting fusion feature, the fusion feature of the current evaluation period is processed by the pre-trained second model to obtain the shallow feature of the current evaluation period. The shallow feature is transformed to obtain the deep feature. Then, multiple two-dimensional convolution kernels are used to fuse the deep feature of the time-aligned cached interest fusion feature with the deep feature of the current evaluation period to obtain the second quality feature of the current evaluation period. The second model adopts the EfficientNet network.

[0271] The first deep learning model is used to evaluate the quality of the first and second quality features, and the multimodal data quality evaluation results are output.

[0272] In the above embodiments, specifically, multiple two-dimensional convolutional kernels are used to fuse the deep features of the time-aligned cached interest fusion features with the deep features of the current evaluation period to obtain the first quality feature of the current evaluation period, including:

[0273] Obtain the deep features of the current evaluation period from the output of the first model. ;

[0274] Calculate the deep features at position b1 in the current evaluation period. Deep features of interest fused with cached features at position b2 The correlation matrix W(b1,b2):

[0275] W(b1,b2) = ,

[0276] in, The deep features at position b1 in the current evaluation period Transform into a two-dimensional matrix. Deep features of the fusion features of interest cached at position b2 in the current evaluation period. Transform into a two-dimensional matrix. For a two-dimensional matrix, the deep features of the cached fused features of interest are the previous deep features of interest;

[0277] Calculate the fusion features of interest for the cache aligned to position b1. :

[0278] = ,

[0279] Interested fusion features for caching with deep features The fusion was performed to obtain the evaluation features at position b1. :

[0280] =R(concat( , )),

[0281] in, (.) represents a 2*2 convolution operation, concat(.) represents a connection operation based on the channel dimension, and R(.) represents a convolution operation with three consecutive two-dimensional convolution kernels, each of which is 2×2 in size.

[0282] Evaluation features of all b1 positions The first quality characteristic that constitutes the current evaluation period .

[0283] It should be noted that the calculation method for the second quality characteristic is the same as that for the first quality characteristic, and will not be repeated in this implementation.

[0284] It should be noted that the preset first deep learning model (e.g., multilayer perceptron) outputs a multimodal data quality assessment result (which can be binary classification: qualified / unqualified, or multi-level scoring).

[0285] It should be understood that the above embodiments are one or more embodiments of the present invention, and there are many other embodiments and variations based on the present invention; any variations and modifications made by those skilled in the art through the present invention without making pioneering innovations are all within the protection scope of the present invention.

Claims

1. A multimodal data quality assessment method based on deep learning, characterized in that, Includes the following steps: The acquisition engine is used to obtain first modality record data and second modality record data marked as aligned in the database, wherein the first modality record data is a structured relational table and the second modality record data is an associated metadata description; Semantic features are extracted from the first modality record data and the second modality record data respectively to obtain the text structured data semantic encoding feature tensor and the metadata description semantic encoding feature vector; The semantic coding feature tensor of the text structured data and the semantic coding feature vector of the metadata description are jointly encoded across modalities to obtain the structured-metadata cross-modal quality assessment joint encoded feature tensor; Data reconstruction is performed based on the structured-metadata cross-modal quality assessment joint encoded feature tensor to obtain reconstructed record data; Calculate the data consistency offset between the reconstructed record data and the first modality record data to generate a first offset feature; calculate the data consistency offset between the reconstructed record data and the second modality record data to generate a second offset feature; evaluate the quality of the first offset feature and the second offset feature using a preset first deep learning model, and output the multimodal data quality evaluation result; The cross-modal joint coding includes: The semantic encoding feature tensor of the text structured data is subjected to local feature decomposition along the channel dimension to obtain a set of local semantic feature matrices of the text. Define the metadata description semantic encoding feature vector as a unified query vector, and treat each matrix in the set of local semantic feature matrices of the text as an independent database key-value matrix; For each database key-value matrix, a fine-grained database attribute hash algorithm query encoding operation is performed. The unified query vector is matched and calculated with the current key-value matrix to generate the corresponding structured-metadata text description local hash query encoding vector. The processing results of all database key-value matrices are aggregated to form a complete set of structured-metadata text description local hash query encoded vectors; Using the complete set of local hash query encoded vectors as descriptive information, feature extraction is performed on the semantic encoded feature tensor of the text structured data to generate the joint encoded feature tensor of structured-metadata cross-modal quality assessment.

2. The method for multimodal data quality assessment based on deep learning according to claim 1, characterized in that, The acquisition engine is used to retrieve first-modality record data and second-modality record data marked as aligned from the database, including: The system first uses the acquisition engine to receive input containing first modality record data and second modality record data. The system then receives instructions provided by the user in natural language, which represent the needs for database name processing and data form processing under the database name. The first modality recording data is subjected to text semantic feature extraction based on the MobileNet convolutional neural network model to obtain the text structured data semantic encoding feature tensor: P(x,y)= , Where P(x,y) represents the text structured data semantic encoding feature tensor output of the first modality record data at the two-dimensional convolution position (x,y), K(•) represents the convolution kernel, I(•) represents the input first modality record data, i and j are the index positions in the convolution kernel, and n is the size of the convolution kernel; The second modality recording data is semantically analyzed using a Long Short-Term Memory (LSTM) network to generate the first semantics. The first semantics is then encoded to obtain the metadata description semantic encoding feature vector. ， in, It is the hidden state at time step t. It is the input vector at time step t. and These are the weight matrices from the input state to the hidden state and from the current hidden state to the next hidden state, respectively. and It is the bias term, and Sigmoid is the activation function.

3. The method for multimodal data quality assessment based on deep learning according to claim 1, characterized in that, For each database key-value matrix, a fine-grained database attribute hash algorithm query encoding operation is performed. The unified query vector is matched and calculated against the current key-value matrix to generate the corresponding structured-metadata text description local hash query encoding vector, including: Calculate the cosine similarity between the query vector and each key vector in the database key matrix to obtain a set of local semantic similarities of the structured-metadata text description; The set of local semantic similarities of the structured-metadata text description is input into the Sigmoid function for normalization to obtain a normalized set of local semantic similarities of the structured-metadata text description.

4. The method for multimodal data quality assessment based on deep learning according to claim 3, characterized in that, The processing results of all database key-value matrices are aggregated to form a complete set of structured-metadata text description local hash query encoded vectors, including: Using the set of normalized structured-metadata text description local semantic similarities as the weight distribution, the position-weighted sum of each key vector in the database key-value matrix is ​​calculated to obtain the complete set of structured-metadata text description local hash query encoding vectors.

5. The method for multimodal data quality assessment based on deep learning according to claim 1, characterized in that, Using the complete set of local hash query encoded vectors as descriptive information, feature extraction is performed on the semantic encoded feature tensor of the text structured data to generate the joint encoded feature tensor of structured-metadata cross-modal quality assessment, including: The local hash query encoding vectors from the complete set of the metadata description semantic encoding feature vectors and the structured-metadata text description local hash query encoding vectors are input into a cross-modal mask weaving network based on the description information; Through the processing of the cross-modal mask weaving network, a set of text local semantic feature weight mask matrices based on descriptive information is generated; Based on the set of weighted masks of local semantic features of the text, feature clustering is performed on the set of local semantic feature matrices of the text. After feature clustering and partitioning, based on the set of text local semantic feature weight mask matrices, feature extraction and aggregation are performed on the set of text local semantic feature matrices to obtain the structured-metadata cross-modal quality assessment joint encoded feature tensor.

6. The method for multimodal data quality assessment based on deep learning according to claim 5, characterized in that, Each local hash query encoding vector in the set of the metadata description semantic encoding feature vector and the structured-metadata text description local hash query encoding vector is input into a cross-modal mask weaving network based on description information; Through the processing of the cross-modal mask weaving network, a set of text local semantic feature weight mask matrices based on descriptive information is generated, including: Calculate the vector product between the semantic encoded feature vector of the metadata description and the transpose of the local hash query encoded vector of the structured-metadata text description to obtain the cross-modal query association matrix of the structured-metadata text description; Divide the cross-modal query association matrix of the structured-metadata text description by the square root of the feature scale of the local hash query encoding vector of the structured-metadata text description, and then input it into the softmax function for normalization to obtain the text local semantic feature weight mask matrix based on the description information.

7. The method for multimodal data quality assessment based on deep learning according to claim 6, characterized in that, After feature clustering and partitioning, based on the set of text local semantic feature weight mask matrices, feature extraction and aggregation are performed on the set of text local semantic feature matrices to obtain the structured-metadata cross-modal quality assessment joint encoded feature tensor, including: The set of weighted masks of local semantic features based on descriptive information and the set of local semantic feature matrices of text are multiplied by position to obtain the set of cross-modal local fine-grained significant transformation matrices of structured-metadata text description. The set of cross-modal local fine-grained significant transformation matrices of the structured-metadata text description is subjected to feature concatenation aggregation to obtain the joint encoded feature tensor of the structured-metadata cross-modal quality assessment.

8. The method for multimodal data quality assessment based on deep learning according to claim 1, characterized in that, The first and second offset features are evaluated using a pre-defined first deep learning model, and the multimodal data quality evaluation results are output, including: Calculate the first and second offset features for the current evaluation period, and then fuse the first and second offset features to generate a fused feature; Calculate the Euclidean similarity between the fusion features of the current evaluation period and the fusion features of the previous evaluation period, and determine whether the fusion features of the current evaluation period are the fusion features of interest based on the Euclidean similarity. If the fusion feature of the current evaluation period is the fusion feature of interest, the fusion feature of the current evaluation period is processed by the first model that has been pre-trained to obtain the deep feature of the current evaluation period. Then, multiple two-dimensional convolutional kernels are used to fuse the deep feature of the time-aligned cached fusion feature of interest with the deep feature of the current evaluation period to obtain the first quality feature of the current evaluation period. The first model adopts the ShuffleNet network. If the fusion feature of the current evaluation period is a non-interesting fusion feature, the fusion feature of the current evaluation period is processed by the pre-trained second model to obtain the shallow feature of the current evaluation period. The shallow feature is transformed to obtain the deep feature. Then, multiple two-dimensional convolution kernels are used to fuse the deep feature of the time-aligned cached interest fusion feature with the deep feature of the current evaluation period to obtain the second quality feature of the current evaluation period. The second model adopts the EfficientNet network. The first deep learning model is used to evaluate the quality of the first and second quality features, and the multimodal data quality evaluation results are output.

9. The method for multimodal data quality assessment based on deep learning according to claim 8, characterized in that, Multiple two-dimensional convolutional kernels are used to fuse the deep features of the time-aligned cached interest fusion features with the deep features of the current evaluation period to obtain the first quality feature of the current evaluation period, including: Obtain the deep features of the current evaluation period from the output of the first model. ; Calculate the deep features at position b1 in the current evaluation period. Deep features of interest fused with cached features at position b2 The correlation matrix W(b1,b2): W(b1,b2) = , in, The deep features at position b1 in the current evaluation period Transform into a two-dimensional matrix. Deep features of the fusion features of interest cached at position b2 in the current evaluation period. Transform into a two-dimensional matrix. For a two-dimensional matrix, the deep features of the cached fused features of interest are the previous deep features of interest; Calculate the fusion features of interest for the cache aligned to position b1. : = ， Interested fusion features for caching with deep features The fusion was performed to obtain the evaluation features at position b1. : =R(concat( , )), in, (·) represents a 2*2 convolution operation, concat(·) represents a connection operation based on the channel dimension, and R(·) represents a convolution operation with three consecutive two-dimensional convolution kernels, each of which is 2×2 in size. Evaluation features of all b1 positions The first quality characteristic that constitutes the current evaluation period .

Images