Multi-modal data quality evaluation method based on deep learning

Abstract

The invention discloses a multi-modal data quality evaluation method based on deep learning. The method comprises the steps of obtaining first modal record data and second modal record data which are marked to be aligned in a database; respectively carrying out semantic feature extraction on the first modal record data and the second modal record data to obtain a text structured data semantic coding feature tensor and a metadata description semantic coding feature vector; performing cross-modal joint coding on the text structured data semantic coding feature tensor and the metadata description semantic coding feature vector to obtain a structured-metadata cross-modal quality evaluation joint coding feature tensor; generating reconstruction record data based on the structured-metadata cross-modal quality evaluation joint coding feature tensor, and calculating offset features of the reconstruction record data; performing quality evaluation on the offset features through a first deep learning model; according to the method, through semantic coding feature extraction, fine-grained hash cross-modal coding and bidirectional reconstruction offset evaluation, the accuracy of data quality evaluation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal data quality evaluation of artificial intelligence, in particular to a multi-modal data quality evaluation method based on deep learning. BACKGROUND

[0002] In the modern information age, databases, as the core tools for data storage and management, carry massive multi-modal data. These multi-modal data include text, images, audio, video, and other forms, widely used in medical, financial, transportation, education, and other fields. For example, medical databases may contain patient medical records, X-ray images, electrocardiogram signals, and other multi-modal data; transportation databases may contain vehicle driving records, surveillance videos, and road condition sensor data.

[0003] However, as multi-modal data continues to accumulate, data quality problems in databases have become increasingly prominent. These problems not only affect the usability and value of data, but also may lead to incorrect decisions and analysis results.

[0004] Traditional database data quality evaluation methods mainly rely on statistical analysis, rule matching, and expert experience. These methods have certain effectiveness in dealing with single-modal data, but have obvious limitations when facing multi-modal data.

[0005] Single-modal data quality evaluation methods: Evaluate data quality by calculating statistical indicators (such as mean, variance, and missing value ratio), which is simple but cannot handle the complex relationships between multi-modal data.

[0006] Check data accuracy based on predefined rules (such as data format, range, etc.), which relies on expert experience and rule completeness, making it difficult to adapt to the diversity of multi-modal data.

[0007] Rely on the experience and knowledge of domain experts to evaluate data quality, which is highly subjective and difficult to standardize and automate.

[0008] Complexity of multi-modal data quality evaluation: Different modalities of data have different characteristics and quality indicators, making traditional single-modal evaluation methods difficult to apply directly to multi-modal data, lacking fine-grained hash cross-modal encoding; The quality problems of multi-modal data may change over time and environment, making traditional static evaluation methods difficult to adapt to dynamically changing data, lacking bidirectional reconstruction offset evaluation; In summary, both single-modal data quality evaluation methods and multi-modal data quality evaluation methods have shortcomings, and lack precision in evaluation methods. SUMMARY

[0009] The application aims to provide a deep learning-based multi-modal data quality evaluation method to solve the above problems in the prior art.

[0010] The application is specifically: A deep learning-based multi-modal data quality evaluation method, comprising the following steps: Using a collection engine to obtain first modal record data and second modal record data marked as aligned in a database, wherein the first modal record data is a structured relational table, and the second modal record data is associated metadata description; Performing semantic feature extraction on the first modal record data and the second modal record data respectively to obtain a text structured data semantic encoding feature tensor and a metadata description semantic encoding feature vector; Performing cross-modal joint encoding on the text structured data semantic encoding feature tensor and the metadata description semantic encoding feature vector to obtain a structured-metadata cross-modal quality evaluation joint encoding feature tensor; Based on the structured-metadata cross-modal quality evaluation joint encoding feature tensor, reconstructing the data to obtain reconstructed record data; Calculating the data consistency offset between the reconstructed record data and the first modal record data to generate a first offset feature; calculating the data consistency offset between the reconstructed record data and the second modal record data to generate a second offset feature; performing quality evaluation on the first offset feature and the second offset feature through a pre-set first deep learning model, and outputting a multi-modal data quality evaluation result; Wherein, the cross-modal joint encoding comprises: Performing fine-grained database attribute hash algorithm query encoding on the text structured data semantic encoding feature tensor and the metadata description semantic encoding feature vector, and obtaining a set of structured-metadata text description local hash query encoding vectors after encoding; Taking the set as description information, performing feature extraction on the text structured data semantic encoding feature tensor to generate the structured-metadata cross-modal quality evaluation joint encoding feature tensor.

[0011] Further, using a collection engine to obtain first modal record data and second modal record data marked as aligned in a database comprises: The system first receives input containing first modal record data and second modal record data using a collection engine, and the system receives instructions provided by a user in natural language form, wherein the instructions represent the needs of database name processing and data table processing under the database name; performing text semantic feature extraction on the first modality record data based on a MobileNet convolutional neural network model to obtain the text structured data semantic encoding feature tensor: P(x,y) = K * I , wherein P(x,y) represents the text structured data semantic encoding feature tensor output of the first modality record data at a two-dimensional convolution position (x,y), K represents a convolution kernel, I represents the input first modality record data, i and j are index positions in the convolution kernel, and n is the size of the convolution kernel; performing semantic analysis on the second modality record data through a long short-term memory (LSTM) network to generate a first semantic, and encoding the first semantic to obtain the metadata description semantic encoding feature vector: , wherein is the hidden state at time step t, is the input vector at time step t, and are weight matrices from the input state to the hidden state and from the current hidden state to the next hidden state, respectively, and are bias terms, and Sigmoid is an activation function.

[0012] Further, performing fine-grained database attribute hashing algorithm query encoding on the text structured data semantic encoding feature tensor and the metadata description semantic encoding feature vector, and obtaining a complete set of structured-metadata text description local hashing query encoding vectors after the encoding, including: performing local feature decomposition along the channel dimension on the text structured data semantic encoding feature tensor to obtain a set of text local semantic feature matrices; defining the metadata description semantic encoding feature vector as a unified query vector, and defining each matrix in the set of text local semantic feature matrices as an independent database key-value matrix; for each database key-value matrix, performing a fine-grained database attribute hashing algorithm query encoding operation, matching and calculating the unified query vector with the current key-value matrix to generate a corresponding structured-metadata text description local hashing query encoding vector; collecting the processing results of all database key-value matrices to form a complete set of structured-metadata text description local hashing query encoding vectors.

[0013] Further, for each database key-value matrix, a fine-grained database attribute hash algorithm query encoding operation is performed to match and calculate the uniform query vector with the current key-value matrix, generating a corresponding structured-metadata text description local hash query encoding vector, including: calculating the cosine similarity between the query vector and each key vector in the database key-value matrix to obtain a set of structured-metadata text description local semantic similarities; inputting the set of structured-metadata text description local semantic similarities into a Sigmoid function for normalization processing to obtain a set of normalized structured-metadata text description local semantic similarities.

[0014] Further, the processing results of all database key-value matrices are collected to form a complete set of structured-metadata text description local hash query encoding vectors, including: using the set of normalized structured-metadata text description local semantic similarities as a weight distribution, calculating the position-weighted sum of each key vector in the database key-value matrix to obtain a set of structured-metadata text description local hash query encoding vectors.

[0015] Further, using the set as description information, feature extraction is performed on the text structured data semantic encoding feature tensor to generate the structured-metadata cross-modal quality assessment joint encoding feature tensor, including: inputting the metadata description semantic encoding feature vector and each local hash query encoding vector in the set of structured-metadata text description local hash query encoding vectors into a cross-modal mask weaving network based on description information; through the processing of the cross-modal mask weaving network, a set of text local semantic feature weight mask matrices based on description information is generated; based on the set of text local semantic feature weight mask matrices, feature clustering division is performed on the set of text local semantic feature matrices; after feature clustering division processing, based on the set of text local semantic feature weight mask matrices, feature extraction and aggregation are performed on the set of text local semantic feature matrices to obtain the structured-metadata cross-modal quality assessment joint encoding feature tensor.

[0016] Further, the metadata description semantic encoding feature vector and each local hash query encoding vector in the set of structured-metadata text description local hash query encoding vectors are input into a cross-modal mask weaving network based on description information; through the processing of the cross-modal mask weaving network, a set of text local semantic feature weight mask matrices based on description information is generated, including: calculating vector products between the semantic coding feature vectors and transposed vectors of the structured-metadata text description local hash query coding vectors to obtain a structured-metadata text description cross-modal query association matrix; inputting the structured-metadata text description cross-modal query association matrix into a softmax function after dividing by square roots of feature scales of the structured-metadata text description local hash query coding vectors to obtain the text local semantic feature weight mask matrix based on description information.

[0017] Further, based on the set of text local semantic feature weight mask matrices, feature extraction and aggregation are performed on the set of text local semantic feature matrices to obtain the structured-metadata cross-modal quality evaluation joint coding feature tensor, including: calculating dot products between the set of text local semantic feature weight mask matrices based on description information and the set of text local semantic feature matrices to obtain a set of structured-metadata text description cross-modal local fine-grained saliency transformation matrices; performing feature concatenation aggregation on the set of structured-metadata text description cross-modal local fine-grained saliency transformation matrices to obtain the structured-metadata cross-modal quality evaluation joint coding feature tensor.

[0018] Further, a first deep learning model is used to perform quality evaluation on the first offset feature and the second offset feature, and output a multi-modal data quality evaluation result, including: calculating the first offset feature and the second offset feature in the current evaluation period, and fusing the first offset feature and the second offset feature to generate a fusion feature; calculating the Euclidean similarity between the fusion feature in the current evaluation period and the fusion feature of interest in the previous evaluation period, and determining whether the fusion feature in the current evaluation period is the fusion feature of interest according to the Euclidean similarity; If the fusion feature in the current evaluation period is the fusion feature of interest, a first model trained in advance is used to process the fusion feature in the current evaluation period to obtain a deep feature in the current evaluation period, and then a plurality of two-dimensional convolution kernels are used to fuse the deep feature of the cached fusion feature of interest and the deep feature in the current evaluation period after time alignment to obtain a first quality feature in the current evaluation period, and the first model uses a ShuffleNet network. If the fusion feature of the current evaluation period is a non-interest fusion feature, the second model trained in advance is used to process the fusion feature of the current evaluation period to obtain the shallow feature of the current evaluation period, the deep feature is obtained by performing feature transformation on the shallow feature, then the deep features of the time-aligned cached interest fusion feature and the current evaluation period are fused by using a plurality of two-dimensional convolution kernels to obtain the second quality feature of the current evaluation period, and the second model adopts an EfficientNet network; The first quality feature and the second quality feature are subjected to quality evaluation by the first deep learning model, and a multi-modal data quality evaluation result is output.

[0019] Further, the deep features of the time-aligned cached interest fusion feature and the current evaluation period are fused by using a plurality of two-dimensional convolution kernels to obtain the first quality feature of the current evaluation period, comprising: obtaining the deep feature of the current evaluation period output by the first model ; calculating the correlation matrix W(b1, b2) of the deep feature at the b1 position of the current evaluation period and the deep feature of the cached interest fusion feature at the b2 position : W(b1, b2) = , wherein, transforming the deep feature at the b1 position of the current evaluation period into a two-dimensional matrix, transforming the deep feature of the cached interest fusion feature at the b2 position of the current evaluation period into a two-dimensional matrix, is a two-dimensional matrix, and the deep feature of the cached interest fusion feature is the last interest deep feature; calculating the cached interest fusion feature aligned to the b1 position : = , fusing the cached interest fusion feature and the deep feature to obtain the evaluation feature at the b1 position : =R(concat( , )), wherein, (.) is a 2*2 convolution operation, concat(.) represents a concatenation operation based on a channel dimension, R(.) represents a convolution operation of 3 consecutive two-dimensional convolution kernels, and the size of the 3 consecutive two-dimensional convolution kernels is 2*2; evaluating features of all b1 positions a first quality feature constituting a current evaluation period .

[0020] Compared with the prior art, the present application has the following beneficial effects: The present application provides a method for acquiring first modality record data and second modality record data marked as aligned in a database by using a collection engine; performing semantic feature extraction on the first modality record data and the second modality record data to obtain a text structured data semantic encoding feature tensor and a metadata description semantic encoding feature vector; performing cross-modality joint encoding on the text structured data semantic encoding feature tensor and the metadata description semantic encoding feature vector to obtain a structured-metadata cross-modality quality evaluation joint encoding feature tensor; performing data reconstruction based on the structured-metadata cross-modality quality evaluation joint encoding feature tensor to obtain reconstructed record data; calculating a data consistency offset between the reconstructed record data and the first modality record data to generate a first offset feature; calculating a data consistency offset between the reconstructed record data and the second modality record data to generate a second offset feature; performing quality evaluation on the first offset feature and the second offset feature by using a preset first deep learning model to output a multi-modality data quality evaluation result; and the present application improves the accuracy of data quality evaluation by means of semantic encoding feature extraction, fine-grained hash cross-modality encoding and bidirectional reconstruction offset evaluation. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a flowchart of a multi-modality data quality evaluation method based on deep learning provided by an embodiment of the present application. DETAILED DESCRIPTION

[0022] The present application will be described in detail below with reference to the accompanying drawings.

[0023] Embodiment 1

[0024] An embodiment of the present application provides a multi-modality data quality evaluation method based on deep learning, which comprises the following steps: Figure 1 , comprising the following steps: S1, acquiring first modality record data and second modality record data marked as aligned in a database by using a collection engine, wherein the first modality record data is a structured relational table, and the second modality record data is associated metadata description; S2, semantic feature extraction is performed on the first modality record data and the second modality record data respectively to obtain a text structured data semantic encoding feature tensor and a metadata description semantic encoding feature vector; S3, cross-modality joint encoding is performed on the text structured data semantic encoding feature tensor and the metadata description semantic encoding feature vector to obtain a structured-metadata cross-modality quality assessment joint encoding feature tensor; S4, data reconstruction is performed based on the structured-metadata cross-modality quality assessment joint encoding feature tensor to obtain reconstructed record data; S5, calculate the data consistency offset between the reconstructed record data and the first modality record data to generate first offset features; calculate the data consistency offset between the reconstructed record data and the second modality record data to generate second offset features; perform quality assessment on the first offset features and the second offset features through a pre-set first deep learning model, and output a multi-modality data quality assessment result; The cross-modality joint encoding comprises: The text structured data semantic encoding feature tensor and the metadata description semantic encoding feature vector are subjected to fine-grained database attribute hash algorithm query encoding, and after encoding, a set of structured-metadata text description local hash query encoding vectors is obtained; The set is taken as description information, and the text structured data semantic encoding feature tensor is subjected to feature extraction to generate the structured-metadata cross-modality quality assessment joint encoding feature tensor.

[0025] It should be noted that the first modality record data (structured relationship table) is a main data carrier, and the second modality record data (associated metadata description) is essentially a descriptive label, just like attaching a classification label (associated metadata description) to the content of a book (structured relationship table), and the core of quality assessment is to verify whether the content conforms to the label description.

[0026] The set is taken as description information, and the text structured data semantic encoding feature tensor is subjected to feature extraction without tensor extraction of the metadata description semantic encoding feature vector, and the specific reason is that: The associated metadata description has completed knowledge distillation in the early hash query, and when the metadata description semantic encoding feature vector is used as a query vector to retrieve the structured feature, the key information has been injected into the local hash query encoding vector set, which is like highlighting (local hash query encoding) in the book (associated metadata description) with keywords (associated metadata description), and only the highlighted content needs to be analyzed subsequently; The structured relationship table needs to be deconstructed, and the high-dimensional characteristics (e.g., 100 columns x 1 million rows) of the table data must be reduced in calculation complexity through feature extraction, and the associated metadata description itself is a low-dimensional vector (e.g., 512 dimensions), and further extraction will lose information; The reconstruction task requires information integrity, and experimental data shows that if the metadata description semantic encoding feature vector is also subjected to feature extraction, because the slight disturbance of the metadata description semantic encoding feature vector will be amplified, for example, "data row field" is incorrectly abstracted as "data column field"; It should be noted that "using a set as description information" limits the role of the associated metadata description to the guidance level and avoids using the associated metadata description itself as a processing object, which is one of the technical highlights of the present application.

[0027] Specifically, the first modality record data and the second modality record data marked as aligned in the database are obtained by using a collection engine; semantic feature extraction is performed on the first modality record data and the second modality record data to obtain a text structured data semantic encoding feature tensor and a metadata description semantic encoding feature vector; cross-modal joint coding is performed on the text structured data semantic encoding feature tensor and the metadata description semantic encoding feature vector to obtain a structured-metadata cross-modal quality assessment joint coding feature tensor; data reconstruction is performed based on the structured-metadata cross-modal quality assessment joint coding feature tensor to obtain reconstructed record data; a data consistency offset between the reconstructed record data and the first modality record data is calculated to generate a first offset feature; a data consistency offset between the reconstructed record data and the second modality record data is calculated to generate a second offset feature; the first offset feature and the second offset feature are subjected to quality assessment by a pre-set first deep learning model to output a multi-modal data quality assessment result; the present application improves the accuracy of data quality assessment through semantic encoding feature extraction, fine-grained hash cross-modal coding, and bidirectional reconstruction offset evaluation.

[0028] In the above embodiment, specifically, obtaining the first modality record data and the second modality record data marked as aligned in the database by using the collection engine includes: The system first receives input containing first modality record data and second modality record data by using a collection engine, and the system receives instructions provided by a user in a natural language form, and the instructions represent the needs of database name processing and data table processing under the database name; It should be noted that the user inputs a natural language instruction, for example: "evaluate the data quality of the 'Process_Records' table in the database 'Oil_Refinery_DB'", a pre-trained BERT model is used to extract the semantic vector of the instruction, and the database name (Oil_Refinery_DB) and table name (Process_Records) are extracted through named entity recognition (NER); The data acquisition of the collection engine includes extracting aligned first modal data (structured relational table) and second modal data (metadata description) from the database: Example: set SELECT * FROM Oil_Refinery_DB.Process_Records to obtain structured table data; Set SELECT column_name, data_type, description; FROM information_schema.columns ; WHERE table_name = 'Process_Records' to obtain metadata description; The text semantic feature extraction based on the MobileNet convolutional neural network model is performed on the first modal record data to obtain the text structured data semantic encoding feature tensor: P(x,y)= , Where P(x,y) represents the text structured data semantic encoding feature tensor output of the first modal record data at the two-dimensional convolution position (x,y), K represents the convolution kernel, I represents the input first modal record data, i and j are index positions in the convolution kernel, and n is the size of the convolution kernel. It should be noted that first, the MobileNet convolutional neural network model is used to convert the structured relational table into an image format, and each row of the table in the structured relational table is converted into a grayscale image (the numerical value is normalized to 0-255). The row height is the image height, and the column width is the image width.

[0029] The second modal record data is analyzed through the long short-term memory (LSTM) network to generate a first semantic, and the first semantic is encoded to obtain the metadata description semantic encoding feature vector: , Where, is the hidden state at time step t, is the input vector at time step t, and are the weight matrices of the input to the hidden state and the hidden to the hidden state, respectively, and is a bias term, Sigmoid is an activation function.

[0030] It should be noted that the metadata description of the second modal record data is used to construct a text sequence, and the constructed text sequence is converted into a word sequence: Input: [ ("temperature", "float", "reaction kettle temperature"), ("pressure", "int", "filter pressure") ]; Output sequence: ["column:temperature", "type:float", "desc:reaction kettle temperature",...]; The first semantic is generated by performing semantic analysis on the word sequence through a long short-term memory (LSTM) network, and the first semantic is encoded. python from tensorflow.keras.layers import LSTM, Embedding vocab_size = 10000 # word sequence corresponding to the size of the vocabulary embedding_dim = 256 lstm_units = 128 # Embedding layer + LSTM model = tf.keras.Sequential([ Embedding(vocab_size, embedding_dim), Bidirectional(LSTM(lstm_units, return_sequences=False)), Dense(256, activation='relu') # Metadata description semantic encoding feature vector [1,256] ])。

[0031] The scheme solves the problem of separation of structured data and text description in the traditional method, provides an end-to-end deep learning framework for multi-modal data quality evaluation, and is one of the technical highlights of the present application.

[0032] In the above embodiment, specifically, the text structured data semantic encoding feature tensor and the metadata description semantic encoding feature vector are subjected to fine-grained database attribute hash algorithm query coding, and after coding, a set of structured-metadata text description local hash query coding vectors is obtained, including: locally decomposing the text structured data semantic encoding feature tensor along the channel dimension to obtain a set of text local semantic feature matrices; defining the metadata description semantic encoding feature vector as a unified query vector, and each matrix in the set of text local semantic feature matrices as an independent database key-value matrix; For each database key-value matrix, performing a fine-grained database attribute hash algorithm query encoding operation, matching and calculating the unified query vector with the current key-value matrix to generate a corresponding structured-metadata text description local hash query encoding vector; Collecting the processing results of all database key-value matrices to form a complete set of structured-metadata text description local hash query encoding vectors.

[0033] It should be noted that the specific implementation of the above steps is as follows: 1. Decompose the text structured data semantic encoding feature tensor along the channel dimension to obtain a set of text local semantic feature matrices.

[0034] 2. The metadata description semantic encoding feature vector is used as a query vector, and each text local semantic feature matrix is regarded as a database key-value matrix.

[0035] 3. Through the fine-grained database attribute hash algorithm query encoding, a query encoding vector is generated for each text local semantic feature matrix to form a set The specific implementation of the above steps is as follows: A1, the text structured data semantic encoding feature tensor is marked as: The structured relationship table features extracted by the MobileNet convolutional neural network, where C is the number of channels, and H*W is the spatial dimension (row and column grid representation of the data table); A2, the metadata description semantic encoding feature vector is marked as: D represents the dimension; A3, decompose along the channel dimension C into a set of local feature matrices, and the mathematical expression is: , wherein, represents the feature vector of each spatial position (h, w), and the feature vector is H*W, which represents the semantic representation of each cell in the data table; The implementation process of the fine-grained hash query encoding includes: Set the metadata description semantic encoding feature vector as the query vector, and set as an independent key-value matrix; ​The similarity score of the query vector and each key-value matrix is calculated: , wherein represents a stable gradient; Hash map the by a lightweight hash function H: ; Generate a local hash query encoding vector: ; Output the set of local hash query encoding vectors .

[0036] In the above embodiment, specifically, for each database key-value matrix, a fine-grained database attribute hash algorithm query encoding operation is performed, the unified query vector is matched and calculated with the current key-value matrix, and a corresponding structured-metadata text description local hash query encoding vector is generated, including: Calculate the cosine similarity between the query vector and each key vector in the database key-value matrix to obtain a set of structured-metadata text description local semantic similarities; Input the set of structured-metadata text description local semantic similarities into a Sigmoid function for normalization processing to obtain a set of normalized structured-metadata text description local semantic similarities It should be noted that the cosine similarity between the query vector and is calculated: wherein represents a similarity set.

[0037] Input the set of structured-metadata text description local semantic similarities into a Sigmoid function for normalization processing to obtain a set of normalized structured-metadata text description local semantic similarities; It should be noted that the set of normalized structured-metadata text description local semantic similarities: ; Take the set of normalized structured-metadata text description local semantic similarities as a weight distribution, calculate the position-weighted sum of each key vector in the database key-value matrix to obtain the set of structured-metadata text description local hash query encoding vectors; It should be noted that the set of normalized structured-metadata text description local semantic similarities is taken as the weight, and the weighted sum of each key vector in each database key-value matrix is calculated. .

[0038] In the above embodiment, specifically, the processing results of all database key-value matrixes are collected to form a complete set of structured-metadata text description local hash query code vectors, including: The position-weighted sum of each key vector in the database key-value matrix is calculated with the set of normalized structured-metadata text description local semantic similarity as the weight distribution to obtain the set of structured-metadata text description local hash query code vectors.

[0039] In the above embodiment, specifically, the set is taken as description information, and feature extraction is performed on the text structured data semantic encoding feature tensor to generate the structured-metadata cross-modal quality assessment joint encoding feature tensor, including: The metadata description semantic encoding feature vector and each local hash query code vector in the set of structured-metadata text description local hash query code vectors are input into a cross-modal mask weaving network based on description information; Through the processing of the cross-modal mask weaving network, a set of text local semantic feature weight mask matrices based on description information is generated; Based on the set of text local semantic feature weight mask matrices, feature clustering division is performed on the set of text local semantic feature matrices; After the feature clustering division processing, based on the set of text local semantic feature weight mask matrices, feature extraction and aggregation are performed on the set of text local semantic feature matrices to obtain the structured-metadata cross-modal quality assessment joint encoding feature tensor.

[0040] In the above embodiment, specifically, the metadata description semantic encoding feature vector and each local hash query code vector in the set of structured-metadata text description local hash query code vectors are input into a cross-modal mask weaving network based on description information; through the processing of the cross-modal mask weaving network, a set of text local semantic feature weight mask matrices based on description information is generated, including: The vector product between the metadata description semantic encoding feature vector and the transpose vector of the structured-metadata text description local hash query code vector is calculated to obtain a structured-metadata text description cross-modal query association matrix; After the structured-metadata text description cross-modal query association matrix is divided by the square root of the feature scale of the structured-metadata text description local hash query code vector, a softmax function is input for normalization processing to obtain the text local semantic feature weight mask matrix based on description information.

[0041] In the above embodiment, specifically, through the feature clustering division processing, based on the set of text local semantic feature weight mask matrices, the set of text local semantic feature matrices is subjected to feature extraction aggregation to obtain the structured-metadata cross-modal quality assessment joint encoding feature tensor, including: Calculating the position point multiplication between the set of text local semantic feature weight mask matrices based on the description information and the set of text local semantic feature matrices to obtain a set of structured-metadata text description cross-modal local fine-grained saliency transformation matrices; The set of structured-metadata text description cross-modal local fine-grained saliency transformation matrices is subjected to feature concatenation aggregation to obtain the structured-metadata cross-modal quality assessment joint encoding feature tensor.

[0042] It should be noted that the structured-metadata text description cross-modal local fine-grained saliency transformation matrix set obtained by the position point multiplication is specifically implemented as follows: Set the input set of text local semantic feature matrices: }, where i=1 to N (N is the number of local features), each is a vector of dimension C (C is the feature dimension); Set the set of text local semantic feature weight mask matrices: }, each is a weight vector of the same dimension as (i.e., the dimension is also C); For each position i, calculate the element-level multiplication (Hadamard product): = ⊙ , where ⊙ represents element-wise multiplication. Therefore, is also a vector of dimension C.

[0043] The feature concatenation aggregation specifically includes: Splice all (i=1 to N) into a large feature vector, since each is C-dimensional, the feature tensor dimension after splicing is (N*C) dimensional, then this vector is regarded as a one-dimensional tensor, or reshaped into a two-dimensional or three-dimensional tensor according to the needs of subsequent processing.

[0044] The specific implementation code example (PyTorch) of the above steps is as follows: mask_set: a list containing N weight mask vectors, each vector is represented as [C]; feature_set: a list containing N local feature vectors, each vector is [C]; python Step 1: Position-wise dot product transformed_set = [] for i in range(N): s_i = mask_set[i] * feature_set[i] transformed_set.append(s_i) Step 2: Feature concatenation aggregation stacked = torch.stack(transformed_set, dim=0) joint_encoding_vector = stacked.view(-1) Final output: joint_encoding_vector is the structured-metadata cross-modal quality assessment joint encoding feature tensor (one-dimensional vector).

[0045] In the above embodiment, specifically, the first offset feature and the second offset feature are quality evaluated by a pre-set first deep learning model, and a multi-modal data quality evaluation result is output, including: The first offset feature and the second offset feature of the current evaluation period are calculated, and the first offset feature and the second offset feature are fused to generate a fused feature; It should be noted that the first offset feature (structured table consistency) is calculated as follows: The structured table T_rec (dimension: HxWxC) and the original structured table T_orig (dimension: HxWxC) are reconstructed, and the difference of each cell is calculated, which can be calculated by absolute error, relative error, etc. The difference is aggregated to generate a feature vector (e.g., row, column, or global statistical feature), which is calculated using global statistical features (mean, variance, maximum, etc.) and difference map convolution features. Code implementation of the first offset feature (structured data offset feature): def calc_structural_offset(original_table, reconstructed_table): param original_table: original structured table param reconstructed_table: reconstructed structured table return: first offset feature 1. Cell-level difference computation cell_diff = torch.abs(original_table - reconstructed_table) 2. Row and column-level statistical features row_features = torch.stack([ cell_diff.mean(dim=2), cell_diff.std(dim=2), cell_diff.max(dim=2).values ], dim=-1) col_features = torch.stack([ cell_diff.mean(dim=1), cell_diff.std(dim=1), cell_diff.max(dim=1).values ], dim=-1) 3. Semantic feature offset semantic_diff = F.cosine_similarity( original_table.flatten(start_dim=1), reconstructed_table.flatten(start_dim=1), dim=1 ) # [B] 4. Feature aggregation row_agg = row_features.mean(dim=1) col_agg = col_features.mean(dim=1) offset_feature = torch.cat([ row_agg, col_agg, semantic_diff.unsqueeze(1) ], dim=1) return offset_feature; Compute the second offset feature (metadata consistency): Reconstructed metadata description (text) and original metadata description (text): Calculate the offset between the two using text similarity: Use a pre-trained language model (such as BERT) to extract sentence vectors, then calculate the Euclidean distance, and determine the offset feature according to the calculation of the Euclidean distance; At the same time, other features (such as edit distance, word overlap rate, etc.) can be calculated to form the second offset feature; Code implementation of the second offset feature (metadata offset feature): def calc_metadata_offset(original_meta, reconstructed_meta): :param original_meta: Original metadata :param reconstructed_meta: Reconstructed metadata :return: Second offset feature 1. Word-level difference word_diff = F.cosine_similarity( original_meta, reconstructed_meta, dim=-1 ) 2. Semantic similarity with torch.no_grad(): orig_emb = bert_model(original_meta)[0][:,0,:] rec_emb = bert_model(reconstructed_meta)[0][:,0,:] semantic_sim = F.cosine_similarity(orig_emb, rec_emb, dim=1) 3. Key entity matching degree orig_entities = ner_model.extract_entities(original_meta) rec_entities = ner_model.extract_entities(reconstructed_meta) entity_match = [] for orig, rec in zip(orig_entities, rec_entities): match_score = len(set(orig) & set(rec)) / (len(set(orig)) +1e-6) entity_match.append(match_score) entity_match = torch.tensor(entity_match) 4. Feature aggregation word_agg = torch.stack([ word_diff.mean(dim=1), word_diff.std(dim=1), word_diff.max(dim=1).values ], dim=1) offset_feature = torch.cat([ word_agg, semantic_sim.unsqueeze(1), entity_match.unsqueeze(1) ], dim=1) return offset_feature; calculate the Euclidean similarity between the fusion feature of the current evaluation period and the fusion feature of interest of the last evaluation period, and determine whether the fusion feature of the current evaluation period is the fusion feature of interest according to the Euclidean similarity; If the fusion feature of the current evaluation period is the fusion feature of interest, process the fusion feature of the current evaluation period using a first model trained in advance to obtain a deep feature of the current evaluation period, and then fuse the deep feature of the cached fusion feature of interest after time alignment and the deep feature of the current evaluation period using multiple two-dimensional convolution kernels to obtain a first quality feature of the current evaluation period, wherein the first model adopts a ShuffleNet network. If the fusion feature of the current evaluation period is a non-interest fusion feature, the second model trained in advance is used to process the fusion feature of the current evaluation period to obtain the shallow feature of the current evaluation period, the deep feature is obtained by performing feature transformation on the shallow feature, then the deep features of the time-aligned cached interest fusion feature and the current evaluation period are fused by using a plurality of two-dimensional convolution kernels to obtain the second quality feature of the current evaluation period, and the second model adopts an EfficientNet network; The first quality feature and the second quality feature are subjected to quality evaluation by the first deep learning model, and a multi-modal data quality evaluation result is output.

[0046] In the above embodiment, specifically, the deep features of the time-aligned cached interest fusion feature and the current evaluation period are fused by using a plurality of two-dimensional convolution kernels to obtain the first quality feature of the current evaluation period, comprising: obtaining the deep feature of the current evaluation period output by the first model ; calculating the correlation matrix W(b1, b2) of the deep feature at the b1 position of the current evaluation period and the deep feature of the cached interest fusion feature at the b2 position : W(b1, b2) = , wherein, transforming the deep feature at the b1 position of the current evaluation period into a two-dimensional matrix, transforming the deep feature of the cached interest fusion feature at the b2 position of the current evaluation period into a two-dimensional matrix, is a two-dimensional matrix, and the deep feature of the cached interest fusion feature is the last interest deep feature; calculating the cached interest fusion feature aligned to the b1 position : = , fusing the cached interest fusion feature and the deep feature to obtain the evaluation feature at the b1 position : =R(concat( , )), wherein, (.) is a 2*2 convolution operation, concat(.) represents a concatenation operation based on the channel dimension, R(.) represents a convolution operation of 3 consecutive two-dimensional convolution kernels, and the size of the 3 consecutive two-dimensional convolution kernels is 2*2; evaluate the features of all b1 positions the first quality feature that constitutes the current evaluation period .

[0047] It should be noted that the second quality feature is calculated in the same way as the first quality feature, and the implementation will not be described again.

[0048] It should be noted that the preset first deep learning model (for example, a multilayer perceptron) outputs a multi-modal data quality evaluation result (which can be a binary classification: qualified / unqualified, or a multi-level score).

[0049] It should be understood that the above embodiments are one or more embodiments of the present application, and there are many other embodiments and variations based on the present application; the modifications and modifications made by the ordinary skilled in the art through the present application without making pioneering innovations all belong to the protection scope of the present application.

Claims

1. A multimodal data quality assessment method based on deep learning, characterized in that: The following steps are involved: Acquire, using a collection engine, first modality record data and second modality record data marked as aligned in a database, wherein the first modality record data is a structured relational table and the second modality record data is an associated metadata description; Performing semantic feature extraction on the first modality recording data and the second modality recording data respectively to obtain a text structured data semantic encoding feature tensor and a metadata description semantic encoding feature vector; Performing cross-modal joint encoding on the text structured data semantic encoding feature tensor and the metadata description semantic encoding feature vector to obtain a structured-metadata cross-modal quality assessment joint encoding feature tensor; Reconstructing data based on the structured-metadata cross-modal quality assessment and the jointly encoded feature tensor to obtain reconstructed recorded data; Calculating a data consistency offset between the reconstructed recorded data and the first modal recorded data to generate a first offset feature; calculating a data consistency offset between the reconstructed recorded data and the second modal recorded data to generate a second offset feature; performing a quality assessment on the first offset feature and the second offset feature using a preset first deep learning model, and outputting a multimodal data quality assessment result; The cross-modal joint encoding includes: Performing fine-grained database attribute hash algorithm query encoding on the text structured data semantic encoding feature tensor and the metadata description semantic encoding feature vector, and obtaining a set of structured-metadata text description local hash query encoding vectors after encoding; Using the set as description information, feature extraction is performed on the text structured data semantic encoding feature tensor to generate the structured-metadata cross-modal quality assessment joint encoding feature tensor.

2. The multimodal data quality assessment method based on deep learning according to claim 1, characterized in that: Acquiring the first modality recording data and the second modality recording data marked as aligned in the database using the acquisition engine includes: The system first receives input including first modality record data and second modality record data using a collection engine, and the system receives instructions provided by a user in a natural language form, wherein the instructions represent requirements for processing a database name and a data form under the database name; Perform text semantic feature extraction based on the MobileNet convolutional neural network model on the first modal recorded data to obtain the semantic encoding feature tensor of the text structured data: P(x,y)= , Where P(x,y) represents the semantic encoding feature tensor output of the text structured data of the first modality recording data at the two-dimensional convolution position (x,y), K represents the convolution kernel, I represents the input first modality recording data, i and j are the index positions in the convolution kernel, and n is the size of the convolution kernel; The second modality recorded data is semantically analyzed using a long short-term memory (LSTM) network to generate a first semantic, and the first semantic is encoded to obtain the metadata description semantic encoding feature vector: ， in, is the hidden state at time step t, is the input vector at time step t, and are the weight matrices from the input state to the hidden state and from the current hidden state to the next hidden state, and is the bias term and Sigmoid is the activation function.

3. The multimodal data quality assessment method based on deep learning according to claim 1, characterized in that: Fine-grained database attribute hash algorithm query encoding is performed on the text structured data semantic encoding feature tensor and the metadata description semantic encoding feature vector, and after encoding, a set of structured-metadata text description local hash query encoding vectors is obtained, including: Performing local feature decomposition along the channel dimension on the semantic encoding feature tensor of the text structured data to obtain a set of text local semantic feature matrices; defining the metadata description semantic encoding feature vector as a unified query vector, and using each matrix in the set of text local semantic feature matrices as an independent database key value matrix; For each database key-value matrix, perform a fine-grained database attribute hash algorithm query encoding operation, match the unified query vector with the current key-value matrix, and generate the corresponding structured-metadata text description local hash query encoding vector; The processing results of all database key-value matrices are collected to form a complete set of structured-metadata text description local hash query encoding vectors.

4. The multimodal data quality assessment method based on deep learning according to claim 3, characterized in that: For each database key-value matrix, perform a fine-grained database attribute hash algorithm query encoding operation, match the unified query vector with the current key-value matrix, and generate the corresponding structured-metadata text description local hash query encoding vector, including: Calculating the cosine similarity between the query vector and each key vector in the database key matrix to obtain a set of structured-metadata text description local semantic similarities; The set of structured-metadata text description local semantic similarities is input into a Sigmoid function for normalization processing to obtain a set of normalized structured-metadata text description local semantic similarities.

5. The multimodal data quality assessment method based on deep learning according to claim 4, characterized in that: Gather the processing results of all database key-value matrices to form a complete set of structured-metadata text description local hash query encoding vectors, including: Using the set of normalized structured-metadata text description local semantic similarities as weight distribution, the position-weighted sum of each key vector in the database key-value matrix is ​​calculated to obtain the set of structured-metadata text description local hash query encoding vectors.

6. The multimodal data quality assessment method based on deep learning according to claim 1, characterized in that: Taking the set as description information, performing feature extraction on the text structured data semantic encoding feature tensor to generate the structured-metadata cross-modal quality assessment joint encoding feature tensor, including: Inputting each local hash query encoding vector in the set of the metadata description semantic encoding feature vector and the structured-metadata text description local hash query encoding vector into a cross-modal mask weaving network based on description information; Generate a set of text local semantic feature weight mask matrices based on the description information through processing of the cross-modal mask weaving network; Based on the set of the text local semantic feature weight mask matrices, performing feature clustering division on the set of the text local semantic feature matrices; After feature clustering and division processing, based on the set of text local semantic feature weight mask matrices, feature extraction and aggregation are performed on the set of text local semantic feature matrices to obtain the structured-metadata cross-modal quality assessment joint encoding feature tensor.

7. The multimodal data quality assessment method based on deep learning according to claim 6, characterized in that: Inputting each local hash query encoding vector in the set of the metadata description semantic encoding feature vector and the structured-metadata text description local hash query encoding vector into a cross-modal mask weaving network based on description information; Through the processing of the cross-modal mask weaving network, a set of text local semantic feature weight mask matrices based on the description information is generated, including: Calculating the vector product between the metadata description semantic encoding feature vector and the transposed vector of the structured-metadata text description local hash query encoding vector to obtain a structured-metadata text description cross-modal query association matrix; The structured-metadata text description cross-modal query association matrix is ​​divided by the square root of the feature scale of the structured-metadata text description local hash query encoding vector, and then input into the softmax function for normalization processing to obtain the text local semantic feature weight mask matrix based on the description information.

8. The multimodal data quality assessment method based on deep learning according to claim 7, characterized in that: After feature clustering processing, based on the set of text local semantic feature weight mask matrices, feature extraction and aggregation are performed on the set of text local semantic feature matrices to obtain the structured-metadata cross-modal quality assessment joint encoding feature tensor, including: Calculating the position-wise multiplication between the set of text local semantic feature weight mask matrices based on the description information and the set of text local semantic feature matrices to obtain a set of structured-metadata text description cross-modal local fine-grained significant transformation matrices; The set of local fine-grained significant transformation matrices of the structured-metadata text description cross-modality is subjected to feature cascade aggregation to obtain the structured-metadata cross-modality quality assessment joint encoding feature tensor.

9. The multimodal data quality assessment method based on deep learning according to claim 1, characterized in that: The first offset feature and the second offset feature are quality evaluated by using a preset first deep learning model, and a multimodal data quality evaluation result is output, including: Calculating a first offset feature and a second offset feature of a current evaluation period, and fusing the first offset feature and the second offset feature to generate a fused feature; Calculate the Euclidean similarity between the fusion features of the current evaluation cycle and the fusion features of the previous evaluation cycle, and determine whether the fusion features of the current evaluation cycle are the fusion features of interest based on the Euclidean similarity; If the fused features of the current evaluation cycle are the fused features of interest, the fused features of the current evaluation cycle are processed using a pre-trained first model to obtain deep features of the current evaluation cycle. Then, multiple two-dimensional convolution kernels are used to fuse the deep features of the time-aligned cached fused features of interest with the deep features of the current evaluation cycle to obtain a first quality feature of the current evaluation cycle. The first model uses a ShuffleNet network. If the fused features of the current evaluation cycle are not fused features of interest, the pre-trained second model is used to process the fused features of the current evaluation cycle to obtain shallow features of the current evaluation cycle, and the shallow features are transformed to obtain deep features. Then, multiple two-dimensional convolution kernels are used to fuse the deep features of the time-aligned cached fused features of interest and the deep features of the current evaluation cycle to obtain the second quality features of the current evaluation cycle. The second model uses the EfficientNet network. The first quality feature and the second quality feature are quality evaluated by the first deep learning model, and a multimodal data quality evaluation result is output.

10. The multimodal data quality assessment method based on deep learning according to claim 9, characterized in that: Multiple two-dimensional convolution kernels are used to fuse the deep features of the time-aligned cached fusion features of interest with the deep features of the current evaluation cycle to obtain the first quality features of the current evaluation cycle, including: Get the deep features of the current evaluation cycle output by the first model ; Calculate the deep features at position b1 in the current evaluation cycle and the deep features of the fused features of interest cached at position b2 The incidence matrix W(b1,b2): W(b1,b2) = , in, The deep features at position b1 in the current evaluation cycle Transformed into a two-dimensional matrix, The deep features of the fused features of interest cached at position b2 in the current evaluation cycle Transformed into a two-dimensional matrix, It is a two-dimensional matrix, and the deep features of the cached fusion features of interest are the previous deep features of interest; Calculate the fused features of interest for the cache aligned to position b1 : = ， Interesting fusion features for cache With deep features Perform fusion to obtain the evaluation features of position b1 : =R(concat( , )), in, (.) is a 2*2 convolution operation, concat(.) represents a connection operation based on the channel dimension, and R(.) represents a convolution operation of three consecutive two-dimensional convolution kernels, and the size of the three consecutive two-dimensional convolution kernels is 2×2; The evaluation features of all b1 positions The first quality characteristic that constitutes the current evaluation cycle .

Images