Medical record quality control method and system based on multi-modal fusion large model

Abstract

The invention discloses a medical record quality control method and system based on a multi-modal fusion large model. Splicing the quality control rule with the text to obtain a spliced text; inputting the image, the spliced text and the time series data into a multi-modal fusion large model, and calculating a first fusion similarity matrix which takes the image as a first query sequence and takes the text as a first key sequence and a first value sequence; distributing a first weight for the query elements of the first query sequence according to the matrix; performing weighted summation on the first value sequence according to the first weight to obtain a fusion output feature; calculating a second fusion similarity matrix which takes the time sequence data as a second query sequence and takes the fusion output feature as a second key sequence and a second value sequence; distributing a second weight for the query elements of the second query sequence according to the matrix; and performing weighted summation on the second value sequence according to the second weight, and outputting a medical record quality control conclusion. By fusing medical data sources of multiple modalities and utilizing an artificial intelligence technology, the connotation quality control of medical records is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of medical record quality control, and particularly to a medical record quality control method and system based on a multi-modal fusion large model. Background Art

[0002] With the development of medical informatization, the application of Electronic Medical Records (EMR) in hospitals has become more and more extensive. However, due to the diversity and complexity of medical data, the content quality control of medical records has become an urgent problem to be solved. At present, the content quality control of medical records mainly relies on manual review and rule-based automated checks, and the current medical record content quality control technology mainly focuses on text data, analyzing and reviewing medical record texts through Natural Language Processing (NLP) technology. However, this method has the following deficiencies:

[0003] Single modality: Only the text data is quality controlled, ignoring other important medical data such as images, electrocardiograms, electroencephalograms, etc., resulting in incomplete content quality control of medical records.

[0004] Lack of intelligence: The existing medical record content quality control methods mainly rely on predefined rules and simple machine learning models, and it is difficult to handle complex medical data and quality control requirements. Summary of the Invention

[0005] The present invention solves the technical problems of single modality and lack of intelligence in the prior art by providing a medical record quality control method and system based on a multi-modal fusion large model.

[0006] The present invention provides a medical record quality control method based on a multi-modal fusion large model, including:

[0007] Concatenate the quality control rules with the text data to obtain the concatenated text data;

[0008] Input the image data, the concatenated text data, and the time series data into a pre-constructed multi-modal fusion large model, and calculate a first fusion similarity matrix between the image data as the first query sequence, the concatenated text data as the first key sequence and first value sequence;

[0009] According to the first fusion similarity matrix, assign a first weight to each query element in the first query sequence;

[0010] Perform weighted summation on the first value sequence according to the first weight to obtain a fusion output feature;

[0011] Calculate a second fusion similarity matrix between the time series data as the second query sequence, the fusion output feature as the second key sequence and second value sequence;

[0012] Allocate a second weight to each query element in the second query sequence according to the second fusion similarity matrix;

[0013] Perform a weighted sum on the second value sequence according to the second weight, and output the medical record quality control conclusion.

[0014] Specifically, the calculation uses the image data as the first query sequence, the spliced text data as the first key sequence and the first value sequence to calculate the first fusion similarity matrix; according to the first fusion similarity matrix, allocate a first weight to each query element in the first query sequence; perform a weighted sum on the first value sequence according to the first weight to obtain the fused output feature, including:

[0015] Through the formula S ij =Q i *K j T Calculate the first fusion similarity matrix S ij ; where Q i is the query element in the first query sequence, and K j T is the transpose of each key element K j in the first key sequence;

[0016] Through the formula Calculate the similarity score S' ij ; where d k is a preset scaling factor;

[0017] Through the formula Normalize the similarity score S' ij to obtain the attention weight α ij ;

[0018] Through the formula Perform a weighted sum to obtain the fused output feature o i ; where V j is each value element in the first value sequence.

[0019] Specifically, inputting the image data, the spliced text data and the time series data into a pre-constructed multi-modal fusion large model, and calculating the first fusion similarity matrix between the image data as the first query sequence, the spliced text data as the first key sequence and the first value sequence, includes:

[0020] Input the image data, the spliced text data, and the time series data into the multi-modal fusion large model. The image processing unit in the multi-modal fusion large model extracts information from the image data to obtain image features. The text processing unit in the multi-modal fusion large model converts the spliced text data into text vector features. The time series processing unit in the multi-modal fusion large model extracts features from the time series data to obtain time series features;

[0021] Calculate a first fusion similarity matrix with the image features as the first query sequence, the text vector features as the first key sequence and first value sequence;

[0022] The calculation of the second fusion similarity matrix with the time series data as the second query sequence, the fusion output features as the second key sequence and second value sequence includes:

[0023] Calculate a second fusion similarity matrix with the time series features as the second query sequence, the fusion output features as the second key sequence and second value sequence.

[0024] Specifically, the time series processing unit in the multi-modal fusion large model extracts features from the time series data to obtain time series features, including:

[0025] Perform normalization processing on the time series data to obtain normalized time series data;

[0026] Perform fitting on the normalized time series data to obtain a fitted time series feature vector;

[0027] Merge the fitted time series feature vectors to obtain a comprehensive feature vector matrix.

[0028] Specifically, after obtaining the comprehensive feature vector matrix, it further includes:

[0029] Through the formula Normalize the comprehensive feature vector matrix F to obtain a normalized matrix X std ; where X stdij is the element in the i-th row and j-th column of the matrix X std , F ij is the element in the i-th row and j-th column of the comprehensive feature vector matrix F, μ j is the mean of the j-th feature in the matrix F, and σ j is the standard deviation of the j-th feature in the matrix F;

[0030] Through the formula Calculate to obtain the matrix X stdThe covariance matrix Σ of T std is the matrix X std The transpose of

[0031] Performing eigenvalue decomposition on the covariance matrix Σ to find a diagonal matrix Λ and an eigenvector matrix V such that ΣV=VΛ;

[0032] According to the size of the eigenvalue, select the eigenvector V corresponding to the first k largest eigenvalues of the eigenvector matrix V k ;

[0033] By formula X pca =V k *X std Calculate the eigenvector matrix X after dimensionality reduction pca ;

[0034] By formula X (n,t,d) pos =X (n,t,d) pca +PE (t,d) Calculate the position-encoded matrix X (n,t,d) pos ; Among them, X (n,t,d) pca is the matrix X pca The elements in, n is the batch size, t is the number of time steps, d is the feature dimension, PE (t,d) is an element in the position encoding matrix PE;

[0035] The matrix X after the position encoding (n,t,d) pos After being processed by multiple layers of Transformer encoders, the final temporal features are output.

[0036] The present invention also provides a medical record quality control system based on a multimodal fusion large model, comprising:

[0037] A text splicing module is used to splice the quality control rules with the text data to obtain the spliced text data;

[0038] A first fusion similarity matrix calculation module is used to input the image data, the spliced text data and the time series data into a pre-built multimodal fusion large model, and calculate a first fusion similarity matrix between the image data as a first query sequence, the spliced text data as a first key sequence and a first value sequence;

[0039] a first weight assignment module, configured to assign a first weight to each query element in the first query sequence according to the first fused similarity matrix;

[0040] A fused feature output module, configured to perform weighted summation on the first value sequence according to the first weight to obtain a fused output feature;

[0041] A second fused similarity matrix calculation module, configured to calculate a second fused similarity matrix between the time series data as the second query sequence, the fused output feature as the second key sequence and the second value sequence;

[0042] A second weight assignment module, configured to assign a second weight to each query element in the second query sequence according to the second fused similarity matrix;

[0043] A medical record quality control module, configured to perform weighted summation on the second value sequence according to the second weight and output a medical record quality control conclusion.

[0044] Specifically, the first fused similarity matrix calculation module is specifically configured to input the image data, the spliced text data and the time series data into a pre-constructed multi-modal fusion large model, and through the formula S ij = Q i * K j T calculate to obtain the first fused similarity matrix S ij ; where Q i is the query element in the first query sequence, and K j T is the transpose of each key element K j in the first key sequence;

[0045] The first weight assignment module includes:

[0046] A similarity score calculation unit, configured to calculate a similarity score S' through the formula ; where d ij is a preset scaling factor; k

[0047] A first weight calculation unit, configured to normalize the similarity score S' through the formula to obtain an attention weight α ij ; ij

[0048] The fused feature output module is specifically configured to perform weighted summation through the formula to obtain a fused output feature o i ; where V j is each value element in the first value sequence.

[0049] Specifically, the first fused similarity matrix calculation module includes: ​​

[0050] A multimodal data input unit for inputting the image data, the spliced text data, and the time series data into the multimodal fusion large model;

[0051] An image processing unit for extracting information from the image data to obtain image features;

[0052] A text processing unit for converting the spliced text data into text vector features;

[0053] A time series processing unit for extracting features from the time series data to obtain time series features;

[0054] The first fusion similarity matrix calculation unit for calculating the first fusion similarity matrix S through the formula S ij =Q i *K j T ; where Q ij ; in which, Q i is the query element in the first query sequence, and K j T is the transpose of each key element K j in the first key sequence;

[0055] The second fusion similarity matrix calculation module is specifically used to calculate the second fusion similarity matrix between the time series features as the second query sequence, the fusion output features as the second key sequence and the second value sequence.

[0056] Specifically, the time series processing unit includes:

[0057] A time series data normalization sub-unit for normalizing the time series data to obtain normalized time series data;

[0058] A data fitting sub-unit for fitting the normalized time series data to obtain a fitted time series feature vector;

[0059] A feature vector merging sub-unit for merging the fitted time series feature vectors to obtain a comprehensive feature vector matrix.

[0060] Specifically, the time series processing unit further includes:

[0061] A comprehensive feature vector matrix normalization sub-unit for normalizing the comprehensive feature vector matrix F through the formula to obtain a normalized matrix X std ; where is the element in the i-th row and j-th column of the matrix X std and Fij is the element in the \(i\)-th row and \(j\)-th column of the comprehensive feature vector matrix \(F\), \(\mu\) j is the mean of the \(j\)-th feature in the matrix \(F\), \(\sigma\) j is the standard deviation of the \(j\)-th feature in the matrix \(F\);

[0062] Covariance matrix calculation sub-unit, used to calculate the matrix \(X\) through the formula and obtain the covariance matrix \(\Sigma\) of the matrix \(X\); where \(N\) is the number of features, and \(X\) std is the transpose of the matrix \(X\) T std ; std ;

[0063] Eigenvalue decomposition sub-unit, used to perform eigenvalue decomposition on the covariance matrix \(\Sigma\) to find the diagonal matrix \(\Lambda\) and an eigenvector matrix \(V\) such that \(\Sigma V = V\Lambda\);

[0064] Eigenvector acquisition sub-unit, used to select the eigenvectors \(V\) corresponding to the top \(k\) largest eigenvalues of the eigenvector matrix \(V\) according to the magnitude of the eigenvalues k ;

[0065] Eigenvector matrix dimensionality reduction sub-unit, used to calculate the dimensionality-reduced eigenvector matrix \(X\) through the formula \(X\) pca = \(V\) k * \(X\) std ; pca ;

[0066] Matrix position encoding sub-unit, used to calculate the position-encoded matrix \(X\) through the formula \(X\) (n,t,d) pos = \(X\) (n,t,d) pca + \(PE\) (t,d) ; where \(X\) (n,t,d) pos is the element in the matrix \(X\) (n,t,d) pca ; \(n\) is the batch size, \(t\) is the time step, \(d\) is the feature dimension, and \(PE\) pca is the element in the position encoding matrix \(PE\); (t,d) ;

[0067] Final temporal feature output sub-unit, used to output the final temporal features after processing the position-encoded matrix \(X\) (n,t,d) pos through multiple layers of Transformer encoders.

[0068] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0069] First, splice the quality control rules with the text data to obtain the spliced text data; input the image data, the spliced text data, and the time series data into a pre-constructed multi-modal fusion large model, and calculate the first fusion similarity matrix between the image data as the first query sequence, the spliced text data as the first key sequence and the first value sequence; then, according to the first fusion similarity matrix, assign the first weight to each query element in the first query sequence; perform weighted summation on the first value sequence according to the first weight to obtain the fusion output feature; then calculate the second fusion similarity matrix between the time series data as the second query sequence, the fusion output feature as the second key sequence and the second value sequence; according to the second fusion similarity matrix, assign the second weight to each query element in the second query sequence; finally, perform weighted summation on the second value sequence according to the second weight, and output the medical record quality control conclusion. By fusing multiple modalities of medical data sources and using advanced artificial intelligence technologies, the present invention realizes the content quality control of medical records, improves the accuracy and consistency of medical documents, and further improves the quality of medical services. Description of the Drawings

[0070] Figure 1 It is a flowchart of the medical record quality control method based on a multi-modal fusion large model provided by an embodiment of the present invention;

[0071] Figure 2 It is a schematic diagram of the medical record quality control method based on a multi-modal fusion large model provided by an embodiment of the present invention;

[0072] Figure 3 It is a module diagram of the medical record quality control system based on a multi-modal fusion large model provided by an embodiment of the present invention. Detailed Embodiments

[0073] An embodiment of the present invention provides a medical record quality control method and system based on a multi-modal fusion large model, which solves the technical problems of single modality and lack of intelligence in the prior art.

[0074] The technical solution in the embodiment of the present invention is as follows to solve the above technical problems:

[0075] 1. First, collect data from various medical data sources such as the hospital information system (HIS), picture archiving and communication system (PACS), and laboratory information system (LIS), including image data, text data, and time series data such as electrocardiogram and electroencephalogram.

[0076] 2. Utilize deep learning technology to construct a large model that integrates multi-modal data. Specifically, the modalities supported include input in three dimensions: text, image, and time-series data. Through the fusion processing of multi-modal data, comprehensive analysis and understanding of medical record data are achieved. At the same time, according to medical norms and standards, a systematic set of quality control rules is established, including rules such as consistency (specifically, inconsistent examination results in physical examinations and specialist examinations), integrity (the onset time in the present illness history is not described or is incorrect), and logic (omission of diagnosis in the preliminary diagnosis), etc. Combining the analysis results of the multi-modal large model, automatic quality control of medical record documents is performed.

[0077] 3. Finally, feedback the medical record quality control results to relevant medical staff, provide detailed quality control reports and optimization suggestions to help medical staff promptly discover and correct problems in medical records and improve the quality of medical record data.

[0078] To better understand the above technical solution, the above technical solution will be described in detail below in combination with the accompanying drawings of the specification and specific implementation manners.

[0079] As Figure 1 and Figure 2 shown, the medical record quality control method based on a multi-modal fusion large model provided by the embodiments of the present invention includes:

[0080] Step S110: Concatenate the quality control rules with the text data to obtain the concatenated text data;

[0081] Step S120: Input the image data, the concatenated text data, and the time-series data into a pre-constructed multi-modal fusion large model, and calculate a first fusion similarity matrix between the image data as the first query sequence, the concatenated text data as the first key sequence, and the first value sequence;

[0082] Step S130: Assign a first weight to each query element in the first query sequence according to the first fusion similarity matrix;

[0083] Step S140: Perform weighted summation on the first value sequence according to the first weight to obtain a fusion output feature;

[0084] Specifically, calculating a first fusion similarity matrix between the image data as the first query sequence, the concatenated text data as the first key sequence, and the first value sequence; assigning a first weight to each query element in the first query sequence according to the first fusion similarity matrix; performing weighted summation on the first value sequence according to the first weight to obtain a fusion output feature, including:

[0085] Through the formula S ij =Q i *K j T calculate to obtain the first fusion similarity matrix Sij ; where Q i is the query element in the first query sequence, and K j T is the transpose of each key element K j in the first key sequence;

[0086] The similarity score S' is calculated through the formula ; where d ij is a preset scaling factor; k

[0087] The similarity score S' is normalized through the formula to obtain the attention weight α ij ; ij

[0088] The weighted sum is performed through the formula to obtain the fused output feature o i ; where V j is each value element in the first value sequence.

[0089] To further illustrate the embodiments of the present invention, image data, spliced text data, and time series data are input into a pre-constructed multi-modal fusion large model to calculate a first fusion similarity matrix between the image data as the first query sequence, the spliced text data as the first key sequence, and the first value sequence, including:

[0090] Input image data, spliced text data, and time series data into the multi-modal fusion large model. The image processing unit in the multi-modal fusion large model extracts information from the image data to obtain image features. The text processing unit in the multi-modal fusion large model converts the spliced text data into text vector features. The time series processing unit in the multi-modal fusion large model extracts features from the time series data to obtain time series features;

[0091] Calculate a first fusion similarity matrix between the image features as the first query sequence, the text vector features as the first key sequence, and the first value sequence;

[0092] Among them, the time series processing unit in the multi-modal fusion large model extracts features from the time series data to obtain time series features, including:

[0093] Perform standardization processing on the time series data to obtain standardized time series data; specifically, perform standardization processing through the formula to obtain the standardized time series data X norm ; where X is the original time series data, μ is the mean of the original time series data, and σ is the standard deviation of the original time series data.

[0094] ​​The standardized time series data is fitted to obtain the fitted time series feature vector. Specifically, the ARMA (AutoRegressive Moving Average) algorithm is used to fit the time series data to obtain the fitted time series feature vector. Suppose X is the input time series data with shape (T, D), where T and D are the number of time steps and the feature dimension respectively. The Akaike Information Criterion (AIC) is used to select the appropriate orders p and q of the ARMA model. In this embodiment, the search range of p and q is set from 0 to 5, and all combinations of p and q are traversed to select the combination with the minimum AIC value as the optimal order of the ARMA model. After fitting the ARMA model, the parameters of the model are used as features. For the ARMA(p, q) model, it includes autoregressive coefficients φ1, φ2, …, φp, moving average coefficients θ1, θ2, …, θq. These parameters reflect the dynamics of the time series. The parameters of the ARMA model and the residual sequence are output as the feature vector. Specifically as follows:

[0095] Feature vector = [φ1, φ2, …, φ p , θ1, θ2, …, θ q , ∈1, ∈2, …, ∈ T ;

[0096] where ∈ T is the residual sequence.

[0097] The fitted time series feature vectors are combined to obtain the comprehensive feature vector matrix. Specifically, the time series data of each lead is standardized separately, and then the ARMA model is applied to fit the time series data of each lead to obtain the feature vector of each lead. The feature vectors of all leads are combined to obtain a comprehensive feature vector matrix. Suppose there are D leads, and the feature vector of each lead is fi, then the comprehensive feature vector matrix F can be expressed as F = [f1, f2, …, fD]. Where fi is the feature vector of the i-th lead.

[0098] Since different time series data have different optimal p and q parameters of the ARMA model, the number of feature vectors of each lead is also different. Therefore, it is necessary to reduce the dimension of each feature vector in the comprehensive feature vector matrix F to unify the dimension of the feature vectors. Specifically, after obtaining the comprehensive feature vector matrix, it further includes:

[0099] The comprehensive feature vector matrix F is standardized through the formula to obtain the standardized matrix X std ; where is the element in the i-th row and j-th column of the matrix X std , F ij is the element in the i-th row and j-th column of the comprehensive feature vector matrix F, μj is the mean of the j-th feature in matrix F, and σ j is the standard deviation of the j-th feature in matrix F;

[0100] Specifically, assume there are D leads, and the feature vector of each lead is fi. Then the shape of the comprehensive feature vector matrix F is (N, M). Among them, N is the number of features, and M is the dimension of the features.

[0101] For each feature dimension, the formula for calculating the mean is as follows:

[0102]

[0103] where μ j is the mean of the j-th feature, and F ij is the element in the i-th row and j-th column of matrix F.

[0104] The formula for calculating the standard deviation is as follows:

[0105]

[0106] Through the formula calculate to obtain the covariance matrix Σ of matrix X std ; where N is the number of features, and X T std is matrix X std 's transpose;

[0107] Perform eigenvalue decomposition on the covariance matrix Σ to find a diagonal matrix Λ and an eigenvector matrix V such that ΣV = VΛ;

[0108] According to the magnitudes of the eigenvalues, select the eigenvectors V corresponding to the first k largest eigenvalues of the eigenvector matrix V k ;

[0109] Through the formula X pca = V k * X std calculate to obtain the feature vector matrix X after dimensionality reduction pca ;

[0110] Through the formula X (n,t,d) pos = X (n,t,d) pca + PE (t,d) calculate to obtain the matrix X after adding position encoding (n,t,d) pos ; where X (n,t,d) pca is the element in matrix X pca , n is the batch size, t is the time step, d is the feature dimension, and PE (t,d)is an element in the position encoding matrix PE;

[0111] The matrix X after position encoding (n,t,d) pos After being processed by multiple layers of Transformer encoders, the final temporal features are output.

[0112] The training process of the temporal processing unit is described below:

[0113] Design the following prediction tasks:

[0114] Mask prediction: Randomly select a part of the temporal data points for masking to form the masked input data Xmasked. Use the temporal processing unit to process Xmasked to obtain the output feature zmasked. Use the mean squared error (MSE) as the loss function to calculate the difference between the masked part predicted by the module and the true value.

[0115] Future prediction: Mask the temporal data after a certain time point, and train the module to predict future temporal data points. For each sample, select a time point tf and mask all time steps after tf to form the input data Xfuture. Use the temporal processing unit to process Xfuture to obtain the output feature zfuture. Also use the mean squared error (MSE) as the loss function to calculate the difference between the future temporal data points predicted by the module and the true value. Combine the mask prediction and future prediction tasks to pre-train the temporal processing unit. Optimize the loss function through the gradient descent and backpropagation algorithms, and update the model parameters of the temporal processing unit so that it can effectively capture the features of the temporal data and generate the initial model parameters.

[0116] In this embodiment, llama3.1-7B is selected as the text processing unit. This model is a decode-only architecture and can efficiently convert the quality control rule text and medical record text data into high-quality vector representations. Use VisionTransformer (ViT) as the image processing unit to process medical image data. ViT can effectively capture the detailed information in medical images. Use Time Series Transformer as the temporal processing unit to process temporal data such as electrocardiograms and electroencephalograms. Time Series Transformer can effectively capture the long-term dependencies in the temporal data while maintaining sensitivity to local features, and can better understand the changes in the patient's physiological state.

[0117] Step S150: Calculate the second fusion similarity matrix between the temporal data as the second query sequence and the fusion output feature as the second key sequence and the second value sequence;

[0118] Specifically, calculating a second fusion similarity matrix between the time-series data as the second query sequence, and the fused output features as the second key sequence and the second value sequence includes:

[0119] Calculating a second fusion similarity matrix between the time-series features as the second query sequence, and the fused output features as the second key sequence and the second value sequence.

[0120] Step S160: Assigning a second weight to each query element in the second query sequence according to the second fusion similarity matrix;

[0121] Step S170: Performing weighted summation on the second value sequence according to the second weight, and outputting a medical record quality control conclusion.

[0122] Specifically, the embodiment of the present invention designs an alignment scheme for image-text features and time-series-text features, as follows:

[0123] Image-Text Feature Alignment Projector: The design goal of this projector is to unify the feature spaces of medical image data and medical record text data spliced with quality control rules. By adopting a fully connected layer and processing it through a non-linear activation function (such as ReLU or GELU), the mapping relationship between image features and text features is learned. This projector can map the image features extracted by Vision Transformer (ViT) to the same space as the text features generated by the llama3.1-7B text processing unit, thereby achieving effective alignment of the two-modal data. Its output dimension is a pre-defined unified dimension. The weights of this projector are learned during the model training process, and it learns to map the features of the image modality to the semantic space of the text dimension. After passing through the projector, the feature vectors of all modalities have the same dimension. These feature vectors with unified dimensions can be used as the input of cross-modal attention. For example, when performing image-text cross-modal attention, the image feature vector and the text feature vector have the same dimension, and then the attention mechanism is calculated in the form of a direct dot product operation.

[0124] Temporal-Text Feature Alignment Projector: The design goal of this projector is to unify the feature spaces of time series data such as electrocardiograms and electroencephalograms with medical record text data. Similarly, a fully connected layer is adopted and processed through a non-linear activation function (such as ReLU or GELU). This projector can map the temporal data features extracted by the Time Series Transformer to the same space as the text features generated by the llama3.1-7B text processing unit, thus achieving effective alignment of temporal data and text data, and its output dimension is also a predefined unified dimension.

[0125] After the alignment of different modality data is completed, the fusion of data features in the three dimensions of text, image, and time series is achieved through 2 cascaded Cross-Attention layers. Specifically, after the image-text features and the temporal-text features are aligned through their respective projectors, the first-level Cross-Attention mechanism is first used to achieve in-depth interaction between the image features and the text features. This mechanism allows the model to refer to the corresponding medical image features when processing text information. Based on the first-level Cross-Attention, the second-level Cross-Attention interaction is performed between the time series data features and the fused features processed by the first-level Cross-Attention. This step further strengthens the fusion between the time series data and the text and image data, enabling the model to perform quality control analysis in a more comprehensive feature space.

[0126] Then, the quality control feedback module can generate a detailed quality control report based on the quality control results and feedback it to the relevant medical staff in a timely manner. The report includes the problems found, detailed descriptions of the problems, potential impacts, and optimization suggestions, helping the medical staff quickly identify and correct the errors in the medical records. In addition, the quality control feedback module can also provide visual result displays, such as charts and trend analyses, so that the medical staff can intuitively understand the data quality status and support the continuous improvement process. This feedback mechanism not only improves the data quality but also enhances the decision-making ability of the medical staff.

[0127] Specifically, the quality control feedback module processes the received analysis results, mainly including the following aspects:

[0128] 1. Store the quality control results in the database, and the results include patient information, doctor information, quality control result information, etc.

[0129] 2. The quality control feedback module classifies problems into logical problems, numerical consistency problems, etc. according to the type of quality control results. The quantity and distribution of each type of problem are counted separately. Obtain month-on-month data and year-on-year data, and analyze the changing trend of medical record quality in different time periods. Specifically, it includes:

[0130] Month-on-month analysis: Compare the changes in the quantity and type of quality control problems between the current time period and the previous time period (such as last month).

[0131] Year-on-year analysis: Compare the changes in the quantity and type of quality control problems between the current time period and the same period last year.

[0132] And generate visual charts, including bar charts, line charts, pie charts, etc., to display the statistical data and changing trend of medical record quality. The charts have the following functions:

[0133] Interactivity: Allow users to click on the chart to view detailed information, support data filtering and custom time range selection.

[0134] Real-time update: The chart data can be updated in real time to ensure that doctors can view the latest quality control results and statistical data in a timely manner.

[0135] Push the quality control results to the quality control system App and web page through the API interface. The API interface includes the following functions:

[0136] Data push: Push the quality control result data to the quality control system App and web page, including patient information, doctor information, and quality control result information.

[0137] Data update: Support the update operation of quality control results to ensure that the data on the App and web page is consistent with the data in the database.

[0138] The quality control feedback module can identify persistent problems by comparing historical quality control data. The historical data should support multi-dimensional analysis, such as by time, problem type, department, etc.

[0139] As Figure 3 shown, the medical record quality control system based on the multi-modal fusion large model provided by the embodiment of the present invention includes:

[0140] A text splicing module 100 for splicing quality control rules with text data to obtain spliced text data;

[0141] A first fusion similarity matrix calculation module 200 for inputting image data, spliced text data, and time series data into a pre-constructed multi-modal fusion large model, and calculating a first fusion similarity matrix between the image data as the first query sequence, the spliced text data as the first key sequence, and the first value sequence;

[0142] Specifically, the first fusion similarity matrix calculation module 200 is specifically configured to input image data, spliced text data, and time series data into a pre-constructed multi-modal fusion large model, and through the formula S ij = Q i * K j T calculate the first fusion similarity matrix S ij ; where Q i is the query element in the first query sequence, and K j T is the transpose of each key element K j in the first key sequence;

[0143] The first weight assignment module 300 is used to assign the first weight to each query element in the first query sequence according to the first fusion similarity matrix;

[0144] Specifically, the first weight assignment module 300 includes:

[0145] The similarity score calculation unit is used to calculate the similarity score S' through the formula ; where d ij is a preset scaling factor; k

[0146] The first weight calculation unit is used to normalize the similarity score S' through the formula to obtain the attention weight α ij ; ij

[0147] The fusion feature output module 400 is used to perform weighted summation on the first value sequence according to the first weight to obtain the fusion output feature;

[0148] Specifically, the fusion feature output module 400 is specifically configured to perform weighted summation through the formula to obtain the fusion output feature o i ; where V j is each value element in the first value sequence.

[0149] The second fusion similarity matrix calculation module 500 is used to calculate the second fusion similarity matrix between the time series data as the second query sequence, the fusion output feature as the second key sequence, and the second value sequence;

[0150] Furthermore, the first fusion similarity matrix calculation module 200 includes:

[0151] The multi-modal data input unit is used to input image data, spliced text data, and time series data into the multi-modal fusion large model;

[0152] ​​An image processing unit for extracting information from image data to obtain image features;

[0153] A text processing unit for converting the spliced text data into text vector features;

[0154] A time series processing unit for extracting features from time series data to obtain time series features;

[0155] A first fusion similarity matrix calculation unit for calculating a first fusion similarity matrix S through the formula S ij =Q i *K j T ; where Q ij is the query element in the first query sequence, and K i is the transpose of each key element K j T in the first key sequence; j norm

[0156] In this embodiment, the time series processing unit includes:

[0157] A time series data normalization subunit for normalizing time series data to obtain normalized time series data; specifically, normalizing through the formula to obtain normalized time series data X norm ; where X is the original time series data, μ is the mean of the original time series data, and σ is the standard deviation of the original time series data.

[0158] A data fitting subunit for fitting the normalized time series data to obtain a fitted time series feature vector; specifically, using the ARMA (AutoRegressive Moving Average) algorithm to fit the time series data to obtain a fitted time series feature vector. Assume that X is the input time series data with shape (T, D), where T and D are the number of time steps and the feature dimension respectively. Use the Akaike Information Criterion (AIC) to select the appropriate orders p and q of the ARMA model. In this embodiment, the search range of p and q is set from 0 to 5, and all p and q are traversed to select the combination with the minimum AIC value as the optimal ARMA model order. After fitting the ARMA model, the parameters of the model are used as features. For the ARMA(p,q) model, it includes autoregressive coefficients φ1, φ2, …, φp, moving average coefficients θ1, θ2, …, θq. These parameters reflect the dynamics of the time series, and the parameters of the ARMA model and the residual sequence are output as a feature vector.

[0159] The eigenvector merging subunit is used to merge the fitted time-series eigenvectors to obtain a comprehensive eigenvector matrix. Specifically, the time-series data of each lead is normalized separately, and then the ARMA model is applied to fit the time-series data of each lead to obtain the eigenvector of each lead. The eigenvectors of all leads are merged to obtain a comprehensive eigenvector matrix. Suppose there are D leads, and the eigenvector of each lead is fi, then the comprehensive eigenvector matrix F can be expressed as F = [f1, f2, …, fD]. Where fi is the eigenvector of the i-th lead.

[0160] Since different time-series data have different optimal p and q parameters of the ARMA model, the number of eigenvectors of each lead is also different. Therefore, it is necessary to reduce the dimension of each eigenvector in the comprehensive eigenvector matrix F to unify the dimension of the eigenvectors. Specifically, the time-series processing unit further includes:

[0161] The comprehensive eigenvector matrix normalization subunit is used to normalize the comprehensive eigenvector matrix F through the formula to obtain the normalized matrix X std ; where X stdij is the element in the i-th row and j-th column of matrix X std , F ij is the element in the i-th row and j-th column of the comprehensive eigenvector matrix F, μ j is the mean of the j-th feature in matrix F, and σ j is the standard deviation of the j-th feature in matrix F;

[0162] The covariance matrix calculation subunit is used to calculate the covariance matrix Σ of matrix X through the formula ; where N is the number of features, and X std is the transpose of matrix X T std ; std The eigenvalue decomposition subunit is used to perform eigenvalue decomposition on the covariance matrix Σ to find a diagonal matrix Λ and an eigenvector matrix V such that ΣV = VΛ;

[0163] The eigenvector acquisition subunit is used to select the eigenvectors V corresponding to the first k largest eigenvalues of the eigenvector matrix V according to the magnitude of the eigenvalues

[0164] ; k ;

[0165] The eigenvector matrix dimension reduction subunit is used to calculate the dimension-reduced eigenvector matrix X through the formula X pca = V k * X std ; pca ;

[0166] Matrix position encoding subunit, used to pass the formula X (n,t,d) pos =X (n,t,d) pca +PE (t,d) Calculate the position-encoded matrix X (n,t,d) pos ; Among them, X (n,t,d) pca is the matrix X pca The elements in, n is the batch size, t is the number of time steps, d is the feature dimension, PE (t,d) is an element in the position encoding matrix PE;

[0167] The final temporal feature output subunit is used to convert the position-encoded matrix X (n,t,d) pos After being processed by multiple layers of Transformer encoders, the final temporal features are output.

[0168] In this case, the second fusion similarity matrix calculation module 500 is specifically used to calculate the second fusion similarity matrix between the time series features as the second query sequence, the fusion output features as the second key sequence and the second value sequence.

[0169] The following describes the training process of the timing processing unit:

[0170] Design the following prediction task:

[0171] Mask prediction: Randomly select a portion of the time series data points to mask, forming the masked input data Xmasked. The time series processing unit processes Xmasked to obtain the output feature zmasked. The mean squared error (MSE) is used as the loss function to calculate the difference between the masked portion predicted by the module and the true value.

[0172] Future prediction: Mask the time series data after a certain time point, and train the module to predict future time series data points. For each sample, select a time point tf, mask all time steps after tf, and form the input number Xfuture. Use the time series processing unit to process Xfuture to obtain the output feature zfuture. The mean square error (MSE) is also used as the loss function to calculate the difference between the future time series data points predicted by the module and the true value. Combining the mask prediction and future prediction tasks, the time series processing unit is pre-trained. The loss function is optimized through the gradient descent and back propagation algorithms, and the model parameters of the time series processing unit are updated so that it can effectively capture the characteristics of the time series data and generate preliminary model parameters.

[0173] In this embodiment, llama3.1-7B is selected as the text processing unit. This model has a decode-only architecture and can efficiently convert quality control rule text and medical record text data into high-quality vector representations. VisionTransformer (ViT) is used as the image processing unit to process medical image data. ViT can effectively capture the detailed information in medical images. Time Series Transformer is adopted as the time series processing unit to process time series data such as electrocardiograms and electroencephalograms. Time Series Transformer can effectively capture the long-term dependencies in time series data while maintaining sensitivity to local features, and can better understand the changes in the patient's physiological state.

[0174] The second weight assignment module 600 is used to assign a second weight to each query element in the second query sequence according to the second fusion similarity matrix;

[0175] The medical record quality control module 700 is used to perform weighted summation on the second value sequence according to the second weight and output the medical record quality control conclusion.

[0176] Specifically, the embodiment of the present invention designs an alignment scheme for image-text features and time series-text features, which is as follows:

[0177] Image-text feature alignment projection layer (Projector): The design goal of this projection layer (Projector) is to unify the feature spaces of medical image data and medical record text data spliced with quality control rules. By adopting a fully connected layer and processing it through a non-linear activation function (such as ReLU or GELU), the mapping relationship between image features and text features is learned. This projection layer (Projector) can map the image features extracted by Vision Transformer (ViT) to the same space as the text features generated by the llama3.1-7B text processing unit, thereby realizing the effective alignment of the two-modal data. Its output dimension is a predefined unified dimension. The weight of this projection layer is learned during the model training process, and it learns to map the features of the image modality to the semantic space of the text dimension. After passing through the projection layer, the feature vectors of all modalities have the same dimension. These feature vectors with unified dimensions can be used as the input of cross-modal attention. For example, when performing image-text cross-modal attention, the image feature vector and the text feature vector have the same dimension, and then the attention mechanism is calculated in the form of a direct dot product operation.

[0178] Temporal-Text Feature Alignment Projector: The design goal of this projector is to unify the feature spaces of time series data such as electrocardiograms and electroencephalograms with medical record text data. Similarly, a fully connected layer is adopted and processed through a non-linear activation function (such as ReLU or GELU). This projector can map the temporal data features extracted by Time Series Transformer to the same space as the text features generated by the llama3.1-7B text processing unit, thus achieving effective alignment between temporal data and text data, and its output dimension is also a predefined unified dimension.

[0179] After aligning different modality data, the fusion of text, image, and temporal dimension data features is achieved through 2 cascaded Cross-Attention layers. Specifically, after the image-text features and temporal-text features are aligned through their respective projectors, the first-level Cross-Attention mechanism is first used to achieve in-depth interaction between image features and text features. This mechanism allows the model to refer to the corresponding medical image features when processing text information. Based on the first-level Cross-Attention, the second-level Cross-Attention interaction is carried out between the temporal data features and the fused features processed by the first-level Cross-Attention. This step further strengthens the fusion between temporal data and text and image data, enabling the model to perform quality control analysis in a more comprehensive feature space.

[0180] Then, the quality control feedback module can generate a detailed quality control report based on the quality control results and feedback it to relevant medical staff in a timely manner. The report includes the problems found, detailed descriptions of the problems, potential impacts, and optimization suggestions, helping medical staff quickly identify and correct errors in medical records. In addition, the quality control feedback module can also provide visual result displays, such as charts and trend analyses, so that medical staff can intuitively understand the data quality status and support the continuous improvement process. This feedback mechanism not only improves data quality but also enhances the decision-making ability of medical staff.

[0181] Specifically, the quality control feedback module processes the received analysis results, mainly including the following aspects:

[0182] 1. Store the quality control results in the database, and the results include patient information, doctor information, quality control result information, etc.

[0183] 2. Based on the types of quality control results, the quality control feedback module classifies problems into logical problems, numerical consistency problems, etc. The quantity and distribution of each type of problem are respectively counted. Obtain month-on-month data and year-on-year data, and analyze the changing trends of medical record quality in different time periods. Specifically include:

[0184] Month-on-month analysis: Compare the changes in the quantity and types of quality control problems between the current time period and the previous time period (such as last month).

[0185] Year-on-year analysis: Compare the changes in the quantity and types of quality control problems between the current time period and the same period last year.

[0186] And generate visual charts, including bar charts, line charts, pie charts, etc., to display the statistical data and changing trends of medical record quality. The charts have the following functions:

[0187] Interactivity: Allow users to click on the chart to view detailed information, support data filtering and custom time range selection.

[0188] Real-time update: The chart data can be updated in real time to ensure that doctors can view the latest quality control results and statistical data in a timely manner.

[0189] Push the quality control results to the quality control system App and web page through the API interface. The API interface includes the following functions:

[0190] Data push: Push the quality control result data to the quality control system App and web page, including patient information, doctor information, and quality control result information.

[0191] Data update: Support the update operation of quality control results to ensure that the data on the App and web page is consistent with the data in the database.

[0192] The quality control feedback module can identify persistent problems by comparing historical quality control data. The historical data should support multi-dimensional analysis, such as by time, problem type, department, etc.

[0193] In summary, through the integration of multiple medical data sources (such as images, texts, laboratory results, electrocardiograms, electroencephalograms, etc.) and the use of advanced artificial intelligence technology, the embodiments of the present invention achieve in-depth quality control of medical records, improve the accuracy and consistency of medical documents, and thus improve the quality of medical services.

[0194] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0195] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0196] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0197] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0198] Where the embodiments of the present invention are not described in detail, they are all well-known technologies to those skilled in the art. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A medical record quality control method based on a multi-modal fusion large model, characterized in that, Including: Concatenate the quality control rules with the text data to obtain the concatenated text data; Input the image data, the concatenated text data, and the time series data into a pre-constructed multi-modal fusion large model, and calculate the first fusion similarity matrix between the image data as the first query sequence, the concatenated text data as the first key sequence and the first value sequence; According to the first fusion similarity matrix, assign a first weight to each query element in the first query sequence; Perform weighted summation on the first value sequence according to the first weight to obtain the fusion output feature; Calculate the second fusion similarity matrix between the time series data as the second query sequence, the fusion output feature as the second key sequence and the second value sequence; According to the second fusion similarity matrix, assign a second weight to each query element in the second query sequence; Perform weighted summation on the second value sequence according to the second weight, and output the medical record quality control conclusion.

2. The medical record quality control method based on the multi-modal fusion large model according to claim 1, wherein, The calculation of the first fusion similarity matrix between the image data as the first query sequence, the concatenated text data as the first key sequence and the first value sequence; according to the first fusion similarity matrix, assign a first weight to each query element in the first query sequence; Performing weighted summation on the first value sequence according to the first weight to obtain the fusion output feature, including: Calculate the first fusion similarity matrix S ij through the formula S i =Q j T *K ij ; where Q i is the query element in the first query sequence, and K j T is the transpose of each key element K j in the first key sequence. The similarity score S’ is calculated through the formula ; where d ij is a preset scaling factor; k ​ Through the formula normalize the similarity score S’ ij to obtain the attention weight α ij ; Through the formula perform weighted summation to obtain the fused output feature o i ; where, V j is each value element in the first value sequence.

3. The medical record quality control method based on the multi-modal fusion large model according to claim 1 or 2, characterized in that The input of the image data, the concatenated text data, and the time series data into a pre-constructed multi-modal fusion large model, and the calculation of the first fusion similarity matrix between the image data as the first query sequence, the concatenated text data as the first key sequence and the first value sequence, including: Input the image data, the concatenated text data, and the time series data into the multi-modal fusion large model. The image processing unit in the multi-modal fusion large model extracts information from the image data to obtain image features. The text processing unit in the multi-modal fusion large model converts the concatenated text data into text vector features. The time series processing unit in the multi-modal fusion large model extracts features from the time series data to obtain time series features; Calculate the first fusion similarity matrix between the image features as the first query sequence, the text vector features as the first key sequence and the first value sequence; The calculation of the second fusion similarity matrix between the time series data as the second query sequence, the fusion output feature as the second key sequence and the second value sequence, including: Calculate the second fusion similarity matrix between the time series features as the second query sequence, the fusion output feature as the second key sequence and the second value sequence.

4. The medical record quality control method based on the multi-modal fusion large model according to claim 3, characterized in that The extraction of features from the time series data by the time series processing unit in the multi-modal fusion large model to obtain time series features, including: Perform standardization processing on the time series data to obtain the standardized time series data; Perform fitting on the standardized time series data to obtain the fitted time series feature vector; Merge the fitted time series feature vectors to obtain the comprehensive feature vector matrix.

5. The medical record quality control method based on the multi-modal fusion large model according to claim 4, wherein After obtaining the comprehensive feature vector matrix, the following steps are also included: Normalize the comprehensive feature vector matrix F through the formula to obtain the normalized matrix X std ; where is the element in the i-th row and j-th column of the matrix X std , F ij is the element in the i-th row and j-th column of the comprehensive feature vector matrix F, μ j is the mean of the j-th feature in the matrix F, and σ j is the standard deviation of the j-th feature in the matrix F; The covariance matrix Σ of matrix X is calculated through the formula ; where N is the number of features, and X std is the transpose of the matrix X T std ; std ​ Perform eigenvalue decomposition on the covariance matrix Σ to find a diagonal matrix Λ and an eigenvector matrix V such that ΣV = VΛ; Select the eigenvectors \(V\) corresponding to the top \(k\) largest eigenvalues of the eigenvector matrix \(V\) according to the magnitudes of the eigenvalues. k ; Through formula X pca =V k *X std The dimensionality-reduced feature vector matrix X is calculated pca ; Through formula X (n,t,d) pos = X (n,t,d) pca + PE (t,d) The matrix X after position encoding is calculated (n,t,d) pos ; where, X (n ,t,d) pca is the element in the matrix X pca n is the batch size, t is the time step, d is the feature dimension, and PE (t,d) is the element in the position encoding matrix PE; The matrix X encoded with the position (n,t,d) pos After being processed by multiple layers of Transformer encoders, the final temporal features are output.

6. A medical record quality control system based on a multi-modal fusion large model, characterized in that, It includes: A text splicing module for splicing quality control rules with text data to obtain spliced text data; A first fusion similarity matrix calculation module for inputting image data, the spliced text data, and time series data into a pre-constructed multi-modal fusion large model to calculate a first fusion similarity matrix between the image data as the first query sequence, the spliced text data as the first key sequence and the first value sequence; A first weight assignment module for assigning a first weight to each query element in the first query sequence according to the first fusion similarity matrix; A fusion feature output module for performing weighted summation on the first value sequence according to the first weight to obtain a fusion output feature; A second fusion similarity matrix calculation module for calculating a second fusion similarity matrix between the time series data as the second query sequence, the fusion output feature as the second key sequence and the second value sequence; A second weight assignment module for assigning a second weight to each query element in the second query sequence according to the second fusion similarity matrix; A medical record quality control module for performing weighted summation on the second value sequence according to the second weight and outputting a medical record quality control conclusion.

7. The medical record quality control system based on the multi-modal fusion large model according to claim 6, wherein, The first fusion similarity matrix calculation module is specifically configured to input the image data, the spliced text data, and the time series data into a pre-constructed multi-modal fusion large model, and through the formula S ij =Q i *K j T calculate to obtain the first fusion similarity matrix S ij ; where Q i is the query element in the first query sequence, and K j T is the transpose of each key element K j in the first key sequence. The first weight assignment module includes: A similarity score calculation unit for calculating a similarity score S' through the formula ; where d ij is a preset scaling factor; k ​ The first weight calculation unit is used to normalize the similarity score S' through the formula to obtain the attention weight α ij ; ij ; The fusion feature output module is specifically configured to perform weighted summation through the formula to obtain the fused output feature o i ; where V j is each value element in the first value sequence.

8. The medical record quality control system based on the multi-modal fusion large model according to claim 7, characterized in that The first fusion similarity matrix calculation module includes: A multi-modal data input unit for inputting the image data, the spliced text data, and the time series data into the multi-modal fusion large model; An image processing unit for extracting information from the image data to obtain image features; A text processing unit for converting the spliced text data into text vector features; A time series processing unit for extracting features from the time series data to obtain time series features; The first fusion similarity matrix calculation unit is used to calculate the first fusion similarity matrix S through the formula S ij = Q i * K j T ; where Q ij is the query element in the first query sequence, and K i is the transpose of each key element K j T in the first key sequence; j ​ The second fusion similarity matrix calculation module is specifically used to calculate a second fusion similarity matrix between the time series features as the second query sequence, the fusion output feature as the second key sequence and the second value sequence.

9. The medical record quality control system based on the multi-modal fusion large model according to claim 8, characterized in that, The time series processing unit includes: A time series data normalization sub-unit for normalizing the time series data to obtain normalized time series data; A data fitting sub-unit for fitting the normalized time series data to obtain a fitted time series feature vector; A feature vector merging sub-unit for merging the fitted time series feature vectors to obtain a comprehensive feature vector matrix.

10. The medical record quality control system based on the multi-modal fusion large model according to claim 9, wherein, The time series processing unit also includes: Comprehensive feature vector matrix standardization subunit, which is used to standardize the comprehensive feature vector matrix F through the formula to obtain the standardized matrix X std ; where is the element in the i-th row and j-th column of the matrix X std , F ij is the element in the i-th row and j-th column of the comprehensive feature vector matrix F, μ j is the mean of the j-th feature in the matrix F, and σ j is the standard deviation of the j-th feature in the matrix F; A covariance matrix calculation subunit, which is used to calculate the covariance matrix Σ of matrix X through the formula where N is the number of features, and X std is the transpose of the matrix X T std ; std ​ An eigenvalue decomposition sub-unit for performing eigenvalue decomposition on the covariance matrix Σ to find a diagonal matrix Λ and an eigenvector matrix V such that ΣV = VΛ; An eigenvector obtaining subunit, configured to select, according to the magnitudes of the eigenvalues, the eigenvectors V corresponding to the top k largest eigenvalues of the eigenvector matrix V k ; The eigenvector matrix dimensionality reduction subunit is used to calculate the dimensionality-reduced eigenvector matrix X by the formula X pca = V k * X std ; pca ​ Matrix position encoding subunit, which is used to calculate the matrix X after position encoding through the formula X (n,t,d) pos = X (n,t,d) pca + PE (t,d) ; where X (n,t,d) pos is the element in the matrix X (n,t,d) pca , n is the batch size, t is the time step, d is the feature dimension, and PE pca (t,d) is the element in the position encoding matrix PE;​ The final temporal feature output subunit is used to output the final temporal feature after processing the matrix X encoded by the position through multiple Transformer encoders. (n,t,d) pos After being processed by multiple Transformer encoders, the final temporal feature is output.

Images