Multi-modal and large-model quality control system based on thoracic pain stroke wound multi-center data

Abstract

The invention discloses a multi-modal and large-model quality control system based on multi-center data of chest pain, stroke and trauma. The quality control system comprises the following steps: collecting and preprocessing multi-modal data of three major centers of chest pain, stroke and trauma; for different types of quality control multi-modal data of the three major centers, performing feature extraction and feature alignment on the different types of quality control multi-modal data by using a multi-modal data encoder; according to cross-modal self-attention in a multi-modal data encoder, carrying out multi-modal feature fusion on the aligned multi-modal features; uniformly mapping and modeling the fused feature vectors to generate a quality control comprehensive model of the three major centers, and outputting quality control indexes and abnormal points of the three major centers through the quality control comprehensive model; and performing multi-dimensional display comparison on an output result of the quality control comprehensive model. According to the invention, quality control data of three major centers of chest pain, stroke and trauma are fused and subjected to modeling analysis, so that the quality control management is more accurate and efficient.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a multi-modal and large-model quality control system based on multi-center data of chest pain, stroke and trauma. Background Art

[0002] With the development of medical technology and the continuous improvement of medical quality control management, the three major centers of chest pain, stroke, and trauma play a vital role in the treatment of patients. However, due to the diverse sources of quality control data of the three major centers, including electronic medical records, images, time series data, etc., and differences in processes and quality control standards, there is currently a lack of a unified modeling method that can efficiently integrate multimodal data and support cross-center quality control analysis. In view of this, the present invention proposes a unified modeling method for quality control data fusion and large models for the three major centers of chest pain, stroke, and trauma, aiming to provide accurate and efficient solutions for quality control management through multimodal data fusion and unified modeling technology. Summary of the invention

[0003] The present invention provides a multi-modal and large-model quality control system based on multi-center data of chest pain, stroke and trauma, including: Acquisition module: used to collect and pre-process different types of quality control multimodal data from three major centers: chest pain, stroke, and trauma; Multimodal data encoder module: used for extracting and aligning features of different types of quality control multimodal data from the three major centers using multimodal data encoders; Modeling and analysis module: used to perform multimodal feature fusion on the aligned multimodal features according to the cross-modal self-attention in the multimodal data encoder, unify the fused feature vectors for mapping and modeling, generate a comprehensive quality control model for the three major centers, and output the quality control indicators and abnormal points of the three major centers through the comprehensive quality control model; Generation module: used to display and compare the quality control indicators and abnormal points of the three major centers output by the quality control comprehensive model in multiple dimensions.

[0004] As described above, a multimodal and large-model quality control system based on multi-center data of chest pain, stroke and trauma is constructed, in which a multimodal data encoder is designed with dedicated encoders for different types of data, including a text data encoder, an image data encoder, a time series data encoder, and a structured data encoder.

[0005] As described above, in the multimodal and large-model quality control system based on multi-center data of chest pain, stroke and trauma, the vector dimensions of different modalities need to be aligned after the features are extracted by the encoder, and the numerical features are adjusted to the same range using the normalization method; the feature mapping network is used to reduce or increase the dimension of each modality feature so that different modalities have a unified vector dimension.

[0006] As described above, a multimodal and large-model quality control system based on multi-center data of chest pain, stroke and trauma is used, in which a multimodal data encoder is used to perform multimodal feature fusion after aligning the vector dimensions of different modalities, and the multimodal data encoder uses cross-modal self-attention for multimodal feature fusion.

[0007] As described above, the multimodal and large-model quality control system based on multi-center data of chest pain, stroke and trauma, in which the fused feature vector is mapped to a unified semantic space, includes the following steps: mapping the fused features into a vector representation of a fixed dimension, compressing high-dimensional features into low-dimensional features, and fine-tuning the feature vector according to quality control analysis.

[0008] As described above, the multimodal and large-model quality control system based on multi-center data of chest pain, stroke and trauma, wherein the comprehensive quality control model includes extracting common features from the mapped fusion vector; building dedicated layers for chest pain, stroke and trauma centers respectively; for each center, the model outputs key quality control indicators according to different quality control standards; and inputting the mapped fusion vector into the clustering algorithm to discover data patterns and anomalies.

[0009] As described above, the multimodal and large-model quality control system based on multi-center data of chest pain, stroke and trauma, wherein the multi-dimensional display of the model quality control results is to combine the model quality control results with the existing quality control report text, input the model generation module to automatically generate a text quality control report, use a multi-dimensional visualization tool to uniformly compare and display the quality control data reports of the three major centers, and output a multi-dimensional visualization result.

[0010] The present invention provides a multi-modal and large-model quality control method based on multi-center data of chest pain, stroke and trauma, including: S10, collect different types of quality control multimodal data from three major centers of chest pain, stroke, and trauma and perform preprocessing; S20. For different types of quality control multimodal data from the three major centers, use multimodal data encoders to extract features and align features for different types of quality control multimodal data; S30, performing multimodal feature fusion on the aligned multimodal features according to the cross-modal self-attention in the multimodal data encoder; S40, uniformly mapping and modeling the fused feature vectors, generating a comprehensive quality control model for the three major centers, and outputting the quality control indicators and abnormal points of the three major centers through the comprehensive quality control model; S50. Display and compare the quality control indicators and abnormal points of the three major centers output by the comprehensive quality control model in multiple dimensions.

[0011] The beneficial effects achieved by the present invention are as follows: the present invention integrates the quality control data of three major centers of chest pain, stroke and trauma and performs modeling analysis to make quality control management more accurate and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0013] Figure 1 This is a flow chart of the multimodal and large-model quality control method based on multi-center data of chest pain, stroke and trauma provided in Example 1 of the present application.

[0014] Figure 2 This is a schematic diagram of a multimodal and large-model quality control system based on multi-center data of chest pain, stroke and trauma provided in Example 2 of the present application. DETAILED DESCRIPTION

[0015] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0016] Embodiment 1 like Figure 1 As shown, Example 1 of the present application provides a multi-modal and large-model quality control method based on multi-center data of chest pain, stroke and trauma, including: S10, collect different types of quality control multimodal data from three major centers of chest pain, stroke, and trauma and perform preprocessing; The quality control data of the three major centers of chest pain, stroke and trauma are extracted from the electronic medical record system, image archiving and communication system, monitoring equipment and laboratory information system. Including: text data, medical records, doctor's diagnosis records; image data, computer tomography, magnetic resonance imaging, X-ray and other images; time series data, monitor ECG data, blood pressure change curve, electroencephalogram, etc.; laboratory data, blood test reports, biochemical indicators, etc.

[0017] It also integrates multimodal data from the three major centers, and uses a data preprocessing module to standardize and clean data in different formats, removing special symbols, redundant punctuation, garbled characters, extra spaces, etc. in the text, retaining only content that has a substantial effect on semantic expression.

[0018] S20. For different types of quality control multimodal data from the three major centers, use multimodal data encoders to extract features and align features for different types of quality control multimodal data; For different types of data, dedicated encoders are designed to extract features of each modality: S21, text data encoder.

[0019] The text data encoder is used to extract features from text data. The text data encoder uses natural language processing technology to convert unstructured text data such as medical records and doctor's diagnosis records into context-related vector representations. The text is semantically parsed to extract key information.

[0020] The input text is divided into words, and the results of word segmentation are converted into input tag sequences that can be recognized by the text data encoder by adding special tags [CLS] and [SEP]. Using the word embedding matrix, each word is converted into a word embedding vector so that each word can be mapped to the pre-trained high-dimensional vector space. The specific formula is: , It's a word The word embedding vector of is the word embedding matrix, size is , is the vocabulary size, is the embedding dimension, is the i-th word in the word sequence that needs to be converted, and the converted word embedding vector sequence is , p is the number of word embedding vectors. Position encoding is added to the vectors in the word embedding vector sequence to provide the position information of the word in the sequence.

[0021] Thus, we get the input word vector sequence A. ,in, is the final input vector of the ith word, It's a word The word embedding vector of It's a word The position encoding of , pos is the position of the word vector in the sequence, and a is the dimension index of the embedding vector.

[0022] The input word vector sequence is input into the text data encoder. The text data encoder contains multiple encoder layers. Each layer contains a multi-head attention mechanism and a feedforward neural network. The input text features are continuously extracted and fused through multiple encoder layers to obtain a text vector representation that incorporates rich contextual information.

[0023] The multi-head attention mechanism contains a heads. For each head, for the input word vector sequence, given the query matrix Q, key matrix K and value matrix V, the importance of each value is determined by calculating the similarity between the query and the key, and a new representation is generated. This new representation contains context information. The specific formula is ,in, Compute the dot product between the query and the key, which measures the similarity between each query and all other keys. Divide by the scale factor The dot product result is scaled to prevent the gradient from vanishing or exploding. The softmax function is applied to convert the dot product result into a probability distribution, i.e., the attention weights. The value matrix V is weighted summed using these weights to obtain the final output vector.

[0024] Concatenate the results of a heads and pass an additional linear transformation To integrate information, the integration formula is ,in is the final output weight matrix, a is the number of heads, represents the dimension of each value vector, and d is the embedding dimension. This results in an output vector of the encoder layer.

[0025] The feedforward neural network is a fully connected layer used to connect the encoder layer. The specific formula is , where A is the input vector, is the weight matrix of this layer, d is the encoder dimension, is the hidden layer dimension of the feedforward neural network, is the bias term of this layer. is the weight matrix of the next layer, is the bias term for the next layer.

[0026] After extraction and fusion of multiple encoder layers, a text vector representation that incorporates rich contextual information is obtained, and the output vector of the last encoder layer is used as the final feature vector.

[0027] S22, image data encoder.

[0028] For image data, an image data encoder is used to extract spatial features and semantic information to generate a high-dimensional image feature vector.

[0029] S221. Extracting two-dimensional image feature vectors.

[0030] Normalize the medical image data to a specific range and set the original pixel value to , the normalized pixel value is , the calculation formula is , is the minimum value of the original data, is the maximum value of the original data.

[0031] According to the encoder input size requirements, the image is cropped to the appropriate size and can be appropriately scaled to ensure uniform image size. The data set is expanded through operations such as rotation, flipping, translation, and adding noise to improve generalization capabilities.

[0032] Apply the filter in the image data encoder to convolve the input image, and perform convolution operations by sliding the convolution kernel on the image to extract local features. The formula is , where I is the input image, K is the convolution kernel, (i, j) is the coordinate of the current pixel position, and the result of the convolution operation is is the output value at (i, j), M is half the height of the convolution kernel, and N is half the width. (m, n) is the offset of the convolution kernel relative to the current position (i, j).

[0033] Adding an activation function after the convolutional layer increases nonlinear characteristics, and reducing the size of the feature map through maximum pooling to reduce computational complexity. The maximum pooling formula is , I is the input feature map, the shape is (C, H, W), C is the number of channels, H is the height, and W is the width. Represents the value of the input feature map at position (i, j), where i and j represent the coordinates of the pixel position currently being processed. In a maximum pooling operation, for the input feature map I, the maximum value is found in each pooling window, and this maximum value is used as the value of the output feature map at the corresponding position.

[0034] The feature maps after multiple convolution and pooling operations are combined and connected into a global feature map, flattened into a one-dimensional vector and connected to a fully connected layer for further feature integration. Suppose the dimension of the input flattened vector b is , the weight matrix of the fully connected layer is , the dimension is , the bias is , dimension , the output vector The calculation formula is , thus obtaining the feature vector of the two-dimensional image data.

[0035] S222. 3D data feature vector extraction.

[0036] For 3D data, use normalization and rotation, cropping, etc. to increase data diversity, and ensure that all input volumes have the same resolution or size.

[0037] Apply the 3D filter in the image data encoder to convolve the input volume and extract local features. The 3D convolution kernel slides in three dimensions, taking into account the information of the X, Y and Z axes at the same time. The specific formula is , is the input 3D data volume, is a 3D convolution kernel, i, j, k represent the coordinates of the pixel position currently being processed, and M, N, P are the radii of the convolution kernel on the X, Y, and Z axes, respectively.

[0038] Adding an activation function after the convolutional layer increases nonlinear characteristics, and reducing the size of the feature map through maximum pooling to reduce computational complexity. The maximum pooling formula is , is the input feature map, i, j, k represent the coordinates of the pixel position currently processed.

[0039] After multiple convolution and pooling operations, the feature map is flattened into a one-dimensional vector and then connected to a fully connected layer to further integrate the features, thereby obtaining the feature vector of the 3D image data.

[0040] S23, time series data encoder.

[0041] Use the time series data encoder for time series data to analyze the continuous data output by the monitoring device Extract dynamic change features in time series.

[0042] Scale the time series data to a fixed range to speed up the training process. Use a sliding window to split the scaled time series data into multiple subsequences of fixed length, each of which is used as an input sample of the encoder. This helps capture local temporal dependencies. The formula for the subsequence after segmentation is , w is the window size, and t is the current time point.

[0043] Assume that the dimension of a subsequence is q, then the number of nodes in the input layer is q. For each subsequence of the input encoder, use the formula Decide which information should be discarded, among which, is the activation function, is the input weight matrix to the forget gate, Represents the input at the current moment, is the weight matrix from hidden state to forget gate, Represents the hidden state of the previous moment, carrying the information from the previous sequence processing. is the bias vector of the forget gate.

[0044] Using the formula Update some information of cell status, is the output of the input gate, is the activation function, is the candidate cell state, tanh is the hyperbolic tangent activation function, , , , are the corresponding weight matrices, Represents the input at the current moment, Indicates the hidden state at the previous moment, , is the bias vector.

[0045] Using the formula Update the cell state, the new cell state It is the part of the cell that is selectively retained from the previous state. and newly added filtered information Combination of. Represents the cell state at time step t-1 before the current time step.

[0046] Using the formula Decide what information should be output. represents the output value of the output gate at time step t, , is the corresponding weight matrix, is the bias vector, is the hidden state at the current moment, from which the feature vector of time series data is obtained.

[0047] S24, structured data encoder.

[0048] For structured information such as laboratory data and test results, a structured data encoder is used to extract the correlation between key indicators and generate feature representation.

[0049] Normalization and standardization are performed on structured data. Normalization scales the numerical features to a fixed range, and standardization converts the numerical features into a standard normal distribution with a mean of 0 and a standard deviation of 1.

[0050] Assume that the structured data has n different categories and the embedding dimension is , that is, to map each category to a dimensional vector, the parameter of the embedding layer is a The embedding matrix For a categorical variable value g, the index of its corresponding category in all categories is i, and the calculation formula of the vector s obtained after the embedding layer is , if there are both numerical indicators and categorical indicators, the categorical indicator vector processed by the embedding layer is concatenated with the numerical indicator data in a certain order to form a unified input vector, and the combined vector is input to the subsequent network layer to capture the correlation between key indicators through network learning. The vector output by the final output layer is the generated feature representation. Suppose the input vector is s, the dimension is m, and after a fully connected layer, the weight matrix is , dimension , the bias is , the dimension is k, then the output feature representation vector The calculation formula is , thus obtaining the feature vector of structured data.

[0051] S25. Align the feature vectors of different modalities.

[0052] Since the data features of different modalities have different dimensions and ranges, it is necessary to normalize and align the features, and adjust the numerical features to the same range through normalization. Suppose the data is normalized to the interval [a, b], and the minimum value of the original data is , the maximum value is , then the standardized data The calculation formula is , u represents the original data value.

[0053] A feature mapping network is used to reduce or increase the dimensionality of each modality feature so that different modalities have a unified vector dimension. The features of the modalities are aligned in the same semantic space. Multimodal features are aligned through a shared embedding space so that the medical record description and the abnormal image area have high similarity in the semantic space.

[0054] Assume the dimension of the input text feature vector is , the target uniform dimension is D. Using a multi-layer fully connected network as the feature mapping network, taking two layers as an example, the weight matrix of the first fully connected layer is , the dimension is , the bias is The specific formula is , Represents the original text vector. Usually, an activation function is added after each layer to increase the nonlinear characteristics. The output after the activation function is , the weight matrix of the second fully connected layer is , the dimension is , the bias is , then the final output vector The other modes are reduced or increased in dimension according to the above steps, so that all modes have a unified vector dimension.

[0055] S30, performing multimodal feature fusion on the aligned multimodal features according to the cross-modal self-attention in the multimodal data encoder; The multimodal data encoder uses cross-modal self-attention to perform multimodal feature fusion, using the text modality vector as the query (Q) and the image modality vector as the key value (K, V), and realizes multimodal feature fusion by calculating the attention weights.

[0056] Introducing a learnable parameter , which is used to dynamically adjust the relative importance between text and image modalities. The outputs of the two modes are weighted summed. The interaction weights between modalities can also be adaptively adjusted according to task requirements to strengthen the correlation between modalities.

[0057] The specific formula is , Y represents the final fusion feature vector, It is the output of the multi-head self-attention mechanism, Q represents the embedding representation from the text modality, and K, V represent the embedding representation from the image modality. represents the feature vector of the imaging modality, This part is the fusion feature calculated through the cross-modal self-attention mechanism, which emphasizes the interactive relationship between the text modality as the query and the image modality as the key value. This part retains the characteristics of the original imaging modality, ensuring that the imaging information is not completely ignored.

[0058] S40, uniformly mapping and modeling the fused feature vectors, generating a comprehensive quality control model for the three major centers, and outputting the quality control indicators and abnormal points of the three major centers through the comprehensive quality control model; S41. The fused feature vector is mapped into a unified semantic space.

[0059] Map the fused features into a vector representation of fixed dimension using a linear transformation matrix Map the fused feature Y to a fixed dimension The mapping formula is , is the mapped vector. is the linear transformation matrix, D is the dimension of the fusion feature, is the target embedding dimension.

[0060] Compress high-dimensional features into low-dimensional features to improve model efficiency. Build an autoencoder model, including the encoder and decoder , , , where H is the hidden layer representation, which captures the key features of the input data. is the reconstructed embedding vector, and are the parameters of the encoder and decoder respectively. These parameters are optimized through the training process so that the encoder can effectively extract useful information from the input data. By training the autoencoder, the reconstruction error is minimized. The specific formula is , G represents the loss function, represents the number of samples in the training set, represents the true embedding vector of the i-th sample, Represents the reconstructed embedding vector of the i-th sample. After training, the encoder part is used to extract low-dimensional features.

[0061] Fine-tune the feature vector based on quality control analysis to ensure semantic relevance and specific task performance. Add task-related loss functions based on specific tasks. The specific formula is , based on the total loss , fine-tune the parameters of the embedding layer and dimensionality reduction module, the formula is , is the reconstruction loss, is the task-specific loss, is the balancing factor.

[0062] S42. The mapped fusion vectors are uniformly modeled to generate a comprehensive quality control model for the three major centers.

[0063] S421. Extract common features from the mapped fusion vector, uniformly encode different modal data using shared weights, and extract common patterns across centers.

[0064] A fully connected network is used as the shared encoding layer, and the mapped feature vector is input , let the shared layer weight matrix be , dimension , the bias is , dimension , then the shared layer output This layer learns the common feature patterns of data from different modalities and centers through shared weights, and extracts common information across centers, including common disease severity representations, common features of basic diagnosis and treatment processes, etc.

[0065] S422. Add a dedicated layer after the shared layer, design specific modeling for the unique quality control needs of each center, and build dedicated layers for chest pain, stroke, and trauma centers respectively.

[0066] For the chest pain center dedicated layer, the weight matrix is , dimension , the bias is , bias dimension , the input is the shared layer output, then the output of this layer ,Based on the quality control needs of the chest pain center, this layer focuses on the time from chest pain onset to the first medical intervention, the correlation between electrocardiogram change characteristics and treatment effects, etc., and learns specific feature representations to output key quality control indicators related to the chest pain center.

[0067] The stroke center-specific layer, assuming the weight matrix is , dimension , the bias is , bias dimension , output This layer focuses on feature learning such as the time from stroke onset to thrombolysis or thrombectomy, and the relationship between changes in brain imaging lesions and neurological function recovery, in order to meet the quality control requirements of the stroke center.

[0068] Trauma center dedicated layer, assuming the weight matrix is , dimension , the bias is , bias dimension , output This layer focuses on the extraction of features such as the time from injury to definitive treatment of trauma patients, the correlation between trauma severity score and treatment effect, etc., which are used for quality control analysis of trauma centers.

[0069] S423. For each center, the model outputs key quality control indicators based on different quality control standards.

[0070] For the chest pain center, key quality control indicators are output according to the chest pain center quality control standards. The specific formula is: ,in, It represents the key quality control indicator of the chest pain center. represents the weight of the gate-balloon dilation time, Indicates the time it takes for the balloon to dilate and open the blood vessels. Indicates the time when the patient arrived at the hospital gate. Represents the weight of the patient's vital signs recovery assessment, represents the weight of the heart rate of the vital signs of patients with chest pain, Indicates the heart rate of the patient with chest pain before treatment, Indicates that after a period of time, patients with chest pain heart rate, represents the weight of blood pressure in the vital signs of patients with chest pain, It indicates the blood pressure of the patient with chest pain before treatment. Indicates that after a period of time, patients with chest pain blood pressure, represents the weight of the patient's ECG improvement assessment, Indicates the weight of the ST segment in the overall ECG improvement assessment. It represents the sum of ST segment elevation amplitude of electrocardiogram characteristic index before treatment. It indicates the sum of ST segment elevation amplitude of electrocardiogram characteristic index after treatment. It indicates the weight of the QRS complex change in the electrocardiogram in the overall electrocardiogram improvement assessment. It represents the sum of QRS wave width of electrocardiogram characteristic index before treatment. It represents the sum of QRS wave widths of the characteristic index of the electrocardiogram after treatment. The above lists three key quality control indicators. In the actual treatment of chest pain, more indicators can be added to the formula.

[0071] For stroke centers, according to the stroke center quality control standards, the specific formula is: , Represents the key quality control indicators of stroke center. represents the weight of thrombolytic therapy, Indicates the time of starting thrombolytic therapy. Indicates the time of admission of the patient. represents the weight of the neurological function score rate of stroke patients, represents the patient's neurological function score before treatment. Indicates a specific time after the patient's treatment Neurological function score, represents the weight of the lesion volume assessment in the image, It indicates the volume of lesions in the images of stroke patients before treatment, such as the volume of cerebral infarction area and hemorrhagic lesion and hematoma calculated by specific algorithms. It represents the volume of lesions in the images of stroke patients after treatment. The above lists three key quality control indicators. In actual stroke treatment, more indicators can be added to the formula.

[0072] For trauma centers, according to the quality control standards of trauma centers, the specific formula is: ,in, It represents the key quality control indicator of the trauma center. represents the weight of post-traumatic treatment time, Indicates the time when definitive treatment is completed, such as the completion of key surgery or major treatment measures such as hemostasis. Indicates the time when the patient was injured. , then it is judged that the golden hour of trauma treatment has been achieved. represents the weight of the patient's vital signs stable time, Indicates the time that the patient's vital signs are stable within the normal range, such as heart rate, blood pressure, blood oxygen saturation, etc. Indicates the time to start monitoring the patient's vital signs. If the patient's vital signs are stable for longer than the standard stabilization time threshold, the rescue efficiency is low, otherwise it is high. represents the weight of the improvement rate of the patient's trauma severity score, Indicates the severity score of the patient's trauma upon admission. It indicates the severity score of the patient's trauma after a period of treatment. The above lists three key quality control indicators. In actual trauma treatment, more indicators can be added to the formula.

[0073] S424: Input the mapped fusion vector into a clustering algorithm to discover data patterns and outliers.

[0074] The mapped feature vector is input into the clustering algorithm for clustering calculation to find data patterns and outliers. Suppose the fused feature vector set is , initially randomly select z cluster centers, and then repeat the following steps until convergence: For each data point , calculate it to each cluster center The distance ,Will Assign to the cluster with the closest cluster center. Recalculate the center of each cluster, that is, for the jth cluster, its new cluster center is the average value of all data points in the cluster, and the calculation formula is ,in is the number of data points in the jth cluster.

[0075] After clustering is completed, the characteristics and data distribution of each cluster are analyzed to discover potential patterns in the data. Different clusters may represent different types of patient groups or different treatment process patterns. At the same time, outliers that are far away from most data points are identified. These outliers correspond to special cases or abnormal treatment situations, which require further in-depth analysis, indicating potential quality problems or situations that require special attention, providing valuable clues and directions for medical quality control.

[0076] S50. Display and compare the quality control indicators and abnormal points of the three major centers output by the comprehensive quality control model in multiple dimensions.

[0077] The output of the model in step S40 is combined with the existing quality control report text and the quality control suggestion text written by the experts, and input into the model generation module to automatically generate a text quality control report. The report content structure includes an overview, key quality control indicators, time trend analysis, outlier analysis, improvement suggestions, etc. The generation module can make up for missing modes, enhance data integrity, and improve the accuracy of quality control analysis.

[0078] Use multi-dimensional visualization tools to compare and display the quality control data reports of the three major centers in a unified manner, output multi-dimensional visualization results, and use time series graphs to show the changing trends of quality control indicators related to time nodes over time or case sequence. Use bar charts to compare the values ​​of quality control indicators between different categories or centers. Use box plots to show the distribution characteristics of data, and pie charts to show the proportion of each part to the total. Radar charts comprehensively display the performance of multiple indicators in different centers or categories. Through multi-dimensional display, the differences in quality control indicators and key improvement points can be intuitively presented.

[0079] Embodiment 2 like Figure 2 As shown, Example 2 of the present application provides a multi-modal and large-model quality control system based on multi-center data of chest pain, stroke and trauma, including: Acquisition module: used to collect quality control data from three major centers of chest pain, stroke, and trauma and perform preprocessing.

[0080] The quality control data of the three major centers of chest pain, stroke and trauma are extracted from the electronic medical record system, image archiving and communication system, monitoring equipment and laboratory information system. Including: text data, medical records, doctor's diagnosis records; image data, computer tomography, magnetic resonance imaging, X-ray and other images; time series data, monitor ECG data, blood pressure change curve, electroencephalogram, etc.; laboratory data, blood test reports, biochemical indicators, etc.

[0081] It also integrates multimodal data from the three major centers, and uses a data preprocessing module to standardize and clean data in different formats, removing special symbols, redundant punctuation, garbled characters, extra spaces, etc. in the text, retaining only content that has a substantial effect on semantic expression.

[0082] Multimodal data encoder module: includes feature extraction submodule, feature alignment submodule, and feature fusion submodule.

[0083] Feature extraction submodule: including text data encoder submodule, image data encoder submodule, time series data encoder submodule, and structured data encoder submodule.

[0084] Text data encoder submodule: performs feature extraction on text data.

[0085] The input text is divided into words, and the results of word segmentation are converted into input tag sequences that can be recognized by the text data encoder by adding special tags [CLS] and [SEP]. Using the word embedding matrix, each word is converted into a word embedding vector so that each word can be mapped to the pre-trained high-dimensional vector space. The specific formula is: , It's a word The word embedding vector of is the word embedding matrix, size is , is the vocabulary size, is the embedding dimension, is the i-th word in the word sequence that needs to be converted, and the converted word embedding vector sequence is , p is the number of word embedding vectors. Position encoding is added to the vectors in the word embedding vector sequence to provide the position information of the word in the sequence.

[0086] Thus, we get the input word vector sequence A. ,in, is the final input vector of the ith word, It's a word The word embedding vector of It's a word The position encoding of , pos is the position of the word vector in the sequence, and a is the dimension index of the embedding vector.

[0087] The input word vector sequence is input into the text data encoder. The text data encoder contains multiple encoder layers. Each layer contains a multi-head attention mechanism and a feedforward neural network. The input text features are continuously extracted and fused through multiple encoder layers to obtain a text vector representation that incorporates rich contextual information.

[0088] The multi-head attention mechanism contains a heads. For each head, for the input word vector sequence, given the query matrix Q, key matrix K and value matrix V, the importance of each value is determined by calculating the similarity between the query and the key, and a new representation is generated. This new representation contains context information. The specific formula is ,in, Compute the dot product between the query and the key, which measures the similarity between each query and all other keys. Divide by the scale factor The dot product result is scaled to prevent the gradient from vanishing or exploding. The softmax function is applied to convert the dot product result into a probability distribution, i.e., the attention weights. The value matrix V is weighted summed using these weights to obtain the final output vector.

[0089] Concatenate the results of a heads and pass an additional linear transformation To integrate information, the integration formula is ,in is the final output weight matrix, a is the number of heads, represents the dimension of each value vector, and d is the embedding dimension. This results in an output vector of the encoder layer.

[0090] The feedforward neural network is a fully connected layer used to connect the encoder layer. The specific formula is , where A is the input vector, is the weight matrix of this layer, d is the encoder dimension, is the hidden layer dimension of the feedforward neural network, is the bias term of this layer. is the weight matrix of the next layer, is the bias term for the next layer.

[0091] After extraction and fusion of multiple encoder layers, a text vector representation that incorporates rich contextual information is obtained, and the output vector of the last encoder layer is used as the final feature vector.

[0092] Image data encoder submodule: includes two-dimensional image submodule and 3D data submodule.

[0093] Two-dimensional image submodule: used for extracting two-dimensional image feature vectors.

[0094] Normalize the medical image data to a specific range and set the original pixel value to , the normalized pixel value is , the calculation formula is , is the minimum value of the original data, is the maximum value of the original data.

[0095] According to the encoder input size requirements, the image is cropped to the appropriate size and can be appropriately scaled to ensure uniform image size. The data set is expanded through operations such as rotation, flipping, translation, and adding noise to improve generalization capabilities.

[0096] Apply the filter in the image data encoder to convolve the input image, and perform convolution operations by sliding the convolution kernel on the image to extract local features. The formula is , where I is the input image, K is the convolution kernel, (i, j) is the coordinate of the current pixel position, and the result of the convolution operation is is the output value at (i, j), M is half the height of the convolution kernel, N is half the width, and (m, n) is the offset of the convolution kernel relative to the current position (i, j).

[0097] Adding an activation function after the convolutional layer increases nonlinear characteristics, and reducing the size of the feature map through maximum pooling to reduce computational complexity. The maximum pooling formula is , I is the input feature map, the shape is (C, H, W), C is the number of channels, H is the height, and W is the width. Represents the value of the input feature map at position (i, j), where i and j represent the coordinates of the pixel position currently being processed. In a maximum pooling operation, for the input feature map I, the maximum value is found in each pooling window, and this maximum value is used as the value of the output feature map at the corresponding position.

[0098] The feature maps after multiple convolution and pooling operations are combined and connected into a global feature map, flattened into a one-dimensional vector and connected to a fully connected layer for further feature integration. Suppose the dimension of the input flattened vector b is , the weight matrix of the fully connected layer is , the dimension is , the bias is , dimension , the output vector The calculation formula is , thus obtaining the feature vector of the two-dimensional image data.

[0099] 3D data submodule: used for 3D data feature vector extraction.

[0100] For 3D data, use normalization and rotation, cropping, etc. to increase data diversity, and ensure that all input volumes have the same resolution or size.

[0101] Apply the 3D filter in the image data encoder to convolve the input volume and extract local features. The 3D convolution kernel slides in three dimensions, taking into account the information of the X, Y and Z axes at the same time. The specific formula is , is the input 3D data volume, is a 3D convolution kernel, i, j, k represent the coordinates of the pixel position currently being processed, and M, N, P are the radii of the convolution kernel on the X, Y, and Z axes, respectively.

[0102] Adding an activation function after the convolutional layer increases nonlinear characteristics, and reducing the size of the feature map through maximum pooling to reduce computational complexity. The maximum pooling formula is , is the input feature map, i, j, k represent the coordinates of the pixel position currently processed.

[0103] After multiple convolution and pooling operations, the feature map is flattened into a one-dimensional vector and then connected to a fully connected layer to further integrate the features, thereby obtaining the feature vector of the 3D image data.

[0104] Time series data encoder submodule: Analyze the continuous data output by the monitoring device for time series data and extract the dynamic change characteristics in the time series.

[0105] Scale the time series data to a fixed range to speed up the training process. Use a sliding window to split the scaled time series data into multiple subsequences of fixed length, each of which is used as an input sample of the encoder. This helps capture local temporal dependencies. The formula for the subsequence after segmentation is , w is the window size, and t is the current time point.

[0106] Assume that the dimension of a subsequence is q, then the number of nodes in the input layer is q. For each subsequence of the input encoder, use the formula Decide which information should be discarded, among which, is the activation function, is the input weight matrix to the forget gate, Represents the input at the current moment, is the weight matrix from hidden state to forget gate, Represents the hidden state of the previous moment, carrying the information from the previous sequence processing. is the bias vector of the forget gate.

[0107] Using the formula Update some information of cell status, is the output of the input gate, is the activation function, is the candidate cell state, tanh is the hyperbolic tangent activation function, , , , are the corresponding weight matrices, Represents the input at the current moment, Indicates the hidden state at the previous moment, , is the bias vector.

[0108] Using the formula Update the cell state, the new cell state It is the part of the cell that is selectively retained from the previous state. and newly added filtered information Combination of. Represents the cell state at time step t-1 before the current time step.

[0109] Using the formula Decide what information should be output. represents the output value of the output gate at time step t, , is the corresponding weight matrix, is the bias vector, is the hidden state at the current moment, from which the feature vector of time series data is obtained.

[0110] Structured data encoder submodule: For structured information such as laboratory data and test results, the structured data encoder is used to extract the correlation between key indicators and generate feature representation.

[0111] Normalization and standardization are performed on structured data. Normalization scales the numerical features to a fixed range, and standardization converts the numerical features into a standard normal distribution with a mean of 0 and a standard deviation of 1.

[0112] Assume that the structured data has n different categories and the embedding dimension is , that is, to map each category to a dimensional vector, the parameter of the embedding layer is a The embedding matrix For a categorical variable value g, the index of its corresponding category in all categories is i, and the calculation formula of the vector s obtained after the embedding layer is , if there are both numerical indicators and categorical indicators, the categorical indicator vector processed by the embedding layer is concatenated with the numerical indicator data in a certain order to form a unified input vector, and the combined vector is input to the subsequent network layer to capture the correlation between key indicators through network learning. The vector output by the final output layer is the generated feature representation. Suppose the input vector is s, the dimension is m, and after a fully connected layer, the weight matrix is , dimension , the bias is , the dimension is k, then the output feature representation vector The calculation formula is , thus obtaining the feature vector of structured data.

[0113] Feature alignment submodule: used to align features of different modalities.

[0114] Since the data features of different modalities have different dimensions and ranges, it is necessary to normalize and align the features, and adjust the numerical features to the same range through normalization. Suppose the data is normalized to the interval [a, b], and the minimum value of the original data is , the maximum value is , then the standardized data The calculation formula is , u represents the original data value.

[0115] A feature mapping network is used to reduce or increase the dimensionality of each modality feature so that different modalities have a unified vector dimension. The features of the modalities are aligned in the same semantic space. Multimodal features are aligned through a shared embedding space so that the medical record description and the abnormal image area have high similarity in the semantic space.

[0116] Assume the dimension of the input text feature vector is , the target uniform dimension is D. Using a multi-layer fully connected network as the feature mapping network, taking two layers as an example, the weight matrix of the first fully connected layer is , the dimension is , the bias is The specific formula is , Represents the original text vector. Usually, an activation function is added after each layer to increase the nonlinear characteristics. The output after the activation function is , the weight matrix of the second fully connected layer is , the dimension is , the bias is , then the final output vector The other modes are reduced or increased in dimension according to the above steps, so that all modes have a unified vector dimension.

[0117] Feature fusion submodule: multimodal feature fusion using cross-modal self-attention.

[0118] The text modality vector is used as the query (Q) and the image modality vector is used as the key value (K, V). Multimodal feature fusion is achieved by calculating the attention weights.

[0119] Introducing a learnable parameter , which is used to dynamically adjust the relative importance between text and image modalities. The outputs of the two modes are weighted summed. The interaction weights between modalities can also be adaptively adjusted according to task requirements to strengthen the correlation between modalities.

[0120] The specific formula is , Y represents the final fusion feature vector, It is the output of the multi-head self-attention mechanism, Q represents the embedding representation from the text modality, and K, V represent the embedding representation from the image modality. represents the feature vector of the imaging modality, This part is the fusion feature calculated through the cross-modal self-attention mechanism, which emphasizes the interactive relationship between the text modality as the query and the image modality as the key value. This part retains the characteristics of the original imaging modality, ensuring that the imaging information is not completely ignored.

[0121] Modeling and analysis module: includes mapping submodule and modeling submodule.

[0122] Mapping submodule: maps the fused feature vector into a unified semantic space.

[0123] Map the fused features into a vector representation of fixed dimension using a linear transformation matrix Map the fused feature Y to a fixed dimension The mapping formula is , is the mapped vector. is the linear transformation matrix, D is the dimension of the fusion feature, is the target embedding dimension.

[0124] Compress high-dimensional features into low-dimensional features to improve model efficiency. Build an autoencoder model, including the encoder and decoder , , , where H is the hidden layer representation, which captures the key features of the input data. is the reconstructed embedding vector, and are the parameters of the encoder and decoder respectively. These parameters are optimized through the training process so that the encoder can effectively extract useful information from the input data. By training the autoencoder, the reconstruction error is minimized. The specific formula is , G represents the loss function, represents the number of samples in the training set, represents the true embedding vector of the i-th sample, Represents the reconstructed embedding vector of the i-th sample. After training, the encoder part is used to extract low-dimensional features.

[0125] Fine-tune the feature vector based on quality control analysis to ensure semantic relevance and specific task performance. Add task-related loss functions based on specific tasks. The specific formula is , based on the total loss , fine-tune the parameters of the embedding layer and dimensionality reduction module, the formula is , is the reconstruction loss, is the task-specific loss, is the balancing factor.

[0126] Modeling submodule: including common feature submodule, dedicated layer submodule, quality control indicator submodule and clustering submodule.

[0127] Common feature submodule: extract common features from the mapped fusion vector, uniformly encode different modal data with shared weights, and extract common patterns across centers.

[0128] A fully connected network is used as the shared encoding layer, and the mapped feature vector is input , let the shared layer weight matrix be , dimension , the bias is , dimension , then the shared layer output This layer learns the common feature patterns of data from different modalities and centers through shared weights, and extracts common information across centers, including common disease severity representations, common features of basic diagnosis and treatment processes, etc.

[0129] Dedicated layer submodules: Design specific modeling for the unique quality control needs of each center, and build dedicated layers for chest pain, stroke, and trauma centers respectively.

[0130] For the chest pain center dedicated layer, the weight matrix is , dimension , the bias is , bias dimension , the input is the shared layer output, then the output of this layer ,Based on the quality control needs of the chest pain center, this layer focuses on the time from chest pain onset to the first medical intervention, the correlation between electrocardiogram change characteristics and treatment effects, etc., and learns specific feature representations to output key quality control indicators related to the chest pain center.

[0131] The stroke center-specific layer, assuming the weight matrix is , dimension , the bias is , bias dimension , output This layer focuses on feature learning such as the time from stroke onset to thrombolysis or thrombectomy, and the relationship between changes in brain imaging lesions and neurological function recovery, in order to meet the quality control requirements of the stroke center.

[0132] Trauma center dedicated layer, assuming the weight matrix is , dimension , the bias is , bias dimension , output This layer focuses on the extraction of features such as the time from injury to definitive treatment of trauma patients, the correlation between trauma severity score and treatment effect, etc., which are used for quality control analysis of trauma centers.

[0133] Quality control indicator submodule: For each center, the model outputs key quality control indicators based on different quality control standards.

[0134] For the chest pain center, key quality control indicators are output according to the chest pain center quality control standards. The specific formula is: in, It represents the key quality control indicator of the chest pain center. represents the weight of the gate-balloon dilation time, Indicates the time it takes for the balloon to dilate and open the blood vessels. Indicates the time when the patient arrived at the hospital gate. Represents the weight of the patient's vital signs recovery assessment, represents the weight of the heart rate of the vital signs of patients with chest pain, Indicates the heart rate of the patient with chest pain before treatment, Indicates that after a period of time, patients with chest pain heart rate, represents the weight of blood pressure in the vital signs of patients with chest pain, It indicates the blood pressure of the patient with chest pain before treatment. Indicates that after a period of time, patients with chest pain blood pressure, represents the weight of the patient's ECG improvement assessment, Indicates the weight of the ST segment in the overall ECG improvement assessment. It represents the sum of ST segment elevation amplitude of electrocardiogram characteristic index before treatment. It indicates the sum of ST segment elevation amplitude of electrocardiogram characteristic index after treatment. It indicates the weight of the QRS complex change in the electrocardiogram in the overall electrocardiogram improvement assessment. It represents the sum of QRS wave width of electrocardiogram characteristic index before treatment. It represents the sum of QRS wave widths of the characteristic index of the electrocardiogram after treatment. The above lists three key quality control indicators. In the actual treatment of chest pain, more indicators can be added to the formula.

[0135] For stroke centers, according to the stroke center quality control standards, the specific formula is: , Represents the key quality control indicators of stroke center. represents the weight of thrombolytic therapy, Indicates the time of starting thrombolytic therapy. Indicates the time of admission of the patient. represents the weight of the neurological function score rate of stroke patients, represents the patient's neurological function score before treatment. Indicates a specific time after the patient's treatment Neurological function score, represents the weight of the lesion volume assessment in the image, It indicates the volume of lesions in the images of stroke patients before treatment, such as the volume of cerebral infarction area and hemorrhagic lesion and hematoma calculated by specific algorithms. It represents the volume of lesions in the images of stroke patients after treatment. The above lists three key quality control indicators. In actual stroke treatment, more indicators can be added to the formula.

[0136] For trauma centers, according to the quality control standards of trauma centers, the specific formula is: ,in, It represents the key quality control indicator of the trauma center. represents the weight of post-traumatic treatment time, Indicates the time when definitive treatment is completed, such as the completion of key surgery or major treatment measures such as hemostasis. Indicates the time when the patient was injured. , then it is judged that the golden hour of trauma treatment has been achieved. represents the weight of the patient's vital signs stable time, Indicates the time that the patient's vital signs are stable within the normal range, such as heart rate, blood pressure, blood oxygen saturation, etc. Indicates the time to start monitoring the patient's vital signs. If the patient's vital signs are stable for longer than the standard stabilization time threshold, the rescue efficiency is low, otherwise it is high. represents the weight of the improvement rate of the patient's trauma severity score, Indicates the severity score of the patient's trauma upon admission. It indicates the severity score of the patient's trauma after a period of treatment. The above lists three key quality control indicators. In actual trauma treatment, more indicators can be added to the formula.

[0137] Clustering submodule: Input the mapped fusion vector into the clustering algorithm to discover data patterns and anomalies.

[0138] The mapped feature vector is input into the clustering algorithm for clustering calculation to find data patterns and outliers. Suppose the fused feature vector set is , initially randomly select z cluster centers, and then repeat the following steps until convergence: For each data point , calculate it to each cluster center The distance ,Will Assign to the cluster with the closest cluster center. Recalculate the center of each cluster, that is, for the jth cluster, its new cluster center is the average value of all data points in the cluster, and the calculation formula is ,in is the number of data points in the jth cluster.

[0139] After clustering is completed, the characteristics and data distribution of each cluster are analyzed to discover potential patterns in the data. Different clusters may represent different types of patient groups or different treatment process patterns. At the same time, outliers that are far away from most data points are identified. These outliers correspond to special cases or abnormal treatment situations, which require further in-depth analysis, indicating potential quality problems or situations that require special attention, providing valuable clues and directions for medical quality control.

[0140] Generation module: used to display the model quality control results in multiple dimensions.

[0141] The output of the step model is combined with the existing quality control report text and the quality control suggestion text written by experts, and the input model generation module automatically generates a text quality control report. The report content structure includes an overview, key quality control indicators, time trend analysis, outlier analysis, improvement suggestions, etc. The generation module can make up for missing modes, enhance data integrity, and improve the accuracy of quality control analysis.

[0142] Use multi-dimensional visualization tools to compare and display the quality control data reports of the three major centers in a unified manner, output multi-dimensional visualization results, and use time series graphs to show the changing trends of quality control indicators related to time nodes over time or case sequence. Use bar charts to compare the values ​​of quality control indicators between different categories or centers. Use box plots to show the distribution characteristics of data, and pie charts to show the proportion of each part to the total. Radar charts comprehensively display the performance of multiple indicators in different centers or categories. Through multi-dimensional display, the differences in quality control indicators and key improvement points can be intuitively presented.

[0143] This system captures data from three major centers for chest pain, stroke, and trauma, conducts in-depth mining and statistical analysis on the multimodal data of chest pain, stroke, and trauma, discovers the correlation between different variables, and predicts the patient's diagnosis results and prognosis; through quality control analysis of the data from the three major centers, it trains a comprehensive quality control model. This model supports the hospital quality control department in quality control analysis and improvement plan generation for the three major centers of chest pain, stroke, and trauma, supports regional medical management agencies in comparing and uniformly managing quality control data of multiple hospitals, supports cross-campus or cross-regional medical collaborative quality control work, and tracks the actual performance of key quality control indicators to ensure compliance with industry standards.

[0144] This system promotes the cooperation between medicine and other disciplines and promotes the application of innovative technologies in the medical field. It can not only improve the quality and efficiency of medical services, but also provide reference for future clinical practice and scientific research.

[0145] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-modal and large-model quality control system based on multi-center data of chest pain, stroke and trauma, characterized by: include: Acquisition module: used to collect and pre-process different types of quality control multimodal data from three major centers: chest pain, stroke, and trauma; Multimodal data encoder module: used for extracting and aligning features of different types of quality control multimodal data from the three major centers using multimodal data encoders; Modeling and analysis module: used to perform multimodal feature fusion on the aligned multimodal features according to the cross-modal self-attention in the multimodal data encoder, unify the mapping model of the fused feature vectors, generate a comprehensive quality control model for the three major centers, and output the quality control indicators and abnormal points of the three major centers through the comprehensive quality control model; Generation module: used to display and compare the quality control indicators and abnormal points of the three major centers output by the quality control comprehensive model in multiple dimensions.

2. The multi-modal and large-model quality control system based on multi-center data of chest pain, stroke and trauma as claimed in claim 1, characterized in that: Multimodal data encoders are designed for different types of data, including text data encoders, image data encoders, time series data encoders, and structured data encoders.

3. The multi-modal and large-model quality control system based on multi-center data of chest pain, stroke and trauma as claimed in claim 2, characterized in that: After the encoder extracts features, the vector dimensions of different modes need to be aligned, and the numerical features need to be adjusted to the same range using normalization methods. The feature mapping network is used to reduce or increase the dimension of each modality feature so that different modalities have a unified vector dimension.

4. The multi-modal and large-model quality control system based on multi-center data of chest pain, stroke and trauma as claimed in claim 3, characterized in that: After aligning the vector dimensions of different modalities, a multimodal data encoder is used to perform multimodal feature fusion. The multimodal data encoder uses cross-modal self-attention to perform multimodal feature fusion.

5. The multi-modal and large-model quality control system based on multi-center data of chest pain, stroke and trauma as claimed in claim 4, characterized in that: The fused feature vector is mapped into a unified semantic space, which includes the following steps: mapping the fused features into a vector representation of fixed dimension, compressing the high-dimensional features into low-dimensional features, and fine-tuning the feature vector according to quality control analysis.

6. The multi-modal and large-model quality control system based on multi-center data of chest pain, stroke and trauma as claimed in claim 5, characterized in that: The comprehensive quality control model includes extracting common features from the mapped fusion vector; building dedicated layers for chest pain, stroke, and trauma centers respectively; for each center, the model outputs key quality control indicators based on different quality control standards; and inputting the mapped fusion vector into the clustering algorithm to discover data patterns and anomalies.

7. The multi-modal and large-model quality control system based on multi-center data of chest pain, stroke and trauma as claimed in claim 6, characterized in that: The multi-dimensional display of model quality control results is to combine the model quality control results with the existing quality control report text, input the model generation module to automatically generate a text quality control report, use multi-dimensional visualization tools to uniformly compare and display the quality control data reports of the three major centers, and output multi-dimensional visualization results.

8. A multimodal and large-model quality control method based on multi-center data of chest pain, stroke and trauma, characterized by: include: S10, collect different types of quality control multimodal data from three major centers of chest pain, stroke, and trauma and perform preprocessing; S20. For different types of quality control multimodal data from the three major centers, use multimodal data encoders to extract features and align features for different types of quality control multimodal data; S30, performing multimodal feature fusion on the aligned multimodal features according to the cross-modal self-attention in the multimodal data encoder; S40, uniformly mapping and modeling the fused feature vectors, generating a comprehensive quality control model for the three major centers, and outputting the quality control indicators and abnormal points of the three major centers through the comprehensive quality control model; S50. Display and compare the quality control indicators and abnormal points of the three major centers output by the comprehensive quality control model in multiple dimensions.

Images