Myocarditis recognition method and system based on multimodal images combined with machine learning

Abstract

The present invention discloses a myocarditis recognition method and system based on multimodal images combined with machine learning. The myocarditis recognition method based on multimodal images combined with machine learning includes the following steps: file data collection; multimodal feature data fusion; model construction; model evaluation; myocarditis recognition. The present invention obtains file data by data annotation and preprocessing the acquired initial file data, and obtains fused modality file feature data by feature extraction and multimodal fusion of the file data, and then inputs the obtained fused modality file feature data into a preset encoder to obtain fused modality file encoding data and construct a myocarditis recognition model to be evaluated, and then evaluates the myocarditis recognition model to be evaluated to obtain a first myocarditis recognition model, and finally combines the first myocarditis recognition model to perform myocarditis recognition to obtain a myocarditis recognition result and feedback, thereby improving the myositis recognition efficiency and solving the problem of low myositis recognition efficiency in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for identifying myocarditis based on multimodal images combined with machine learning. Background Art

[0002] With the development of multimodality, the current research field mainly focuses on the processing of three modalities: image, text, and voice. Different modalities have different ways of expression and different perspectives on the fifteen, and some cross-over and complementary phenomena will occur. There are also many different ways of information interaction between modalities, and more abundant feature information can be obtained under the condition of reasonable processing of multimodal information. Machine learning is an artificial intelligence technology that allows machines to automatically improve their performance by learning data without human programming. It generally includes decision trees, random forests, K-nearest neighbor algorithms, logistic regression, clustering algorithms, dimensionality reduction methods, integration methods, neural networks, and deep learning. These methods have their own characteristics and are suitable for different learning tasks and data types. Myocarditis refers to a disease with localized or diffuse inflammatory lesions of the myocardium as the main manifestation. According to the established Dallas criteria, the histological evidence of myocardial infiltration is myocardial inflammatory cell infiltration, accompanied by adjacent myocardial cell degeneration and necrosis.

[0003] The existing multimodal recognition system promotes each other through the external demand of multimodal data analysis and the internal demand of advanced cognitive intelligence. In biometrics, it refers to the integration or fusion of two or more biometric technologies, using the unique advantages of multiple biometric technologies and combining data fusion technology to make the authentication and recognition process more accurate and secure. Multimodal biometric technology can collect different biometric features through independent or multiple collection methods, and identify and authenticate by analyzing and judging the feature values ​​of multiple biometric methods.

[0004] For example, the multimodal emotion recognition method announced in the invention patent with announcement number: CN112559835B includes inputting an audio file, a video file and a corresponding text file of a sample to be tested, extracting features of the audio file, the video file and the text file to obtain frame-level audio features, frame-level video features and word-level text features, and then inputting them into an audio feature encoder, a video feature encoder and a text feature encoder for local context modeling to obtain audio coding features, video coding features and text coding features; respectively passing the audio coding features, the video coding features and the text coding features through their respective self-attention modules to model the interaction relationship within the modality to obtain intra-modal features; respectively performing temporal pooling on the intra-modal features and the inter-modal features to obtain global intra-modal features; performing weighted fusion on the global intra-modal features and the global inter-modal features to obtain the intra-modal feature representation and the inter-modal feature representation of the entire sample to be tested, and then splicing the intra-modal feature representation and the inter-modal feature representation, and then passing through a fully connected network to obtain the final emotion classification result.

[0005] For example, the invention patent with announcement number CN113590852B discloses a training method, a multimodal recognition method, and a device, comprising: obtaining sample data, wherein the sample data comprises at least one of first image data and first text data; identifying the sample data to obtain target data; training the multimodal recognition model based on the similarity between the target data and the sample data; the multimodal recognition model comprises a text recognition model and an image recognition model, identifying the sample data to generate target data; and training the multimodal recognition model based on the similarity between the target data and the sample data.

[0006] However, in the process of implementing the technical solution of the invention in the embodiments of the present application, the present application found that the above technology has at least the following technical problems:

[0007] In the existing technology, the traditional recognition method is single, and the same recognition technology has large differences in the false recognition rate and rejection rate in different modalities. Each modality has different acquisition adaptation scenarios, data security levels and privacy sensitivity, which has great limitations and the problem of low efficiency in myocarditis recognition. Summary of the invention

[0008] The embodiments of the present application solve the problem of low efficiency of myocarditis recognition in the prior art by providing a myocarditis recognition method and system based on multimodal images combined with machine learning, thereby improving the efficiency of myocarditis recognition.

[0009] The embodiment of the present application provides a method for identifying myocarditis based on multimodal images combined with machine learning, comprising the following steps: S1, performing data annotation and preprocessing on the acquired initial file data to obtain file data, wherein the initial file data includes initial historical multimodal file data and initial multimodal file data; S2, performing feature extraction and multimodal fusion on the acquired file data to obtain fused modality file feature data; S3, inputting the acquired fused modality file feature data into a preset encoder to perform feature data encoding to obtain corresponding fused modality file encoding data, and using the historical fused modality encoding data to construct a myocarditis recognition model to be evaluated, wherein the fused modality encoding data is image encoding data, audio The result of multimodal fusion of coded data and text coded data, the myocarditis recognition model to be evaluated is used to identify the myocarditis condition of the person to be identified; S4, model evaluation is performed on the constructed myocarditis recognition model to be evaluated to obtain a first evaluation result, and the first evaluation result is compared with the myocarditis recognition evaluation index. If the myocarditis recognition evaluation index is met, the corresponding model is recorded as the first myocarditis recognition model, otherwise the model is reconstructed, and the first evaluation result is used to evaluate the model accuracy and recognition ability of the myocarditis recognition model to be evaluated; S5, myocarditis recognition is performed on the acquired fused modal coded data in combination with the first myocarditis recognition model to obtain the myocarditis recognition result of the person to be identified.

[0010] Furthermore, the specific steps of the preprocessing are as follows: analyzing the data duplication of the corresponding data after the initial text data is annotated, and processing it to obtain the first file data; analyzing the data noise of the first text data after deduplication processing, and processing it to obtain the second file data; analyzing the data annotation of the second text data after deduplication processing, and processing it to obtain the third file data; analyzing the data format of the third text data after error correction processing of data annotation, and processing it to obtain the fourth file data; analyzing the data normalization of the fourth text data after format conversion processing, and processing it to obtain the fifth file data; analyzing the data size of the fifth text data after normalization processing, and processing it to obtain file data.

[0011] Furthermore, the specific steps of the multimodal fusion are as follows: obtaining a first multimodal fusion result, the first multimodal fusion result represents a result obtained by concatenating the data obtained after feature extraction of the file data in a preset order; obtaining a second multimodal fusion result, the second multimodal fusion result represents a result of converting the data obtained after feature extraction of the file data into corresponding feature vectors, and fusing the corresponding feature vectors using weighted summation; obtaining a third multimodal fusion result, the third multimodal fusion result represents a result of inputting the data obtained after feature extraction of the file data into a preset deep neural network, and then fusing them through a preset specific structure; combining the first multimodal fusion result, the second multimodal fusion result and the third multimodal fusion result.

[0012] Furthermore, a specific method for obtaining the fused modal file encoding data is as follows: inputting the fused modal file feature data into a corresponding preset data encoder for feature data encoding, and converting the data in combination with a corresponding encoding method to obtain the fused modal file encoding data to be evaluated, wherein the fused modal file encoding data to be evaluated includes image encoding data to be evaluated, audio encoding data to be evaluated, and text encoding data to be evaluated; comparing the fused modal file encoding data to be evaluated with the reference fused modal file encoding data to obtain a fusion encoding data evaluation index; if the fusion encoding data evaluation index meets the corresponding threshold, it indicates that the feature data encoding is successful, and the corresponding data is recorded as the fused modal file encoding data; otherwise, the feature data encoding is performed again, and the fusion encoding data evaluation index is used to describe the feature data encoding situation.

[0013] Furthermore, the specific method for obtaining the fusion coding data evaluation index is as follows: obtaining the coding data vector of the fusion modality file coding data to be evaluated, the coding data vector includes an image coding data vector, an audio coding data vector and a text coding data vector; calculating the fusion coding data evaluation index according to the obtained coding data vector and in combination with the fusion coding data weight, the fusion coding data weight represents the influence of the coding data vector on the fusion coding data evaluation index, the fusion coding data weight includes an image coding data weight, an audio coding data weight and a text coding data weight, and the fusion coding data evaluation index is calculated using the following formula:

[0014] , where represents a natural constant, Indicates the number of the coded data of the fusion modality file to be evaluated. , is the total number of encoded data in the fusion modality file to be evaluated, Indicates The number of the image encoding data of the fused modality file encoding data to be evaluated, , For the The total number of image encoding data of the fusion modality file encoding data to be evaluated, Indicates The number of the audio coded data of the fused modality file coded data to be evaluated, , Indicates The number of the text encoding data of the fusion modality file encoding data to be evaluated, , For the The fusion coding data evaluation index of the fusion modality file coding data to be evaluated, For the The first of the coded data of the fusion modality files to be evaluated image encoding data vector of image encoding data, For the The first of the coded data of the fusion modality files to be evaluated A reference image coding data vector of image coding data, For the The first of the coded data of the fusion modality files to be evaluated An audio coded data vector of audio coded data, For the The first of the coded data of the fusion modality files to be evaluated A reference audio coded data vector of audio coded data, For the The first of the coded data of the fusion modality files to be evaluated A text encoding data vector of text encoding data, For the The first of the coded data of the fusion modality files to be evaluated A reference text encoding data vector of text encoding data, Encode the data weights for the image, is the audio coding data weight, Encode data weights for text.

[0015] Furthermore, the specific construction method of the myocarditis recognition model to be evaluated is as follows: step one, constructing a feature data set, the feature data set is a collection of historical fusion modal coding data, and dividing the feature data set into a training feature data set and a verification feature data set according to a preset ratio; step two, obtaining a training machine model, the training machine model is a converged model obtained by training a preset machine learning model using the training feature data set; step three, using the verification feature data set to perform model verification on the training machine model to obtain the myocarditis recognition model to be evaluated.

[0016] Furthermore, the specific evaluation method of the model accuracy is as follows: the model training and model verification process of the myocarditis recognition model to be evaluated is analyzed to obtain the performance evaluation parameters of the myocarditis recognition model to be evaluated, and the performance evaluation parameters include accuracy, precision and recall rate; the accuracy evaluation index of the myocarditis recognition model to be evaluated is calculated according to the obtained performance evaluation parameters and the influence of the corresponding data on the accuracy evaluation index, and the accuracy evaluation index is used to describe the accuracy of the myocarditis recognition model to be evaluated, and the accuracy evaluation index is calculated using the following formula: , where represents a natural constant, Indicates the number of the myocarditis identification model to be evaluated, , is the total number of myocarditis recognition models to be evaluated, For the The accuracy evaluation index of the myocarditis recognition model to be evaluated, For the The accuracy of the myocarditis recognition model to be evaluated, is the model reference accuracy, is the model accuracy reference deviation, For the The accuracy of the myocarditis recognition model to be evaluated, is the model reference accuracy, is the model accuracy reference deviation, For the The recall rate of the myocarditis recognition model to be evaluated, is the model reference recall, is the model recall reference deviation, is the influence of absolute deviation of accuracy on the accuracy evaluation index, is the influence of absolute deviation of precision on the accuracy evaluation index, It is the influence of the absolute deviation of recall rate on the accuracy evaluation index.

[0017] Furthermore, the specific evaluation method of the recognition ability is as follows: obtaining recognition model data and recognition distribution score of the myocarditis recognition model to be evaluated, the recognition model data including data quality data and model complexity data, and the recognition distribution score being an evaluation result of the distribution of recognition ability data of the myocarditis recognition model to be evaluated by a preset professional; obtaining a recognition ability evaluation index of the myocarditis recognition model to be evaluated based on the acquired recognition model data and recognition distribution score, and the recognition ability evaluation index is used to describe the recognition ability of the myocarditis recognition model to be evaluated.

[0018] Furthermore, the specific method for obtaining the myocarditis recognition result of the person to be identified is as follows: according to the acquired fusion modal coding data, and in combination with the first myocarditis recognition model, the myocarditis recognition result is obtained by analysis; the acquired myocarditis recognition result of the person to be identified is fed back to the corresponding preset medical staff for myocarditis situation analysis.

[0019] The embodiment of the present application provides a myocarditis recognition system based on multimodal images combined with machine learning, including a file data collection module, a multimodal fusion module, a model construction module, a model evaluation module and a myocarditis recognition module; wherein the file data collection module is used to perform data annotation and preprocessing on the acquired initial file data to obtain file data, and the initial file data includes initial historical multimodal file data and initial multimodal file data; the multimodal fusion module is used to perform feature extraction and multimodal fusion on the acquired file data to obtain fused modality file feature data; the model construction module is used to input the acquired fused modality file feature data into a preset encoder to perform feature data encoding to obtain corresponding fused modality file encoding data, and use the historical fused modality encoding data to construct a myocarditis recognition model to be evaluated. type, the fused modal coding data is the result of multimodal fusion of image coding data, audio coding data and text coding data, the myocarditis recognition model to be evaluated is used to identify the myocarditis condition of the person to be identified; the model evaluation module is used to perform model evaluation on the constructed myocarditis recognition model to be evaluated to obtain a first evaluation result, and compare the first evaluation result with the myocarditis recognition evaluation index, if the myocarditis recognition evaluation index is met, the corresponding model is recorded as the first myocarditis recognition model, otherwise the model is reconstructed, and the first evaluation result is used to evaluate the model accuracy and recognition ability of the myocarditis recognition model to be evaluated; the myocarditis recognition module is used to perform myocarditis recognition on the acquired fused modal coding data in combination with the first myocarditis recognition model to obtain the myocarditis recognition result of the person to be identified.

[0020] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0021] 1. The fused modality file feature data is obtained by performing feature extraction and multimodal fusion on the acquired file data, and then the fused modality file encoding data is obtained in combination with a preset encoder and a myocarditis recognition model to be evaluated is constructed. Then, the myocarditis recognition model to be evaluated is evaluated to obtain a first myocarditis recognition model. Finally, myocarditis recognition is performed in combination with the first myocarditis recognition model to obtain a myocarditis recognition result and feedback is provided, thereby achieving accurate acquisition of the first myocarditis recognition model, thereby achieving improved myocarditis recognition efficiency, and effectively solving the problem of low myocarditis recognition efficiency in the prior art.

[0022] 2. By obtaining the encoded data vector of the encoded data of the fused modal file to be evaluated, and then calculating the fused encoded data evaluation index in combination with the fused encoded data weight, finally evaluating the characteristic data encoding situation of the fused modal file encoded data to be evaluated through the calculated fused encoded data evaluation index, thereby realizing the digitization of the characteristic data encoding situation of the fused modal file encoded data to be evaluated, and then realizing a more accurate evaluation of the characteristic data encoding situation of the fused modal file encoded data to be evaluated.

[0023] 3. By obtaining the performance evaluation parameters of the myocarditis recognition model to be evaluated, and then combining the influence of the corresponding data on the accuracy evaluation index to calculate the accuracy evaluation index of the myocarditis recognition model to be evaluated, and finally evaluating the accuracy of the myocarditis recognition model to be evaluated through the calculated accuracy evaluation index, the accuracy of the myocarditis recognition model to be evaluated is quantified, thereby achieving a more accurate evaluation of the accuracy of the myocarditis recognition model to be evaluated. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flow chart of a method for identifying myocarditis based on multimodal images combined with machine learning provided in an embodiment of the present application;

[0025] Figure 2 A flowchart for constructing a myocarditis recognition model to be evaluated provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the structure of a myocarditis recognition system based on multimodal images combined with machine learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The embodiment of the present application solves the problem of low efficiency of myocarditis recognition in the prior art by providing a myocarditis recognition method and system based on multimodal images combined with machine learning. The method obtains file data by performing data annotation and preprocessing on the acquired initial file data, and performs feature extraction and multimodal fusion on the acquired file data to obtain fused modality file feature data. The acquired fused modality file feature data is then input into a preset encoder for feature data encoding to obtain corresponding fused modality file encoding data, and historical fused modality encoding data is used to construct a myocarditis recognition model to be evaluated. Then, the constructed myocarditis recognition model to be evaluated is model evaluated to obtain a first myocarditis recognition model. Finally, myocarditis recognition is performed on the acquired fused modality encoding data in combination with the first myocarditis recognition model to obtain a myocarditis recognition result of the person to be identified and the result is fed back to the preset medical staff, thereby improving the efficiency of myocarditis recognition.

[0028] The technical solution in the embodiment of the present application is to solve the problem of low efficiency in identifying myocarditis, and the overall idea is as follows:

[0029] The acquired file data is subjected to feature extraction and multimodal fusion to obtain fused modality file feature data, which is then input into a preset encoder to obtain fused modality file encoding data and to construct a myocarditis recognition model to be evaluated. The myocarditis recognition model to be evaluated is then evaluated to obtain a first myocarditis recognition model. Finally, myocarditis recognition is performed in combination with the first myocarditis recognition model to obtain a myocarditis recognition result and fed back to the preset medical staff, thereby achieving the effect of improving the efficiency of myocarditis recognition.

[0030] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0031] like Figure 1 As shown, it is a flow chart of a myocarditis recognition method based on multimodal images combined with machine learning provided in an embodiment of the present application, and the method comprises the following steps: S1, file data collection: data annotation and preprocessing are performed on the acquired initial file data to obtain file data, and the initial file data includes initial historical multimodal file data and initial multimodal file data, and the initial multimodal file data includes initial image file data, initial audio file data and initial text file data; S2, multimodal feature data fusion: feature extraction and multimodal fusion are performed on the acquired file data to obtain fused modality file feature data, and the fused modality file feature data includes historical fused modality feature data and fused modality feature data; S3, model construction: the acquired fused modality file feature data is input into a preset encoder for feature data encoding to obtain corresponding fused modality file encoding data, and the fused modality file encoding data includes historical fusion Modal coding data and fused modal coding data, and use historical fused modal coding data to construct a myocarditis recognition model to be evaluated, the fused modal coding data is the result of multimodal fusion of image coding data, audio coding data and text coding data, and the myocarditis recognition model to be evaluated is used to identify the myocarditis condition of the person to be identified; S4, model evaluation: perform model evaluation on the constructed myocarditis recognition model to be evaluated to obtain a first evaluation result, and compare the first evaluation result with the myocarditis recognition evaluation index. If the myocarditis recognition evaluation index is met, the corresponding model is recorded as the first myocarditis recognition model, otherwise the model is reconstructed, and the first evaluation result is used to evaluate the model accuracy and recognition ability of the myocarditis recognition model to be evaluated; S5, myocarditis recognition: combine the first myocarditis recognition model to perform myocarditis recognition on the acquired fused modal coding data to obtain the myocarditis recognition result of the person to be identified.

[0032] In this embodiment, the file data includes location and category information in image file data, classification and sentiment tendency in text file data, and audio emotion in audio file data; the initial historical multimodal file data is file data within a preset time interval, and the initial multimodal file data is file data of the person to be identified acquired in real time; the initial file data of the person to be identified is collected through crawlers and API interface methods; the efficiency of myocarditis recognition is improved.

[0033] Furthermore, the specific steps of preprocessing are as follows: analyzing the data duplication of the corresponding data after the initial text data is annotated, and processing it to obtain the first file data; analyzing the data noise of the first text data after deduplication processing, and processing it to obtain the second file data; analyzing the data annotation of the second text data after deduplication processing, and processing it to obtain the third file data; analyzing the data format of the third text data after error correction processing of data annotation, and processing it to obtain the fourth file data; analyzing the data normalization of the fourth text data after format conversion processing, and processing it to obtain the fifth file data; analyzing the data size of the fifth text data after normalization processing, and processing it to obtain file data.

[0034] In this embodiment, the first file data represents file data after deduplication processing is performed on the initial file data after data annotation, the second file data represents file data after denoising processing is performed on the first file data, the third file data represents file data after error correction processing is performed on the second file data after data annotation, the fourth file data represents file data after format conversion processing is performed on the third file data, the fifth file data represents file data after normalization processing is performed on the fourth file data, and the file data represents file data after scaling processing is performed on the fifth file data; the accuracy of the subsequently constructed myocarditis recognition model to be evaluated is improved.

[0035] Furthermore, the specific steps of multimodal fusion are as follows: obtaining a first multimodal fusion result, the first multimodal fusion result represents the result of concatenating the data after the obtained file data features are extracted in a preset order; obtaining a second multimodal fusion result, the second multimodal fusion result represents the result of converting the data after the obtained file data features are extracted into corresponding feature vectors, and fusing the corresponding feature vectors using weighted summation; obtaining a third multimodal fusion result, the third multimodal fusion result represents the result of inputting the data after the obtained file data features are extracted into a preset deep neural network, and then fusing them through a preset specific structure; combining the first multimodal fusion result, the second multimodal fusion result and the third multimodal fusion result.

[0036] In this embodiment, the preset specific structure includes an attention mechanism and a gating mechanism, and realizes the fusion of three modalities of image feature data, text feature data and audio feature data.

[0037] Furthermore, a specific method for obtaining fused modal file encoding data is as follows: inputting the fused modal file feature data into a corresponding preset data encoder for feature data encoding, and converting the data in combination with a corresponding encoding method to obtain fused modal file encoding data to be evaluated, the preset data encoder includes an image feature data encoder, an audio feature data encoder, and a text feature data encoder, and the fused modal file encoding data to be evaluated includes image encoding data to be evaluated, audio encoding data to be evaluated, and text encoding data to be evaluated; comparing the fused modal file encoding data to be evaluated with the reference fused modal file encoding data to obtain a fused encoding data evaluation index, if the fused encoding data evaluation index meets the corresponding threshold, it indicates that the feature data encoding is successful, and the corresponding data is recorded as fused modal file encoding data, otherwise the fused modal file feature data is re-encoded, and the fused encoding data evaluation index is used to describe the feature data encoding situation.

[0038] In this embodiment, the encoding method is a dimensionality reduction technique, such as principal component analysis (PCA), autoencoder, other compression and representation learning methods, which improves the encoding efficiency of multimodal feature data.

[0039] Furthermore, a specific method for obtaining the fusion coding data evaluation index is as follows: obtaining a coding data vector of the fusion modality file coding data to be evaluated, the coding data vector including an image coding data vector, an audio coding data vector and a text coding data vector; calculating the fusion coding data evaluation index based on the obtained coding data vector and in combination with the fusion coding data weight, the fusion coding data weight represents the influence of the coding data vector on the fusion coding data evaluation index, the fusion coding data weight includes the image coding data weight, the audio coding data weight and the text coding data weight; the fusion coding data evaluation index describes the feature data coding situation, the feature data coding situation can be directly obtained by the feature extraction algorithm, and can also be more accurately calculated by the following formula, the fusion coding data evaluation index is calculated using the following formula:

[0040] , where represents a natural constant, Indicates the number of the coded data of the fusion modality file to be evaluated. , is the total number of encoded data in the fusion modality file to be evaluated, Indicates The number of the image encoding data of the fused modality file encoding data to be evaluated, , For the The total number of image encoding data of the fusion modality file encoding data to be evaluated, Indicates The number of the audio coded data of the fused modality file coded data to be evaluated, , Indicates The number of the text encoding data of the fusion modality file encoding data to be evaluated, , For the The fusion coding data evaluation index of the fusion modality file coding data to be evaluated, For the The first of the coded data of the fusion modality files to be evaluated image encoding data vector of image encoding data, For the The first of the coded data of the fusion modality files to be evaluated A reference image coding data vector of image coding data, For the The first of the coded data of the fusion modality files to be evaluated An audio coded data vector of audio coded data, For the The first of the coded data of the fusion modality files to be evaluated A reference audio coded data vector of audio coded data, For the The first of the coded data of the fusion modality files to be evaluated A text encoding data vector of text encoding data, For the The first of the coded data of the fusion modality files to be evaluated A reference text encoding data vector of text encoding data, Encode the data weights for the image, is the audio coding data weight, Encode data weights for text.

[0041] In this embodiment, For the The total number of audio coded data of the fused modality file coded data to be evaluated, For the The total number of text encoding data of the fusion modality file encoding data to be evaluated; the calculation of the fusion encoding data evaluation index is based on the dot multiplication of the vectors, where , and Represents the dot product of vectors; the dot product is also called the scalar product, and its result is the length of the projection of a vector in the direction of another vector, which is a scalar; it realizes the numerical evaluation of the encoding of feature data.

[0042] Further, such as Figure 2 As shown, it is a flow chart for constructing the myocarditis recognition model to be evaluated provided in an embodiment of the present application, and the specific method for constructing the myocarditis recognition model to be evaluated is as follows: step one, constructing a feature data set, the feature data set is a collection of historical fusion modal coding data, and dividing the feature data set into a training feature data set and a verification feature data set according to a preset ratio; step two, obtaining a training machine model, the training machine model is a converged model obtained by training a preset machine learning model using the training feature data set, and the preset machine learning model is a model selected according to the characteristics of the historical fusion modal coding data; step three, using the verification feature data set to perform model verification on the training machine model to obtain the myocarditis recognition model to be evaluated.

[0043] In this embodiment, the training feature data set represents a set of feature data used to train a preset machine learning model, and the verification feature data set represents a set of feature data used to verify the training machine model; the model is trained using the training feature data set, and the model parameters are adjusted through an optimization algorithm to minimize the loss function; the model performance is evaluated using the verification feature data set, and the model hyperparameters are adjusted to improve the performance; thereby improving the efficiency of identifying the myocarditis condition of the person to be identified.

[0044] Furthermore, the specific evaluation method of the model accuracy is as follows: the model training and model verification process of the myocarditis recognition model to be evaluated is analyzed to obtain the performance evaluation parameters of the myocarditis recognition model to be evaluated, and the performance evaluation parameters include accuracy, precision and recall rate; the accuracy evaluation index of the myocarditis recognition model to be evaluated is calculated according to the obtained performance evaluation parameters and the influence of the corresponding data on the accuracy evaluation index, and the accuracy evaluation index is used to describe the accuracy of the myocarditis recognition model to be evaluated; the accuracy evaluation index describes the accuracy of the model. In addition to being directly obtained through the accuracy rate, the model accuracy can also be more accurately calculated through the following formula. The accuracy evaluation index is calculated using the following formula:

[0045] , where represents a natural constant, Indicates the number of the myocarditis recognition model to be evaluated, , is the total number of myocarditis recognition models to be evaluated, For the The accuracy evaluation index of the myocarditis recognition model to be evaluated, For the The accuracy of the myocarditis recognition model to be evaluated, is the model reference accuracy, is the model accuracy reference deviation, For the The accuracy of the myocarditis recognition model to be evaluated, is the model reference accuracy, is the model accuracy reference deviation, For the The recall rate of the myocarditis recognition model to be evaluated, is the model reference recall, is the model recall reference deviation, is the influence of absolute deviation of accuracy on the accuracy evaluation index, is the influence of absolute deviation of precision on the accuracy evaluation index, It is the influence of the absolute deviation of recall rate on the accuracy evaluation index.

[0046] In this embodiment, accuracy is used to describe the proportion of samples correctly predicted by the myocarditis recognition model to be evaluated, precision is used to describe the proportion of samples that are truly positive among all samples predicted to be positive, and recall is used to describe the proportion of samples that are correctly predicted to be positive among all actual positive samples; thus, a numerical evaluation of the accuracy of the myocarditis recognition model to be evaluated is achieved.

[0047] Furthermore, a specific evaluation method for recognition ability is as follows: obtaining recognition model data and recognition distribution score of the myocarditis recognition model to be evaluated, the recognition model data including data quality data and model complexity data, and the recognition distribution score being an evaluation result of a preset professional on the distribution of recognition ability data of the myocarditis recognition model to be evaluated; obtaining a recognition ability evaluation index of the myocarditis recognition model to be evaluated based on the obtained recognition model data and recognition distribution score, and the recognition ability evaluation index is used to describe the recognition ability of the myocarditis recognition model to be evaluated.

[0048] In this embodiment, the recognition ability evaluation index describes the model recognition ability. In addition to being directly obtained through the confusion matrix and the AUC-ROC curve, the model recognition ability can also be more accurately calculated through the following formula. The recognition ability evaluation index is calculated using the following formula:

[0049] , where Indicates the number of the preset professional. , is the total number of preset professionals, For the The recognition ability evaluation index of the myocarditis recognition model to be evaluated, For the Data quality data of myocarditis recognition models to be evaluated, is the model reference data quality data, For the Model complexity data of the myocarditis recognition model to be evaluated, is the reference model complexity data, For the reason The first preset professional The recognition distribution scores of the myocarditis recognition models to be evaluated, is the correction factor for the absolute deviation of data quality data, is the correction factor for the absolute deviation of the model complexity data, It is the correction factor of the average recognition distribution score; it realizes the numerical evaluation of the recognition ability of the myocarditis recognition model to be evaluated.

[0050] Furthermore, a specific method for obtaining the myocarditis recognition result of the person to be identified is as follows: based on the acquired fusion modal coding data, and combined with the first myocarditis recognition model, an analysis is performed to obtain the myocarditis recognition result; the acquired myocarditis recognition result of the person to be identified is fed back to the corresponding preset medical staff for myocarditis situation analysis.

[0051] In this embodiment, the efficiency of the first myocarditis identification model in identifying myocarditis is improved.

[0052] like Figure 3As shown, it is a structural schematic diagram of a myocarditis recognition system based on multimodal images combined with machine learning provided in an embodiment of the present application. The myocarditis recognition system based on multimodal images combined with machine learning provided in an embodiment of the present application includes: a file data collection module, a multimodal fusion module, a model construction module, a model evaluation module and a myocarditis recognition module; wherein the file data collection module is used to perform data annotation and preprocessing on the acquired initial file data to obtain file data, the initial file data includes initial historical multimodal file data and initial multimodal file data, and the initial multimodal file data includes initial image file data, initial audio file data and initial text file data; the multimodal fusion module is used to perform feature extraction and multimodal fusion on the obtained file data to obtain fused modal file feature data, and the fused modal file feature data includes historical fused modal feature data and fused modal feature data; the model construction module is used to input the acquired fused modal file feature data into a preset encoder for feature data encoding The corresponding fusion modality file encoding data is obtained, and the fusion modality file encoding data includes historical fusion modality encoding data and fusion modality encoding data, and the historical fusion modality encoding data is used to construct a myocarditis recognition model to be evaluated, and the fusion modality encoding data is the result of multimodal fusion of image encoding data, audio encoding data and text encoding data. The myocarditis recognition model to be evaluated is used to identify the myocarditis condition of the person to be identified; the model evaluation module is used to perform model evaluation on the constructed myocarditis recognition model to be evaluated to obtain a first evaluation result, and compare the first evaluation result with the myocarditis recognition evaluation index. If the myocarditis recognition evaluation index is met, the corresponding model is recorded as the first myocarditis recognition model, otherwise the model is reconstructed, and the first evaluation result is used to evaluate the model accuracy and recognition ability of the myocarditis recognition model to be evaluated; the myocarditis recognition module is used to perform myocarditis recognition on the acquired fusion modality encoding data in combination with the first myocarditis recognition model to obtain the myocarditis recognition result of the person to be identified.

[0053] In this embodiment, systematization of myocarditis identification is achieved.

[0054] The technical solution in the above-mentioned embodiment of the present application has at least the following technical effects or advantages: relative to the multimodal emotion recognition method announced in the invention patent with announcement number: CN112559835B, the embodiment of the present application obtains the encoded data vector of the fused modal file encoded data to be evaluated, and then calculates the fused encoded data evaluation index in combination with the fused encoded data weight, and finally evaluates the characteristic data encoding situation of the fused modal file encoded data to be evaluated through the calculated fused encoded data evaluation index, thereby realizing the digitization of the characteristic data encoding situation of the fused modal file encoded data to be evaluated, and then realizing the characteristic data encoding situation of the fused modal file encoded data to be evaluated More accurate evaluation of data encoding conditions; relative to the training method, multimodal recognition method and device of the multimodal recognition model announced in the invention patent with announcement number: CN113590852B, the embodiment of the present application obtains the performance evaluation parameters of the myocarditis recognition model to be evaluated, and then calculates the accuracy evaluation index of the myocarditis recognition model to be evaluated in combination with the influence of the corresponding data on the accuracy evaluation index, and finally evaluates the accuracy of the myocarditis recognition model to be evaluated through the calculated accuracy evaluation index, thereby realizing the digitization of the accuracy of the myocarditis recognition model to be evaluated, and then realizing a more accurate evaluation of the accuracy of the myocarditis recognition model to be evaluated.

[0055] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0057] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0058] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0059] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0060] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A myocarditis recognition method based on multimodal images combined with machine learning, characterized in that: The following steps are involved: S1, performing data annotation and preprocessing on the acquired initial file data to obtain file data, wherein the initial file data includes initial historical multimodal file data and initial multimodal file data; S2, performing feature extraction and multi-modal fusion on the obtained file data to obtain fused modality file feature data; S3, inputting the acquired fusion modality file feature data into a preset encoder to perform feature data encoding to obtain corresponding fusion modality file encoding data, and using the historical fusion modality encoding data to construct a myocarditis recognition model to be evaluated, wherein the fusion modality encoding data is a result of multimodal fusion of image encoding data, audio encoding data and text encoding data, and the myocarditis recognition model to be evaluated is used to identify the myocarditis condition of the person to be identified; S4, performing model evaluation on the constructed myocarditis recognition model to be evaluated to obtain a first evaluation result, and comparing the first evaluation result with the myocarditis recognition evaluation index. If the myocarditis recognition evaluation index is met, the corresponding model is recorded as the first myocarditis recognition model; otherwise, the model is reconstructed, and the first evaluation result is used to evaluate the model accuracy and recognition ability of the myocarditis recognition model to be evaluated; S5, performing myocarditis recognition on the acquired fusion modality coded data in combination with the first myocarditis recognition model to obtain a myocarditis recognition result of the person to be recognized; The specific evaluation method of the accuracy of the model is as follows: Analyzing the model training and model verification process of the myocarditis recognition model to be evaluated to obtain performance evaluation parameters of the myocarditis recognition model to be evaluated, wherein the performance evaluation parameters include accuracy, precision and recall rate; The accuracy evaluation index of the myocarditis recognition model to be evaluated is calculated according to the obtained performance evaluation parameters and in combination with the influence of the corresponding data on the accuracy evaluation index. The accuracy evaluation index is used to describe the accuracy of the myocarditis recognition model to be evaluated. The accuracy evaluation index is calculated using the following formula: ， In the formula, represents a natural constant, Indicates the number of the myocarditis recognition model to be evaluated, , is the total number of myocarditis recognition models to be evaluated, For the The accuracy evaluation index of the myocarditis recognition model to be evaluated, For the The accuracy of the myocarditis recognition model to be evaluated, is the model reference accuracy, is the model accuracy reference deviation, For the The accuracy of the myocarditis recognition model to be evaluated, is the model reference accuracy, is the model accuracy reference deviation, For the The recall rate of the myocarditis recognition model to be evaluated, is the model reference recall, is the model recall reference deviation, is the influence of absolute deviation of accuracy on the accuracy evaluation index, is the influence of absolute deviation of precision on the accuracy evaluation index, is the influence of the absolute deviation of recall rate on the accuracy evaluation index; The specific evaluation method of the recognition ability is as follows: Acquire recognition model data and recognition distribution score of the myocarditis recognition model to be evaluated, wherein the recognition model data includes data quality data and model complexity data, and the recognition distribution score is an evaluation result of the recognition ability data distribution of the myocarditis recognition model to be evaluated by a preset professional; A recognition capability evaluation index of the myocarditis recognition model to be evaluated is obtained according to the acquired recognition model data and the recognition distribution score, and the recognition capability evaluation index is used to describe the recognition capability of the myocarditis recognition model to be evaluated.

2. The method for identifying myocarditis based on multimodal images combined with machine learning as claimed in claim 1, characterized in that: The specific steps of the pretreatment are as follows: Analyze the data duplication of the corresponding data after the data annotation of the initial text data, and process it to obtain the first file data; Analyzing the data noise of the first text data after deduplication processing, and processing the data to obtain second file data; Analyze the data annotation of the second text data after the deduplication processing, and process it to obtain the third file data; Analyzing the data format of the third text data after the data-annotated error correction processing, and processing the data to obtain fourth file data; Analyzing the data normalization of the fourth text data after the format conversion processing, and processing the data to obtain fifth file data; The data size of the fifth text data after the normalization processing is analyzed and processed to obtain file data.

3. The method for identifying myocarditis based on multimodal images combined with machine learning as claimed in claim 1, characterized in that ,The specific steps of the multimodal fusion are as follows: Obtaining a first multimodal fusion result, where the first multimodal fusion result represents a result obtained by concatenating the acquired file data feature-extracted data in a preset order; Obtaining a second multimodal fusion result, where the second multimodal fusion result represents a result of converting the acquired file data feature-extracted data into corresponding feature vectors, and fusing the corresponding feature vectors using weighted summation; Obtaining a third multimodal fusion result, wherein the third multimodal fusion result represents a result of inputting the acquired file data after feature extraction into a preset deep neural network and then fusing the data through a preset specific structure; The acquired first multimodal fusion result, the second multimodal fusion result and the third multimodal fusion result are combined.

4. The method for identifying myocarditis based on multimodal images combined with machine learning as claimed in claim 1, characterized in that ,The specific method for obtaining the fusion modality file encoding data is as follows: Inputting the fusion modality file feature data into a corresponding preset data encoder to encode the feature data, and converting the data in combination with a corresponding encoding method to obtain the fusion modality file encoding data to be evaluated, wherein the fusion modality file encoding data to be evaluated includes image encoding data to be evaluated, audio encoding data to be evaluated, and text encoding data to be evaluated; The fused modality file encoding data to be evaluated is compared with the reference fused modality file encoding data to obtain a fused encoding data evaluation index. If the fused encoding data evaluation index meets the corresponding threshold, it indicates that the feature data encoding is successful, and the corresponding data is recorded as the fused modality file encoding data. Otherwise, the feature data encoding is performed again. The fused encoding data evaluation index is used to describe the feature data encoding situation.

5. The method for identifying myocarditis based on multimodal images combined with machine learning as claimed in claim 4, characterized in that: The specific method for obtaining the fusion coding data evaluation index is as follows: Obtaining a coded data vector of coded data of the fused modality file to be evaluated, wherein the coded data vector includes an image coded data vector, an audio coded data vector, and a text coded data vector; According to the acquired coding data vector, the fused coding data evaluation index is calculated in combination with the fused coding data weight, wherein the fused coding data weight represents the influence of the coding data vector on the fused coding data evaluation index, and the fused coding data weight includes the image coding data weight, the audio coding data weight and the text coding data weight. The fused coding data evaluation index is calculated using the following formula: ， In the formula, Indicates the number of the coded data of the fusion modality file to be evaluated. , is the total number of encoded data in the fusion modality file to be evaluated, Indicates The number of the image encoding data of the fused modality file encoding data to be evaluated, , For the The total number of image encoding data of the fusion modality file encoding data to be evaluated, Indicates The number of the audio coded data of the fused modality file coded data to be evaluated, , Indicates The number of the text encoding data of the fusion modality file encoding data to be evaluated, , For the The fusion coding data evaluation index of the fusion modality file coding data to be evaluated, For the The first of the coded data of the fusion modality files to be evaluated image encoding data vector of image encoding data, For the The first of the coded data of the fusion modality files to be evaluated A reference image coding data vector of image coding data, For the The first of the coded data of the fusion modality files to be evaluated An audio coded data vector of audio coded data, For the The first of the coded data of the fusion modality files to be evaluated A reference audio coded data vector of audio coded data, For the The first of the coded data of the fusion modality files to be evaluated A text encoding data vector of text encoding data, For the The first of the coded data of the fusion modality files to be evaluated A reference text encoding data vector of text encoding data, is the image encoding data weight, is the audio coding data weight, Encode data weights for text.

6. The method for identifying myocarditis based on multimodal images combined with machine learning as claimed in claim 1, characterized in that: The specific method for constructing the myocarditis recognition model to be evaluated is as follows: Step 1: construct a feature data set, where the feature data set is a collection of historical fusion modal coding data, and divide the feature data set into a training feature data set and a verification feature data set according to a preset ratio; Step 2: Obtain a training machine model, where the training machine model is a converged model obtained by training a preset machine learning model using a training feature data set; Step three, use the verification feature data set to verify the trained machine model to obtain the myocarditis recognition model to be evaluated.

7. The method for identifying myocarditis based on multimodal images combined with machine learning as claimed in claim 1, characterized in that: The specific method for obtaining the myocarditis recognition result of the person to be recognized is as follows: The myocarditis recognition result is obtained by analyzing the acquired fusion modality coding data in combination with the first myocarditis recognition model; The myocarditis identification results of the person to be identified are fed back to the corresponding preset medical staff for myocarditis situation analysis.

8. Myocarditis recognition system based on multimodal images combined with machine learning, characterized in that: include: Document data collection module, multimodal fusion module, model building module, model evaluation module and myocarditis identification module; The file data collection module is used to perform data annotation and preprocessing on the acquired initial file data to obtain file data, wherein the initial file data includes initial historical multimodal file data and initial multimodal file data; The multimodal fusion module is used to perform feature extraction and multimodal fusion on the obtained file data to obtain fused modality file feature data; The model building module is used to input the acquired fusion modality file feature data into a preset encoder to perform feature data encoding to obtain corresponding fusion modality file encoding data, and use the historical fusion modality encoding data to build a myocarditis recognition model to be evaluated, wherein the fusion modality encoding data is the result of multimodal fusion of image encoding data, audio encoding data and text encoding data, and the myocarditis recognition model to be evaluated is used to identify the myocarditis condition of the person to be identified; The model evaluation module is used to perform model evaluation on the constructed myocarditis recognition model to be evaluated to obtain a first evaluation result, and compare the first evaluation result with the myocarditis recognition evaluation index. If the myocarditis recognition evaluation index is met, the corresponding model is recorded as the first myocarditis recognition model, otherwise the model is reconstructed, and the first evaluation result is used to evaluate the model accuracy and recognition ability of the myocarditis recognition model to be evaluated; The myocarditis recognition module is used to perform myocarditis recognition on the acquired fusion modality coded data in combination with the first myocarditis recognition model to obtain a myocarditis recognition result of the person to be recognized; The specific evaluation method of the accuracy of the model is as follows: Analyzing the model training and model verification process of the myocarditis recognition model to be evaluated to obtain performance evaluation parameters of the myocarditis recognition model to be evaluated, wherein the performance evaluation parameters include accuracy, precision and recall rate; The accuracy evaluation index of the myocarditis recognition model to be evaluated is calculated according to the obtained performance evaluation parameters and in combination with the influence of the corresponding data on the accuracy evaluation index. The accuracy evaluation index is used to describe the accuracy of the myocarditis recognition model to be evaluated. The accuracy evaluation index is calculated using the following formula: ， In the formula, represents a natural constant, Indicates the number of the myocarditis recognition model to be evaluated, , is the total number of myocarditis recognition models to be evaluated, For the The accuracy evaluation index of the myocarditis recognition model to be evaluated, For the The accuracy of the myocarditis recognition model to be evaluated, is the model reference accuracy, is the model accuracy reference deviation, For the The accuracy of the myocarditis recognition model to be evaluated, is the model reference accuracy, is the model accuracy reference deviation, For the The recall rate of the myocarditis recognition model to be evaluated, is the model reference recall, is the model recall reference deviation, is the influence of absolute deviation of accuracy on the accuracy evaluation index, is the influence of absolute deviation of precision on the accuracy evaluation index, is the influence of the absolute deviation of recall rate on the accuracy evaluation index; The specific evaluation method of the recognition ability is as follows: Acquire recognition model data and recognition distribution score of the myocarditis recognition model to be evaluated, wherein the recognition model data includes data quality data and model complexity data, and the recognition distribution score is an evaluation result of the recognition ability data distribution of the myocarditis recognition model to be evaluated by a preset professional; A recognition capability evaluation index of the myocarditis recognition model to be evaluated is obtained according to the acquired recognition model data and the recognition distribution score, and the recognition capability evaluation index is used to describe the recognition capability of the myocarditis recognition model to be evaluated.

Images