A lightweight multi-modal in-vitro diagnosis information fusion analysis method and system

Abstract

The application discloses a kind of lightweight multi-modal in-vitro diagnosis information fusion analysis method and system, comprising: the data preprocessing of each modality in-vitro diagnosis data is extracted to obtain pre-processing feature;According to the pre-processing feature, different error classification cost initialization cost sensitive matrix is generated using cost-sensitive learning strategy;According to the initialization cost sensitive matrix and the output matrix of each modality under hidden layer, weighted negative correlation learning is carried out, and each modality negative correlation learning weight is obtained;According to each modality negative correlation learning weight, the classification result under each modality and each modality contribution score are calculated, and balanced processing is carried out according to each modality contribution score, and multi-modal data fusion analysis result is obtained;According to the multi-modal data fusion analysis result, disease screening and guidance suggestion are output.The application can effectively alleviate the problem of multi-modal data sample imbalance and modality contribution imbalance, improve data analysis precision.

Description

Technical Field

[0001] This invention relates to the field of medical data analysis technology, and in particular to a lightweight multimodal in vitro diagnostic information fusion analysis method and system. Background Technology

[0002] Traditional Chinese medicine (TCM) instruments, such as pulse diagnosis devices, tongue image analyzers, and electronic noses, have seen rapid development. These instruments enable rapid, convenient, non-invasive, and painless perception of various modalities of external diagnostic information, helping to reduce the risk of transmission of contact-borne infectious diseases. Simultaneously, utilizing artificial intelligence methods to process visual (tongue image / facial color), auditory (voice information), olfactory (exhaled breath concentration), and tactile (pulse wave) external diagnostic information facilitates early health monitoring and warning of diseases, aids in disease screening and diagnosis, and reduces the national economic burden.

[0003] Currently, single-modal in vitro diagnostic data analysis has achieved high classification accuracy in single-disease data classification problems. However, different symptoms of diseases have varying correlations with different modalities, resulting in an imbalance of contributions. Unlike single-modal disease diagnosis methods, multimodal fusion methods can fully explore the correlations and differences between modalities, further improving diagnostic accuracy and reliability. Meanwhile, multi-source heterogeneous in vitro diagnostic information based on visual, auditory, olfactory, and tactile sensory signals suffers from the sample class imbalance commonly found in health and medical data, further impacting the accuracy of disease data classification.

[0004] In the context of real-time diagnostic analysis of medical data, balancing diagnostic efficiency and accuracy has become a prominent challenge in in vitro diagnostic information analysis. Randomized ad hoc networks, as a novel neural network model, employ data-dependent supervised mechanisms to allocate node parameters and adaptively incrementally generate network structures. They not only possess high learning accuracy and efficiency but also do not rely on high-performance GPUs or TPUs for computation, making them easily implementable in embedded artificial intelligence and suitable for real-time cognitive scenarios involving in vitro diagnostic information. Therefore, designing a lightweight, multimodal in vitro diagnostic information fusion and analysis method and system is essential.

[0005] Existing deep learning models suffer from problems such as large number of parameters, complex model structure, and low training efficiency. In the process of multimodal medical data analysis, existing methods ignore the class imbalance and modality contribution imbalance of data samples, affecting the data classification accuracy. At the same time, direct concatenation and fusion can easily lead to the curse of dimensionality due to the increase in data dimensionality, resulting in feature redundancy.

[0006] Therefore, existing technologies still need improvement. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a lightweight multimodal in vitro diagnostic information fusion and analysis method and system to address the shortcomings of existing technologies, thereby solving the problem of low data classification accuracy caused by the imbalance of data sample categories and the imbalance of modal contributions in existing multimodal medical data analysis methods.

[0008] The technical solution adopted by this invention to solve the technical problem is as follows:

[0009] In a first aspect, the present invention provides a lightweight multimodal in vitro diagnostic information fusion and analysis method, including:

[0010] Data preprocessing was performed on the in vitro diagnostic data of each modality to extract preprocessed features;

[0011] Based on the preprocessed features, an initial cost-sensitive matrix with different misclassification costs is generated using a cost-sensitive learning strategy;

[0012] Based on the initial cost-sensitive matrix and the hidden layer output matrix of each modality, weighted negative correlation learning is performed to obtain the negative correlation learning weights for each modality.

[0013] The classification results and contribution scores of each modality are calculated based on the negative correlation learning weights of each modality, and the data is balanced based on the contribution scores of each modality to obtain the multimodal data fusion analysis results.

[0014] Based on the results of the multimodal data fusion analysis, disease screening and guidance suggestions are output.

[0015] In one implementation, the data preprocessing of the in vitro diagnostic data for each modality to extract preprocessed features includes:

[0016] Acquire in vitro diagnostic data for each modality;

[0017] Determine the feature extraction methods corresponding to the in vitro diagnostic data of each modality;

[0018] Based on the determined feature extraction method, features are extracted from the in vitro diagnostic data of each modality, and the extracted features are normalized to obtain the preprocessed features.

[0019] In one implementation, generating an initial cost-sensitive matrix with different misclassification costs using a cost-sensitive learning strategy based on the preprocessed features includes:

[0020] Based on the preprocessed features, a cost-sensitive learning strategy is used to minimize the total cost of the classifier, and an initial cost-sensitive matrix with different misclassification costs is generated in a random manner.

[0021] In one implementation, the step of performing weighted negative correlation learning based on the initialized cost-sensitivity matrix and the hidden layer output matrix for each modality to obtain the negative correlation learning weights for each modality includes:

[0022] The preprocessed features corresponding to the in vitro diagnostic information of each modality are input into multiple independent randomly configured network models, and the hidden layer output matrix of each modality is calculated.

[0023] The hidden layer output matrix for each mode is obtained based on the cost sensitivity matrix of each mode, and the obtained hidden layer output matrix is ​​used as the input of the weighted negative correlation learning framework.

[0024] With the goal of minimizing the cost function of the base model, a weighted negative correlation learning objective function is constructed and solved to obtain the negative correlation learning weights for each modality.

[0025] In one implementation, the step of calculating the classification results and contribution scores of each modality based on the negative correlation learning weights of each modality, and performing balancing processing based on the contribution scores of each modality to obtain the multimodal data fusion analysis results includes:

[0026] The classification results for each modality are calculated based on the negative correlation learning weights for each modality, and the prediction loss for each modality is also calculated.

[0027] Based on the prediction loss of each modality, a pseudo-label of the contribution score for each base model is generated;

[0028] A multimodal contribution learner based on a deep randomized network is constructed using the pseudo-labels of the contribution scores, and the contribution scores of each modality base model are learned through the multimodal contribution learner.

[0029] The results of multimodal data fusion analysis are obtained by balancing the contribution scores of each modal basis model to obtain the multimodal learning results.

[0030] In one implementation, the step of constructing a multimodal contribution learner based on a deep stochastic configuration network according to the pseudo-label of the contribution score, and learning the contribution scores of each modality base model through the multimodal contribution learner, includes:

[0031] Initialize the contribution scores for each modality and construct a multimodal contribution learner based on a deep stochastic configuration network with the goal of minimizing the loss;

[0032] The multimodal contribution learner learns the input data of each modality, and uses an incremental multi-layer construction method to iteratively generate the contribution scores of each modality base model. The softmax function is then used to normalize the contributions of each modality to obtain the normalized contribution scores of each modality base model.

[0033] In one implementation, the step of outputting disease screening and guidance suggestions based on the multimodal data fusion analysis results includes:

[0034] Based on the results of the multimodal data fusion analysis, the health and medical knowledge base is searched, and disease screening and guidance suggestions are output based on the search results.

[0035] Secondly, the present invention provides a lightweight multimodal in vitro diagnostic information fusion and analysis system, comprising:

[0036] The data preprocessing module is used to preprocess the in vitro diagnostic data of each modality and extract preprocessed features;

[0037] The cost-sensitive learning module is used to generate an initial cost-sensitive matrix with different misclassification costs based on the preprocessed features using a cost-sensitive learning strategy.

[0038] The weighted negative correlation learning module is used to perform weighted negative correlation learning based on the initial cost sensitivity matrix and the hidden layer output matrix under each mode, so as to obtain the negative correlation learning weights for each mode.

[0039] The multimodal contribution learner module is used to calculate the classification results and contribution scores of each modality based on the negative correlation learning weights of each modality, and to perform balancing processing based on the contribution scores of each modality to obtain the multimodal data fusion analysis results.

[0040] The disease screening and guidance suggestion module is used to output disease screening and guidance suggestions based on the results of the multimodal data fusion analysis;

[0041] An integrated sensing device system includes a visual sensor, an auditory sensor, an olfactory sensor, and a tactile sensor; wherein the visual sensor, the auditory sensor, the olfactory sensor, and the tactile sensor are respectively used to acquire corresponding sensor signals to obtain in vitro diagnostic data for each modality.

[0042] Thirdly, the present invention provides a terminal, comprising: a processor and a memory, wherein the memory stores a lightweight multimodal in vitro diagnostic information fusion analysis program, and the lightweight multimodal in vitro diagnostic information fusion analysis program, when executed by the processor, is used to implement the operation of the lightweight multimodal in vitro diagnostic information fusion analysis method as described in the first aspect.

[0043] Fourthly, the present invention also provides a medium, which is a computer-readable storage medium, storing a lightweight multimodal in vitro diagnostic information fusion analysis program, which, when executed by a processor, is used to implement the operation of the lightweight multimodal in vitro diagnostic information fusion analysis method as described in the first aspect.

[0044] The present invention, by employing the above technical solution, has the following effects:

[0045] This invention mitigates the impact of sample distribution on disease diagnosis results by extracting and normalizing features from in vitro diagnostic information of each modality, and learning cost-sensitive matrices for each modality's in vitro diagnostic information. Furthermore, it reduces the negative impact of imbalanced samples by utilizing the cost-sensitive matrices and output matrices learned by a stochastic network for each modality, learns the differences between modalities, and updates the output weights of each modality. By calculating the classification loss for each modality, it uses an adaptive incremental deep stochastic network to learn the contribution of each modality, updates the contribution scores of each modality, and obtains the adaptive weights and fusion analysis results for each modality. Based on the fusion analysis results and comparative analysis with a database, it provides users with disease screening and guidance suggestions. This invention effectively alleviates the problems of imbalanced multimodal data samples and imbalanced modal contributions, improving the accuracy of data analysis. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0047] Figure 1 This is a flowchart of the lightweight multimodal in vitro diagnostic information fusion and analysis method in this invention.

[0048] Figure 2 This is a flowchart of the lightweight multimodal in vitro diagnostic information fusion and analysis system of the present invention.

[0049] Figure 3 This is a flowchart of the data preprocessing module in this invention.

[0050] Figure 4 This is a flowchart of the cost-sensitive learning module in this invention.

[0051] Figure 5 This is a flowchart of the weighted negative correlation learning module in this invention.

[0052] Figure 6 This is a flowchart of the processing of the multimodal contribution learner module in this invention.

[0053] Figure 7 This is a flowchart of the disease screening and guidance module in this invention.

[0054] Figure 8This is a functional schematic diagram of the terminal in one implementation of the present invention.

[0055] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0057] Exemplary methods

[0058] Existing deep learning models suffer from problems such as large number of parameters, complex model structure, and low training efficiency. In the process of multimodal medical data analysis, existing methods ignore the class imbalance and modality contribution imbalance of data samples, affecting the data classification accuracy. At the same time, direct concatenation and fusion can easily lead to the curse of dimensionality due to the increase in data dimensionality, resulting in feature redundancy.

[0059] To address the above-mentioned technical problems, this invention provides a lightweight multimodal in vitro diagnostic information fusion and analysis method. This method preprocesses in vitro diagnostic data from each modality to extract preprocessed features. Based on these preprocessed features, an initial cost-sensitive matrix with different misclassification costs is generated using a cost-sensitive learning strategy. Weighted negative correlation learning is then performed on the initial cost-sensitive matrix and the hidden layer output matrix for each modality to obtain the negative correlation learning weights for each modality. The classification results and modality contribution scores for each modality are calculated based on these negative correlation learning weights, and a balancing process is performed based on the modality contribution scores to obtain the multimodal data fusion analysis results. Finally, disease screening and guidance suggestions are output based on the multimodal data fusion analysis results. Therefore, this invention can effectively alleviate the problems of imbalanced multimodal data samples and modality contribution imbalance, improving the accuracy of data analysis.

[0060] like Figure 1 As shown, this embodiment of the invention provides a lightweight multimodal in vitro diagnostic information fusion and analysis method, including the following steps:

[0061] Step S100: Perform data preprocessing on the in vitro diagnostic data of each modality and extract the preprocessed features.

[0062] In this embodiment, a lightweight multimodal in vitro diagnostic information fusion and analysis system is designed for the aforementioned lightweight multimodal in vitro diagnostic information fusion and analysis method. This system can improve the accuracy of disease classification based on in vitro diagnostic information. The system includes: a data preprocessing module, a cost-sensitive learning module, a weighted negative correlation learning module, a multimodal contribution learner module, a disease screening and guidance suggestion module, and an integrated sensing device system. The integrated sensing device system includes a visual sensor, an auditory sensor, an olfactory sensor, and a tactile sensor.

[0063] The visual sensor is used to acquire images of the patient's face and tongue to obtain visual signals; the auditory sensor is used to acquire the patient's vocalizations to obtain auditory signals; the olfactory sensor is used to acquire the concentration of the patient's exhaled gas to obtain olfactory signals; and the tactile sensor is used to acquire the patient's pulse wave signals to obtain tactile signals. The patient signals acquired by the above different sensors are called different modalities of in vitro diagnostic information. In this embodiment, the different modalities of in vitro diagnostic information are preprocessed by a data preprocessing module, which uses different feature extraction methods to learn the features of each modality of in vitro diagnostic information (preprocessed features).

[0064] Specifically, in one implementation of this embodiment, step S100 includes the following steps:

[0065] Step S101: Obtain in vitro diagnostic data for each modality;

[0066] Step S102: Determine the feature extraction method corresponding to the in vitro diagnostic data of each modality;

[0067] Step S103: Extract features from the in vitro diagnostic data of each modality according to the determined feature extraction method, and normalize the extracted features to obtain the preprocessed features.

[0068] In this embodiment, different feature extraction methods are used to learn the features of in vitro diagnostic information for different modalities, as follows:

[0069] 1) Visual signals: The face and tongue images were segmented using the large language segmentation inference model LISA. Then, the color and texture features of the face and tongue images were extracted using CIE Lab uniform color space clustering based on density peaks and Gabor wavelet transform.

[0070] 2) Auditory signals: Extract speech features such as Mel-frequency cepstral coefficients (MFCC), jitter, and shimmer from the auditory signals, and learn semantic features through Transformer, BERT, and GPT models.

[0071] 3) Olfactory signals: Extract geometric features such as waveform peaks, curve mean, and topological features, and learn time-frequency domain features through methods such as Fast Fourier Transform (FFT) and Wavelet Packet Transform (Wavelet).

[0072] 4) Tactile signals: Use Gabor wavelet transform, short-time Fourier transform (STFT), principal component analysis (PCA), wavelet packet transform (Wavelet), and wavelet packet energy spectrum to learn time-frequency domain, spectral entropy, and spectral band energy ratio characteristics.

[0073] After extracting data features for the corresponding modality using different feature extraction methods, the extracted features are normalized to obtain the normalized preprocessed features.

[0074] In this embodiment, by extracting the preprocessed features corresponding to the in vitro diagnostic data of each modality, cost-sensitive learning and weighted negative correlation learning can be performed based on these preprocessed features. Thus, based on the corresponding learning results, multimodal fusion analysis of the in vitro diagnostic features of each modality is performed to obtain a comprehensive analysis result based on multimodal in vitro diagnostic data.

[0075] like Figure 1 As shown, in one implementation of this invention, the lightweight multimodal in vitro diagnostic information fusion and analysis method further includes the following steps:

[0076] Step S200: Based on the preprocessed features, generate an initial cost-sensitive matrix with different misclassification costs using a cost-sensitive learning strategy.

[0077] In this embodiment, the preprocessed features obtained by the data preprocessing module under each modality are input into the cost-sensitive learning module, and the cost-sensitive learning module is used to generate an initial cost-sensitive matrix for different misclassification costs under each modality.

[0078] Specifically, in one implementation of this embodiment, step S200 includes the following steps:

[0079] Step S201: Based on the preprocessed features, minimize the total cost of the classifier using a cost-sensitive learning strategy, and generate an initial cost-sensitive matrix with different misclassification costs in a random manner.

[0080] In this embodiment, the total cost of the classifier is minimized by using a cost-sensitive learning strategy through a cost-sensitive learning module; wherein, the cost-sensitive learning strategy refers to providing a larger penalty weight when dealing with imbalanced classification problems, thereby obtaining an unbiased classification surface.

[0081] In this embodiment, in the cost-sensitive learning strategy, initial cost-sensitive matrices with different misclassification costs can be generated randomly:

[0082]

[0083] In the formula, s ij Indicates sample x i Misdiagnosed as X j The cost; i,j=1,2,...,N; N represents the number of samples; t=1,2,...,M,M represents the number of modes.

[0084] In this embodiment, the initial cost-sensitive matrix with different misclassification costs is generated by the cost-sensitive learning module, which can alleviate the impact of sample distribution on disease diagnosis results; then, based on the cost-sensitive matrix of each modality, the hidden layer output matrix of the randomly configured network model under each modality is obtained, and weighted negative correlation learning is performed using the hidden layer output matrix.

[0085] like Figure 1 As shown, in one implementation of this invention, the lightweight multimodal in vitro diagnostic information fusion and analysis method further includes the following steps:

[0086] Step S300: Perform weighted negative correlation learning based on the initial cost sensitivity matrix and the hidden layer output matrix under each modality to obtain the negative correlation learning weights for each modality.

[0087] In this embodiment, based on the initial cost-sensitive matrix, the weighted negative correlation learning module is used to learn the relevant weights of the preprocessed features of each modality of in vitro diagnostic information. In the weighted negative correlation learning module, the in vitro diagnostic information of each modality is input into the deep randomized network to obtain the hidden layer output matrix. The weights of the multimodal fusion model are calculated using the cost-sensitive matrix and the negative correlation learning strategy to obtain the calculation results of each modality.

[0088] Specifically, in one implementation of this embodiment, step S300 includes the following steps:

[0089] Step S301: Input the preprocessed features corresponding to the in vitro diagnostic information of each modality into multiple independent randomized network models, and calculate the hidden layer output matrix of each modality.

[0090] Step S302: Obtain the hidden layer output matrix for each mode based on the cost sensitivity matrix for each mode, and use the obtained hidden layer output matrix as the input of the weighted negative correlation learning framework;

[0091] Step S303: With the goal of minimizing the base model cost function, construct a weighted negative correlation learning objective function and solve it to obtain the negative correlation learning weights for each modality.

[0092] In this embodiment, multiple independent randomized network models are configured. By inputting the preprocessed features of in vitro diagnostic information for each modality into these multiple independent randomized network models, the hidden layer output matrix H for each modality can be calculated. t Simultaneously, based on the cost sensitivity matrix θ of each mode... t Obtain the hidden layer output matrix in each modality. The hidden layer output matrix can be used in each mode. As input to the weighted negative correlation learning module, the output weights of each modality are learned.

[0093] In the weighted negative correlation learning module, for each input modality, the hidden layer output matrix... Minimize the cost function of the base model (i.e., the stochastically configured network model) To achieve the objective (as shown in Equation (2)), construct and solve the weighted negative correlation learning objective function as shown in Equation (3) to obtain the output weights {B1, B2, ..., B} for each modality. M}, and obtain the diagnostic classification result P for each modality based on the output weights. t .

[0094]

[0095]

[0096] P t =H t B t ,t=1,2,...,M(4);

[0097] In the formula, This represents the weighted output matrix of the base model t; β represents the weighted label of the base model t; t Represents the original output weights of the base model; {B1,B2,...,B M} represents the output weight of the entire set; L is the total number of hidden nodes; γ represents the constraint parameter of the negative correlation learning penalty term.

[0098] In this embodiment, the weighted negative correlation learning module can reduce the negative impact of imbalanced samples by utilizing the cost-sensitive matrix and the output matrices of each modality learned by the randomly configured network, learn the differences between modalities, and update the output weights of each modality.

[0099] like Figure 1 As shown, in one implementation of this invention, the lightweight multimodal in vitro diagnostic information fusion and analysis method further includes the following steps:

[0100] Step S400: Calculate the classification results and contribution scores of each modality based on the negative correlation learning weights of each modality, and perform balancing processing based on the contribution scores of each modality to obtain the multimodal data fusion analysis results.

[0101] In this embodiment, the classification results of each modality are input into the multimodal contribution learner module to obtain the contribution weight of each modality and calculate the final multimodal data fusion analysis result.

[0102] Specifically, in one implementation of this embodiment, step S400 includes the following steps:

[0103] Step S401: Calculate the classification results for each modality based on the negative correlation learning weights for each modality, and calculate the prediction loss for each modality;

[0104] Step S402: Generate pseudo-labels for the contribution scores of each base model based on the prediction loss of each modality;

[0105] Step S403: Construct a multimodal contribution learner based on a deep randomized network according to the pseudo-label of the contribution score, and learn the contribution scores of each modality base model through the multimodal contribution learner.

[0106] In one implementation of this embodiment, the step of constructing a multimodal contribution learner based on a deep stochastic network according to the pseudo-label of the contribution score, and learning the contribution scores of each modality base model through the multimodal contribution learner, includes: initializing each modality contribution score, constructing a multimodal contribution learner based on a deep stochastic network with the goal of minimizing loss; learning each modal input data through the multimodal contribution learner, iteratively generating the contribution scores of each modality base model using an incremental multilayer construction method, and normalizing each modality contribution using a softmax function to obtain the normalized contribution scores of each modality base model.

[0107] Specifically, in one implementation of this embodiment, step S400 further includes the following steps:

[0108] Step S404: Balance the multimodal learning results based on the contribution scores of each modal basis model to obtain the multimodal data fusion analysis results.

[0109] In this embodiment, the output weights {B1,B2,...,B...} of each mode are obtained from the weighted negative correlation learning module. M} and the diagnostic classification results P for each modality t The balancing process is performed in the multimodal contribution learning module to obtain the final fusion analysis result of the preprocessed features of multimodal in vitro diagnostic information.

[0110] Specifically, in this embodiment, the classification result P for each modality is calculated based on the ensemble learning weights of negative correlations among the modalities. t (as shown in Equation (4)), and calculate the prediction loss λ for each mode. t =Loss(P t ,Y t The loss function can be calculated using the mean absolute error. Then, pseudo-labels for the contribution scores of each base model are generated based on the predicted loss, as shown in formula (5).

[0111]

[0112] In the formula, α represents a hyperparameter with a positive value, which controls the range of pseudo-label values ​​and prevents the denominator from being 0.

[0113] Then, initialize the contribution scores c of each mode. t =1 / M, to minimize To achieve this goal, a multimodal contribution learner based on a deep stochastically configured network is constructed to learn from the input data of each modality. An incremental multilayer construction method (which involves continuously increasing the number of hidden layers and updating the contributions) is used to iteratively generate the contribution scores c of the base models for each modality. t The softmax function is used to normalize the modal contributions.

[0114]

[0115] In the formula, exp represents an exponential function with base e of the natural logarithm;

[0116] Finally, contribution scores can effectively reflect the classification confidence of each modality. Based on the contribution scores of each modality... By balancing the multimodal learning results (i.e. the final fusion analysis results) as shown in the definition formula (7), the problem of unbalanced modal contributions in the disease diagnosis process can be effectively alleviated, and the accuracy of the final health calculation can be improved by utilizing more discriminative modalities.

[0117]

[0118] In this embodiment, the multimodal contribution learner model can calculate the classification loss of each modality, learn the contribution of each modality using an adaptive incremental deep stochastic configuration network, update the contribution score of each modality, and obtain the adaptive weight of each modality, thereby obtaining the fusion analysis results of the preprocessed features of multimodal in vitro diagnostic data.

[0119] like Figure 1 As shown, in one implementation of this invention, the lightweight multimodal in vitro diagnostic information fusion and analysis method further includes the following steps:

[0120] Step S500: Output disease screening and guidance suggestions based on the results of the multimodal data fusion analysis.

[0121] In this embodiment, after obtaining the results of multimodal data fusion analysis, the data is input into the disease screening and guidance module to provide patients with health advice and guidance.

[0122] Specifically, in one implementation of this embodiment, step S500 includes the following steps:

[0123] Step S501: Based on the results of the multimodal data fusion analysis, search the health and medical knowledge base, and output the disease screening and guidance suggestions based on the search results.

[0124] In the disease screening and guidance module, the results of multimodal data fusion analysis are uploaded to the health and medical knowledge base for further evaluation of the data analysis results. Based on the results, disease screening and guidance suggestions are provided (for example, based on the analyzed disease type and database information, guidance suggestions on diet, follow-up examination, and medical treatment are provided).

[0125] The technical solution in this embodiment will be illustrated below through specific application examples:

[0126] like Figure 2 As shown, the overall flow of the lightweight multimodal in vitro diagnostic information fusion and analysis method in this embodiment includes the following steps:

[0127] S10, input visual signals, auditory signals, olfactory signals and tactile signals;

[0128] S20, Data Preprocessing;

[0129] S30, cost-sensitive learning;

[0130] S40, weighted negative correlation learning;

[0131] S50, obtain visual classification results, auditory classification results, olfactory classification results and tactile classification results;

[0132] S60 performs contribution learning based on the classification results.

[0133] S70, Multimodal Adaptive Fusion Analysis;

[0134] S80, Disease Screening and Guidance Recommendations.

[0135] like Figure 3 As shown, steps S10 to S20 above are processed by the data preprocessing module, specifically including:

[0136] S21, Input in vitro diagnostic information for each modality;

[0137] S22, Determine feature extraction methods for different signals;

[0138] S23, Feature normalization;

[0139] S24, Output the preprocessing results.

[0140] In this embodiment, the above data preprocessing module processes the in vitro diagnostic information of each modality to obtain the preprocessed features corresponding to the in vitro diagnostic information of each modality.

[0141] like Figure 4 As shown, step S30 above is processed by a cost-sensitive learning module, specifically including:

[0142] S31, Input the preprocessing features for each modality;

[0143] S32, a cost-sensitive learning strategy;

[0144] S33, outputs the cost-sensitive matrix;

[0145] In this embodiment, for the preprocessed features corresponding to each modality of in vitro diagnostic information, the cost-sensitive learning module outputs the corresponding cost-sensitive matrix.

[0146] like Figure 5 As shown, steps S40 to S50 above are processed through a weighted negative correlation learning module, specifically including:

[0147] S41, Input the preprocessing features for each modality;

[0148] S42, random configuration network processing;

[0149] S43, obtain the hidden layer output matrix;

[0150] S44, based on cost-sensitive weighted negative correlation ensemble learning;

[0151] S45 outputs the classification results for each modality.

[0152] In this embodiment, the cost-sensitivity matrix corresponding to the in vitro diagnostic information of each modality is uniformly processed by a weighted negative correlation learning module to output the classification results of each modality.

[0153] like Figure 6 As shown, steps S60 to S70 above are processed by the multimodal contribution learner module, specifically including:

[0154] S51, Input the classification results for each modality;

[0155] S52, calculate the prediction loss for each modality;

[0156] S53, Construct a multimodal contribution learner based on a deep randomized network;

[0157] S54, calculate the adaptive weights of modal contributions;

[0158] S55 outputs the fusion result.

[0159] In this embodiment, for each modality classification result, the modality contribution adaptive weight is calculated by a multimodal contribution learner based on a deep randomized network, and then the multimodal learning results are balanced according to the modality contribution adaptive weight to obtain the multimodal data fusion analysis results.

[0160] like Figure 7 As shown, step S80 above is processed through the disease screening and guidance module, specifically including:

[0161] S61, Input the fusion result;

[0162] S62, Search the health and medical knowledge base;

[0163] S63 provides disease screening and guidance recommendations.

[0164] This embodiment uses Alzheimer's disease as an example. Research results from both Western and Traditional Chinese Medicine indicate that patients' multimodal in vitro diagnostic information, such as facial features, tongue appearance, voice, exhaled breath, and pulse waves, differs from that of healthy individuals. Therefore, disease data analysis based on the fusion of multimodal in vitro diagnostic information is feasible. Furthermore, utilizing a lightweight multimodal in vitro diagnostic information fusion analysis method and system, the collection of multimodal in vitro diagnostic information can be achieved. Through data preprocessing modules, cost-sensitive learning modules, weighted negative correlation learning modules, and multimodal contribution learner modules, key disease information can be learned fully based on interpretable features, overcoming sample imbalance and modal contribution imbalance problems, improving disease diagnostic accuracy. The disease screening and guidance suggestion module enables symptom analysis and report printing.

[0165] This embodiment achieves the following technical effects through the above technical solution:

[0166] This embodiment mitigates the impact of sample distribution on disease diagnosis results by extracting and normalizing features from in vitro diagnostic information of each modality and learning cost-sensitive matrices for each modality. Furthermore, it reduces the negative impact of imbalanced samples by utilizing the cost-sensitive matrices and output matrices learned by a stochastic network, learns the differences between modalities, and updates the output weights of each modality. By calculating the classification loss for each modality, it uses an adaptive incremental deep stochastic network to learn the contribution of each modality, updates the contribution scores of each modality, and obtains the adaptive weights and fusion analysis results for each modality. Based on the fusion analysis results and comparative analysis with the database, it provides users with disease screening and guidance suggestions. This embodiment effectively alleviates the problems of imbalanced multimodal data samples and imbalanced modal contributions, improving the accuracy of data analysis.

[0167] Exemplary device

[0168] Based on the above embodiments, the present invention also provides a lightweight multimodal in vitro diagnostic information fusion and analysis system, comprising:

[0169] The data preprocessing module is used to preprocess the in vitro diagnostic data of each modality and extract preprocessed features;

[0170] The cost-sensitive learning module is used to generate an initial cost-sensitive matrix with different misclassification costs based on the preprocessed features using a cost-sensitive learning strategy.

[0171] The weighted negative correlation learning module is used to perform weighted negative correlation learning based on the initial cost sensitivity matrix and the hidden layer output matrix under each mode, so as to obtain the negative correlation learning weights for each mode.

[0172] The multimodal contribution learner module is used to calculate the classification results and contribution scores of each modality based on the negative correlation learning weights of each modality, and to perform balancing processing based on the contribution scores of each modality to obtain the multimodal data fusion analysis results.

[0173] The disease screening and guidance suggestion module is used to output disease screening and guidance suggestions based on the results of the multimodal data fusion analysis;

[0174] An integrated sensing device system includes a visual sensor, an auditory sensor, an olfactory sensor, and a tactile sensor; wherein the visual sensor, the auditory sensor, the olfactory sensor, and the tactile sensor are respectively used to acquire corresponding sensor signals to obtain in vitro diagnostic data for each modality.

[0175] This embodiment achieves the following technical effects through the above technical solution:

[0176] This embodiment mitigates the impact of sample distribution on disease diagnosis results by extracting and normalizing features from in vitro diagnostic information of each modality and learning cost-sensitive matrices for each modality. Furthermore, it reduces the negative impact of imbalanced samples by utilizing the cost-sensitive matrices and output matrices learned by a stochastic network, learns the differences between modalities, and updates the output weights of each modality. By calculating the classification loss for each modality, it uses an adaptive incremental deep stochastic network to learn the contribution of each modality, updates the contribution scores of each modality, and obtains the adaptive weights and fusion analysis results for each modality. Based on the fusion analysis results and comparative analysis with the database, it provides users with disease screening and guidance suggestions. This embodiment effectively alleviates the problems of imbalanced multimodal data samples and imbalanced modal contributions, improving the accuracy of data analysis.

[0177] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 8 As shown.

[0178] The terminal includes: a processor, a memory, an interface, a display screen, and a communication module connected via a system bus; wherein, the processor of the terminal provides computing and control capabilities; the memory of the terminal includes a storage medium and internal memory; the storage medium stores the operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the storage medium; the interface is used to connect to external devices; the display screen is used to display relevant information; and the communication module is used to communicate with a cloud server or other devices.

[0179] When executed by a processor, this computer program is used to implement a lightweight, multimodal in vitro diagnostic information fusion and analysis method.

[0180] It will be understood by those skilled in the art that Figure 8 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0181] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing a lightweight multimodal in vitro diagnostic information fusion analysis program, which, when executed by the processor, is used to implement the operation of the lightweight multimodal in vitro diagnostic information fusion analysis method described above.

[0182] In one embodiment, a storage medium is provided, wherein the storage medium stores a lightweight multimodal in vitro diagnostic information fusion analysis program, which, when executed by a processor, is used to implement the operation of the lightweight multimodal in vitro diagnostic information fusion analysis method described above.

[0183] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, database, or other media used in the embodiments provided by this invention can include both non-volatile and volatile memory.

[0184] In summary, this invention provides a lightweight multimodal in vitro diagnostic information fusion analysis method and system, comprising: preprocessing in vitro diagnostic data of each modality to extract preprocessed features; generating an initial cost-sensitive matrix with different misclassification costs using a cost-sensitive learning strategy based on the preprocessed features; performing weighted negative correlation learning based on the initial cost-sensitive matrix and the hidden layer output matrix of each modality to obtain the negative correlation learning weights for each modality; calculating the classification results and modality contribution scores for each modality based on the negative correlation learning weights for each modality, and performing balancing processing based on the modality contribution scores to obtain the multimodal data fusion analysis results; and outputting disease screening and guidance suggestions based on the multimodal data fusion analysis results. This invention can effectively alleviate the problems of multimodal data sample imbalance and modality contribution imbalance, and improve the accuracy of data analysis.

[0185] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A lightweight, multimodal in vitro diagnostic information fusion and analysis method, characterized in that, include: Data preprocessing was performed on the in vitro diagnostic data of each modality to extract preprocessed features; Based on the preprocessed features, an initial cost-sensitive matrix with different misclassification costs is generated using a cost-sensitive learning strategy; Based on the initial cost-sensitive matrix and the hidden layer output matrix of each modality, weighted negative correlation learning is performed to obtain the negative correlation learning weights for each modality. This includes: inputting the preprocessed features corresponding to the in vitro diagnostic information of each modality into multiple independent randomly configured network models, and calculating the hidden layer output matrix of each modality. Based on the cost sensitivity matrix of each mode Obtain the hidden layer output matrix in each mode and the obtained hidden layer output matrix As input to the weighted negative correlation learning framework; with the goal of minimizing the base model cost function, construct the weighted negative correlation learning objective function and solve it to obtain the negative correlation learning weights for each modality; The classification results for each modality are calculated based on the negative correlation learning weights of each modality. The prediction loss for each modality is calculated. The contribution score of each modality is calculated based on the prediction loss of each modality. The contribution scores of each modality are then balanced to obtain the multimodal data fusion analysis results. Based on the results of the multimodal data fusion analysis, disease screening and guidance suggestions are output.

2. The lightweight multimodal in vitro diagnostic information fusion and analysis method according to claim 1, characterized in that, The data preprocessing of in vitro diagnostic data for each modality, and the extraction of preprocessed features, includes: Acquire in vitro diagnostic data for each modality; Determine the feature extraction methods corresponding to the in vitro diagnostic data of each modality; Based on the determined feature extraction method, features are extracted from the in vitro diagnostic data of each modality, and the extracted features are normalized to obtain the preprocessed features.

3. The lightweight multimodal in vitro diagnostic information fusion and analysis method according to claim 1, characterized in that, The step of generating an initial cost-sensitive matrix with different misclassification costs using a cost-sensitive learning strategy based on the preprocessed features includes: Based on the preprocessed features, a cost-sensitive learning strategy is used to minimize the total cost of the classifier, and an initial cost-sensitive matrix with different misclassification costs is generated in a random manner.

4. The lightweight multimodal in vitro diagnostic information fusion and analysis method according to claim 1, characterized in that, The process involves calculating the contribution score of each modality based on the prediction loss of each modality, and then performing a balancing process based on the contribution scores of each modality to obtain the multimodal data fusion analysis results, including: A contribution score pseudo-label is generated for each base model based on the prediction loss of each modality; wherein the contribution score pseudo-label is a weight calculated based on the prediction loss. A multimodal contribution learner based on a deep randomized network is constructed using the pseudo-labels of the contribution scores, and the contribution scores of each modality base model are learned through the multimodal contribution learner. The results of multimodal data fusion analysis are obtained by balancing the contribution scores of each modal basis model to obtain the multimodal learning results.

5. The lightweight multimodal in vitro diagnostic information fusion and analysis method according to claim 4, characterized in that, The step of constructing a multimodal contribution learner based on a deep stochastic configuration network according to the pseudo-label of the contribution score, and learning the contribution scores of each modality base model through the multimodal contribution learner, includes: Initialize the contribution scores for each modality and construct a multimodal contribution learner based on a deep stochastic configuration network with the goal of minimizing the loss; The multimodal contribution learner learns the input data of each modality, and uses an incremental multi-layer construction method to iteratively generate the contribution scores of each modality base model. The softmax function is then used to normalize the contributions of each modality to obtain the normalized contribution scores of each modality base model.

6. The lightweight multimodal in vitro diagnostic information fusion and analysis method according to claim 1, characterized in that, The step of outputting disease screening and guidance suggestions based on the results of the multimodal data fusion analysis includes: Based on the results of the multimodal data fusion analysis, the health and medical knowledge base is searched, and disease screening and guidance suggestions are output based on the search results.

7. A lightweight multimodal in vitro diagnostic information fusion and analysis system, used to implement the lightweight multimodal in vitro diagnostic information fusion and analysis method as described in any one of claims 1-6, characterized in that, include: The data preprocessing module is used to preprocess the in vitro diagnostic data of each modality and extract preprocessed features; The cost-sensitive learning module is used to generate an initial cost-sensitive matrix with different misclassification costs based on the preprocessed features using a cost-sensitive learning strategy. The weighted negative correlation learning module is used to perform weighted negative correlation learning based on the initial cost sensitivity matrix and the hidden layer output matrix under each mode, so as to obtain the negative correlation learning weights for each mode. The multimodal contribution learner module is used to calculate the classification results and contribution scores of each modality based on the negative correlation learning weights of each modality, and to perform balancing processing based on the contribution scores of each modality to obtain the multimodal data fusion analysis results. The disease screening and guidance suggestion module is used to output disease screening and guidance suggestions based on the results of the multimodal data fusion analysis; An integrated sensing device system includes a visual sensor, an auditory sensor, an olfactory sensor, and a tactile sensor; wherein the visual sensor, the auditory sensor, the olfactory sensor, and the tactile sensor are respectively used to acquire corresponding sensor signals to obtain in vitro diagnostic data for each modality.

8. A terminal, characterized in that, include: The processor and memory, wherein the memory stores a lightweight multimodal in vitro diagnostic information fusion analysis program, which, when executed by the processor, is used to implement the operation of the lightweight multimodal in vitro diagnostic information fusion analysis method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a lightweight multimodal in vitro diagnostic information fusion analysis program, which, when executed by a processor, is used to implement the operation of the lightweight multimodal in vitro diagnostic information fusion analysis method as described in any one of claims 1-6.

Images