Three-branch diagnosis and evaluation method and device based on multimodal telemedicine data

Abstract

The present invention relates to the field of auxiliary medical diagnosis technology, and specifically discloses a three-branch diagnosis and evaluation method and device based on multimodal telemedicine data. By collecting the patient's multimodal telemedicine data, it is input into the three-branch diagnosis and evaluation model for analysis. If the diagnosis result is clear, that is, it is confirmed whether the patient is ill, the corresponding result is directly output; if the diagnosis result is unclear, the patient's disease risk is further evaluated and the evaluation result is output. This method can not only effectively diagnose the patient's diseased and non-disease status, but also break through the limitation of the traditional intelligent diagnosis model that cannot accurately evaluate the risk of disease when the patient's disease status is uncertain or does not fully meet the diagnostic criteria.

Description

Technical Field

[0001] The present application relates to the technical field of auxiliary medical diagnosis, and specifically discloses a three-branch diagnosis and evaluation method and device based on multimodal telemedicine data. Background Art

[0002] Although clinical diagnosis and treatment methods have gradually improved with the rapid development of medical technology, doctors still face huge challenges in diagnosis due to the multimodal characteristics of patient information (such as voice, text, physiological signals, etc.) and the complexity of the disease. Especially in the telemedicine environment, doctors lack the opportunity to have direct contact with patients, and it is more difficult to make accurate diagnoses based solely on remotely collected medical data.

[0003] Therefore, how to use modern intelligent technology and combine multimodal remote data to develop a method that can effectively assist doctors in intelligent diagnosis and evaluation has become an important research direction in the medical field.

[0004] Most of the current mainstream intelligent diagnosis models are binary classification models, which classify patients into two categories: "sick" or "healthy". This binary classification method has significant limitations in actual clinical applications, especially for patients with uncertain disease status and those who do not fully meet the diagnostic criteria. The model either mistakenly judges them as healthy or ill, increasing clinical risks.

[0005] In the existing technology, when encountering patients who cannot be clearly classified, there is a lack of intelligent auxiliary tools that can further assess their disease risk. This makes it difficult for doctors to obtain effective auxiliary diagnosis suggestions from the model when facing complex or uncertain cases. Therefore, there is an increasing demand for fuzzy assessment of these patients to better provide doctors with judgments based on risk probability and reduce the possibility of misdiagnosis or missed diagnosis.

[0006] In view of this, the inventor provides a three-branch diagnosis and evaluation method and device based on multimodal telemedicine data to solve the above problems. Summary of the invention

[0007] The purpose of the present invention is to achieve effective diagnosis of the patient's diseased or non-disease status, while breaking through the limitations of traditional intelligent diagnostic models that cannot accurately assess the risk of disease when the patient's disease status is unclear or does not fully meet the diagnostic criteria.

[0008] In order to achieve the above object, the basic scheme of the present invention provides a three-branch diagnosis and evaluation method based on multimodal telemedicine data, and the three-branch diagnosis and evaluation method based on multimodal telemedicine data includes the following steps:

[0009] By collecting multimodal telemedicine data from patients, it is input into a three-branch diagnosis and assessment model for analysis;

[0010] If the diagnosis result is clear, that is, it is confirmed whether the patient is ill, the corresponding result will be directly output; if the diagnosis result is unclear, the patient's risk of illness will be further evaluated and the evaluation result will be output;

[0011] The method for constructing the three-branch diagnosis and evaluation model includes the following steps:

[0012] Step S1: collect data and preprocess the collected data to obtain a multimodal telemedicine dataset;

[0013] Step S2: cluster the multimodal telemedicine dataset using the shadow set clustering algorithm to obtain R fuzzy clusters, establish a partition threshold for each fuzzy cluster to divide the fuzzy cluster into a core domain, a shadow domain, and an exclusion domain, and construct a sphere with the core domain of each fuzzy cluster to obtain R spheres;

[0014] Step S3: Identify and find the spheres in the overlapping state and the nested state, and perform de-overlapping and de-nesting operations on the spheres in the overlapping state and the nested state respectively to obtain refined spheres;

[0015] Step S4: Identify and find abnormal spheres in the refined spheres, delete the abnormal spheres, and obtain spheres with specific semantic information: specific spheres;

[0016] Step S5: Establish a stable set of specific granules, and construct three classification rules based on the specific granules, thereby constructing three intelligent classification and evaluation models based on the specific granules.

[0017] Further, in step S2, R is taken as , N is the number of samples in the multimodal telemedicine dataset;

[0018] The partition threshold is calculated according to the optimization principle. The fuzzy cluster is divided into core domain, shadow domain and exclusion domain according to the partition threshold. The granular ball constructed by the core domain of the i-th fuzzy cluster is represented as GB i =(C i ,r i ,O i ), where C i Indicates GB i The center of i Indicates GB i Radius, O i Indicates included in GB i Internal samples.

[0019] Further, in the implementation process of step S3, the following steps are included:

[0020] Step S31: Identification and processing of overlapping spheres;

[0021] Step S311: Identify overlapping spheres, and generate sphere GB for the i-th fuzzy cluster i =(C i ,r i ,O i ) and the granular ball GB generated by the jth fuzzy cluster j =(C j ,r j ,O j ), when the two balls satisfy:

[0022] ;

[0023] Then it means the sphere GB i GB j There is an overlapping relationship;

[0024] Step S312: Adjust the radius of the two overlapping balls so that the two balls are separated. The radius of the two balls after adjustment are:

[0025] ;

[0026] ;

[0027] in, Represents the distance between the centers of the two spheres;

[0028] Step S32: identification and processing of nested pellets;

[0029] Step S321: Identify the nested balls. For the ball GB i =(C i ,r i ,O i ) and ball GB j =(C j ,r j ,O j ), when the two balls satisfy:

[0030] ;

[0031] Then it means the sphere GB i GB j There is a nested relationship;

[0032] Step S322: Merge two balls in a nested relationship. The merged ball is recorded as GB. mer =(C mer ,r mer ,Omer ),in

[0033] ;

[0034] ;

[0035] ;

[0036] in, Represents the distance between the samples in the two spheres and the center of the merged sphere. and Represents the sphere GB i and GB j The number of samples in x k represents the kth sample.

[0037] Further, in the implementation process of step S4, the following steps are included:

[0038] Step S41: Identification of abnormal spheres: for the sphere GB generated by the i-th fuzzy cluster i ,

[0039] When the identified ball GB i =(C i ,r i ,O i ) i >2r, it means that the identified sphere is a super sphere, where r represents the average value of the radius of all spheres obtained after step S3;

[0040] When the identified ball GB i =(C i ,r i ,O i ) i ≦1 or r i =0, it means that the identified sphere is a solitary sphere;

[0041] When the identified ball GB i =(C i ,r i ,O i ) i When <θ, it means that the identified ball is an impure ball, where P i It represents the proportion of samples of any category inside the identified sphere to the total samples, θ∈[0,1];

[0042] Step S42: Processing abnormal spheres, deleting the identified super spheres, isolated spheres and impure spheres, and obtaining spheres with specific semantic information: specific spheres.

[0043] Further, in the implementation process of step S5, the following steps are included:

[0044] Step S51: Establish a stable collection of specific spheres;

[0045] Step S52: Determine three classification rules based on pellets and balls;

[0046] Step S53: construct three intelligent classification and evaluation models based on particles and balls.

[0047] Further, in step S51, the center set of the first obtained specific granules is expressed as:

[0048] ;

[0049] in, represents the center of the specific sphere obtained after the operations of step S2, step S3, and step S4, R' represents the number of newly formed specific spheres, and R'≦R;

[0050] The center set center is used as the initial clustering center of the multimodal telemedicine dataset, and steps S2 to S4 are iterated until the generated spheres do not intersect with each other, that is, a stable set of specific spheres is generated. At this time, R new Each ball is represented by:

[0051] GB new =(C new ,r new ,O new ),

[0052] Among them, C new Represents GB new The center of new GB new The radius, O new is included in GB new Internal samples.

[0053] Further, in step S52, the three classification rules include:

[0054] Rule 1: If a test sample x t In the ball GB new The interior of

[0055] ,

[0056] Then the label of the sample to be tested is the label of the sphere, that is, the labels of most similar samples in the sphere, where t=1,2,…;

[0057] Rule 2: If a sample x to be tested tNot in any ball GB new The interior of the sample x does not belong to any sphere. t In the ball GB new The outside of the sample to be tested x is calculated t The activation membership of each sphere:

[0058] ;

[0059] Among them, e represents a specific sphere GB e =(C e ,r e ,O e );

[0060] At the same time, let u et and u bt Represents the sample x to be tested t The two maximum memberships of the sphere and u bt >u et ,

[0061] (a) If GB e and GB b If they are two balls of the same category, the label of the sample to be tested is ball GB e and GB b label b ;

[0062] (b) If GB e and GB b are two spheres of different categories that satisfy

[0063] u bt -u et > δ;

[0064] Where δ∈[0,1], the label of the sample to be tested is label b ;

[0065] (c) Otherwise, the sample to be tested is divided into the shadow area, which has a negative effect on label b The membership degree is u bt ;

[0066] The above rules are interpreted in the telemedicine diagnosis scenario as follows: Based on the multimodal telemedicine data of the subject,

[0067] If rule 1 is satisfied, that is, the subject as the sample to be tested is located inside the sphere, then whether the patient is sick is determined by the label of the sphere in which he is located;

[0068] If rule 2(a) is satisfied, the diagnostic result is label b ;

[0069] If rule 2(b) is satisfied, then the diagnosis result is also label b ;

[0070] If rule 2(c) is satisfied, the subject is assigned to the shadow domain and the label is output. b The membership degree of label b If u bt Indicates the risk level of the subject's disease, otherwise 1-u bt Indicates the risk level of the subject to develop the disease.

[0071] The basic scheme of the present invention also provides a three-branch diagnosis and evaluation device based on multimodal telemedicine data, including a data acquisition module, a data processing module, a model building module, a model training module and a diagnosis and evaluation module for executing the three-branch diagnosis and evaluation method based on multimodal telemedicine data as described above.

[0072] The principle and effect of this scheme are:

[0073] 1. Compared with the prior art, the present invention can improve the accuracy of diagnosis: through the fusion processing of multimodal data, the present invention can make full use of various medical information such as voice data, physiological signals and text records, and combine the three-branch intelligent classification model to significantly improve the accuracy of diagnosis of the patient's condition, and provide a disease assessment for patients who are uncertain whether they are sick or not. This method can not only effectively diagnose the patient's diseased and non-disease status, but also break through the limitation of traditional intelligent diagnostic models that cannot accurately assess the risk of disease when the patient's disease status is uncertain or does not fully meet the diagnostic criteria.

[0074] 2. Compared with the prior art, the present invention can enhance the robustness of the diagnostic model: the present invention generates and refines granular balls, identifies and removes overlapping, nested and abnormal granular balls, effectively reduces the impact of data redundancy and noise, and ensures the robustness of the model and its application reliability in complex medical data scenarios.

[0075] 3. Compared with the prior art, the present invention is suitable for telemedicine scenarios: the present invention can obtain the patient's multimodal medical information through a remote data acquisition module, and provide instant and accurate diagnosis services in combination with an intelligent classification model, which greatly reduces the need for face-to-face diagnosis between patients and doctors and enhances the practicality and effectiveness of telemedicine. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0077] Figure 1 A flowchart of a three-branch diagnosis and evaluation method based on multimodal telemedicine data proposed in an embodiment of the present application is shown;

[0078] Figure 2 The multimodal data feature extraction and feature selection technology roadmap proposed in the embodiment of the present application is shown;

[0079] Figure 3 A schematic diagram of a three-branch diagnosis and evaluation device based on multimodal telemedicine data proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0080] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0081] Three-pronged diagnosis and evaluation method based on multimodal telemedicine data, implementing e.g. Figure 1 As shown: The specific steps include:

[0082] By collecting the patient's multimodal telemedicine data, it is input into the three-branch diagnosis and evaluation model for analysis. If the diagnosis result is clear, that is, confirming whether the patient is sick, the corresponding result is directly output; if the diagnosis result is unclear, the patient's disease risk is further evaluated and the evaluation result is output.

[0083] Specifically, taking the multimodal telemedicine data of Parkinson's disease patients, the multimodal telemedicine data of diabetes patients, and the multimodal telemedicine data of chronic obstructive pulmonary disease (COPD) patients as the input of the three diagnosis and evaluation models, the specific implementation is as follows:

[0084] Collect multimodal telemedicine data from Parkinson's patients, including motion data collected through wearable devices; voice data reflecting the patient's speech speed, intonation and pronunciation changes; behavioral data collected by using smart home sensors; and subjective descriptions, diary records or symptom report text data of the patient, etc. The collected data are input into three diagnosis and evaluation models to diagnose the patient. If the diagnosis result is clear, that is, it is confirmed that the patient has Parkinson's disease or not, the diagnosis result is directly output; if the diagnosis result is unclear, the assessment result of the patient's risk of Parkinson's disease is output.

[0085] Collect multimodal telemedicine data from diabetic patients, including physiological data such as blood sugar, blood pressure, and heart rate collected through portable devices; lifestyle data that records patients' eating habits and exercise conditions; daily activity levels and behavioral data monitored by smart devices, as well as text data such as patients' subjective descriptions, diet records, or symptom reports. Input the collected data into the three-branch diagnosis and evaluation model to diagnose the patient. If the diagnosis result is clear, that is, it is confirmed that the patient has diabetes or not, the diagnosis result is directly output; if the diagnosis result is unclear, the assessment result of diabetes risk or disease progression risk is output.

[0086] Collect multimodal telemedicine data from patients with chronic obstructive pulmonary disease (COPD), including physiological data such as respiratory rate, oxygen saturation, and peak expiratory flow rate collected by portable pulmonary function monitoring devices; patients' lifestyle data, such as exercise intensity and frequency; behavioral data such as daily activity level and sleep quality monitored by smart home sensors, as well as text data such as patients' subjective descriptions, dyspnea records, or symptom reports. Input the collected data into the three-branch diagnosis and evaluation model to diagnose the patient. If the diagnosis is clear, that is, it is confirmed that the patient has COPD or not, the diagnosis result is directly output; if the diagnosis is unclear, the assessment result of the patient's COPD risk or the risk of severe disease is output.

[0087] In this embodiment, the method for constructing the three-branch diagnosis and evaluation model includes the following steps:

[0088] Step S1: collect data and preprocess the collected data to obtain a multimodal telemedicine dataset.

[0089] like Figure 2 As shown, the specific steps are as follows:

[0090] Step S11: Collecting data. This includes collecting multimodal telemedicine information of patients obtained through remote data sources, online medical platforms, or human-computer interaction methods, including but not limited to patient voice data, physiological signals, text records, and other data. Multimodal telemedicine data can come from different medical sensors or patient interaction platforms, covering a variety of medical features to provide a comprehensive description of health status;

[0091] Step S12: perform data preprocessing, including operations such as feature extraction, feature reduction, and data normalization, and finally generate a multimodal telemedicine dataset for the patient. Including:

[0092] The preprocessing of the speech data, specifically the preprocessing of the patient's speech data, includes the following two steps:

[0093] Step S1211: extracting features of speech data, capturing the core features of speech signals by using methods such as Mel Frequency Cepstral Coefficient (MFCC), Linear Prediction Coefficient (LPC) or spectrum feature extraction;

[0094] Step S1212: Use techniques such as principal component analysis (PCA) or linear discriminant analysis (LDA) to simplify the extracted speech features to reduce data dimensions while retaining key features related to diagnosis.

[0095] The preprocessing of physiological signals, specifically the processing of the patient's physiological signals (such as electrocardiogram, blood pressure, pulse, etc.), includes the following three steps:

[0096] Step S1221: Noise filtering of physiological signals, using filtering technology (such as bandpass filter, Kalman filter) to remove background noise;

[0097] Step S1222: feature extraction, extracting key features in physiological signals, such as heart rate variability, P wave features, QRS complex waves, etc., through time domain, frequency domain or time-frequency domain analysis methods;

[0098] Step S1223: Perform feature selection on the extracted physiological features, and select the most diagnostically significant physiological features through statistical methods or deep learning models (such as autoencoder AE).

[0099] Preprocessing of text records, specifically natural language processing (NLP) of patients’ medical text records (such as medical records, doctor’s reports, etc.), includes the following three steps:

[0100] Step S1231: word segmentation and stem extraction, pre-processing the text data to extract important medical terms and keywords;

[0101] Step S1232: TF-IDF or word embedding model (such as Word2Vec, BERT) is used to convert text data into feature vectors that can be used for analysis;

[0102] Step S1233: Use a feature selection method based on statistical or semantic analysis to extract key information that best reflects the patient's condition.

[0103] Step S124: Fusion of multimodal remote data refers to normalizing the above multimodal remote data, using Min-Max normalization or Z-Score normalization and other technologies to adjust the data to a unified range to ensure the stability and consistency of subsequent algorithm processing. Then the pre-processed voice data, physiological signals, text records and other data are fused to construct a unified multimodal telemedicine data set with a data volume of N. The fusion process can maximize the use of diagnostic information from multi-source data by means of feature splicing, weighted fusion, etc.

[0104] Step S2: Generate pellets.

[0105] The specific steps include:

[0106] Step S21: Formation of fuzzy clusters.

[0107] According to the multimodal telemedicine dataset obtained in step S1, the number of samples (data volume) in the multimodal telemedicine dataset is N. The multimodal telemedicine dataset is clustered using the shadow set clustering algorithm (SCM) to form R clusters. According to the number of clusters in most models during clustering, R= .

[0108] Among them, the specific steps of the SCM clustering algorithm are:

[0109] Step S211: Initialize cluster centers: randomly select R cluster centers. In other embodiments, the output of algorithms such as K-Means can also be used as the initial centers;

[0110] Step S212: Calculate the membership degree: for the kth sample x k , calculate the cluster center v of the corresponding i-th fuzzy cluster i The membership degree u ik , the membership reflects the membership between the sample and the cluster center, and the calculation formula is:

[0111] ;

[0112] Among them, u ik ∈[0,1] represents sample x k Belongs to cluster center v i The membership degree, d is the Euclidean distance, and R is the number of clusters;

[0113] Step S213: Calculate the partition threshold α of the i-th fuzzy cluster according to the optimization principle i In this embodiment, a classifier is established by a partition threshold to determine the division of the corresponding fuzzy cluster, and the fuzzy cluster is divided into a core domain, a shadow domain, and an exclusion domain. In a fuzzy cluster, the core domain represents a set of samples that are definitely belonging to the cluster, the shadow domain represents a set of samples that may belong to the cluster, and the exclusion domain represents a set of samples that do not belong to the cluster. Therefore, a granular sphere-based classifier is formed.

[0114] Specifically, the partition threshold α i The calculation is based on the minimum distance principle, and the calculation formula is as follows:

[0115] ;

[0116] In the partition threshold α i In the process of solving , it can be based on the uncertainty invariance principle, and the calculation formula is as follows:

[0117] ;

[0118] Among them, un is the uncertainty measure of the fuzzy value, T is the three-value set of the fuzzy set mapping, and the expression of T is as follows:

[0119] ;

[0120] Step S214: Update cluster center: The cluster center is recalculated based on the membership and weights. The influence of samples in the core domain, shadow domain, and excluded domain on the cluster center is weighted and calculated according to different weights. The updated cluster center v i The calculation is as follows:

[0121] ;

[0122] Among them, m is the fuzzification coefficient, m>1, and usually takes the value of 2, card is the cardinality, α i is the partition threshold and α i ∈[0,0.5];

[0123] Among them, according to the partition threshold, the sample is divided according to the distinction threshold relative to the cluster center to obtain the partition to which the sample belongs and the weight to which it belongs. The specific distinction method is as follows:

[0124] When the membership of the sample is greater than 1-α i When , it means that the samples are homogeneous and unambiguous, and they belong to the core domain of the cluster, so their weights are set to 1;

[0125] When the membership of the sample is in α i and 1-α i When , it means that the samples are assigned to the boundary domain of the cluster, that is, the shadow domain. When calculating the cluster center, their weights are the mth power of the membership value;

[0126] When the membership value of the sample is less than α i When , it means that the samples are assigned to the exclusion domain of the cluster. They do not belong to the cluster, and their weight values ​​are to the mth power.

[0127] Step S215: Minimize the objective function J:

[0128] ;

[0129] in is the Euclidean distance.

[0130] Step S22: constructing a sphere.

[0131] Specifically, according to the shadow clustering algorithm, the data set is divided into R fuzzy clusters, and each fuzzy cluster is divided into a core domain, a shadow domain, and an exclusion domain according to the partition threshold of each cluster. Since the samples in the core domain of each fuzzy cluster are definitely belonging to the cluster, the core area of ​​each cluster is used to construct the granule ball. Therefore, the granule ball derived from the i-th fuzzy cluster is represented as GB i =(C i ,r i ,O i ),

[0132] Among them, C i Indicates GB i The center of i Indicates GB i Radius, O i Indicates included in GB i Internal samples, and C i =v i , .

[0133] In this embodiment, the new method for constructing a sphere based on the SCM clustering algorithm is provided above. By using the fuzzy cluster center obtained by clustering as the center of the sphere, the samples in the sphere are the samples in the core domain of the fuzzy cluster, and the radius of the sphere is the maximum value of the distance between the samples in the sphere and the center of the sphere. In this way, the generated sphere can accurately reflect the structure of the data set, which is convenient for subsequent classification and analysis.

[0134] Step S3: Refine the spheres. When a fixed number of spheres are used to reflect the data structure, there may be complex positional relationships between the spheres, which may lead to inaccurate representation of the data structure. Therefore, the generated spheres are refined to obtain more specific and representative spheres, thereby improving the description of the data structure.

[0135] The complex positional relationship between the spheres is divided into overlapping and nesting, where overlapping refers to the intersection between two spheres, and nesting refers to a sphere being completely contained within another sphere. In order to make the refined spheres more consistent with the original data distribution and to make the subsequent construction of the sphere-based classifier more concise, the sphere refinement task is divided into de-overlapping and de-nesting, which specifically includes the following steps:

[0136] S31: Identification and processing of overlapping spheres;

[0137] S311: Identify overlapping spheres;

[0138] For the generated granular ball GB of the i-th fuzzy cluster i =(C i ,ri ,O i ) and the granular ball GB of the jth fuzzy cluster j =(C j ,r j ,O j ), if two balls satisfy:

[0139] ;

[0140] Then it means the sphere GB i GB j There is an overlapping relationship between two balls. When two balls are in an overlapping relationship, it indicates that there is ambiguity between their semantics, so the two balls in a nested relationship need to be adjusted to balls with clear semantics.

[0141] S312: Processing overlapping pellets;

[0142] When the two spheres satisfy the overlapping relationship of step S311, the radii of the two spheres are adjusted to:

[0143] ;

[0144] ;

[0145] in, Represents the distance between the centers of two spheres.

[0146] Through the above de-overlapping operation, the two overlapping spheres become separated.

[0147] In addition, after the de-overlapping operation, if there is no sample in the sphere, the radius of the sphere is set to 0.

[0148] Step S32: Identify and process the nested pellets.

[0149] S321: Identify nested granules;

[0150] For the generated granular ball GB of the i-th fuzzy cluster i =(C i ,r i ,O i ) and the granular ball GB of the jth fuzzy cluster j =(C j ,r j ,O j ), when the two balls satisfy:

[0151] ;

[0152] Then it means the sphere GB i GB jThere is a nested relationship. When two spheres are in a nested relationship, it means that they have the same semantics. In order to avoid the redundancy of the subsequent sphere-based classifiers, it is necessary to de-nest the two spheres in such a relationship.

[0153] S322: merging nested spheres, merging two spheres that are in a nested relationship into one sphere.

[0154] When the ball GB i =(C i ,r i ,O i ) and ball GB j =(C j ,r j ,O j ) When the nested relationship of step S321 is satisfied, the two balls are merged, and the merged ball is recorded as GB mer =(C mer ,r mer ,O mer ),in

[0155] ;

[0156] ;

[0157] ;

[0158] in, Represents the distance between the samples in the two spheres and the center of the merged sphere. and Represents the sphere GB i and GB j The number of samples within .

[0159] Step S4: Identify and process abnormal spheres. In this embodiment, abnormal spheres include super spheres, solitary spheres and impure spheres. By processing abnormal spheres, the refined spheres can become more representative and more robust. Specifically, the following steps are included:

[0160] Step S41: Identify the super-granular ball.

[0161] If the radius of a sphere is larger than the mean of the radii of all refined spheres, the sphere is a super sphere. Since the radius of a super sphere is too large, it may contain anomalies that are too far away from the center of the sphere.

[0162] The judgment criteria of the super sphere are as follows: for the sphere GB of the generated ith fuzzy cluster i , when the ball GB is identified i =(C i,r i ,O i ) i >2r, it indicates that the identified sphere is a super sphere, wherein r represents the mean value of the radius of all spheres obtained after step S3.

[0163] Step S42: Identify isolated spheres.

[0164] If the interior of a sphere contains only one point or the radius of the sphere is 0, the sphere is called an isolated sphere. Therefore, an isolated sphere means that the sphere has no sample points or only one sample point.

[0165] There are two criteria for judging solitary spheres, as follows:

[0166] For the generated granular ball GB of the i-th fuzzy cluster i , when the ball GB is identified i O i ≦1, it means that the identified ball is a solitary ball; when the ball GB is identified i R i =0, it also means that the identified sphere is an isolated sphere.

[0167] When a solitary sphere has no sample point, the sphere has no actual semantics. When a solitary sphere has only one sample point, the sphere only represents one sample point and has no general meaning.

[0168] Step S43: Identify impure balls.

[0169] If the proportion of samples of any category in a sphere to the total samples is P i If θ is less than θ, the sphere is called an impure sphere.

[0170] The criterion for judging impure spheres is as follows: for the sphere GB of the generated i-th fuzzy cluster i , when the ball GB is identified i P i <θ, it means that the identified ball is an impure ball.

[0171] Setting the purity value of the impure sphere to be smaller than θ (θ∈[0,1]) can increase the robustness of the model.

[0172] Step S44: Processing of abnormal pellets.

[0173] When a sphere is a super sphere, an isolated sphere or an impure sphere, it should be deleted. Through the above abnormal sphere identification and deletion operations, the refined sphere becomes more representative and more robust.

[0174] Step S5: Establish three intelligent classification models based on particles and balls.

[0175] After the above operations of generating spheres, identifying and deleting refined spheres and abnormal spheres, the spheres obtained do not intersect with each other, so they are spheres with specific semantic information and are defined as specific spheres. The specific steps for establishing the three intelligent classification models are as follows:

[0176] Step S51: Generate stable specific spheres.

[0177] According to the obtained specific granules, the center set of the assembly specific granules is:

[0178] ;

[0179] in, represents the center of the specific sphere obtained after the operations of step S2, step S3, and step S4, R' represents the number of newly formed specific spheres, and R'≦R;

[0180] Iterate steps S2-S4 and use the center set center as the initial clustering center of the multimodal telemedicine dataset until the generated spheres do not intersect with each other, that is, a specific set of spheres is generated. At this point, R new Each ball is represented by:

[0181] GB new =(C new ,r new ,O new ),

[0182] Among them, C new Represents GB new The center of new GB new The radius, O new is included in GB new Internal samples.

[0183] GB new The collection describes in detail the structure of the original multimodal telemedicine datasets.

[0184] In the formation of GB new In the previous step, R has been obtained with the help of SCM clustering algorithm. new fuzzy clusters, through the partition threshold α new Each fuzzy cluster is divided into core domain, shadow domain and excluded domain.

[0185] Step S52: Determine three classification rules based on particles and balls.

[0186] The above steps are completed on the training set of the data set. For the samples to be tested in the test set, the following three classification rules are used for classification.

[0187] Rule 1: If a test sample x t In the ball GB new =(C new ,r new ,O new ), that is,

[0188] ,

[0189] Then the label of the sample to be tested is the label of the sphere, that is, the labels of most similar samples in the sphere, where t=1,2,…;

[0190] Rule 2: If a sample to be tested is not inside any sphere, that is, it does not belong to any sphere, when it is in sphere GB new The outside of the sample to be tested x is calculated t The activation membership of each sphere:

[0191] ;

[0192] Where p=1,2,…,R new , represents the number of specific granules, e represents a specific granule GB e =(C e ,r e ,O e ).

[0193] At the same time, let u et and u bt Represents the sample x to be tested t The two maximum memberships of the sphere and u bt >u et ,

[0194] (a) If GB e and GB b If they are two balls of the same category, the label of the sample to be tested is ball GB e and GB b label b ;

[0195] (b) If GB e and GB b are two spheres of different categories that satisfy

[0196] u bt -u et > δ;

[0197] Where δ∈[0,1], the label of the sample to be tested is label b ;

[0198] (c) Otherwise, the sample to be tested is divided into the shadow area, which has a negative effect on label b The membership degree is u bt ;

[0199] The above rules are interpreted in the telemedicine diagnosis scenario as follows: Based on the multimodal telemedicine data of the subject,

[0200] If rule 1 is satisfied, that is, the subject as the sample to be tested is located inside the sphere, then whether the patient is sick is determined by the label of the sphere in which he is located;

[0201] If rule 2(a) is satisfied, the diagnostic result is label b ;

[0202] If rule 2(b) is satisfied, then the diagnosis result is also label b ;

[0203] If rule 2(c) is satisfied, the subject is assigned to the shadow domain and the label is output. b The membership degree of label b If u bt Indicates the risk level of the subject's disease, otherwise 1-u bt Indicates the risk level of the subject to develop the disease.

[0204] In this embodiment, a three-branch diagnosis and evaluation device based on multimodal telemedicine data is also provided. It should be understood that the device is used to execute the construction method of the three-branch diagnosis and evaluation model, and is used to execute the three-branch diagnosis and evaluation method based on multimodal telemedicine data. Specifically, it includes a data acquisition module, a data processing module, a model construction module for executing the construction method of the three-branch diagnosis and evaluation model, and a model training module and a diagnosis and evaluation module for executing the three-branch diagnosis and evaluation method based on multimodal telemedicine data.

[0205] The data acquisition module is to obtain the multimodal telemedicine information of patients through the specified data source or human-computer interaction. Then the data processing module performs feature extraction and feature selection on the multimodal information, and constructs a unified multimodal telemedicine data set through data fusion;

[0206] The model construction module is specifically divided into: the first construction module: generating granules. SCM clustering is performed on the preprocessed multimodal telemedicine data set to generate multiple granules; the second construction module: granule refinement, which is used to perform de-overlapping and de-nesting processing on the generated granules; the third construction module: identification and processing of abnormal granules, identifying and deleting super granules, solitary granules and impure granules; the fourth construction module: iterating the first construction block-the third construction module to generate stable specialized granules, and constructing a three-branch intelligent classification and evaluation model based on granules according to the three-branch classification rules based on granules.

[0207] The model training module refers to training three intelligent classification and evaluation models through medical history diagnosis and treatment information to meet the usage standards.

[0208] The diagnosis and evaluation module refers to the input of multimodal telemedicine information of new patients, and the classification and evaluation of disease risks by using the three-branch intelligent classification and evaluation model based on particles and spheres.

[0209] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A three-branch diagnosis and evaluation method based on multimodal telemedicine data, characterized in that: The three-branch diagnosis and evaluation method based on multimodal telemedicine data includes the following steps: By collecting multimodal telemedicine data from patients, it is input into a three-branch diagnosis and assessment model for analysis; If the diagnosis result is clear, that is, it is confirmed whether the patient is ill, the corresponding result will be directly output; if the diagnosis result is unclear, the patient's risk of illness will be further evaluated and the evaluation result will be output; The method for constructing the three-branch diagnosis and evaluation model includes the following steps: Step S1: collect data and preprocess the collected data to obtain a multimodal telemedicine dataset; Step S2: cluster the multimodal telemedicine dataset using the shadow set clustering algorithm to obtain R fuzzy clusters, establish the partition threshold of each fuzzy cluster, divide the fuzzy cluster into a core domain, a shadow domain, and an exclusion domain, and construct a sphere with the core domain of each fuzzy cluster to obtain R spheres; Step S3: Identify and find the spheres in the overlapping state and the nested state, and perform de-overlapping and de-nesting operations on the spheres in the overlapping state and the nested state respectively to obtain refined spheres; Step S4: Identify and find abnormal spheres in the refined spheres, delete the abnormal spheres, and obtain spheres with specific semantic information: specific spheres; Step S5: Establish a stable set of specific granules, and construct three classification rules based on the specific granules, thereby constructing three intelligent classification and evaluation models based on the specific granules.

2. The three-branch diagnosis and evaluation method based on multimodal telemedicine data according to claim 1 is characterized in that: In step S2, R is , N is the number of samples in the multimodal telemedicine dataset; The partition threshold is calculated according to the optimization principle. The fuzzy cluster is divided into core domain, shadow domain and exclusion domain according to the partition threshold. The granular ball constructed by the core domain of the i-th fuzzy cluster is represented as GB i =(C i ,r i ,O i ), where C i Indicates GB i The center of i Indicates GB i Radius, O i Indicates included in GB i Internal samples.

3. The three-branch diagnosis and evaluation method based on multimodal telemedicine data according to claim 1, characterized in that: The implementation process of step S3 includes the following steps: Step S31: Identification and processing of overlapping spheres; Step S311: Identify overlapping spheres, and generate sphere GB for the i-th fuzzy cluster i =(C i ,r i ,O i ) and the granular ball GB generated by the jth fuzzy cluster j =(C j ,r j ,O j ), when the two balls satisfy: ； Then it means the sphere GB i GB j There is an overlapping relationship; Step S312: Adjust the radius of the two overlapping balls so that the two balls are separated. The radius of the two balls after adjustment are: ； ； in, Represents the distance between the centers of the two spheres; Step S32: identification and processing of nested pellets; Step S321: Identify the nested balls. For the ball GB i =(C i ,r i ,O i ) and ball GB j =(C j ,r j ,O j ), when the two balls satisfy: ； Then it means the sphere GB i GB j There is a nested relationship; Step S322: Merge two balls in a nested relationship. The merged ball is recorded as GB. mer =(C mer ,r mer ,O mer ),in ； ； ； in, Represents the distance between the samples in the two spheres and the center of the merged sphere. and Represents the particle sphere GB i and GB j The number of samples in x k represents the kth sample.

4. The three-branch diagnosis and evaluation method based on multimodal telemedicine data according to claim 1, characterized in that: The implementation process of step S4 includes the following steps: Step S41: Identification of abnormal spheres: for the sphere GB generated by the i-th fuzzy cluster i , When the identified ball GB i =(C i ,r i ,O i ) i >2r, it means that the identified sphere is a super sphere, where r represents the average value of the radius of all spheres obtained after step S3; When the identified ball GB i =(C i ,r i ,O i ) i ≦1 or r i =0, it means that the identified sphere is a solitary sphere; When the identified ball GB i =(C i ,r i ,O i ) i When <θ, it means that the identified ball is an impure ball, where P i It represents the proportion of samples of any category inside the identified sphere to the total samples, θ∈[0,1]; Step S42: Processing abnormal spheres, deleting the identified super spheres, isolated spheres and impure spheres, and obtaining spheres with specific semantic information: specific spheres.

5. The three-branch diagnosis and evaluation method based on multimodal telemedicine data according to claim 1, characterized in that: The implementation process of step S5 includes the following steps: Step S51: Establish a stable collection of specific spheres; Step S52: Determine three classification rules based on pellets and balls; Step S53: construct three intelligent classification and evaluation models based on particles and balls.

6. The three-branch diagnosis and evaluation method based on multimodal telemedicine data according to claim 1, characterized in that: In step S51, the center set of the first obtained specific spheres is expressed as: ； in, represents the center of the specific sphere obtained after the operations of step S2, step S3, and step S4, R' represents the number of newly formed specific spheres, and R'≦R; The center set center is used as the initial clustering center of the multimodal telemedicine dataset, and steps S2 to S4 are iterated until the generated spheres do not intersect with each other, that is, a stable set of specific spheres is generated. At this time, R new Each ball is represented by: GB new =(C new ,r new ,O new ), Among them, C new Represents GB new The center of new GB new The radius, O new is included in GB new Internal samples.

7. The three-branch diagnosis and evaluation method based on multimodal telemedicine data according to claim 1, characterized in that: In step S52, the three classification rules include: Rule 1: If a test sample x t In the ball GB new The interior of ， Then the label of the sample to be tested is the label of the sphere, that is, the labels of most similar samples in the sphere, where t=1,2,…; Rule 2: If a test sample x t Not in any ball GB new The interior of the sample x does not belong to any sphere. t In the ball GB new The outside of the sample to be tested x is calculated t The activation membership of each sphere: ； Among them, e represents a specific ball GB e =(C e ,r e ,O e ); At the same time, let u et and u bt Represents the sample x to be tested t The two maximum memberships of the sphere and u bt >u et , (a) If GB e and GB b If they are two balls of the same category, the label of the sample to be tested is ball GB e and GB b label b ; (b) If GB e and GB b are two spheres of different categories that satisfy in bt -in et >δ; Where δ∈[0,1], the label of the sample to be tested is label b ; (c) Otherwise, the sample to be tested is divided into the shadow area, which has a negative effect on label b The membership degree is u bt ; The above rules are interpreted in the telemedicine diagnosis scenario as follows: Based on the multimodal telemedicine data of the subject, If rule 1 is satisfied, that is, the subject as the sample to be tested is located inside the sphere, then whether the patient is sick is determined by the label of the sphere in which he is located; If rule 2(a) is satisfied, the diagnostic result is label b ; If rule 2(b) is satisfied, then the diagnosis result is also label b ; If rule 2(c) is satisfied, the subject is assigned to the shadow domain and the label is output. b The membership degree of label b If u bt Indicates the risk level of the subject's disease, otherwise 1-u bt Indicates the risk level of the subject to develop the disease.

8. A three-branch diagnosis and evaluation device based on multimodal telemedicine data, characterized in that: It includes a data acquisition module, a data processing module, a model building module, a model training module and a diagnosis and evaluation module for executing the three-branch diagnosis and evaluation method based on multimodal telemedicine data as described in claim 1.

Images