Multi-modal evidence fusion and high-risk early warning method and device for dizziness inquiry

Abstract

The embodiment of the invention provides a multi-mode evidence fusion and high-risk early warning method and device for dizziness inquiry, and relates to the technical field of data analysis and intelligent inquiry. The method comprises the following steps: receiving multi-modal data which is input by a patient and comprises inquiry text information and examination related data, and acquiring feature vector information of the multi-modal data; obtaining candidate category information based on the inquiry text information; inputting the feature vector information and the candidate category information into a fusion analysis model to obtain patient condition analysis data, the patient condition analysis data at least comprising high risk characterization information and consistency evaluation information; and calculating a risk assessment value and determining a risk assessment level based on the high-risk risk characterization information, the consistency assessment information and the red flag rule triggering and cross-modal conflict condition. According to the invention, the high risk is quantified and pre-warned by using the consistency and conflict information of the multi-modal evidence, the single-modal deviation is reduced, and the accuracy and risk pre-warning capability in a dizziness inquiry scene are improved.

Description

Technical Field

[0001] This invention relates to the field of data analysis and intelligent consultation technology, specifically to a multimodal evidence fusion and high-risk early warning method and device for vertigo consultation. Background Technology

[0002] Vertigo is a type of motor or vertigo caused by spatial disorientation. It is a discomfort caused by balance disorders due to various reasons such as ear and positional diseases, and often presents with symptoms such as tinnitus, nausea, vomiting, severe instability, and gait instability.

[0003] Vertigo can be caused by the following factors: First, ear diseases, such as Meniere's disease and benign paroxysmal positional vertigo, can affect the inner ear's balance function; second, neurological diseases, such as cerebellar infarction and cerebral hemorrhage, can damage the areas of the brain that control balance; third, cardiovascular diseases, such as low blood pressure and arrhythmia, can lead to insufficient blood supply to the brain; and fourth, psychological factors, such as long-term anxiety and depression, can also trigger vertigo symptoms.

[0004] Dizziness is a common physical symptom that is easily overlooked by patients. If patients can relieve dizziness by resting, they usually choose not to go to the hospital. However, if dizziness is caused by cardiovascular diseases such as myocardial infarction, this will lead to missing the best time for treatment and have a significant impact on the patient's health and even life. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal evidence fusion and high-risk early warning method and device for vertigo consultation. It uses a fusion analysis model to evaluate the consistency and detect conflicts between two different modalities of evidence: consultation text information and examination-related data. It calculates high-risk and outputs the risk assessment level, thereby providing timely early warning and reducing the bias of single-modal diagnosis in vertigo consultation scenarios.

[0006] To achieve the above objectives, this invention provides a multimodal evidence fusion and high-risk early warning method for vertigo diagnosis, comprising: Receive multimodal data from patients, including consultation text information and examination-related data, and obtain feature vector information of the multimodal data; Based on the consultation text information, the patient's candidate category information is obtained, and the candidate category information includes: the probability of at least one candidate category corresponding to each candidate category; The feature vector information of the multimodal data and the candidate category information are both input into a preset fusion analysis model to obtain patient condition analysis data output by the fusion analysis model. The patient condition analysis data includes at least high-risk characterization information and consistency assessment information. Based on the high-risk characterization information and the consistency assessment information, the patient's current risk assessment level is determined and output.

[0007] The present invention also provides a multimodal evidence fusion and high-risk early warning device, comprising: at least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform the above-described multimodal evidence fusion and high-risk early warning method for vertigo diagnosis.

[0008] The present invention also provides a computer-readable storage medium, which is a non-volatile or non-transient storage medium, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the multimodal evidence fusion and high-risk early warning method for vertigo diagnosis as described above.

[0009] In one embodiment, determining the patient's current risk assessment level based on the high-risk characterization information and the consistency assessment information includes: Obtain the target feature vector that triggers the preset red flag rule from the feature vector information of the multimodal data; Based on the consistency assessment information, the cross-modal conflict situation among the multimodal data is determined; wherein, the consistency assessment information includes a consistency score, which is a numerical value output by the fusion analysis model to characterize the degree of consistency among the multimodal data; Calculate the risk assessment value, and determine the risk assessment level based on the risk assessment value and a preset threshold range; The risk assessment value is obtained by weighting a normalized high-risk probability term, a normalized consistency inverse value term, a normalized red flag score term, and a normalized cross-modal conflict score term according to preset weights. The high-risk probability term is the high-risk probability value in the high-risk representation information. The consistency inverse value term is a value obtained by subtracting the consistency score. The red flag score term is determined according to the red flag rule triggered by the target feature vector and mapped to a dimension of 0-100. The cross-modal conflict score term takes a preset conflict score and is mapped to a dimension of 0-100 when cross-modal conflict exists.

[0010] In one embodiment, after determining the risk assessment level, the method further includes: providing corresponding handling suggestions based on the risk assessment level; the handling suggestions are generated by matching a preset rule base or by being generated by the fusion analysis model after being verified by the rule base.

[0011] In one embodiment, the training process of the fusion analysis model is as follows: A training set is constructed comprising multiple training samples, each training sample comprising: a feature vector of multimodal sample data, a true diagnostic type corresponding to the consultation text sample, and a high-risk label; the multimodal sample data comprises: the consultation text sample and the examination image sample; Establish a fusion analysis model that takes the training samples as input and patient condition analysis data as output. The patient condition analysis data includes at least: high-risk characterization information and consistency assessment information. The established fusion analysis model is trained using multiple training samples in the training set to obtain the final fusion analysis model. During the training process, a multi-task loss function based on candidate diagnostic classification loss term and high-risk classification loss term is adopted.

[0012] In one embodiment, each training sample further includes: a consistency label characterizing the degree of consistency among the multimodal data; the patient condition analysis data further includes: consistency assessment information among the multimodal data; and the multi-task loss function further includes: a consistency assessment loss term.

[0013] In one embodiment, obtaining the feature vector information of the multimodal data includes: Based on the feature extraction model for each single modality, single modal features are extracted from the multimodal data to obtain the feature vector of each single modality data. The feature vector information of the multimodal data includes the feature vectors of all single modality data.

[0014] In one embodiment, the method further includes: Generate and output the patient's source tracing report information, which includes: the risk assessment level, the rule identifier and evidence fragment corresponding to the target feature vector, the cross-modal conflict situation, and treatment recommendation information.

[0015] In one embodiment, the examination-related data includes at least one of the following: eye movement and nystagmus examination videos / images, hearing examination diagrams, and vestibular evoked myogenic potential examination data.

[0016] In one embodiment, the fusion analysis model includes a cross-attention layer and a gating layer. The cross-attention layer is used to calculate the association weight between the text features corresponding to the consultation text information and the objective features corresponding to the examination-related data to achieve cross-modal alignment. The gating layer is used to assign weights to each modal feature according to the quality of each modal evidence and output the high-risk characterization information and consistency assessment information. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the multimodal evidence fusion and high-risk early warning method for vertigo diagnosis in the first embodiment of the present invention. Figure 2 This is a flowchart of the training process of the fusion analysis model in the first embodiment of the present invention; Figure 3 This is a flowchart illustrating the multimodal evidence fusion and high-risk early warning method for vertigo diagnosis in the second embodiment of the present invention. Detailed Implementation

[0018] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings to provide a clearer understanding of the purpose, features, and advantages of the present invention. It should be understood that the embodiments shown in the drawings are not intended to limit the scope of the present invention, but are merely illustrative of the essential spirit of the technical solution of the present invention.

[0019] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known apparatuses, structures, and techniques associated with this application may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0020] Unless the context requires otherwise, throughout the specification and claims, the word “comprising” and its variations, such as “including” and “having”, shall be understood to have an open, inclusive meaning, that is, to be interpreted as “including, but not limited to”.

[0021] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0022] The singular forms “a” and “the” used in this specification and the appended claims include plural references unless otherwise expressly stated herein. It should be noted that the term “or” is generally used to include the meaning of “or / and” unless otherwise expressly stated herein.

[0023] In the following description, in order to clearly demonstrate the structure and working method of the present invention, a number of directional terms will be used. However, terms such as "front", "back", "left", "right", "outer", "inner", "outer", "inner", "up", and "down" should be understood as convenient terms and not as limiting terms.

[0024] The first embodiment of this invention relates to a multimodal evidence fusion and high-risk early warning method for vertigo diagnosis, applied to a multimodal evidence fusion and high-risk early warning device. This device can be a user's electronic device, a server, an edge workstation, etc. The multimodal evidence fusion and high-risk early warning device can analyze the patient's multimodal data based on the vertigo diagnosis method of this embodiment to promptly determine whether the patient currently has a high risk.

[0025] The specific process of the multimodal evidence fusion and high-risk early warning method for vertigo diagnosis in this embodiment is as follows: Figure 1 As shown. This embodiment and subsequent embodiments are based on the patient's dizziness symptoms for illustrative purposes, but any other patient symptoms are also analyzed for high risk using the multimodal evidence fusion and high-risk early warning method for dizziness consultation proposed in this application.

[0026] Step 101: Receive multimodal data from the patient, including consultation text information and examination-related data, and obtain the feature vector information of the multimodal data.

[0027] Specifically, patients can input consultation text by accessing the multimodal evidence fusion and high-risk early warning device through their personal electronic devices, such as by using the keyboard or voice on their personal electronic devices. For example, a patient might input consultation text as, "Doctor, I woke up this morning feeling dizzy, nauseous, and unsteady on my feet. I feel a little better now, but I still get dizzy when I move."

[0028] Patients can also upload examination-related data to the multimodal evidence fusion and high-risk early warning device via their personal electronic devices. The examination-related data includes at least one of the following: eye movement and nystagmus examination videos / images, hearing test images, and vestibular evoked myogenic potential examination data.

[0029] For example, eye movement and nystagmus examination videos / images can be eye movement videos of a preset duration recorded by the user through a personal electronic device. The preset duration can be set in advance, such as 10 seconds, 15 seconds, etc.

[0030] A hearing chart is a graph that records a patient's hearing sensitivity at different sound frequencies and intensities, used to visually display the degree and type of hearing loss.

[0031] Vestibular evoked myogenic potential (VEMP) test data are recorded by stimulating the patient's vestibular system and recording the electromyographic activity on specific muscles, which is used to reflect the function of the neural pathway between the vestibule and muscles.

[0032] Subsequently, feature extraction is performed on the multimodal data to obtain feature vector information of the multimodal data. In one example, single-modal feature extraction can be performed on the multimodal data based on a feature extraction model for each single modality to obtain feature vectors for each single modality. The feature vector information of the multimodal data includes: feature vectors of all single modality data. The feature vector information corresponding to the consultation text information is used at least to characterize dizziness-related key information, and the feature vector information corresponding to the examination-related data is used at least to characterize objective examination indicators. For example, for consultation text information, a pre-trained text understanding model is used. This model employs a language model with secure alignment and coverage of Chinese medical terminology. For instance, it is derived by fine-tuning a general medical language model using pre-annotated consultation text samples. The text understanding model can extract key entities related to dizziness from the consultation text information, such as extracting key entities related to chronology, triggers, and accompanying ear / neurological signs, obtaining feature vectors for each key entity.

[0033] For eye movement and nystagmus examination videos / images, a pre-trained eye movement / nystagmus analysis model can be trained using pre-labeled eye movement videos on a temporal model (such as a 3D-CNN neural network model or a video Transformer model) to obtain an eye movement / nystagmus analysis model. This model can classify the input eye movement and nystagmus examination videos / images (which can be videos or images of multiple consecutive frames) and output key indicators such as the direction, frequency, phase, presence of fixation suppression failure, and HIT index of the head impulse test, obtaining structured feature vectors for these indicators. It can also indicate nystagmus types such as horizontal, vertical / downbeat, and rotational.

[0034] For hearing test maps, a lightweight regression or classification model can be pre-trained. For example, a pre-labeled hearing test map can be used to train the model to detect key indicators such as hearing threshold and symmetry of the input hearing test map, and obtain structured feature vectors of these indicators.

[0035] Vestibular evoked myogenic potential (VEMP) test data can also be used to pre-train lightweight regression or classification models. For example, pre-labeled VEMP test data can be used to train the model to detect key indicators such as latency, fluctuation amplitude, threshold, and left-right ratio of the input VEMP test data, and obtain structured feature vectors of these indicators.

[0036] Step 102: Based on the consultation text information, obtain the patient's candidate category information, which includes: the probability of at least one candidate category corresponding to each candidate category.

[0037] Specifically, based on the feature vectors of key entities such as chronology, triggers, and accompanying symptoms extracted from the consultation text, preliminary candidate category information for the patient can be obtained. This candidate category information includes: the probability of at least one candidate category and each candidate category. For example, the consultation text is "Doctor, I suddenly felt dizzy, nauseous, and unsteady when I woke up this morning. I feel a little better now, but I still get dizzy with the slightest movement." The key entities extracted from this text include: onset: "sudden onset," nature: "rotational vertigo," and accompanying symptoms: "nausea" and "unsteadiness." Based on these key entities, the candidate category information obtained is: vestibular neuritis (65%), posterior circulation stroke (20%), and Meniere's disease (10%).

[0038] Based on the consultation text information, the candidate category information of the patient can be obtained, or it can be achieved using a pre-trained diagnostic model.

[0039] Step 103: Input the feature vector information of the multimodal data and the candidate category information into a preset fusion analysis model to obtain the patient condition analysis data output by the fusion analysis model. The patient condition analysis data includes at least high-risk characterization information and consistency assessment information.

[0040] For example, please refer to Figure 2 The training process of the fusion analysis model is as follows: Step 201: Construct a training set including multiple training samples. Each training sample includes: a feature vector of multimodal sample data, a true diagnosis type corresponding to the consultation text sample, and a high-risk label; the multimodal sample data includes: the consultation text sample and the examination image sample.

[0041] Step 202: Establish a fusion analysis model with the training samples as input and patient condition analysis data as output. The patient condition analysis data includes at least: high-risk characterization information and consistency assessment information.

[0042] Step 203: Train the established fusion analysis model using multiple training samples in the training set to obtain the final fusion analysis model; during the training process, a multi-task loss function based on candidate diagnostic classification loss term and high-risk classification loss term is adopted.

[0043] Specifically, training samples are first collected. For example, each training sample represents a patient's vertigo case, which includes at least: the feature vector of the patient's medical history text sample and the corresponding true diagnostic type (e.g., benign paroxysmal positional vertigo, vestibular migraine, and posterior circulation stroke); the feature vector of examination image samples, including eye movement video samples and audiogram samples; and a high-risk label characterizing whether the case belongs to a high-risk type, which is a binary label indicating whether the case belongs to a preset high-risk type, such as central vertigo. A portion of these training samples (e.g., 80% of the training samples) can be used to form the training set, and the remaining training samples can form the validation set. It should be noted that the training samples here can also be used for the aforementioned text understanding model, eye movement / nystagmus analysis model, and regression or classification models used to analyze audiograms or vestibular evoked myogenic potentials (VEMP) test data.

[0044] In some embodiments, each training sample further includes a consistency label characterizing the degree of consistency between the multimodal data. The consistency label indicates the degree of consistency between the consultation text sample and the examination image sample. For example, it can be labeled as "consistent", "inconsistent", or "cannot be determined", or a continuous score between 0 and 1, where 0 indicates inconsistency and the closer to 1, the higher the degree of consistency.

[0045] For example, in a patient's training sample, the consultation text information is "sudden onset of dizziness and nausea, inability to stand, lasting for 2 hours", the examination image sample is an eye movement video, and the nystagmus type indicated by the eye movement video is pure downbeat nystagmus; the true diagnosis type is posterior circulation stroke, the high-risk label is 1, and the consistency label is 0.2; the reason for the low consistency score here is that the symptoms in the consultation text are severe vestibular neuritis (peripheral), but the nystagmus type is a typical central marker.

[0046] A fusion analysis model is established with the training samples as input and patient condition analysis data as output. The patient condition analysis data includes at least: high-risk characterization information and consistency assessment information. The high-risk characterization information indicates the joint confidence of candidate categories of high-risk types in multimodal data; the consistency assessment information indicates the degree of consistency between multimodal data.

[0047] The fusion analysis model includes a cross-attention layer and a gating layer. The cross-attention layer calculates the association weights between the text features corresponding to the consultation text information and the objective features corresponding to the examination-related data to achieve cross-modal alignment. The gating layer assigns weights to each modality feature based on the quality of evidence from each modality and outputs the high-risk characterization information and consistency assessment information. Specifically, based on the differences in evidence quality among different modalities, the weights of different modalities are adaptively adjusted to improve the robustness of the model.

[0048] When training the fusion analysis model using training samples, an end-to-end joint training method is adopted. During the training process, a multi-task loss function based on candidate diagnostic classification loss term and high-risk classification loss term is used.

[0049] For example, the multi-task loss function further includes a consistency evaluation loss term.

[0050] For example, the multi-task loss function L_total used during model training is: L_total=w1×L_diagnosis+w2×L_consistency+w3×L_risk; Here, L_diagnosis represents the candidate diagnosis classification loss, which is used to minimize the difference between the candidate diagnosis predicted by the model and the true diagnosis label. For example, it is the cross-entropy loss, and its weight value is w1.

[0051] L_consistency represents the consistency prediction loss between multimodal data, used to learn the consistency between medical text samples and examination image samples, such as for classification or regression loss, with a weight value of w2.

[0052] L_risk represents the classification loss for high-risk categories, which primarily penalizes underreporting of high-risk types. For example, it could be a weighted cross-entropy loss with a weight value of w3. Increasing the weight value w3 can increase the penalty for high-risk types. For instance, w1=0.3, w2=0.3, w3=0.4.

[0053] The training samples in the training set are then used to train the fusion analysis model. Through joint optimization, the model learns to assess the intrinsic relationships between different modalities while simultaneously making diagnoses, thereby improving its sensitivity to potential risk signals. Training stops when the training loss of the multi-task loss function meets the requirements. After training the fusion analysis model using the training samples for a set number of epochs, a trained fusion analysis model is obtained. The trained fusion analysis model is then validated using training samples from the validation set.

[0054] The trained fusion analysis model can output patient condition analysis data based on the feature vector information and candidate category information of the current patient's multimodal data. The patient condition analysis data includes at least high-risk characterization information and consistency assessment information. The high-risk characterization information is the joint confidence of the candidate category that belongs to the high-risk type in the at least one candidate category under the multimodal data.

[0055] Furthermore, the patient condition analysis data also includes: consistency assessment information among the multimodal data.

[0056] The feature vectors of the patient's consultation text information, the feature vectors of the examination-related data, and the candidate category information are all input into the trained fusion analysis model. The fusion analysis model is based on the cross-attention mechanism to determine the association weight between the feature vectors of the consultation text information and the feature vectors of the examination-related data. For example, it can determine the association weight between the description of "dizziness like the house spinning" in the consultation text and the "horizontal rotational nystagmus" feature parsed from the eye movement / nystagmus examination image. In this way, cross-modal feature alignment is performed on the consultation text information and the examination-related data.

[0057] The fusion analysis model will output: The consistency assessment information between the consultation text information and the examination-related data can be categorized into three types: "consistent," "inconsistent," and "cannot be determined," or a continuous score between 0 and 1. Taking the consistency assessment information as an example, the fusion analysis model outputs a consistency score between 0 and 1. For instance, when the consultation text information describes the patient as having benign peripheral vertigo, but the oculomotor / nystagmus examination images show central characteristics (such as pure vertical nystagmus), the consistency score will be significantly reduced.

[0058] High-risk characterization information indicates the joint confidence level of the candidate category belonging to the high-risk type among the at least one candidate category under the multimodal data; that is, if there is a high-risk type among the initially determined patient candidate categories, the joint confidence level is calculated for the candidate category under the multimodal data, and the joint confidence level value is the high-risk characterization information; if there are multiple high-risk types among the initially determined patient candidate categories, the joint confidence level of these multiple high-risk types under the multimodal data is calculated for each, and the highest joint confidence level and its corresponding high-risk type are used as the output.

[0059] The high-risk characterization information output is based on the joint confidence of the candidate categories of high-risk types under the multimodal data. It combines evidence from all modalities and represents the patient's high-risk probability value. Therefore, when the consistency between multimodal data is low, even if a certain high-risk type has a high confidence in a single modality, the joint confidence output by the model will be lowered accordingly.

[0060] For example, the patient's current consultation text is: "Doctor, I woke up this morning feeling dizzy, nauseous, and unsteady on my feet. I feel a little better now, but I still get dizzy with the slightest movement." The key entities extracted include: onset: "sudden onset," nature: "rotational vertigo," and accompanying symptoms: "nausea" and "unsteadiness." Based on these key entities, the candidate categories are: vestibular neuritis (65%), posterior circulation stroke (20%), and Meniere's disease (10%).

[0061] The relevant data, including the eye movement and nystagmus examination video / images, showed that the nystagmus classification result was "downbeat nystagmus".

[0062] The fusion analysis model receives feature vectors from the consultation text information, feature vectors from eye movement and nystagmus examination videos / images, and candidate category information. It judges that the consultation text information is more likely to indicate the periphery, while the eye movement and nystagmus examination videos / images point to the central nervous system. The cross-attention mechanism of the fusion analysis model finds that the nystagmus type of "vertigo" described in the consultation text information and "downpick nystagmus" indicated by the eye movement / nystagmus examination images are weakly correlated, and there is even cross-modal conflict.

[0063] Output of the fusion analysis model: The consistency assessment information includes a consistency score of 0.25. This consistency score is a numerical value output by the fusion analysis model, characterizing the degree of consistency among the multimodal data.

[0064] The high-risk characterization information includes a high-risk probability of 0.88.

[0065] In some embodiments, after step 103, the method further includes: Step 104: Based on the high-risk characterization information and the consistency assessment information, determine and output the patient's current risk assessment level.

[0066] Specifically, if high-risk characterization information indicates the existence of candidate categories belonging to the high-risk type, then a high-risk warning is required. For example, this can be achieved by normalizing the high-risk probability included in the high-risk characterization information and the consistency score included in the consistency assessment information, and then calculating a risk assessment value based on the red flag rule triggering and cross-modal conflict situations. Finally, based on the risk assessment value, the risk assessment level is determined. This approach utilizes the consistency and conflict information of multimodal evidence to quantify and warn of high-risk risks, reducing single-modal bias and improving the accuracy and risk warning capabilities in vertigo diagnosis scenarios.

[0067] In this embodiment, a fusion analysis model is used to analyze and learn the consistency between two different modalities of data: consultation text information and examination-related data. This allows for the determination of the joint confidence value of the initially identified patient candidate category under multiple modalities. In other words, the consistency between multimodal data is used to correct the accuracy of the candidate category, avoiding the bias of single-modal diagnosis. This enables the indication of the degree of high risk when the patient's candidate category belongs to a high-risk type. As a result, patients can promptly discover high-risk symptoms through online consultation, reducing health and even life risks.

[0068] The second embodiment of the present invention relates to a multimodal evidence fusion and high-risk early warning method for vertigo diagnosis. Compared with the first embodiment, this embodiment adds red flag rule and cross-modal conflict to perform high-risk early warning.

[0069] The specific process of the multimodal evidence fusion and high-risk early warning method for vertigo diagnosis in this embodiment is as follows: Figure 3 As shown.

[0070] Step 301: Receive multimodal data from the patient, including consultation text information and examination-related data, and obtain the feature vector information of the multimodal data. This is largely the same as step 101 in the first embodiment and will not be described again here.

[0071] Step 302: Based on the consultation text information, obtain the patient's candidate category information, which includes: the probability of at least one candidate category corresponding to each candidate category. This is largely the same as step 102 in the first embodiment and will not be described again here.

[0072] Step 303 involves inputting the feature vector information of the multimodal data and the candidate category information into a preset fusion analysis model to obtain patient condition analysis data output by the fusion analysis model. This patient condition analysis data includes at least high-risk characterization information and consistency assessment information. This is largely the same as step 103 in the first embodiment and will not be repeated here.

[0073] Step 304 includes the following sub-steps: Sub-step 3041: Obtain the target feature vector that triggers the preset red flag rule from the feature vector information of the multimodal data.

[0074] Sub-step 3042: Based on the consistency assessment information, determine the cross-modal conflict situation among the multimodal data; wherein, the consistency assessment information includes a consistency score, which is a numerical value output by the fusion analysis model to characterize the degree of consistency among the multimodal data.

[0075] Sub-step 3043: Calculate the risk assessment value and determine the risk assessment level based on the risk assessment value and a preset threshold range. The risk assessment value is obtained by weighting a normalized high-risk probability term, a normalized consistency inverse value term, a normalized red flag score term, and a normalized cross-modal conflict score term according to preset weights. The high-risk probability term is the high-risk probability value in the high-risk representation information; the consistency inverse value term is a value obtained by subtracting the consistency score; the red flag score term is determined according to the red flag rule triggered by the target feature vector and mapped to a dimension of 0-100; the cross-modal conflict score term takes a preset conflict score when cross-modal conflict exists and is mapped to a dimension of 0-100.

[0076] Specifically, the multimodal evidence fusion and high-risk early warning device has a pre-set red flag rule library, which includes multiple red flag rules. For example, the keywords for red flag rules corresponding to medical history text information include: unclear speech, severe headache, unsteady gait, etc.; and the red flag rules corresponding to oculomotor / nystagmus examination images include: central nystagmus, abnormal eye movement, etc.

[0077] Therefore, after obtaining the feature vector information of the multimodal data, it can be determined whether there are target feature vectors that trigger the preset red flag rule. If they exist, these target feature vectors are extracted; if they do not exist, it is determined that the red flag rule has not been triggered, and the red flag score is 0.

[0078] Based on the consistency assessment information output by the fusion analysis model, the cross-modal conflict status among the multimodal data is determined. For example, if the consistency assessment information is inconsistent, a cross-modal conflict is determined to exist; if the consistency assessment information is consistent, a cross-modal conflict is determined not to exist. Alternatively, if the consistency assessment information is a consistency score, a score threshold can be preset (e.g., 0.4). If the consistency score is less than this score threshold, a cross-modal conflict is determined to exist; otherwise, a cross-modal conflict is determined not to exist.

[0079] Based on the target feature vector, cross-modal conflict status, high-risk probability term, and consistency assessment term obtained above, the patient's risk assessment value is calculated. Specifically, the risk assessment value is obtained by weighting the normalized high-risk probability term, the normalized consistency reverse value term, the normalized red flag score term, and the normalized cross-modal conflict score term according to preset weights. Among them, the high-risk probability term is the high-risk probability value in the high-risk risk representation information; the consistency reverse value term is a value obtained by subtracting the consistency score; the red flag score term is determined according to the red flag rule triggered by the target feature vector and mapped to the dimension of 0-100; the cross-modal conflict score term takes a preset conflict score and is mapped to the dimension of 0-100 when cross-modal conflict exists.

[0080] Furthermore, the risk assessment level is determined based on the risk assessment value and the preset threshold range; that is, different risk assessment levels correspond to preset threshold ranges with different risk assessment values. By comparing the risk assessment value with the preset threshold range corresponding to each risk assessment level, the patient's risk assessment level can be determined.

[0081] For example, the Risk_Score is calculated as follows: Risk_Score = W4 × normalized high-risk probability + W5 × normalized (1 - consistency score) + normalized red flag score + normalized cross-modal conflict score; where W4 + W5 = 1.

[0082] Different red flag rules have corresponding original scores. If multiple red flag rules are triggered simultaneously, the original red flag scores can be obtained by weighting or averaging. The red flag scores are then mapped to a scale of 0-100. The cross-modal conflict score takes a preset conflict score (e.g., 30) and is mapped to a scale of 0-100 when cross-modal conflict exists; it is 0 when no cross-modal conflict exists.

[0083] For example, the "downbeat nystagmus" indicated by the patient's oculomotor / nystagmus examination images triggered the red flag sign rule for central nystagmus, which has a score of 60.

[0084] Output of the fusion analysis model: Consistency score, 0.25.

[0085] High-risk probability: 0.88.

[0086] If the consistency score is less than the preset score threshold of 0.4, cross-modal conflict is identified.

[0087] Taking W4 as 0.6 and W5 as 0.4 as examples, if the high-risk probability is 0.88, the consistency score is 0.25, the red flag score is mapped to 60 points, there is cross-modal conflict and the conflict score is mapped to 30 points, then the risk assessment value Risk_Score is: Risk_Score=(0.6×88)+(0.4×75)+60+30=52.8+30+60+30=172.8.

[0088] For example, different preset threshold ranges are defined with corresponding risk assessment levels, such as: alert, warning, and high risk. The risk assessment value range of the preset threshold range is 0-80, corresponding to an alert risk assessment level; the risk assessment value range of the preset threshold range is 80-140, corresponding to a warning risk assessment level; the risk assessment value range of the preset threshold range is 140-200, corresponding to a high risk risk level. The patient's risk assessment value is 172.8, which falls into the high risk range, so the patient's risk assessment level is high risk.

[0089] Furthermore, based on the target feature vector, cross-modal conflict situation, and high-risk probability term obtained above, it can be determined whether to trigger a high-risk warning; a high-risk warning is triggered when at least one of the following conditions is met: the high-risk characterization information indicates the existence of a candidate category belonging to the high-risk type, there is a cross-modal conflict, or a target feature vector that triggers the preset red flag rule appears.

[0090] Step 305: Based on the risk assessment level, provide corresponding handling suggestions; the handling suggestions are generated by matching a preset rule base or by being generated by the output of the fusion analysis model and then verified by the rule base.

[0091] Step 306: Generate and output the patient's source tracing report information, which includes: the risk assessment level, the rule identifier and evidence fragment corresponding to the target feature vector, the cross-modal conflict situation, and treatment recommendation information.

[0092] Specifically, after determining the risk assessment level, corresponding treatment recommendations are given based on the risk assessment level. For example, if the risk assessment level is "warning," it is recommended that the patient seek medical attention as soon as possible to determine the cause of the dizziness. If the risk assessment level is "high risk," it is recommended that the patient seek medical attention immediately, as there may be a risk of stroke, and an emergency head MRI should be arranged.

[0093] Subsequently, treatment recommendations will be sent to the patient's personal electronic device or the electronic devices of other designated important contacts.

[0094] Furthermore, based on the patient information obtained and analyzed in this embodiment, a patient tracing report can be generated and output. The tracing report can be in text or JSON format, etc. It includes: the risk assessment level, the rule identifier and evidence fragment corresponding to the target feature vector, the cross-modal conflict situation, and treatment recommendations. For example, for the patient in the example above, the output tracing report information is: high-risk type exists, high-risk probability is 88%, red flag sign rule for triggering central nystagmus exists, cross-modal conflict exists, consultation text "suddenly felt dizzy this morning", eye movement video indication: downbeat nystagmus, occurrence time 00:05-00:12, and a screenshot of downbeat nystagmus is provided; cross-modal conflict analysis: the patient's text complaint is similar to common peripheral vertigo, but the nystagmus video shows typical signs of central lesion, and the two are highly inconsistent; current treatment recommendations: central lesion (especially posterior circulation stroke) is highly suspected, and it is recommended to immediately initiate the stroke procedure and perform emergency neurological examination and cranial imaging examination (MRI preferred).

[0095] This enables the provision of clear and traceable reporting information to assist doctors in making rapid diagnoses and treatments.

[0096] The third embodiment of the present invention relates to a multimodal evidence fusion and high-risk early warning device, such as a computer, laptop computer or other electronic device with computing capabilities.

[0097] A multimodal evidence fusion and high-risk early warning device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a multimodal evidence fusion and high-risk early warning method for vertigo consultation as described in the first or second embodiment.

[0098] The fourth embodiment of the present invention relates to a computer-readable storage medium, which is a non-volatile or non-transient storage medium, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the multimodal evidence fusion and high-risk early warning method for vertigo diagnosis as described in the first embodiment.

[0099] In light of the detailed description above, these and other changes can be made to the embodiments. Generally, the terminology used in the claims should not be considered limited to the specific embodiments disclosed in the specification and claims, but should be understood to include all possible embodiments together with the full scope of equivalents enjoyed by these claims.

Claims

1. A multimodal evidence fusion and high-risk early warning method for vertigo diagnosis, characterized in that, include: Receive multimodal data from patients, including consultation text information and examination-related data, and obtain feature vector information of the multimodal data; Based on the consultation text information, the patient's candidate category information is obtained, and the candidate category information includes: the probability of at least one candidate category corresponding to each candidate category; The feature vector information of the multimodal data and the candidate category information are both input into a preset fusion analysis model to obtain patient condition analysis data output by the fusion analysis model. The patient condition analysis data includes at least high-risk characterization information and consistency assessment information. Based on the high-risk characterization information and the consistency assessment information, the patient's current risk assessment level is determined and output.

2. The multimodal evidence fusion and high-risk early warning method for vertigo diagnosis according to claim 1, characterized in that, The process of determining the patient's current risk assessment level based on the high-risk characterization information and consistency assessment information includes: Obtain the target feature vector that triggers the preset red flag rule from the feature vector information of the multimodal data; Based on the consistency assessment information, the cross-modal conflict situation among the multimodal data is determined; wherein, the consistency assessment information includes a consistency score, which is a numerical value output by the fusion analysis model to characterize the degree of consistency among the multimodal data; Calculate the risk assessment value, and determine the risk assessment level based on the risk assessment value and a preset threshold range; The risk assessment value is obtained by weighting a normalized high-risk probability term, a normalized consistency inverse value term, a normalized red flag score term, and a normalized cross-modal conflict score term according to preset weights. The high-risk probability term is the high-risk probability value in the high-risk representation information. The consistency inverse value term is a value obtained by subtracting the consistency score. The red flag score term is determined according to the red flag rule triggered by the target feature vector and mapped to a dimension of 0-100. The cross-modal conflict score term takes a preset conflict score and is mapped to a dimension of 0-100 when cross-modal conflict exists.

3. The multimodal evidence fusion and high-risk early warning method for vertigo diagnosis according to claim 1, characterized in that, After determining the risk assessment level, the method further includes: providing corresponding handling suggestions based on the risk assessment level; the handling suggestions are generated by matching a preset rule base or by being generated by the fusion analysis model after being verified by the rule base.

4. The multimodal evidence fusion and high-risk early warning method for vertigo diagnosis according to claim 1, characterized in that, The training process of the fusion analysis model is as follows: A training set is constructed comprising multiple training samples, each training sample comprising: a feature vector of multimodal sample data, a true diagnostic type corresponding to the consultation text sample, and a high-risk label; the multimodal sample data comprises: the consultation text sample and the examination image sample; Establish a fusion analysis model that takes the training samples as input and patient condition analysis data as output. The patient condition analysis data includes at least: high-risk characterization information and consistency assessment information. The established fusion analysis model is trained using multiple training samples in the training set to obtain the final fusion analysis model. During the training process, a multi-task loss function based on candidate diagnostic classification loss term and high-risk classification loss term is adopted.

5. The multimodal evidence fusion and high-risk early warning method for vertigo diagnosis according to claim 4, characterized in that, Each training sample further includes: a consistency label characterizing the degree of consistency among the multimodal data; the multi-task loss function further includes: a consistency evaluation loss term.

6. The multimodal evidence fusion and high-risk early warning method for vertigo diagnosis according to claim 1, characterized in that, Obtaining the feature vector information of the multimodal data includes: Based on the feature extraction model for each single modality, single modal features are extracted from the multimodal data to obtain the feature vector of each single modality data. The feature vector information of the multimodal data includes the feature vectors of all single modality data.

7. The multimodal evidence fusion and high-risk early warning method for vertigo diagnosis according to claim 2, characterized in that, The method further includes: Generate and output the patient's source tracing report information, which includes: the risk assessment level, the rule identifier and evidence fragment corresponding to the target feature vector, the cross-modal conflict situation, and treatment recommendation information.

8. The multimodal evidence fusion and high-risk early warning method for vertigo diagnosis according to claim 1, characterized in that, The examination-related data includes at least one of the following: eye movement and nystagmus examination videos / images, hearing test images, and vestibular evoked myogenic potential examination data.

9. The multimodal evidence fusion and high-risk early warning method for vertigo diagnosis according to claim 1, characterized in that, The fusion analysis model includes a cross-attention layer and a gating layer. The cross-attention layer is used to calculate the association weight between the text features corresponding to the consultation text information and the objective features corresponding to the examination-related data to achieve cross-modal alignment. The gating layer is used to assign weights to each modal feature according to the quality of each modal evidence and output the high-risk characterization information and consistency assessment information.

10. A multimodal evidence fusion and high-risk early warning device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the multimodal evidence fusion and high-risk early warning method for vertigo diagnosis as described in any one of claims 1 to 9.

Images