Auxiliary diagnosis method and device based on multi-modal data and electronic equipment

Abstract

The invention provides an auxiliary diagnosis method and device based on multi-modal data and electronic device.The method comprises the steps that a multi-modal sample set and a sample diagnosis report are obtained, and the multi-modal sample set comprises sample eye movement features, sample electroencephalogram features, sample heart rate features and sample galvanic skin features; the sample diagnosis report comprises a sample diagnosis result, a sample clinical record and a sample treatment record; according to the multi-modal sample set and the sample diagnosis report, training to obtain a disease diagnosis model; acquiring multi-modal information of the patient; and inputting the multi-modal information of the patient into the disease diagnosis model to obtain a diagnosis result of the patient. According to the technical scheme provided by the embodiment of the invention, the disease diagnosis model is trained through the multi-modal information of a large number of patients and the corresponding diagnosis reports, and disease diagnosis is carried out through the disease diagnosis model, so that the multi-modal information of the patients is fully utilized, and the efficiency of disease diagnosis is also improved.

Description

Technical Field

[0001] The present disclosure belongs to the field of medical technology, and in particular relates to an auxiliary diagnosis method, device and electronic equipment based on multimodal data. Background Art

[0002] With the development of medical technology, various diseases can be diagnosed through different technical means. In the field of cognitive diseases, the main diagnostic methods rely mainly on the experience of clinicians and traditional assessment methods. For example, clinicians conduct relevant psychiatric and psychological assessments on patients and evaluate the functions, living abilities, emotional states, and mental conditions of patients in various cognitive domains through formatted questionnaires and scales. This type of diagnosis and treatment method is highly subjective, and the accuracy and efficiency of the diagnosis are low. Summary of the invention

[0003] The embodiments of the present disclosure provide a solution to solve the problem in the related art that the traditional diagnosis method of cognitive diseases is highly subjective and has low accuracy and efficiency in diagnosis and treatment.

[0004] In a first aspect, the present disclosure provides an auxiliary diagnosis method based on multimodal data, the method comprising:

[0005] Obtaining a multimodal sample set and a sample diagnosis report, wherein the multimodal sample set includes sample eye movement features, sample EEG features, sample heart rate features, and sample electrodermal features, and the sample diagnosis report includes sample diagnosis results, sample clinical records, and sample medical records;

[0006] Training a disease diagnosis model according to the multimodal sample set and the sample diagnosis report;

[0007] Obtain multimodal information about patients;

[0008] The multimodal information of the patient is input into the disease diagnosis model to obtain the diagnosis result of the patient.

[0009] In a second aspect, the present disclosure provides an auxiliary diagnosis device based on multimodal data, the device comprising:

[0010] An acquisition unit, used to acquire a multimodal sample set and a sample diagnosis report, wherein the multimodal sample set includes sample eye movement features, sample EEG features, sample heart rate features, and sample electrodermal features, and the sample diagnosis report includes sample diagnosis results, sample clinical records, and sample medical records;

[0011] A training unit, used for training a disease diagnosis model according to the multimodal sample set and the sample diagnosis report;

[0012] The acquisition unit is further used to acquire multimodal information of the patient;

[0013] The input unit is used to input the multimodal information of the patient into the disease diagnosis model to obtain the diagnosis result of the patient.

[0014] In a third aspect, the present disclosure provides an electronic device, including:

[0015] Processor; and

[0016] A memory, configured to store executable instructions of the processor;

[0017] The processor is configured to execute any method in the first aspect or any possible implementation of the first aspect by executing the executable instructions.

[0018] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements any method in the first aspect or any possible implementation of the first aspect.

[0019] The technical solution provided by the present disclosure obtains a multimodal sample set and a sample diagnosis report, wherein the multimodal sample set includes sample eye movement features, sample EEG features, sample heart rate features, and sample electrodermal features, and the sample diagnosis report includes sample diagnosis results, sample clinical records, and sample medical records. A disease diagnosis model is trained based on the multimodal sample set and the sample diagnosis report, and the multimodal information of the patient is obtained. The multimodal information of the patient is input into the disease diagnosis model to obtain the diagnosis result of the patient. The technical solution provided by each embodiment of the present disclosure obtains a disease diagnosis model through the acquired multimodal sample set and sample diagnosis report, and the diagnosis result of the patient can be determined based on the multimodal information of the patient. This process makes full use of the multimodal information of the patient, and can improve the accuracy of diagnosis and the efficiency of disease diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0021] Figure 1 A flowchart of an auxiliary diagnosis method based on multimodal data provided by an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of the structure of an auxiliary diagnosis device based on multimodal data provided by an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, but should not be understood as limiting the present disclosure.

[0025] The terms "first" and "second" and the like in the specification, claims, and drawings of the disclosed embodiments are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the disclosed embodiments described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0026] The auxiliary diagnosis method based on multimodal data provided by the embodiment of the present disclosure can be run on an edge device, a terminal device or a server. Among them, the terminal device can be a local terminal device, including wearable devices such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality). Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0027] With the development of medical technology, various diseases can be diagnosed through different technical means. In the field of cognitive diseases, the main diagnostic methods rely mainly on the experience of clinicians and traditional assessment methods. For example, clinicians conduct relevant psychiatric and psychological assessments on patients and evaluate the functions, living abilities, emotional states, and mental conditions of patients in various cognitive domains through formatted questionnaires and scales. This type of diagnosis and treatment method is highly subjective, and the accuracy and efficiency of the diagnosis are low.

[0028] The following specific embodiments are used to describe in detail the technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0029] Figure 1 A flowchart of an auxiliary diagnosis method based on multimodal data provided by an exemplary embodiment of the present disclosure is provided. The method can be applied to an electronic device having a data processing function. Taking the method applied to a medical device as an example, the solution at least includes the following steps S101-S104:

[0030] S101, obtaining a multimodal sample set and a sample diagnosis report.

[0031] In some embodiments, the multimodal sample set includes sample eye movement features, sample EEG features, sample heart rate features, and sample electrodermal features.

[0032] Among them, multimodality refers to the way of expression, communication and understanding using information in a variety of different forms or perceptual channels, usually including multiple sensory input and output methods such as vision, hearing, text, and touch.

[0033] In some embodiments, the eye movement feature refers to the distribution of eye fixation points, eye saccade frequency, etc. Specifically, the eye movement feature is used because the distribution of fixation points can reflect the patient's attention focus area and shift pattern in cognitive tasks, and the eye saccade frequency is related to the speed and efficiency of cognitive processing.

[0034] In some embodiments, the EEG feature refers to power spectrum density, event-related potential, etc. Specifically, the EEG feature is used because the power spectrum density can reveal the energy distribution of different frequency bands of the brain, and the event-related potential (ERP) such as the P300 component is closely related to cognitive decision-making, information processing, and other processes.

[0035] In some embodiments, the heart rate feature refers to electrocardiogram, heart rate variability, etc. Specifically, the heart rate feature is used because heart rate variability (HRV) can reflect the regulatory function of the autonomic nervous system and is associated with cognitive load and emotional state.

[0036] In some embodiments, the galvanic skin feature refers to the galvanic skin response amplitude, etc. Specifically, the galvanic skin feature is used because the galvanic skin response amplitude can be used as an indicator of emotional arousal and cognitive stress.

[0037] Specifically, the multimodal sample set is preferably a set of multimodal information of patients suffering from cognitive diseases, wherein cognitive diseases at least include cognitive impairment, Alzheimer's disease, etc.

[0038] In some embodiments, the sample diagnostic report includes the sample diagnostic result, the sample clinical record, and the sample medical record. Specifically, the sample diagnostic report is preferably a diagnostic report of a patient suffering from a cognitive disease.

[0039] In some embodiments, obtaining a multimodal sample set includes: obtaining the multimodal sample set from a disease database.

[0040] In some embodiments, the disease database stores multimodal information and diagnosis reports of different patients.

[0041] In this embodiment, the method further includes: classifying and labeling the data in the disease database so that the multimodal information and diagnosis reports corresponding to patients with different diseases are classified and stored.

[0042] Specifically, during the medical diagnosis and treatment process in medical institutions, a large number of patients' diagnostic reports and multimodal information are accumulated and stored in the disease database. In this way, a large amount of training data can be provided for the subsequent training of disease diagnosis models, thereby ensuring the accuracy of disease diagnosis models.

[0043] In some embodiments, after obtaining the multimodal information of the patient, the method includes: storing the multimodal information of the patient in the disease database. Specifically, in the daily diagnosis process, the newly obtained multimodal information of the patient is stored in the disease database in real time, and the disease database is continuously enriched and updated to provide guarantee for the subsequent update and optimization of the disease diagnosis model.

[0044] In some embodiments, after storing the multimodal information of the patient in the disease database, the method includes steps S11-S12:

[0045] S11, reacquiring a new multimodal sample set and a new sample diagnosis report from the disease database.

[0046] S12: training a new disease diagnosis model based on the new multimodal sample set and the new sample diagnosis report.

[0047] Specifically, in the actual process of training the disease diagnosis model, since the multimodal information in the disease database is updated in real time, in order to ensure the accuracy of the disease training model output, the disease diagnosis model needs to be trained in real time. Therefore, after the disease database is updated, it is necessary to re-acquire a new multimodal sample set and a new sample diagnosis report from the disease database, and train a new disease diagnosis model based on the new multimodal sample set and the new sample diagnosis report.

[0048] S102: training a disease diagnosis model according to the multimodal sample set and the sample diagnosis report.

[0049] In some embodiments, the training of a disease diagnosis model based on the multimodal sample set and the sample diagnosis report comprises steps S21-S23:

[0050] S21, filtering out abnormal multimodal information from the multimodal sample set by using a support vector machine.

[0051] Among them, Support Vector Machine (SVM) is a generalized linear classifier that performs binary classification of data in a supervised learning manner. It has good classification performance when processing small samples and high-dimensional data, so Support Vector Machine is preferred when performing preliminary screening of multimodal sample sets.

[0052] In some embodiments, the abnormal multimodal information includes abnormal sample eye movement features, abnormal sample EEG features, abnormal sample heart rate features, and abnormal sample electrodermal features.

[0053] In some embodiments, abnormal multimodal information refers to information that is highly correlated with the patient's disease.

[0054] Specifically, the comprehensive analysis of abnormal multimodal information can construct a comprehensive and accurate representation of cognitive states. For example, in the early diagnosis of Alzheimer's disease patients, the study found that when they perform memory tasks, the power spectral density of specific brain areas changes, the saccade latency is prolonged, and the heart rate variability decreases. These feature combinations can effectively distinguish patients from healthy people.

[0055] S22, performing deep feature learning on the abnormal multimodal information and the sample diagnosis report through a convolutional neural network to determine a mapping relationship between the diagnosis result and the abnormal multimodal information.

[0056] Specifically, the diagnosis result corresponding to patient A is Alzheimer's disease, and the abnormal multimodal information corresponding to the Alzheimer's disease includes: sample eye movement feature a, abnormal sample EEG feature a, abnormal sample heart rate feature a and abnormal sample electrodermal feature a.

[0057] S23, based on the mapping relationship, construct the disease diagnosis model through a random forest algorithm.

[0058] Specifically, in the actual process of training the disease diagnosis model, in addition to using a large number of patients' multimodal information for learning and training, the parameters of the disease diagnosis model can also be adjusted through cross-validation and other techniques to ensure the accuracy and reliability of the disease diagnosis model.

[0059] S103, obtaining multimodal information of the patient.

[0060] Specifically, the methods of obtaining multimodal information from patients include memory tests, attention tests, eye movement tasks, emotional tasks, etc., which can accurately stimulate multimodal physiological responses of patients in different cognitive states. For example, in a memory test, when the subject tries to recall information, his EEG signal will show a specific waveform change, and the brain characteristics will be recorded. At the same time, the eye movement pattern may show the characteristics of focusing on a specific area, and the eye movement characteristics will be recorded. The heart rate and skin electricity will also fluctuate accordingly due to changes in cognitive load, and the skin electricity characteristics and heart rate characteristics will be recorded.

[0061] In other embodiments, after obtaining the multimodal information of the patient, the method further comprises: preprocessing the multimodal information to obtain new multimodal information, wherein the preprocessing includes a denoising operation and a de-interference operation.

[0062] Specifically, in the actual process of acquiring multimodal information, the collected multimodal information may be interfered by environmental noise, equipment interference, and artifacts of the physiological signal itself, so the collected multimodal information needs to be preprocessed to make the obtained multimodal information more accurate.

[0063] S104, inputting the multimodal information of the patient into the disease diagnosis model to obtain the diagnosis result of the patient.

[0064] In some embodiments, the multimodal information of the patient is input into the disease diagnosis model to obtain the diagnosis result of the patient, including steps S31-S32:

[0065] S31, obtaining the patient's historical diagnosis report.

[0066] S32, inputting the patient's multimodal information and the historical diagnosis report into the disease diagnosis model to obtain the patient's diagnosis result.

[0067] In some embodiments, the historical diagnosis report refers to the diagnosis report of the patient obtained from the disease database at different consultation times.

[0068] In order to ensure the accuracy of the diagnosis results output by the disease diagnosis model, it is necessary to obtain the patient's historical diagnosis report, and input the patient's multimodal information and the historical diagnosis report into the disease diagnosis model to obtain the patient's diagnosis results. Specifically, when processing the patient's multimodal information, combining the patient's historical diagnosis report can improve the accuracy of the diagnosis.

[0069] In some embodiments, after obtaining the diagnosis result of the patient, the method includes: sending the diagnosis result to a client so that the user can determine a treatment plan according to the diagnosis result.

[0070] The technical solution provided by the present disclosure obtains a multimodal sample set and a sample diagnosis report, wherein the multimodal sample set includes sample eye movement features, sample EEG features, sample heart rate features, and sample electrodermal features, and the sample diagnosis report includes sample diagnosis results, sample clinical records, and sample medical records. A disease diagnosis model is trained based on the multimodal sample set and the sample diagnosis report, and the multimodal information of the patient is obtained. The multimodal information of the patient is input into the disease diagnosis model to obtain the diagnosis result of the patient. The technical solution provided by each embodiment of the present disclosure obtains a disease diagnosis model through the acquired multimodal sample set and sample diagnosis report, and the diagnosis result of the patient can be determined based on the multimodal information of the patient. This process makes full use of the multimodal information of the patient, and can improve the accuracy of diagnosis and the efficiency of disease diagnosis.

[0071] Figure 2 A schematic diagram of the structure of an auxiliary diagnosis device based on multimodal data provided by an exemplary embodiment of the present disclosure;

[0072] The device includes: an acquisition unit 201, a training unit 202, and an input unit 203;

[0073] An acquisition unit 201 is used to acquire a multimodal sample set and a sample diagnosis report, wherein the multimodal sample set includes sample eye movement features, sample EEG features, sample heart rate features, and sample electrodermal features, and the sample diagnosis report includes sample diagnosis results, sample clinical records, and sample medical records;

[0074] A training unit 202, configured to train a disease diagnosis model based on the multimodal sample set and the sample diagnosis report;

[0075] The acquisition unit 201 is also used to acquire multimodal information of the patient;

[0076] The input unit 203 is used to input the multimodal information of the patient into the disease diagnosis model to obtain the diagnosis result of the patient.

[0077] In some embodiments, the device is used to input the multimodal information of the patient into the disease diagnosis model to obtain the patient's diagnosis result, and the device is specifically used to:

[0078] Obtain the patient's historical diagnostic report;

[0079] The patient's multimodal information and the historical diagnosis report are input into the disease diagnosis model to obtain the patient's diagnosis result.

[0080] In some embodiments, the device is used to train a disease diagnosis model based on the multimodal sample set and the sample diagnosis report, and the device is specifically used to:

[0081] Screening out abnormal multimodal information from the multimodal sample set by using a support vector machine, wherein the abnormal multimodal information includes abnormal sample eye movement features, abnormal sample EEG features, abnormal sample heart rate features, and abnormal sample electrodermal features;

[0082] Performing deep feature learning on the abnormal multimodal information and the sample diagnosis report through a convolutional neural network to determine a mapping relationship between the diagnosis result and the abnormal multimodal information;

[0083] Based on the mapping relationship, the disease diagnosis model is constructed by a random forest algorithm.

[0084] In some embodiments, the apparatus is used to obtain a multimodal sample set, and the apparatus is specifically used to:

[0085] A multimodal sample set is obtained from a disease database, where multimodal information and diagnosis reports of different patients are stored.

[0086] In some embodiments, after the device is used to obtain the multimodal information of the patient, the device is specifically used to: store the multimodal information of the patient in the disease database.

[0087] In some embodiments, after the device is used to store the multimodal information of the patient in the disease database, the device is specifically used to:

[0088] Re-acquire a new multimodal sample set and a new sample diagnosis report from the disease database; and train a new disease diagnosis model based on the new multimodal sample set and the new sample diagnosis report.

[0089] In some embodiments, after the device is used to obtain a diagnosis result of a patient, the device is specifically used to: send the diagnosis result to a client so that a user can determine a treatment plan based on the diagnosis result.

[0090] The technical solution provided by the present disclosure obtains a multimodal sample set and a sample diagnosis report, wherein the multimodal sample set includes sample eye movement features, sample EEG features, sample heart rate features, and sample electrodermal features, and the sample diagnosis report includes sample diagnosis results, sample clinical records, and sample medical records. A disease diagnosis model is trained based on the multimodal sample set and the sample diagnosis report, and the multimodal information of the patient is obtained. The multimodal information of the patient is input into the disease diagnosis model to obtain the diagnosis result of the patient. The technical solution provided by each embodiment of the present disclosure obtains a disease diagnosis model through the acquired multimodal sample set and sample diagnosis report, and the diagnosis result of the patient can be determined based on the multimodal information of the patient. This process makes full use of the multimodal information of the patient, and can improve the accuracy of diagnosis and the efficiency of disease diagnosis.

[0091] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the device may perform the above method embodiment, and the above and other operations and / or functions of each module in the device are the corresponding processes in each method in the above method embodiment, respectively, and no further description is given here for the sake of brevity.

[0092] The above describes the device of the embodiment of the present disclosure from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present disclosure can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in conjunction with the embodiment of the present disclosure can be directly embodied as a hardware decoding processor to execute, or a combination of hardware and software modules in the decoding processor to execute. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in conjunction with its hardware.

[0093] Figure 3 is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure, and the electronic device may include:

[0094] The memory 301 and the processor 302, the memory 301 is used to store the computer program and transmit the program code to the processor 302. In other words, the processor 302 can call and run the computer program from the memory 301 to implement the method in the embodiment of the present disclosure.

[0095] For example, the processor 302 may be configured to execute the above method embodiments according to instructions in the computer program.

[0096] In some embodiments of the present disclosure, the processor 302 may include but is not limited to:

[0097] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0098] In some embodiments of the present disclosure, the memory 301 includes but is not limited to:

[0099] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0100] In some embodiments of the present disclosure, the computer program may be divided into one or more modules, which are stored in the memory 301 and executed by the processor 302 to complete the method provided by the present disclosure. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0101] like Figure 3 As shown, the electronic device may also include:

[0102] The transceiver 303 may be connected to the processor 302 or the memory 301 .

[0103] The processor 302 may control the transceiver 303 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 303 may include a transmitter and a receiver. The transceiver 303 may further include an antenna, and the number of antennas may be one or more.

[0104] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0105] The present disclosure also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present disclosure embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0106] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state disk (solid state disk, SSD)), etc.

[0107] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0108] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0109] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present disclosure may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0110] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An auxiliary diagnosis method based on multimodal data, characterized in that: The method comprises: Obtaining a multimodal sample set and a sample diagnosis report, wherein the multimodal sample set includes sample eye movement features, sample EEG features, sample heart rate features, and sample electrodermal features, and the sample diagnosis report includes sample diagnosis results, sample clinical records, and sample medical records; Training a disease diagnosis model according to the multimodal sample set and the sample diagnosis report; Obtain multimodal information about patients; The multimodal information of the patient is input into the disease diagnosis model to obtain the diagnosis result of the patient.

2. The method according to claim 1, characterized in that: Inputting the multimodal information of the patient into the disease diagnosis model to obtain the patient's diagnosis result includes: Obtain the patient's historical diagnostic report; The patient's multimodal information and the historical diagnosis report are input into the disease diagnosis model to obtain the patient's diagnosis result.

3. The method according to claim 1, characterized in that: The training of a disease diagnosis model based on the multimodal sample set and the sample diagnosis report includes: Screening out abnormal multimodal information from the multimodal sample set by using a support vector machine, wherein the abnormal multimodal information includes abnormal sample eye movement features, abnormal sample EEG features, abnormal sample heart rate features, and abnormal sample electrodermal features; Performing deep feature learning on the abnormal multimodal information and the sample diagnosis report through a convolutional neural network to determine a mapping relationship between the diagnosis result and the abnormal multimodal information; Based on the mapping relationship, the disease diagnosis model is constructed by a random forest algorithm.

4. The method according to claim 1, characterized in that Get a multimodal sample set, including: A multimodal sample set is obtained from a disease database, where multimodal information and diagnosis reports of different patients are stored.

5. The method according to claim 4, characterized in that After acquiring the multimodal information of the patient, the method includes: storing the multimodal information of the patient in the disease database.

6. The method according to claim 5, characterized in that After storing the multimodal information of the patient in the disease database, the method includes: Re-acquire a new multimodal sample set and a new sample diagnosis report from the disease database; and train a new disease diagnosis model based on the new multimodal sample set and the new sample diagnosis report.

7. The method according to claim 1, characterized in that After obtaining the diagnosis result of the patient, the method includes: sending the diagnosis result to the client, so that the user can determine a treatment plan according to the diagnosis result.

8. An auxiliary diagnosis device based on multimodal data, characterized in that: The device comprises: An acquisition unit, used to acquire a multimodal sample set and a sample diagnosis report, wherein the multimodal sample set includes sample eye movement features, sample EEG features, sample heart rate features, and sample electrodermal features, and the sample diagnosis report includes sample diagnosis results, sample clinical records, and sample medical records; A training unit, used for training a disease diagnosis model according to the multimodal sample set and the sample diagnosis report; The acquisition unit is further used to acquire multimodal information of the patient; The input unit is used to input the multimodal information of the patient into the disease diagnosis model to obtain the diagnosis result of the patient.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 8 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images