Disease early warning method and device

Abstract

The invention discloses a disease early warning method and device, and relates to the technical field of disease early warning, the method utilizes rich target disease related knowledge in clinical examination result data, and through a cross-modal knowledge distillation method, knowledge learned by a multi-modal teacher model is transferred to a student model only accessing a man-machine interaction mode, so that the disease early warning is realized. The disease early warning capability of the man-machine interaction student model is improved. Knowledge highly related to the target disease in the clinical examination result data can be migrated to the man-machine interaction data, so that a more accurate early warning result can be obtained under the condition that only the man-machine interaction data is applicable. A low-cost and low-burden nervous system disease early warning mode based on human-computer interaction data is realized, and early recognition and long-term monitoring can be performed on large-scale crowds with onset risks.

Description

Technical Field

[0001] The present invention relates to the field of disease early warning technology, and in particular to an early disease warning method and device based on a privileged knowledge distillation cross-modal knowledge transfer method. Background Art

[0002] With the global aging trend, the prevalence of neurological diseases continues to rise. Due to the current lack of effective treatments, it is too late to treat patients when they show obvious symptoms. Therefore, early detection and targeted intervention are crucial.

[0003] Currently commonly used methods for early detection of neurological diseases, such as magnetic resonance imaging or cerebrospinal fluid sampling, are invasive and costly. They are not suitable for frequent testing to provide early warnings or continuous tracking of disease progression, and pose an obstacle to the risk assessment of large-scale populations. Therefore, an alternative method is needed to identify large numbers of people at risk of disease in an early stage in a low-burden, low-cost manner.

[0004] In recent years, a series of digitally measured behavioral and physiological indicators based on human-computer interaction technology, including breathing, eye movement, voice and gait, have been incorporated into the early detection system of neurological diseases. Current research has also demonstrated the effectiveness of deep learning methods that use human-computer interaction data to assist in early warning of neurological diseases. However, digital measurements obtained only through human-computer interaction are unlikely to accurately distinguish specific neurological diseases, and further diagnosis is still required in combination with brain imaging results or clinical markers in clinical practice.

[0005] Recently, some research has combined brain imaging data with human-computer interaction data to achieve early diagnosis and risk assessment of neurodegenerative diseases through multimodal fusion and complementation, and has achieved good results. However, this method requires the patient's brain imaging results as input to the model, which violates the basic principle of low burden and low cost for early warning. Summary of the invention

[0006] In view of the above problems, the present invention provides a disease early warning method and device for overcoming the above problems or at least partially solving the above problems. The complete multimodal privileged knowledge in the teacher model is refined by using a cross-modal knowledge transfer framework to enhance the early warning capability of neurological diseases in the single-modal human-computer interaction data in the student model.

[0007] The present invention provides the following scheme:

[0008] A disease early warning method comprising:

[0009] Obtain sample clinical examination result data and sample human-computer interaction data for training;

[0010] Inputting the sample clinical examination result data and the sample human-computer interaction data into a teacher model to train the teacher model, so that the teacher model learns the semantic correlation information between the sample clinical examination result data and the sample human-computer interaction data and the strong correlation information with the early warning of the target disease;

[0011] Acquire the accurate warning results, soft labels and shared feature vectors output by the trained teacher model; perform knowledge distillation on the accurate warning results and soft labels by means of warning target distillation, and perform knowledge distillation on the shared feature vectors by means of multimodal semantic distillation; so as to transfer the accurate warning results, soft labels and shared feature vectors to a student model that only receives the sample human-computer interaction data to train the student model, so that after training, the student model has the ability to provide early warning for the target disease when only receiving the sample human-computer interaction data;

[0012] The human-computer interaction data to be warned is obtained, and the human-computer interaction data to be warned is input into the trained student model so that the student model outputs an early warning result of the target disease.

[0013] Preferably: the teacher model includes a shared modality encoder, two first specific modality encoders for processing specific modalities, a fusion module and a first Softmax classifier, the shared modality encoder is used to decouple the sample clinical examination result data and the sample human-computer interaction data to obtain the shared feature vector; the two first specific modality encoders are used to decouple the sample clinical examination result data and the sample human-computer interaction data to obtain a modality unique feature vector;

[0014] The fusion module is used to fuse the shared feature vector and the modality-specific feature vector and input the resultant into the first Softmax classifier so that the classifier outputs the accurate warning result and soft label.

[0015] Preferably: during the teacher model training process, a cross entropy loss function is used to train the shared modality encoder and the first specific modality encoder.

[0016] Preferably: the student network includes a second specific modality encoder and a second Softmax classifier; the second specific modality encoder is used to decouple the sample human-computer interaction data to obtain a coded feature vector; the second Softmax classifier is used to output a predicted label according to the coded feature vector.

[0017] Preferably: the warning target distillation method includes minimizing the warning target loss function obtained by weighting the prediction loss and the corresponding soft label loss; the soft label loss is the cross entropy loss between the probability distributions output by the teacher model and the student model, and the prediction loss is the cross entropy loss between the probability distribution output by the student model and the true label.

[0018] Preferably: the minimization warning target loss function is expressed by the following formula:

[0019] L target =L h +αL s

[0020] Where: L h represents the prediction loss, L s represents the soft label loss;

[0021] The prediction loss is expressed as follows:

[0022]

[0023] Where: They are the true label and output probability distribution of the student network when the temperature coefficient is 1;

[0024] The soft label loss is expressed as follows:

[0025]

[0026] Where: They are the output probability distributions of the teacher model and the student model under the temperature coefficient T respectively.

[0027] Preferably: the multimodal semantic distillation method is used to perform a nonlinear transformation on the shared feature vector output by the shared modality encoder in the teacher model, project the shared feature vector into the joint representation space of the clinical examination result data and the human-computer interaction data, and obtain a first joint space embedding vector; the encoded feature vector output by the student model encoding module is projected into the joint representation space of the clinical examination result data and the human-computer interaction data, and obtain a second joint space embedding vector;

[0028] A multimodal semantic loss function is constructed by the distance between the embedding vectors of the teacher model and the student model in the subspace, and the distance is minimized by calculating the cosine similarity between two feature vectors, so that the student model can imitate the multimodal joint feature representation learned by the teacher model.

[0029] Preferably: the multimodal semantic loss function is expressed by the following formula:

[0030]

[0031] Where: are the joint space embedding vectors obtained by the teacher model and the student model for the j-th sample respectively.

[0032] Preferably, the sample clinical examination result data includes sample brain image data, the sample human-computer interaction data includes sample gait data, and the target disease includes a nervous system disease.

[0033] A disease early warning device, used to implement the above-mentioned disease early warning method, comprising:

[0034] A data acquisition unit, used to acquire sample clinical examination result data and sample human-computer interaction data for training;

[0035] A teacher model training unit, used for inputting the sample clinical examination result data and the sample human-computer interaction data into a teacher model to train the teacher model, so that the teacher model learns the semantic correlation information between the sample clinical examination result data and the sample human-computer interaction data and the strong correlation information with the early warning of the target disease;

[0036] The knowledge distillation and student model training unit is used to obtain the accurate warning results, soft labels and shared feature vectors output by the trained teacher model; perform knowledge distillation on the accurate warning results and soft labels by warning target distillation, and perform knowledge distillation on the shared feature vectors by multimodal semantic distillation; so as to transfer the accurate warning results, soft labels and shared feature vectors to the student model that only receives the sample human-computer interaction data to train the student model, so that the student model has the early warning capability of the target disease after training when only receiving the sample human-computer interaction data;

[0037] The warning result output unit is used to obtain the human-computer interaction data to be warned, and input the human-computer interaction data to be warned into the trained student model so that the student model can output the early warning result of the target disease.

[0038] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0039] An embodiment of the present application provides an early disease warning method and device, which utilizes the rich target disease-related knowledge in clinical examination result data, and transfers the knowledge learned by the multimodal teacher model to the student model that only accesses the human-computer interaction modality through a cross-modal knowledge distillation method, so as to enhance the disease warning capability of the human-computer interaction student model. Since the knowledge that is strongly related to the target disease in the clinical examination result data can be transferred to the human-computer interaction data, more accurate warning results can be obtained when only human-computer interaction data is used. A low-cost, low-burden early warning method for neurological diseases based on human-computer interaction data is realized, which can be used for early identification and long-term monitoring of large-scale people at risk of disease.

[0040] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 is a flow chart of a disease early warning method provided by an embodiment of the present invention;

[0043] Figure 2 It is an overall framework flow chart provided by an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of a distillation module provided in an embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of an early warning device for a disease provided by an embodiment of the present invention;

[0046] Figure 5 It is a schematic diagram of a disease early warning device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0048] See also Figure 1 , is a disease early warning method provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0049] S101: Obtain sample clinical examination result data and sample human-computer interaction data for training;

[0050] S102: Input the sample clinical examination result data and the sample human-computer interaction data into the teacher model to train the teacher model, so that the teacher model learns the semantic correlation information between the sample clinical examination result data and the sample human-computer interaction data and the strong correlation information with the early warning of the target disease; in specific implementation, the embodiment of the present application can provide that the teacher model includes a shared modality encoder, two first specific modality encoders for processing a specific modality, a fusion module and a first Softmax classifier, the shared modality encoder is used to decouple the sample clinical examination result data and the sample human-computer interaction data to obtain the shared feature vector; the two first specific modality encoders are used to decouple the sample clinical examination result data and the sample human-computer interaction data to obtain a modality unique feature vector;

[0051] The fusion module is used to fuse the shared feature vector and the modality-specific feature vector and input the resultant into the first Softmax classifier so that the classifier outputs the accurate warning result and soft label.

[0052] Furthermore, during the teacher model training process, a cross entropy loss function is used to train the shared modality encoder and the first specific modality encoder.

[0053] S103: Obtain the accurate warning results, soft labels and shared feature vectors output by the trained teacher model; perform knowledge distillation on the accurate warning results and soft labels by using a warning target distillation method, and perform knowledge distillation on the shared feature vectors by using a multimodal semantic distillation method; so as to transfer the accurate warning results, soft labels and shared feature vectors to a student model that only receives the sample human-computer interaction data to train the student model, so that after training, the student model has the ability to provide early warning for the target disease when only receiving the sample human-computer interaction data;

[0054] The student model provided in the embodiment of the present application adopts a similar structure to the teacher model, but because the input is a unimodal training data set containing only gait data, the encoding module only retains the specific modality encoder corresponding to the gait, and no feature fusion is required. In specific implementation, the embodiment of the present application can provide that the student network includes a second specific modality encoder and a second Softmax classifier; the second specific modality encoder is used to decouple the sample human-computer interaction data to obtain a coded feature vector; the second Softmax classifier is used to output a predicted label based on the coded feature vector.

[0055] The distillation module contains two distillation methods, which perform knowledge distillation by minimizing the warning target loss function and the multimodal semantic loss function respectively.

[0056] In specific implementation, the embodiment of the present application can provide that the warning target distillation method includes minimizing the warning target loss function obtained by weighting the prediction loss and the corresponding soft label loss; the soft label loss is the cross entropy loss between the probability distribution output by the teacher model and the student model, and the prediction loss is the cross entropy loss between the probability distribution output by the student model and the true label.

[0057] The minimization warning target loss function is expressed by the following formula:

[0058] L target =L h +αL s

[0059] Where: L h represents the prediction loss, L s represents the soft label loss;

[0060] The prediction loss is expressed as follows:

[0061]

[0062] Where: They are the true label and output probability distribution of the student network when the temperature coefficient is 1;

[0063] The soft label loss is expressed as follows:

[0064]

[0065] Where: They are the output probability distributions of the teacher model and the student model under the temperature coefficient T respectively.

[0066] The multimodal semantic distillation method is used to perform a nonlinear transformation on the shared feature vector output by the shared modality encoder in the teacher model, project the shared feature vector into the joint representation space of the clinical examination result data and the human-computer interaction data, and obtain a first joint space embedding vector; project the encoded feature vector output by the student model encoding module into the joint representation space of the clinical examination result data and the human-computer interaction data, and obtain a second joint space embedding vector;

[0067] A multimodal semantic loss function is constructed by the distance between the embedding vectors of the teacher model and the student model in the subspace, and the distance is minimized by calculating the cosine similarity between two feature vectors, so that the student model can imitate the multimodal joint feature representation learned by the teacher model.

[0068] The multimodal semantic loss function is expressed as follows:

[0069]

[0070] Where: are the joint space embedding vectors obtained by the teacher model and the student model for the j-th sample respectively.

[0071] The overall loss function is the weighted combination of the warning target loss function and the multimodal semantic loss function, namely:

[0072] L=L h +αL s +βL cos

[0073] In the formula, α and β are weight coefficients, which can be adjusted according to actual conditions to balance the contribution of each loss term.

[0074] S104: Obtain the human-computer interaction data to be warned, and input the human-computer interaction data to be warned into the trained student model, so that the student model can output the early warning result of the target disease.

[0075] It is understandable that the clinical examination result data provided by the embodiments of the present application may be invasive examination result data of the patient through instruments or blood tests, and the sample human-computer interaction data may be a series of digitally measured behavioral and physiological indicators, including breathing, eye movement, voice, gait, etc. For example, in one implementation, the embodiments of the present application may provide that the sample clinical examination result data includes sample brain imaging data, the sample human-computer interaction data includes sample gait data, and the target disease includes a nervous system disease.

[0076] The student module trained by the method provided in the embodiment of the present application can achieve early and accurate warning of the target disease under the premise of only human-computer interaction data input, without the need to use clinical examination results such as brain imaging examination results as input to the model, which complies with the basic principles of low burden and low cost for early warning.

[0077] The following uses the early warning of nervous system diseases as an example to describe the method provided in the embodiments of the present application in detail. Figure 2 shown.

[0078] The method provided in the embodiment of the present application refers to and comprehensively applies the knowledge transfer method of cross-modal distillation, and proposes an early warning method for neurological diseases through human-computer interaction data based on privileged knowledge distillation. The complete multimodal input including brain imaging data and human-computer interaction data is formulated as privileged knowledge that is only available during the training process of the teacher model, so that the teacher model can learn the semantic correlation between brain imaging data and human-computer interaction data and information that is strongly related to early warning of neurological diseases, and encode the above knowledge into additional supervision through two distillation schemes, and transfer it to the student model that only receives human-computer interaction data, so as to enhance the disease warning ability of the student model when only human-computer interaction data is used, and achieve low-burden, low-cost accurate warning of neurological diseases.

[0079] The specific implementation process is as follows:

[0080] Experimental data set construction: The initial data includes brain imaging data and human-computer interaction data (gait data). The sample data is divided into training set and test set. The human-computer interaction data (gait data) is used as the target modality and can be used in both training and testing. Brain imaging data is a privileged modality that only exists in the training data set and cannot be accessed during model testing.

[0081] The training dataset containing the complete modality is fed into the teacher model, and the data of each modality is scaled to the same feature dimension so that feature decoupling and fusion can be performed on the same dimension.

[0082] Teacher model pre-training: The teacher model inputs complete modal information including brain imaging data and human-computer interaction data, and uses a shared modality encoder and two encoders for processing specific modalities (specific modality encoders) to decouple multimodal data into modality common information and modality specific information. Furthermore, these two types of features are passed to the fusion module and connected through a fully connected layer to obtain a joint representation of multimodal features.

[0083] The encoder is trained through cross entropy loss. The trained encoder module can extract the modality-independent common representation patterns between multimodal data through the shared modality encoder, so as to compensate the representation ability of the target modality (human-computer interaction data) through the privileged modality during the distillation process, and balance the contribution differences between different modalities through the specific modality encoder, enhance the discriminative features of each modality, and obtain high-quality warning results and soft labels. Finally, the teacher model can learn the joint representation of multimodal data that is strongly related to neurological diseases. This knowledge will be refined into the target modality student model that cannot access the privileged modality to improve the learning process of the student model.

[0084] In the specific implementation, the scaled multimodal features are encoded. The feature encoding module contains two specific modality encoders for encoding brain images and gait features respectively. and a shared modality encoder E for encoding shared features between modalities com The two modality-specific encoders have separate backbones and architectures and use the corresponding modality expert network to extract modality-specific features; the shared encoder shares parameters across all input modalities to extract common information across modalities. Through the feature encoding module, input information of different modalities is simultaneously fed forward into the corresponding modality-specific encoder and shared modality encoder, and outputs unique features for each modality and a shared feature vector across modalities.

[0085] The modality-specific features and shared modality features are input into the fusion module to learn multimodal joint representation. The fusion module includes a fully connected layer and a 1×1 convolutional layer. By connecting the feature vectors extracted by the encoder module, it aggregates knowledge strongly related to neurological diseases, enhances the discriminative features of each modality, and improves the early warning ability of the model.

[0086] The Softmax layer is connected to the last layer of the teacher model as a classifier, and the output category probability is used as the soft label of the teacher model. The teacher model is pre-trained using the cross entropy loss. The teacher model is trained until the loss function converges to ensure that the teacher model can provide accurate warning results and high-quality soft labels. After pre-training, the teacher model is no longer updated, and knowledge is transferred to the student network through cross-modal knowledge distillation to support the training process of the student model.

[0087] Knowledge distillation process: After pre-training, the teacher model is no longer updated, and the training data containing only the target modality is fed into the student model, and the student model is trained through knowledge distillation.

[0088] The student model uses a similar architecture to the teacher model, but because the input is a unimodal training dataset containing only gait data, the encoding module only retains the specific modality encoder corresponding to the gait, and no feature fusion is required. The student network training process is guided by the true label and two knowledge distillation losses in the distillation module to refine the knowledge in the multimodal teacher model into the unimodal student network.

[0089] The distillation module contains two distillation methods, which perform knowledge distillation by minimizing the warning target loss function and the multimodal semantic loss function. Figure 3 shown.

[0090] The warning target loss function is constructed by weighting the prediction loss and the corresponding soft label loss, where the soft label loss is the cross entropy loss between the probability distribution output by the teacher model and the student model, and the prediction loss is the cross entropy loss between the probability distribution output by the student model and the true label.

[0091] The warning target distillation method utilizes the accurate warning results and high-quality soft labels obtained by the multimodal teacher model. By introducing knowledge distillation loss, the student model is encouraged to learn the softened category distribution containing rich knowledge in the teacher model to output high-precision warning results similar to the teacher model; the warning target loss function is obtained by weighting the prediction loss and the corresponding soft label loss, that is:

[0092] L target =L h +αL s

[0093] Among them, L h , L s Represent the corresponding prediction loss and soft label loss respectively.

[0094] The prediction loss is the cross entropy loss between the probability distribution of the student model at temperature 1 and the true label, that is:

[0095]

[0096] in, They are the true label and output probability distribution of the student network when the temperature coefficient is 1.

[0097] The corresponding soft label loss is the cross entropy loss between the probability distribution of the teacher model output by the Softmax layer and the probability distribution of the student model at the same temperature, that is:

[0098]

[0099] in, They are the output probability distributions of the teacher model and the student network under the temperature coefficient T respectively.

[0100] The multimodal semantic distillation method utilizes the explicit construction of the correlation and alignment semantic information between the privileged modality and the target modality by the shared modality encoder in the teacher model. By constructing a multimodal joint representation space to aggregate samples with similar semantic information, the knowledge transfer between unpaired modalities from the teacher model to the student network is realized. By constructing the cosine similarity loss in the shared space, the student model can imitate the multimodal joint feature representation learned by the teacher model, and compensate the representation ability of human-computer interaction data through the rich semantic knowledge in the field of neurological diseases in brain imaging data, thus realizing knowledge transfer between unpaired modalities.

[0101] The multimodal semantic loss function projects the strongly correlated privileged modality and target modality features into a subspace by performing a nonlinear transformation on the feature vector output by the shared modality encoder, where semantically similar samples are closer in the subspace than dissimilar samples. The multimodal semantic loss function is then constructed using the distance between the feature vectors of the teacher model and the student model in the subspace, and the distance is minimized by calculating the cosine similarity between the two feature vectors.

[0102] The feature vector X output by the shared modality encoder in the teacher model com Perform nonlinear transformation and project it into the joint representation space of brain image data and gait data to obtain the joint space embedding vector Similarly, the same operation is performed on the feature vector output by the encoding module of the student model to obtain the joint space embedding vector

[0103] The multimodal semantic loss function is further constructed by the distance between the embedding vectors of the teacher model and the student model in the subspace. The distance is minimized by calculating the cosine similarity between the two feature vectors, so that the student model can imitate the multimodal joint feature representation learned by the teacher model. The constructed loss function is:

[0104]

[0105] in, are the joint space embedding vectors obtained by the teacher model and the student model for the j-th sample respectively.

[0106] The teacher-student model framework is trained as a whole, in which the teacher model is no longer updated. The training process of the student model is guided by the true label and two distillation losses. The overall loss function is the weighted sum of the warning target loss function and the multimodal semantic loss function, that is:

[0107] L=L h +αL s +βL cos

[0108] Among them, α and β can be adjusted according to actual conditions to balance the contribution of each loss term.

[0109] Through the distillation module, the student model is continuously supervised by the accurate warning results of the multimodal teacher model and the rich semantic information from the privileged modality during the training process, and finally obtains more accurate warning results.

[0110] The parameters of the student network are updated by back-propagation to minimize the overall loss function. After the loss function converges, the teacher model can be separated, and the student model can obtain more accurate warning results when only gait data is input.

[0111] When early warning is needed, it is only necessary to collect the human-computer interaction data to be warned, such as the gait data of the personnel, and only input the gait data into the trained student model to obtain the early warning results of the god-level systemic disease. The entire warning process does not require the provision of any invasive examination results data, which conforms to the basic principle of low burden and low cost for early warning.

[0112] In summary, the early disease warning method provided by this application utilizes the rich target disease-related knowledge in the clinical examination result data, and through a cross-modal knowledge distillation method, transfers the knowledge learned by the multimodal teacher model to the student model that only accesses the human-computer interaction modality, so as to enhance the disease warning ability of the human-computer interaction student model. Since the knowledge that is strongly related to the target disease in the clinical examination result data can be transferred to the human-computer interaction data, more accurate warning results can be obtained when only human-computer interaction data is applicable. A low-cost, low-burden early warning method for neurological diseases based on human-computer interaction data is realized, which can be used for early identification and long-term monitoring of large-scale people at risk of disease.

[0113] See also Figure 4 , the embodiment of the present application can also provide a disease early warning device, such as Figure 4 As shown, the device for executing the above-mentioned disease early warning method may include:

[0114] A data acquisition unit 401 is used to acquire sample clinical examination result data and sample human-computer interaction data for training;

[0115] The teacher model training unit 402 is used to input the sample clinical examination result data and the sample human-computer interaction data into the teacher model to train the teacher model, so that the teacher model learns the semantic correlation information between the sample clinical examination result data and the sample human-computer interaction data and the strong correlation information with the early warning of the target disease;

[0116] The knowledge distillation and student model training unit 403 is used to obtain the accurate warning results, soft labels and shared feature vectors output by the trained teacher model; perform knowledge distillation on the accurate warning results and soft labels by a warning target distillation method, and perform knowledge distillation on the shared feature vector by a multimodal semantic distillation method; so as to transfer the accurate warning results, soft labels and shared feature vectors to a student model that only receives the sample human-computer interaction data to train the student model, so that after training, the student model has the ability to provide early warning for the target disease when only receiving the sample human-computer interaction data;

[0117] The warning result output unit 404 is used to obtain the human-computer interaction data to be warned, and input the human-computer interaction data to be warned into the trained student model so that the student model outputs the early warning result of the target disease.

[0118] The embodiment of the present application may also provide a disease early warning device, the device comprising a processor and a memory:

[0119] The memory is used to store program codes and transmit the program codes to the processor;

[0120] The processor is used to execute the steps of the above-mentioned disease early warning method according to the instructions in the program code.

[0121] like Figure 5 As shown, an early warning device for a disease provided by an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 all communicate with each other through the communication bus 13.

[0122] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0123] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in an embodiment of the disease early warning method.

[0124] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0125] Obtain sample clinical examination result data and sample human-computer interaction data for training;

[0126] Inputting the sample clinical examination result data and the sample human-computer interaction data into a teacher model to train the teacher model, so that the teacher model learns the semantic correlation information between the sample clinical examination result data and the sample human-computer interaction data and the strong correlation information with the early warning of the target disease;

[0127] Acquire the accurate warning results, soft labels and shared feature vectors output by the trained teacher model; perform knowledge distillation on the accurate warning results and soft labels by means of warning target distillation, and perform knowledge distillation on the shared feature vectors by means of multimodal semantic distillation; so as to transfer the accurate warning results, soft labels and shared feature vectors to a student model that only receives the sample human-computer interaction data to train the student model, so that after training, the student model has the ability to provide early warning for the target disease when only receiving the sample human-computer interaction data;

[0128] The human-computer interaction data to be warned is obtained, and the human-computer interaction data to be warned is input into the trained student model so that the student model outputs an early warning result of the target disease.

[0129] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.

[0130] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0131] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0132] Of course, it should be noted that Figure 5 The structure shown does not constitute a limitation on the disease early warning device in the embodiment of the present application. In actual applications, the disease early warning device may include Figure 5 More or fewer components than shown, or combinations of certain components.

[0133] The embodiment of the present application may also provide a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the steps of the above-mentioned disease early warning method.

[0134] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0135] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0136] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A disease early warning method, characterized in that: include: Obtain sample clinical examination result data and sample human-computer interaction data for training; Inputting the sample clinical examination result data and the sample human-computer interaction data into a teacher model to train the teacher model, so that the teacher model learns the semantic correlation information between the sample clinical examination result data and the sample human-computer interaction data and the strong correlation information with the early warning of the target disease; Obtaining accurate warning results, soft labels, and shared feature vectors output by the trained teacher model; Performing knowledge distillation on the accurate warning result and the soft label by using a warning target distillation method, and performing knowledge distillation on the shared feature vector by using a multimodal semantic distillation method; In order to transfer the accurate warning result and the soft label and the shared feature vector to the student model that only receives the sample human-computer interaction data to train the student model, so that the student model has the early warning capability of the target disease after being trained when only receiving the sample human-computer interaction data; The human-computer interaction data to be warned is obtained, and the human-computer interaction data to be warned is input into the trained student model so that the student model outputs an early warning result of the target disease.

2. The disease early warning method according to claim 1, characterized in that: The teacher model includes a shared modality encoder, two first specific modality encoders for processing specific modalities, a fusion module and a first Softmax classifier, wherein the shared modality encoder is used to decouple the sample clinical examination result data and the sample human-computer interaction data to obtain the shared feature vector; the two first specific modality encoders are used to decouple the sample clinical examination result data and the sample human-computer interaction data to obtain a modality unique feature vector; The fusion module is used to fuse the shared feature vector and the modality-specific feature vector and input the resultant into the first Softmax classifier so that the classifier outputs the accurate warning result and soft label.

3. The disease early warning method according to claim 2, characterized in that: During the teacher model training process, a cross entropy loss function is used to train the shared modality encoder and the first specific modality encoder.

4. The disease early warning method according to claim 1, characterized in that: The student network includes a second specific modality encoder and a second Softmax classifier; the second specific modality encoder is used to decouple the sample human-computer interaction data to obtain a coded feature vector; the second Softmax classifier is used to output a predicted label according to the coded feature vector.

5. The disease early warning method according to claim 4, characterized in that: The warning target distillation method includes minimizing the warning target loss function obtained by weighting the prediction loss and the corresponding soft label loss; the soft label loss is the cross entropy loss between the probability distribution output by the teacher model and the student model, and the prediction loss is the cross entropy loss between the probability distribution output by the student model and the true label.

6. The disease early warning method according to claim 5, characterized in that: The minimization warning target loss function is expressed by the following formula: L target =L h +αL s Where: L h represents the prediction loss, L s represents the soft label loss; The prediction loss is expressed as follows: Where: They are the true label and output probability distribution of the student network when the temperature coefficient is 1; The soft label loss is expressed as follows: Where: They are the output probability distributions of the teacher model and the student model under the temperature coefficient T respectively.

7. The disease early warning method according to claim 4, characterized in that: The multimodal semantic distillation method is used to perform a nonlinear transformation on the shared feature vector output by the shared modality encoder in the teacher model, project the shared feature vector into the joint representation space of the clinical examination result data and the human-computer interaction data, and obtain a first joint space embedding vector; Projecting the encoded feature vector output by the student model encoding module into the joint representation space of clinical examination result data and human-computer interaction data to obtain a second joint space embedding vector; A multimodal semantic loss function is constructed by the distance between the embedding vectors of the teacher model and the student model in the subspace, and the distance is minimized by calculating the cosine similarity between two feature vectors, so that the student model can imitate the multimodal joint feature representation learned by the teacher model.

8. The disease early warning method according to claim 7, characterized in that: The multimodal semantic loss function is expressed as follows: Where: are the joint space embedding vectors obtained by the teacher model and the student model for the j-th sample respectively.

9. The disease early warning method according to claim 1, characterized in that: The sample clinical examination result data includes sample brain image data, the sample human-computer interaction data includes sample gait data, and the target disease includes a nervous system disease.

10. An early warning device for a disease, characterized in that: Used to implement the disease early warning method according to claims 1 to 9, comprising: A data acquisition unit, used to acquire sample clinical examination result data and sample human-computer interaction data for training; A teacher model training unit, used for inputting the sample clinical examination result data and the sample human-computer interaction data into a teacher model to train the teacher model, so that the teacher model learns the semantic correlation information between the sample clinical examination result data and the sample human-computer interaction data and the strong correlation information with the early warning of the target disease; The knowledge distillation and student model training unit is used to obtain the accurate warning results, soft labels and shared feature vectors output by the trained teacher model; perform knowledge distillation on the accurate warning results and soft labels by warning target distillation, and perform knowledge distillation on the shared feature vectors by multimodal semantic distillation; so as to transfer the accurate warning results, soft labels and shared feature vectors to the student model that only receives the sample human-computer interaction data to train the student model, so that the student model has the early warning capability of the target disease after training when only receiving the sample human-computer interaction data; The warning result output unit is used to obtain the human-computer interaction data to be warned, and input the human-computer interaction data to be warned into the trained student model so that the student model can output the early warning result of the target disease.

Images