Early warning method and device for disease

Abstract

The application discloses a disease early warning method and device, and relates to the technical field of disease warning. The method uses rich target disease related knowledge in clinical examination result data, transfers the knowledge learned by a multi-modal teacher model to a student model which only accesses a human-computer interaction mode through a cross-modal knowledge distillation method, and improves the disease warning capability of the human-computer interaction student model. Since the knowledge strongly related to the target disease in the clinical examination result data can be transferred to the human-computer interaction data, more accurate warning results can be obtained under the condition that only the human-computer interaction data is applied. A low-cost and low-burden early warning method of nervous system diseases based on human-computer interaction data is realized, and large-scale people with the risk of diseases can be early identified and long-term monitored.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of disease early warning, in particular to a disease early warning method and device based on a privileged knowledge distillation cross-modal knowledge transfer method. BACKGROUND

[0002] With the global aging trend, the prevalence of nervous system diseases is rising, which has brought a heavy social and economic burden to our country. Due to the lack of effective treatment methods at present, when the patient has obvious symptoms, it is too late to treat, so it is crucial to discover and implement targeted intervention as early as possible.

[0003] The current commonly used early detection method of nervous system diseases, such as magnetic resonance imaging examination or cerebrospinal fluid sampling, is invasive and costly, and is not suitable for frequent detection to provide early warning or continuous tracking of disease progression, which constitutes an obstacle to the risk assessment of large-scale population, so an alternative method is needed to identify early large-scale people at risk in a low-burden and low-cost manner.

[0004] In recent years, a series of digital measurement behavior and physiological indicators based on human-computer interaction technology, including breathing, eye movement, voice and gait, have been included in the early detection system of nervous system diseases. Current research work has also proven the effectiveness of deep learning methods that use human-computer interaction data to assist in early warning of nervous system diseases, but the digital measurement values obtained only through human-computer interaction are unlikely to accurately distinguish specific nervous system diseases, and further diagnosis still needs to be combined with brain imaging examination results or clinical markers in clinical practice.

[0005] Recently, some research work combines brain imaging data with human-computer interaction data to perform early diagnosis and risk assessment of neurodegenerative diseases through multi-modal fusion and complementary methods, and has achieved good results. But this method needs to use the patient's brain imaging examination results as the input of the model, which violates the basic principle of low burden and low cost of early warning. SUMMARY

[0006] In view of the above problems, the present application provides a disease early warning method and device for overcoming the above problems or at least partially solving the above problems. By using a cross-modal knowledge transfer framework, the complete multi-modal privileged knowledge in the teacher model is refined to enhance the nervous system disease warning capability of the single-modal human-computer interaction data in the student model.

[0007] The present application provides the following solutions:

[0008] A disease early warning method, comprising:

[0009] Obtaining sample clinical examination result data and sample human-computer interaction data for training;

[0010] inputting the sample clinical examination result data and the sample human-computer interaction data into the teacher model, training the teacher model, so that the teacher model learns semantic correlation information between the sample clinical examination result data and the sample human-computer interaction data and information strongly related to early warning of a target disease;

[0011] obtaining accurate warning results and soft labels and a shared feature vector output by the trained teacher model, performing knowledge distillation on the accurate warning results and the soft labels by using a warning target distillation manner, and performing knowledge distillation on the shared feature vector by using a multi-modal semantic distillation manner, so as to transfer the accurate warning results and the soft labels and the shared feature vector to a student model that only receives the sample human-computer interaction data, train the student model, and make the student model have early warning capability of the target disease after being trained under the condition of only receiving the sample human-computer interaction data;

[0012] obtaining to-be-warned human-computer interaction data, inputting the to-be-warned human-computer interaction data into the trained student model, and outputting early warning results of the target disease by the student model.

[0013] Preferably, the teacher model comprises a shared modal encoder, two first specific modal encoders for processing specific modalities, a fusion module, and a first Softmax classifier, the shared modal encoder is used for decoupling the sample clinical examination result data and the sample human-computer interaction data to obtain the shared feature vector, and the two first specific modal encoders are used for decoupling the sample clinical examination result data and the sample human-computer interaction data to obtain modal unique feature vectors.

[0014] The fusion module is used for fusing the shared feature vector and the modal unique feature vectors and inputting the fused feature vectors into the first Softmax classifier, so that the classifier outputs the accurate warning results and the soft labels.

[0015] Preferably, a cross-entropy loss function is used to train the shared modal encoder and the first specific modal encoders in the training process of the teacher model.

[0016] Preferably, the student network comprises a second specific modal encoder and a second Softmax classifier, the second specific modal encoder is used for decoupling the sample human-computer interaction data to obtain an encoded feature vector, and the second Softmax classifier is used for outputting a predicted label according to the encoded feature vector.

[0017] Preferably, the early warning target distillation mode comprises a minimized early warning target loss function obtained by weighting and constructing a prediction loss and a corresponding soft label loss; the soft label loss is a cross-entropy loss between probability distributions output by the teacher model and the student model, and the prediction loss is a cross-entropy loss between a probability distribution output by the student model and a true label.

[0018] Preferably, the minimized early warning target loss function is represented by the following formula:

[0019] L target =L h +αL s

[0020] In the formula, L h represents the prediction loss, L s represents the soft label loss.

[0021] The prediction loss is represented by the following formula:

[0022]

[0023] In the formula, y are respectively a true label and an output probability distribution of the student network under a temperature coefficient of 1;

[0024] The soft label loss is represented by the following formula:

[0025]

[0026] In the formula, y are respectively output probability distributions of the teacher model and the student model under a temperature coefficient T.

[0027] Preferably, the multi-modal semantic distillation mode is used to perform a nonlinear transformation on the shared feature vector output by the shared modal encoder in the teacher model, project the shared feature vector into a joint representation space of clinical examination result data and human-computer interaction data, and obtain a first joint space embedding vector; project the encoded feature vector output by the encoding module of the student model into the joint representation space of the clinical examination result data and the human-computer interaction data, and obtain a second joint space embedding vector.

[0028] A multi-modal semantic loss function is constructed by a distance between embedding vectors of the teacher model and the student model in a subspace, and the distance is minimized by calculating a cosine similarity between two feature vectors, so that the student model can imitate the multi-modal joint feature representation learned by the teacher model.

[0029] Preferably, the multi-modal semantic loss function is represented by the following formula:

[0030]

[0031] In the formula: respectively are the joint space embedding vectors obtained by the teacher model and the student model respectively for the jth sample.

[0032] Preferably, the sample clinical examination result data comprises sample brain image data, the sample human-computer interaction data comprises sample gait data, and the target disease comprises a nervous system disease.

[0033] A disease early warning device for performing the disease early warning method described above, comprising:

[0034] A data acquisition unit for acquiring sample clinical examination result data and sample human-computer interaction data for training;

[0035] A teacher model training unit for inputting the sample clinical examination result data and the sample human-computer interaction data into a teacher model to train the teacher model, so that the teacher model learns semantic correlation information between the sample clinical examination result data and the sample human-computer interaction data and early warning strong correlation information with a target disease;

[0036] A knowledge distillation and student model training unit for acquiring accurate warning results and soft labels and shared feature vectors output by the trained teacher model; performing knowledge distillation on the accurate warning results and soft labels through a warning target distillation mode, and performing knowledge distillation on the shared feature vectors through a multi-modal semantic distillation mode; so as to transfer the accurate warning results and soft labels and the shared feature vectors to a student model that only receives the sample human-computer interaction data to train the student model, so that the student model has the ability of early warning of the target disease after being trained under the condition of only receiving the sample human-computer interaction data;

[0037] A warning result output unit for acquiring human-computer interaction data to be warned and inputting the human-computer interaction data to be warned into the trained student model, so that the student model outputs early warning results of the target disease.

[0038] According to the specific embodiments provided by the present application, the following technical effects are disclosed:

[0039] This application provides a method and apparatus for early disease warning. The method utilizes rich disease-related knowledge from clinical examination results data. Through cross-modal knowledge distillation, it transfers knowledge learned by a multimodal teacher model to a student model that only accesses the human-computer interaction modality, thereby improving the disease warning capability of the human-computer interaction student model. Because it can transfer knowledge strongly correlated with the target disease from clinical examination results data to human-computer interaction data, it can obtain more accurate warning results when only human-computer interaction data is used. This achieves a low-cost, low-burden early warning method for neurological diseases based on human-computer interaction data, enabling early identification and long-term monitoring of a large population at risk of developing the disease.

[0040] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0042] Figure 1 This is a flowchart of an early disease warning method provided in an embodiment of the present invention;

[0043] Figure 2 This is an overall framework flowchart provided by an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the distillation module provided in an embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of an early disease warning device provided in an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of an early disease warning device provided in an embodiment of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0048] See Figure 1 This invention provides an early disease warning method, such as...Figure 1 As shown, the method may include:

[0049] S101: Obtain clinical examination results data and human-computer interaction data of the training samples;

[0050] S102: The sample clinical examination result data and the sample human-computer interaction data are input into the teacher model to train the teacher model, so that the teacher model learns the semantic correlation information between the sample clinical examination result data and the sample human-computer interaction data, as well as the strong correlation with early warning of the target disease; in specific implementation, the embodiments of this application can provide that the teacher model includes a shared modality encoder, two first specific modality encoders for processing specific modalities, a fusion module, and a first Softmax classifier. The shared modality encoder is used to decouple the sample clinical examination result data and the sample human-computer interaction data to obtain the shared feature vector; the two first specific modality encoders are used to decouple the sample clinical examination result data and the sample human-computer interaction data to obtain modality-unique feature vectors.

[0051] The fusion module is used to fuse the shared feature vector and the modality-unique feature vector and input them into the first Softmax classifier so that the classifier can output the accurate warning result and soft label.

[0052] Furthermore, during the training process of the teacher model, the cross-entropy loss function is used to train the shared modality encoder and the first specific modality encoder.

[0053] S103: Obtain the accurate early warning results, soft labels, and shared feature vectors output by the trained teacher model; perform knowledge distillation on the accurate early warning results and soft labels using the early warning target distillation method, and perform knowledge distillation on the shared feature vectors using the multimodal semantic distillation method; so that the accurate early warning results, soft labels, and shared feature vectors can be transferred to the student model that only receives the sample human-computer interaction data to train the student model, so that the student model, after training, has the ability to provide early warning of target diseases when only receiving the sample human-computer interaction data;

[0054] The student model provided in this embodiment adopts a similar structure to the teacher model, but because the input is a single-modal training dataset containing only gait data, the encoding module only retains the specific modality encoder corresponding to the gait and does not require feature fusion. Specifically, this embodiment can provide the student network including a second specific modality encoder and a second Softmax classifier; the second specific modality encoder is used to decouple the sample human-computer interaction data to obtain an encoded feature vector; the second Softmax classifier is used to output a predicted label based on the encoded feature vector.

[0055] The distillation module includes two distillation methods, which perform knowledge distillation by minimizing the early warning target loss function and the multimodal semantic loss function, respectively.

[0056] In a specific implementation, the embodiments of this application may provide that the early warning target distillation method includes minimizing the early warning target loss function obtained by weighting the prediction loss and the corresponding soft label loss; the soft label loss is the cross-entropy loss between the probability distributions output by the teacher model and the student model, and the prediction loss is the cross-entropy loss between the probability distribution output by the student model and the true label.

[0057] The loss function for minimizing the early warning target is expressed by the following formula:

[0058] L target =L h +αL s

[0059] In the formula: L h Indicates predicted loss, L s Indicates soft label loss;

[0060] The predicted loss is expressed by the following formula:

[0061]

[0062] In the formula: These are the true labels and output probability distributions of the student network under a temperature coefficient of 1;

[0063] The soft tag loss is expressed by the following formula:

[0064]

[0065] In the formula: These represent the output probability distributions of the teacher model and the student model at a temperature coefficient T, respectively.

[0066] The multimodal semantic distillation method is used to perform a nonlinear transformation on the shared feature vector output by the shared modality encoder in the teacher model, projecting the shared feature vector onto the joint representation space of clinical examination result data and human-computer interaction data to obtain a first joint space embedding vector; and projecting the encoded feature vector output by the student model encoding module onto the joint representation space of clinical examination result data and human-computer interaction data to obtain a second joint space embedding vector.

[0067] A multimodal semantic loss function is constructed by the distance between the embedding vectors of the teacher model and the student model in the subspace. This distance is minimized by calculating the cosine similarity between the two feature vectors, so that the student model can mimic the multimodal joint feature representation learned by the teacher model.

[0068] The multimodal semantic loss function is expressed by the following equation:

[0069]

[0070] In the formula: These are the joint spatial embedding vectors obtained by the teacher model and the student model for the j-th sample, respectively.

[0071] The overall loss function is a weighted sum of the warning target loss function and the multimodal semantic loss function, that is:

[0072] L = L h +αL s +βL cos

[0073] In the formula, α and β are both weighting coefficients, which can be adjusted according to the actual situation to balance the contribution of each loss term.

[0074] S104: Obtain the human-computer interaction data to be warned and input the human-computer interaction data to be warned into the trained student model so that the student model can output the early warning result of the target disease.

[0075] It is understood that the clinical examination results data provided in this application embodiment may be data from invasive tests performed on patients using instruments or blood tests, and the sample human-computer interaction data may be a series of digitally measured behavioral and physiological indicators, including respiration, eye movement, speech, and gait, etc. For example, in one implementation, this application embodiment may provide the sample clinical examination results data including sample brain imaging data, the sample human-computer interaction data including sample gait data, and the target disease including neurological diseases.

[0076] The student module trained using the method provided in this application can achieve early and accurate warning of target diseases with only human-computer interaction data input, without the need to use clinical examination results such as brain imaging results as input to the model, which is in line with the basic principles of low burden and low cost of early warning.

[0077] The following uses early warning of nervous system diseases as an example to describe in detail the method provided in the embodiments of this application. Figure 2 As shown.

[0078] The method provided in this application references and comprehensively utilizes the cross-modal distillation knowledge transfer method, proposing a method for early warning of neurological diseases based on privileged knowledge distillation using human-computer interaction data. The complete multimodal input, including brain imaging data and human-computer interaction data, is defined as privileged knowledge usable only during the training of the teacher model. This allows the teacher model to learn the semantic correlation between brain imaging data and human-computer interaction data, as well as information strongly correlated with early warning of neurological diseases. This knowledge is then encoded as additional supervision through two distillation schemes and transferred to the student model, which only receives human-computer interaction data. This enhances the student model's disease warning capability when only human-computer interaction data is available, achieving accurate early warning of neurological diseases with low burden and low cost.

[0079] The specific implementation process is as follows:

[0080] Experimental dataset construction: The initial data includes brain imaging data and human-computer interaction data (gait data). The sample data is divided into training set and test set. The human-computer interaction data (gait data) is used as the target modality and can be used in both training and testing. The brain imaging data is used as the privileged modality and exists only in the training dataset. It cannot be accessed during model testing.

[0081] The training dataset containing the complete modalities is fed into the teacher model, and the data for each modality is scaled to the same feature dimension so that feature decoupling and fusion can be performed on the same dimension.

[0082] Teacher model pre-training: The teacher model is input with complete modal information including brain imaging data and human-computer interaction data. It uses a shared modality encoder and two modality-specific encoders (modality-specific encoders) to decouple the multimodal data into common modality information and modality-specific information. Furthermore, these two types of features are passed to the fusion module and connected through a fully connected layer to obtain a joint representation of the multimodal features.

[0083] The encoder is trained using cross-entropy loss. The trained encoder module can extract modality-independent common representation patterns among multimodal data through a shared modality encoder. This allows privileged modality compensation for the target modality's (human-computer interaction data) representational capabilities during distillation, while specific modality encoders balance the contribution differences between different modalities, enhancing the discriminative features of each modality and obtaining high-quality warning results and soft labels. Ultimately, the teacher model learns joint representations of multimodal data strongly correlated with neurological diseases. This knowledge is then refined into the target modality student model, which lacks access to privileged modalities, to improve the student model's learning process.

[0084] In practice, the scaled multimodal features are encoded. The feature encoding module contains two modality encoders, one for encoding brain images and the other for encoding gait features. And a shared modal encoder E for encoding shared features between modalities. com Two modality-specific encoders have separate backbones and architectures, and use corresponding modality expert networks to extract modality-specific features; the shared encoder shares parameters across all input modalities to extract cross-modal common information. Through the feature encoding module, input information from different modalities is simultaneously fed forward into the corresponding modality-specific encoder and the shared modality encoder, and outputs unique features for each modality as well as a shared feature vector across modalities.

[0085] Specific modality features and shared modality features are input into a fusion module to learn a multimodal joint representation. The fusion module consists of a fully connected layer and a 1×1 convolutional layer. By concatenating the feature vectors extracted by the encoder module, it aggregates knowledge strongly correlated with neurological diseases, enhances the discriminative features of each modality, and improves the model's early warning capability.

[0086] A Softmax layer is added as the classifier at the last layer of the teacher model, and the output class probability is used as the soft label for the teacher model. The teacher model is then pre-trained using cross-entropy loss. The teacher model is trained until the loss function converges to ensure it can provide accurate warnings and high-quality soft labels. After pre-training, the teacher model is no longer updated; instead, knowledge is transferred to the student network through cross-modal knowledge distillation to support the training process of the student model.

[0087] Knowledge distillation process: After pre-training, the teacher model is no longer updated. Training data containing only the target modality is fed into the student model, and the student model is trained through knowledge distillation.

[0088] The student model employs a similar architecture to the teacher model, but because the input is a single-modal training dataset containing only gait data, the encoding module retains only the specific modality encoder corresponding to the gait and does not require feature fusion. The student network training process is guided by both the ground truth labels and two knowledge distillation losses from the distillation module, to extract knowledge from the multimodal teacher model into the single-modal student network.

[0089] The distillation module includes two distillation methods: knowledge distillation is performed by minimizing the warning target loss function and the multimodal semantic loss function, respectively. For example... Figure 3 As shown.

[0090] The warning target loss function is constructed by weighting the prediction loss and the corresponding soft label loss. The soft label loss is the cross-entropy loss between the probability distributions output by the teacher model and the student model, and the prediction loss is the cross-entropy loss between the probability distribution output by the student model and the true label.

[0091] The early warning target distillation method utilizes the accurate early warning results and high-quality soft labels obtained from the multimodal teacher model. By introducing knowledge distillation loss, it encourages the student model to learn the softened category distribution rich in knowledge contained in the teacher model, so as to output high-precision early warning results similar to those of the teacher model. The early warning target loss function is obtained by weighting the prediction loss and the corresponding soft label loss, i.e.:

[0092] L target =L h +αL s

[0093] Among them, L h L s These represent the corresponding prediction loss and soft label loss, respectively.

[0094] The prediction loss is the cross-entropy loss between the student model's probability distribution at temperature 1 and the true label, i.e.:

[0095]

[0096] in, These are the true labels and output probability distributions of the student network at a temperature coefficient of 1.

[0097] The corresponding soft-label loss is the cross-entropy loss between the probability distribution output by the teacher model after the Softmax layer and the probability distribution of the student model at the same temperature, i.e.:

[0098]

[0099] in, These represent the output probability distributions of the teacher model and the student network under the temperature coefficient T, respectively.

[0100] The multimodal semantic distillation approach leverages the shared modality encoder in the teacher model to explicitly construct the correlation and alignment semantic information between privileged and target modalities. By constructing a multimodal joint representation space, it aggregates samples with similar semantic information, achieving knowledge transfer between unpaired modalities from the teacher model to the student network. Through the construction of a cosine similarity loss in the shared space, the student model can mimic the multimodal joint feature representation learned by the teacher model. Furthermore, it compensates for the representational capabilities of human-computer interaction data with rich semantic knowledge from neurological disease domains in brain imaging data, thus achieving knowledge transfer between unpaired modalities.

[0101] The multimodal semantic loss function projects strongly correlated privileged modal and target modal features into a subspace by performing a nonlinear transformation on the feature vectors of the shared modality encoder. Semantically similar samples are closer to each other in the subspace than dissimilar samples. The multimodal semantic loss function is then constructed by the distance between the feature vectors of the teacher model and the student model in the subspace, and this distance is minimized by calculating the cosine similarity between the two feature vectors.

[0102] The feature vector X output by the shared modal encoder in the teacher model com A nonlinear transformation is performed, projecting the data onto the joint representation space of brain imaging and gait data to obtain the joint spatial embedding vector. Similarly, the same operation is performed on the feature vectors output by the student model's encoding module to obtain the joint space embedding vector.

[0103] Furthermore, a multimodal semantic loss function is constructed based on the distance between the embedding vectors of the teacher model and the student model in the subspace. This distance is minimized by calculating the cosine similarity between two feature vectors, allowing the student model to mimic the multimodal joint feature representation learned by the teacher model. The constructed loss function is as follows:

[0104]

[0105] in, These are the joint spatial embedding vectors obtained by the teacher model and the student model for the j-th sample, respectively.

[0106] The teacher-student model framework is trained as a whole, with the teacher model no longer updated. The student model's training process is guided by the real labels and two distillation losses. The overall loss function is a weighted sum of the warning target loss function and the multimodal semantic loss function, i.e.:

[0107] L = L h +αL s +βL cos

[0108] Among them, α and β can be adjusted according to the actual situation to balance the contribution of each loss term.

[0109] Through the distillation module, the student model is continuously supervised by the accurate early warning results from the multimodal teacher model and the rich semantic information from the privileged modality during the training process, ultimately resulting in more accurate early warning results.

[0110] The parameters of the student network are updated by minimizing the overall loss function through backpropagation. Once the loss function converges, the student model can be separated from the teacher model, and with only gait data as input, it can obtain more accurate warning results.

[0111] When early warning is needed, only human-computer interaction data, such as gait data, needs to be collected. Simply inputting the gait data into a trained student model is sufficient to obtain early warning results for systemic diseases. The entire warning process requires no invasive examination results, adhering to the basic principles of low burden and low cost in early warning.

[0112] In summary, the disease early warning method provided in this application utilizes rich target disease-related knowledge from clinical examination results data. Through a cross-modal knowledge distillation method, it transfers knowledge learned by a multimodal teacher model to a student model that only accesses the human-computer interaction modality, thereby enhancing the disease early warning capability of the human-computer interaction student model. Because it can transfer knowledge strongly correlated with the target disease from clinical examination results data to human-computer interaction data, more accurate early warning results can be obtained when only human-computer interaction data is used. This achieves a low-cost, low-burden early warning method for neurological diseases based on human-computer interaction data, enabling early identification and long-term monitoring of large populations at risk of disease.

[0113] See Figure 4 This application embodiment can also provide an early disease warning device, such as... Figure 4 As shown, the device for performing the above-described early disease warning method may include:

[0114] The data acquisition unit 401 is used to acquire clinical examination results data of training samples and human-computer interaction data of samples.

[0115] The teacher model training unit 402 is used to input the sample clinical examination result data and the sample human-computer interaction data into the teacher model to train the teacher model, so that the teacher model can learn the semantic correlation information between the sample clinical examination result data and the sample human-computer interaction data, as well as the strong correlation with the early warning of the target disease.

[0116] The knowledge distillation and student model training unit 403 is used to acquire the accurate early warning results, soft labels, and shared feature vectors output by the trained teacher model; to perform knowledge distillation on the accurate early warning results and soft labels through early warning target distillation, and to perform knowledge distillation on the shared feature vectors through multimodal semantic distillation; so as to transfer the accurate early warning results, soft labels, and shared feature vectors to the student model that only receives the sample human-computer interaction data for training, so that the student model, after training, has the ability to provide early warning of target diseases when only receiving the sample human-computer interaction data;

[0117] The early warning result output unit 404 is used to acquire human-computer interaction data to be warned and input the human-computer interaction data to be warned into the trained student model so that the student model can output early warning results of the target disease.

[0118] This application embodiment can also provide a disease early warning device, the device including a processor and a memory:

[0119] The memory is used to store program code and transmit the program code to the processor;

[0120] The processor is used to execute the steps of the above-described early disease warning method according to the instructions in the program code.

[0121] like Figure 5 As shown in the embodiment of this application, an early disease warning device may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0122] In the embodiments of this application, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0123] The processor 10 can call programs stored in the memory 11. Specifically, the processor 10 can execute operations in the embodiments of the disease early warning method.

[0124] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0125] Acquire clinical examination results data and human-computer interaction data of the training samples;

[0126] The clinical examination results data and human-computer interaction data of the sample are input into the teacher model to train the teacher model, so that the teacher model can learn the semantic correlation information between the clinical examination results data and the human-computer interaction data of the sample, as well as the strong correlation with the early warning of the target disease.

[0127] The system acquires the accurate early warning results, soft labels, and shared feature vectors output by the trained teacher model; it performs knowledge distillation on the accurate early warning results and soft labels using a target distillation method, and performs knowledge distillation on the shared feature vectors using a multimodal semantic distillation method; so that the accurate early warning results, soft labels, and shared feature vectors can be transferred to a student model that only receives the sample human-computer interaction data for training, so that the student model, after training, has the ability to provide early warning of target diseases when only receiving the sample human-computer interaction data;

[0128] Acquire human-computer interaction data to be warned, and input the human-computer interaction data to be warned into the trained student model so that the student model can output early warning results of the target disease.

[0129] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0130] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0131] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0132] Of course, it should be noted that, Figure 5 The structure shown does not constitute a limitation on the early disease warning device in the embodiments of this application. In practical applications, the early disease warning device may include more than Figure 5 More or fewer components as shown, or combinations of certain components.

[0133] This application embodiment may also provide a computer-readable storage medium for storing program code for performing the steps of the above-described early disease warning method.

[0134] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0135] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0136] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0137] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for early warning of a disease, characterized by, The method comprises the following steps: obtaining sample clinical examination result data and sample human-computer interaction data; the sample clinical examination result data comprises brain image data, and the sample human-computer interaction data comprises gait data; inputting the sample clinical examination result data and the sample human-computer interaction data into a teacher model to train the teacher model, so that the teacher model learns semantic correlation information between the sample clinical examination result data and the sample human-computer interaction data and information strongly related to early warning of a target disease; obtaining accurate warning results, soft labels and shared feature vectors output by the trained teacher model; performing knowledge distillation on the accurate warning results and the soft labels by using a warning target distillation mode, and performing knowledge distillation on the shared feature vectors by using a multi-modal semantic distillation mode; so as to transfer the accurate warning results, the soft labels and the shared feature vectors to a student model which only receives the sample human-computer interaction data, train the student model, and make the student model have the ability of early warning of the target disease after being trained under the condition of only receiving the sample human-computer interaction data; obtaining human-computer interaction data to be warned, and inputting the human-computer interaction data to be warned into the trained student model, so that the student model outputs early warning results of the target disease; The teacher model comprises a shared modality encoder, two first specific modality encoders for processing specific modalities, a fusion module, and a first classifier, the shared modality encoder is used to decouple the sample clinical examination result data and the sample human-computer interaction data to obtain the shared feature vector; the two first specific modality encoders are used to decouple the sample clinical examination result data and the sample human-computer interaction data to obtain modality unique feature vectors; The fusion module is configured to input the shared feature vector and the modality unique feature vector after fusion into the first classifier, so that the classifier outputs the accurate early warning result and a soft label. The student model comprises a second specific modal encoder and a second classifier; the second specific modal encoder is used for decoupling the sample human-computer interaction data to obtain an encoded feature vector; the second classifier is used for outputting a predicted label according to the encoded feature vector. the multi-modal semantic distillation mode is used to perform nonlinear transformation on the shared feature vectors output by a shared modal encoder in the teacher model, project the shared feature vectors into a joint representation space of clinical examination result data and human-computer interaction data, and obtain a first joint space embedding vector; project the encoding feature vectors output by an encoding module of the student model into the joint representation space of the clinical examination result data and the human-computer interaction data, and obtain a second joint space embedding vector; construct a multi-modal semantic loss function by using distances between embedding vectors of the teacher model and the student model in a subspace, minimize the distances by calculating cosine similarity between the two feature vectors, and make the student model imitate multi-modal joint feature representations learned by the teacher model.

2. The disease early warning method of claim 1, wherein, In the training process of the teacher model, a cross-entropy loss function is used to train the shared modal encoder and the first specific modal encoder.

3. The disease early warning method of claim 1, wherein, The warning target distillation mode comprises a minimum warning target loss function obtained by weighting and constructing a prediction loss and a corresponding soft label loss; the soft label loss is a cross-entropy loss between probability distributions output by the teacher model and the student model, and the prediction loss is a cross-entropy loss between a probability distribution output by the student model and a real label.

4. The method of claim 3, wherein the disease early warning method is characterized by, The minimum warning target loss function is represented by the following formula: In the formula: represents a prediction loss, represents a soft label loss; the prediction loss is represented by the following formula: In the formula: , are the true labels and output probability distributions of the student model at a temperature of 1, respectively. the soft label loss is represented by the following formula: In the formulae: , are the output probability distributions of the teacher model and the student model, respectively, at the temperature coefficient T.

5. The disease early warning method of claim 1, wherein, the multi-modal semantic loss function is represented by the following formula: In the formula, are the first are the joint space embedding vectors obtained by the teacher model and the student model, respectively.

6. The method of claim 1, wherein the disease early warning method is characterized by, the sample clinical examination result data comprises sample brain image data, the sample human-computer interaction data comprises sample gait data, and the target disease comprises a nervous system disease.

7. A disease early warning device characterized by, The method is used for executing the disease early warning method in claims 1 to 6, and comprises: The data acquisition unit is configured to acquire sample clinical examination result data and sample human-computer interaction data for training. The teacher model training unit is configured to input the sample clinical examination result data and the sample human-computer interaction data into a teacher model, train the teacher model, so that the teacher model learns semantic correlation information between the sample clinical examination result data and the sample human-computer interaction data and information strongly related to early warning of a target disease. The knowledge distillation and student model training unit is configured to acquire accurate warning results and soft labels and shared feature vectors output by the trained teacher model; perform knowledge distillation on the accurate warning results and the soft labels by using a warning target distillation manner, and perform knowledge distillation on the shared feature vectors by using a multi-modal semantic distillation manner; and transfer the accurate warning results and the soft labels and the shared feature vectors to a student model that only receives the sample human-computer interaction data, train the student model, so that the student model has the ability of early warning of the target disease after being trained and only receives the sample human-computer interaction data. The warning result output unit is configured to acquire human-computer interaction data to be warned, input the human-computer interaction data to be warned into the trained student model, so that the student model outputs early warning results of the target disease.

Images