Alzheimer's disease early diagnosis system

Abstract

The invention discloses an Alzheimer's disease early diagnosis system, which comprises an LLM construction module used for constructing a similar natural language program LLM and comprising a data processing module and a ChatGPT module; the data acquisition module is used for acquiring to-be-tested audio data; the data processing module is used for converting the audio to be detected into spectrogram and text data through LLM; the data extraction module is used for extracting sampling data from the Alzheimer's disease data set; the model training module is used for training a Plug-in model multi-modal model by using the sampling data and outputting a prediction label and a confidence coefficient score; the construction module is used for connecting the real labels and the priori knowledge or rules as well as the predicted labels and the confidence scores to construct a context background; the initial prediction module is used for inputting the audio, the spectrogram and the text data to be detected into the trained Plug-in model multi-modal model and outputting an initial prediction result; and the final prediction module is used for fusing the context background and the initial prediction result, and outputting a final early diagnosis result through the ChatGPT module.

Description

Technical Field

[0001] The present application relates to the technical field of early diagnosis of Alzheimer's disease, and in particular to an early diagnosis system for Alzheimer's disease. Background Art

[0002] In recent years, large language models (LLMs) for various natural language processing tasks have continued to emerge, with prominent examples including GPT-3, LLaMA, ChatGPT, Wenyanyixin and the newly developed GPT-4. They have excellent emergent reasoning capabilities and have achieved remarkable results in solving various types of natural language processing tasks. With the release of ChatGPTplugins by Google this year, which can connect to hundreds of APIs, ChatGPT has set a precedent by calling third-party plug-ins to enhance its functionality and allow it to perform a wider range of operations. More LLMs will be invested in it in the future.

[0003] Alzheimer's disease (AD) is a common neurological disease in the elderly with a slow course and gradually worsening symptoms, which seriously affects the quality of life of patients. At present, the diagnosis of AD mainly relies on the doctor's clinical experience judgment of various medical tests and some traditional neuropsychological tests, which has a high misdiagnosis rate. At the same time, early detection technology for AD is more expensive imaging technology or invasive methods, which is a great burden on patients. With the rapid development of artificial intelligence technology, multimodal analysis of speech, image, and text data has been increasingly used in the field of disease diagnosis, and more and more AD diagnosis research in various directions using deep learning has been carried out. The successful use of these technologies confirms the possibility of non-invasive methods for early diagnosis of Alzheimer's disease in medical clinics, which has a broad application market.

[0004] Multimodal methods refer to combining multiple different sources of information (such as text, images, voice, etc.) to understand and process data more comprehensively and accurately. By integrating information from different sources, multimodal methods can improve the accuracy and robustness of the model and avoid the limitations of a single modality. It can also generate richer and more accurate semantic representations, thereby improving the performance of the model. At the same time, multimodal methods can make the results of the model more interpretable and explainable, making the model easier to understand and apply.

[0005] Although the current LLM has solved the limitations of being unable to access the latest information, being unable to use external tools, and being unable to perform mathematical and logical reasoning, there is almost no practical application of large language models combined with external plug-ins to use multimodal methods to solve problems in the medical field, especially in the application of early diagnosis of AD.

[0006] In summary, the current technology suffers from the following shortcomings:

[0007] 1. The large language model combined with external plug-ins and multimodal models that solve specific problems has a large gap in its ability to solve problems in specific fields and cannot play a practical role;

[0008] 2. Early diagnosis of AD rarely relies on non-invasive methods of artificial intelligence;

[0009] 3. Large language models have a common disadvantage, that is, usually the intermediate execution process of large language models will be discarded, and the context length is limited. Although it significantly improves the reasoning ability, this approach is suboptimal. Summary of the invention

[0010] In view of the above shortcomings of the prior art, the embodiment of the present application provides an early diagnosis system for Alzheimer's disease to solve the problems existing in the background technology. The present application combines LLM with other tools to build a bridge between different types of data and various models and tools to achieve early diagnosis of Alzheimer's disease.

[0011] According to an embodiment of the present application, a system for early diagnosis of Alzheimer's disease is provided, comprising:

[0012] An LLM construction module is used to use the ChatGPT model to perform feature learning on the data in the Alzheimer's disease dataset to obtain prior knowledge or rules of Alzheimer's disease, and then construct a natural language program LLM based on the ChatGPT model. The LLM includes a data processing module, a model tool module and a ChatGPT module. The data processing module is used to process audio data; the model tool module is used to store HuggingFace, PyTorch and TensorFlow model tools, and provide models and algorithms to the Plug-in model multimodal model; the ChatGPT module is used to predict output results;

[0013] A data acquisition module, used to acquire audio data to be tested in which the subject describes the contents of the picture he sees under the guidance of a doctor;

[0014] A data processing module, used for transferring the audio data to be tested into the LLM, and generating spectrogram data and text data to be tested through the data processing module;

[0015] A data extraction module, used for randomly sampling a predetermined amount of audio data, spectrogram data and text data from the Alzheimer's disease dataset as sampling data;

[0016] The model training module is used to train the plug-in model multimodal model by using the sampled data as a training set and the model tool module, and output the predicted label and confidence score of the sampled data, wherein:

[0017] The plug-in model multimodal model has a spectrogram module, an audio module, a text module, a feature fusion module, and an SVM classifier. The input of the spectrogram module is the spectrogram data, and the output is the image feature ε d , perform image classification tasks, the input of the audio module is audio data, and the output is conventional acoustic features ε c and the deep acoustic feature ε a , the input of the text module is text data, and the output is deep language features ε t , the feature fusion module combines the image feature ε d , conventional acoustic features ε c , deep acoustic features ε a and deep language features ε t After fusion, the SVM classifier is used to predict and classify the above four features;

[0018] A construction module, used to connect the real label and prior knowledge or rules of the sampled data itself and the predicted label and confidence score of the sampled data to construct a context;

[0019] An initial prediction module is used to input the audio data to be tested, the spectrogram data to be tested, and the text data to be tested into the trained Plug-in model multimodal model, and output an initial prediction result;

[0020] The final prediction module is used to fuse the context background and the initial prediction result, and output the final early diagnosis result through the ChatGPT module.

[0021] Optionally, the spectrogram module in the plug-in model multimodal model is used to convert the incoming audio data into a spectrogram through the model tool plug-in and separate it, and then input it into the fine-tuned AlexNet model. When the fine-tuned AlexNet model projects the features into the class distribution, it is necessary to add a projection layer (fully connected layer) before the fully connected layer of the original AlexNet to map the embedded features to the class distribution to obtain the image features ε of the picture information. d , is one of the features used to diagnose Alzheimer's disease.

[0022] Optionally, the audio module in the plug-in model multimodal model is used to perform noise reduction processing on the incoming audio data through the data processing plug-in, and pass the processed complete audio to the opensmile tool to extract conventional acoustic features ε c On the other hand, the audio data is segmented through the data processing plug-in and put into the fine-tuned Wav2Vec2.0 model. The embedded features extracted by the Wav2Vec2.0 model are mapped to the AD classification label using the fully connected layer to obtain the deep acoustic features ε of the audio data. a , both of which are important features in predicting Alzheimer's disease.

[0023] Optionally, the text module in the plug-in model multimodal model is used to generate transcripts of the incoming audio data using a data processing plug-in, and a fine-tuned BERT model is used to retain the CLS tag as a feature representation for the downstream text classification task, and an embedding layer is used to extract low-dimensional features, and the embedded features are projected into the category distribution through a fully connected layer to extract deep language features ε t ,This feature is observed through the logic of the text and used to diagnose Alzheimer's disease.

[0024] Optionally, training the plug-in model multimodal model to output the predicted labels and confidence scores of the sampled data includes:

[0025] The data, true labels, and prior knowledge or rules of the training set are passed into the plug-in model multimodal model for training;

[0026] The data of the training set is input into the trained Plug-in model multimodal model, and the predicted label and the corresponding confidence score are output.

[0027] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0028] It can be seen from the above embodiments that the present application combines ChatGPT + third-party plug-ins + small models + early diagnosis of Alzheimer's disease, overcoming the problem that large language models combined with external plug-ins and multimodal models that solve specific problems cannot play a practical value in solving problems in specific fields, thereby achieving the effect of playing the value of large and small models and early diagnosis of Alzheimer's disease. The method of diagnosing Alzheimer's disease using audio, text, and spectrograms overcomes the previous problem of invasive means and expensive examinations, thereby achieving the effect of using convenient non-invasive methods to diagnose Alzheimer's disease. The present application uses the SuperICL method to overcome the common shortcomings of large language model reasoning intermediate process discarding and context length limitations, thereby achieving the effect of providing rich context information and better utilizing the capabilities of large language models.

[0029] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0031] Figure 1 It is a block diagram of an early diagnosis system for Alzheimer's disease according to an exemplary embodiment.

[0032] Figure 2 The figure is a schematic diagram of using a plug-in according to an exemplary embodiment.

[0033] Figure 3 is a schematic diagram of data expansion according to an exemplary embodiment.

[0034] Figure 4 It is a workflow diagram of the combination of LLM and various plug-ins according to an exemplary embodiment.

[0035] Figure 5 It is a schematic diagram of a plug-in model multimodal model according to an exemplary embodiment.

[0036] Figure 6 It is a diagram showing a final display effect according to an exemplary embodiment. DETAILED DESCRIPTION

[0037] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0038] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0039] Figure 1 is a block diagram of an early diagnosis system for Alzheimer's disease according to an exemplary embodiment. Figure 1As shown, the system includes: LLM construction module 1, data acquisition module 2, data processing module 3, data extraction module 4, model training module 5, construction module 6, initial prediction module 7, and final prediction module 8. Each module is described in further detail below.

[0040] LLM construction module 1 is used to use the ChatGPT model to perform feature learning on the data in the Alzheimer's disease dataset to obtain prior knowledge or rules of Alzheimer's disease, and then construct a natural language program LLM based on the ChatGPT model. The LLM includes a data processing module, a model tool module and a ChatGPT module. The data processing module is used to process audio data; the model tool module is used to store HuggingFace, PyTorch and TensorFlow model tools, and provide models and algorithms to the Plug-in model multimodal model; the ChatGPT module is used to predict output results;

[0041] To achieve the above goals, the following measures are taken when combining the ChatGPT model and the natural language planner: Each module processes the query and cached context during execution and returns the result determined by the module itself. At the same time, the module updates the query and context for subsequent execution. These modules are combined into a sequential program so that subsequent modules can take advantage of the previously cached context and updated queries, that is, it is regarded as a natural language program LLM.

[0042] Since ChatGPT is not a domain-specific programming program, it is used to create a solution similar to a natural language program so that it can be extended based on natural language instructions and easily understood by users.

[0043] Here is how the natural language planner works: Given an input query a, a module library b, and constraints c, the natural language program d selects a set of modules that can be executed in sequence to generate a natural language-like program to answer the query. The module library b consists of a set of pre-built modules, denoted as {b i}, each module corresponds to a different type of tool. Constraints c are constraints generated by the program, such as the concurrency relationship and serial order of modules. In subsequent work, the natural language program d is an LLM based on a small number of examples, which is prompted to generate module names in a sequence setting. The planner is prompted with natural language to plan the task instructions e, the module description c with corresponding constraints in b, and several demonstration examples f. The plan of length g sampled from d can be expressed as p = b 1 ,…,b g , where b g represents the gth element in the generated plan, b g∈b. Formally, given an input query (question statement) a, the generated plan p is as follows:

[0044] p←d(a;e,b,c,f)

[0045] Given the generated plan, the modules corresponding to each step are then executed in sequence. The plan can be viewed as a natural language program where each module is bound only by string matching. g When evaluating, execute h g The output calculation formula is:

[0046] h g ←b g (i g-1 ;k g-1 )

[0047] where i g-1 Is the current module b g The input, k g-1 It is the cache information generated by the module execution history (such as image semantics, retrieved knowledge, and executed programs). g and the next module b g+1 Cache k g Updated by:

[0048] i g ←update_input(i g-1 ;h g ),k g ←update_cache(k g-1 ;h g )

[0049] The update_input and update_cache functions are for each b g hand-crafted. Specifically, update_input is applied to elements in the input query. These elements are updated after the module is executed. Update_cache corresponds to the generation of new information, such as a description of the input image or knowledge retrieved from a web resource. Finally, the response m to the query is generated by the last module b. g generate:

[0050] m=h g ←b g (i g-1 ;k g-1 )

[0051] In this embodiment, each module plug-in of the LLM uses Figure 2As shown, since Google has released ChatGPT plugins, which can connect APIs to call third-party plug-ins, it can be customized and called according to the task to help ChatGPT perform a wider range of operations. According to the function and purpose of the plug-in, it is divided into model tool library plug-ins, data processing plug-ins and other plug-ins, which are placed in the data processing module, model tool module and ChatGPT module respectively; model tool library plug-ins include HuggingFace, PyTorch and TensorFlow, data processing plug-ins include iFlytek, UVR5, KALDI, whisper and opensimle, and other plug-ins include answer generator and Plug-in model. At the same time, OpenAI provides two plug-ins and open-sources the code of a knowledge base retrieval plug-in, so that any developer can build plug-ins by themselves, and will use the Plug-in model multimodal model custom written in Python as a plug-in.

[0052] Call the API of the third-party plug-in. The specific construction method is as follows:

[0053] 1. Import the plugin library of the third-party plugin: According to the language and type of the plugin, import the plugin library corresponding to the plugin in the code. This can be done by using the appropriate import statement, such as import;

[0054] 2. Install dependencies: If the plugin library depends on other libraries or software packages, make sure these dependencies are installed in the environment. You can use the package manager to install the required dependencies;

[0055] 3. Initialize the plug-in: According to the requirements of the plug-in library, perform necessary initialization settings, which involve setting configuration parameters, creating instances, and loading model operations;

[0056] 4. Calling the plug-in function: using the function, class or method provided by the plug-in library to call the plug-in function;

[0057] 5. Integrate the plug-in into each module of the LLM: Integrate the calling code of the plug-in function into the corresponding position. Add the plug-in code in the code block, function or class of a specific module;

[0058] 6. Testing and debugging: Ensure that the plugin functions properly. Test and debug the code to ensure that the plugin is integrated correctly and produces the expected results.

[0059] This application introduces the LLM by calling a third-party plug-in API to process the data in the LLM to obtain a data set, and the data set can be directly stored in the database or used to train the model to obtain a model configuration file and store it in the database, such as Figure 3 As shown in the figure, the final prediction effect is achieved by using LLM.

[0060] The data acquisition module 2 is used to acquire the audio data to be tested of the subject describing the content of the picture he / she sees under the guidance of the doctor. The acquisition process is as follows:

[0061] Install the Voice Control for Chat (GPT) extension in Google. The patient will describe what he sees in the picture under the guidance of the doctor, and the audio of the description will be saved and input into the computer in real time.

[0062] This application can reflect the patient's true feelings and emotions by collecting natural reaction data and provide real-time data. Such real-time nature helps to discover potential problems and make real-time adjustments to the experimental design to ensure data quality and effectiveness.

[0063] The data processing module 3 is used to transfer the audio data to be tested into the LLM, and generate the spectrogram data and text data to be tested through the data processing module. The processing process is as follows:

[0064] Use the API of the UVR5 noise reduction tool to reduce the noise of the audio, use the API of iFlytek and whisper transcription tools to extract the transcripts of all speech audio, use the KALDI plug-in to process the audio, generate spectrograms and segment them, and all processed data will flow to the next step specified by LLM.

[0065] This application helps clean, filter and remove noise, outliers and inconsistencies in the data during data processing, thereby improving the quality and reliability of the data, which helps the model better capture patterns and regularities in the data. After data processing, while generating the audio, spectrogram and text data to be tested, the processed data is stored in a local database to solve the problem of few and small public data sets and the almost non-existent Chinese data sets, so as to achieve the purpose of expanding the Alzheimer's disease data set.

[0066] In this embodiment, since the weights of large language models are not publicly accessible, it is difficult to fine-tune the model directly. At the same time, LLM usually has a maximum context length limit. Therefore, the SuperCL method is used in the implementation process of Alzheimer's disease diagnosis, and the fine-tuned plug-in model multimodal model is combined to bypass this problem. The specific process of the method is as follows Figure 4 As shown, the method involves a data extraction module 4, a model training module 5, a construction module 6, an initial prediction module 7 and a final prediction module 8.

[0067] The data extraction module 4 is used to randomly sample a predetermined amount of audio data, spectrogram data and text data from the Alzheimer's disease dataset as sampling data. The random sampling method is used to select data samples from the Alzheimer's disease dataset to ensure the representativeness of the samples and reduce the result error caused by selection bias.

[0068] The model training module 5 is used to train the plug-in model multimodal model using the sampled data as a training set and the model tool module, and output the predicted label and confidence score of the sampled data, wherein: the plug-in model multimodal model has a spectrogram module, an audio module, a text module, a feature fusion module, and an SVM classifier, the input of the spectrogram module is the spectrogram data, and the output is the image feature ε d , perform image classification tasks, the input of the audio module is audio data, and the output is conventional acoustic features ε c and the deep acoustic feature ε a , the input of the text module is text data, and the output is deep language features ε t , the feature fusion module combines the image feature ε d , conventional acoustic features ε c , deep acoustic features ε a and deep language features ε t After fusion, the SVM classifier is used to predict and classify the above four features;

[0069] The overall structure of the plug-in model multimodal model is as follows Figure 5 As shown, the Plug-in model multimodal model includes:

[0070] Spectrogram module: used to convert the incoming audio data into a spectrogram through the model tool plug-in and separate it, and then input it into the fine-tuned AlexNet model. When the fine-tuned AlexNet model projects features into class distribution, it is necessary to add a projection layer (fully connected layer) before the fully connected layer of the original AlexNet to map the embedded features to the class distribution to obtain the image features ε of the picture information d , is one of the features used to diagnose Alzheimer's disease.

[0071] For image classification tasks, the spectrogram module can use the cross entropy loss function to measure the difference between the prediction result and the target category. For target detection tasks, the mean square error loss function can be used to measure the difference between the prediction result and the target position. Finally, the image classification loss and target detection loss can be weighted and summed to obtain the overall loss function:

[0072] L cls =-tcls *log(y cls )-(1-t cls )*log(1-y cls )

[0073] Among them, t cls is the one-hot encoding of the target category, y cls The prediction result of the classification task.

[0074] L det =(y det -t det ) 2

[0075] Among them, y det is the prediction result of the target detection task, t det is the true value of the target position.

[0076] L img =α*L cls +β*L det

[0077] Among them, L cls is the target classification result, L det is the prediction result of the target detection task, α and β are the weight coefficients of the two task losses, which are used to balance the importance of the two tasks.

[0078] Audio module: used to perform noise reduction on the incoming audio data through the data processing plug-in UVR5, and pass the processed complete audio to the opensmile tool to use IS10_paraling to extract conventional acoustic features ε c On the other hand, the audio data is segmented through the data processing plug-in and put into the fine-tuned Wav2Vec2.0 model. The embedded features extracted by the Wav2Vec2.0 model are mapped to the AD classification label using the fully connected layer to obtain the deep acoustic features ε of the audio data. a , both of which are important features in predicting Alzheimer's disease.

[0079] Fine-tune the Wav2vec2.0 encoder to generate a context feature sequence so that all context features of the long audio are pooled through the average pooling layer. In this way, the longer original audio is compressed into a 192-dimensional acoustic feature vector with the first fully connected layer (embedding layer) of the GELU activation function, and then the embedding layer projects the context features to a fixed dimension, that is, the embedded features extracted by Wav2Vec2.0 are projected onto the AD classification label through the fully connected layer. The BERT and AlexNet model fine-tuning methods are similar to Wav2vec2.0.

[0080] To be applicable to the Chinese Alzheimer's disease dataset, the cross-language speech version of Wav2Vec2.0 was fine-tuned and pre-trained on the Chinese dataset. In addition, for the analysis of the English dataset, a pre-trained model was built based on the 960-hour version of the LibriSpeech dataset, which was obtained from Hugging face.

[0081] The audio module uses the contrastive loss function for the fine-tuning part of wav2vec2.0. By minimizing the contrastive loss function, the model can be trained so that different slices of the same speech segment are closer in the embedding space, while the embeddings of different speech segments should be far away from each other, thereby obtaining an embedding representation with more speech characteristics:

[0082]

[0083] Among them, sim i,j Indicates the similarity of positive samples, sim i,k Represents the similarity of the kth negative sample. The goal of the formula is to make the similarity of positive samples higher and the similarity of negative samples lower. The exponential function exp is used to calculate the relative probability of positive and negative samples, and the logarithm is taken after normalization.

[0084] Text module: used to generate transcripts of incoming audio data using the iFlytek open platform and whisper data processing plug-in. It uses a fine-tuned BERT model, retains the CLS tag as a feature representation for downstream text classification tasks, uses an embedding layer to extract low-dimensional features, and projects the embedded features into the category distribution through a fully connected layer to extract deep language features ε t ,This feature is observed through the logic of the text and used to diagnose Alzheimer's disease.

[0085] The text module can use the cross entropy loss function to calculate the difference in text classification:

[0086]

[0087] Among them, N is the number of training samples, C is the number of categories, and y i,j is the jth element of the true label of sample i, is the probability that the sample i belongs to the jth category as predicted by the model.

[0088] Comprehensively speaking, the loss function of the plug-in model multimodal model can be expressed as:

[0089] L total =L img +L aud +L txt

[0090] Feature fusion module: Direct splicing method is used to extract the image features ε d 、IS10 parallel feature ε c , speech embedding feature ε a and text embedding feature ε t , forming the fusion feature ε.

[0091] SVM classifier: The fused feature ε is input into the SVM classifier, mapped into the feature space, and its category is determined based on its position relative to the hyperplane.

[0092] Construction module 6, used to connect the real label and prior knowledge or rules of the sampled data itself and the predicted label and confidence score of the sampled data to construct a context;

[0093] The initial prediction module 7 is used to input the audio data to be tested, the spectrogram data to be tested and the text data to be tested into the trained Plug-in model multimodal model, and output the initial prediction result;

[0094] The final prediction module 8 is used to fuse the context background and the initial prediction result, and output the final early diagnosis result through the ChatGPT module.

[0095] In the SuperICL method used in the diagnosis of Alzheimer's disease in this application, the LLM combined with the plug-in model multimodal model for prediction is not an independent model or program, but can be regarded as a combination of technologies or methods. Since the plug-in model has been fine-tuned and trained according to specific needs, it can be used as a bridge between large pre-trained models and task-specific data, providing task-specific language knowledge and predictions to achieve effective knowledge transfer and improve performance. The prediction label in the method is used to provide additional contextual information to guide LLM to learn better and provide explanation capabilities; the confidence score is used to guide LLM to make better use of the prediction label in the decision-making process, and pay more attention to contextual examples related to uncertain predictions when the plug-in model confidence is low, and can also be used as part of the explanation of the prediction results; the real label is used for the evaluation, training and improvement of ILM: prior knowledge or rules are used to improve model performance and robustness to a certain extent, provide more information and guidance, accelerate the LLM learning and reasoning process, and enhance the generalization ability of the model.

[0096] Figure 6 is a diagram showing the final display effect according to an exemplary embodiment. Figure 6As shown, the predicted results are finally combined with the final predicted probability distribution to determine the patient's Alzheimer's disease level. Specifically, the level with the highest probability can be used as the predicted result, or a certain threshold can be used to determine whether the patient has Alzheimer's disease. The optional explanation will feedback other diseases predicted by the model and give exclusion opinions.

[0097] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0098] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An early diagnosis system for Alzheimer's disease, characterized in that: include: An LLM construction module is used to use the ChatGPT model to perform feature learning on the data in the Alzheimer's disease dataset to obtain prior knowledge or rules of Alzheimer's disease, and then construct a natural language program LLM based on the ChatGPT model. The LLM includes a data processing module, a model tool module and a ChatGPT module. The data processing module is used to process audio data; the model tool module is used to store Hugging Face, PyTorch and TensorFlow model tools, and provide models and algorithms to the Plug-in model multimodal model; the ChatGPT module is used to predict output results; A data acquisition module, used to acquire audio data to be tested in which the subject describes the contents of the picture he sees under the guidance of a doctor; A data processing module, used for transferring the audio data to be tested into the LLM, and generating spectrogram data and text data to be tested through the data processing module; A data extraction module, used for randomly sampling a predetermined amount of audio data, spectrogram data and text data from the Alzheimer's disease dataset as sampling data; The model training module is used to train the plug-in model multimodal model using the sampled data as a training set and the model tool module, and output the predicted label and confidence score of the sampled data, wherein: the plug-in model multimodal model has a spectrogram module, an audio module, a text module, a feature fusion module, and an SVM classifier, the input of the spectrogram module is the spectrogram data, and the output is the image feature ε d , perform image classification tasks, the input of the audio module is audio data, and the output is conventional acoustic features ε c and the deep acoustic feature ε a , the input of the text module is text data, and the output is deep language features ε t , the feature fusion module combines the image feature ε d , conventional acoustic features ε c , deep acoustic features ε a and deep language features ε t After fusion, the SVM classifier is used to predict and classify the above four features; A construction module, used to connect the real label and prior knowledge or rules of the sampled data itself and the predicted label and confidence score of the sampled data to construct a context; An initial prediction module is used to input the audio data to be tested, the spectrogram data to be tested, and the text data to be tested into the trained Plug-in model multimodal model, and output an initial prediction result; The final prediction module is used to fuse the context background and the initial prediction result, and output the final early diagnosis result through the ChatGPT module.

2. The system according to claim 1, characterized in that The spectrogram module in the plug-in model multimodal model is used to convert the incoming audio data into a spectrogram through the model tool plug-in and separate it, and then input it into the fine-tuned AlexNet model. When the fine-tuned AlexNet model projects the features into the class distribution, it is necessary to add a projection layer (fully connected layer) before the fully connected layer of the original AlexNet to map the embedded features to the class distribution to obtain the image features ε of the picture information d , is one of the features used to diagnose Alzheimer's disease.

3. The system according to claim 1, characterized in that The audio module in the plug-in model multimodal model is used to perform noise reduction processing on the incoming audio data through the data processing plug-in, and pass the processed complete audio to the opensmile tool to extract conventional acoustic features ε c On the other hand, the audio data is segmented through the data processing plug-in and put into the fine-tuned Wav2Vec2.0 model. The embedded features extracted by the Wav2Vec2.0 model are mapped to the AD classification label using the fully connected layer to obtain the deep acoustic features ε of the audio data. a , both of which are important features in predicting Alzheimer's disease.

4. The system according to claim 1, characterized in that The text module in the plug-in model multimodal model is used to generate transcripts of the incoming audio data using the data processing plug-in. The fine-tuned BERT model is used to retain the CLS tag as the feature representation of the downstream text classification task, and the embedding layer is used to extract low-dimensional features. The embedded features are projected into the category distribution through the fully connected layer to extract deep language features ε t ,This feature is observed through the logic of the text and used to diagnose Alzheimer's disease.

5. The system according to claim 1, characterized in that The plug-in model multimodal model is trained to output the predicted labels and confidence scores of the sampled data, including: The data, true labels, and prior knowledge or rules of the training set are passed into the plug-in model multimodal model for training; The data of the training set is input into the trained Plug-in model multimodal model, and the predicted label and the corresponding confidence score are output.

Images