Parkinson's disease assessment model training method, system and Parkinson's disease assessment method

Abstract

The present application relates to the field of artificial intelligence-assisted disease detection technology, and discloses a Parkinson's disease assessment model training method, system, and Parkinson's disease assessment method. The model training method includes: obtaining multiple types of facial expression images of multiple sample objects, and marking each sample object with a true category label (PD patient or non-PD patient); extracting multi-scale features through the branch network module of the initial model, and using the attention fusion module of the initial model to adaptively weighted fuse the multi-scale features to form facial expression fusion features; classifying the facial expression fusion features through the classification module of the initial model to obtain a prediction result; constructing a joint loss function based on the difference between the prediction result and the true label to optimize the classification performance of the model; training the initial model based on the joint loss function, and finally obtaining a target Parkinson's disease assessment model. The target Parkinson's disease assessment model can further improve the assessment of Parkinson's disease.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence-assisted disease detection, and in particular to a Parkinson's disease assessment model training method, system and Parkinson's disease assessment method. Background Art

[0002] At present, the diagnosis of Parkinson's disease (PD) is usually divided into two methods: in vivo diagnosis and in vitro diagnosis. In vivo diagnosis mainly relies on professional imaging equipment, such as positron emission tomography (PET). Although the in vivo diagnosis method has a high diagnostic accuracy, it also has the following limitations: on the one hand, professional imaging equipment such as PET is often scarce in developing countries, and it is very inconvenient for the elderly to travel long distances to large hospitals in developed areas for treatment; on the other hand, the cost of PET examination is high, and not every family can afford it. In contrast, in vitro diagnosis has attracted more and more attention from researchers in recent years, because it only requires the collection of in vitro biomarkers for diagnosis, such as voice signals, gait signals, and facial expressions. It can be seen that in vitro diagnosis is more convenient and has a lower cost. Among the three in vitro biomarkers mentioned above, gait signals require subjects to wear professional gait sensors, the characteristics of voice signals will be affected by the type of language the subject speaks, and the use of facial expressions to diagnose Parkinson's disease only requires the collection of facial expression images of the subject. The clinical basis for this is that Parkinson's disease patients usually have "mask face" features, that is, Parkinson's disease patients have obstacles in their ability to make facial expressions, resulting in a reduction or loss of their facial expressions. In summary, the use of facial expressions to diagnose Parkinson's disease is convenient, economical, and reliable.

[0003] The existing technology for diagnosing Parkinson's disease through facial expressions has the following main shortcomings: First, the diagnostic model uses a traditional neural network structure, such as the residual network ResNet, which cannot effectively extract facial expression features, and the extracted features are easily affected by local interference such as facial occlusion and posture changes in the face image; Second, the existing technology usually diagnoses Parkinson's disease through a specific facial expression image, such as a happy expression. However, some Parkinson's disease patients have not lost all facial expressions in the early stage of the disease, which easily leads to the model misdiagnosing Parkinson's disease patients as normal people; Third, the existing technology usually uses a simple splicing method to fuse features, without considering the importance of different types of facial expressions for Parkinson's disease diagnosis, and thus cannot achieve effective fusion of extracted features; Fourth, the current scale of facial expression data of Parkinson's disease patients is usually small, resulting in the problem of unbalanced sample category distribution when the model learns Parkinson's disease diagnosis tasks, and the existing technology usually uses a conventional binary cross entropy loss function to train the diagnostic model, without considering the negative impact of unbalanced sample category distribution on model feature learning. Therefore, there is an urgent need for an accurate and reliable intelligent assessment method for Parkinson's disease, which, on the one hand, can extract multi-scale features of the subject's facial expression image, and on the other hand, can reduce the distance between intra-class features and class centers to improve the separability between classes. Summary of the invention

[0004] Based on this, the present application proposes a Parkinson's disease assessment model training method, system and Parkinson's disease assessment method, aiming to extract multi-scale features of the subject's facial expression image and to reduce the distance between intra-class features and class centers to improve the separability between classes.

[0005] The first aspect of the present application provides a Parkinson's disease assessment model training method, the method comprising:

[0006] Acquire multiple categories of facial expression images of each sample object among a plurality of sample objects, each of the sample objects having a corresponding true category label, wherein the true category label includes PD patients and non-PD patients;

[0007] Extracting multi-scale features of the multi-class facial expression images of each sample object respectively through a branch network module in the initial Parkinson's disease assessment model, wherein the branch network module includes a branch network corresponding to each class of facial expression images;

[0008] The multi-scale features are adaptively weighted fused through the attention fusion module in the initial Parkinson's disease assessment model to obtain facial expression fusion features;

[0009] The facial expression fusion features are classified by the classification module in the initial Parkinson's disease assessment model to obtain a predicted classification result of the sample object;

[0010] According to the difference between the predicted classification results of the plurality of sample objects and the true category labels corresponding to the plurality of sample objects, a joint loss function is constructed, wherein the joint loss function includes a binary cross entropy loss function and a center loss function, wherein the binary cross entropy loss function is used to calculate the binary cross entropy loss function value between the probability that each of the sample objects is predicted to be a PD patient and the true category label, and the center loss function is used to calculate the center loss function value between the predicted sample category of each of the sample objects and the category center vector;

[0011] Based on the joint loss function, the initial Parkinson's disease assessment model is trained to obtain a target Parkinson's disease assessment model.

[0012] As an optional implementation manner of the first aspect, multi-scale features of multiple categories of facial expression images of each of the sample objects are extracted respectively through a branch network module in an initial Parkinson's disease assessment model, wherein the branch network module includes a branch network corresponding to each category of facial expression image in the step of: dividing the branch network into an initial feature extraction unit and a multi-scale feature extraction unit, wherein: the initial feature extraction unit is used to perform initial feature extraction on the facial expression image to obtain initial facial expression features; and the multi-scale feature extraction unit is used to perform multi-scale feature extraction on the initial facial expression features to obtain the multi-scale features.

[0013] As an optional implementation of the first aspect, in the step of dividing the branch network into an initial feature extraction unit and a multi-scale feature extraction unit: a convolution layer and a plurality of residual network blocks are set in the initial feature extraction unit, each of the residual network blocks includes a first convolution layer group and a first jump connection layer, wherein: the facial expression image is feature extracted through the convolution layer to obtain a first feature; the first feature is feature extracted through the first convolution layer group to obtain a second feature; the first feature and the second feature are connected through the jump connection layer to obtain the initial facial expression feature; a second convolution layer group, a dual-branch network block, a splicing layer and a second jump connection are set in the multi-scale feature extraction unit. Layer, the double-branch network block includes a left branch network and a right branch network, wherein: through the second convolutional layer group, the initial facial expression feature is extracted to obtain an intermediate feature; through the group segmentation mechanism, the intermediate feature is divided into a plurality of sub-feature groups; through the left branch network, the plurality of sub-feature groups are extracted from the order from front to back to obtain a third feature; through the right branch network, the plurality of sub-feature groups are extracted from the order from back to front to obtain a fourth feature; through the splicing layer, the third feature and the fourth feature are spliced ​​to obtain a fifth feature; through the second jump connection layer, the initial facial expression feature and the fifth feature are connected to obtain the multi-scale feature.

[0014] As an optional implementation manner of the first aspect, in the step of performing adaptive weighted fusion on the multi-scale features through the attention fusion module in the initial Parkinson's disease assessment model to obtain facial expression fusion features: a splicing layer, two fully connected layer groups and a Sigmoid function layer are set in the attention fusion module, and the two fully connected layer groups are respectively a first fully connected layer group and a second fully connected layer group, wherein: through the splicing layer, the multi-scale features of the multiple categories of facial expression images are spliced ​​to obtain splicing features; through the first fully connected layer group, the splicing features are average pooled and fully connected to obtain first connection features; through the second fully connected layer group, the splicing features are maximum pooled and fully connected to obtain second connection features; through the Sigmoid function layer, the first connection features and the second connection features are spliced ​​and mapped to obtain attention weights; through an element-by-element multiplication operation, the splicing features and the attention weights are adaptively weighted fused to obtain the facial expression fusion features.

[0015] As an optional implementation of the first aspect, according to the difference between the predicted classification results of the plurality of sample objects and the true category labels corresponding to the plurality of sample objects, in the step of constructing a joint loss function: selecting the plurality of sample objects as training samples, each of the training samples containing a true category label; calculating the probability of each of the training samples being predicted as a PD patient through the initial Parkinson's disease assessment model; calculating the binary cross entropy loss function value between the probability of each training sample being predicted as a PD patient and the true category label through the binary cross entropy loss function; selecting the plurality of sample objects as training samples and determining a category center vector; calculating the predicted sample category of each of the training samples through the initial Parkinson's disease assessment model; calculating the center loss function value between the predicted sample category of each training sample and the category center vector through the center loss function; weighting the binary cross entropy loss function value and the center loss function value to obtain a joint loss function value.

[0016] As an optional implementation manner of the first aspect, in the step of obtaining multiple categories of facial expression images of each sample object among multiple sample objects: the multiple categories of facial expression images are determined to be six different categories of facial expression images, including facial expression images corresponding to anger, facial expression images corresponding to disgust, facial expression images corresponding to fear, facial expression images corresponding to happiness, facial expression images corresponding to sadness, and facial expression images corresponding to surprise.

[0017] A second aspect of the present application provides a Parkinson's disease assessment model training system, the system comprising:

[0018] A data acquisition module, used for acquiring multiple types of facial expression images of each sample object among a plurality of sample objects, each of the sample objects having a corresponding true category label, wherein the true category label includes PD patients and non-PD patients;

[0019] Constructing an initial model module, for respectively extracting multi-scale features of multiple categories of facial expression images of each sample object through a branch network module in the initial Parkinson's disease assessment model, wherein the branch network module includes a branch network corresponding to each category of facial expression image; performing adaptive weighted fusion on the multi-scale features through an attention fusion module in the initial Parkinson's disease assessment model to obtain facial expression fusion features; performing classification processing on the facial expression fusion features through a classification module in the initial Parkinson's disease assessment model to obtain a predicted classification result of the sample object;

[0020] Constructing a loss function module, for constructing a joint loss function according to the difference between the predicted classification results of the plurality of sample objects and the true category labels corresponding to the plurality of sample objects, wherein the joint loss function includes a binary cross entropy loss function and a center loss function, wherein the binary cross entropy loss function is used to calculate the binary cross entropy loss function value between the probability of each training sample being predicted as a PD patient and the true category label, and the center loss function is used to calculate the center loss function value between the predicted sample category of each training sample and the category center vector;

[0021] The initial model training module is used to train the initial Parkinson's disease assessment model based on the joint loss function to obtain a target Parkinson's disease assessment model.

[0022] The third aspect of the present application provides a method for evaluating Parkinson's disease, the method comprising:

[0023] Obtain multiple types of facial expression images of the object to be tested;

[0024] Inputting the multi-class facial expression images into a target Parkinson's disease assessment model, wherein the target Parkinson's disease assessment model is obtained by training using the above-mentioned Parkinson's disease assessment model training method;

[0025] A prediction classification result corresponding to the object to be tested is obtained based on the target Parkinson's disease assessment model.

[0026] The fourth aspect of the present application provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement the above-mentioned Parkinson's disease assessment model training method or the above-mentioned Parkinson's disease assessment method.

[0027] The fifth aspect of the present application provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above-mentioned Parkinson's disease assessment model training method or the above-mentioned Parkinson's disease assessment method.

[0028] In summary, the present application provides a method for training a Parkinson's disease assessment model: obtaining multiple categories of facial expression images from multiple sample objects, and assigning a true category label to each sample object, which are divided into two categories: PD patients and non-PD patients; this step ensures the diversity and pertinence of the training data, which is conducive to the model learning the specificity of Parkinson's disease in facial expressions. Using the branch network module in the initial Parkinson's disease assessment model, multi-scale features are extracted for each type of facial expression image; this design can capture information at different levels of detail and improve the model's ability to recognize complex expression changes. Through the attention fusion module, the extracted multi-scale features are adaptively weighted and fused to generate facial expression fusion features; this process can give different weights according to the importance of features of different scales, so that the final feature representation is more accurate and comprehensive. The fused features are input into the classification module, and the sample objects are predicted and classified to obtain the predicted classification results of each sample object; this step is the basis for the practical application of the model and directly determines the performance of the model in actual scenarios. According to the difference between the predicted classification results and the true category labels, a joint loss function is constructed, which consists of a binary cross entropy loss function and a central loss function; the binary cross entropy loss function measures the consistency between the model prediction results and the true labels, while the central loss function focuses on the compactness of samples within each category, which helps to reduce inter-class confusion and enhance intra-class clustering. Finally, the constructed joint loss function is used to optimize the initial Parkinson's disease assessment model to obtain the target Parkinson's disease assessment model; this process continuously adjusts the model parameters so that it can more accurately predict the Parkinson's disease state when processing new facial expression data. Therefore, this model can not only identify the facial expression characteristics of Parkinson's disease patients, but also effectively distinguish the expressions of normal people, thereby improving the efficiency and accuracy of early diagnosis of Parkinson's disease. The present application also provides a Parkinson's disease assessment method, by inputting multiple categories of facial expression images of the subject to be tested into a trained target Parkinson's disease assessment model, the model will automatically analyze and give a predicted classification result, helping doctors or researchers to quickly determine whether the tested person has Parkinson's disease, thereby assisting clinical decision-making and improving the efficiency of diagnosis and treatment.

[0029] Additional aspects and advantages of the present application will be given in part in the following description, and in part will become apparent from the following description, or will be understood through the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flow chart of a Parkinson's disease assessment model training method proposed in the first embodiment of the present application;

[0031] Figure 2 This is a schematic diagram of the overall structure of the Parkinson's disease assessment model PD-Net proposed in the first embodiment of the present application;

[0032] Figure 3 This is a schematic diagram of the structure of the initial feature extraction unit proposed in the first embodiment of the present application;

[0033] Figure 4 This is a schematic diagram of the structure of the multi-scale feature extraction module proposed in the first embodiment of the present application;

[0034] Figure 5 This is a schematic diagram of the structure of the residual network block proposed in the first embodiment of the present application;

[0035] Figure 6 This is a schematic diagram of the structure of the attention fusion module proposed in the first embodiment of the present application;

[0036] Figure 7 This is a schematic diagram of the structure of the PD / non-PD classifier proposed in the first embodiment of the present application;

[0037] Figure 8 A schematic diagram of the joint loss function training method proposed in the first embodiment of the present application;

[0038] Fig. 9 A schematic diagram of the structure of a Parkinson's disease assessment model training system proposed in the second embodiment of the present application;

[0039] Fig.10 This is a flow chart of a Parkinson's disease assessment method proposed in the third embodiment of the present application.

[0040] The following specific implementation methods will further illustrate the present application in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here. In addition, the "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated with each other are in an "or" relationship.

[0043] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0044] Example 1

[0045] See also Figure 1 , is a flow chart of a Parkinson's disease assessment model training method proposed in the first embodiment of the present application. The proposed method includes S01 to S06, as follows.

[0046] S01: Acquire multiple categories of facial expression images of each sample object among multiple sample objects, each sample object having a corresponding true category label, wherein the true category label includes PD patients and non-PD patients.

[0047] In this embodiment, the multiple facial expression images specifically include an angry facial expression image, a disgusted facial expression image, a fearful facial expression image, a happy facial expression image, a sad facial expression image, and a surprised facial expression image.

[0048] Exemplarily, the present application selects the PDFE dataset, the public Oulu dataset, the public RaFD dataset, and the public TFED dataset as experimental data. Specifically, for the training set, we select facial expression images of 112 Parkinson's disease patients (PD patients) in the PDFE dataset, facial expression images of 64 normal people (non-PD patients) in the public Oulu dataset, facial expression images of 52 normal people in the public RaFD dataset, and facial expression images of 88 normal people in the public TFED dataset as the training set; for the test set, the present application selects facial expression images of 28 Parkinson's disease patients in the PDFE dataset, facial expression images of 16 normal people in the public Oulu dataset, facial expression images of 15 normal people in the public RaFD dataset, and facial expression images of 22 normal people in the public TFED dataset as the test set.

[0049] S02: extracting multi-scale features of multiple categories of facial expression images of each sample object respectively through a branch network module in the initial Parkinson's disease assessment model, wherein the branch network module includes a branch network corresponding to each category of facial expression image.

[0050] In some embodiments, the branch network is divided into an initial feature extraction unit and a multi-scale feature extraction unit, wherein: the initial feature extraction unit performs initial feature extraction on the facial expression image to obtain initial facial expression features; and the multi-scale feature extraction unit performs multi-scale feature extraction on the initial facial expression features to obtain multi-scale features.

[0051] See also Figure 2 , is the overall structure diagram of the Parkinson's disease assessment model PD-Net proposed in the first embodiment of this application. Figure 3, which is a schematic diagram of the structure of the initial feature extraction unit proposed in the first embodiment of the present application. Figure 4 , which is a schematic diagram of the structure of the multi-scale feature extraction unit proposed in the first embodiment of the present application.

[0052] In some embodiments, a convolutional layer and a plurality of residual network blocks are set in an initial feature extraction unit, and each residual network block includes a first convolutional layer group and a first jump connection layer; wherein: the facial expression image is feature extracted through the convolutional layer to obtain a first feature; the first feature is feature extracted through the first convolutional layer group to obtain a second feature; the first feature and the second feature are connected through the jump connection layer to obtain an initial facial expression feature.

[0053] Specifically, see Figure 5 , which is a schematic diagram of the structure of the residual network block proposed in the first embodiment of the present application, wherein the first convolutional layer group includes two 3x3 convolutional layers, two BN layers and two ReLU layers, the convolutional layer extracts features through convolution operations, and the jump connection connects the input and output of the residual network block, which fuses the features extracted by the convolution operation in the network block with the original input features by element-by-element addition. If the number of channels of the input feature and the output feature is different, the jump connection adjusts the number of channels of the input feature to the same as the output feature by 1x1 convolution, and then adds the two. The jump connection allows the training gradient to be transferred more smoothly from the output layer to the input layer, which can effectively alleviate the gradient vanishing problem in neural network training and avoid the phenomenon that the diagnostic performance of the PD-Net model decreases instead of increases with the increase of network depth. The working principle of the residual network block is shown in formula (1):

[0054]

[0055] In formula (1), x Represents input features; and Represents two 3x3 convolutional layers in the residual network block; Represents batch normalization operation; represents a rectified linear unit; Represents input features x Output features obtained after convolution operation, batch normalization and rectified linear unit; represents a skip connection. If the output feature With input features x The number of channels is the same, then For the identity mapping, directly convert the output features and input features x Add element by element. If the output feature is the same as the input feature x If the number of channels is different, It is a 1x1 convolution operation. First, the input feature is convolved by 1x1 convolution. x The number of channels is adjusted to be the same as the number of output feature channels. Represents the input features after adjusting the number of channels, and then the output features and Add element by element; Represents the output features of the entire residual network block.

[0056] In the initial feature extraction unit, the input facial expression image can be effectively extracted to provide high-quality initial feature representation for the subsequent multi-scale feature extraction module and attention fusion module, which is a necessary prerequisite for the entire PD-Net model to perform accurate Parkinson's disease assessment.

[0057] In some embodiments, a second convolutional layer group, a dual-branch network block, a splicing layer and a second jump connection layer are set in the multi-scale feature extraction unit, and the dual-branch network block includes a left branch network and a right branch network, wherein: the second convolutional layer group is used to extract the initial facial expression features to obtain intermediate features; the intermediate features are divided into multiple sub-feature groups through a group segmentation mechanism; the left branch network is used to extract features from the multiple sub-feature groups in a front-to-back order to obtain a third feature; the right branch network is used to extract features from the multiple sub-feature groups in a back-to-front order to obtain a fourth feature; the third feature and the fourth feature are spliced ​​through the splicing layer to obtain a fifth feature; the initial facial expression feature and the fifth feature are connected through the second jump connection layer to obtain a multi-scale feature.

[0058] Specifically, Figure 4 As shown, the second convolutional layer group includes a 3x3 convolutional layer, a BN layer and a ReLU layer. First, in the multi-scale feature extraction unit, a 3x3 convolutional layer performs a convolution operation on the input features. The process is shown in formula (2):

[0059]

[0060] In formula (2), represents the input features of the multi-scale feature extraction unit, represents a 3x3 convolutional layer, express The output features obtained after convolution operation, batch normalization and rectified linear unit are also the intermediate features of the multi-scale feature extraction unit.

[0061] Through the group segmentation mechanism, the intermediate features The channel dimension is evenly divided into 4 sub-feature groups, using Represents the first i sub-feature groups, among which The dual-branch network block contains a left-branch network and a right-branch network, which respectively perform sub-feature grouping in the order from front to back and from back to front. Multi-scale features are extracted. In each sub-branch, each sub-feature group is first element-wise added to the output obtained by the convolution operation of the previous sub-feature group, and then processed by a 3x3 convolution operation. The working principles of the left branch network and the right branch network of the dual-branch network block are shown in equations (3) and (4), respectively:

[0062]

[0063]

[0064] In formula (3) and formula (4), Indicates i Sub-feature groups, Indicates the left branch network i The convolutional layer that processes the sub-feature groups. Indicates the right branch network i The convolutional layer that processes the sub-feature groups. Represents the convolutional layer in the left branch network The output features of Represents the convolutional layer in the right branch network After processing each sub-feature group, both the left branch network and the right branch network concatenate the four extracted feature groups along the channel dimension, and then add the two concatenated features element by element. The process is shown in formula (5):

[0065]

[0066] In formula (5), represents the feature concatenation operation along the channel dimension, Represents the feature obtained by adding two concatenated features element by element. Then through the jump connection and input features Add together to obtain the output features of the multi-scale feature extraction module. The process is shown in formula (6):

[0067]

[0068] In formula (6), represents the input features of the multi-scale feature extraction unit, represents a skip connection, Represents the output features of the multi-scale feature extraction unit.

[0069] The multi-scale feature extraction unit of the present application divides the feature map into groups, then extracts features of different scales for each sub-feature group through convolution operation, and fuses the multi-scale features through jump connections to extract richer and more diverse feature representations and more comprehensively capture the information in the input feature map, thereby improving the robustness of the network model to local interferences such as facial occlusion and posture changes in facial expression images, and making the model perform better in the Parkinson's disease intelligent assessment task.

[0070] S03: Through the attention fusion module in the initial Parkinson's disease assessment model, multi-scale features are adaptively weighted fused to obtain facial expression fusion features.

[0071] It should be noted that after the six sub-branch networks of the PD-Net network extract features from the input image through the initial feature extraction unit and the multi-scale feature extraction unit, the six extracted features are fused through the attention fusion module of this application. Figure 6 , which is a structural diagram of the attention fusion module proposed in the first embodiment of the present application, wherein the first fully connected layer group includes an average pooling layer, two fully connected layers and a ReLU layer, and the second fully connected layer group includes a maximum pooling layer, two fully connected layers and a ReLU layer.

[0072] In some embodiments, a splicing layer, two fully connected layer groups and a Sigmoid function layer are set in the attention fusion module, and the two fully connected layer groups are respectively a first fully connected layer group and a second fully connected layer group, wherein: through the splicing layer, the multi-scale features of the multiple categories of facial expression images are spliced ​​to obtain splicing features; through the first fully connected layer group, the splicing features are average pooled and fully connected to obtain first connection features; through the second fully connected layer group, the splicing features are maximum pooled and fully connected to obtain second connection features; through the Sigmoid function layer, the first connection features and the second connection features are spliced ​​and mapped to obtain attention weights; through the element-by-element multiplication operation, the splicing features and the attention weights are adaptively weighted fused to obtain the facial expression fusion features.

[0073] Specifically, the attention fusion module first concatenates the features extracted by the six sub-branch networks along the channel dimension. The process is shown in formula (7):

[0074]

[0075] In formula (7), Indicates the first i The features extracted by the sub-branch network are represents the feature concatenation operation along the channel dimension, Represents the concatenated features.

[0076] Secondly, for the splicing features Perform average pooling and maximum pooling operations respectively, and then input the pooled features into the fully connected layer to obtain the corresponding attention weights. The input features are adaptively weighted by the obtained attention weights. The working principle of this part is shown in formula (8) and formula (9):

[0077]

[0078]

[0079] In formula (8), , Represent two fully connected layers, represents the average pooling operation, represents the maximum pooling operation, Represents the Sigmoid function, which can map all input values ​​to Within the range, represents the attention weight output by the Sigmoid function. In formula (9), represents the element-by-element multiplication operation, Represents the output features of the attention fusion module, that is, the features weighted by the attention weights.

[0080] In the attention fusion module, the attention fusion module of the present application calculates the attention weights of the feature channels through the average pooling layer, the maximum pooling layer and the fully connected layer, adaptively enhances the important feature channels while suppressing the irrelevant feature channels, and realizes the effective fusion and utilization of the extracted features of facial expression images of 6 basic different expressions, thereby improving the feature learning ability of the PD-Net model for the intelligent assessment task of Parkinson's disease.

[0081] S04: Classify the facial expression fusion features through the classification module in the initial Parkinson's disease assessment model to obtain the predicted classification results of the sample objects.

[0082] The PD / non-PD classifier is used as the classification module. The PD / non-PD classifier includes an average pooling layer and two fully connected layers. For the output features of the attention fusion module , and then processed by the PD / non-PD classifier to obtain the final output of the PD-Net network. Figure 7 , which is a structural diagram of the PD / non-PD classifier proposed in the first embodiment of the present application.

[0083] The processing formula of PD / non-PD classifier is expressed as:

[0084]

[0085] in, Indicates the output PD / non-PD classification result, represents the Sigmoid function, , Represent two fully connected layers, Represents an average pooling operation.

[0086] S05: Based on the difference between the predicted classification results of multiple sample objects and the true category labels corresponding to the multiple sample objects, a joint loss function is constructed. The joint loss function includes a binary cross entropy loss function and a center loss function. The binary cross entropy loss function is used to calculate the binary cross entropy loss function value between the probability of each sample object being predicted as a PD patient and the true category label. The center loss function is used to calculate the center loss function value between the predicted sample category of each sample object and the category center vector.

[0087] S06: Based on the joint loss function, train the initial Parkinson's disease assessment model to obtain the target Parkinson's disease assessment model.

[0088] In some embodiments, multiple sample objects are selected as training samples, each training sample contains a true category label; the probability of each training sample being predicted as a PD patient is calculated through an initial Parkinson's disease assessment model; the binary cross entropy loss function value between the probability of each training sample being predicted as a PD patient and the true category label is calculated through a binary cross entropy loss function; multiple sample objects are selected as training samples and a category center vector is determined; the predicted sample category of each training sample is calculated through an initial Parkinson's disease assessment model; the center loss function value between the predicted sample category of each training sample and the category center vector is calculated through a center loss function; the binary cross entropy loss function value and the center loss function value are weighted to obtain a joint loss function value.

[0089] It should be noted that the training data set consists of two parts: the first part is the facial expression images of Parkinson's disease patients, and the second part is the facial expression images of normal people. Figure 8 , which is a schematic diagram of the joint loss function training method proposed in the first embodiment of the present application.

[0090] The joint loss function consists of two loss functions, namely the binary cross entropy loss function and the center loss function. The calculation method of the binary cross entropy loss function can be expressed by formula (11):

[0091]

[0092] In formula (11), N represents the total number of samples in the training data set. Indicates i PD labels for samples, Indicates that the PD-Net model judges i The probability that a sample is a PD patient, Represents the calculated binary cross entropy loss function value.

[0093] The calculation method of the center loss function can be expressed by formula (12):

[0094]

[0095] In formula (12), represents the total number of training samples, Indicates i samples, Represents the samples obtained by the PD-Net model The characteristic representation of Representation sample The corresponding category, The value range is ,in Represents the total number of categories, here The value of is 2, corresponding to the two categories of "PD" and "non-PD". Indicates category The center vector of represents the square of the L2 norm, Represents the calculated center loss function value.

[0096] Based on the above two loss functions, the calculation method of the joint loss function used in this step can be expressed by formula (13):

[0097]

[0098] In formula (13), Represents the weight coefficient corresponding to the center loss, Represents the calculated joint loss function value.

[0099] In this step, the joint loss function can make up for the deficiency of binary cross entropy loss that ignores the distribution of intra-class sample features. The center loss function contained therein minimizes the distance between the intra-class sample features and the class center in the model feature space, thereby achieving the effect of reducing the intra-class distance and enhancing the distinction between categories, thereby alleviating the problem of unbalanced sample category distribution caused by the small scale of facial expression data of Parkinson's patients and its negative impact on the feature learning of the PD-Net model, and further improving the intelligent evaluation performance of the PD-Net model for Parkinson's disease.

[0100] During the model training process, the facial expression images of Parkinson's disease patients were labeled as "PD patients", and the facial expression images of normal people were labeled as "non-PD patients". The optimizer for model training was set to Adam optimizer, and the training learning rate was set to , the training rounds are set to 150 rounds, and the weight coefficient in the joint loss function is The value is set to 1.0. A five-fold cross-validation strategy is used to evaluate the diagnostic performance of Parkinson's disease, and the diagnostic accuracy is used as the performance evaluation indicator. In addition, as shown in Table 1 below, we selected five mainstream deep learning models, Swin-Transformer, CoAtNet, ResNet, ConvNext, and EfficientNetV2, as comparison methods.

[0101] Table 1 Comparison of PD diagnostic accuracy (%) between this application and other PD diagnostic methods

[0102]

[0103] As can be seen from Table 1, compared with the other five PD diagnosis methods based on deep learning models, this application has achieved a higher PD diagnosis accuracy, and the PD diagnosis accuracy of the second best EfficientNetV2 model has increased by 4.73%. The advantages of this application can be attributed to the following four aspects: First, this application adds an original multi-scale feature extraction module to the model network structure to extract richer and more diverse feature representations and more comprehensively capture the information in the input feature map; Second, this application uses a facial expression image of a subject under 6 basic different expressions as the model input to avoid misdiagnosis caused by the fact that some PD patients can normally make certain expressions in the early stage of the disease; Third, this application uses an original attention fusion module to adaptively weight the extracted features to promote the model to effectively fuse and utilize the features extracted from the facial expression images under 6 basic different expressions; Fourth, this application uses an original joint loss function to train and construct the Parkinson's disease intelligent assessment model PD-Net, which minimizes the distance between the sample features within the class and the class center, thereby reducing the distance within the class and enhancing the distinction between classes, thereby further improving the classification performance of the PD-Net model.

[0104] In summary, the innovations of this application are as follows:

[0105] 1. This application adds an original multi-scale feature extraction module to the network structure, extracts features of different scales from the segmented sub-feature groups through convolution operations, and fuses multi-scale features through jump connections to extract richer and more diverse feature representations, and more comprehensively capture the information in the input feature map, thereby improving the robustness of the network model to local interference such as facial occlusion and posture changes in facial expression images;

[0106] 2. This application uses facial expression images of a subject under 6 basic different expressions as model input to perform intelligent evaluation of Parkinson's disease. Compared with diagnosing Parkinson's disease based on images of only one specific facial expression, this application uses facial expression images under 6 basic different expressions for evaluation, which helps to avoid the situation where some Parkinson's patients can make certain expressions normally in the early stages of the disease, thereby reducing the risk of model misdiagnosis and improving the reliability of the evaluation results;

[0107] 3. This application proposes an original attention fusion module. Compared with the fusion method of simply splicing multiple features, the attention fusion module calculates the attention weight of the feature channel through the average pooling layer, the maximum pooling layer and the fully connected layer, so as to achieve the effect of adaptively enhancing important feature channels and suppressing irrelevant feature channels, and realize the effective fusion and utilization of features extracted from facial expression images under 6 basic different expressions, thereby improving the feature learning ability of the PD-Net model for the intelligent assessment task of Parkinson's disease;

[0108] 4. This application proposes an original joint loss function for training the constructed Parkinson's disease intelligent assessment model PD-Net. The joint loss function combines the binary cross entropy loss function and the center loss function by weighted combination. By minimizing the distance between the sample features within the class and the class center in the model feature space, it can achieve the effect of reducing the distance within the class and enhancing the distinction between classes, alleviating the problem of unbalanced sample class distribution caused by the small scale of facial expression data of Parkinson's patients and its negative impact on the feature learning of the PD-Net model, thereby further improving the accuracy of the model's Parkinson's disease assessment.

[0109] Example 2

[0110] See also Fig. 9 , shown is a schematic diagram of the structure of a Parkinson's disease assessment model training system proposed in the second embodiment of the present application, the system comprising:

[0111] A data acquisition module 100 is used to acquire multiple types of facial expression images of each sample object among a plurality of sample objects, each of the sample objects having a corresponding true category label, wherein the true category label includes PD patients and non-PD patients;

[0112] Constructing an initial model module 200, for respectively extracting multi-scale features of multiple categories of facial expression images of each sample object through a branch network module in the initial Parkinson's disease assessment model, wherein the branch network module includes a branch network corresponding to each category of facial expression image; performing adaptive weighted fusion of the multi-scale features through an attention fusion module in the initial Parkinson's disease assessment model to obtain facial expression fusion features; performing classification processing on the facial expression fusion features through a classification module in the initial Parkinson's disease assessment model to obtain a predicted classification result of the sample object;

[0113] A loss function construction module 300 is used to construct a joint loss function according to the difference between the predicted classification results of the plurality of sample objects and the true category labels corresponding to the plurality of sample objects, wherein the joint loss function includes a binary cross entropy loss function and a center loss function, wherein the binary cross entropy loss function is used to calculate the binary cross entropy loss function value between the probability of each training sample being predicted as a PD patient and the true category label, and the center loss function is used to calculate the center loss function value between the predicted sample category of each training sample and the category center vector;

[0114] The initial model training module 400 is used to train the initial Parkinson's disease assessment model based on the joint loss function to obtain a target Parkinson's disease assessment model.

[0115] Example 3

[0116] See also Fig.10 , is a flow chart of a Parkinson's disease assessment method proposed in the third embodiment of the present application, and the proposed method includes S01 to S04.

[0117] S101: Acquire multiple types of facial expression images of the object to be tested;

[0118] S102: inputting multiple types of facial expression images into a target Parkinson's disease assessment model, where the target Parkinson's disease assessment model is obtained by training using the Parkinson's disease assessment model training method of the first embodiment;

[0119] S103: Obtaining a prediction classification result corresponding to the object to be tested based on the target Parkinson's disease assessment model.

[0120] Specifically, the facial expression images of a subject to be evaluated (i.e., the object to be tested) under 6 basic different expressions are input into the trained original PD-Net model to obtain the model's intelligent evaluation results (i.e., predicted classification results) on whether the subject suffers from Parkinson's disease. The 6 basic different expressions here include anger, disgust, fear, happiness, sadness, and surprise. The working principle of this part can be expressed by formula (14):

[0121]

[0122] In formula (14), , , , , , They represent the facial expression images of the input subject under six basic different expressions: anger, disgust, fear, happiness, sadness, and surprise. represents the trained PD-Net model (i.e. the target Parkinson's disease assessment model), It represents the intelligent assessment result of Parkinson's disease output by the PD-Net model. It is a two-dimensional vector representing the probability that the PD-Net model judges the input subject as PD / non-PD.

[0123] On the other hand, the present application also proposes an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement the above-mentioned Parkinson's disease assessment model training method or the above-mentioned Parkinson's disease assessment method.

[0124] On the other hand, the present application also proposes a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above-mentioned Parkinson's disease assessment model training method or the above-mentioned Parkinson's disease assessment method.

[0125] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0126] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0127] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A Parkinson's disease assessment model training method, characterized in that: The method comprises: Acquire multiple categories of facial expression images of each sample object among a plurality of sample objects, each of the sample objects having a corresponding true category label, wherein the true category label includes PD patients and non-PD patients; The multi-scale features of the multi-class facial expression images of each sample object are extracted respectively through the branch network module in the initial Parkinson's disease assessment model, wherein the branch network module includes a branch network corresponding to each class of facial expression images; the branch network is divided into an initial feature extraction unit and a multi-scale feature extraction unit; the initial feature extraction unit is used to perform initial feature extraction on the facial expression image to obtain initial facial expression features; the multi-scale feature extraction unit is used to perform multi-scale feature extraction on the initial facial expression features to obtain the multi-scale features; A convolution layer and a plurality of residual network blocks are arranged in the initial feature extraction unit, each of the residual network blocks comprises a first convolution layer group and a first jump connection layer, wherein: the facial expression image is feature extracted through the convolution layer to obtain a first feature; the first feature is feature extracted through the first convolution layer group to obtain a second feature; the first feature and the second feature are connected through the jump connection layer to obtain the initial facial expression feature; A second convolution layer group, a dual-branch network block, a splicing layer, and a second jump connection layer are set in the multi-scale feature extraction unit, and the dual-branch network block includes a left branch network and a right branch network, wherein: the initial facial expression feature is extracted through the second convolution layer group to obtain an intermediate feature; the intermediate feature is divided into a plurality of sub-feature groups through a group segmentation mechanism; the plurality of sub-feature groups are extracted from the left branch network in a sequence from front to back to obtain a third feature; the plurality of sub-feature groups are extracted from the right branch network in a sequence from back to front to obtain a fourth feature; the third feature and the fourth feature are spliced ​​through the splicing layer to obtain a fifth feature; the initial facial expression feature and the fifth feature are connected through the second jump connection layer to obtain the multi-scale feature; The multi-scale features are adaptively weighted fused through the attention fusion module in the initial Parkinson's disease assessment model to obtain facial expression fusion features; The facial expression fusion features are classified by the classification module in the initial Parkinson's disease assessment model to obtain a predicted classification result of the sample object; According to the difference between the predicted classification results of the plurality of sample objects and the true category labels corresponding to the plurality of sample objects, a joint loss function is constructed, wherein the joint loss function includes a binary cross entropy loss function and a center loss function, wherein the binary cross entropy loss function is used to calculate the binary cross entropy loss function value between the probability that each of the sample objects is predicted to be a PD patient and the true category label, and the center loss function is used to calculate the center loss function value between the predicted sample category of each of the sample objects and the category center vector; Based on the joint loss function, the initial Parkinson's disease assessment model is trained to obtain a target Parkinson's disease assessment model.

2. A Parkinson's disease assessment model training method according to claim 1, characterized in that: In the step of performing adaptive weighted fusion on the multi-scale features through the attention fusion module in the initial Parkinson's disease assessment model to obtain facial expression fusion features: In the attention fusion module, a splicing layer, two fully connected layer groups and a Sigmoid function layer are set, and the two fully connected layer groups are respectively a first fully connected layer group and a second fully connected layer group, wherein: Through the splicing layer, the multi-scale features of the multi-class facial expression images are spliced ​​to obtain splicing features; Performing average pooling and full connection on the concatenated features through the first fully connected layer group to obtain a first connected feature; Through the second fully connected layer group, performing maximum pooling and full connection on the concatenated features to obtain a second connected feature; Through the Sigmoid function layer, the first connection feature and the second connection feature are concatenated and mapped to obtain an attention weight; The concatenated features and the attention weights are adaptively weighted fused by element-by-element multiplication to obtain the facial expression fusion features.

3. A Parkinson's disease assessment model training method according to claim 1, characterized in that: In the step of constructing a joint loss function according to the difference between the predicted classification results of the plurality of sample objects and the true category labels corresponding to the plurality of sample objects: Selecting a plurality of the sample objects as training samples, each of the training samples containing a true category label; calculating the probability of each of the training samples being predicted as a PD patient through the initial Parkinson's disease assessment model; By using the binary cross entropy loss function, a binary cross entropy loss function value between the probability of each training sample being predicted as a PD patient and the true category label is calculated; Selecting a plurality of the sample objects as training samples and determining a category center vector; calculating the predicted sample category of each of the training samples by using the initial Parkinson's disease assessment model; calculating the center loss function value between the predicted sample category of each training sample and the category center vector by using the center loss function; The binary cross entropy loss function value and the center loss function value are weighted to obtain a joint loss function value.

4. A Parkinson's disease assessment model training method according to claim 1, characterized in that: In the step of obtaining a multi-class facial expression image of each sample object in a plurality of sample objects: The multiple categories of facial expression images are determined to be six different categories of facial expression images, including facial expression images corresponding to anger, facial expression images corresponding to disgust, facial expression images corresponding to fear, facial expression images corresponding to happiness, facial expression images corresponding to sadness, and facial expression images corresponding to surprise.

5. A Parkinson's disease assessment model training system, characterized in that: The system comprises: A data acquisition module, used for acquiring multiple types of facial expression images of each sample object among a plurality of sample objects, each of the sample objects having a corresponding true category label, wherein the true category label includes PD patients and non-PD patients; An initial model module is constructed, which is used to extract multi-scale features of multiple categories of facial expression images of each sample object through a branch network module in an initial Parkinson's disease assessment model, wherein the branch network module includes a branch network corresponding to each category of facial expression image; the branch network is divided into an initial feature extraction unit and a multi-scale feature extraction unit; the initial feature extraction unit is used to perform initial feature extraction on the facial expression image to obtain initial facial expression features; the multi-scale feature extraction unit is used to perform multi-scale feature extraction on the initial facial expression features to obtain the multi-scale features; a convolution layer and a plurality of residual network blocks are set in the initial feature extraction unit, and each residual network block includes a first convolution layer group and a first jump connection layer, wherein: the convolution layer is used to perform feature extraction on the facial expression image to obtain a first feature; the first convolution layer group is used to perform feature extraction on the first feature to obtain a second feature; the first feature and the second feature are connected through the jump connection layer to obtain the initial facial expression feature; a convolution layer is set in the multi-scale feature extraction unit. A second convolution layer group, a dual-branch network block, a splicing layer and a second jump connection layer are arranged, and the dual-branch network block includes a left branch network and a right branch network, wherein: the initial facial expression feature is extracted through the second convolution layer group to obtain an intermediate feature; the intermediate feature is divided into a plurality of sub-feature groups through a group segmentation mechanism; the plurality of sub-feature groups are extracted from the left branch network in a forward-to-backward order to obtain a third feature; the plurality of sub-feature groups are extracted from the right branch network in a backward-to-front order to obtain a fourth feature; the third feature and the fourth feature are spliced ​​through the splicing layer to obtain a fifth feature; the initial facial expression feature and the fifth feature are connected through the second jump connection layer to obtain the multi-scale feature; the multi-scale feature is adaptively weighted fused through the attention fusion module in the initial Parkinson's disease assessment model to obtain a facial expression fusion feature; the facial expression fusion feature is classified through the classification module in the initial Parkinson's disease assessment model to obtain a predicted classification result of the sample object; Constructing a loss function module, for constructing a joint loss function according to the difference between the predicted classification results of the plurality of sample objects and the true category labels corresponding to the plurality of sample objects, wherein the joint loss function includes a binary cross entropy loss function and a center loss function, wherein the binary cross entropy loss function is used to calculate the binary cross entropy loss function value between the probability of each training sample being predicted as a PD patient and the true category label, and the center loss function is used to calculate the center loss function value between the predicted sample category of each training sample and the category center vector; The initial model training module is used to train the initial Parkinson's disease assessment model based on the joint loss function to obtain a target Parkinson's disease assessment model.

6. A method for evaluating Parkinson's disease, characterized in that: The method comprises: Obtain multiple types of facial expression images of the object to be tested; Inputting the multi-class facial expression images into a target Parkinson's disease assessment model, wherein the target Parkinson's disease assessment model is obtained by training the Parkinson's disease assessment model training method according to any one of claims 1 to 4; A prediction classification result corresponding to the object to be tested is obtained based on the target Parkinson's disease assessment model.

7. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the executable instructions to implement the Parkinson's disease assessment model training method according to any one of claims 1 to 4 or the Parkinson's disease assessment method according to claim 6.

8. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the Parkinson's disease assessment model training method as described in any one of claims 1 to 4 or the Parkinson's disease assessment method as described in claim 6.

Images