Alzheimer's disease facial data collection method and device based on game interaction

Abstract

The present invention discloses a facial data collection method and device for Alzheimer's disease based on game interaction, and relates to the technical field of data collection. The method comprises: obtaining human-computer interaction data and facial video data generated by Alzheimer's disease users when completing game interaction tasks, wherein the facial video data includes facial feature data and eye movement trajectory data; inputting the human-computer interaction data and facial video data into a constructed data collection model; obtaining a database based on the human-computer interaction data, facial video data and data collection model of Alzheimer's disease users; wherein the database includes human-computer interaction data and facial video data. The present invention can solve the problem of the lack of a database of Alzheimer's disease population characteristics in the prior art.

Description

Technical Field

[0001] The present invention relates to the technical field of data collection, and in particular to a method and device for collecting facial data of Alzheimer's disease based on game interaction. Background Art

[0002] AD (Alzheimer's disease) is the most common dementia. The continuous progression of the disease interferes with the daily life of the elderly, and the proportion of deaths each year increases significantly. In 2006, the global prevalence of Alzheimer's disease was 26.6 million. By 2050, the prevalence of the disease will quadruple, and 1 in 85 people worldwide will suffer from the disease. Some Alzheimer's patients develop symptoms in their 40s, 50s, and 60s and deteriorate rapidly, often losing their language ability. Alzheimer's disease is not part of normal aging, and the average age of onset is 75 years old. More and more institutions have made large investments in the early detection, prevention, and disease management of dementia, so it is crucial to have an economical, efficient, and large-scale initial screening method for Alzheimer's disease.

[0003] At present, early diagnostic tools for AD are usually some large equipment, such as FMRI (functional magnetic resonance imaging) and CT (Computed Tomography). These tools are expensive and time-consuming, and are not suitable for large-scale, wide-ranging screening in the community. In addition, facial apraxia refers to the patient's impairment in facial expression, and there are deviations in motor plans and results. Apraxia is one of the cognitive deficits characteristic of Alzheimer's disease. Although the disease is prevalent and relevant in the diagnosis of Alzheimer's disease, this topic has received little attention. There is no suitable computer task paradigm to define the relationship between facial features and the disease, as well as the relationship between the incidence of apraxia of the lower face (mouth, tongue, throat) and upper face (eyes, eyebrows) in AD patients and the severity of dementia.

[0004] Currently, AD lacks methods for collecting facial videos and establishing facial databases, which hinders its possible transformation into clinical practice. Summary of the invention

[0005] The present invention is proposed to solve the problem that the prior art lacks a database of characteristics of the Alzheimer's disease population.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In one aspect, the present invention provides a method for collecting facial data of Alzheimer's disease based on game interaction, which is implemented by an electronic device and includes:

[0008] S1. Obtaining human-computer interaction data and facial video data generated by an Alzheimer's disease user when completing a game interaction task, wherein the facial video data includes facial feature data and eye movement trajectory data.

[0009] S2. Input the human-computer interaction data and facial video data into the constructed data acquisition model.

[0010] S3. Based on the human-computer interaction data, facial video data and data acquisition model of Alzheimer's disease users, a database is obtained; wherein the database includes the human-computer interaction data and facial video data.

[0011] Optionally, the game interaction tasks include visual-spatial comprehension tasks, executive ability assessment tasks, attention assessment tasks, and memory ability assessment tasks.

[0012] The visual-spatial comprehension task includes: fruits fall one after another from the top of the smart mobile terminal display screen, and the Alzheimer's disease user catches the fruits by clicking in the specified upper and lower areas.

[0013] The executive ability assessment task includes: the Alzheimer's user slides the received fruit into the fruit basket according to the designated route.

[0014] The attention judgment task includes: the user with Alzheimer's disease continues to pay attention to the fruits in the fruit basket. If a fruit ripens and changes color, the user with Alzheimer's disease clicks on the ripe fruit within a specified time.

[0015] The memory ability assessment task includes: the smart mobile terminal display screen displays a fruit basket containing different fruits in sequence, and the Alzheimer's user memorizes the order in which the fruit basket appears within the reserved time. After the reserved time ends, the Alzheimer's user clicks on the order in which the fruit basket appears.

[0016] The human-computer interaction data include: visual-spatial comprehension task scores, executive ability assessment task scores, attention assessment task scores, and memory ability assessment task scores.

[0017] Optionally, the data acquisition model includes a data processing module, a targeted training module and a database construction module.

[0018] Based on the human-computer interaction data, facial video data and data acquisition model of Alzheimer's disease users in S3, the database obtained includes:

[0019] S31, inputting the facial video data into a data processing module to obtain the facial video data after data processing.

[0020] S32, inputting the human-computer interaction data and the facial video data into a targeted training module, and performing targeted training on the game interaction tasks whose scores are lower than a preset threshold; wherein the targeted training is repeated execution of the game interaction tasks.

[0021] S33, inputting the processed facial video data into a database construction module to obtain a database.

[0022] Optionally, the data processing module includes a video data preprocessing module, a face detection module and a feature representation module.

[0023] The data processing module in S31 processes the facial video data to obtain the facial video data after data processing, including:

[0024] S311, inputting the facial video data into a data preprocessing module to obtain preprocessed facial video data; wherein the preprocessed facial video data is the facial video data after video frame cutting, image geometry change and data augmentation are performed on the facial video data;

[0025] S312: Input the pre-processed facial video data into a face detection module to obtain facial video data after face positioning.

[0026] S313, input the facial video data after face positioning into the feature representation module to obtain feature information.

[0027] Optionally, in S313, the facial video data after face positioning is input into a feature representation module to obtain feature information including:

[0028] S3131. Extract fine-grained change features of the local region of interest of the facial video data after face localization, and establish high-dimensional features.

[0029] S3132. Obtain the mutual influence between deep-level frame features of high-dimensional features and select efficient frame features.

[0030] S3133. Perform feature weighted fusion based on efficient frame features and human-computer interaction data.

[0031] S3134. Input the weighted fused features into the support vector machine model for training to verify whether the feature information is representative.

[0032] On the other hand, the present invention provides an Alzheimer's disease facial data collection device based on game interaction, which is used to implement an Alzheimer's disease facial data collection method based on game interaction, and the device includes:

[0033] The acquisition module is used to obtain human-computer interaction data and facial video data generated by an Alzheimer's disease user when completing a game interaction task, wherein the facial video data includes facial feature data and eye movement trajectory data.

[0034] The input module is used to input human-computer interaction data and facial video data into the constructed data acquisition model.

[0035] The output module is used to obtain a database based on the human-computer interaction data, facial video data and data acquisition model of Alzheimer's disease users; wherein the database includes the human-computer interaction data and facial video data.

[0036] Optionally, the game interaction tasks include visual-spatial comprehension tasks, executive ability assessment tasks, attention assessment tasks, and memory ability assessment tasks.

[0037] The visual-spatial comprehension task includes: fruits fall one after another from the top of the smart mobile terminal display screen, and the Alzheimer's disease user catches the fruits by clicking in the specified upper and lower areas.

[0038] The executive ability assessment task includes: the Alzheimer's user slides the received fruit into the fruit basket according to the designated route.

[0039] The attention judgment task includes: the user with Alzheimer's disease continues to pay attention to the fruits in the fruit basket. If a fruit ripens and changes color, the user with Alzheimer's disease clicks on the ripe fruit within a specified time.

[0040] The memory ability assessment task includes: the smart mobile terminal display screen displays a fruit basket containing different fruits in sequence, and the Alzheimer's user memorizes the order in which the fruit basket appears within the reserved time. After the reserved time ends, the Alzheimer's user clicks on the order in which the fruit basket appears.

[0041] The human-computer interaction data include: visual-spatial comprehension task scores, executive ability assessment task scores, attention assessment task scores, and memory ability assessment task scores.

[0042] Optionally, the output module is further configured to:

[0043] S31, inputting the facial video data into a data processing module to obtain the facial video data after data processing.

[0044] S32, inputting the human-computer interaction data and the facial video data into a targeted training module, and performing targeted training on the game interaction tasks whose scores are lower than a preset threshold; wherein the targeted training is repeated execution of the game interaction tasks.

[0045] S33, inputting the processed facial video data into a database construction module to obtain a database.

[0046] Optionally, the output module is further configured to:

[0047] S311, inputting the facial video data into a data preprocessing module to obtain preprocessed facial video data; wherein the preprocessed facial video data is the facial video data after video frame cutting, image geometry change and data augmentation.

[0048] S312: Input the pre-processed facial video data into a face detection module to obtain facial video data after face positioning.

[0049] S313, input the facial video data after face positioning into the feature representation module to obtain feature information.

[0050] Optionally, the output module is further configured to:

[0051] S3131. Extract fine-grained change features of the local region of interest of the facial video data after face localization, and establish high-dimensional features.

[0052] S3132. Obtain the mutual influence between deep-level frame features of high-dimensional features and select efficient frame features.

[0053] S3133. Perform feature weighted fusion based on efficient frame features and human-computer interaction data.

[0054] S3134. Input the weighted fused features into the support vector machine model for training to verify whether the feature information is representative.

[0055] On the one hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned Alzheimer's disease facial data collection method based on game interaction.

[0056] On the one hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned Alzheimer's disease facial data collection method based on game interaction.

[0057] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0058] In the above scheme, facial data collection for Alzheimer's disease is based on facial features. Android games are used as stimulation to record facial changes during the entire game interaction process. Facial features are extracted and calculated through algorithms. Human-computer interaction features and eye movement features are fused with facial features in the form of local attention and related attention, respectively. Therefore, this application is a data collection process with facial features, pupil trajectories and human-computer interaction behaviors as the core. The database source for Alzheimer's disease recognition in this application is different from the general database. It abandons functional magnetic resonance imaging or EEG image classification. These tools are expensive and time-consuming, and are not suitable for large-scale detection and screening. Instead, it uses game interaction to obtain facial video data. This method has simple collection equipment and short detection time, and is widely used in the community.

[0059] This application collects facial video data of users in game interactions and analyzes the collected data. Game tasks are designed based on visual-spatial comprehension ability, execution, sustained attention, memory, etc. The method described in this application can comprehensively obtain the state of changes in user facial features in game interactions. It provides a dimensional reference for medical testing, helps future research and clinical applications, and provides new means and new ways to break through the establishment of a database of Alzheimer's disease population characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 It is a schematic diagram of the process of the facial data collection method of Alzheimer's disease based on game interaction of the present invention;

[0062] Figure 2 It is a schematic diagram of the process of the facial data collection method of Alzheimer's disease based on game interaction of the present invention;

[0063] Figure 3 It is a schematic diagram of obtaining human-computer interaction data and facial video data of the present invention;

[0064] Figure 4 It is a schematic diagram of the targeted training process of the present invention;

[0065] Figure 5 It is a block diagram of the facial data collection device for Alzheimer's disease based on game interaction of the present invention;

[0066] Figure 6It is a structural schematic diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0067] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0068] like Figure 1 As shown, an embodiment of the present invention provides a method for collecting facial data of Alzheimer's disease based on game interaction, which can be implemented by an electronic device. Figure 1 The flowchart of the method for collecting facial data of Alzheimer's disease based on game interaction is shown in the figure. The processing flow of the method may include the following steps:

[0069] S11. Obtaining human-computer interaction data and facial video data generated by an Alzheimer's disease user when completing a game interaction task, wherein the facial video data includes facial feature data and eye movement trajectory data.

[0070] S12. Input the human-computer interaction data and facial video data into the constructed data acquisition model.

[0071] S13. Obtain a database based on the human-computer interaction data, facial video data and data acquisition model of Alzheimer's disease users; wherein the database includes the human-computer interaction data and facial video data.

[0072] Optionally, the game interaction tasks include visual-spatial comprehension tasks, executive ability assessment tasks, attention assessment tasks, and memory ability assessment tasks.

[0073] The visual-spatial comprehension task includes: fruits fall one after another from the top of the smart mobile terminal display screen, and the Alzheimer's disease user catches the fruits by clicking in the specified upper and lower areas.

[0074] The executive ability assessment task includes: the Alzheimer's user slides the received fruit into the fruit basket according to the designated route.

[0075] The attention judgment task includes: the user with Alzheimer's disease continues to pay attention to the fruits in the fruit basket. If a fruit ripens and changes color, the user with Alzheimer's disease clicks on the ripe fruit within a specified time.

[0076] The memory ability assessment task includes: the smart mobile terminal display screen displays a fruit basket containing different fruits in sequence, and the Alzheimer's user memorizes the order in which the fruit basket appears within the reserved time. After the reserved time ends, the Alzheimer's user clicks on the order in which the fruit basket appears.

[0077] The human-computer interaction data include: visual-spatial comprehension task scores, executive ability assessment task scores, attention assessment task scores, and memory ability assessment task scores.

[0078] Optionally, the data acquisition model includes a data processing module, a targeted training module and a database construction module.

[0079] In S13, based on the human-computer interaction data of Alzheimer's disease users, facial video data and data acquisition model, the database obtained includes:

[0080] S131, inputting facial video data into a data processing module to obtain facial video data after data processing.

[0081] S132. Input the human-computer interaction data and the facial video data into a targeted training module, and perform targeted training on the game interaction tasks whose scores are lower than a preset threshold; wherein the targeted training is repeated execution of the game interaction tasks.

[0082] S133, inputting the processed facial video data into a database construction module to obtain a database.

[0083] Optionally, the data processing module includes a video data preprocessing module, a face detection module and a feature representation module.

[0084] The data processing module in S131 processes the facial video data to obtain the facial video data after data processing, including:

[0085] S1311, inputting the facial video data into a data preprocessing module to obtain preprocessed facial video data; wherein the preprocessed facial video data is the facial video data after video frame cutting, image geometry change, and data augmentation are performed on the facial video data;

[0086] S1312: Input the pre-processed facial video data into a face detection module to obtain facial video data after face positioning.

[0087] S1313, input the facial video data after face positioning into the feature representation module to obtain feature information.

[0088] Optionally, in S1313, the facial video data after face positioning is input into a feature representation module to obtain feature information including:

[0089] S13131. Extract fine-grained change features of the local region of interest of the facial video data after face localization, and establish high-dimensional features.

[0090] S13132. Obtain the mutual influence between deep-level frame features of high-dimensional features and select efficient frame features.

[0091] S13133. Perform feature weighted fusion based on efficient frame features and human-computer interaction data.

[0092] S13134. Input the weighted fused features into the support vector machine model for training to verify whether the feature information is representative.

[0093] In the embodiment of the present invention, facial data of Alzheimer's disease is collected based on facial features, using Android games as stimulation, recording facial changes during the entire game interaction process, extracting and calculating facial features through algorithms, and integrating human-computer interaction features and eye movement features with facial features in the form of local attention and related attention, respectively. Therefore, this application is a data collection process with facial features, pupil trajectories, and human-computer interaction behaviors as the core. The database source for Alzheimer's disease recognition in this application is different from the general database. It abandons functional magnetic resonance imaging or EEG image classification, which are expensive and time-consuming, and are not suitable for large-scale detection and screening. Instead, it uses game interaction to obtain facial video data. This method has simple collection equipment and short detection time, and is widely used in the community.

[0094] This application collects facial video data of users in game interactions and analyzes the collected data. Game tasks are designed based on visual-spatial comprehension ability, execution, sustained attention, memory, etc. The method described in this application can comprehensively obtain the state of changes in user facial features in game interactions. It provides a dimensional reference for medical testing, helps future research and clinical applications, and provides new means and new ways to break through the establishment of a database of Alzheimer's disease population characteristics.

[0095] like Figure 2 As shown, an embodiment of the present invention provides a method for collecting facial data of Alzheimer's disease based on game interaction, which can be implemented by an electronic device. Figure 2 The flowchart of the method for collecting facial data of Alzheimer's disease based on game interaction is shown in the figure. The processing flow of the method may include the following steps:

[0096] S21. Obtain human-computer interaction data and facial video data generated by an Alzheimer's disease user when completing a game interaction task.

[0097] Among them, facial video data includes facial feature data and eye movement trajectory data. The game scores, facial features and eye movement trajectories of Alzheimer's users during game interaction are a kind of feedback on the current cognitive level, which has a certain universality. Real-time game interaction reflects the subject's visual-spatial understanding, execution ability, concentration ability and memory ability of the current task. Facial features reflect the subject's stimulation feedback on the success and failure of the current game. Eye movement trends reflect the subject's concentration during the game interaction. The fusion of the three constitutes the subject's state during the interaction process.

[0098] Specifically, the eye movement trajectory data can be obtained by the eye movement trajectory processing module, which is an eye movement trajectory tracking and feature extraction module. The eye movement tracking system can track the movement trajectory of the eye, thereby obtaining indicators such as the gaze point position, gaze trajectory, gaze time, and gaze frequency of the human eye movement, and by observing and studying the sight line trajectory diagram, the user's concentration on the current task can be obtained. For eye movement feature extraction, the image of the pupil of the eye is first segmented by an image processing method, and then the texture feature of the pupil image is extracted. The image of the eye area is obtained by face detection, eye key point positioning, and image segmentation. Then, based on the principle that the pupil is always darker than other parts of the eye, the located eye area is gray-scale converted, and the threshold obtained by the experimental data is used to separate the pupil from the eye area according to the threshold. For eye movement trajectory movement, only the pupil needs to be paid attention to, because the pupil can reflect the user's focus point, and then feature extraction is performed.

[0099] like Figure 3 As shown, the overall process of the game interaction task may include user information entry, game tasks, video recording, and generation of a game task database.

[0100] Among them, user information entry mainly records personal information. Users log in to the system by entering their name, age, and ID number. This information is required and plays a key role in subsequent data recording and identity authentication. After the user successfully logs into the account, the system automatically enters the basic information into the user database for storage before the user can enter the subsequent game tasks.

[0101] Optionally, the game tasks in the game interaction task include visual-spatial understanding tasks, executive ability assessment tasks, attention assessment tasks, and memory ability assessment tasks. Correspondingly, the human-computer interaction data may include: visual-spatial understanding task scores, executive ability assessment task scores, attention assessment task scores, and memory ability assessment task scores. The following introduces these four game tasks respectively:

[0102] (1) The visual-spatial comprehension task includes: fruits are dropped from the top of the display screen of a smart mobile terminal, and the Alzheimer's disease user catches the fruits by clicking in the specified upper and lower areas.

[0103] In a feasible implementation, the visual-spatial understanding task collects facial video data and interactive behavior data in this process by the user's judgment of the relative position of the fruit. The test is divided into three levels. The user is considered successful only if he catches the fruit within the interval indicated by the upper and lower horizontal lines. The difficulty of the current module task is achieved by changing the speed at which the fruit falls. There is a hand-shaped symbol on the right end of the screen. The user moves the hand-shaped symbol by clicking on the screen. After contacting the object, a pleasant prompt sound is triggered; if the user clicks the fruit outside the interval or clicks in the wrong position, a harsh sound prompts the failure of the task. In the case of successful contact by the subject, the subject's score will be continuously updated on the screen, with success corresponding to plus points and failure corresponding to minus points. The number of clicks by the user, the degree of deviation of the click position, and the time spent on each fruit are automatically recorded.

[0104] (2) The execution ability assessment task includes: the Alzheimer's disease user slides the received fruit into the fruit basket according to the designated route.

[0105] In a feasible implementation, the ability to perform evaluation tasks collects facial video data and interactive behavior data during the process by the user moving along a predetermined route. The test is divided into three levels, and the difficulty of each level is achieved through the complexity of the route. The user needs to drag the fruit along the route given by the system until the fruit enters the fruit basket to indicate success. If the deviation between the fruit and the route exceeds the threshold during the entire dragging process, a harsh sound will be triggered to prompt the task failure. After the fruit enters the fruit basket, a pleasant prompt sound will be triggered to indicate success. The time it takes the user to drag each fruit and the number of fruits that successfully enter the fruit basket are automatically recorded.

[0106] (3) The attention assessment task includes: the user with Alzheimer's disease continuously pays attention to the fruits in the fruit basket. If a fruit ripens and changes color, the user with Alzheimer's disease clicks on the ripe fruit within a specified time.

[0107] In a feasible implementation, the attention assessment task collects facial video data and interactive behavior data during the process by having the user pay attention to the changing fruit. The test is divided into three levels, and the difficulty of each level is achieved by controlling the reaction time. The ripening and color change of the fruit occurs randomly, and the user needs to click on the ripe fruit within the allowed reaction time. If the time exceeds the threshold or the click is wrong, a harsh sound will be triggered to prompt the task failure, and a pleasant prompt sound will be triggered if it is successful. The time from the ripening and color change of the fruit to the user's reaction and click is automatically recorded, as well as the number of successful clicks by the user and the degree of click deviation.

[0108] (4) The memory ability assessment task includes: the smart mobile terminal display screen displays a fruit basket containing different fruits in sequence, and the Alzheimer's user memorizes the order in which the fruit basket appears within a reserved time. After the reserved time ends, the Alzheimer's user clicks on the order in which the fruit basket appears.

[0109] In a feasible implementation, the memory ability evaluation task collects facial video data and interactive behavior data during the process by stimulating the user's memory ability. The test is divided into three levels, and the difficulty is manipulated by changing the number of items that need to be remembered. The system will randomly place some fruit baskets on the screen, and the user has a certain amount of time to remember the order in which the fruit baskets appear. Then the screen is cleared, and the system will shuffle all the fruit baskets and require the user to select the fruit baskets in the order just now. Finally, the user's time and the order of selection are automatically recorded to judge the user's accuracy.

[0110] When the user completes the above-mentioned game tasks, the electronic device can record videos. The video recording mainly records the facial video data and interactive behavior data during each task of the game. The data is automatically recorded throughout the process. The video recording function is automatically started at the beginning of the game to save the user's facial performance in the game interaction in the form of a video. The entire recording interface is hidden to ensure the true display of the user's facial video. In addition, the number of clicks, the time interval between clicks, the number of clicks in the wrong position, and the straight-line distance between each click and the object are recorded. These operation data are convenient for analyzing the user's execution ability. The length of time the user takes is also recorded throughout the process to facilitate the analysis of the user's memory ability.

[0111] In a feasible implementation, generating a game task database mainly involves storing various data in game interactions in the form of a database. The game task database mainly includes input personal information, facial video data of each task and corresponding interaction data, and the user's corresponding cognitive state.

[0112] S22: input the facial video data into a data preprocessing module to obtain preprocessed facial video data.

[0113] The preprocessed facial video data is the facial video data after video frame capture, image geometry change, and data augmentation.

[0114] In a feasible implementation, appropriate preprocessing can reduce the impact of image quality on recognition results and also improve the robustness of the algorithm. The above preprocessing is a common processing method in the prior art and need not be elaborated in detail in the embodiments of the present invention.

[0115] S23, inputting the pre-processed facial video data into a face detection module to obtain facial video data after face positioning.

[0116] In a feasible implementation, face detection needs to locate the position of the face in the image. Face detection is the premise of face alignment and directly determines the performance of subsequent key point positioning. The above face positioning technology is a common processing method in the prior art, and the embodiments of the present invention need not be described in detail here.

[0117] S24, inputting the facial video data after face positioning into a feature representation module to obtain feature information.

[0118] In a feasible implementation manner, the above step S24 may include the following steps S241-S244:

[0119] S241, extracting fine-grained variation features of the local region of interest of the facial video data after face localization, and establishing high-dimensional features.

[0120] In a feasible implementation, for feature extraction of the region of interest, the embodiment of the present invention refers to the region in the image that plays a major role in cognitive classification. The upper half of the face consisting of eyes and nose and the lower half of the face consisting of mouth and surroundings, these two regions almost cover all important features of the face, so the relationship between cognitive state and local facial regions is focused on through this region.

[0121] In a feasible implementation, the feature extraction of the region of interest is divided into two parts. First, the original features of the face need to be obtained, including geometric features, appearance features and sequence features. Secondly, the original features need to be reduced in dimension to eliminate redundant information, excessive dimensions and other problems to improve the distinguishability. In order to obtain feature data that is more favorable for classification, the features need to be decomposed to remove interference factors. Considering that the distinction of cognitive states is only related to the local area of ​​the face, this step will further focus on the relationship between cognitive states and local areas of the face. Extracting the appearance features of the local area of ​​interest of the face can further assist in the classification of cognitive states. At the level of feature extraction methods. The local binary pattern is an effective image texture feature description operator. It characterizes the spatial structure of the local texture in the image by using the size relationship between the grayscale values ​​of any point in the image and its neighborhood points. It has certain resistance to light and dark changes, noise interference, rotation and the like.

[0122] S242. Obtain the mutual influence between deep-level frame features of high-dimensional features and select efficient frame features.

[0123] In a feasible implementation, the features of each frame extracted are not simply stacked up to express the features of the entire video, but a larger weight is given to the efficient frames. The change of facial movements is a gradual process, in which obvious facial movements often occur when there is stimulation. The moment when the prompt sound is triggered in the game is selected, and the frames before and after the current moment are selected to give a larger weight to improve the contribution to the final facial video features.

[0124] S243. Perform feature weighted fusion based on efficient frame features and human-computer interaction data.

[0125] In a feasible implementation, using a single feature cannot solve the current problem well. Combining the above individual features to form a feature combination helps to express nonlinear relationships and is more conducive to classifying cognitive states. The above feature weighted fusion adopts a processing method commonly used in the prior art, and the embodiment of the present invention does not need to be described in detail here.

[0126] S244, inputting the weighted fused features into a support vector machine model for training to verify whether the feature information is representative.

[0127] Among them, check whether the feature information is representative. If so, inputting the feature information into the classifier can obtain better results, further indicating that the feature extraction method is the optimal method; if not, select other feature information to input into the classifier.

[0128] In a feasible implementation, in view of the complexity of cognitive state recognition during game interaction, facial features and eye movement trajectory features are used to combine interactive data such as game scores and game time. Among them, facial features reflect the facial apraxia of users when they are stimulated by the success or failure of the game, eye movement trajectory features reflect the user's concentration during the whole process, and game scores reflect the user's overall execution ability and memory ability. The introduction of kernel function in support vector machine can solve the problem of excessive dimensional expansion when low-dimensional space is mapped to high-dimensional space, and is considered to be one of the most reliable and accurate data classification methods. In an embodiment of the present invention, the combined features are input into a support vector machine model with a kernel function. On this basis, an extended study can be carried out to observe the upper half of the face composed of eyes and nose and the lower half of the face composed of mouth and surroundings, and the influence of these two areas on the final cognitive classification, so as to obtain the contribution of local facial areas of Alzheimer's patients to the classification results under external stimulation.

[0129] S25. Input the human-computer interaction data into a targeted training module, and perform targeted training on the game interaction tasks with scores below a preset threshold.

[0130] Among them, targeted training is the repeated execution of game interaction tasks.

[0131] In a feasible implementation, Figure 4 As shown, taking the scores of visual-spatial ability module, executive ability module, sustained attention module and memory module as indicators, and the eye movement trajectory as an auxiliary evaluation criterion, targeted training is arranged for the lower modules, which can effectively improve visual-spatial comprehension ability, executive ability, concentration and memory ability.

[0132] Specifically, when a user's score for a certain game task is lower than a preset threshold, the user needs to be arranged to perform repeated training on this game task, and the difficulty of the game task can be adaptively adjusted according to the user's game task score.

[0133] S26, inputting the processed facial video data into a database construction module to obtain a database.

[0134] In a feasible implementation, the user's age, gender, cognitive status, etc. are used as basic data of the database, the facial changes recorded in each module of the game task are used as the main video data, and the user's task score, click frequency and click offset in game interaction are used as auxiliary data. The above data are used to establish a facial video database of Alzheimer's disease in the current game task.

[0135] In a feasible implementation, the device collected in this experiment is a Huawei tablet M5, which is 10.1 inches in size, weighs about 480 grams, has a front camera of 800w, and a screen resolution of 1920*1200. From the beginning to the end of each task, the subject's face is automatically recorded using the camera on the Android tablet. In addition, in order to make the subject's facial expression more realistic, the camera interface during the task is hidden. Finally, the name, age, score of each level, and self-evaluation of each subject are formed into a text document, and the text document and video data are saved in the local memory.

[0136] In the embodiment of the present invention, facial data of Alzheimer's disease is collected based on facial features, using Android games as stimulation, recording facial changes during the entire game interaction process, extracting and calculating facial features through algorithms, and integrating human-computer interaction features and eye movement features with facial features in the form of local attention and related attention, respectively. Therefore, this application is a data collection process with facial features, pupil trajectories, and human-computer interaction behaviors as the core. The database source for Alzheimer's disease recognition in this application is different from the general database. It abandons functional magnetic resonance imaging or EEG image classification, which are expensive and time-consuming, and are not suitable for large-scale detection and screening. Instead, it uses game interaction to obtain facial video data. This method has simple collection equipment and short detection time, and is widely used in the community.

[0137] This application collects facial video data of users in game interactions and analyzes the collected data. Game tasks are designed based on visual-spatial comprehension ability, execution, sustained attention, memory, etc. The method described in this application can comprehensively obtain the state of changes in user facial features in game interactions. It provides a dimensional reference for medical testing, helps future research and clinical applications, and provides new means and new ways to break through the establishment of a database of Alzheimer's disease population characteristics.

[0138] like Figure 5 As shown, an embodiment of the present invention provides an Alzheimer's disease facial data collection device 500 based on game interaction, and the device 500 is applied to implement an Alzheimer's disease facial data collection method based on game interaction. The device 500 includes:

[0139] The acquisition module 510 is used to acquire the human-computer interaction data and facial video data generated by the Alzheimer's disease user when completing the game interaction task, wherein the facial video data includes facial feature data and eye movement trajectory data.

[0140] The input module 520 is used to input the human-computer interaction data and the facial video data into the constructed data acquisition model.

[0141] The output module 530 is used to obtain a database based on the human-computer interaction data, facial video data and data acquisition model of the Alzheimer's disease user; wherein the database includes the human-computer interaction data and facial video data.

[0142] Optionally, the game interaction tasks include visual-spatial comprehension tasks, executive ability assessment tasks, attention assessment tasks, and memory ability assessment tasks.

[0143] The visual-spatial comprehension task includes: fruits fall one after another from the top of the smart mobile terminal display screen, and the Alzheimer's disease user catches the fruits by clicking in the specified upper and lower areas.

[0144] The executive ability assessment task includes: the Alzheimer's user slides the received fruit into the fruit basket according to the designated route.

[0145] The attention judgment task includes: the user with Alzheimer's disease continues to pay attention to the fruits in the fruit basket. If a fruit ripens and changes color, the user with Alzheimer's disease clicks on the ripe fruit within a specified time.

[0146] The memory ability assessment task includes: the smart mobile terminal display screen displays a fruit basket containing different fruits in sequence, and the Alzheimer's user memorizes the order in which the fruit basket appears within the reserved time. After the reserved time ends, the Alzheimer's user clicks on the order in which the fruit basket appears.

[0147] The human-computer interaction data include: visual-spatial comprehension task scores, executive ability assessment task scores, attention assessment task scores, and memory ability assessment task scores.

[0148] Optionally, the output module 530 is further configured to:

[0149] S31, inputting the facial video data into a data processing module to obtain the facial video data after data processing.

[0150] S32, inputting the human-computer interaction data and the facial video data into a targeted training module, and performing targeted training on the game interaction tasks whose scores are lower than a preset threshold; wherein the targeted training is repeated execution of the game interaction tasks.

[0151] S33, inputting the processed facial video data into a database construction module to obtain a database.

[0152] Optionally, the output module 530 is further configured to:

[0153] S311, inputting the facial video data into a data preprocessing module to obtain preprocessed facial video data; wherein the preprocessed facial video data is the facial video data after video frame cutting, image geometry change and data augmentation.

[0154] S312: Input the pre-processed facial video data into a face detection module to obtain facial video data after face positioning.

[0155] S313, input the facial video data after face positioning into the feature representation module to obtain feature information.

[0156] Optionally, the output module 530 is further configured to:

[0157] S3131. Extract fine-grained change features of the local region of interest of the facial video data after face localization, and establish high-dimensional features.

[0158] S3132. Obtain the mutual influence between deep-level frame features of high-dimensional features and select efficient frame features.

[0159] S3133. Perform feature weighted fusion based on efficient frame features and human-computer interaction data.

[0160] S3134. Input the weighted fused features into a support vector machine model for training to verify whether the feature information is representative.

[0161] In the embodiment of the present invention, facial data of Alzheimer's disease is collected based on facial features, using Android games as stimulation, recording facial changes during the entire game interaction process, extracting and calculating facial features through algorithms, and integrating human-computer interaction features and eye movement features with facial features in the form of local attention and related attention, respectively. Therefore, this application is a data collection process with facial features, pupil trajectories, and human-computer interaction behaviors as the core. The database source for Alzheimer's disease recognition in this application is different from the general database. It abandons functional magnetic resonance imaging or EEG image classification, which are expensive and time-consuming, and are not suitable for large-scale detection and screening. Instead, it uses game interaction to obtain facial video data. This method has simple collection equipment and short detection time, and is widely used in the community.

[0162] This application collects facial video data of users in game interactions and analyzes the collected data. Game tasks are designed based on visual-spatial comprehension ability, execution, sustained attention, memory, etc. The method described in this application can comprehensively obtain the state of changes in user facial features in game interactions. It provides a dimensional reference for medical testing, helps future research and clinical applications, and provides new means and new ways to break through the establishment of a database of Alzheimer's disease population characteristics.

[0163] Figure 6 6 is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present invention. The electronic device 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 601 and one or more memories 602, wherein the memory 602 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 601 to implement the following Alzheimer's disease facial data collection method based on game interaction:

[0164] S1. Acquire human-computer interaction data and facial video data generated by an Alzheimer's disease user when completing a game interaction task, wherein the facial video data includes facial feature data and eye movement trajectory data;

[0165] S2, inputting the human-computer interaction data and facial video data into the constructed data acquisition model;

[0166] S3. Based on the human-computer interaction data, facial video data and data acquisition model of Alzheimer's disease users, a database is obtained; wherein the database includes the human-computer interaction data and facial video data.

[0167] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to complete the above-mentioned Alzheimer's disease facial data collection method based on game interaction. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0168] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for collecting facial data of Alzheimer's disease based on game interaction, characterized in that: The method comprises: S1. Obtaining human-computer interaction data and facial video data generated by users with Alzheimer's disease when completing game interaction tasks, where the facial video data includes facial feature data and eye movement trajectory data; S2, input the generated data into the constructed data acquisition model; S3, obtaining a database based on the generated data and the data acquisition model; The game interaction tasks include visual-spatial comprehension tasks, executive ability assessment tasks, attention assessment tasks, and memory ability assessment tasks; The visual-spatial comprehension task includes fruits falling from the top of the smart mobile terminal display screen one after another, and the user catches the fruits by clicking in the specified upper and lower areas; The execution ability assessment task includes the user sliding the received fruit into the fruit basket according to the specified route; The attention judgment task involves the user continuously paying attention to the fruits in the fruit basket. If a fruit ripens and changes color, the user clicks on the ripe fruit within a specified time. The memory ability assessment task includes the display screen of the smart mobile terminal sequentially displaying fruit baskets containing different fruits, and the user memorizes the order in which the fruit baskets appear within a reserved time. After the reserved time is over, the user clicks on the order in which the fruit baskets appear; The human-computer interaction data include visual-spatial comprehension task scores, executive ability assessment task scores, attention assessment task scores, and memory ability assessment task scores; The data collection model includes a data processing module, a targeted training module, and a database construction module; S3 includes: S31, inputting the facial video data into a data processing module to obtain the facial video data after data processing; S32, inputting the human-computer interaction data into a targeted training module, and performing targeted training on the game interaction tasks whose scores are lower than a preset threshold; S33, inputting the processed facial video data into a database construction module to obtain a database.

2. The method according to claim 1, characterized in that The data processing module includes a video data preprocessing module, a face detection module and a feature representation module; The data processing module in S31 processes the facial video data to obtain the facial video data after data processing, including: S311, inputting the facial video data into the data preprocessing module to obtain preprocessed facial video data; wherein the preprocessed facial video data is the facial video data after video frame capture, image geometry change and data augmentation are performed on the facial video data; S312, inputting the pre-processed facial video data into the face detection module to obtain facial video data after face positioning; S313, inputting the facial video data after face positioning into the feature representation module to obtain feature information.

3. The method according to claim 2, characterized in that The facial video data after face positioning in S313 is input into the feature representation module to obtain feature information including: S3131, extracting fine-grained change features of the local region of interest of the facial video data after face positioning, and establishing high-dimensional features; S3132, obtaining the mutual influence between the deep-level frame features of the high-dimensional features, and selecting efficient frame features; S3133, performing feature weighted fusion according to the efficient frame features and human-computer interaction data; S3134. Input the weighted fused features into a support vector machine model for training to verify whether the feature information is representative.

4. A facial data collection device for Alzheimer's disease based on game interaction, characterized in that: The device comprises: An acquisition module is used to acquire human-computer interaction data and facial video data generated by Alzheimer's disease users when completing game interaction tasks, and the facial video data includes facial feature data and eye movement trajectory data; An input module, used to input the generated data into the constructed data acquisition model; The output module is used to obtain a database based on the generated data and the data acquisition model; the game interaction tasks include visual-spatial comprehension tasks, executive ability assessment tasks, attention assessment tasks, and memory ability assessment tasks; The visual-spatial comprehension task includes fruits falling from the top of the smart mobile terminal display screen one after another, and the user catches the fruits by clicking in the specified upper and lower areas; The execution ability assessment task includes the user sliding the received fruit into the fruit basket according to the specified route; The attention judgment task involves the user continuously paying attention to the fruits in the fruit basket. If a fruit ripens and changes color, the user clicks on the ripe fruit within a specified time. The memory ability assessment task includes the display screen of the smart mobile terminal sequentially displaying fruit baskets containing different fruits, and the user memorizes the order in which the fruit baskets appear within a reserved time. After the reserved time is over, the user clicks on the order in which the fruit baskets appear; The human-computer interaction data include visual-spatial comprehension task scores, executive ability assessment task scores, attention assessment task scores, and memory ability assessment task scores; The data collection model includes a data processing module, a targeted training module, and a database construction module; Output modules for: S31, inputting the facial video data into a data processing module to obtain the facial video data after data processing; S32, inputting the human-computer interaction data and the facial video data into a targeted training module, and performing targeted training on the game interaction tasks whose scores are lower than a preset threshold; S33, inputting the processed facial video data into a database construction module to obtain a database.

5. The device according to claim 4, characterized in that The output module is further used for: S311, inputting the facial video data into a data preprocessing module to obtain preprocessed facial video data; wherein the preprocessed facial video data is the facial video data after video frame capture, image geometry change and data augmentation are performed on the facial video data; S312, inputting the pre-processed facial video data into a face detection module to obtain facial video data after face positioning; S313, inputting the facial video data after face positioning into the feature representation module to obtain feature information.

6. The device according to claim 5, characterized in that The output module is further used for: S3131, extracting fine-grained change features of the local region of interest of the facial video data after face positioning, and establishing high-dimensional features; S3132, obtaining the mutual influence between the deep-level frame features of the high-dimensional features, and selecting efficient frame features; S3133, performing feature weighted fusion according to the efficient frame features and human-computer interaction data; S3134. Input the weighted fused features into a support vector machine model for training to verify whether the feature information is representative.

Images