Method and device for determining dementia level of user

Abstract

This method for determining the level of dementia of a user involves: outputting first content via a user terminal; acquiring, via the user terminal, first reaction information about a reaction of the user to the first content; outputting second content via the user terminal; acquiring, via the user terminal, second reaction information about a reaction of the user to the second content; and determining the level of dementia of the user by inputting the first reaction information and the second reaction information to a dementia level classification model based on an artificial neural network in order to determine the level of dementia.

Description

Method and device for determining the degree of dementia in a user

[0001] The present disclosure relates to a technology for determining the degree of dementia of a user, and more particularly, to a method and device for instructing a user to perform a specific action and determining the degree of dementia of the user based on the user's reaction to the instructed action.

[0002] Dementia, one of the most serious illnesses affecting older adults, has seen a rapid increase over the past decade, fueling a surge in social and economic costs. Furthermore, it hinders patients' independence, and causes significant distress not only to the individual but also to their caregivers, with instances of disappearance and suicide. Early diagnosis and appropriate treatment can prevent or delay further cognitive decline in dementia. However, existing early diagnosis methods for this condition have inherent challenges. Because they typically require visits to specialized medical institutions like hospitals, many patients who present with worsening forgetfulness often already have advanced stages of Mild Cognitive Impairment (MCI) or Alzheimer's Disease (AD). Diagnostic neurocognitive tests (e.g., SNSB-II, CERAD-K) require experienced and skilled medical professionals to ensure high reliability. Diagnostic tests such as magnetic resonance imaging (MRI), single-photon emission computed tomography (SPECT), positron emission tomography (PET), and cerebrospinal fluid (CSF) tests are not only expensive but also inconvenient for patients.

[0003] One embodiment may provide a method and device for determining the degree of dementia of a user.

[0004] One embodiment may provide a method and device for determining the degree of dementia of a user based on a picture of the user.

[0005] According to one embodiment, a method for determining a degree of dementia of a user, performed by an electronic device, comprises: an operation of outputting first content through a user terminal, wherein the first content instructs the user to perform a first task through a first touch action on the user terminal; an operation of obtaining first reaction information of the user with respect to the first content through the user terminal, wherein the first reaction information includes at least one of first picture information for a first picture created by the user as the first task, first time series data obtained with respect to the first touch action, and first structured data associated with the first picture; an operation of outputting second content through the user terminal, wherein the second content instructs the user to perform a second task through a second touch action on the user terminal; The method may include: obtaining second reaction information of the user for the second content through the user terminal; obtaining second reaction information of the user for the second content through the user terminal; wherein the second reaction information includes at least one of second picture information for a second picture created by the user as the second task, second time series data obtained for the second touch action, and second structured data associated with the second picture; and determining the degree of dementia of the user by inputting the first reaction information and the second reaction information into a dementia degree classification model based on an artificial neural network to determine the degree of dementia.

[0006] The first content and the second content may include instructions in the form of voice or text that direct the user to take an action.

[0007] The first time series data may include at least one of the time that the digital pen used by the user touches the floor in a pen-down state while the user performs the first task of the first content, the time that the digital pen moves from one stroke to another in a pen-up state, or the distance that the digital pen moves from one stroke to another.

[0008] The first structured data includes a performance score for the first task, and the method for determining the degree of dementia further includes an operation of calculating the performance score for the first task based on the first picture information, and the operation of calculating the performance score may include an operation of dividing the first picture into a plurality of regions based on the first picture information; an operation of normalizing a first target symbol included in a first region among the plurality of regions; an operation of determining a first performance score for the first region based on the normalized first target symbol and a first reference symbol preset for the first region; and an operation of determining the performance score for the first task based on the first performance score.

[0009] The operation of determining a first performance score for the first region based on the normalized first target symbol and the first reference symbol preset for the first region may include an operation of determining the first performance score by inputting the normalized first target symbol into a pre-updated convolutional neural network (CNN).

[0010] The first structured data includes parameter information obtained by a digital pen used by the user while the user performs the first task of the first content, and the parameter information may include at least one of a first movement speed of the digital pen during a first time period when the digital pen is in contact with a touch display of the user terminal, a pen pressure during the first time period, the number of lines drawn during a preset time period, or a second time period when no drawing is performed.

[0011] The operation of determining the degree of dementia of the user may include an operation of generating a first feature by inputting at least one of the first time series data and the second time series data into a first artificial neural network that has been updated in advance; an operation of generating a second feature by inputting at least one of the first structured data and the second structured data into a second artificial neural network that has been updated in advance; an operation of generating a concatenated feature by inputting the first feature and the second feature into a concatenated layer based on an artificial neural network; and an operation of determining the degree of dementia of the user by inputting the concatenated feature into a fully connected layer based on an artificial neural network.

[0012] The method for determining the degree of dementia may further include: an operation of outputting third content through the user terminal, wherein the third content instructs the user to perform a third task through a third touch action on the user terminal; and an operation of obtaining third reaction information of the user for the third content through the user terminal, wherein the third reaction information includes at least one of third picture information for a third picture created by the user as the first task, third time-series data obtained for a third touch action, and third structured data related to the third picture.

[0013] The above-mentioned dementia level can be any of normal, subjective cognitive impairment (SCI), mild cognitive impairment (MCI), and Alzheimer's disease (AD).

[0014] The determined level of dementia can be output through the user terminal.

[0015] According to one embodiment, a computer-readable recording medium can store a program for performing the method for determining the degree of dementia.

[0016] According to one embodiment, an electronic device for determining a degree of dementia of a user comprises: a memory having a program for determining a degree of dementia of the user recorded therein; and a processor for executing the program, wherein the program comprises: an operation for outputting first content through a user terminal, wherein the first content instructs the user to perform a first task through a first touch action on the user terminal; an operation for obtaining first reaction information of the user with respect to the first content through the user terminal, wherein the first reaction information includes at least one of first picture information for a first picture created by the user as the first task, first time series data obtained with respect to the first touch action, and first structured data associated with the first picture; an operation for outputting second content through the user terminal, wherein the second content instructs the user to perform a second task through a second touch action on the user terminal; An operation of obtaining second reaction information of the user for the second content through the user terminal - An operation of obtaining second reaction information of the user for the second content through the user terminal - The second reaction information includes at least one of second picture information for a second picture created by the user as the second task, second time series data obtained for the second touch action, and second structured data associated with the second picture -; and an operation of determining the degree of dementia of the user by inputting the first reaction information and the second reaction information into a dementia degree classification model based on an artificial neural network to determine the degree of dementia.

[0017] A method and device for determining the degree of dementia of a user may be provided.

[0018] A device and method for determining the degree of dementia of a user based on a picture of the user may be provided.

[0019] Figure 1 is a schematic diagram of a system for determining the degree of dementia of a user, according to an example.

[0020] Figure 2 illustrates images output to a user terminal to determine the user's dementia level according to an example.

[0021] FIG. 3 is a schematic diagram of an electronic device for determining the degree of dementia of a user, according to one embodiment.

[0022] FIG. 4 is a flowchart of a method for determining a user's dementia level according to one embodiment.

[0023] FIG. 5 is a flowchart of a method for determining a user's dementia level according to one embodiment.

[0024] FIGS. 6, 7a, 7b, and 8 illustrate pre-fabricated content according to various examples.

[0025] FIG. 9 is a flowchart of a method for calculating a performance score of structured data according to one embodiment.

[0026] FIG. 10 is a flowchart of a method for determining the degree of dementia using a dementia degree classification model according to one embodiment.

[0027] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Therefore, the actual implementation is not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or alternatives within the technical concepts described in the embodiments.

[0028] Although terms such as "first" or "second" may be used to describe various components, these terms should be interpreted solely to distinguish one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0029] When it is said that a component is "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0030] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "has" should be understood to indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0031] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0032] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0033]

[0034] Figure 1 is a schematic diagram of a system for determining the degree of dementia of a user, according to an example.

[0035] According to one embodiment, a system for determining a user's dementia level may include an electronic device (110) for determining the user's dementia level, a user terminal (120) for outputting content, and a monitoring terminal (130) of a medical institution. For example, the electronic device (110) may be a server.

[0036] The electronic device (110) can provide pre-created content to the user terminal (120) to determine the user's dementia level. The content may be content for obtaining a picture (or drawing) from the user. The content may instruct the user to perform a task through a touch action. The user may draw a picture using the user terminal (120) as a task. The content output through the user terminal (120) to obtain a picture from the user is described in detail below with reference to FIGS. 6 to 8.

[0037] For example, the user terminal (120) may be a mobile terminal such as a tablet PC (personal computer) or a smartphone with a touch display. As another example, the user terminal (120) may be an electronic device connected to a tablet as an input tool. The user may draw on the user terminal (120) using a tool such as a digital pen or by directly touching the device with their hand.

[0038] The user terminal (120) can be connected to the electronic device (110) offline or online and communicate with each other. The electronic device (110) provides content to the user terminal (120), and the user terminal (120) outputs the content to the user through a display. For example, the user terminal (120) can receive a picture from the user through a touch display or tablet. The user terminal (120) can output a picture drawn by the user through the display (or touch display) of the user terminal (120).

[0039] When a user draws a picture using a digital pen, the user terminal (120) can receive pen information from the digital pen. The user terminal (120) can transmit the picture information and pen information for the obtained picture to the electronic device (110).

[0040] If the user terminal (120) is a mobile terminal, the user is not restricted by time and place and can measure the degree of dementia at a low cost.

[0041] Below, a method for determining the degree of dementia of a user is described in detail with reference to FIGS. 2 to 10.

[0042] Figure 2 illustrates images output to a user terminal to determine the user's dementia level according to an example.

[0043] The images below (210 to 240) may be images of applications for determining the degree of dementia. For example, a user of an electronic device (110) can create and distribute an application, and the user can run the application via a user terminal (120).

[0044] The first image (210) is the start screen of the application.

[0045] The second image (220) shows the functions supported by the application.

[0046] The third image (230) is an example of content provided to the user. One or more contents may be provided to the user.

[0047] The fourth image (240) indicates the determined level of dementia of the user. For example, the user's dementia level may be determined as normal, subjective cognitive impairment (SCI), mild cognitive impairment (MCI), or Alzheimer's disease (AD). For example, a score or probability for the user's dementia level may be output. In addition to the level of attention given to individual diseases, a comprehensive assessment may also be output.

[0048] FIG. 3 is a schematic diagram of an electronic device for determining the degree of dementia of a user according to one embodiment.

[0049] The electronic device (300) includes a communication unit (310), a processor (320), and a memory (330). For example, the electronic device (300) may be the electronic device (110) described above with reference to FIG. 1.

[0050] The communication unit (310) is connected to the processor (320) and the memory (330) to transmit and receive data. The communication unit (310) can be connected to other external devices to transmit and receive data. Hereinafter, the expression "transmitting and receiving "A" may refer to transmitting and receiving "information or data representing A."

[0051] The communication unit (310) may be implemented as a circuitry within the electronic device (300). For example, the communication unit (310) may include an internal bus and an external bus. As another example, the communication unit (310) may be an element that connects the electronic device (300) to an external device. The communication unit (310) may be an interface. The communication unit (310) may receive data from an external device and transmit the data to the processor (320) and the memory (330).

[0052] The processor (320) processes data received by the communication unit (310) and data stored in the memory (330). A "processor" may be a data processing device implemented as hardware having a circuit with a physical structure for executing desired operations. For example, the desired operations may include code or instructions included in a program. For example, a data processing device implemented as hardware may include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), or a field programmable gate array (FPGA).

[0053] The processor (320) executes computer-readable code (e.g., software) stored in memory (e.g., memory (330)) and instructions generated by the processor (320).

[0054] The memory (330) stores data received by the communication unit (310) and data processed by the processor (320). For example, the memory (330) may store a program (or application, software). The stored program may be a set of syntaxes that are coded to determine the user's level of dementia and can be executed by the processor (320).

[0055] According to one aspect, the memory (330) may include one or more volatile memory, non-volatile memory, and random access memory (RAM), flash memory, a hard disk drive, and an optical disk drive.

[0056] The memory (330) stores a set of instructions (e.g., software) for operating the electronic device (300). The set of instructions for operating the electronic device (300) is executed by the processor (320).

[0057] Figure 4 is a flowchart of a method for determining the degree of dementia of a user according to one embodiment.

[0058] The following operations 410 to 450 can be performed by the electronic device (300) (hereinafter, electronic device) described above with reference to FIG. 3.

[0059] In operation 410, the electronic device may output first content via a user terminal (e.g., the user terminal (120) of FIG. 1 ). The first content may instruct the user to perform a first task via a first touch action on the user terminal. For example, the first content may include instructions expressed in voice or text that instruct the user to perform an action. The first content may be output to the user terminal, and the user may draw a first picture using the user terminal as a first task for the first content.

[0060] According to one embodiment, the first touch action may be obtained by the user drawing a picture by directly touching the touch display or tablet of the user terminal with his / her hand.

[0061] According to one embodiment, the first touch action may be obtained by the user drawing on the touch display or tablet of the user terminal using a digital pen.

[0062] The user terminal may receive a first drawing generated as a first task for the first content via a touch display or tablet. Additionally, the user terminal may receive pen information regarding the first drawing from a digital pen connected to the user terminal via a wired or wireless connection.

[0063] According to one embodiment, the pen information for the first drawing may include at least one of first time series data and first parameter information acquired by the digital pen while the user performs the first task.

[0064] The user terminal can generate first drawing information for the first drawing. The user terminal can transmit at least one of the first drawing information and pen information for the first drawing drawn by the user to the electronic device.

[0065] In operation 420, the electronic device may obtain first user reaction information regarding first content via the user terminal. The first reaction information may include at least one of first image information regarding a first image created by the user as a first task, first time series data obtained by a first touch action, and first structured data associated with the first image.

[0066] According to one embodiment, the first picture information for the first picture may include information such as information about the overall size of the first picture and pixel information.

[0067] According to one embodiment, the first time series data acquired by the first touch action for the first picture may include the feature elements of [Table 1] below.

[0068] Feature Element Contents time on surface the time the digital pen is in pen-down state and touching the floor time in air the time the digital pen is in pen-up state and moving from one stroke to another pen-up stroke length the distance from one stroke to another

[0069] In [Table 1], time on surface may be the time during which the digital pen is in contact with the touch display of the user terminal. Time in air may be the time during which the digital pen is not in contact with the touch display.

[0070] According to one embodiment, the first structured data associated with the first picture may include at least one of a performance score for the first task and first parameter information. The first structured data may include data in the form of a scalar or vector.

[0071] The method for determining the performance score for the first task is described in detail below with reference to Figure 9.

[0072] The first parameter information may include the characteristic elements of [Table 2] below.

[0073] Feature Element Content Velocity on surface Speed ​​of drawing Velocity in air Speed ​​of movement of digital pen Pressure Pressure pressure_velocity_relation Relationship between pressure and speed Strokes per minute Number of lines drawn per minute Time not painting Intermission Pressure variability Variation in pressure Horizontal inclination variability Variation in the parallel angle of the digital pen Vertical inclination variability Variation in the vertical angle of the digital pen Total time Sum of Time in air and Time on surface

[0074] In [Table 2], total time may be the sum of time on surface and time in air. Velocity on surface may be the first movement speed of the digital pen during the time on surface when the digital pen is in contact with the touch display of the user terminal. Velocity in air may be the second movement speed of the digital pen during the time in air. The second movement speed may be calculated based on time in air and pen-up stroke length. Pressure may be the pen pressure during the time on surface. Strokes per minute may be the number of lines drawn during a preset time (e.g., 1 minute). Time not painting may be the time when no drawing is done.

[0075] In operation 430, the electronic device may output second content via the user terminal. The second content may instruct the user to perform a second task, different from the first task of the aforementioned first content, through a second touch action on the user terminal. For example, the second content may include instructions expressed in voice or text that instruct the user to perform an action. The second content may be output to the user terminal, and the user may draw a second picture using the user terminal as a second task for the second content.

[0076] According to one embodiment, the second touch action may be obtained by the user drawing a picture by directly touching the touch display or tablet of the user terminal with his or her hand.

[0077] In one embodiment, the second touch action may be obtained by the user drawing on the touch display or tablet of the user terminal using a digital pen.

[0078] The user terminal can receive a second drawing generated as a second task for the second content via a touch display or tablet. Additionally, the user terminal can receive pen information regarding the second drawing from a digital pen connected to the user terminal via a wired or wireless connection.

[0079] According to one embodiment, the pen information for the second drawing may include at least one of second time series data and second parameter information acquired by the digital pen while the user performs the second task.

[0080] The user terminal can generate second drawing information for the second drawing. The user terminal can transmit at least one of the second drawing information and pen information for the second drawing drawn by the user to the electronic device.

[0081] In operation 440, the electronic device may obtain second reaction information of the user regarding the second content through the user terminal. The second reaction information may include at least one of first image information regarding the second image generated by the user as a second task, second time series data obtained by the second touch action, and second structured data associated with the second image.

[0082] In one embodiment, the second picture information for the second picture may include information such as information about the overall size of the second picture and pixel information.

[0083] According to one embodiment, the second time series data acquired by the second touch action for the second picture may include the feature elements of [Table 1].

[0084] According to one embodiment, the second structured data associated with the second drawing may include at least one of performance information for the second task and second parameter information. The second structured data may include data in the form of a scalar or vector.

[0085] Performance information for the second task is described in detail below with reference to FIGS. 7a and 7b. The second parameter information may include the characteristic elements of [Table 2].

[0086] In operation 450, the electronic device can determine the degree of dementia of the user by inputting the first reaction information and the second reaction information into a dementia degree classification model based on an artificial neural network to determine the degree of dementia.

[0087] A method for determining a user's dementia level by inputting first reaction information and second reaction information into a dementia level classification model based on an artificial neural network according to one embodiment is described in detail with reference to FIG. 10.

[0088] According to one embodiment, the degree of dementia may be any one of normal, subjective cognitive impairment (SCI), mild cognitive impairment (MCI), and Alzheimer's disease (AD).

[0089] In one embodiment, the dementia severity may include a score or probability for at least one of normal, SCI, MCI, and AD.

[0090] After operation 450 is performed, the electronic device can output the determined dementia level through the user terminal.

[0091] Figure 5 is a flowchart of a method for determining the degree of dementia of a user according to one embodiment.

[0092] According to one embodiment, the method for determining the degree of dementia described above with reference to FIG. 4 may further include operations 510 and 520 below. For example, operations 510 and 520 may be performed after operation 440 of FIG. 4.

[0093] The operations 510 and 520 below can be performed by the electronic device (300) (hereinafter, electronic device) described above with reference to FIG. 3.

[0094] In operation 510, the electronic device may output third content via a user terminal (e.g., the user terminal (120) of FIG. 1 ). The third content may instruct the user to perform a third task, different from the first task of the first content and the second task of the second content, via a third touch action on the user terminal. For example, the third content may include instructions in the form of voice or text that instruct the user to perform an action. The third content may be output to the user terminal, and the user may draw a third picture as a third task for the third content.

[0095] According to one embodiment, the third touch action may be obtained by the user drawing a picture by directly touching the touch display or tablet of the user terminal with his / her hand.

[0096] In one embodiment, the third touch action may be obtained by the user drawing on the touch display or tablet of the user terminal using a digital pen.

[0097] The user terminal may receive a third drawing generated as a third task for third content through a touch display or tablet, and additionally receive pen information for the third drawing from a digital pen connected to the user terminal by wire or wirelessly.

[0098] According to one embodiment, the pen information for the third drawing may include at least one of third time series data and third parameter information acquired by the digital pen while the user performs the third task.

[0099] The user terminal can generate third picture information for the third picture. The user terminal can transmit at least one of the third picture information and pen information for the third picture drawn by the user to the electronic device.

[0100] In operation 520, the electronic device may obtain third reaction information of the user regarding third content via the user terminal. The third reaction information may include at least one of third image information regarding a third image created by the user as a third task, third time series data obtained by a third touch action, and third structured data associated with the third image.

[0101] According to one embodiment, the third picture information for the third picture may include information such as information about the overall size of the third picture and pixel information.

[0102] According to one embodiment, the third time series data acquired by the third touch action for the third picture may include the feature elements of [Table 1].

[0103] According to one embodiment, the third structured data associated with the third drawing may include at least one of performance information for the third task and third parameter information. The third structured data may include data in the form of a scalar or vector.

[0104] The performance information for the third task is described in detail below with reference to Figure 8. The third parameter information may include the characteristic elements of [Table 2].

[0105] After operation 520 is performed, the electronic device may perform operation 450 of FIG. 4 to determine the user's dementia level. In operation 450, the electronic device may determine the user's dementia level by inputting the first reaction information, the second reaction information, and the third reaction information into a dementia level classification model based on an artificial neural network to determine the dementia level.

[0106] A method for determining a user's dementia level by inputting first reaction information, second reaction information, and third reaction information into a dementia level classification model based on an artificial neural network according to one embodiment is described in detail with reference to FIG. 10.

[0107] After operation 450 is performed, the electronic device can output the determined dementia level through the user terminal.

[0108] Figures 6, 7a, 7b, and 8 illustrate pre-fabricated content according to various examples.

[0109] The content may be output on a display (610, 710, 720, 810) (e.g., a touch display) of a user terminal (e.g., the user terminal (120) of FIG. 1). The content may convey instructions to the user regarding the content. The content may instruct the user to perform a task through a touch action. For example, the instructions may be output via text (611, 711, 721, 811). As another example, the instructions may be output via sound.

[0110] FIG. 6 illustrates first content provided to a user according to one embodiment.

[0111] In one embodiment, the first content may be content that prompts the user to perform a task of looking at a given symbol table and drawing a symbol (or symbols) corresponding to a specific number.

[0112] In one embodiment, the first content may pre-output examples of preset reference symbol tables and pictures. For example, the first content may pre-output a table divided into multiple areas in which the user can draw pictures of each symbol.

[0113] According to one embodiment, a user may use a digital pen to draw symbols corresponding to each given number on a display (610) or a tablet connected to a user terminal. For example, the user may draw a symbol corresponding to the number 7 in area (613). For example, after completing the drawings of symbols corresponding to all numbers, the user may touch a submit button (612) to terminate the provision of the first content. The user terminal may generate image information for the drawings drawn by the user. The user terminal may transmit the image information to an electronic device (e.g., electronic device (300)).

[0114] Drawings drawn by each user may have various characteristics, but the higher the level of dementia, the lower the performance score for correctly drawn symbols may appear.

[0115] FIGS. 7A and 7B illustrate second content provided to a user, respectively, according to one embodiment.

[0116] In one embodiment, the second content may be content that prompts the user to perform a task of viewing numbers and / or letters and drawing lines connecting them in order.

[0117] According to one embodiment, the second content may pre-output numbers and / or letters with preset start and end points.

[0118] In one embodiment, a user may use a digital pen to draw lines connecting numbers and / or letters in the order indicated by text (711, 721) on a display (710, 720) or a tablet connected to a user terminal. For example, the user may draw a picture (713) connecting numbers in order or a picture (723) connecting numbers and letters in alternating order. Depending on the task of the second content, the user may draw a picture connecting numbers in ascending or descending order. As another example, the user may draw a picture connecting numbers of different colors in alternating order among duplicate numbers of different colors.

[0119] In one embodiment, if the user connects numbers and / or letters in the wrong order, the second content may provide feedback, such as making an alarm sound.

[0120] According to one embodiment, the user can terminate the provision of the second content by touching the submit button (712, 722) after completing the drawing by drawing lines connecting all numbers and / or letters. The user terminal can generate drawing information for the drawing (713, 723) drawn by the user.

[0121] In one embodiment, if the user fails to complete the task within a set time, the provision of the second content may be forcibly terminated upon expiration of the set time. The user terminal may generate image information for an unfinished drawing drawn by the user. The user terminal may transmit the image information to an electronic device.

[0122] According to one embodiment, the user terminal may generate performance information of the second content. The performance information of the second content may include scalar or vector data regarding lines in the drawing. The performance information of the second content may include at least one of information regarding whether the user has completed all tasks, the total time taken to perform the task, and the level (or point) to which the user has completed the task. The user terminal may transmit the performance information of the second content to an electronic device.

[0123] The drawings made by each user may have various characteristics, but the higher the level of dementia, the longer it may take for the user to perform the task or the lower the level of performance may appear.

[0124] FIG. 8 illustrates third content provided to a user according to one embodiment.

[0125] In one embodiment, the third content may be content that prompts the user to perform a task of drawing a clock with hands pointing to a specific time (e.g., 11:10).

[0126] In one embodiment, the third content may pre-output the outer shape of the watch (e.g., circular).

[0127] In one embodiment, a user may use a digital pen to draw a picture (813) of a clock with a needle pointing to a specific time on a display (810) or a tablet connected to a user terminal. For example, after completing the drawing, the user may terminate the provision of third content by touching a submit button (812). The user terminal may generate drawing information for the drawing (813) drawn by the user. The user terminal may transmit the drawing information to an electronic device.

[0128] According to one embodiment, an electronic device may determine performance information for a drawing (hereinafter, "third drawing") created by a user as a task of third content. The electronic device may normalize the third drawing based on third drawing information about the third drawing. The electronic device may set the area of ​​the entire area of ​​the third drawing where the actual drawing is drawn as a region of interest (ROI). The electronic device may normalize the drawing by adjusting the ROI to a normalized size.

[0129] In one embodiment, normalization information, which is normalized information, may be further generated. The size and / or ratio of the adjusted ROI may be generated as normalization information.

[0130] In one embodiment, the electronic device may input a normalized image into a pre-updated neural network. For example, the pre-updated neural network may be a neural network based on a convolutional neural network (CNN) and / or a deep neural network (DNN). In one embodiment, normalization information may be further input into the pre-updated neural network.

[0131] According to one embodiment, an electronic device may calculate a balance score for a normalized image as performance information for a third task. For example, the calculated balance score may be a value within a preset range (e.g., 0 to 5). The electronic device may calculate the balance score by separating each element of the image and determining the degree of balance between the separated elements. Segmentation of the image may be performed first by determining multiple elements. For example, a first element representing the arrangement of numbers indicating the times of a clock and a second element representing the arrangement of hands may be determined, and a balance score may be calculated based on the balance between the first and second elements. For example, the balance score may increase as the arranged times become more symmetrical. As another example, the balance score may increase as the sizes of the first and second elements become more appropriate.

[0132] The drawings made by each user can have various characteristics, but as the level of dementia increases, the shape of the clock and the time it represents may appear unclear.

[0133] Figure 9 is a flowchart of a method for calculating a performance score of structured data according to one embodiment.

[0134] According to one embodiment, the method for determining the degree of dementia described above with reference to FIG. 4 or FIG. 5 may further include operation 900 below. Operation 900 may be performed after operation 420 of FIG. 4.

[0135] The operation 900 below can be performed by the electronic device (300) (hereinafter, electronic device) described above with reference to FIG. 3.

[0136] In one embodiment, the first structured data associated with the first picture may include a performance score for the first task. In operation 900, the electronic device may calculate a performance score for the first task based on the first picture information.

[0137] According to one embodiment, operation 900 may include operations 910 to 940 below.

[0138] In operation 910, the electronic device may divide the first image into a plurality of regions based on the first image information. The first image may represent only images of symbols drawn by the user, excluding preset reference symbol tables, example images, and tables for drawing symbols pre-output by the first content. For example, the first image may represent an image of a second layer among the first layer for elements pre-output by the first content and the second layer generated by the user's first touch input.

[0139] In one embodiment, the electronic device may segment the first image into multiple regions, each region containing a symbol corresponding to a number. For example, the electronic device may segment the first image into multiple regions corresponding to the segmented regions of a table for drawing pre-printed symbols. Accordingly, the multiple regions may be regions at fixed locations, rather than regions at variable locations where actual images are drawn within the first image.

[0140] In operation 920, the electronic device can normalize a first target symbol contained within a first region among the plurality of regions.

[0141] Hereinafter, for convenience of explanation, a case will be described where the first area is an area (613) in which a picture of a symbol corresponding to the number 7 of FIG. 6 is drawn. The first area may include a first target symbol corresponding to the number 7.

[0142] According to one aspect, for users with a high level of dementia, the first target symbol may partially extend beyond the first area or be included in an area other than the first area. The electronic device may perform the following processing on a portion of the first target symbol drawn within the first area. If the first target symbol significantly extends beyond the first area or is included in an area other than the first area, the electronic device may determine the first performance score for the first area to be 0.

[0143] According to one embodiment, the electronic device may set a region of interest (ROI) in which a picture of an actual first target symbol is drawn among the first regions. The electronic device may adjust the ROI to a preset size. The electronic device may normalize the first target symbol by normalizing pixel values ​​for the adjusted ROI. For example, the electronic device may normalize the first target symbol in the range [0, 1]. For example, the electronic device may generate a normalized picture of the first target symbol or generate normalization information including each pixel value of the normalized first target symbol.

[0144] In operation 930, the electronic device may determine a first performance score for the first region based on a normalized first target symbol and a first reference symbol preset for the first region. For example, as illustrated in the reference symbol table of FIG. 6, a triangle symbol may be preset as the first reference symbol for the first region.

[0145] In one embodiment, the electronic device may input a normalized first target symbol to a pre-updated neural network. For example, the pre-updated neural network may be a neural network based on a convolutional neural network (CNN). The electronic device may input the output data generated by inputting the normalized first target symbol to the CNN to a fully connected layer.

[0146] According to one embodiment, the electronic device may determine a first performance score as a preset score for a correct answer if the first target symbol corresponds to the first reference symbol (e.g., if the similarity is greater than a threshold level) based on a pre-updated neural network.

[0147] In one embodiment, the electronic device may determine the first performance score as 0 if the first target symbol does not correspond to the first reference symbol based on the pre-updated neural network.

[0148] In operation 940, the electronic device may determine a performance score for the first task based on the first performance score. The electronic device may determine performance scores for each of the multiple regions of the first picture and sum the performance scores for all regions to determine the performance score for the first task. For example, the electronic device may determine the number of regions among the multiple regions that are correctly answered as the performance score.

[0149] Figure 10 is a flowchart of a method for determining the degree of dementia using a dementia degree classification model according to one embodiment.

[0150] According to one embodiment, operation 450 described above with reference to FIG. 4 may further include operations 1010 to 1040 below.

[0151] The following operations 1010 to 1040 can be performed by the electronic device (300) (hereinafter, electronic device) described above with reference to FIG. 3.

[0152] According to one embodiment, an electronic device may determine the user's dementia level using a dementia level classification model. The dementia level classification model may include a first artificial neural network, a second artificial neural network, a concatenated layer, and a fully connected layer. The electronic device may update the dementia level classification model in advance to determine the user's dementia level.

[0153] According to one embodiment, an electronic device may obtain test reaction information of a test user regarding first content and second content. The electronic device may input the test reaction information, labeled with the test user's ground truth (GT) dementia level, into a dementia level classification model to determine the test dementia level of the test user. The electronic device may update the dementia level classification model based on the test dementia level and the GT dementia level.

[0154] In operation 1010, the electronic device can generate a first feature by inputting at least one of the first time series data and the second time series data into a pre-updated first artificial neural network. For example, the first artificial neural network can include a transformer or a long short-term memory layer (LSTM).

[0155] According to one embodiment, the electronic device may preprocess first time series data and / or second time series data and input them into a first artificial neural network. The electronic device may generate the preprocessed first time series data and / or second time series data by performing min-max scaling and differencing on the first time series data and / or second time series data. The electronic device may generate the first feature by inputting the preprocessed first time series data and / or second time series data into the first artificial neural network.

[0156] In operation 1020, the electronic device may generate a second feature by inputting at least one of the first structured data and the second structured data into a pre-updated second artificial neural network. For example, the second artificial neural network may include a fully connected layer (or dense layer).

[0157] According to one embodiment, the electronic device may preprocess first structured data and / or second structured data and input them into a second artificial neural network. The electronic device may normalize the first structured data and / or second structured data by performing min-max scaling or standard scaling, and filter the normalized first structured data and / or second structured data based on correlations to generate preprocessed first structured data and / or second structured data. The electronic device may remove data with low correlation through preprocessing or remove variables with multicollinearity by measuring variance inflation factors (VIF), and adjust the data so that outliers or kurtosis do not occur. The electronic device may generate second features by inputting the preprocessed first structured data and / or second structured data into the second artificial neural network.

[0158] In operation 1030, the electronic device can generate a combined feature by inputting the first feature and the second feature into a combined layer based on an artificial neural network.

[0159] According to one embodiment, the order of combining the first feature and the second feature to determine the degree of dementia may be the same as the order of combining the features generated by inputting the time series data and structured data included in the test reaction information into the first artificial neural network and the second artificial neural network, respectively, in the process of pre-updating the dementia degree classification model.

[0160] According to one embodiment, as described with reference to FIG. 5, an electronic device may obtain third user reaction information regarding third content. The electronic device may obtain test user reaction information regarding the first content, second content, and third content, and update a dementia severity classification model as described above. In this case, the dementia severity classification model may further include a third artificial neural network.

[0161] According to one embodiment, the electronic device can generate a third feature by inputting a third image (or third image information about the third image) generated by a user for third content into a pre-updated third artificial neural network. For example, the third artificial neural network may be the same as or different from the pre-updated neural network described with reference to FIG. 8. For example, the third artificial neural network may include a vision transformer. As described with reference to FIG. 8, a normalized image and / or normalization information about the third image may be input into the third artificial neural network.

[0162] The electronic device can generate a first feature by inputting at least one of the first time series data, the second time series data, and the third time series data into a pre-updated first artificial neural network, as described in operations 1010 to 1030, generate a second feature by inputting at least one of the first structured data, the second structured data, and the third structured data into a pre-updated second artificial neural network, and generate a combined feature by inputting the first feature, the second feature, and the third feature into a combined layer based on the artificial neural network.

[0163] According to one embodiment, the order of combining the first feature, the second feature, and the third feature to determine the degree of dementia may be the same as the order of combining the features generated by inputting time series data, structured data, and picture information included in the test reaction information into the first artificial neural network, the second artificial neural network, and the third artificial neural network, respectively, in the process of pre-updating the dementia degree classification model.

[0164] In the work 1040, the electronic device can determine the degree of dementia of the user by inputting the combined features into a fully connected layer based on an artificial neural network.

[0165] In one embodiment, the degree of dementia may be any of normal, SCI, MCI, and AD.

[0166] In one embodiment, the dementia severity may include a score or probability for at least one of normal, SCI, MCI, and AD.

[0167] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0168] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or, independently or collectively, command the processing device. The software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0169] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0170] The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0171] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the described embodiments. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0172] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. A method for determining the degree of dementia of a user, performed by an electronic device, An action of outputting a first content through a user terminal, wherein the first content instructs the user to perform a first task through a first touch action on the user terminal; An operation of obtaining first reaction information of the user for the first content through the user terminal, wherein the first reaction information includes at least one of first picture information for a first picture created by the user as the first task, first time series data obtained for the first touch action, and first structured data associated with the first picture; An action of outputting second content through the user terminal, wherein the second content instructs the user to perform a second task through a second touch action on the user terminal; An operation of obtaining second reaction information of the user for the second content through the user terminal - An operation of obtaining second reaction information of the user for the second content through the user terminal - The second reaction information includes at least one of second picture information for a second picture created by the user as the second task, second time series data obtained for the second touch action, and second structured data associated with the second picture -; and An operation of determining the degree of dementia of the user by inputting the first reaction information and the second reaction information into a dementia degree classification model based on an artificial neural network to determine the degree of dementia. Including, How to determine the degree of dementia.

2. In paragraph 1, The above first content and the above second content, Including instructions in the form of voice or text that direct the user to take action; How to determine the degree of dementia.

3. In paragraph 1, The above first time series data is, The digital pen used by the user while performing the first task of the first content includes at least one of the time that the digital pen is in contact with the floor in a pen-down state, the time that the digital pen moves from one stroke to another in a pen-up state, or the distance that the digital pen moves from one stroke to another. How to determine the degree of dementia.

4. In paragraph 1, The above first structured data includes a performance score for the first task, An operation of calculating the performance score for the first task based on the first picture information. Including more, The operation of calculating the above performance score is: An operation of dividing the first picture into a plurality of regions based on the first picture information; An operation of normalizing a first target symbol included in a first region among the above multiple regions; An operation of determining a first performance score for the first region based on the normalized first target symbol and a first reference symbol preset for the first region; and An operation of determining the performance score for the first task based on the first performance score. Including, How to determine the degree of dementia.

5. In paragraph 4, An operation of determining a first performance score for the first region based on the normalized first target symbol and the first reference symbol preset for the first region, An operation of determining the first performance score by inputting the normalized first target symbol into a pre-updated convolutional neural network (CNN). Including, How to determine the degree of dementia.

6. In paragraph 1, The first structured data includes first parameter information obtained by a digital pen used by the user while the user performs the first task of the first content, The first parameter information includes at least one of a first movement speed of the digital pen during a first time period when the digital pen touches the touch display of the user terminal, a pressure during the first time period, a number of lines drawn during a preset time period, or a second time period when no drawing is performed. How to determine the degree of dementia.

7. In paragraph 1, The actions that determine the degree of dementia of the above user are: An operation of generating a first feature by inputting at least one of the first time series data and the second time series data into a pre-updated first artificial neural network; An operation of generating a second feature by inputting at least one of the first structured data and the second structured data into a pre-updated second artificial neural network; An operation of generating a concatenated feature by inputting the first feature and the second feature into a concatenated layer based on an artificial neural network; and An operation for determining the degree of dementia of the user by inputting the above combination features into a fully connected layer based on an artificial neural network. Including, How to determine the degree of dementia.

8. In paragraph 1, An action of outputting third content through the user terminal, wherein the third content instructs the user to perform a third task through a third touch action on the user terminal; and An operation of obtaining third reaction information of the user for the third content through the user terminal, wherein the third reaction information includes at least one of third picture information for a third picture created by the user as the first task, third time series data obtained for a third touch action, and third structured data related to the third picture. Including more, How to determine the degree of dementia.

9. In paragraph 1, The above dementia level is one of normal, subjective cognitive impairment (SCI), mild cognitive impairment (MCI), and Alzheimer's disease (AD). How to determine the degree of dementia.

10. In paragraph 1, The level of dementia determined above is output through the user terminal. How to determine the degree of dementia.

11. A computer-readable recording medium storing a program for performing the method of paragraph 1.

12. An electronic device that determines the degree of dementia of a user, A memory containing a program that determines the degree of dementia of the user; and Processor executing the above program Including, The above program is, An action of outputting a first content through a user terminal, wherein the first content instructs the user to perform a first task through a first touch action on the user terminal; An operation of obtaining first reaction information of the user for the first content through the user terminal, wherein the first reaction information includes at least one of first picture information for a first picture created by the user as the first task, first time series data obtained for the first touch action, and first structured data associated with the first picture; An action of outputting second content through the user terminal, wherein the second content instructs the user to perform a second task through a second touch action on the user terminal; An operation of obtaining second reaction information of the user for the second content through the user terminal - An operation of obtaining second reaction information of the user for the second content through the user terminal - The second reaction information includes at least one of second picture information for a second picture created by the user as the second task, second time series data obtained for the second touch action, and second structured data associated with the second picture -; and An operation of determining the degree of dementia of the user by inputting the first reaction information and the second reaction information into a dementia degree classification model based on an artificial neural network to determine the degree of dementia. To perform, Electronic devices.

13. In paragraph 12, The above first content and the above second content, Including instructions in the form of voice or text that direct the user to take action; Electronic devices.

14. In paragraph 12, The above first time series data is, The digital pen used by the user while performing the first task of the first content includes at least one of the time that the digital pen is in contact with the floor in a pen-down state, the time that the digital pen moves from one stroke to another in a pen-up state, or the distance that the digital pen moves from one stroke to another. Electronic devices.

15. In paragraph 12, The actions that determine the degree of dementia of the above user are: An operation of generating a first feature by inputting at least one of the first time series data and the second time series data into a pre-updated first artificial neural network; An operation of generating a second feature by inputting at least one of the first structured data and the second structured data into a pre-updated second artificial neural network; An operation of generating a concatenated feature by inputting the first feature and the second feature into a concatenated layer based on an artificial neural network; and An operation for determining the degree of dementia of the user by inputting the above combination features into a fully connected layer based on an artificial neural network. Including, Electronic devices.

Images