Mental disorder analysis device

Abstract

To provide a mental disorder analysis device capable of analyzing whether a person has a mental disorder.SOLUTION: A mental disorder analysis device comprises: a storage unit that stores a mental disorder learning model that has learned a moving image for learning including a patient diagnosed with a mental disorder as teacher data; an acquisition unit that acquires an analysis target moving image including an analysis target person; and an analysis unit that applies the mental disorder learning model to the analysis target moving image to analyze whether the analysis target person has a mental disorder.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a mental disorder analysis device. [Background technology]

[0002] A technique for analyzing the emotions felt by others in response to a speaker's statement is known (see, for example, Patent Document 1). A technique for chronologically analyzing changes in a target's facial expression over a long period of time and estimating the emotions felt during that time is also known (see, for example, Patent Document 2). A technique for identifying the factor that most influenced the change in emotion is also known (see, for example, Patent Documents 3 to 5). A technique for comparing a target's usual facial expression with a current facial expression and issuing an alert if the facial expression is gloomy is also known (see, for example, Patent Document 6). A technique for comparing a target's normal (expressionless) facial expression with a current facial expression to determine the degree of the target's emotion is also known (see, for example, Patent Documents 7 to 9). A technique for analyzing the emotions of an organization and the atmosphere within a group that an individual feels is also known (see, for example, Patent Documents 10 and 11). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-58625 A [Patent Document 2] JP 2016-149063 A [Patent Document 3] JP 2020-86559 A [Patent Document 4] JP 2000-76421 A [Patent Document 5] JP 2017-201499 A [Patent Document 6] JP 2018-112831 A [Patent Document 7] JP 2011-154665 A [Patent Document 8] JP 2012-8949 A [Patent Document 9] JP 2013-300 A [Patent Document 10] JP 2011-186521 A [Patent Document 11] WO15 / 174426 publication Summary of the Invention [Problem to be solved by the invention]

[0004] All of the above technologies can analyze a person from multiple angles, but they cannot predict a specific condition. In particular, improvements are needed to determine whether a person is in a disease-related condition.

[0005] In particular, the present invention aims to make it possible to determine whether a subject is in a state related to a mental disorder. [Means for solving the problem]

[0006] According to the present invention A memory unit that stores a mental disorder learning model that has been trained using training videos including patients diagnosed with a mental disorder as training data; An acquisition unit that acquires a video to be analyzed that includes a person to be analyzed; an analysis unit that applies the mental disability learning model to the analysis target video to analyze whether the analysis target has the mental disability; A mental disorder analyzer is obtained. Effect of the Invention

[0007] According to the present disclosure, it is possible to reduce the burden of medical interviews when determining whether or not a mental disorder exists. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an overall system according to an embodiment of the present invention. [Diagram 2] FIG. 1 is an example of a functional block diagram according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a video analysis device according to an embodiment of the present invention. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of a mental disorder analysis device according to an embodiment of the present invention. [Diagram 5] FIG. 13 is a diagram showing the flow of a mental disorder device. [Figure 6] FIG. 13 is a diagram showing the flow of a mental disorder device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The contents of the embodiments of the present disclosure will be described below. The present disclosure has the following configuration. [Item 1] A memory unit that stores a mental disorder learning model that has been trained using training videos including patients diagnosed with a mental disorder as training data; An acquisition unit that acquires a video to be analyzed that includes a person to be analyzed; an analysis unit that applies the mental disability learning model to the analysis target video to analyze whether the analysis target has the mental disability; Psychological disorder analyzer. [Item 2] The mental disorder analysis apparatus according to claim 1, The learning video is a recording of both the first subject and the second subject undergoing a diagnosis process for the mental disorder by an expert, The first subject has been previously diagnosed with the mental disorder, and the second subject has not been previously diagnosed with the mental disorder. Psychological disorder analyzer. [Item 3] The mental disorder analysis apparatus according to claim 1, The video to be analyzed includes at least facial expressions and voices of the person to be analyzed. Psychological disorder analyzer. [Item 4] The mental disorder analysis apparatus according to claim 1, The video to be analyzed is a video about a conversation that the subject is having with another person. Psychological disorder analyzer. [Item 5] The mental disorder analysis apparatus according to claim 1, The video to be analyzed was acquired using an online session. Psychological disorder analyzer. [Item 6] The mental disorder analysis apparatus according to claim 1, The analysis unit includes information regarding whether the degree of the mental disorder of the analysis subject exceeds a predetermined threshold. Psychological disorder analyzer. [Item 7] The mental disorder analysis apparatus according to claim 1, A questioning unit that asks the subject of analysis a predetermined additional question; An answer acquisition unit for acquiring answers to the additional questions, The analysis unit analyzes the answer to analyze whether the subject has the specific mental disorder. Psychological disorder analyzer. [Item 8] A step of storing a mental disorder learning model, which is trained using training videos including patients diagnosed with a mental disorder as training data, in a storage unit of a computer; A step of acquiring a video to be analyzed that includes a person to be analyzed; and applying the mental disability learning model to the analysis target video to analyze whether the analysis target has the mental disability. Methods of analysis of mental disorders. [Item 9] A memory unit that stores a mental disorder learning model that has been trained using training videos including patients diagnosed with a mental disorder as training data; An acquisition unit that acquires a video to be analyzed that includes a person to be analyzed; an analysis unit that applies the mental disability learning model to the analysis target video to analyze whether the analysis target has the mental disability; Psychiatric Disorder Analysis System. [Item 10] Computer A memory unit that stores a mental disorder learning model that has been trained using training videos including patients diagnosed with a mental disorder as training data; an acquisition unit that acquires a video to be analyzed including a person to be analyzed; applying the mental disability learning model to the analysis target video to function as an analysis unit that analyzes whether the analysis target has the mental disability; Psychiatric Analysis Program.

[0010] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0011] The present invention relates to an analysis system that analyzes video recorded of a person being analyzed communicating, and analyzes whether or not the person is in a specific mental state (for example, but not limited to, a mental disorder).

[0012] This system generates a mental disability learning model that is trained using training videos including patients diagnosed with mental disorders as training data, and applies the mental disability learning model to videos including the subject of analysis to analyze whether the subject of analysis has a mental disorder or not. The generation of the mental disability learning model and the analysis of the subject of analysis are performed using a video analysis device described below.

[0013] The embodiment of the present system will be described below in the following order. 1.Video Analysis Device Basic functions of video analysis equipment Hardware configuration example How to get the video Analysis flow 2. Generate the model overview Model generation method Other ways to generate models 3. Analysis of the subjects Acquiring videos Applying analytical models Getting a Number Threshold setting and judgment Output format Examples of analysis

[0014] 1.Video Analysis Device <Basic functions> The video analysis device of the present embodiment is a system that analyzes and evaluates the unique emotions (feelings caused by one's own or other people's words and actions, such as pleasantness or unpleasantness or the degree of discomfort) of an analysis target person among a plurality of people in an environment where the video session (hereinafter, one-way and two-way sessions are both referred to as online sessions) is held by a plurality of people. An online session is, for example, an online conference, an online class, an online chat, etc., and terminals installed in a plurality of places are connected to a server via a communication network such as the Internet, and video images can be exchanged between the plurality of terminals through the server. The video images handled in the online session include facial images and voices of users using the terminals. In addition, the video images also include images such as materials shared and viewed by a plurality of users. It is possible to switch between facial images and document images on the screen of each terminal to display only one of them, or to divide the display area to display facial images and document images simultaneously. It is also possible to display an image of one of the plurality of people in full screen, or to display some or all of the images of the users divided into small screens. It is possible to specify one or more of the plurality of users who participate in the online session using terminals as the analysis target. For example, the leader, facilitator, or manager of the online session (hereinafter collectively referred to as the organizer) designates a user as the analysis target. The organizer of the online session may be, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a session aimed at coaching. The organizer of the online session is usually one of multiple users who participate in the online session, but may be a different person who does not participate in the online session. Note that the analysis target may not be designated and all participants may be the analysis target. It is also possible for the leader, facilitator, or manager of the online session (hereinafter collectively referred to as the organizer) to designate a user as the analysis target. The organizer of the online session may be, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a session aimed at coaching.The host of an online session is typically one of multiple users participating in the online session, but may be a different person who does not participate in the online session.

[0015] In the video analysis device according to the present embodiment, when a video session is established between a plurality of terminals, at least a video image acquired from the video session is displayed. The displayed video image is acquired by the terminal, and at least a face image included in the video image is identified for each predetermined frame unit. An evaluation value for the identified face image is then calculated. The evaluation value is shared as necessary. In particular, in the present embodiment, the acquired video image is stored in the terminal, analyzed and evaluated on the terminal, and the result is provided to the user of the terminal. Therefore, even if a video session includes personal information or confidential information, for example, the video itself can be analyzed and evaluated without providing the video itself to an external evaluation agency or the like. Furthermore, by providing only the evaluation result (evaluation value) to an external terminal as necessary, the result can be visualized, cross-analysis, etc. can be performed.

[0016] As shown in Figure 1, the video analysis device of this embodiment includes user terminals 10 and 20 having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display, and an output unit such as a speaker, a video session service terminal 30 that provides a two-way video session to the user terminals 10 and 20, and an evaluation terminal 40 that performs part of the evaluation of the video session.

[0017] <Hardware configuration example> Each of the functional blocks, functional units, and functional modules described below can be configured, for example, by hardware, DSP (Digital Signal Processor), or software provided in a computer. For example, when configured by software, the computer is actually configured with a CPU, RAM, ROM, etc., and is realized by the operation of a program stored in a recording medium such as a RAM, ROM, hard disk, or semiconductor memory. A series of processes by the system and terminal described in this specification may be realized by using any of software, hardware, and a combination of software and hardware. A computer program for realizing each function of the information sharing support device 10 according to this embodiment can be created and implemented in a PC or the like. In addition, a computer-readable recording medium in which such a computer program is stored can also be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, etc. In addition, the above computer program may be distributed, for example, via a network, without using a recording medium.

[0018] The evaluation terminal according to this embodiment acquires moving images from a video session service terminal, identifies at least facial images contained in the moving images for each predetermined frame, and calculates an evaluation value for the facial images (described in detail later).

[0019] <How to get the video> As shown in FIG. 2, the video session service provided by the video session service terminal (hereinafter, sometimes simply referred to as "this service") enables two-way communication with user terminals 10 and 20 using images and audio. This service is capable of displaying video images captured by the camera unit of the other user terminal on the display of the user terminal, and outputting audio captured by the microphone unit of the other user terminal from the speaker. This service is also configured to enable both or any one of the user terminals to record video images and audio (collectively referred to as "video images, etc.") in the memory unit of at least one of the user terminals. The recorded video information Vs (hereinafter referred to as "recorded information") is cached in the user terminal that started the recording, and is recorded only locally in one of the user terminals. If necessary, the user can view the recorded information himself or share it with others within the scope of use of this service.

[0020] <Analysis flow> Fig. 3 is a block diagram showing a configuration example according to this embodiment. As shown in Fig. 3, the video analysis device of this embodiment is realized as a functional configuration of a user terminal 10. That is, the user terminal 10 includes, as its functions, a video image acquisition unit 11, a biological reaction analysis unit 12, a peculiar determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16.

[0021] The video acquisition unit 11 acquires from each terminal video images obtained by photographing multiple people (multiple users) with a camera provided in each terminal during an online session. The video images acquired from each terminal may or may not be set to be displayed on the screen of each terminal. In other words, the video acquisition unit 11 acquires video images from each terminal, including video images being displayed and video images not being displayed on each terminal.

[0022] The biological reaction analysis unit 12 analyzes changes in biological reactions for each of the multiple people based on the moving images (whether or not they are being displayed on the screen) acquired by the moving image acquisition unit 11. In this embodiment, the biological reaction analysis unit 12 separates the moving images acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and audio, and analyzes changes in biological reactions from each of them.

[0023] For example, the biological reaction analysis unit 12 analyzes a change in a biological reaction related to at least one of facial expression, gaze, pulse rate, and facial movement by analyzing a facial image of the user using frame images separated from the video acquired by the video acquisition unit 11. In addition, the biological reaction analysis unit 12 analyzes a change in a biological reaction related to at least one of the user's remarks and voice quality by analyzing a sound separated from the video acquired by the video acquisition unit 11.

[0024] When a person's emotions change, this is reflected in changes in biological reactions such as facial expressions, gaze, pulse rate, facial movements, speech, and voice quality. In this embodiment, the change in the user's emotions is analyzed by analyzing the change in the user's biological reactions. In this embodiment, the emotion analyzed is, for example, the degree of comfort / discomfort. In this embodiment, the biological reaction analysis unit 12 quantifies the change in the biological reaction according to a predetermined standard, thereby calculating a biological reaction index value that reflects the content of the change in the biological reaction.

[0025] The analysis of changes in facial expression is performed, for example, as follows. That is, for each frame image, a facial region is identified from within the frame image, and the identified facial expressions are classified into a plurality of categories according to an image analysis model that has been trained by machine learning in advance. Then, based on the classification results, it is analyzed whether a positive or negative facial expression change has occurred between consecutive frame images, and to what extent the facial expression change has occurred, and a facial expression change index value according to the analysis results is output.

[0026] The analysis of the change in the line of sight is performed, for example, as follows. That is, for each frame image, the eye area is identified from within the frame image, and the direction of both eyes is analyzed to analyze where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, the shared material being displayed, or looking outside the screen. It may also be analyzed whether the movement of the line of sight is large or small, and whether the movement is frequent or infrequent. The change in the line of sight is also related to the concentration level of the user. The biological reaction analysis unit 12 outputs a line of sight change index value according to the analysis result of the change in the line of sight.

[0027] The analysis of changes in pulse rate is performed, for example, as follows. That is, for each frame image, a facial area is identified from within the frame image. Then, using a trained image analysis model that captures the numerical value of facial color information (G of RGB), changes in G color on the face surface are analyzed. The results are arranged along the time axis to form a waveform that represents changes in color information, and the pulse rate is identified from this waveform. When a person is nervous, their pulse rate increases, and when they feel calm, their pulse rate decreases. The biological response analysis unit 12 outputs a pulse rate change index value according to the analysis result of the changes in pulse rate.

[0028] The analysis of the change in facial movement is performed, for example, as follows. That is, for each frame image, the facial area is identified from within the frame image, and the facial direction is analyzed to analyze where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, the shared material being displayed, or looking outside the screen. It is also possible to analyze whether the facial movement is large or small, and whether the frequency of the movement is high or low. It is also possible to analyze the facial movement together with the eye movement. For example, it is also possible to analyze whether the user is looking straight at the speaker being displayed, looking up or down, or looking at an angle. The biological reaction analysis unit 12 outputs a facial direction change index value according to the analysis result of the change in the facial direction.

[0029] The analysis of the utterance contents is performed, for example, as follows. That is, the biological response analysis unit 12 converts the voice into a character string by performing a known voice recognition process on the voice for a specified time (for example, about 30 to 150 seconds), and removes words such as particles and articles that are unnecessary for expressing the conversation by performing a morphological analysis of the character string. Then, it vectorizes the remaining words, analyzes whether a positive or negative emotional change has occurred, and the extent of the emotional change, and outputs a utterance content index value according to the analysis result.

[0030] The voice quality is analyzed, for example, as follows. That is, the biological response analysis unit 12 identifies the acoustic features of the voice by performing a known voice analysis process on the voice for a specified time (for example, about 30 to 150 seconds). Then, based on the acoustic features, it analyzes whether a positive or negative voice quality change has occurred and to what extent the voice quality change has occurred, and outputs a voice quality change index value according to the analysis result.

[0031] The biological reaction analysis unit 12 calculates a biological reaction index value using at least one of the facial expression change index value, eye gaze change index value, pulse rate change index value, face direction change index value, statement content index value, and voice quality change index value calculated as described above. For example, the biological reaction index value is calculated by weighting the facial expression change index value, eye gaze change index value, pulse rate change index value, face direction change index value, statement content index value, and voice quality change index value.

[0032] The unique determination unit 13 determines whether or not the change in the biological reaction analyzed for the subject of analysis is unique compared to the change in the biological reaction analyzed for other people other than the subject of analysis. In this embodiment, the unique determination unit 13 determines whether or not the change in the biological reaction analyzed for the subject of analysis is unique compared to other people based on the biological reaction index value calculated for each of the multiple users by the biological reaction analysis unit 12.

[0033] For example, the unique judgment unit 13 calculates the variance of the biological reaction index values ​​calculated for each of multiple individuals by the biological reaction analysis unit 12, and by comparing the biological reaction index value calculated for the subject of analysis with the variance, judges whether or not the changes in the biological reactions analyzed for the subject of analysis are unique compared to others.

[0034] There are three possible cases where the changes in the analyzed biological reactions of the subject are unique compared to others. The first is when no particularly significant changes in biological reactions occur in others, but a relatively large change in biological reactions occurs in the subject. The second is when no particularly significant changes in biological reactions occur in the subject, but a relatively large change in biological reactions occurs in others. The third is when a relatively large change in biological reactions occurs in both the subject and others, but the content of the change differs between the subject and others.

[0035] The associated event identification unit 14 identifies an event occurring with respect to at least one of the subject to be analyzed, other people, and the environment when a change in the biological reaction determined to be unique by the unique determination unit 13 occurs. For example, the associated event identification unit 14 identifies the behavior of the subject to be analyzed himself / herself from a video when a unique change in the biological reaction occurs in the subject to be analyzed. The associated event identification unit 14 also identifies the behavior of other people from the video when a unique change in the biological reaction occurs in the subject to be analyzed. The associated event identification unit 14 also identifies the environment when a unique change in the biological reaction occurs in the subject to be analyzed from the video. The environment is, for example, a shared document being displayed on the screen, something that appears in the background of the subject to be analyzed, etc.

[0036] The clustering unit 15 analyzes the degree of correlation between a change in biological reaction determined to be unique by the unique determination unit 13 (for example, one or more combinations of eye contact, pulse rate, facial movement, speech content, and voice quality) and an event occurring when the unique change in biological reaction occurs (an event identified by the related event identification unit 14), and if the correlation is determined to be at a certain level or above, the clustering unit 15 clusters the person or event being analyzed based on the results of the correlation analysis.

[0037] For example, if a specific change in biological reaction corresponds to a negative emotional change, and the event occurring when the specific change in biological reaction occurs is also a negative event, a correlation of a certain level or higher is detected. The clustering unit 15 clusters the analysis subject or event into one of a plurality of pre-segmented classifications according to the content of the event, the degree of negativity, the magnitude of correlation, etc.

[0038] Similarly, if a specific change in biological reaction corresponds to a positive emotional change, and the event occurring when the specific change in biological reaction occurs is also a positive event, a correlation of a certain level or higher is detected. The clustering unit 15 clusters the analysis subject or event into one of a plurality of pre-segmented classifications according to the content of the event, the degree of positivity, the magnitude of correlation, etc.

[0039] The analysis result notification unit 16 notifies the person designating the analysis subject (the analysis subject or the organizer of the online session) of at least one of the changes in biological reactions determined to be specific by the unique determination unit 13, the events identified by the related event identification unit 14, and the classifications clustered by the clustering unit 15.

[0040] For example, the analysis result notification unit 16 notifies the analysis subject of his / her own words and actions as an event occurring when a unique change in biological reaction occurs in the analysis subject that differs from that of others (any of the three patterns described above; the same applies below). This allows the analysis subject to understand that when he / she behaves in a certain way, he / she has different feelings from others. At this time, the analysis subject may also be notified of the unique changes in biological reaction identified for the analysis subject. Furthermore, the analysis subject may be further notified of the changes in biological reaction of others for comparison.

[0041] For example, when the emotion received by others for the words and actions of the analysis subject unconsciously performed with normal emotions, or the words and actions of the analysis subject performed with particular consciousness accompanied by a certain emotion differs from the emotion felt by the analysis subject himself at the time of the words and actions, the analysis subject is notified of the words and actions of the analysis subject at that time. This makes it possible to discover words and actions that are well received by others or words and actions that are not well received by others, despite one's own awareness.

[0042] In addition, the analysis result notification unit 16 notifies the host of the online session of events that occur when a unique change in the biological reaction of the analysis subject occurs that is different from that of others, together with the unique change in the biological reaction. This allows the host of the online session to know what events are influencing what emotional changes as a phenomenon unique to the specified analysis subject. Then, it becomes possible to take appropriate measures for the analysis subject according to the content thus understood.

[0043] In addition, the analysis result notification unit 16 notifies the organizer of the online session of events occurring when a change in a specific biological reaction occurs in the analysis subject that is different from that of others, or the clustering results of the analysis subject. This allows the organizer of the online session to grasp the behavioral tendencies specific to the analysis subject and predict possible future behaviors and conditions, etc., depending on which classification the specified analysis subject is clustered into. Then, it becomes possible to take appropriate measures for the analysis subject.

[0044] 2. Generate the model <Summary> The system of this embodiment includes a memory unit that stores a mental disability learning model trained using training videos including patients diagnosed with mental disorders as teacher data, an acquisition unit that acquires videos including the subject of analysis, and an analysis unit that applies the mental disability learning model to the videos to analyze whether the subject of analysis has a mental disorder or not.

[0045] <Model generation method> The mental disorder learning model uses videos showing the behavior, facial expressions, speech, and other characteristics of patients diagnosed with mental disorders as training data. This allows the model to learn the characteristics and patterns of mental disorders and become capable of analyzing the presence or absence of mental disorders in new video data.

[0046] In particular, the learning video according to this embodiment is a recording of a first subject and a second subject undergoing a mental disorder diagnosis process by a specialist, where the first subject is a person who has been previously diagnosed with a mental disorder, and the second subject is a person who has not been previously diagnosed with a mental disorder.

[0047] The first subject was a person who had already been diagnosed with a mental disorder before the video was taken. The video of this subject serves as the primary training data for the model to learn the defining characteristics of a mental disorder. From the video of the first subject, the model can learn typical symptoms, reactions, facial expressions, speech, and other characteristics of a mental disorder.

[0048] The second subject was not diagnosed with a mental disorder before the video was taken. The video of this subject helps the model learn normal reactions and behavioral characteristics. It is also essential to prevent overfitting and to improve the model's ability to learn normal behavioral characteristics and determine the absence of a mental disorder.

[0049] As shown in Figure 5, the specific generation procedure of the mental disability learning model is as follows:

[0050] <Step 1> The first and second subjects are asked to talk to the expert and the conversation is recorded, and both video (visual information) and audio (auditory information) data are acquired.

[0051] As an example of the diagnostic process mentioned above, the "diagnostic criteria for depression" are as follows. That is, depression is diagnosed based on the patient's self-reported symptoms, clinical observation, and standard diagnostic criteria (e.g., DSM-5, ICD-10, etc.). For example, a specialist will ask whether the following symptoms are present by questioning the patient. 1. Low mood or depressed mood 2. Loss of interest or pleasure 3. Fatigue and loss of energy 4. Feelings of worthlessness or excessive or inappropriate guilt 5. Poor concentration and difficulty making decisions 6. Insomnia or hypersomnia 7. Weight changes with increased or decreased appetite 8. Thoughts of self-harm or suicide or plans

[0052] <Step 2> More specifically, the conversation between the expert and the subject is conducted in accordance with a clinically recognized diagnostic process for mental disorders. This process clearly captures the signs and characteristics of mental disorders, such as the subject's reactions, facial expressions, choice of words, and tone, and contributes to model generation. The expert may evaluate the presence or absence and degree of the above-mentioned symptoms of depression through conversation with the patient, or may label the subject. In addition, information such as the patient's daily life and level of activity, past medical history, family medical history, medications being used, and other treatments may be collected.

[0053] Examples of conversations with experts that may be obtained include the following: 1. Expert: "How have you been feeling lately? How has your mood changed from day to day?" Subject: "I feel depressed all the time and I don't feel like doing anything. I feel like I can't enjoy anything." 2. Expert: "How is your sleep at night? Do you wake up multiple times during the night?" Subject: "I wake up many times during the night and feel tired and have difficulty getting up in the morning." 3. Expert: "Have you noticed any changes in your diet recently? Has your appetite increased or decreased?" Subject: "I have almost no appetite and I feel like it's too much of a bother to eat." 4. Expert: “How do you feel about yourself? Has your sense of self-worth or worth changed?” Subject: "I often feel like I can't do anything and I have a strong sense of worthlessness."

[0054] <Step 3> Multimodal AI is used to analyze the features of all subjects' conversations and generate a learning model. By using multimodal AI, it is possible to make the most of the information obtained from both video and audio to generate a highly accurate mental disorder learning model.

[0055] <Other model generation methods> The model according to this embodiment may be generated, for example, in the following manner. Supervised learning: Videos of patients with diagnosed mental disorders are combined with the diagnoses (labels) of those disorders. The model learns by extracting features from the videos and associating them with the labels. Transfer learning: Using existing video and face recognition models as a basis, we train models that are specific to mental disorders. We use models that have been pre-trained with a large amount of general video data, and then fine-tune them with a small amount of mental disorder data. Deep Learning: Using neural networks to capture deep features of image and audio data in a video. CNN (Convolutional Neural Network) is used to capture visual features in the video, and RNN (Recurrent Neural Network) and LSTM (Long Term The method learns features with temporal continuity using Short Term Memory (Short Term Memory). Reinforcement learning: Based on the diagnosis results, the model is trained repeatedly by giving rewards when it makes correct decisions, so that the model learns which actions will produce the most accurate analysis results. By appropriately combining these learning methods, a learning model can be generated that can capture the characteristics of mental disorders with high accuracy.

[0056] 3. Analysis of the subjects <Video Acquisition> The subject of analysis is recorded engaged in everyday conversations. For example, the recording may be of an everyday online meeting. It may mimic everyday conversations with friends and family, or a conversation with a computer-generated agent. In the case of the agent, general everyday questions or questions that elicit deeper emotions may be programmed, and the subject's reactions may be observed.

[0057] <Application of analytical model> The acquired video is then input into the multimodal AI. For analysis, information is obtained from both the video's visual data (facial expressions, body language, etc.) and audio data (speaking style, choice of words, etc.). Using the previously generated mental disorder learning model, the features and patterns contained in the subject's video are analyzed to determine whether there are signs or characteristics of a mental disorder.

[0058] <Getting numerical values> The AI ​​outputs the results of the analysis as numerical data. This numerical value may be between 0 and 1, for example, and the closer to 1 the value, the stronger the characteristics of a mental disorder. This numerical value can be used as an index showing the probability and degree of a mental disorder.

[0059] <Threshold setting and judgment> Based on a pre-set threshold, the system judges whether the subject of the analysis is likely to have a mental disorder. For example, if the value is 0.7 or higher, it is judged that there is a high probability that there are signs of a mental disorder. This threshold is set based on previous research and clinical experience, and is expected to be updated according to the times and progress of research. Ultimately, this system can detect mental disorders at an early stage.

[0060] <Output format> The above-mentioned judgment result is output in a predetermined format. Examples of the output format include a PDF report, a notification by e-mail, display on a dashboard of a human resources cloud service, API integration, etc. Examples of recipients of the output include a doctor, a counselor, or other specialist, the subject, his / her family, the management department of an organization such as a company or school, an insurance company, or a rehabilitation facility.

[0061] <Examples of analysis subjects> Examples of diseases that can be treated by this system include: depression, bipolar disorder, schizophrenia, generalized anxiety disorder, obsessive-compulsive disorder, panic disorder, social anxiety disorder, PTSD, borderline personality disorder, antisocial personality disorder, dependent personality disorder, avoidant personality disorder, attention-deficit hyperactivity disorder, autism spectrum disorder, alcohol use disorder, drug use disorder, Alzheimer's disease, dementia with Lewy bodies, vascular dementia, anorexia nervosa, bulimia nervosa, insomnia, sleep apnea syndrome, hyposexuality, sexual dysfunction, conversion disorder, dissociative disorder, samatoform disorder, pseudo-disorder caused by others, and pseudo-disorder caused by self.

[0062] The processes described herein using flow chart diagrams need not necessarily be performed in the order shown, some process steps may be performed in parallel, additional process steps may be employed, and some process steps may be omitted.

[0063] The above-described embodiments may be appropriately combined and implemented. The effects described in this specification are merely explanatory or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are obvious to a person skilled in the art from the description of this specification, in addition to or in place of the above effects. [Explanation of symbols]

[0064] 10, 20 user terminals 30 Video Session Service Terminal 40 Evaluation Devices

Claims

1. A memory unit that stores a mental disorder learning model that has been trained using training videos including patients diagnosed with a mental disorder as training data; An acquisition unit that acquires a video to be analyzed that includes a person to be analyzed; an analysis unit that applies the mental disability learning model to the analysis target video to analyze whether the analysis target has the mental disability; Psychological disorder analyzer.

2. 2. The mental disorder analysis apparatus according to claim 1, The learning video is a recording of the first subject and the second subject undergoing a diagnosis process for the mental disorder by an expert, The first subject is a person who has been previously diagnosed with the mental disorder, and the second subject is a person who has not been previously diagnosed with the mental disorder. Psychological disorder analyzer.

3. 2. The mental disorder analysis apparatus according to claim 1, The video to be analyzed includes at least facial expressions and voices of the person to be analyzed. Psychological disorder analyzer.

4. 2. The mental disorder analysis apparatus according to claim 1, The video to be analyzed is a video about a conversation that the subject is having with another person. Psychological disorder analyzer.

5. 2. The mental disorder analysis apparatus according to claim 1, The video to be analyzed was acquired using an online session. Psychological disorder analyzer.

6. 2. The mental disorder analysis apparatus according to claim 1, The analysis unit includes information regarding whether the degree of the mental disorder of the analysis subject exceeds a predetermined threshold. Psychological disorder analyzer.

7. 2. The mental disorder analysis apparatus according to claim 1, A questioning unit that asks the subject of analysis a predetermined additional question; An answer acquisition unit for acquiring answers to the additional questions, The analysis unit analyzes the answer to analyze whether the subject has the specific mental disorder. Psychological disorder analyzer.

8. A step of storing a mental disorder learning model, which is trained using training videos including patients diagnosed with a mental disorder as training data, in a storage unit of a computer; A step of acquiring a video to be analyzed that includes a person to be analyzed; and applying the mental disability learning model to the analysis target video to analyze whether the analysis target has the mental disability. Methods of analysis of mental disorders.

9. A memory unit that stores a mental disorder learning model that has been trained using training videos including patients diagnosed with a mental disorder as training data; An acquisition unit that acquires a video to be analyzed that includes a person to be analyzed; an analysis unit that applies the mental disability learning model to the analysis target video to analyze whether the analysis target has the mental disability; Psychiatric Disorder Analysis System.

10. Computer A memory unit that stores a mental disorder learning model that has been trained using training videos including patients diagnosed with a mental disorder as training data; an acquisition unit that acquires a video to be analyzed including a person to be analyzed; applying the mental disability learning model to the analysis target video to function as an analysis unit that analyzes whether the analysis target has the mental disability; Psychiatric Analysis Program.

Images