Mental disorder analysis device, mental disorder analysis method, mental disorder analysis system, and mental disorder analysis program

Abstract

[Problem] To provide a device for analyzing the presence of a mental disorder. [Solution] A system according to the present disclosure comprises: a storage unit that stores a mental disorder learning model trained using, as teacher data, a moving image for learning including a patient diagnosed as having a mental disorder; an acquisition unit that acquires a moving image for analysis including a subject for analysis; and an analysis unit that applies the mental disorder learning model to the moving image for analysis to analyze whether the subject for analysis has the mental disorder.

Description

Mental disorder analysis device, mental disorder analysis method, mental disorder analysis system, and mental disorder analysis program

[0001] The present invention relates to a mental disorder analyzer.

[0002] A technique for analyzing the emotions felt by others in response to a speaker's utterances is known (see, for example, Patent Document 1). A technique for chronologically analyzing changes in a subject's facial expression over a long period of time and estimating the emotions felt during that time is also known (see, for example, Patent Document 2). A technique for identifying the factors that most influenced changes in emotions is also known (see, for example, Patent Documents 3 to 5). A technique for comparing a subject's usual facial expression with their current facial expression and issuing an alert if the facial expression is gloomy is also known (see, for example, Patent Document 6). A technique for comparing a subject's normal (expressionless) facial expression with their current facial expression to determine the subject's level of emotion is also known (see, for example, Patent Documents 7 to 9). Furthermore, a technique for analyzing organizational emotions and the atmosphere felt by individuals within a group is also known (see, for example, Patent Documents 10 and 11).

[0003] JP 2019-58625 A JP 2016-149063 A JP 2020-86559 A JP 2000-76421 A JP 2017-201499 A JP 2018-112831 A JP 2011-154665 A JP 2012-8949 A JP 2013-300 A JP 2011-186521 A WO15 / 174426

[0004] All of the above technologies can analyze a person from multiple angles, but they cannot predict a specific condition. Improvements are needed, especially to determine whether a person is in a disease state.

[0005] The present invention particularly aims to make it possible to determine whether a subject is in a state related to a mental disorder.

[0006] According to the present invention, a mental disorder analysis device is provided which comprises: a memory unit which stores a mental disorder learning model trained using training videos including patients diagnosed with a mental disorder as training data; an acquisition unit which acquires videos to be analyzed including the person being analyzed; and an analysis unit which applies the mental disorder learning model to the videos to be analyzed to analyze whether the person being analyzed has the mental disorder.

[0007] According to the present disclosure, the burden of medical interviews for determining mental disorders can be reduced.

[0008] Fig. 1 is a diagram showing an overall system according to an embodiment of the present invention; Fig. 2 is an example of a functional block diagram according to an embodiment of the present invention; Fig. 3 is a diagram showing an example of the functional configuration of a video analysis device according to an embodiment of the present invention; Fig. 4 is a diagram showing an example of the functional configuration of a mental disorder analysis device according to an embodiment of the present invention; Fig. 5 is a diagram showing the flow of the mental disorder device; Fig. 6 is a diagram showing the flow of the mental disorder device.

[0009] The contents of embodiments of the present disclosure will be described below. The present disclosure has the following configuration. [Item 1] A mental disorder analysis device comprising: a memory unit that stores a mental disorder learning model trained using training videos including patients diagnosed with a mental disorder as training data; an acquisition unit that acquires analysis target videos including a subject to be analyzed; and an analysis unit that applies the mental disorder learning model to the analysis target videos to analyze whether the subject to be analyzed has the mental disorder. [Item 2] The mental disorder analysis device according to claim 1, wherein the training videos are video recordings of both a first subject and a second subject undergoing a mental disorder diagnosis process by a specialist, the first subject having been diagnosed with the mental disorder in advance, and the second subject having not been diagnosed with the mental disorder in advance. [Item 3] The mental disorder analysis device according to claim 1, wherein the analysis target videos include at least facial expressions and voices of the subject to be analyzed. [Item 4] The mental disorder analysis device according to claim 1, wherein the video to be analyzed is related to a conversation the subject is having with another person. [Item 5] The mental disorder analysis device according to claim 1, wherein the video to be analyzed is acquired using an online session. [Item 6] The mental disorder analysis device according to claim 1, wherein the analysis unit includes information regarding whether the degree of the mental disorder of the subject exceeds a predetermined threshold. [Item 7] The mental disorder analysis device according to claim 1, further comprising: a questioning unit that asks the subject predetermined additional questions; and an answer acquisition unit that acquires answers to the additional questions, wherein the analysis unit analyzes the answers to analyze whether the subject has the specific mental disorder.[Item 8] A mental disorder analysis method comprising the steps of storing in a computer memory a mental disorder learning model trained using training videos including patients diagnosed with a mental disorder as training data, acquiring analysis target videos including a subject to be analyzed, and applying the mental disorder learning model to the analysis target videos to analyze whether or not the subject to be analyzed has the mental disorder. [Item 9] A mental disorder analysis system comprising: a memory unit that stores a mental disorder learning model trained using training videos including patients diagnosed with a mental disorder as training data, an acquisition unit that acquires analysis target videos including the subject to be analyzed, and an analysis unit that applies the mental disorder learning model to the analysis target videos to analyze whether or not the subject to be analyzed has the mental disorder. [Item 10] A mental disorder analysis program that causes a computer to function as a memory unit that stores a mental disorder learning model trained using training videos including patients diagnosed with a mental disorder as training data, an acquisition unit that acquires analysis target videos including the subject to be analyzed, and an analysis unit that applies the mental disorder learning model to the analysis target videos to analyze whether or not the subject to be analyzed has the mental disorder.

[0010] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0011] The present invention relates to an analysis system that analyzes video recordings of a person being analyzed communicating and analyzes whether or not the person is in a specific mental state (for example, but not limited to, a mental disorder).

[0012] This system generates a mental disorder learning model trained using training videos including patients diagnosed with mental disorders as training data, and applies the mental disorder learning model to videos including the subject of analysis to analyze whether the subject of analysis has a mental disorder. The generation of the mental disorder learning model and the analysis of the subject of analysis are performed using a video analysis device described below.

[0013] The following describes the embodiment of this system in the following order: 1. Video analysis device Basic functions of the video analysis device Example of hardware configuration Video acquisition method Analysis flow 2. Model generation Overview Model generation method Other model generation methods 3. Analysis of analysis subjects Video acquisition Application of analysis model Acquisition of numerical values ​​Threshold setting and judgment Output format Example of analysis subjects

[0014] 1. Video Analysis Device <Basic Functions> The video analysis device of this embodiment is a system that analyzes and evaluates the unique emotions (e.g., feelings of pleasure or discomfort in response to one's own or another's words or actions, or the degree of such feelings) of a person being analyzed, which differ from those of others, in an environment where multiple people are participating in a video session (hereinafter, both one-way and two-way sessions are referred to as an online session). An online session, such as an online conference, online class, or online chat, involves connecting terminals installed in multiple locations to a server via a communication network such as the Internet, enabling the exchange of video images between the multiple terminals through the server. Video images handled in an online session include facial images and audio of users using the terminals. Video images also include images of documents shared and viewed by multiple users. It is possible to switch between facial images and document images on the screen of each terminal, displaying only one of them, or to split the display area to simultaneously display both facial images and document images. It is also possible to display an image of one of the multiple people in full screen, or to split the images of some or all of the users into smaller screens. It is possible to designate one or more of the multiple users participating in an online session using terminals as analysis targets. For example, the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the organizer) designates one of the users as the analysis target. The organizer of the online session is, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a session intended for coaching. The organizer of the online session is typically one of the multiple users participating in the online session, but may also be a different person who does not participate in the online session. Note that it is also possible to designate no analysis target and to treat all participants as analysis targets. It is also possible for the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the organizer) to designate one of the users as the analysis target. The organizer of the online session is, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a session intended for coaching.The host of an online session is typically one of multiple users participating in the online session, but may also be a different person who is not participating in the online session.

[0015] In the video analysis device according to this embodiment, when a video session is established between multiple terminals, at least a video acquired from the video session is displayed. The displayed video is acquired by the terminal, and at least facial images contained in the video are identified for each predetermined frame. An evaluation value for the identified facial images is then calculated. The evaluation value is shared as needed. In particular, in this embodiment, the acquired video is stored on the terminal, analyzed and evaluated on the terminal, and the results are provided to the user of the terminal. Therefore, even if a video session contains personal information or confidential information, for example, the video itself can be analyzed and evaluated without providing the video itself to an external evaluation agency, etc. Furthermore, if necessary, the evaluation results (evaluation values) can be provided to an external terminal, allowing the results to be visualized and cross-analysis, etc. to be performed.

[0016] As shown in Figure 1, the video analysis device of this embodiment includes user terminals 10 and 20 having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display, and an output unit such as a speaker, a video session service terminal 30 that provides a two-way video session to the user terminals 10 and 20, and an evaluation terminal 40 that performs part of the evaluation of the video session.

[0017] <Hardware Configuration Example> Each functional block, functional unit, and functional module described below can be configured using, for example, hardware, a DSP (Digital Signal Processor), or software provided in a computer. For example, when configured using software, the system is actually configured with a computer's CPU, RAM, ROM, etc., and is realized by running a program stored in a recording medium such as RAM, ROM, a hard disk, or a semiconductor memory. The series of processes performed by the system and terminal described herein can be realized using software, hardware, or a combination of software and hardware. A computer program for implementing each function of the information sharing support device 10 according to this embodiment can be created and installed on a PC or the like. A computer-readable recording medium storing such a computer program can also be provided. Examples of the recording medium include a magnetic disk, an optical disk, a magneto-optical disk, and a flash memory. The computer program may also be distributed, for example, via a network, without using a recording medium.

[0018] The evaluation terminal according to this embodiment acquires moving images from a video session service terminal, identifies at least facial images contained in the moving images for each predetermined frame unit, and calculates an evaluation value for the facial images (details will be described later).

[0019] <Video Acquisition Method> As shown in Figure 2, the video session service (hereinafter simply referred to as "this service") provided by the video session service terminal enables two-way image and audio communication with user terminals 10 and 20. This service displays video captured by the camera of the other user terminal on the display of the user terminal, and can output audio captured by the microphone of the other user terminal from the speaker. This service is also configured to enable both or either user terminal to record video and audio (collectively referred to as "video, etc.") in the memory of at least one of the user terminals. The recorded video information Vs (hereinafter referred to as "recorded information") is cached on the user terminal that initiated the recording and is recorded only locally on one of the user terminals. If necessary, users can view the recorded information themselves or share it with others within the scope of their use of this service.

[0020] <Analysis flow> Fig. 3 is a block diagram showing an example configuration according to this embodiment. As shown in Fig. 3, the video analysis device of this embodiment is realized as a functional configuration possessed by a user terminal 10. That is, the user terminal 10 has, as its functions, a video image acquisition unit 11, a biological reaction analysis unit 12, a peculiar determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16.

[0021] The video acquisition unit 11 acquires video from each terminal, which is obtained by capturing images of multiple people (multiple users) using a camera provided in each terminal during an online session. The video acquired from each terminal may or may not be set to be displayed on the screen of each terminal. In other words, the video acquisition unit 11 acquires video from each terminal, including video currently being displayed and video currently not being displayed on each terminal.

[0022] The biological response analysis unit 12 analyzes changes in biological responses for each of multiple people based on the moving images (whether or not they are being displayed on the screen) acquired by the moving image acquisition unit 11. In this embodiment, the biological response analysis unit 12 separates the moving images acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and audio, and analyzes changes in biological responses from each.

[0023] For example, the biological response analysis unit 12 analyzes changes in biological responses related to at least one of facial expression, eye movement, pulse rate, and facial movement by analyzing the user's facial image using frame images separated from the video acquired by the video acquisition unit 11. Furthermore, the biological response analysis unit 12 analyzes changes in biological responses related to at least one of the user's speech content and voice quality by analyzing the audio separated from the video acquired by the video acquisition unit 11.

[0024] When a person's emotions change, this is reflected in changes in biological reactions such as facial expressions, eye movements, pulse rate, facial movements, speech content, and voice quality. In this embodiment, changes in the user's emotions are analyzed by analyzing changes in the user's biological reactions. One example of the emotion analyzed in this embodiment is the degree of comfort / discomfort. In this embodiment, the biological reaction analysis unit 12 quantifies changes in biological reactions according to a predetermined standard, thereby calculating a biological reaction index value that reflects the details of the changes in biological reactions.

[0025] The analysis of facial expression changes is performed, for example, as follows: For each frame image, a facial region is identified within the frame image, and the identified facial expressions are classified into multiple categories according to an image analysis model that has been trained in advance by machine learning. Based on the classification results, the system analyzes whether a positive or negative facial expression change has occurred between consecutive frame images, and the magnitude of the change, and outputs a facial expression change index value according to the analysis results.

[0026] The analysis of changes in gaze is performed, for example, as follows. That is, for each frame image, the eye area is identified within the frame image, and the direction of both eyes is analyzed to analyze where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker currently being displayed, at the shared document currently being displayed, or looking off-screen. It may also be possible to analyze whether the gaze movements are large or small, and whether the movements are frequent or infrequent. The gaze changes are also related to the user's concentration level. The biological response analysis unit 12 outputs a gaze change index value according to the analysis results of the gaze changes.

[0027] The analysis of pulse rate changes is performed, for example, as follows. That is, for each frame image, the facial area is identified within the frame image. Then, using a trained image analysis model that captures the numerical value of facial color information (G in RGB), changes in the G color of the facial surface are analyzed. The results are arranged along the time axis to form a waveform representing changes in color information, and the pulse is identified from this waveform. When a person is nervous, their pulse rate increases, and when they feel calm, their pulse rate decreases. The biological response analysis unit 12 outputs a pulse rate change index value according to the analysis results of the pulse rate changes.

[0028] The analysis of changes in facial movement is performed, for example, as follows. That is, for each frame image, a facial area is identified within the frame image, and the facial direction is analyzed to analyze where the user is looking. For example, it is analyzed whether the user is looking at the face of the currently displayed speaker, the currently displayed shared material, or looking off-screen. It may also be analyzed whether the facial movement is large or small, or whether the movement is frequent or infrequent. It may also be analyzed by combining facial movement and eye movement. For example, it may be analyzed whether the user is looking directly at the currently displayed speaker's face, looking up or down, or looking at an angle. The biological response analysis unit 12 outputs a facial direction change index value according to the analysis result of the change in facial direction.

[0029] The analysis of speech content is performed, for example, as follows. That is, the biological response analysis unit 12 converts speech for a specified period of time (for example, approximately 30 to 150 seconds) into a string of characters by performing known speech recognition processing, and then performs morphological analysis on the string of characters to remove words unnecessary for expressing the conversation, such as particles and articles. The remaining words are then vectorized, and an analysis is performed to determine whether a positive or negative emotional change has occurred, and the extent of the emotional change, and a speech content index value corresponding to the analysis result is output.

[0030] Voice quality analysis is performed, for example, as follows: The biological response analysis unit 12 identifies the acoustic features of the voice by performing known voice analysis processing on the voice for a specified period of time (for example, approximately 30 to 150 seconds). Based on the acoustic features, the unit then analyzes whether a positive or negative voice quality change has occurred and the magnitude of the voice quality change, and outputs a voice quality change index value according to the analysis results.

[0031] The biological response analysis unit 12 calculates a biological response index value using at least one of the facial expression change index value, eye direction change index value, pulse rate change index value, facial direction change index value, speech content index value, and voice quality change index value calculated as described above. For example, the biological response index value is calculated by weighting the facial expression change index value, eye direction change index value, pulse rate change index value, facial direction change index value, speech content index value, and voice quality change index value.

[0032] The peculiar determination unit 13 determines whether or not the change in biological reaction analyzed for the subject of analysis is peculiar compared to the change in biological reaction analyzed for other people other than the subject of analysis. In this embodiment, the peculiar determination unit 13 determines whether or not the change in biological reaction analyzed for the subject of analysis is peculiar compared to other people based on the biological reaction index values ​​calculated for each of the multiple users by the biological reaction analysis unit 12.

[0033] For example, the unique determination unit 13 calculates the variance of the biological reaction index values ​​calculated for each of multiple people by the biological reaction analysis unit 12, and by comparing the biological reaction index value calculated for the person being analyzed with the variance, determines whether the changes in the biological reactions analyzed for the person being analyzed are unique compared to others.

[0034] The following three patterns can be considered when changes in the analyzed biological reactions of the subject are unique compared to others. The first is when no particularly large changes in biological reactions occur in others, but a relatively large change in biological reactions occurs in the subject. The second is when no particularly large changes in biological reactions occur in the subject, but a relatively large change in biological reactions occurs in others. The third is when relatively large changes in biological reactions occur in both the subject and others, but the content of the change differs between the subject and others.

[0035] The associated event identification unit 14 identifies an event occurring with respect to at least one of the subject, other people, and the environment when a change in a biological reaction determined to be unique by the unique determination unit 13 occurs. For example, the associated event identification unit 14 identifies, from video images, the words and actions of the subject when a unique change in a biological reaction occurs in the subject. The associated event identification unit 14 also identifies, from video images, the words and actions of other people when a unique change in a biological reaction occurs in the subject. The associated event identification unit 14 also identifies, from video images, the environment when a unique change in a biological reaction occurs in the subject. The environment may be, for example, shared materials displayed on the screen or something that appears in the background of the subject.

[0036] The clustering unit 15 analyzes the degree of correlation between a change in biological reaction determined to be unique by the unique determination unit 13 (for example, one or more combinations of eye contact, pulse rate, facial movement, speech content, and voice quality) and an event occurring when the unique change in biological reaction occurs (an event identified by the related event identification unit 14), and if it is determined that the correlation is at a certain level or above, it clusters the person or event being analyzed based on the analysis results of the correlation.

[0037] For example, if a specific change in a biological reaction corresponds to a negative emotional change and the event occurring when the specific change in the biological reaction occurs is also a negative event, a correlation of a certain level or higher is detected. The clustering unit 15 clusters the analysis subject or event into one of a plurality of pre-segmented classifications according to the content of the event, the degree of negativity, the magnitude of correlation, etc.

[0038] Similarly, if a specific change in biological reaction corresponds to a positive emotional change and the event occurring when the specific change in biological reaction occurs is also a positive event, a correlation of a certain level or higher is detected. The clustering unit 15 clusters the analysis subject or event into one of a plurality of pre-segmented classifications according to the content, degree of positivity, magnitude of correlation, etc. of the event.

[0039] The analysis result notification unit 16 notifies the person designating the subject of analysis (the subject of analysis or the organizer of the online session) of at least one of the changes in biological reactions determined to be specific by the specific determination unit 13, the events identified by the related event identification unit 14, and the classifications clustered by the clustering unit 15.

[0040] For example, the analysis result notification unit 16 notifies the analysis subject of the analysis subject's own words and actions as an event occurring when a unique change in biological reaction occurs in the analysis subject that is different from that of others (one of the three patterns described above; the same applies below). This allows the analysis subject to understand that when he or she behaves in a certain way, he or she has different emotions than others. At this time, the analysis subject may also be notified of the unique changes in biological reaction identified for the analysis subject. Furthermore, the analysis subject may also be notified of changes in the biological reaction of others to be compared.

[0041] For example, if the emotions felt by others in response to words or actions made by the subject without any particular awareness and with normal emotions, or words or actions made by the subject with a particular awareness and with a certain emotion differ from the emotions felt by the subject himself at the time of the words or actions, the subject will be notified of his or her own words or actions at that time. This makes it possible to discover words or actions that are well-received by others or that are not well-received by others, despite the subject's own awareness.

[0042] Furthermore, the analysis result notification unit 16 notifies the organizer of the online session of events occurring when a unique change in biological reaction occurs in the analysis subject that is different from that of others, along with the unique change in biological reaction. This allows the organizer of the online session to know what events are influencing what emotional changes as phenomena unique to the designated analysis subject. Then, it becomes possible to take appropriate measures for the analysis subject based on the information obtained.

[0043] Furthermore, the analysis result notification unit 16 notifies the organizer of the online session of events occurring when a unique change in the biological reaction of the analysis subject occurs that is different from that of others, or of the clustering results of the analysis subject. This allows the organizer of the online session to understand the behavioral tendencies unique to the analysis subject and predict possible future behaviors and conditions, etc., depending on which category the specified analysis subject is clustered into. This then makes it possible to take appropriate measures for the analysis subject.

[0044] 2. Model Generation <Overview> The system according to this embodiment includes a storage unit that stores a mental disorder learning model trained using training videos that include patients diagnosed with mental disorders as training data, an acquisition unit that acquires videos that include the subject of analysis, and an analysis unit that applies the mental disorder learning model to the videos to analyze whether the subject of analysis has a mental disorder.

[0045] <Model generation method> The mental disorder learning model uses videos that show the behavior, facial expressions, speech, and other characteristics of patients diagnosed with mental disorders as training data. This allows the model to learn the characteristics and patterns of mental disorders and gain the ability to analyze the presence or absence of mental disorders in new video data.

[0046] In particular, the training video according to this embodiment is a recording of a first subject and a second subject undergoing a mental disorder diagnosis process by a specialist, where the first subject has been diagnosed with a mental disorder in advance, and the second subject has not been diagnosed with a mental disorder in advance.

[0047] The first subject was a person who had already been diagnosed with a mental disorder before the video was taken. The video of this subject served as the primary training data for the model to learn the defining characteristics of the mental disorder. From the video of the first subject, typical symptoms, reactions, facial expressions, speech, and other characteristics of the mental disorder can be learned.

[0048] The second subject was not diagnosed with a mental disorder before the video was taken. The video of this subject helps the model learn normal responses and behavioral characteristics. This is essential to prevent overfitting and improve the model's ability to learn normal behavioral characteristics and determine the absence of a mental disorder.

[0049] As shown in FIG. 5, the specific procedure for generating the mental disorder learning model is as follows.

[0050] <Step 1> The first and second subjects are asked to converse with an expert, and the conversation is recorded, obtaining both video (visual information) and audio (auditory information) data.

[0051] As part of the diagnostic process described above, examples of the "diagnostic criteria for depression" are as follows: That is, the diagnosis of depression is made based on the patient's self-reported symptoms, clinical observation, and standard diagnostic criteria (e.g., DSM-5, ICD-10, etc.). For example, a specialist will ask whether the following symptoms are present through a medical interview: 1. Low mood or depressed mood 2. Loss of interest or pleasure 3. Fatigue or loss of energy 4. Feelings of worthlessness or excessive or inappropriate guilt 5. Poor concentration or difficulty making decisions 6. Insomnia or hypersomnia 7. Weight change accompanied by an increase or decrease in appetite 8. Thoughts or plans of self-harm or suicide

[0052] <Step 2> More specifically, the conversation between the expert and the subject is conducted in accordance with a clinically recognized diagnostic process for mental disorders. This process clearly captures the signs and characteristics of mental disorders, such as the subject's reactions, facial expressions, word choice, and tone, which contribute to model generation. Through the conversation with the patient, the expert may assess the presence and severity of the above-mentioned depression symptoms or label the subject. Additionally, information such as the patient's daily life and activity level, past medical history, family medical history, and current medications and other treatments may also be collected.

[0053] Examples of conversations with experts that can be obtained include the following: 1. Expert: "How are you feeling lately? How has your mood changed from day to day?" Subject: "I've been feeling depressed and don't feel like doing anything. I feel like I can't enjoy anything." 2. Expert: "How is your sleep at night? Do you wake up several times during the night?" Subject: "I wake up several times during the night, and I'm so tired in the morning it's hard for me to get up." 3. Expert: "Have you noticed any changes in your eating habits lately? Has your appetite increased or decreased?" Subject: "I have almost no appetite, and even eating seems like a chore." 4. Expert: "How are you feeling about yourself? Has your self-esteem or sense of self-worth changed?" Subject: "I often feel like I can't do anything, and my sense of worthlessness is increasing."

[0054] <Step 3> Multimodal AI is used to analyze the features of all subject conversations and generate a learning model. By using multimodal AI, it is possible to maximize the use of information obtained from both video and audio to generate a highly accurate learning model of mental disorders.

[0055] <Other Model Generation Methods> The model according to this embodiment may be generated by, for example, the following methods: Supervised learning: Videos of patients diagnosed with a mental disorder are used in combination with the diagnosis (label). The model learns by extracting features from the videos and associating them with labels. Transfer learning: Features specific to mental disorders are learned based on existing video recognition models or face recognition models. A model pre-trained with a large amount of general video data is used, and fine-tuned with a small amount of mental disorder data. Deep learning: A neural network is used to capture deep features of the image and audio data in the video. A convolutional neural network (CNN) is used to learn visual features in the video, and a recurrent neural network (RNN) or long short-term memory (LSTM) is used to learn features with temporal continuity. Reinforcement learning: Based on the diagnosis, the model is repeatedly trained by providing a reward when it makes a correct decision. This allows the model to learn behaviors that will produce the most accurate analysis results. By appropriately combining these learning methods, a learning model capable of capturing the characteristics of mental disorders with high accuracy is generated.

[0056] 3. Analysis of the Subject <Video Acquisition> Record the subject engaging in everyday conversation. For example, the recording may be of an everyday online meeting. It may also be used to simulate everyday conversations with friends or family, or a conversation with a computer-generated agent. In the case of an agent, it may be programmed to ask general everyday questions or questions that elicit deep emotions, and observe the subject's reactions.

[0057] <Application of analytical model> The acquired video is input into the multimodal AI. For analysis, information is obtained from both the video's visual data (facial expressions, body language, etc.) and audio data (speaking style, word choice, etc.). Using the previously generated mental disorder learning model, the characteristics and patterns contained in the subject's video are analyzed to determine whether there are signs or characteristics of a mental disorder.

[0058] <Obtaining Numerical Values> The AI ​​outputs the results of the analysis as numerical data. This numerical value may be, for example, between 0 and 1, and the closer to 1 the value is, the stronger the characteristics of a mental disorder are observed. This numerical value is used as an index showing the probability and degree of a mental disorder.

[0059] <Threshold setting and judgment> Based on a pre-set threshold, the system judges whether the subject of analysis is likely to have a mental disorder. For example, if the value is 0.7 or higher, it is judged that there is a high probability that there are signs of a mental disorder. This threshold is set based on previous research and clinical experience, and is expected to be updated as time goes by and research progresses. Ultimately, this system can detect mental disorders at an early stage.

[0060] <Output Format> The above-described determination results are output in a predetermined format. Examples of the output format include a PDF report, a notification by email, a display on a dashboard of a human resources cloud service, and API integration. Examples of recipients of the output include a doctor, a counselor, or other specialist, the subject, their family, the management department of an organization such as a company or school, an insurance company, or a rehabilitation facility.

[0061] <Examples of analysis targets> Examples of diseases that can be analyzed by this system include: depression, bipolar disorder, schizophrenia, generalized anxiety disorder, obsessive-compulsive disorder, panic disorder, social anxiety disorder, PTSD, borderline personality disorder, antisocial personality disorder, dependent personality disorder, avoidant personality disorder, attention deficit hyperactivity disorder, autism spectrum disorder, alcohol use disorder, drug use disorder, Alzheimer's disease, dementia with Lewy bodies, vascular dementia, anorexia nervosa, bulimia nervosa, insomnia, sleep apnea syndrome, hyposexuality, sexual dysfunction, conversion disorder, dissociative disorder, samatoform disorder, pseudo-disorder imposed by others, and pseudo-disorder imposed by self.

[0062] The processes described herein using flowchart diagrams do not necessarily have to be performed in the order shown, some process steps may be performed in parallel, additional process steps may be employed, and some process steps may be omitted.

[0063] The above-described embodiments may be combined as appropriate. Furthermore, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0064] 10, 20 User terminal 30 Video session service terminal 40 Evaluation terminal

Claims

1. A mental disorder analysis device comprising: a memory unit for storing a mental disorder learning model trained using training videos including patients diagnosed with a mental disorder as teacher data; an acquisition unit for acquiring analysis target videos including a subject of analysis; and an analysis unit for applying the mental disorder learning model to the analysis target videos to analyze whether the subject of analysis has the mental disorder.

2. A mental disorder analysis device as described in claim 1, wherein the learning video is a recording of both a first subject and a second subject undergoing a mental disorder diagnosis process by an expert, the first subject being a person who has been diagnosed in advance with the mental disorder, and the second subject being a person who has not been diagnosed in advance with the mental disorder.

3. A mental disorder analysis device according to claim 1, wherein the video to be analyzed includes at least facial expressions and voice of the person to be analyzed.

4. A mental disorder analysis device according to claim 1, wherein the video to be analyzed is a video relating to a conversation the subject is having with another person.

5. A mental disorder analysis device according to claim 1, wherein the video to be analyzed is acquired using an online session.

6. A mental disorder analysis apparatus as claimed in claim 1, wherein said analysis unit includes information on whether the degree of said mental disorder of said subject exceeds a predetermined threshold.

7. A mental disorder analysis device as described in claim 1, further comprising a questioning unit that asks the subject of analysis certain additional questions, and an answer acquisition unit that acquires answers to the additional questions, wherein the analysis unit analyzes the answers to determine whether the subject of analysis has the specific mental disorder.

8. A method for analyzing mental disorders, comprising the steps of: storing in a computer's memory a mental disorder learning model trained using training videos including patients diagnosed with mental disorders as training data; acquiring a target video for analysis including a person to be analyzed; and applying the mental disorder learning model to the target video for analysis to analyze whether the person to be analyzed has the mental disorder.

9. A mental disorder analysis system comprising: a memory unit for storing a mental disorder learning model trained using training videos including patients diagnosed with a mental disorder as teacher data; an acquisition unit for acquiring analysis target videos including a subject of analysis; and an analysis unit for applying the mental disorder learning model to the analysis target videos to analyze whether the subject of analysis has the mental disorder.

10. A mental disorder analysis program which causes a computer to function as: a memory unit which stores a mental disorder learning model trained using training videos including patients diagnosed with a mental disorder as teacher data; an acquisition unit which acquires analysis target videos including a person to be analyzed; and an analysis unit which applies the mental disorder learning model to the analysis target videos to analyze whether the person to be analyzed has the mental disorder.

Images