Emotion estimation program, emotion estimation method, and information processing device.
The emotion estimation method addresses the accuracy issues in existing technologies by integrating physiological and psychological data through machine learning, enhancing fraud detection accuracy.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- FUJITSU LTD
- Filing Date
- 2022-09-09
- Publication Date
- 2026-06-02
Smart Images
Figure 0007868465000001 
Figure 0007868465000002 
Figure 0007868465000003
Abstract
Description
Technical Field
[0001] The present invention relates to an emotion estimation program, an emotion estimation method, and an information processing device.
Background Art
[0002] In recent years, with the development of wearable devices and the like, it has become easier to obtain physiological reaction information such as heart rate and respiratory rate, and emotion estimation technologies that utilize physiological reaction information have been used. For example, there is a known technology for obtaining the physiological reaction information of a person during a conversation on the phone, estimating emotions from the obtained physiological reaction information, and using the estimated emotions to prevent special fraud.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Patent Document 2
Patent Document 3
Patent Document 4
Patent Document 5
Summary of the Invention
Problems to be Solved by the Invention
[0004] However, it is difficult to say that the emotions estimated from the above physiological reaction information are highly accurate. For example, there are large individual differences in the correlation between mental states and physiological reactions. Some people's heart rates increase when they become anxious, while others' heart rates do not change even when they become anxious. Therefore, the emotions estimated from physiological reactions lack accuracy.
[0005] One objective is to provide an emotion estimation program, an emotion estimation method, and an information processing device that can estimate emotions with high accuracy. [Means for solving the problem]
[0006] In the first proposal, the emotion estimation program is characterized by causing a computer to perform the following processes: acquire physiological response information of a person identified from the person's vital data, acquire physical information of the person identified from video data of the person, input the acquired physiological response information and the physical information into a first machine learning model to generate psychological characteristics that indicate the person's unique personality, and estimate the person's emotions based on the generated psychological characteristics. [Effects of the Invention]
[0007] According to one embodiment, emotions can be estimated with high accuracy. [Brief explanation of the drawing]
[0008] [Figure 1] Figure 1 is a diagram illustrating the information processing device according to Example 1. [Figure 2] Figure 2 is a diagram illustrating the reference technology. [Figure 3] Figure 3 illustrates the problems with the reference technology. [Figure 4] Figure 4 illustrates specific examples of psychological characteristics. [Figure 5] Figure 5 is a diagram illustrating the system configuration according to Example 1. [Figure 6] Figure 6 is a functional block diagram showing the functional configuration of the information processing device according to Embodiment 1. [Figure 7] Figure 7 is a diagram illustrating the first training data database. [Figure 8] Figure 8 is a diagram illustrating the second training data database. [Figure 9] Figure 9 is a diagram illustrating the third training data database. [Figure 10]FIG. 10 is a diagram showing an example of the victim's feelings. [Figure 11] FIG. 11 is a diagram showing an example of the feelings at the time of the crime. [Figure 12] FIG. 12 is a diagram showing the relationship between refund fraud and emotional patterns. [Figure 13] FIG. 13 is a diagram showing the relationship between self-made fraud and emotional patterns. [Figure 14] FIG. 14 is a diagram for explaining the generation of training data. [Figure 15] FIG. 15 is a diagram for explaining the machine learning of the characteristic estimation model. [Figure 16] FIG. 16 is a diagram for explaining the machine learning of the emotion estimation model. [Figure 17] FIG. 17 is a diagram for explaining the machine learning of the crime risk estimation model. [Figure 18] FIG. 18 is a diagram for explaining the estimation of psychological characteristics. [Figure 19] FIG. 19 is a diagram for explaining the estimation of emotions. [Figure 20] FIG. 20 is a diagram for explaining the estimation of crime risk. [Figure 21] FIG. 21 is a diagram for explaining a specific example of an emotional pattern. [Figure 22] FIG. 22 is a flowchart for explaining the flow of machine learning processing. [Figure 23] FIG. 23 is a flowchart for explaining the flow of crime risk estimation processing. [Figure 24] FIG. 24 is a diagram for explaining an example of hardware configuration.
MODE FOR CARRYING OUT THE INVENTION
[0009] Hereinafter, embodiments of the emotion estimation program, emotion estimation method, and information processing apparatus disclosed in the present application will be described in detail based on the drawings. Note that the present invention is not limited by this embodiment. Also, each embodiment can be appropriately combined within a non-contradictory range.
EXAMPLE
[0010] (Description of information processing device) Figure 1 is a diagram illustrating the information processing device 10 according to Embodiment 1. The information processing device 10 shown in Figure 1 is an example of a computer device that uses imaging data, including video and images of a target person, to estimate emotions with high accuracy when detecting special fraud using telephones, detecting suspicious persons in stores, or detecting employee stress in the workplace.
[0011] In recent years, telephone fraud, such as "ore-ore" (impersonation) scams and refund scams, has increased dramatically. Methods have been proposed to prevent fraud by analyzing the audio content of phone calls. However, the methods used by perpetrators are becoming more complex and sophisticated to match, so these methods quickly become inapplicable. Therefore, there is a need for detection methods that do not depend on the perpetrator's methods.
[0012] Therefore, by focusing on the emotional changes (such as tension and anxiety) that victims experience in all types of fraud, it is possible to detect when someone is involved in fraud through emotion estimation. In recent years, due to the development of wearable devices and other technologies that make it easy to obtain such information, emotion estimation technologies exist that utilize physiological response information such as heart rate and respiratory rate.
[0013] Figure 2 illustrates the reference technology. As shown in Figure 2, the reference technology acquires physiological response information from a wearable device worn by the person during a call, inputs this information into a trained machine learning model, and estimates the person's emotions. The reference technology then uses the estimated emotions to determine whether or not the person is being subjected to a special fraud. However, there are significant individual differences in the correlation between physiological response information and emotions, and there are limitations to estimating emotions based solely on physiological response information.
[0014] Figure 3 illustrates the problems with the reference technology. The left axis of Figure 3 shows heart rate, and the right axis represents an index used in psychology to indicate psychological state (such as level of excitement), with higher values indicating a more excited state. Figure 3(a) shows a person in whom the excitement level decreases and changes to a calm state as the heart rate decreases, demonstrating a correlation between physiological response and psychological state. On the other hand, Figure 3(b) shows a person in whom the psychological state does not change even when the heart rate decreases, demonstrating a correlation between physiological response and psychological state.
[0015] Thus, there are differences between individuals in physiological responses and psychological states. In particular, for individuals whose physiological responses and psychological states are not correlated, it is considered difficult to estimate emotions based solely on physiological responses, and we consider using individual characteristic information as features for emotion estimation. Therefore, in this embodiment, we develop a method that absorbs the above-mentioned individual differences in emotion estimation using physiological responses, thereby improving accuracy and enabling fraud detection through highly accurate emotion estimation.
[0016] Here, we will explain psychological characteristics, which are an example of personal characteristic information used in this embodiment. Figure 4 is a diagram illustrating a specific example of psychological characteristics. Psychological characteristics are traits inherent to a person, regardless of their emotions or psychological state. These psychological characteristics are similar to information about a person's personality, individuality, and traits, such as being prone to worry, not worrying about small details, or having a tendency to be suspicious of others. Therefore, there are two aspects to psychological characteristics: those perceived by the person themselves and those perceived by people related to that person. These can be identified through questionnaires to the person, questionnaires to related parties, psychological experiments, etc.
[0017] In this embodiment, as shown in Figure 4, suspicion and anxiety traits are used. For example, User A has suspicion (50) and anxiety traits (24) as psychological traits, and User B has suspicion (44) and anxiety traits (24) as psychological traits.
[0018] Furthermore, the scale used in POMS2 (Profile of Mood States Second Edition) can be used to measure suspicion. For anxiety traits, the trait anxiety scale measured by STAI (State-Trait Anxiety Inventory) can be used. Note that the calculated values of these scales may be used as they are, or as normalized values. However, other indicators can be used if they represent traits inherent to the person, regardless of emotion or psychological state.
[0019] (System Configuration) Next, a special fraud detection system using the information processing device 10 described above will be explained. Figure 5 is a diagram illustrating the system configuration according to Example 1. As shown in Figure 5, this system is a public-private partnership system that includes a special fraud prevention solution 1, a special fraud group 2, a user's home 3, and a local government 4. This system is an example of a special fraud prevention system that uses a crime detection model trained with criminal psychology and other factors to detect special fraud in real time, prevent the occurrence of special fraud, and trains the evolving characteristics of special fraud to provide effective guidance to local governments and other organizations.
[0020] Special Fraud Prevention Solution 1 is a service implemented by companies and other organizations, which generates and provides a crime detection model that combines criminal psychology and machine learning. This crime detection model is trained on patterns of user emotions at the time of special fraud, as identified by criminal psychology. For example, the crime detection model includes a machine learning model that estimates user emotions from sensing data, which is an example of feature data that shows the characteristics of the user, and a machine learning model that estimates the possibility of special fraud from emotional patterns, thereby estimating the possibility of special fraud from the user's emotions in real time during a phone call.
[0021] Special fraud group 2 consists of criminals who make phone calls to the homes of users 3 and commit crimes such as special fraud. For example, special fraud group 2 carries out known special frauds, including impersonation fraud, deposit fraud, ATM card fraud, fictitious fee fraud, refund fraud, financial product fraud, gambling fraud, and matchmaking fraud, as well as unknown special frauds, including combinations of these frauds and new types of fraud.
[0022] User's home 3 is, for example, the home of an elderly user, and is the home of a user targeted by a special fraud group 2. User's home 3 is equipped with an information processing device 10 and non-contact sensing terminals, including a camera that captures video and facial images, a microphone that collects sound, a millimeter-wave sensor that measures heart rate and respiratory rate, and a wearable terminal that is worn on the arm or finger to measure pulse waves, etc. Furthermore, the information processing device 10 has a crime detection model created by the special fraud prevention solution 1 and is connected to communicate with the user's electronic devices (e.g., a telephone) and each sensing terminal.
[0023] Local government 4 will implement various measures to prevent special fraud, such as providing guidance and holding workshops for the elderly. Note that the workshops exemplified here are not limited to those conducted by local governments such as local government 4, but also include services provided by private companies, workshops conducted by neighborhood associations, and workshops held at schools.
[0024] In such a system, the information processing device 10 acquires physiological response information (respiratory rate, pulse rate, heart rate, etc.) of a person identified from the person's vital data. The information processing device 10 also acquires physical information (gender, age, etc.) of a person identified from video data of the person. Then, the information processing device 10 inputs the acquired physiological response information and physical information into a first machine learning model to generate psychological characteristics that indicate the person's unique personality. Based on the generated psychological characteristics, the information processing device 10 estimates the person's emotions.
[0025] Subsequently, the information processing device 10 estimates whether or not a special fraud has occurred against the person based on the patterns of emotional changes of the person that are estimated as they occur during the call. As a result, the information processing device 10 can more accurately detect whether the target person is a victim of a special fraud using improved emotion estimation.
[0026] (Functional Configuration) Figure 6 is a functional block diagram showing the functional configuration of the information processing device 10 according to Embodiment 1. As shown in Figure 6, the information processing device 10 has a communication unit 11, a storage unit 12, and a control unit 20.
[0027] The communication unit 11 is a processing unit that controls communication with other devices, and is implemented, for example, by a communication interface. For example, the communication unit 11 receives sensing data from sensing terminals such as wearable devices and millimeter-wave radar, receives video data from cameras, and transmits alarms and messages to users.
[0028] The memory unit 12 is an example of a processing unit that stores various data, including sensing data and imaging data, as well as programs executed by the control unit 20, and can be implemented by, for example, memory or a hard disk. This memory unit 12 stores the first training data DB 13, the second training data DB 14, the third training data DB 15, the characteristic estimation model 16, the emotion estimation model 17, and the crime risk estimation model 18.
[0029] The first training data DB13 is a database that stores the first training data used for machine learning of the characteristic estimation model 16. Figure 7 is a diagram illustrating the first training data DB13. As shown in Figure 7, the first training data stored in the first training data DB13 is data that associates "physical information" and "physiological response information," which are explanatory variables during machine learning, with "psychological characteristics," which are the target variables during machine learning.
[0030] The "physical information" stored here is information about a person's body, such as gender, age, and height, and is set as numerical information, for example, representing males as 1 and females as 0. "Physiological response information" is an example of information identified from sensing data and vital data that can be sensed from a person, such as heart rate and respiratory rate, and can be set as the measured value itself or as normalized information. "Psychological characteristics" indicate characteristics inherent to a person, regardless of emotions and psychological state, such as scales indicating suspicion or anxiety characteristics. In the example in Figure 7, a "male in his 70s (1)" with the psychological characteristics of "suspiciousness scale of 50" and "anxiety characteristic scale of 20" is set to have a heart rate of 70 and a respiratory rate of 13 (normalized value).
[0031] The second training data DB14 is a database that stores the second training data used for machine learning of the emotion estimation model 17. Figure 8 is a diagram illustrating the second training data DB14. As shown in Figure 8, the second training data stored in the second training data DB14 is data that associates "physical information," "physiological response information," and "psychological characteristics," which are explanatory variables during machine learning, with "emotion," which is the target variable during machine learning.
[0032] The "physical information," "physiological response information," and "psychological characteristics" stored here are the same as those in Figure 7, so a detailed explanation will be omitted. "Emotion" is information that indicates the person's feelings, and it is also possible to set numerical information for each emotion such as "anger" and "sadness." For example, "0" can be set for "anger," and "1" for "anxiety." In the example in Figure 8, when a "male in his 70s (1)" with psychological characteristics of "suspiciousness scale of 50" and "anxiety trait scale of 20" has a heart rate of 70 and a respiratory rate of 13 (normalized value), his emotion is set to "emotion A."
[0033] The "emotions" defined here refer to the user's facial expressions during the phone call. Common indicators such as happy or sad can be used, or the seven scales defined in POM2—"anger-hostility," "confusion-embarrassment," "depression-downcast," "fatigue-apathy," "tension-anxiety," "liveliness-energy," and "friendliness"—can be adopted.
[0034] The third training data DB15 is a database that stores the third training data used for machine learning of the crime risk estimation model 18. Figure 9 is a diagram illustrating the third training data DB15. As shown in Figure 9, the third training data stored in the third training data DB15 is data that associates "emotional patterns," which are explanatory variables during machine learning, with "special fraud," which is the target variable during machine learning. The "emotional patterns" stored here represent patterns of emotional changes in a user during a phone call, and are patterns of emotional changes in a person within a predetermined time after a predetermined operation, such as answering a call on the phone, is detected. "Special fraud" is information that identifies the special fraud associated with that emotional pattern, and is the criminal act that corresponds to that emotional pattern among multiple criminal acts. In the example in Figure 9, it is shown that the special fraud when the emotion changes from emotion A to emotion B to emotion A is "fraud AA."
[0035] Here, we will explain the "emotions" used in Example 1 from both the perspective of the user (victim) receiving a call from the fraud group and the fraud group making the call to the user (perpetrator). First, we will explain emotions in detail. Figure 10 shows examples of the victim's emotions. Here, as an example, we will explain "friendliness," "liveliness / vitality," and "confusion / bewilderment." As shown in Figure 10, each emotion can be identified by the user's state, the user's tone of voice and atmosphere, and the content of the user's emotions.
[0036] For example, the "friendly" emotional state is one in which the user shows attention and interest in the person they are talking to, such as through introduction and trust. In this "friendly" state, the user's tone of voice and demeanor are "polite, cheerful, and accommodating," and the emotional content of the user is characterized by "having trust and interest" in the person they are talking to.
[0037] The "confused-bewildered" emotional state is characterized by the user becoming agitated and experiencing heart rate variability in response to their conversation partner. In this state, the user's tone of voice and demeanor become "polite and flat," and their emotional content is characterized by "anxiety and increased desire" towards their conversation partner.
[0038] The "lively to energetic" emotional state is characterized by an increased level of arousal, such as prompting action from the person the user is interacting with. In this "lively to energetic" state, the user's tone of voice and demeanor become "polite and neutral," and the emotional content of the user is characterized by a desire to "take action."
[0039] Next, we will explain the perpetrator's manipulation and the victim's emotions at the time of the crime. Figure 11 shows an example of emotions at the time of the crime. As shown in Figure 11, in order to make the victim feel the aforementioned "friendly" emotion, the perpetrator uses polite, cheerful, and sympathetic tones and atmosphere, and engages in conversation that includes keywords such as "This is a government office, you will get your money back, have you received any documents?" with the aim of "building trust and getting the victim interested." As a result, the victim develops unconditional trust in public institutions and expectations of a refund, and comes to feel "friendly."
[0040] Furthermore, in order to make the victim feel the aforementioned "confusion and bewilderment," the perpetrator uses polite and neutral language and demeanor, and engages in conversations that include keywords such as "the refund deadline has passed," "special offer, just for you," "passbook and cash card," and "refund at the ATM," with the aim of "making the victim anxious and stimulating their desires." As a result, the victim becomes anxious, develops suspicion, and is persuaded, leading to feelings of "confusion and bewilderment."
[0041] Furthermore, in order to make the victim feel the aforementioned "liveliness and vitality," the perpetrator uses polite and neutral language and demeanor, and engages in conversations that include keywords such as "speak to an employee on the phone and follow their instructions" with the aim of getting the victim to "take their passbook and go to an ATM." As a result, the victim feels anxious about operating the ATM, tries to follow the instructions, and develops feelings of "liveliness and vitality."
[0042] As mentioned above, perpetrators of special frauds use skillful rhetoric and manipulate emotions in different ways depending on the type of fraud, thereby influencing the victim's feelings. In other words, the emotional changes of victims often differ depending on the type of fraud. Therefore, the machine learning of the crime risk estimation model 18 uses training data that associates "emotional patterns" with "special frauds" based on past history, analysis of criminal psychology, and questionnaires from perpetrators and victims.
[0043] Here, as an example, we will explain the combination of special fraud and emotional patterns. Figure 12 shows the relationship between refund fraud and emotional patterns. As shown in Figure 12, in refund fraud, the perpetrator often calls the user and starts with something that will pique their interest, such as "You'll get your money back," then makes a urging statement such as "The refund deadline has passed," and finally makes a statement that will encourage action, such as "It's okay if you pay the fee." In this case, the user's emotions become "friendly" due to the interesting statement, then "confused to bewildered" due to the urging statement, and finally "lively to energetic" due to the statement that encourages action. In other words, in "refund fraud," a shift from positive to negative emotions occurs. As a result, "refund fraud" is associated with the emotional patterns of "friendly, confused to bewildered, lively to energetic."
[0044] Figure 13 illustrates the relationship between "ore-ore" (impersonation) scams and emotional patterns. As shown in Figure 13, in "ore-ore" scams, the first perpetrator, posing as a family member of the victim, calls the victim and starts with a story designed to create anxiety, such as "I've lost company money." Midway through, the second perpetrator, posing as the victim's boss, takes over and makes reassuring remarks such as "I'll cover the cost, so don't worry." After that, the first perpetrator often makes another anxious request, such as "I need you to transfer the money," and the second perpetrator often makes a statement encouraging action, such as "I'll support you over the phone." In this case, the victim's emotions shift from "confused to bewildered" due to the anxious remarks, to "friendly" due to the reassuring remarks, then back to "confused to bewildered" again, and finally to "lively to energetic" due to the action-encouraging remarks. In other words, "ore-ore" scams involve a shift from negative to positive emotions. As a result, "ore-ore" scams are associated with the emotional patterns of "confused to bewildered, friendly, confused to bewildered, lively to energetic."
[0045] Returning to Figure 6, the characteristic estimation model 16 is a machine learning model included in the crime detection model that estimates psychological characteristics. Specifically, the characteristic estimation model 16 is a first machine learning model that outputs psychological characteristics in response to input physical information and physiological response information, and is a multi-class judgment model that outputs estimated values (probabilities) corresponding to each indicator showing psychological characteristics. Various mathematical models, such as neural networks, can be employed in the characteristic estimation model 16.
[0046] The emotion estimation model 17 is a machine learning model that estimates emotions and is included in the crime detection model. Specifically, the emotion estimation model 17 is a second machine learning model that outputs emotions in response to input physical information, physiological response information, and psychological characteristics, and is a multi-class judgment model that outputs estimated values (probabilities) corresponding to each of multiple emotions. The emotion estimation model 17 can employ various mathematical models, such as neural networks.
[0047] The crime risk estimation model 18 is a machine learning model included in the crime detection model that estimates the risk of crimes such as special fraud occurring. Specifically, the crime risk estimation model 18 is a machine learning model that outputs crime risk in response to an input of an emotional pattern that includes at least one emotion, and is a multi-class judgment model that estimates the risk (probability) of each crime occurring among multiple crimes. The crime risk estimation model 18 can employ various mathematical models, such as neural networks.
[0048] The control unit 20 is the processing unit that oversees the entire information processing device 10, and is implemented by, for example, a processor. This control unit 20 includes a machine learning unit 30, an operation unit 40, and a notification unit 50. The machine learning unit 30, the operation unit 40, and the notification unit 50 are implemented by electronic circuits and processes executed by the processor.
[0049] The machine learning unit 30 comprises a data generation unit 31, a first training unit 32, a second training unit 33, and a third training unit 34, and is a processing unit that generates various machine learning models prior to real-time detection of special fraud.
[0050] The data generation unit 31 is a processing unit that generates training data for machine learning by conducting psychological experiments. Figure 14 is a diagram illustrating the generation of training data. As shown in Figure 14, the data generation unit 31 obtains the psychological characteristics and physical information of the subjects through a questionnaire administered to the subjects before the experiment (S1). Subsequently, the subjects wear wearable devices capable of measuring physiological responses (S2).
[0051] Subsequently, the data generation unit 31 provides the subject with an arbitrary stimulus, such as listening to a fraudulent voice, and records the physiological response before and after the stimulus, measured by the wearable device, for a certain period of time (S3). The data generation unit 31 also similarly records the subject's emotions before and after the stimulus through a questionnaire (S4).
[0052] In this way, the data generation unit 31 collects data for each subject that associates physical information, psychological characteristics, a certain emotion, and physiological response information when that emotion is present. The data generation unit 31 then stores the training data that associates "physical information, physiological response information, and psychological characteristics" in the first training data DB 13, and the training data that associates "physical information, physiological response information, psychological characteristics, and emotions" in the second training data DB 14.
[0053] The first training unit 32 is a processing unit that generates a characteristic estimation model 16 by machine learning using the first training data stored in the first training data DB 13. Specifically, the first training unit 32 generates a characteristic estimation model 16 that estimates "psychological characteristics" in response to inputs of "physical information and physiological response information" through supervised learning.
[0054] Figure 15 illustrates the machine learning process of the characteristic estimation model 16. As shown in Figure 15, the first training unit 32 inputs first training data, which includes explanatory variables "physical information (gender: 1, age: 70), physiological response information (heart rate: 70, respiratory rate: 13)" and target variable "psychological characteristics (suspiciousness: 50, anxiety characteristics: 20)", into the characteristic estimation model 16 and obtains the output result of the characteristic estimation model 16. The first training unit 32 then updates various parameters of the characteristic estimation model 16 to minimize the error between the output result of the characteristic estimation model 16 and the target variable "psychological characteristics (suspiciousness: 50, anxiety characteristics: 20)".
[0055] The second training unit 33 is a processing unit that generates an emotion estimation model 17 by machine learning using the second training data stored in the second training data DB 14. Specifically, the second training unit 33 generates an emotion estimation model 17 that estimates "emotions" in response to inputs of "physical information, physiological response information, and psychological characteristics" through supervised learning.
[0056] Figure 16 illustrates the machine learning process of the emotion estimation model 17. As shown in Figure 16, the second training unit 33 inputs second training data, which includes explanatory variables "physical information (gender: 1, age: 70), physiological response information (heart rate: 70, respiratory rate: 13), psychological characteristics (suspiciousness: 50, anxiety characteristics: 20)" and the target variable "emotion (emotion A)," into the emotion estimation model 17 and obtains the output result of the emotion estimation model 17. The second training unit 33 then updates various parameters of the emotion estimation model 17 to minimize the error between the output result of the emotion estimation model 17 and the target variable "emotion (emotion A)."
[0057] The third training unit 34 is a processing unit that generates a crime risk estimation model 18 using machine learning with the third training data stored in the third training data DB 15. Specifically, the third training unit 35 generates a crime risk estimation model 18 that estimates the "probability of special fraud occurring" in response to the input of "emotional patterns," which are time-series changes in emotions, using supervised learning.
[0058] Figure 17 illustrates the machine learning process of the crime risk estimation model 18. As shown in Figure 17, the third training unit 34 inputs third training data, which includes the explanatory variable "emotion pattern (emotion A → emotion B → emotion A)" and the target variable "crime AA", into the crime risk estimation model 18 and obtains the output result of the crime risk estimation model 18. The third training unit 34 then updates various parameters of the crime risk estimation model 18 so as to minimize the error between the output result of the crime risk estimation model 18 and the target variable "crime AA".
[0059] The operation unit 40 comprises an acquisition unit 41, a characteristic estimation unit 42, an emotion estimation unit 43, and a risk estimation unit 44. It is a processing unit that detects the risk of special fraud from the content of a user's phone call using various machine learning models generated by the machine learning unit 30. In other words, the operation unit 40 estimates the risk of known and unknown special fraud occurring in real time.
[0060] The acquisition unit 41 is a processing unit that acquires sensing data from the user. Specifically, the acquisition unit 41 acquires various sensing data from a camera installed in the user's home 3, a millimeter-wave radar installed in the user's home 3, and a wearable device worn by the user.
[0061] For example, the acquisition unit 41 detects the start of a call by detecting telephone operations, such as a ringtone in the case of a landline or a call initiation operation via wireless communication in the case of a mobile phone, and acquires video data captured by the camera attached to the telephone. The acquisition unit 41 then uses known image analysis or machine learning models that estimate physical information from image data to acquire the user's physical information from the acquired video data.
[0062] Furthermore, when the acquisition unit 41 detects the start of a call, it continuously acquires physiological response information measured by the user's wearable device while the call is in progress. The acquisition unit 41 then associates the acquired physical information with the physiological response information, stores it in the storage unit 12, and outputs it to the characteristic estimation unit 42.
[0063] The characteristic estimation unit 42 is a processing unit that inputs the physical information and physiological response information acquired by the acquisition unit 41 into the characteristic estimation model 16 to estimate the user's psychological characteristics. Figure 18 is a diagram illustrating the estimation of psychological characteristics. As shown in Figure 18, the characteristic estimation unit 42 inputs the user's physical information acquired by analyzing the user's video data during a call, and the user's physiological response information during the call, into the trained characteristic estimation model 16.
[0064] The characteristic estimation unit 42 then estimates psychological characteristics based on the output results of the trained characteristic estimation model 16. For example, the characteristic estimation unit 42 identifies the scale with the highest estimated probability among the scales and estimated probabilities of suspicion included in the output results as the estimated result of suspicion, and identifies the scale with the highest estimated probability among the scales and estimated probabilities of anxiety characteristics included in the output results as the estimated result of anxiety characteristics. The characteristic estimation unit 42 then outputs the physical information and physiological response information acquired by the acquisition unit 41, along with the estimated psychological characteristics, to the emotion estimation unit 43.
[0065] The emotion estimation unit 43 is a processing unit that estimates the user's emotions by inputting the physical information and physiological response information acquired by the acquisition unit 41 and the psychological characteristics estimated by the characteristic estimation unit 42 into a trained emotion estimation model 17.
[0066] Figure 19 illustrates the estimation of emotions. As shown in Figure 19, when psychological characteristics are estimated using the characteristic estimation model 16, the emotion estimation unit 43 inputs the estimated psychological characteristics, along with the physical information and physiological response information used for estimation, into the emotion estimation model 17 to obtain the emotion estimation result. The emotion estimation unit 43 then identifies the emotion with an estimated probability above a threshold or the emotion with the highest estimated probability as the estimated result. The emotion estimation unit 43 stores the estimation results in chronological order in the memory unit 12, etc.
[0067] The risk estimation unit 44 is a processing unit that estimates the risk of special fraud occurring using the emotions estimated by the emotion estimation unit 43 and the crime risk estimation model 18. For example, the risk estimation unit 44 generates emotion patterns by combining the estimated emotions in a time series, inputs them into the trained crime risk estimation model 18, and obtains the estimated result of the risk of special fraud occurring.
[0068] For example, the risk estimation unit 44 estimates that a person has been subjected to a special fraud call when the person's emotions, as estimated by the emotion estimation unit 43, meet the condition that the person's emotions transition from a negative state to a positive state over a predetermined period of time.
[0069] Furthermore, the risk estimation unit 44 inputs estimated emotion patterns to the crime risk estimation model 18 to obtain the magnitude of the risk of occurrence for each of multiple criminal acts, and identifies a criminal act against a person from among the multiple criminal acts based on the magnitude of the risk of occurrence. For example, the risk estimation unit 44 identifies a crime with an estimated probability above a threshold or the crime with the highest estimated probability as the estimated result in the risk estimation results obtained from the crime risk estimation model 18. When a crime is detected, the risk estimation unit 44 outputs information about the estimated crime to the notification unit 50.
[0070] Here, we will explain the estimation of crime risk. Figure 20 is a diagram illustrating the estimation of crime risk. As shown in Figure 20, the operation unit 40 acquires (1) physical information and (2) physiological response information as external and internal information of the user during a phone conversation, and (3) psychological characteristics inherent to the person regardless of emotions and psychological state. The operation unit 40 associates each of the acquired information (1), (2), and (3) and inputs them into the emotion estimation model 17 to estimate the person's emotions in real time.
[0071] The operations unit 40 then generates emotion patterns using information about each emotion that is estimated sequentially from the start of the conversation. For example, the operations unit 40 generates emotion patterns such as "liveliness to vitality," "confusion to bewilderment," and "liveliness to vitality" in the order of the conversation time, inputs them into the crime risk estimation model 18, and obtains estimation results such as "Estimated likelihood of special fraud: ○%."
[0072] The above emotion patterns are generated by using emotion information estimated within a predetermined time interval, such as every 5 minutes. Figure 21 illustrates an example of emotion pattern identification. As shown in Figure 21, when a call is initiated at time T0, the operation unit 40 performs emotion estimation as needed, estimating emotion A at T1, emotion B at T2, emotion A at T3, emotion C at T4, and emotion B at T5.
[0073] In this case, the operations unit 40 generates an emotion pattern 1 containing emotion A, emotion B, and emotion A, using emotions B and emotion C estimated within a specified time, starting from T1 when emotion A was first estimated, and performs crime risk estimation. For example, the operations unit 40 inputs the emotion pattern "emotion A, emotion B, emotion A" into the crime risk estimation model 18 and estimates "no crime risk".
[0074] Next, the operations unit 40 generates emotion pattern 2, which includes emotion B, emotion A, and emotion C, using emotion A and emotion C estimated within a specified time, starting from T2, when emotion B was estimated after T1, and performs crime risk estimation. For example, the operations unit 40 inputs the emotion pattern "emotion B, emotion A, emotion C" into the crime risk estimation model 18 and estimates "no crime risk".
[0075] Next, the operations unit 40 generates an emotion pattern 3 containing emotion A, emotion C, and emotion B, using emotion C and emotion B estimated within a specified time, starting from T3, when emotion A was estimated after T2, and performs crime risk estimation. For example, the operations unit 40 inputs the emotion pattern "emotion A, emotion C, emotion B" into the crime risk estimation model 18 and estimates "crime risk present".
[0076] In this way, the operation unit 40 generates emotion patterns using emotions within predetermined intervals while the telephone conversation is ongoing, and repeatedly performs crime risk estimation. Note that the emotion pattern generation interval shown in Figure 21 is just an example and can be changed arbitrarily. For example, an emotion pattern can be generated using T, T2, and T3, and then an emotion pattern can be generated using T4, T5, and T6.
[0077] Returning to Figure 6, the notification unit 50 is a processing unit that notifies relevant parties when the operation unit 40 estimates a crime risk. For example, if the notification unit 50 estimates a "risk of crime occurrence" where the estimated probability is above a threshold, it will send a message such as "The person on the other end of the phone may be part of a fraud group" to a speaker installed in the user's home 3, or it will vibrate the user's mobile phone or wearable device to warn them. The notification unit 50 can also send an emergency message such as "Mr. / Ms. XX is on the phone and it may be a scam" to a pre-registered emergency contact. The notification unit 50 can also notify the police or local government.
[0078] (Machine learning processing flow) Figure 22 is a flowchart illustrating the flow of machine learning processing. As shown in Figure 22, when the machine learning unit 30 is instructed to start processing (S101: Yes), it administers a questionnaire about psychological characteristics to the subject to obtain psychological characteristics (S102).
[0079] Next, the machine learning unit 30 plays audio of a special fraud case to the subject to collect physiological response information (S103), and after the audio ends, it administers an emotional state questionnaire to the subject to obtain their emotions when they experienced the special fraud (S104). At this point, if the machine learning unit 30 decides to continue the experiment (S105: No), it repeats steps S102 onwards.
[0080] Meanwhile, when the experiment is completed (S105:Yes), the machine learning unit 30 uses the experimental results to generate first training data for the characteristic estimation model 16 (S106) and second training data for the emotion estimation model 17 (S107).
[0081] Subsequently, the machine learning unit 30 generates a characteristic estimation model 16 using the generated first training data (S108), generates an emotion estimation model 17 using the generated second training data (S109), and generates a crime risk estimation model 18 using the pre-prepared third training data (S110).
[0082] (Estimated processing flow) Figure 23 is a flowchart illustrating the process for estimating crime risk. As shown in Figure 23, when the operation unit 40 is instructed to start processing (S201: Yes), it acquires video data (S202), analyzes the video data, and obtains the physical characteristics of the people shown in the video (S203).
[0083] Next, the operations unit 40 acquires physiological response information from the wearable device of the person shown in the video data (S204). Then, the operations unit 40 inputs the physical information and physiological response information into the characteristic estimation model 16 to estimate the person's psychological characteristics (S205). Furthermore, the operations unit 40 inputs the physical information, physiological response information, and psychological characteristics into the emotion estimation model 17 to estimate the person's emotions (S206). Here, if a predetermined amount of time has not elapsed since the start of the call (S207: No), the operations unit 40 repeats steps S202 onwards.
[0084] Meanwhile, when a predetermined amount of time has elapsed since the start of the phone call (S207: Yes), the operations unit 40 generates an emotion pattern using the estimated emotion (S208). Subsequently, the operations unit 40 inputs the emotion pattern into the crime risk estimation model 18 to estimate the crime risk (S209).
[0085] Then, if the operation unit 40 detects a crime risk (S210:Yes), the notification unit 50 notifies the relevant parties of the crime risk (S211). Subsequently, if the operation unit 40 continues processing (S212:No), it executes S202 onwards, and if it terminates processing (S212:Yes), it terminates the estimation of the crime risk. If no crime risk to be reported is detected in S210 (S210:No), S211 is not executed and S212 is executed.
[0086] (effect) As described above, the information processing device 10 estimates psychological characteristics based on physical characteristics and physiological responses estimated from video data, etc. In emotion estimation, the information processing device 10 uses physiological response information and individual characteristic information (physical characteristic information and psychological characteristic information) as features. The information processing device 10 can minimize the impact of individual differences on estimation accuracy by using an emotion estimation model 17 that takes psychological characteristics into account, thereby improving estimation accuracy.
[0087] Furthermore, psychological trait information, which is part of personal trait information, requires the administration of psychological questionnaires, making its acquisition difficult and costly. Conventional methods have not yet enabled emotion estimation using both personal trait information and physiological response information.
[0088] On the other hand, by adding psychological characteristics, the information processing device 10 can improve the accuracy of emotion estimation based on physiological responses and achieve accurate fraud detection that focuses on changes in the victim's emotions.
[0089] The information processing device 10 can infer crimes from a person's emotional patterns, thereby preventing victims from becoming victims of both known and unknown types of special fraud. Furthermore, the information processing device 10 can generate machine learning models that perform accurate inferences using accurate training data, enabling faster estimation processing compared to using unnecessary training data.
[0090] The information processing device 10 estimates the risk of a criminal act against a person using a crime risk estimation model 18 that has been trained on the relationship between emotional patterns and crime. Therefore, even when a new crime occurs, the information processing device 10 can estimate the risk of the crime occurring from the emotional patterns of the person associated with the crime, and thus can effectively prevent unknown types of fraud.
[0091] The information processing device 10 generates emotional patterns derived from past performance and criminal psychology for each specific crime and trains the crime risk estimation model 18 with them. Therefore, the information processing device 10 can train not only on past performance but also on unknown emotional patterns during crimes that can be inferred from past performance. [Examples]
[0092] Now, although embodiments of the present invention have been described, the present invention may be implemented in various other forms besides those described above.
[0093] (Numerical values, etc.) The example datasets, training data, sensors, number of data points, time intervals, examples and number of emotions, number and types of special frauds, emotion patterns, and combinations of special frauds and emotion patterns used in the above embodiment are merely examples and can be changed as needed. Furthermore, the processing flow described in each flowchart can also be modified as appropriate within a consistent range.
[0094] (Model form) In the above embodiment, an example was described in which a multi-class decision model (multi-class classification model) was used as each machine learning model, but it is not limited to this. For example, the characteristic estimation model 16 could also use a binary classification model that determines whether or not physiological responses and psychological states are linked.
[0095] (Estimation of emotions) In the above embodiment, an example of estimating a person's emotions using the emotion estimation model 17 was described, but the invention is not limited to this. For example, the information processing device 10 can estimate emotions using psychological characteristics by using rules that associate psychological characteristics with emotions, or rules that associate changes in psychological characteristics with emotions.
[0096] (Examples of using emotions) The above embodiment describes an example of using highly accurate estimated emotions to detect special fraud, but it is not limited to this. For example, estimated emotions can be used to detect suspicious individuals in stores, or to detect employee stress in the workplace.
[0097] (An example of a special type of fraud) In the above embodiment, an example was described in which the crime risk estimation model 18 is used to estimate the crime risk of special fraud from emotional patterns, but the invention is not limited to this. For example, the information processing device 10 can maintain judgment rules that associate emotional patterns with special fraud, and use these judgment rules to estimate special fraud that corresponds to a specified emotional pattern.
[0098] (Example of electronic equipment) In the above embodiment, the example of a phone call at the user's home 3 was used, but the invention is not limited to this. For example, the information processing device 10 can provide a similar service to users of ATMs (Automatic Teller Machines) installed in stores such as banks and convenience stores. In this case, the information processing device 10 acquires sensing data from cameras and microphones installed around the ATM to estimate the risk of crime, and if a risk of crime is detected, it makes an announcement within the store or notifies the person in charge.
[0099] (system) Unless otherwise specified, the processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed at will.
[0100] Furthermore, each component of the illustrated device is a functional concept and does not necessarily have to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown. That is, all or part of it can be functionally or physically distributed and integrated in any unit depending on various loads and usage conditions. For example, the machine learning unit 30 and the operation unit 40 can be implemented in separate computers (enclosures). In other words, it can be implemented with an information processing device that performs the same functions as the machine learning unit 30 and an information processing device that performs the same functions as the operation unit 40.
[0101] Furthermore, each processing function performed by each device may be implemented, in whole or in part, by a CPU and a program executed for analysis by that CPU, or by hardware using wired logic.
[0102] (Hardware) Figure 24 illustrates an example of a hardware configuration. As shown in Figure 24, the information processing device 10 includes a communication device 10a, an HDD (Hard Disk Drive) 10b, memory 10c, and a processor 10d. Furthermore, each component shown in Figure 24 is interconnected by a bus or the like.
[0103] The communication device 10a is a network interface card or the like, and communicates with other devices. The HDD 10b stores programs and databases that operate the functions shown in Figure 6.
[0104] The processor 10d reads a program from the HDD 10b or the like that performs the same processing as each processing unit shown in Figure 6, and loads it into memory 10c, thereby operating a process that performs each of the functions described in Figure 6. For example, this process performs the same functions as each processing unit of the information processing device 10. Specifically, the processor 10d reads a program from the HDD 10b or the like that has the same functions as the machine learning unit 30, the operation unit 40, the notification unit 50, etc. Then, the processor 10d executes a process that performs the same processing as the machine learning unit 30, the operation unit 40, the notification unit 50, etc.
[0105] Thus, the information processing device 10 operates as an information processing device that executes an information processing method by reading and executing a program. Furthermore, the information processing device 10 can also achieve the same functionality as the embodiment described above by reading the program from the recording medium using a media reader and executing the read program. Note that the program referred to in this other embodiment is not limited to being executed by the information processing device 10. For example, the above embodiment may also be applied to cases where another computer or server executes the program, or where these computers or servers collaborate to execute the program.
[0106] This program may be distributed via a network such as the Internet. Alternatively, this program may be recorded on a computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO (Magneto-Optical disk), or DVD (Digital Versatile Disc), and executed by being read from the recording medium by a computer. [Explanation of symbols]
[0107] 10 Information Processing Devices 11 Communications Department 12 Storage section 13. First Training Data Database 14. Second Training Data Database 15. Third Training Data Database 16. Characteristic Estimation Models 17. Emotion Estimation Models 18. Crime Risk Estimation Models 20 Control Unit 30 Machine Learning Department 31 Data Generation Unit 32 1st Training Department 33 2nd Training Department 34 3rd Training Department 40 Operations Department 41 Acquisition Department 42 Characteristic Estimation Unit 43 Emotion estimation part 44 Risk Estimation Department 50 Hochi Department
Claims
1. On the computer, By obtaining physiological response information of a person identified from their vital data, Physical information of the person identified from video data of the person is obtained. By inputting the acquired physiological response information and physical information into a first machine learning model, psychological characteristics that represent the individual's unique personality are generated. Based on the generated psychological characteristics, the emotions of the person are estimated. Execute the process, The aforementioned estimation process is, An emotion estimation program characterized by estimating a person's emotions by inputting the aforementioned psychological characteristics, physiological response information, and physical information into a second machine learning model.
2. The emotion estimation program according to claim 1, characterized in that the aforementioned psychological characteristics are characteristics inherent to the person, regardless of their emotions and psychological state.
3. The aforementioned acquisition process is, Based on the vital data of a person using an electronic device, physiological response information of the person is generated. The aforementioned acquisition process is, Physical information of a person identified from video data of a person using the aforementioned electronic device is obtained. The aforementioned generation process is, By inputting the physiological response information and physical information of the person into the first machine learning model, the psychological characteristics are generated. The aforementioned estimation process is, Based on the patterns of emotional changes of the person estimated based on the aforementioned psychological characteristics, the presence or absence of special fraud occurring among the person using the electronic device is estimated. The emotion estimation program according to feature 1.
4. Computers By obtaining physiological response information of a person identified from their vital data, Physical information of the person identified from video data of the person is obtained. By inputting the acquired physiological response information and physical information into a first machine learning model, psychological characteristics that represent the individual's unique personality are generated. Based on the generated psychological characteristics, the emotions of the person are estimated. Execute the process, The aforementioned estimation process is, An emotion estimation method characterized by estimating the emotions of a person by inputting the aforementioned psychological characteristics, the aforementioned physiological response information, and the aforementioned physical information into a second machine learning model.
5. By obtaining physiological response information of a person identified from their vital data, Physical information of the person identified from video data of the person is obtained. By inputting the acquired physiological response information and physical information into a first machine learning model, psychological characteristics that represent the individual's unique personality are generated. Based on the generated psychological characteristics, the emotions of the person are estimated. It has a control unit, The control unit, An information processing device characterized by estimating the emotions of a person by inputting the aforementioned psychological characteristics, the aforementioned physiological response information, and the aforementioned physical information into a second machine learning model.