Portable information terminal equipment

Abstract

To provide an information processor capable of selecting an appropriate operation according to user's feelings and executing the operation, and an information processing method.SOLUTION: An information processing method includes: detecting the user's face and extracting the feature of the user's face from information on the detected face; estimating the user's feelings from the extracted face feature; generating information according to the estimated user's feelings; determining whether to send to an external device, information generated according to positional information of the user and an external device including an information reception part, the positional information included in a radio wave emitted from a global positioning system; and sending the generated information to the external device if it is determined to send the same. The external device that received the information selects and executes the operation based on the received information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] One aspect of the present invention relates to an information processing device and an information processing method.

[0002] Note that one embodiment of the present invention is not limited to the above technical field. Examples of technical fields of one embodiment of the present invention disclosed in this specification and the like include semiconductor devices, display devices, light-emitting devices, power storage devices, memory devices, electronic devices, lighting devices, input devices, input / output devices, driving methods thereof, and manufacturing methods thereof. Note that in this specification and the like, a semiconductor device refers to any device that can function by utilizing semiconductor characteristics. [Background technology]

[0003] There is known a technique for recognizing facial expressions from captured images of a face. For example, facial expression recognition is applied to a technique for automatically capturing an image of a person when they smile or look toward the camera in a digital camera.

[0004] As a technique for facial expression recognition, for example, Patent Document 1 discloses a technique for detecting facial feature points and recognizing facial expressions with high accuracy based on the feature points. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2007-87346 A Summary of the Invention [Problem to be solved by the invention]

[0006] A person's actions (behavior, actions) reflect their emotions at any given time to a certain extent, and even if the person intends to perform the same action, subtle differences often appear. For example, when driving a car, a driver can drive calmly and safely when he or she is in a normal state of mind, but if the driver is feeling intense anger, he or she may lose control and drive recklessly, such as steering the wheel or pressing the accelerator pedal. Also, if the driver is feeling deep sadness, his or her judgment ability may become impaired, and there may be a slight delay in the timing of various driving operations. Also, if the driver encounters some unexpected situation while driving and feels a great sense of surprise, he or she may lose his or her ability to make calm judgments and make an unusual mistake, such as accidentally stepping on the accelerator instead of the brake.

[0007] In driving a car, even a slight operational error can directly affect a person's life. Therefore, drivers are required to always have high concentration, calm judgment, and perform stable driving operations without being influenced by their emotions at the time. However, even if the driving operations caused by the driver's emotional differences as described above are obvious to others, the driver himself is often unaware of them, making it difficult for him to perform stable driving operations by his own will alone.

[0008] Although driving a car is given as an example here, actions that may be influenced by human emotions are not limited to this. For example, operating heavy machinery or equipment on a factory production line are also actions that may be influenced by human emotions.

[0009] An object of one embodiment of the present invention is to provide a device or a method capable of preventing an abnormal operation caused by a person's emotion, or to provide a device or a method capable of selecting and executing an appropriate operation in accordance with a person's emotion.

[0010] Another object of one embodiment of the present invention is to provide a novel information processing device. Another object of one embodiment of the present invention is to provide a novel information processing method.

[0011] Note that the description of these problems does not preclude the existence of other problems. Note that one embodiment of the present invention does not necessarily solve all of these problems. Note that problems other than these can be extracted from the description of the specification, drawings, claims, etc. [Means for solving the problem]

[0012] One aspect of the present invention is an information processing device having a subject detection unit that detects a user's face, a feature extraction unit that extracts facial features, an emotion estimation unit that estimates the user's emotions from the features, an information generation unit that generates first information corresponding to the estimated emotion, a sensor unit that receives radio waves from a global positioning system, an information processing unit that receives the first information and second information contained in the radio waves transmitted from the sensor unit, and generates third information corresponding to the first information and the second information, and an information transmission unit that transmits the third information.

[0013] In the above, it is preferable that the third information is transmitted to an external device having an information receiving unit and whose position is specified by the global positioning system.

[0014] In the above, the features preferably include at least one of the user's eye shape, eyebrow shape, mouth shape, line of sight, and facial color.

[0015] In the above, the extraction of features is preferably performed by inference using a neural network.

[0016] In the above, the emotion preferably includes at least one of anger, sadness, pain, impatience, anxiety, dissatisfaction, fear, surprise, and emptiness.

[0017] In the above, it is preferable that the emotion estimation is performed by inference using a neural network.

[0018] In the above, it is preferable that the second information includes a distance between the user and the external device.

[0019] In the above, it is preferable that the third information includes the first information.

[0020] In the above, it is preferable that the external device includes either a car or a building.

[0021] Another aspect of the present invention is an information processing method having the steps of detecting a user's face, extracting facial features from information of the detected face, estimating the user's emotions from the features, generating first information corresponding to the emotion, generating second information corresponding to the first information based on the first information, determining whether to transmit the second information to the outside based on third information contained in radio waves from a global positioning system, and transmitting the second information to the outside or not transmitting the second information to the outside depending on the determination.

[0022] In the above method, it is preferable that the method further includes, after the determining step, a step of transmitting the second information to an external device having an information receiving unit and whose position is specified by the global positioning system.

[0023] In the above, the features preferably include at least one of the user's eye shape, eyebrow shape, mouth shape, line of sight, and facial color.

[0024] In the above, the extraction of features is preferably performed by inference using a neural network.

[0025] In the above, the emotion preferably includes at least one of anger, sadness, pain, impatience, anxiety, dissatisfaction, fear, surprise, and emptiness.

[0026] In the above, it is preferable that the emotion estimation is performed by inference using a neural network.

[0027] In the above, it is preferable that the third information includes a distance between the user and the external device.

[0028] In the above, it is preferable that the second information includes the first information.

[0029] In the above, it is preferable that the external device includes either a car or a building. Effect of the Invention

[0030] According to one aspect of the present invention, it is possible to provide a device or method capable of preventing an abnormal operation caused by a person's emotion, or to provide a device or method capable of selecting and executing an appropriate operation according to a person's emotion.

[0031] According to one embodiment of the present invention, a novel information processing device or a novel information processing method can be provided.

[0032] Note that the description of these effects does not preclude the existence of other effects. Note that one embodiment of the present invention does not necessarily have all of these effects. Note that effects other than these can be extracted from the description in the specification, drawings, claims, etc. [Brief description of the drawings]

[0033] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an information processing device according to an embodiment of the present invention. [Diagram 2]2A and 2B are diagrams illustrating a neural network used in an information processing device according to an embodiment of the present invention, and FIG. 2C is a diagram showing an example of an output result of the neural network used in the information processing device according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a flowchart illustrating an example of an information processing method according to an embodiment of the present invention. [Figure 4] FIG. 4 is a block diagram illustrating an example of the configuration of an information processing device according to an embodiment of the present invention. [Diagram 5] 5A to 5F are diagrams illustrating examples of electronic devices to which one embodiment of the present invention can be applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] Hereinafter, the embodiments will be described with reference to the drawings. However, it will be easily understood by those skilled in the art that the embodiments can be implemented in many different ways, and that the modes and details can be changed in various ways without departing from the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited to the description of the following embodiments.

[0035] In addition, in each figure described in this specification, the size or area of ​​each component may be exaggerated for clarity, and therefore, the present invention is not necessarily limited to the scale.

[0036] In addition, ordinal numbers such as "first" and "second" in this specification are used to avoid confusion of components and do not limit the numbers.

[0037] (Embodiment 1) In one aspect of the present invention, a user's face is detected, and facial features of the user are extracted from information on the detected face. Then, the user's emotions are estimated from the extracted features, and information corresponding to the estimated emotions is generated. The information is received by an external device having an information receiving unit. In another aspect of the present invention, radio waves transmitted from a Global Positioning System (GPS) are received. The radio waves include location information of the user and the external device. Information generated by estimating the user's emotions is transmitted according to the location information, and is received by the external device. This makes it possible to prevent unusual or erroneous operations from being performed due to the user's emotions. Also, an appropriate operation can be selected and executed according to the user's emotions.

[0038] For example, suppose that an information processing device according to an embodiment of the present invention is a mobile information terminal device such as a mobile phone (including a smartphone) or a tablet terminal, and an external device that receives information transmitted from the mobile information terminal device is a car having an information receiving unit. Then, consider a case where a user brings the mobile information terminal device into the car and drives with the device set and installed in a state where the user's face can be detected at all times.

[0039] Facial information includes a great many features, ranging from global information such as facial contours and complexion, to the shapes and relative positions of facial parts such as the eyes, eyebrows, nose, and mouth, as well as local information such as the degree to which the eyes are open, the degree to which the nostrils are flared, the degree to which the mouth is open (closed), the angle of the eyebrows, wrinkles between the eyebrows, and the position of the gaze. Furthermore, since a person's facial expression is formed by a combination of these various features, by continuously detecting the user's face, a mobile information terminal device can detect changes in facial expression that reflect the user's emotions while driving. For this reason, it is desirable for the mobile information terminal device to detect as many facial expressions as possible.

[0040] Furthermore, there is a certain degree of correlation between human emotions and facial expressions, although there are individual differences and differences in the degree of correlation. For example, when a person is feeling intense anger, characteristics such as the corners of the eyes and eyebrows rising, the gaze becoming stricter, and the face becoming flushed may be displayed when compared to normal. For example, when a person is feeling deep sadness, characteristics such as the corners of the eyes and eyebrows tending to droop, the gaze becoming unfocused, the eyes looking down, and the face becoming paler may be displayed when compared to normal. For example, when a person is feeling great surprise, characteristics such as the eyes opening wider than normal, the mouth opening wide, and staring at one point may be displayed when compared to normal.

[0041] Therefore, an information processing device according to one aspect of the present invention can detect information about a user's face and read the user's facial expression from that information, i.e., extract facial features, thereby making it possible to estimate the user's emotions at any given time.

[0042] As mentioned above, the correlation between human emotions and facial expressions varies from person to person and varies in degree, and is not necessarily limited to the above. Also, characteristics other than those mentioned above may appear.

[0043] In addition, although anger, sadness, and surprise are given as examples of human emotions in the above, the emotions that the information processing device according to one embodiment of the present invention can estimate are not limited to these emotions. In addition to anger, sadness, and surprise, the emotions that the information processing device according to one embodiment of the present invention can estimate include pain, impatience, anxiety, dissatisfaction, fear, emptiness, and the like.

[0044] For example, if a user is driving a car with strong anger, a mobile information terminal device installed in the car extracts features from the detected facial information of the user, such as raised eye corners or eyebrows, a sharp gaze, a flushed face, etc. From these features, it can be inferred that the user is angry. As will be described later, the extraction of facial features and estimation of emotions of the user can be performed by inference using a neural network.

[0045] If the estimated emotion of the user is intense anger, the user's continued driving may lead to driving errors or accidents, so in such a case, it is preferable for the user to temporarily stop driving.

[0046] In one aspect of the present invention, when a mobile information terminal device installed in a vehicle estimates that the user is feeling angry, for example, as described above, the mobile information terminal device generates information including appropriate actions that the user should take in response to this (for example, slowing down the vehicle's traveling speed, stopping the vehicle, etc.), and transmits this information to an information receiving unit in the vehicle. When the information receiving unit receives the above-mentioned information, measures are taken, such as controlling the vehicle's speed so that it does not exceed a certain speed when the vehicle is traveling, or preventing the engine from starting if the user has just gotten in. This makes it possible to prevent the user from behaving differently from normal even if the user is feeling different from normal. It is also possible to select and execute appropriate actions according to the user's emotions.

[0047] In addition, if the user makes a mistake in driving the car, for example, if the user mistakenly starts the car backwards while trying to move it forward, the mobile information terminal device installed in the car extracts features from the detected facial information of the user, such as eyes wide open, mouth wide open, staring at one point, etc. From these features, it can be estimated that the user is feeling surprised.

[0048] In one aspect of the present invention, when a mobile information terminal device installed in a vehicle estimates that the user is feeling surprised, as described above, the mobile information terminal device generates information including an appropriate action to be taken by the user in response (e.g., immediately stopping, etc.) and transmits the information to an information receiving unit in the vehicle. When the information receiving unit receives the information, measures are taken, such as automatically stopping the vehicle. This makes it possible to prevent the user from performing an unusual action even if a sudden change in emotion occurs in the user. Also, an appropriate action can be selected and executed according to the user's emotion.

[0049] Moreover, an information processing device according to an embodiment of the present invention receives radio waves transmitted from a global positioning system. In the above example, the radio waves include location information of the mobile information terminal device and the car identified by the global positioning system (i.e., information indicating the distance between the user and the car, etc.). In one embodiment of the present invention, the mobile information terminal device estimates the user's emotions in response to the location information and transmits the generated information (e.g., stop the car, do not start the engine, etc.) to an information receiving unit of the car.

[0050] For example, when it is found that a user is in a car in which a mobile information terminal device is installed, based on location information contained in radio waves transmitted from a global positioning system, it is estimated that the user is about to drive the car or is currently driving the car. In this case, depending on the user's emotions, starting or continuing driving may be unsafe, so information generated by the mobile information terminal device estimating the user's emotions is transmitted to an information receiving unit in the car. Then, the car is controlled in accordance with the information (for example, not to start the engine, not to allow the car to travel faster than a certain speed, etc.).

[0051] On the other hand, for example, if a user is carrying a mobile information terminal device and is indoors, such as at home, the user will not cause an accident while driving a car having an information receiving unit. Therefore, if it is determined that there is a certain distance between the user and the car having the information receiving unit based on the location information contained in the radio waves transmitted from the global positioning system, the information generated by the mobile information terminal device by estimating the user's emotions will not be transmitted to the information receiving unit of the car.

[0052] As described above, an information processing device according to an embodiment of the present invention estimates an emotion from a user's facial expression and creates information corresponding to the estimated emotion. The information is then transmitted to an external device having an information receiving unit whose position is identified by a global positioning system. However, if the global positioning system determines that there is a certain distance or more between the user and the external device, the information is not transmitted. If the external device receives the information, the external device executes an operation according to the information.

[0053] In this way, the information processing device according to one embodiment of the present invention can prevent an unusual operation caused by the user's emotions by operating in cooperation with an external device having an information receiving unit and a global positioning system. Alternatively, an appropriate operation can be selected and executed according to the user's emotions. For example, when the information processing device according to one embodiment of the present invention is a mobile information terminal device and the external device having an information receiving unit is a car driven by the user, the user can always drive safely (for example, drive at an appropriate speed, stop temporarily, etc.) even if the user has an emotion different from normal (for example, anger, sadness, surprise, etc.).

[0054] In the above, a car is given as an example of an external device having an information receiving unit, but is not limited thereto. In one aspect of the present invention, the external device having an information receiving unit may be a building. Specific examples of buildings include stores in commercial facilities such as convenience stores, supermarkets, and department stores, buildings in public facilities such as banks, schools, and hospitals, residential buildings such as houses, apartments, and condominiums, and office buildings such as buildings.

[0055] Hereinafter, a more specific example of one embodiment of the present invention will be described with reference to the drawings and a flowchart.

[0056] <Configuration example of information processing device> 1 is a block diagram showing an example of a configuration of an information processing device 10 according to an embodiment of the present invention. The information processing device 10 includes, as an example, a subject detection unit 11, a feature extraction unit 12, a feeling estimation unit 13, an information generation unit 14, a sensor unit 15, an information processing unit 16, and an information transmission unit 17.

[0057] As described above, the information processing device 10 can operate in cooperation with an external device 28 having an information receiving unit 18 and a global positioning system 29.

[0058] In the drawings attached to this specification, the components are classified by function and shown in block diagrams as independent blocks. However, in reality, it is difficult to completely separate the components by function, and one component may be involved in multiple functions, or one function may be realized by multiple components.

[0059] [Subject detection unit 11] The subject detection unit 11 has a function of acquiring information about a part or the whole of the user's face, and outputting the information to the feature extraction unit 12 .

[0060] As the subject detection unit 11, typically, an imaging device equipped with an image sensor can be used. In that case, an infrared imaging device that irradiates infrared rays onto the user's face to capture an image may be used. Note that the subject detection unit 11 is not limited to an imaging device as long as it is a device that can detect the state of a part or the whole of the subject's face. An optical distance measuring device that measures the distance between a device and a part of the face using infrared rays or the like may also be used. Also, a detection device that electrically detects the movement of the user's facial muscles by contacting electrodes with the user's face may be used.

[0061] [Feature extraction unit 12] The feature extraction unit 12 has a function of extracting feature points from the face information output from the subject detection unit 11, extracting some or all of the facial features from the positions of the feature points, and outputting the extracted feature information to the emotion estimation unit 13.

[0062] When the face information acquired by the subject detection unit 11 is information on the eyes and their surroundings, the features extracted by the feature extraction unit 12 include, for example, the pupil, iris, cornea, conjunctiva (white of the eye), inner corner of the eye, outer corner of the eye, upper eyelid, lower eyelid, eyelashes, eyebrows, between the eyebrows, inner corner of the eyebrow, and inner corner of the eyebrow. In addition, features other than the eyes and their surroundings include the root of the nose, the tip of the nose, the bridge of the nose, the nostrils, the lips (upper lip, lower lip), corners of the mouth, cleft palate, teeth, cheeks, chin, gills, and forehead. The feature extraction unit 12 recognizes the shapes and positions of these facial parts and extracts the position coordinates of the feature points in each part. Then, the extracted position coordinate data can be output to the emotion estimation unit 13 as information on the facial features.

[0063] In one embodiment of the present invention, it is preferable that the feature extraction unit 12 extracts at least one of the following features from the facial information acquired by the subject detection unit 11: eye shape, eyebrow shape, mouth shape, gaze, and facial color.

[0064] As a method of feature extraction by the feature extraction unit 12, various algorithms can be applied for extracting feature points from the image etc. acquired by the subject detection unit 11. For example, algorithms such as SIFT (Scale Invariant Feature Transform), SURF (Speed ​​Up Robust Features), and HOG (Histograms of Oriented Gradients) can be used.

[0065] In one embodiment of the present invention, the feature extraction by the feature extraction unit 12 is preferably performed by inference of a neural network. In particular, it is preferable to use a convolutional neural network (CNN). The following describes the case where a neural network is used.

[0066] 2A shows a schematic diagram of a neural network NN1 that can be used in the feature extraction unit 12. The neural network NN1 has an input layer 51, three intermediate layers 52, and an output layer 53. Note that the number of intermediate layers 52 is not limited to three, and may be one or more.

[0067] Data 61 is input to the neural network NN1 from the subject detection unit 11. The data 61 is data including coordinates and values ​​corresponding to the coordinates. Typically, the data 61 may be image data including coordinates and gradation values ​​corresponding to the coordinates. Data 62 is output from the neural network NN1. The data 62 is data including the position coordinates of the above-mentioned feature points.

[0068] The neural network NN1 is trained in advance to extract the above-mentioned feature points from data 61 such as image data and output their coordinates. The neural network NN1 is trained to perform edge processing using various filters in the intermediate layer 52 so that the neuron values ​​in the output layer 53 corresponding to the coordinates where the above-mentioned feature points exist become high.

[0069] [Emotion estimation unit 13] The emotion estimation unit 13 has a function of estimating the emotion of the user from the facial feature information input from the feature extraction unit 12, and outputting the estimated emotion information to the information generation unit 14.

[0070] The emotion estimation unit 13 can estimate whether the user has a negative emotion (e.g., anger, sadness, pain, impatience, fear, anxiety, dissatisfaction, surprise, irritation, resentment, excitement, emptiness, etc.) by using the information on the user's facial features. In addition, if the user has a negative emotion, it is preferable to estimate the degree (level) of the emotion.

[0071] In one aspect of the present invention, it is preferable that the emotion estimation unit 13 be able to estimate at least one of anger, sadness, suffering, impatience, anxiety, dissatisfaction, fear, surprise, and emptiness.

[0072] In addition, in one aspect of the present invention, it is preferable that the emotion estimation in the emotion estimation unit 13 is performed by inference using a neural network. In particular, it is preferable that the emotion estimation is performed using a CNN.

[0073] 2B is a schematic diagram of a neural network NN2 that can be used in the emotion estimation unit 13. Here, an example is shown in which the neural network NN2 has a configuration generally similar to that of the neural network NN1. Note that the number of neurons in the input layer 51 of the neural network NN2 can be smaller than that of the neural network NN1.

[0074] The neural network NN2 receives data 62 from the feature extraction unit 12. The data 62 includes information related to the coordinates of the extracted feature points.

[0075] Moreover, data obtained by processing the data 62 may be used as data to be input to the neural network NN2. For example, a vector connecting any two feature points may be calculated, and the vector may be calculated for all or some of the feature points, and may be used as data to be input to the neural network NN2. The calculated vector may also be normalized as data. Note that hereinafter, data processed based on the data 62 output by the neural network NN1 will also be referred to as data 62.

[0076] Data 63 is output from the neural network NN2 to which the data 62 is input. The data 63 corresponds to a neuron value output from each neuron in the output layer 53. Each neuron in the output layer 53 is associated with one emotion. As shown in FIG. 2B, the data 63 is data including neuron values ​​of neurons corresponding to predetermined negative emotions (anger, sadness, suffering, impatience, fear, etc.).

[0077] The neural network NN2 is trained in advance to estimate the degree of negative emotion from the data 62 and output it as a neuron value. Since the relative positions of multiple feature points on the user's face can determine the user's facial expression, the neural network NN2 can estimate the emotion the user is feeling from that facial expression.

[0078] FIG. 2C is a diagram showing data 63. The height of the neuron value corresponding to each emotion indicates the degree of the estimated emotion. The data 63 also shows a threshold value T by a dashed line. For example, when the height of the neuron value corresponding to each emotion is below the threshold value T, it can be determined that the user does not feel that emotion or that the degree of that emotion is low. Conversely, when the height of the neuron value corresponding to each emotion is above the threshold value T, it can be determined that the degree of that emotion is high.

[0079] For example, from Figure 2C, it can be inferred that the emotion is a mixture of "anger" and "impatience," and that the "anger" is felt particularly strongly.

[0080] In FIG. 2C, only one threshold value is set to distinguish the high and low degree of each emotion, but multiple threshold values ​​may be set according to the magnitude of the neuron value. For example, a threshold value T1 may be set for any neuron value smaller than the threshold value T, and a threshold value T2 may be set for any neuron value larger than the threshold value T. This allows for more detailed discrimination of the high and low degree of each emotion, for example, when the height of the neuron value corresponding to each emotion is below the threshold value T1, it is determined that the emotion is low (calm), when it is between the threshold values ​​T1 and T2, it is determined that the emotion is somewhat high, and when it is above the threshold value T2, it is determined that the emotion is very high.

[0081] In this way, by configuring the emotion estimation unit 13 to estimate only negative emotions and output the result to the information generation unit 14, the scale of calculation in the emotion estimation unit 13 can be reduced, and power consumption related to the calculation can be reduced. In addition, since the amount of data used in the information generation unit 14 can be reduced, the transmission of data from the emotion estimation unit 13 to the information generation unit 14 and the power consumption related to the calculation in the information generation unit 14 can also be reduced. Note that the emotion estimation unit 13 can estimate not only negative emotions but also emotions opposite thereto, such as joy, gratitude, happiness, familiarity, satisfaction, and love, and output the result to the information generation unit 14.

[0082] Incidentally, emotion estimation can also be performed without using a neural network. For example, an image of a part of the user's face acquired by the subject detection unit 11 can be compared with a template image, and the estimation can be performed by a template matching method or a pattern matching method that uses the similarity between the image and a template image. In that case, the feature extraction unit 12 may not be included in the configuration.

[0083] [Information generation unit 14] The information generation unit 14 has a function of determining or generating information (first information) according to the emotion estimated by the emotion estimation unit 13, and outputting the information to the information processing unit 16.

[0084] The first information is information that is the basis of information that is finally transmitted to an external device 28 having an information receiving unit 18 via an information processing unit 16 and an information transmitting unit 17 described later. For example, consider a case where the external device 28 is a car driven by a user. If the emotion estimation unit 13 estimates that the user has a negative emotion that exceeds the above-mentioned threshold T, the information generation unit 14 that receives the estimation result determines or generates an appropriate action according to the user's emotion, such as "prevent the car from exceeding a certain speed," "slow down the car," or "stop the car," as the first information, and outputs it to the information processing unit 16. Also, if the emotion estimation unit 13 estimates that the user's negative emotion is below the above-mentioned threshold T, the information generation unit 14 that receives the estimation result determines or generates an action that the user should take, such as "continue driving as is," as the first information, and outputs it to the information processing unit 16.

[0085] The first information may be linked in advance to the emotion estimated by the emotion estimation unit 13. For example, information linking the emotion "anger" with the action "preventing the car from exceeding a certain speed" may be created in advance and registered in the information generation unit 14. In this way, when the emotion estimation unit 13 estimates the emotion "anger", the information generation unit 14 can output the action "preventing the car from exceeding a certain speed" as the first information. In this way, by creating a large amount of information (data set) in advance linking each emotion with an appropriate action corresponding to each emotion, the information generation unit 14 can instantly output appropriate first information even if a sudden change occurs in the user's emotion.

[0086] [Sensor unit 15] The sensor unit 15 has a function of receiving radio waves transmitted from the global positioning system 29 and outputting information included in the radio waves (second information) to the information processing unit 16.

[0087] The radio waves received by the sensor unit 15 from the global positioning system 29 contain position information of the information processing device 10 and an external device 28 having an information receiving unit 18 described below. The second information is information that contains at least the distance between the user and the external device 28 among the above-mentioned position information. The sensor unit 15 extracts the second information from the above-mentioned position information contained in the radio waves transmitted from the global positioning system 29, and outputs it to the information processing unit 16.

[0088] [Information Processing Section 16] The information processing unit 16 has the function of receiving the first information output from the information generation unit 14 and the second information output from the sensor unit 15, determining or generating information (third information) according to the received content, and outputting this to the information transmission unit 17.

[0089] The third information is final information output from the information processing device 10 via the information transmission unit 17 described later, and includes the first information. The information processing unit 16 determines or generates the third information based on the first information input from the information generation unit 14. The third information includes all or at least a part of the first information.

[0090] Furthermore, the information processing unit 16 judges whether or not to output the third information to the information transmission unit 17 based on the second information input from the sensor unit 15. For example, consider a case where the information processing device 10 is a mobile information terminal device carried by the user, and the external device 28 is a car owned by the user. If the user is carrying the mobile information terminal device and is indoors, such as at home, the information processing unit 16 receives the second information from the sensor unit 15, which indicates that there is a certain distance or more between the user and the car (i.e., the user is not in the car).

[0091] In the above case, no matter what emotions the user may have, the user will not cause an accident while driving the car. Therefore, the mobile terminal device does not necessarily need to transmit the third information to the car. In such a case, the information processing unit 16 selects the decision not to output the third information to the information transmitting unit 17.

[0092] In this way, the information processing unit 16 generates or determines the third information based on the first information input from the information generating unit 14, and determines whether or not to output the generated or determined third information to the information transmitting unit 17 based on the second information input from the sensor unit 15. Then, when the output determination is selected, the third information is output to the information transmitting unit 17.

[0093] The information processing unit 16 may have two calculation units. For example, one calculation unit may determine or generate the third information based on the first information input from the information generating unit 14, and the other calculation unit may determine whether or not to output the third information to the information transmitting unit 17 based on the second information input from the sensor unit 15, and output the third information to the information transmitting unit 17 when the output determination is selected.

[0094] [Information Dissemination Section 17] The information transmitting unit 17 has a function of transmitting the third information input from the information processing unit 16 to an external device 28 having an information receiving unit 18.

[0095] When the information transmitting unit 17 receives the third information from the information processing unit 16, the information transmitting unit 17 transmits the third information to an external device 28 having an information receiving unit 18, the position of which has been specified by a global positioning system 29. In one embodiment of the present invention, the external device 28 having an information receiving unit 18 that receives the third information includes either a car or a building.

[0096] The above is a description of a configuration example of the information processing device 10 according to one embodiment of the present invention. By applying this configuration example, it is possible to provide an information processing device that can prevent an unusual operation caused by a user's emotion. Alternatively, it is possible to provide an information processing device that can select and execute an appropriate operation according to the user's emotion.

[0097] <Examples of information processing methods> 3 is a flowchart showing an example of an information processing method according to an embodiment of the present invention. A series of processes according to the flowchart can be performed by the information processing device 10 according to the embodiment of the present invention described above.

[0098] First, in step S1, a process for detecting a part or the whole of a user's face is performed. In the information processing device 10, the subject detection unit 11 can perform this process.

[0099] Next, in step S2, a process is performed to extract some or all of the facial features from the information on the user's face detected in step S1. In one aspect of the present invention, it is preferable to extract at least one of the following features from the detected information on the user's face: eye shape, eyebrow shape, mouth shape, gaze, and facial color. It is also preferable that the extraction of features is performed by inference using a neural network. In the information processing device 10, the feature extraction unit 12 can perform this process.

[0100] Next, in step S3, a process of estimating the user's emotion is performed based on the facial features of the user extracted in step S2. In one aspect of the present invention, it is preferable that at least one of anger, sadness, pain, impatience, anxiety, dissatisfaction, fear, surprise, and emptiness can be estimated as the user's emotion. It is also preferable that the emotion estimation is performed by inference using a neural network. In the information processing device 10, the emotion estimation unit 13 can perform this process.

[0101] Next, in step S4, a process is performed to determine or generate information (first information) according to the user's emotion estimated in step S3. The first information referred to here corresponds to the first information described in the above <Configuration example of information processing device>. In the information processing device 10, the information generating unit 14 can perform this process.

[0102] Next, in step S5, a process is performed to determine or generate information (second information) based on the first information determined or generated in step S4. In one aspect of the present invention, the second information includes all or at least a part of the first information. Note that the second information referred to here corresponds to the third information described in the above <Configuration example of information processing device>. In the information processing device 10, the information processing unit 16 can perform this process.

[0103] Next, in step S6, a process is performed to determine whether or not to transmit the second information determined or generated in step S5 to the outside based on information (third information) included in the radio waves transmitted from the global positioning system. The global positioning system referred to here corresponds to the global positioning system 29 described in the above <Configuration example of information processing device>. The third information referred to here corresponds to the second information described in the above <Configuration example of information processing device>. In the information processing device 10, the information processing unit 16 can perform this process.

[0104] If it is determined in step S6 that the second information is to be transmitted to the outside, a process of transmitting the second information to the outside is performed in accordance with the determination (step S7). In one aspect of the present invention, it is preferable that the second information is transmitted to an external device having an information receiving unit, the position of which is identified by the global positioning system. The information receiving unit referred to here corresponds to the information receiving unit 18 described in the above <Configuration example of information processing device>. The external device referred to here corresponds to the external device 28 described in the above <Configuration example of information processing device>. In the information processing device 10, the information transmitting unit 17 can perform the process.

[0105] On the other hand, if it is determined in step S6 that the second information is not to be transmitted to the outside, then in accordance with the determination, the second information is not transmitted to the outside (step S8).

[0106] In one aspect of the present invention, the third information is preferably information including a distance between the user and the external device. Also, in one aspect of the present invention, the external device is preferably one of a car and a building.

[0107] The above is a description of an example of an information processing method according to one embodiment of the present invention. By applying the example of the processing method, it is possible to provide an information processing method that can prevent an abnormal operation caused by a user's emotion from being performed in advance. Alternatively, it is possible to provide an information processing method that can select an appropriate operation according to a user's emotion and execute the operation.

[0108] This embodiment mode can be implemented by appropriately combining at least a part of it with other embodiment modes described in this specification.

[0109] (Embodiment 2) In this embodiment, an example of a hardware configuration of an information processing device according to an embodiment of the present invention will be described. As described in the first embodiment, the information processing device according to an embodiment of the present invention can be applied as a mobile information terminal device such as a mobile phone (including a smartphone) or a tablet terminal.

[0110] 4 shows a block diagram of an information processing device 100 exemplified below. The information processing device 100 includes a processing unit 101, a processing unit 102, a memory module 103, a display module 104, a sensor module 105, a sound module 106, a communication module 108, a battery module 109, a camera module 110, and an external interface 111.

[0111] The calculation unit 102, memory module 103, display module 104, sensor module 105, sound module 106, communication module 108, battery module 109, camera module 110, external interface 111, etc. are each connected to the calculation unit 101 via a bus line 107.

[0112] The display module 104 can function as an image display unit of an information processing device according to an embodiment of the present invention (for example, a mobile information terminal device such as a mobile phone or a tablet terminal). The sound module 106 can function as a call unit or a voice output unit of the information processing device according to an embodiment of the present invention. The sensor module 105 or the camera module 110 can function as the subject detection unit 11 of the information processing device 10 described in the first embodiment. The calculation unit 101, the calculation unit 102, and the memory module 103 can function as the feature extraction unit 12, the emotion estimation unit 13, the information generation unit 14, the information processing unit 16, and the like of the information processing device 10. The communication module 108 can function as the sensor unit 15 of the information processing device 10. The external interface 111 can function as the information transmission unit 17 of the information processing device 10.

[0113] 4, the calculation unit 101 is shown as one block, but may be configured to include two calculation units. For example, when the calculation unit 101 functions as the information processing unit 16 of the information processing device 10 described in the first embodiment, one of the two calculation units determines or generates information to be output to the information transmission unit 17 based on information input from the information generation unit 14, and the other of the two calculation units determines whether or not to output the information determined or generated by one of the calculation units to the information transmission unit 17 based on information input from the sensor unit 15.

[0114] The calculation unit 101 can function as, for example, a central processing unit (CPU). The calculation unit 101 has a function of controlling each component, such as the calculation unit 102, the memory module 103, the display module 104, the sensor module 105, the sound module 106, the communication module 108, the battery module 109, the camera module 110, and the external interface 111.

[0115] Signals are transmitted between the calculation unit 101 and each component via a bus line 107. The calculation unit 101 has a function of processing signals input from each component connected via the bus line 107, a function of generating signals to be output to each component, and the like, and can comprehensively control each component connected to the bus line 107.

[0116] The computing unit 101 performs various data processing and program control by interpreting and executing commands from various programs using a processor. The programs that can be executed by the processor may be stored in a memory area of ​​the processor or may be stored in the memory module 103.

[0117] As the calculation unit 101, in addition to a CPU, other microprocessors such as a DSP (Digital Signal Processor) and a GPU (Graphics Processing Unit) can be used alone or in combination. Furthermore, these microprocessors may be realized by a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array) or an FPA (Field Programmable Analog Array).

[0118] The calculation unit 101 may have a main memory. The main memory may include a volatile memory such as a random access memory (RAM) or a non-volatile memory such as a read only memory (ROM).

[0119] The RAM provided in the main memory may be, for example, a dynamic random access memory (DRAM), and a virtual memory space is allocated and used as a working space for the calculation unit 101. The operating system, application programs, program modules, program data, etc. stored in the memory module 103 are loaded into the RAM for execution. The data, programs, program modules, etc. loaded into the RAM are directly accessed and operated by the calculation unit 101.

[0120] On the other hand, ROM can store BIOS (Basic Input / Output System) and firmware that do not require rewriting. As ROM, mask ROM, OTPROM (One Time Programmable Read Only Memory), EPROM (Erasable Programmable Read Only Memory), etc. can be used. As EPROM, UV-EPROM (Ultra-Violet Erasable Programmable Read Only Memory), which allows the stored data to be erased by exposure to ultraviolet light, EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory, etc. can be used.

[0121] It is preferable to use a processor specialized for parallel calculation rather than a CPU as the calculation unit 102. For example, it is preferable to use a processor having a large number (tens to hundreds) of processor cores capable of parallel processing, such as a GPU, a TPU (Tensor Processing Unit), or an NPU (Neural Processing Unit). This enables the calculation unit 102 to perform calculations, particularly related to neural networks, at high speed.

[0122] The memory module 103 may be, for example, a storage device using a nonvolatile storage element such as a flash memory, MRAM (Magnetoresistive Random Access Memory), PRAM (Phase change Random Access Memory), ReRAM (Resistive Random Access Memory), or FeRAM (Ferroelectric Random Access Memory), or a storage device using a volatile storage element such as DRAM or SRAM (Static Random Access Memory). Also, for example, a recording media drive such as a hard disk drive (HDD) or a solid state drive (SSD) may be used.

[0123] Also, a storage device such as an HDD or SSD that is detachable by a connector via the external interface 111, or a media drive for a recording medium such as a flash memory, a Blu-ray disc, or a DVD can be used as the memory module 103. Note that the memory module 103 may not be built into the information processing device 100, and an external storage device may be used as the memory module 103. In that case, the memory module 103 may be connected via the external interface 111, or may exchange data wirelessly via the communication module 108.

[0124] The display module 104 includes a display panel, a display controller, a source driver, a gate driver, and the like. An image can be displayed on the display surface of the display panel. The display module 104 may further include a projection unit (screen) and project the image displayed on the display surface of the display panel onto the screen. In this case, if a material that transmits visible light is used as the screen, an AR device that displays an image superimposed on a background image can be realized.

[0125] Display elements that can be used in the display panel include liquid crystal elements, organic EL elements, inorganic EL elements, LED elements, microcapsules, electrophoretic elements, electrowetting elements, electrofluidic elements, electrochromic elements, MEMS elements, and the like.

[0126] Also, a touch panel having a touch sensor function can be used as the display panel. In that case, the display module 104 may be configured to have a touch sensor controller, a sensor driver, and the like. The touch panel is preferably an on-cell type touch panel in which the display panel and the touch sensor are integrated, or an in-cell type touch panel. The on-cell type or in-cell type touch panel can be made thin and lightweight. Furthermore, the on-cell type or in-cell type touch panel can reduce the number of parts, thereby reducing costs.

[0127] The sensor module 105 includes a sensor unit and a sensor controller. The sensor controller receives an input from the sensor unit, converts it into a control signal, and outputs it to the calculation unit 101 via the bus line 107. The sensor controller may perform error management of the sensor unit, or may perform calibration processing of the sensor unit. The sensor controller may be configured to include a plurality of controllers for controlling the sensor units.

[0128] The sensor unit of the sensor module 105 preferably includes a photoelectric conversion element that detects visible light, infrared light, ultraviolet light, or the like and outputs the detection intensity. In this case, the sensor unit can be called an image sensor unit.

[0129] In addition to the sensor unit, the sensor module 105 preferably has a light source that emits visible light, infrared light, or ultraviolet light. In particular, when the sensor module 105 is used to detect a part of the user's face, having a light source that emits infrared light allows imaging with high sensitivity without making the user feel dazzled.

[0130] The sensor module 105 may also be configured to include various sensors having the function of measuring, for example, force, displacement, position, speed, acceleration, angular velocity, rotation speed, distance, light, liquid, magnetism, temperature, chemicals, sound, time, hardness, electric field, current, voltage, power, radiation, flow rate, humidity, gradient, vibration, odor, or infrared light.

[0131] The sound module 106 has an audio input unit, an audio output unit, a sound controller, and the like. The audio input unit has, for example, a microphone and an audio input connector. The audio output unit has, for example, a speaker and an audio output connector. The audio input unit and the audio output unit are each connected to the sound controller and connected to the calculation unit 101 via a bus line 107. The audio data input to the audio input unit is converted into a digital signal in the sound controller and processed in the sound controller and the calculation unit 101. Meanwhile, the sound controller generates an analog audio signal audible to the user in response to a command from the calculation unit 101 and outputs it to the audio output unit. An audio output device such as an earphone, a headphone, or a headset can be connected to the audio output connector of the audio output unit, and the audio generated by the sound controller is output to the device.

[0132] The communication module 108 can perform communication via an antenna. For example, the communication module 108 can have a function of receiving radio waves from the global positioning system 29 described in the first embodiment and outputting information contained in the radio waves to the information processing unit 16 of the information processing device 10. For example, the communication module 108 can have a function of controlling a control signal for connecting the information processing device 100 to a computer network in response to an instruction from the calculation unit 101 and transmitting the signal to the computer network. This allows the information processing device 100 to be connected to a computer network such as the Internet, an intranet, an extranet, a PAN (Personal Area Network), a LAN (Local Area Network), a CAN (Campus Area Network), a MAN (Metropolitan Area Network), a WAN (Wide Area Network), or a GAN (Global Area Network) to perform communication. In addition, when a plurality of methods are used as the communication method, the communication module 108 may have a plurality of antennas according to the communication method.

[0133] The communication module 108 may be provided with, for example, a high-frequency circuit (RF circuit) for transmitting and receiving RF signals. The high-frequency circuit is a circuit for converting between electromagnetic signals and electric signals in a frequency band defined by the laws of each country, and for wirelessly communicating with other communication devices using the electromagnetic signals. A practical frequency band is generally several tens of kHz to several tens of GHz. The high-frequency circuit connected to the antenna has a high-frequency circuit section corresponding to a plurality of frequency bands, and the high-frequency circuit section may have an amplifier, a mixer, a filter, a DSP, an RF transceiver, and the like. When performing wireless communication, a communication standard such as LTE (Long Term Evolution) or a specification standardized by IEEE such as Wi-Fi (registered trademark) or Bluetooth (registered trademark) may be used as a communication protocol or communication technology.

[0134] The communication module 108 may also have a function of connecting the information processing device 100 to a telephone line. The communication module 108 may also have a tuner that generates a video signal to be output to the display module 104 from a broadcasting wave received by an antenna.

[0135] The battery module 109 may have a configuration including a secondary battery and a battery controller. Representative examples of the secondary battery include a lithium ion secondary battery and a lithium ion polymer secondary battery. The battery controller may have a function of supplying the power stored in the battery to each component, a function of receiving power supplied from the outside and charging the battery, a function of controlling the charging operation according to the battery charging state, and the like. For example, the battery controller may have a configuration including a BMU (Battery Management Unit) and the like. The BMU collects data on the cell voltage and cell temperature of the battery, monitors overcharging and overdischarging, controls the cell balancer, manages the battery degradation state, calculates the remaining battery capacity (State Of Charge: SOC), controls fault detection, and the like.

[0136] The camera module 110 may have an imaging element and a controller. For example, a still image or a video can be captured by pressing a shutter button or by operating a touch panel of the display module 104. The captured image or video data can be stored in the memory module 103. The image or video data can be processed by the calculation unit 101 or the calculation unit 102. The camera module 110 may also have a light source for capturing images. For example, a lamp such as a xenon lamp, a light-emitting element such as an LED or an organic EL, or the like can be used. Alternatively, light emitted by a display panel of the display module 104 may be used as a light source for capturing images. In this case, light of various colors, not just white, may be used for capturing images.

[0137] Examples of the external port of the external interface 111 include a configuration in which a transceiver for optical communication using infrared light, visible light, ultraviolet light, or the like is provided, and a configuration in which a transceiver for RF signals is provided like the above-mentioned communication module 108. With this configuration, the external interface 111 can have the function of the information transmission unit 17 of the information processing device 10 described in the first embodiment, and can transmit information determined or generated by the information processing unit 16 to an external device 28 having an information receiving unit 18.

[0138] In addition to the above configuration, for example, a physical button provided on the housing of the information processing device 100 or an external port to which other input components can be connected may be provided. In this case, the external port of the external interface 111 may be configured to be connected via a cable to devices such as input means such as a keyboard or a mouse, output means such as a printer, and storage means such as a HDD. A representative example is a USB terminal. In addition, the external port may be configured to have a terminal for connecting to a LAN, a terminal for receiving digital broadcasts, a terminal for connecting an AC adapter, etc.

[0139] The above is a description of an example of the hardware configuration of the information processing device according to one aspect of the present invention.

[0140] This embodiment mode can be implemented by appropriately combining at least a part of it with other embodiment modes described in this specification.

[0141] (Embodiment 3) Examples of electronic devices to which one embodiment of the present invention can be applied include display devices, personal computers, image storage devices or image playback devices equipped with a recording medium, mobile phones (including smartphones), game machines including portable types, portable data terminals (tablet terminals), electronic book terminals, cameras such as video cameras and digital still cameras, goggle-type displays (head-mounted displays), navigation systems, audio playback devices (car audio, digital audio players, etc.), copiers, facsimiles, printers, printer-combined machines, automated teller machines (ATMs), vending machines, etc. Specific examples of these electronic devices are shown in Figures 5A to 5F.

[0142] 5A shows an example of a mobile phone, which includes a housing 981, a display unit 982, operation buttons 983, an external connection port 984, a speaker 985, a microphone 986, a camera 987, and the like. The mobile phone includes a touch sensor in the display unit 982. Any operation, such as making a call or inputting characters, can be performed by touching the display unit 982 with a finger, a stylus, or the like. An information processing device and an information processing method according to one embodiment of the present invention can be applied to an element for acquiring an image (acquiring information on a user's face) in the mobile phone.

[0143] 5B shows an example of a portable data terminal, which includes a housing 911, a display unit 912, a speaker 913, a camera 919, and the like. Information can be input and output using a touch panel function of the display unit 912. Characters and the like can be recognized from an image acquired by the camera 919, and the characters can be output as voice by the speaker 913. An information processing device and an information processing method according to one embodiment of the present invention can be applied to an element for acquiring an image (acquiring information on a user's face) in the portable data terminal.

[0144] FIG. 5C shows an example of a surveillance camera (security camera), which includes a support base 951, a camera unit 952, a protective cover 953, and the like. The camera unit 952 is provided with a rotation mechanism and the like, and can capture images of the entire periphery by installing it on a ceiling. An information processing device and an information processing method according to an embodiment of the present invention can be applied to an element for acquiring an image (acquiring information on a user's face) in the camera unit. Note that the term "surveillance camera" is a common name and does not limit the use. For example, a device having a function as a surveillance camera is also called a camera or a video camera.

[0145] 5D shows an example of a video camera, which has a first housing 971, a second housing 972, a display unit 973, operation keys 974, a lens 975, a connection unit 976, a speaker 977, a microphone 978, and the like. The operation keys 974 and the lens 975 are provided in the first housing 971, and the display unit 973 is provided in the second housing 972. An information processing device and an information processing method according to an embodiment of the present invention can be applied to elements for acquiring an image (acquiring information about a user's face) in the video camera.

[0146] 5E shows an example of a digital camera, which includes a housing 961, a shutter button 962, a microphone 963, a light-emitting unit 967, and a lens 965. An information processing device and an information processing method according to an embodiment of the present invention can be applied to elements for acquiring an image (acquiring information on a user's face) in the digital camera.

[0147] 5F shows an example of a wristwatch-type information terminal, which includes a display unit 932, a housing / wristband 933, a camera 939, and the like. The display unit 932 includes a touch panel for operating the information terminal. The display unit 932 and the housing / wristband 933 are flexible and have excellent wearability on the body. An information processing device and an information processing method according to one embodiment of the present invention can be applied to elements for acquiring images (acquiring information on the user's face) in the information terminal.

[0148] For example, consider a case where the information processing device according to an embodiment of the present invention is a mobile phone as shown in Fig. 5A. For example, when a user drives a car having the information receiving unit 18 described in the first embodiment, by placing the mobile phone in a position where the user's face can be detected, the information processing method according to an embodiment of the present invention allows the user to always drive safely regardless of the user's emotions.

[0149] Note that an information processing device according to an embodiment of the present invention that can be applied when driving a car is not limited to a mobile phone as shown in Fig. 5A, but may be a mobile data terminal as shown in Fig. 5B, a video camera as shown in Fig. 5D, a digital camera as shown in Fig. 5E, or a wristwatch-type information terminal as shown in Fig. 5F.

[0150] Also, for example, consider a case where an information processing device according to one embodiment of the present invention is a mobile phone as shown in FIG. 5A, and a user uses the mobile phone to make a call inside a building having the information receiving unit 18 described in the first embodiment. In this case, the mobile phone can detect the face of the user during the call. For example, when the mobile phone estimates from the user's facial expression that the user's emotions have suddenly changed to intense anger, it transmits information to the effect that the call will be stopped. Then, the building having the information receiving unit 18 that has received the information can take measures such as forcibly terminating the user's call (for example, transmitting a signal to automatically disconnect the call). This can prevent deterioration of human relationships and loss of business opportunities.

[0151] Also, for example, consider a case where the information processing device according to one aspect of the present invention is a mobile phone as shown in FIG. 5A, and the external device 28 having the information receiving unit 18 described in the first embodiment is a building in which an automated teller machine is installed (for example, a bank or a convenience store). For example, assume that the user notices the receipt of an email on the mobile phone urging the user to make a transfer. At this time, the mobile phone infers from the user's facial expression that the user is feeling strong anxiety. Then, by the information processing method according to one aspect of the present invention, when the user is near an automated teller machine, the mobile phone can take measures such as sending information prohibiting the use of the automated teller machine to the building in which the automated teller machine is installed. This makes it possible to prevent damage such as the above-mentioned transfer fraud from occurring.

[0152] Also, for example, consider a case where an information processing device according to an embodiment of the present invention is a surveillance camera as shown in FIG. 5C. For example, consider a case where the surveillance camera is installed at a cash register of a convenience store, and an external device 28 having the information receiving unit 18 described in the first embodiment is a crisis management room of the convenience store. For example, if a convenience store clerk is having difficulty dealing with a troublesome customer, and the surveillance camera estimates from the clerk's facial expression that the clerk is feeling a strong sense of distress, measures can be taken such as sending information from the surveillance camera to the crisis management room requesting the dispatch of support personnel. This makes it possible to prevent damage such as the clerk getting involved in trouble.

[0153] This embodiment mode can be implemented by appropriately combining at least a part of it with other embodiment modes described in this specification. [Explanation of symbols]

[0154] 10: Information processing device, 11: Object detection unit, 12: Feature extraction unit, 13: Emotion estimation unit, 14: Information generation unit, 15: Sensor unit, 16: Information processing unit, 17: Information transmission unit, 18: Information reception unit, 28: External device, 29: Global positioning system, 51: Input layer, 52: Middle layer, 53: Output layer, 61: Data, 62: Data, 63: Data, 100: Information processing device, 101: Calculation unit, 102: Calculation unit, 103: Memory module, 104: Display module, 105: Sensor module, 106: Sound module, 107: Bus line, 108: Communication module, 109: Battery module, 110: Camera module, 1 11: external interface, 911: housing, 912: display unit, 913: speaker, 919: camera, 932: display unit, 933: housing / wristband, 939: camera, 951: support stand, 952: camera unit, 953: protective cover, 961: housing, 962: shutter button, 963: microphone, 965: lens, 967: light emitting unit, 971: first housing, 972: second housing, 973: display unit, 974: operation keys, 975: lens, 976: connection unit, 977: speaker, 978: microphone, 981: housing, 982: display unit, 983: operation button, 984: external connection port, 985: speaker, 986: microphone, 987: camera

Claims

[Claim 1] A portable information terminal device that can be carried by a user, A subject detection unit having a function of acquiring image data of the user's face; a feature extraction unit having a function of extracting feature information from the face image data; a feeling estimation unit having a function of estimating a feeling of the user by using information on the features; an information generating unit having a function of selecting one piece of first information corresponding to the estimated emotion of the user from a data set having a plurality of pieces of first information, the first information being information on an appropriate action to be performed by an external device and linked to emotion information; a sensor unit that generates second information including distance information between the portable information terminal device and the external device; an information processing unit having a function of receiving one of the first information and the second information, a function of generating third information including the one of the first information, and a function of determining whether or not to transmit the third information to the external device based on the second information; A mobile information terminal device having the above configuration.

Images