Determination device, determination method, and program

Abstract

This determination device comprises: an acquisition unit which acquires sensing data indicating movement and / or vital sign states of a target user, the sensing data changing in response to an internal state of the target user; a learning unit which executes machine learning by using sensing data intended for learning and information indicating internal states of the target user, and which generates a trained model for outputting information indicating the internal state in response to an input of the sensing data; and a determination unit which determines the internal state of the target user on the basis of the trained model and sensing data intended for inference.

Description

Discrimination device, discrimination method, and program

[0001] The present invention relates to a discrimination device, a discrimination method, and a program.

[0002] In traditional offices where work is primarily done face-to-face, it is easy for team members, including team leaders, to keep track of each other's schedules, work progress, physical condition, and other information. This makes it relatively easy to check whether each member is able to carry out their work while maintaining an appropriate level of mental relaxation.

[0003] However, in recent years, modern offices that are not limited to face-to-face work, such as offices with free address systems and offices that have introduced remote work, are becoming more common. Even in modern offices with such diverse work styles, it is easy to check each member's schedule and work status online, just like in traditional offices. However, it can be difficult to check online things related to the internal state of members, such as their physical condition and mood, which traditionally required physical inspection.

[0004] In response to this, there are conventional techniques for determining a person's internal state by sensing with a sensor. For example, the sensing technique described in Non-Patent Document 1 estimates a person's emotions, such as "happiness," from facial expressions.

[0005] Tomori Hachisuka, "[Research Introduction] The Challenge of Sensing Human Internal States and Inducing Emotions," University of Tokyo Graduate School of Interdisciplinary Information Studies, website, October 25, 2022, [Retrieved September 8, 2023], Internet (URL: https: / / www.iii.u-tokyo.ac.jp / research / 221025hachisuka)

[0006] However, in public situations such as work, a person may appear to be smiling in a facial image, but may be feeling stressed and anxious inside. In such cases where there is a discrepancy between the emotional expression on the surface and the person's inner state, the accuracy of the conventional techniques for determining a person's inner state from a facial image may be low.

[0007] In view of the above circumstances, the present invention aims to provide a discrimination device, a discrimination method, and a program that can suppress a decrease in discrimination accuracy that can occur when discriminating a person's internal state through sensing, due to a discrepancy between outward emotional expression and the internal state.

[0008] One aspect of the present invention is a discrimination device that includes an acquisition unit that acquires sensing data indicating at least one of the movements and vital state of a target user, which changes depending on the internal state of the target user; a learning unit that performs machine learning using the sensing data for learning and information indicating the internal state of the target user, and generates a learning model that outputs information indicating the internal state in response to input of the sensing data; and a discrimination unit that discriminates the internal state of the target user based on the learning model and the sensing data for inference.

[0009] Another aspect of the present invention is a computer-based discrimination method, comprising: an acquisition step of acquiring sensing data indicating at least one of the target user's movements and vital state, which change depending on the target user's internal state; a learning step of performing machine learning using the sensing data for learning and information indicating the target user's internal state, to generate a learning model that outputs information indicating the internal state in response to input of the sensing data; and a discrimination step of discriminating the internal state of the target user based on the learning model and the sensing data for inference.

[0010] Another aspect of the present invention is a program for causing a computer to execute an acquisition step of acquiring sensing data indicating at least one of the target user's movements and vital state, which change depending on the target user's internal state; a learning step of performing machine learning using the sensing data for learning and information indicating the target user's internal state, to generate a learning model that outputs information indicating the internal state in response to input of the sensing data; and a determination step of determining the internal state of the target user based on the learning model and the sensing data for inference.

[0011] The present invention makes it possible to suppress a decrease in the accuracy of discrimination that can occur due to a discrepancy between outward emotional expression and the inner state when discriminating between the inner state of a person through sensing.

[0012] FIG. 1 is a diagram showing the overall configuration of a discrimination system 1 in a first embodiment of the present invention. FIG. 2 is a block diagram showing the functional configuration of a discrimination device 10a in the first embodiment of the present invention. FIG. 3 is a flowchart showing the operation of the discrimination device 10a in the first embodiment of the present invention during learning. FIG. 4 is a flowchart showing the operation of the discrimination device 10a in the first embodiment of the present invention during inference. FIG. 4 is a block diagram showing the functional configuration of a discrimination device 10b in the first embodiment of the present invention. FIG. 5 is a flowchart showing the operation of the discrimination device 10b in the first embodiment of the present invention during learning. FIG. 6 is a flowchart showing the operation of the discrimination device 10b in the first embodiment of the present invention during inference. FIG. 7 is a block diagram showing the functional configuration of a discrimination device 10c in the first embodiment of the present invention. FIG. 8 is a flowchart showing the operation of the discrimination device 10c in the first embodiment of the present invention during discrimination rule registration. FIG. 9 is a flowchart showing the operation of the discrimination device 10c in the first embodiment of the present invention during inference. FIG. 10 is a block diagram showing the functional configuration of a discrimination device 10d in a second embodiment of the present invention. FIG. 11 is a flowchart showing the operation of the discrimination device 10d in the second embodiment of the present invention during learning. FIG. 12 is a flowchart showing the operation of the discrimination device 10d in the second embodiment of the present invention during inference. FIG. 13 is a block diagram showing the functional configuration of a discrimination device 10e in a third embodiment of the present invention. FIG. 14 is a flowchart showing the operation of the discrimination device 10e in the third embodiment of the present invention. FIG. 10 is a flowchart showing the operation of the discrimination device 10e during inference in the third embodiment of the present invention. FIG. 11 is a block diagram showing the functional configuration of the discrimination device 10f in the fourth embodiment of the present invention. FIG. 12 is a flowchart showing the operation of the discrimination device 10f during learning in the fourth embodiment of the present invention. FIG. 13 is a block diagram showing the functional configuration of the discrimination device 10g in the fourth embodiment of the present invention. FIG. 14 is a flowchart showing the operation of the discrimination device 10g during inference in the fourth embodiment of the present invention. FIG. 15 is a block diagram showing the functional configuration of the discrimination device 10h in the fourth embodiment of the present invention. FIG. 16 is a flowchart showing the operation of the discrimination device 10h in the fourth embodiment of the present invention during inference. FIG. 17 is an overall configuration diagram of a chat system 1i in a fifth embodiment of the present invention.FIG. 1 is a block diagram showing the functional configuration of a chat server 20i in a fifth embodiment of the present invention. FIG. 2 is a flowchart showing the operation of the chat server 20i in the fifth embodiment of the present invention. FIG. 3 is an overall configuration diagram of an interview system 1j in a sixth embodiment of the present invention. FIG. 4 is a block diagram showing the functional configuration of an interview server 20j in a sixth embodiment of the present invention. FIG. 5 is a flowchart showing the operation of the interview server 20j in the sixth embodiment of the present invention.

[0013] Hereinafter, several embodiments of the discrimination device, discrimination method, and program of the present invention will be described in detail with reference to the drawings.

[0014] First Embodiment A first embodiment of the present invention will be described below.

[0015] [Configuration of the Discrimination System] The following describes the configuration of the discrimination system 1 according to the first embodiment of the present invention. The discrimination system 1 is an information system that uses a discrimination device 10 to determine the inner state of a target user, and uses a service providing device 20 to perform information processing such as providing various services utilizing the determination results.

[0016] Fig. 1 is a diagram showing the overall configuration of a discrimination system 1 according to a first embodiment of the present invention. As shown in Fig. 1, the discrimination system 1 includes a discrimination device 10, a service providing device 20, a sensor device 40, and a network 50. Note that the number of sensor devices 40 may be one.

[0017] The discrimination device 10 includes an information processing device such as a general-purpose computer. The discrimination device 10 is connected to a network 50. The discrimination device 10 can communicate with the service providing device 20 and each of the plurality of sensor devices 40 via the network 50.

[0018] The discrimination device 10 is a device that discriminates the inner state of a user using information collected by sensing using the sensor devices 40 and provides the discrimination result to the service providing device 20. The user here refers to a person who is the target of discrimination of the inner state (hereinafter referred to as the "target user"). The discrimination device 10 discriminates the inner state of the target user using AI (Artificial Intelligence).

[0019] For example, the determination device 10 is a server device that provides a cloud-type service via a network 50 in cooperation with a service providing device 20. For example, a cloud-type service is a service that determines the inner state of a target user in response to a request and provides information in an appropriate manner depending on the determination result. Note that the recipient of such a service is not limited to the target user (i.e., the person whose inner state is to be determined). For example, the recipient of such a service may be a business operator or the like that provides services to the target user.

[0020] The discrimination device 10 acquires information (hereinafter referred to as "sensing data") obtained by sensing using various sensors provided in the sensor device 40. The discrimination device 10 discriminates the inner state of the target user who is the subject of sensing based on the acquired sensing data.

[0021] In this embodiment, determining the inner state means determining whether the target user is in a relaxed state (hereinafter simply referred to as a "relaxed state") or in a stressed state (hereinafter simply referred to as a "distressed state"). However, the determination of the inner state performed by the determination device 10 is not limited to determining whether the target user is in a relaxed state, and may be a determination of other types of inner state.

[0022] A relaxed state refers to a state in which the target user is relatively under pressure from external factors, both physically and mentally. When in a relaxed state, the target user is relaxed and has time to spare, and can make appropriate decisions on their own, even in matters that require judgment. In addition, when in a relaxed state, the target user can, for example, accept new tasks other than those currently in hand.

[0023] On the other hand, a stressful state refers to a state in which the target user is under relatively high pressure due to external factors in at least one of their minds and bodies. When in a stressful state, the target user feels tense and stressed, and may find it difficult to make appropriate decisions on their own, for example, in matters that require judgment. Also, when in a stressful state, the target user may find it difficult to accept new tasks other than those they currently have.

[0024] For example, the discrimination device 10 stores information indicating a plurality of predefined inner states, and outputs one of the pieces of information indicating the plurality of inner states as the discrimination result. For example, if the predefined inner states are classified into two categories, the two inner states are a "comfortable state" and a "tight state." Furthermore, if the predefined inner states are classified into five categories, the five inner states are a "comfortable state," a "slightly comfortable state," a "neutral state," a "slightly tight state," and a "tight state." The discrimination device 10 outputs information indicating one of these inner states as the discrimination result.

[0025] The discrimination device 10 outputs information indicating the discrimination result to the service providing device 20. The service providing device 20 provides the service while controlling the content and timing of provision according to the discrimination result based on the acquired information. Note that the discrimination device 10 and the service providing device 20 may be integrated into one device.

[0026] The configuration of the discrimination device 10 will be described in detail later.

[0027] The service providing device 20 includes an information processing device such as a general-purpose computer. The service providing device 20 is connected to a network 50. The service providing device 20 can communicate with the discrimination device 10 and each of the plurality of sensor devices 40 via the network 50.

[0028] The service providing device 20 is a server device that provides various services depending on the determination result of the inner state of the target user based on the information output from the determination device 10. As described above, for example, the service providing device 20 is a server device that provides a cloud-type service via the network 50 in cooperation with the determination device 10. For example, the cloud-type service is a service that determines the inner state of the target user in response to a request and provides information in an appropriate manner depending on the determination result.

[0029] For example, if the cloud-based service is a service that provides appropriate notifications to the target user, the service providing device 20 controls the notifications depending on the inner state of the target user determined by the determination device 10.

[0030] For example, suppose that when the target user is in a relaxed state, the target user is willing to receive notifications containing information about work, etc., but when the target user is in a stressful state, the target user does not want to receive notifications containing information about work, etc. In such a case, the determination device 10 determines whether the current target user is in a relaxed state or a stressful state, and the service providing device 20 determines whether to execute a notification to the target user based on the determination result output from the determination device 10.

[0031] An example of a service using the result of determining the inner state will be described in more detail in the embodiments (especially the fifth and sixth embodiments) described later.

[0032] The service providing device 20 pre-stores information associating the inner state of the target user with service provision contents. The service providing device 20 acquires, from the discrimination device 10, determination result information including information indicating the inner state of the target user and identification information identifying the target user or identification information identifying a terminal (not shown) held by the target user (hereinafter collectively referred to as "user identification information"). The service providing device 20 provides, to the target user identified by the user identification information included in the acquired determination result information, a service according to the service provision contents associated with the inner state based on the information included in the acquired determination result information.

[0033] The determination result information may be instruction information further including information instructing the content of the service to be provided. That is, the service providing device 20 may be configured to only execute the service, and the determination device 10 may be configured to determine the inner state of the target user and also determine the content of the service to be provided according to the determination result.

[0034] In this case, the service providing device 20 provides a service according to the service provision content included in the acquired instruction information. For example, the service provision content included in the instruction information may include whether or not to notify, the content of the notification, the people to be notified and the notification range, the device or application used for the notification, and the service provision level. The instruction information may also include information indicating the timing of the service provision, such as the date and time when the service is provided. In this case, the service providing device 20 provides the service according to the timing based on the information included in the acquired instruction information.

[0035] The network 50 is an information communication network that connects the discrimination device 10, the service providing device 20, and each of the plurality of sensor devices 40 to one another for communication. For example, the network 50 is configured to include the Internet, a local area network (LAN), etc. Also, for example, a wireless communication base station (not shown) is connected to the network 50. For example, the sensor device 40, which is a wireless communication terminal, is connected to the network 50 via the wireless communication base station.

[0036] The sensor devices 40 are various sensors equipped with communication functions or terminal devices equipped with various sensors. The terminal devices may be terminals owned by users (hereinafter referred to as "user terminals"), such as smartphones, tablet terminals, or laptop computers.

[0037] The various sensors included in the sensor device 40 include, for example, a camera and a microphone that measure the target user's movements, and a sensor that measures the vital status (hereinafter referred to as a "vital status measurement sensor"). The vital status here refers to the status indicated by the index values ​​of indicators (vital signs) that indicate signs of life and health status, such as pulse rate, breathing, body temperature, blood pressure, and level of consciousness. Below, cases where the sensor device 40 is these devices will be described.

[0038] [When a Camera is Used] The following describes a case where the sensor provided in the sensor device 40 is a camera. In this case, the sensor device 40 is, for example, a camera equipped with a communication function or a terminal device equipped with a camera. The camera captures an image of a target user who is the subject of determining the internal state. The camera captures, for example, a video of a part of the target user's body or the entire body. Note that the camera may also be a camera that detects movement of an object, such as an event camera.

[0039] The sensor device 40 transmits sensing data obtained by the camera to the discrimination device 10 via the network 50. That is, the sensing data in this embodiment is image data (video data) of the target user captured by the camera. Note that the camera capturing the image of the target user and the device transmitting the image data to the discrimination device 10 may be separate devices.

[0040] The sensor device 40 assigns user identification information that identifies the target user photographed by the camera to the sensing data, and transmits the sensing data to the discrimination device 10. Note that, in cases where the target user photographed by each sensor device 40 is uniquely determined, the sensor device 40 may assign identification information (e.g., a terminal ID) that identifies the sensor device 40 to the sensing data, and transmit the sensing data to the discrimination device 10. In other words, the sensor device 40 assigns information that enables the discrimination device 10 to identify which target user the sensing data relates to, and transmits the sensing data to the discrimination device 10.

[0041] In addition, the sensing data may be configured to be transmitted directly from the sensor device 40 to the discrimination device 10, or may be configured to be transmitted from the sensor device 40 to the discrimination device 10 via the service providing device 20 or other devices (not shown).

[0042] The discrimination device 10 receives sensing data of a specific target user transmitted from the sensor device 40. As described above, the sensing data here is image data captured of the target user. The discrimination device 10 analyzes the image data and uses AI to detect specific actions of the target user and discriminate the internal state of the target user.

[0043] It should be noted that the detection of the specific behavior of the target user and the determination of the internal state can be performed using any of unsupervised machine learning AI, supervised machine learning AI, and rule-based AI. The configuration of each of the above-mentioned discrimination devices 10 using AI will be described in detail later.

[0044] For example, when the discrimination device 10 detects the following specific behavior of the target user from the image data, it determines that the target user's inner state is stressed. The specific behavior of the target user shown below is generally recognized as being likely to occur in people who are feeling stressed, and is listed as an example of the behavior of the target user in a stressed state.

[0045] - When fidgeting is detected (i.e., when it is detected that only one part of the target user's body is moving unnaturally). - When it is detected that the target user is restlessly touching something unrelated to work (for example, something other than a mouse, keyboard, or writing utensil) with their hands, but other parts of the body are stable. - When it is detected that the knees are continuously moving, but other parts of the body are stable. - When it is detected that the head is shaking, but other parts of the body are stable. - When it is detected that the gaze is unsteady (Note that in this case, the discrimination device 10 may be configured to obtain and detect observed values ​​of eye movement and measured values ​​of electrooculography from the sensor device 40).

[0046] Any existing behavior analysis technology can be used to detect the specific behavior of the target user as described above. For example, OpenCV (registered trademark), an open source computer vision library developed and released by Intel Corporation in the United States, can be used to detect the specific behavior of the target user as described above.

[0047] [When a microphone is used] The following describes a case where the sensor provided in the sensor device 40 is a microphone (hereinafter, abbreviated as "microphone"). In this case, the sensor device 40 is, for example, a microphone equipped with a communication function or a terminal device equipped with a microphone. The microphone collects the voice of the target user whose inner state is to be determined.

[0048] The sensor device 40 transmits sensing data obtained by the microphone to the discrimination device 10 via the network 50. That is, the sensing data in this embodiment is voice data of the target user collected by the microphone. Note that the microphone that collects the voice of the target user and the device that transmits the voice data to the discrimination device 10 may be separate devices.

[0049] The sensor device 40 assigns user identification information for identifying the target user whose voice is collected by the microphone to the sensing data, and transmits the sensing data to the discrimination device 10. Note that, in cases where the target user photographed by each sensor device 40 is uniquely determined, the sensor device 40 may assign identification information for identifying the sensor device 40 (e.g., a terminal ID, etc.) to the sensing data, and transmit the sensing data to the discrimination device 10. In other words, the sensor device 40 assigns information that enables the discrimination device 10 to identify which target user the sensing data relates to, and transmits the sensing data to the discrimination device 10.

[0050] In addition, the sensing data may be configured to be transmitted directly from the sensor device 40 to the discrimination device 10, or may be configured to be transmitted from the sensor device 40 to the discrimination device 10 via the service providing device 20 or other devices (not shown).

[0051] The discrimination device 10 receives sensing data of a specific target user transmitted from the sensor device 40. As described above, the sensing data here is audio data obtained by collecting the target user's voice. The discrimination device 10 analyzes the audio data and uses AI to detect the target user's specific conversational voice and discriminate the target user's internal state.

[0052] It should be noted that the detection of the specific conversational voice of the target user and the determination of the internal state can be performed using any of unsupervised machine learning AI, supervised machine learning AI, and rule-based AI. The configuration of each of the above-mentioned AI-based determination devices 10 will be described in detail later.

[0053] For example, when the discrimination device 10 detects the following specific conversational voice of the target user from the voice data, it determines that the target user is in a stressful state. The specific conversational voice of the target user shown below is generally recognized as being likely to occur in people who are feeling stressed, and is listed as an example of the behavior of the target user in a stressful state.

[0054] - When a hesitation in speech is detected. - When frequent periods of silence of 2 seconds or more are detected. - When an unnatural tremor in the voice is detected.

[0055] Furthermore, the discrimination device 10 determines that the target user's inner state is relaxed when, for example, the following specific conversational voice of the target user is detected from the voice data: The specific conversational voice of the target user shown below is generally recognized as being likely to occur in people who are not feeling stressed, and is listed as an example of behavior of a target user in a relaxed state.

[0056] ・When it is detected that the person is speaking loudly and smoothly. ・When it is detected that the person is speaking continuously without pauses in speech.

[0057] Any existing voice analysis technology can be used to detect the specific conversational voice of the target user as described above.

[0058] Furthermore, for example, by using sensing data obtained by a microphone, the discrimination device 10 can also detect the state of the environment around the target user, which may further improve the accuracy of discriminating the inner state of the target user.

[0059] [When a vital sign measurement sensor is used] The following describes a case where the sensor included in the sensor device 40 is a vital sign measurement sensor. In this case, the sensor device 40 is, for example, a vital sign measurement sensor equipped with a communication function, or a terminal device equipped with a vital sign measurement sensor. The vital sign measurement sensor here is, for example, a sensor that can measure heart rate, pulse rate, or brain waves (and further alpha waves and beta waves), etc. The vital sign measurement sensor may also be a terminal device that can detect fingerprints, irises, etc. In this case, the sensor device 40 may be a wearable terminal that can measure vital signs, such as a smart watch or smart glasses.

[0060] The vital sign measurement sensor measures the vital state of the target user, who is the subject of inner state discrimination. The sensor device 40 transmits sensing data obtained by the vital sign measurement sensor to the discrimination device 10 via the network 50. That is, the sensing data in this embodiment is data indicating vital information of the target user measured by the vital sign measurement sensor (hereinafter referred to as "vital data"). Note that the vital sign measurement sensor that measures the vital state of the target user and generates vital sign data and the device that transmits the vital sign data to the discrimination device 10 may be separate devices.

[0061] The sensor device 40 assigns user identification information that identifies a target user whose vital status has been measured by the vital measurement sensor to the sensing data, and transmits the sensing data to the discrimination device 10. Note that, in cases where the target user whose vital status is measured by each sensor device 40 is uniquely determined, the sensor device 40 may assign identification information that identifies the sensor device 40 (e.g., a terminal ID, etc.) to the sensing data and transmit the sensing data to the discrimination device 10. In other words, the sensor device 40 assigns information that enables the discrimination device 10 to identify which target user the sensing data relates to, and transmits the sensing data to the discrimination device 10.

[0062] In addition, the sensing data may be configured to be transmitted directly from the sensor device 40 to the discrimination device 10, or may be configured to be transmitted from the sensor device 40 to the discrimination device 10 via the service providing device 20 or other devices (not shown).

[0063] The discrimination device 10 receives sensing data of a specific target user transmitted from the sensor device 40. As described above, the sensing data here is vital data indicating the measurement results of the target user's vital state. The discrimination device 10 analyzes the vital data and uses AI to detect the target user's specific vital state and discriminate the internal state.

[0064] It should be noted that detection of a specific vital state and determination of an internal state of a target user can be performed using any of unsupervised machine learning AI, supervised machine learning AI, and rule-based AI. The configuration of each of the above-mentioned discrimination devices 10 using AI will be described in detail later.

[0065] For example, when the discrimination device 10 detects the following specific vital states of the target user from the vital data, it determines that the target user's inner state is stressed. The specific vital states of the target user shown below are generally recognized as being likely to occur in people who are feeling stressed, and are listed as examples of behavior of the target user in a stressed state.

[0066] ・When it is detected that the heart rate value is stable at a lower value than normal. ・When it is detected that there is a disturbance in the pulse (for example, large fluctuations in the pulse, or an especially fast pulse). ・When it is detected that the proportion of alpha waves in the brain waves is relatively large.

[0067] Any existing analysis technology can be used to detect the specific vital status of the target user as described above.

[0068] Note that if the sensor device 40 is equipped with a device capable of measuring position information, such as a positioning system such as a GPS (Global Positioning System), a gyro, or an acceleration sensor, the discrimination device 10 may be configured to perform machine learning using the position information obtained by these devices to discriminate the target user's inner state. For example, by using sensing data to which position information is attached, it becomes possible to detect the target user's position and movement in association with each other. This may allow the discrimination device 10 to determine the target user's inner state taking the target user's position into consideration, thereby further improving the accuracy of discriminating the inner state.

[0069] Furthermore, if the sensor device 40 is equipped with a device capable of measuring the environment surrounding the target user, such as a thermometer, hygrometer, barometer, anemometer, and photometer (hereinafter referred to as "environmental measurement device"), the discrimination device 10 may be configured to further use sensing data obtained by these environmental measurement devices to perform machine learning and discriminate the inner state of the target user.

[0070] The discrimination device 10 may further include a configuration for detecting an environmental measurement device installed near the sensor device 40 based on the location information of the sensor device 40, and for acquiring sensing data from the detected environmental measurement device. This makes it possible to discriminate the inner state of the target user taking into account the environment, such as the weather, and it is thought that this may further improve the accuracy of discriminating the inner state.

[0071] As such, sensing data can be broadly divided into sensing data indicating the target user's behavior (movements, voice, etc.) and location, sensing data indicating the target user's vital status, sensing data indicating the environment surrounding the target user, etc. It is believed that the more types of sensing data are used, the more accurate the determination of the target user's inner state can be improved.

[0072] Below, we will explain in detail each of the configuration examples (first to third configuration examples) of the discrimination device 10 that uses unsupervised machine learning type AI, supervised machine learning type AI, and rule-based type AI.

[0073] [First Configuration Example of Discrimination Device] The configuration of a discrimination device 10a that uses unsupervised machine learning AI will be described in detail below. Fig. 2 is a block diagram showing the functional configuration of the discrimination device 10a according to the first embodiment of the present invention. As shown in Fig. 2, the discrimination device 10 includes a learning unit 11, an inference unit 12, and a discrimination result output unit 13.

[0074] The learning unit 11 acquires learning data and performs unsupervised machine learning to cluster the learning data. Hereinafter, machine learning will also be simply referred to as "learning." As a clustering method, for example, hierarchical clustering or non-hierarchical clustering using a k-means algorithm or the like may be used.

[0075] As shown in FIG. 2, the learning unit 11 includes a learning data acquisition unit 111 , a learning data storage unit 112 , a learning execution unit 113 , an inner state registration unit 114 , and a learning model storage unit 115 .

[0076] The learning data acquisition unit 111 acquires learning data. Note that, because the discrimination device 10a is a device that performs unsupervised machine learning, this learning data does not include correct answer labels. The learning data referred to here is sensing data for learning collected by the sensor device 40. As described above, the sensing data may include image data captured by a camera of the body of a target user who is the subject of inner state discrimination, audio data collected by a microphone of the target user's conversation, or vital data measured by a vital sensor on the vital state of the target user. Note that the sensing data may include multiple types of data from among these image data, audio data, and vital data.

[0077] The learning data acquiring unit 111 acquires, in bulk as learning data, part or all of a large amount of sensing data collected in the past by each of the plurality of sensor devices 40, for example, from an external server (not shown) or the like that accumulates the large amount of sensing data. Note that the learning data acquiring unit 111 may also acquire sensing data as needed directly from each of the plurality of sensor devices 40. The learning data acquiring unit 111 stores the acquired sensing data in the learning data storage unit 112 as learning data.

[0078] In addition, if the learning unit 11 is configured not to include the learning data storage unit 112, the learning data acquisition unit 111 may output the acquired learning data directly to the learning execution unit 113 described below.

[0079] The learning data storage unit 112 stores the learning data acquired by the learning data acquisition unit 111. The learning data storage unit 112 stores the learning data, for example, by linking the learning data with user identification information that identifies a target user who is the target of inner state discrimination.

[0080] The learning execution unit 113 acquires the learning data stored in the learning data storage unit 112. The learning execution unit 113 performs unsupervised machine learning using the acquired learning data, thereby clustering the learning data.

[0081] Each cluster classified by clustering is interpreted by the internal state registration unit 114. Specifically, the interpretation is performed by assigning information indicating the internal state of the target user to each cluster classified by clustering. The information indicating the internal state here is, for example, information indicating whether the internal state is a "comfortable state" or a "constrained state" when the internal state is classified into two categories as described above. Furthermore, the information indicating the internal state here is, for example, information indicating whether the internal state is a "comfortable state," a "slightly comfortable state," a "neutral state," a "slightly constrained state," or a "constrained state" when the internal state is classified into five categories as described above.

[0082] For example, the interpretation of each cluster may be performed by an operator of the discrimination system 1 or by the target user himself / herself. For example, the operator or the target user refers to information indicating the clustering results obtained by unsupervised machine learning by the learning execution unit 113, and inputs information indicating the inner state to be assigned to each cluster to the inner state registration unit 114. The inner state registration unit 114 outputs the input information indicating the inner state to the learning execution unit 113, and the learning execution unit 113 associates the input information indicating the inner state with each cluster obtained as a result of the clustering.

[0083] The learning execution unit 113 stores the trained learning model generated as a result of the unsupervised machine learning in the learning model storage unit 115. The learning model here includes, for example, a machine learning algorithm, parameter values, and information indicating the inner state of the target user associated with each cluster.

[0084] The learning model storage unit 115 stores the learned learning model generated by the learning execution unit 113.

[0085] The learning execution unit 113 is a functional unit realized by a processor (not shown), such as a CPU (Central Processing Unit) included in the discrimination device 10a, reading and executing a program pre-stored in a storage medium (not shown) included in the discrimination device 10a. The learning data storage unit 112 and the learning model storage unit 115 include, for example, semiconductor memories such as RAM (Random Access Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), and ROM (Read Only Memory), flash memories such as SSDs (Solid State Drives), magnetic disks such as HDDs (Hard Disk Drives), optical disks, or any combination of these storage media (not shown). The learning data acquisition unit 111 and the internal state registration unit 114 include, for example, a communication interface for communication with external devices via a network 50.

[0086] As shown in FIG. 2, the inference unit 12 includes a sensing data acquisition unit 121 and an inner state determination unit 124 .

[0087] The sensing data acquisition unit 121 acquires sensing data from the sensor device 40. As described above, the sensing data here includes image data captured by a camera of the body of the target user who is the subject of inner state determination, audio data collected by a microphone of the target user's conversation, and vital data measured by a vital sensor on the target user's vital state. In other words, the sensing data acquired by the sensing data acquisition unit 121 is sensing data for inference that includes information on the same items as the learning sensing data, which is the learning data acquired by the learning data acquisition unit 111 described above. Note that the sensing data acquisition unit 121 and the learning data acquisition unit 111 may be configured as an integrated hardware device.

[0088] The sensing data acquisition unit 121 outputs the acquired sensing data to the inner state determination unit 124. Note that the sensing data acquisition unit 121 may store the acquired sensing data in the learning data storage unit 112 of the learning unit 11. In other words, the sensing data acquired by the sensing data acquisition unit 121 may be used not only for inference but also as learning data.

[0089] When the sensing data acquired by the sensing data acquisition unit 121 is used as learning data, for example, the sensing data is used in the next unsupervised machine learning. Alternatively, each time the sensing data acquisition unit 121 acquires sensing data, the learning execution unit 113 may execute unsupervised machine learning using learning data including the acquired sensing data.

[0090] The inner state determination unit 124 acquires the sensing data output from the sensing data acquisition unit 121. The inner state determination unit 124 also acquires the trained learning model stored in the learning model storage unit 115 of the learning unit 11. The inner state determination unit 124 determines the inner state of the target user based on the acquired sensing data and the trained learning model. The inner state determination unit 124 outputs information indicating the inner state of the identified target user to the determination result output unit 13.

[0091] Specifically, for example, the internal state determination unit 124 inputs the acquired sensing data into a trained learning model and identifies a cluster to which the input sensing data belongs. The internal state determination unit 124 identifies the internal state assigned to the identified cluster. The internal state determination unit 124 outputs information indicating the internal state of the identified target user to the determination result output unit 13.

[0092] The inner surface state determination unit 124 is a functional unit realized by a processor (not shown), such as a CPU, included in the determination device 10 a reading and executing a program stored in advance in a storage medium (not shown) included in the determination device 10 a. The sensing data acquisition unit 121 includes a communication interface that communicates with the sensor device 40 via the network 50, for example.

[0093] The determination result output unit 13 acquires information indicating the inner state of the target user output from the inner state determination unit 124. The information acquired by the determination result output unit 13 from the inner state determination unit 124 includes, for example, user identification information that identifies the target user. The determination result output unit 13 transmits the acquired determination result information, including the information indicating the inner state of the target user and the user identification information, to the service providing device 20 via the network 50.

[0094] The determination result output unit 13 may transmit instruction information for instructing the service providing device 20 to the service providing device 20. The instruction information here refers to information for instructing the service providing device 20 to provide a service related to the target user based on the user identification information, with service content and timing according to the inner state of the target user.

[0095] The service providing device 20 provides a service related to the target user based on the user identification information in accordance with the received determination result information or instruction information. Note that the service related to the target user here refers to a service that controls the content and timing of the service provided depending on the inner state of the target user. Specific examples of services related to the target user will be described in detail in the embodiments described below (especially the fifth and sixth embodiments).

[0096] The determination result output unit 13 includes a communication interface that connects to the service providing device 20 via the network 50, for example.

[0097] Note that the determination result output unit 13 may be a device that displays information indicating the inner state of the target user. In this case, the determination result output unit 13 is configured to include a display device such as a liquid crystal display (LCD), an organic electroluminescence (EL) display, or a cathode ray tube (CRT). In this case, for example, an operator of the service providing device 20 refers to the information indicating the inner state of the target user displayed on the determination result output unit 13 and manually inputs the information indicating the inner state into the service providing device 20. As a result, a service is provided in accordance with the determined inner state of the target user.

[0098] [First Operation Example of Discrimination Device] Hereinafter, an example of the operation of the discrimination device 10a during unsupervised machine learning learning and an example of the operation during inference of the inner state of the target user will be described.

[0099] 3 is a flowchart showing the operation of the discrimination device 10a during learning according to the first embodiment of the present invention. The operation of the discrimination device 10a shown in the flowchart of FIG. 3 is started when training data that does not include a correct label is input to the training data acquisition unit 111 from, for example, an external device (not shown).

[0100] The learning data acquisition unit 111 of the learning unit 11 acquires learning data that does not include a correct label (step S001). As described above, the learning data here refers to sensing data such as image data captured with a camera of the body of a target user who is the subject of inner state discrimination, audio data collected with a microphone of the target user's conversation, or vital data measured with a vital sensor on the target user's vital state. The learning data storage unit 112 stores the learning data acquired by the learning data acquisition unit 111 (step S002).

[0101] The learning execution unit 113 acquires the learning data stored in the learning data storage unit 112. The learning execution unit 113 performs unsupervised machine learning using the acquired learning data, thereby clustering the learning data (step S003).

[0102] Each cluster classified by clustering is interpreted by the internal state registration unit 114. For example, the operator or target user refers to information indicating the clustering results obtained by unsupervised machine learning by the learning execution unit 113, and inputs information indicating the internal state to be assigned to each cluster to the internal state registration unit 114. The internal state registration unit 114 outputs the input information indicating the internal state to the learning execution unit 113, and the learning execution unit 113 associates the input information indicating the internal state with each cluster obtained as a result of the clustering (step S004).

[0103] The learning model storage unit 115 stores the trained learning model generated as a result of the unsupervised machine learning by the learning execution unit 113 (step S005). This completes the learning operation of the discrimination device 10a shown in the flowchart of FIG.

[0104] 4 is a flowchart showing the operation of the discrimination device 10a during inference according to the first embodiment of the present invention. The operation of the discrimination device 10a shown in the flowchart of FIG. 4 is started, for example, when sensing data output from a certain sensor device 40 is input to the sensing data acquisition unit 121.

[0105] The sensing data acquisition unit 121 of the inference unit 12 acquires sensing data from the sensor device 40 (step S006). As described above, the sensing data here refers to sensing data such as image data captured with a camera of the body of a target user who is the subject of inner state determination, audio data collected with a microphone of the target user's conversation, or vital data measured with a vital sensor on the target user's vital state. The sensing data acquisition unit 121 outputs the acquired sensing data to the inner state determination unit 124.

[0106] The inner state determination unit 124 acquires the sensing data output from the sensing data acquisition unit 121. The inner state determination unit 124 also acquires the trained learning model stored in the learning model storage unit 115 of the learning unit 11 (step S007). The inner state determination unit 124 inputs the acquired sensing data into the trained learning model and identifies a cluster to which the input sensing data belongs. The inner state determination unit 124 identifies the inner state assigned to the identified cluster, thereby determining the inner state of the target user (step S008). The inner state determination unit 124 outputs information indicating the inner state of the target user to the determination result output unit 13 as a determination result.

[0107] The determination result output unit 13 acquires information indicating the inner state of the target user output from the inner state determination unit 124. The information acquired by the determination result output unit 13 from the inner state determination unit 124 includes user identification information. The determination result output unit 13 outputs the acquired determination result information including the information indicating the inner state of the target user and the user identification information to the service providing device 20 via the network 50 (step S009). This completes the inference operation of the determination device 10a shown in the flowchart of FIG. 4.

[0108] [Second Configuration Example of Discrimination Device] The configuration of a discrimination device 10b that uses supervised machine learning AI will be described in detail below. Fig. 5 is a block diagram showing the functional configuration of the discrimination device 10b according to the first embodiment of the present invention. As shown in Fig. 5, the discrimination device 10b includes a learning unit 11b, an inference unit 12b, and a discrimination result output unit 13.

[0109] The learning unit 11b acquires learning data and performs supervised machine learning to generate a trained learning model that enables output of information indicating a determination result of the target user's inner state in response to input sensing data. Note that any method such as a deep neural network (DNN), a support vector machine, or linear regression can be used as the supervised machine learning method.

[0110] As shown in FIG. 5, the learning unit 11b includes a learning data acquiring unit 111b, a learning data storage unit 112b, a learning execution unit 113b, and a learning model storage unit 115b.

[0111] The learning data acquisition unit 111b acquires learning data (teacher data) including correct labels. The learning data here refers to data in which sensing data collected by the sensor device 40 is associated with information indicating the inner state of the target user (i.e., correct labels). As described above, the sensing data may include image data captured by a camera of the body of the target user, who is the subject of inner state determination, audio data collected by a microphone of the target user's conversation, or vital data measured by a vital sensor on the vital state of the target user. Note that the sensing data may include multiple types of data from among the image data, audio data, and vital data.

[0112] The learning data acquiring unit 111b acquires part or all of the learning data in a lump from, for example, an external server (not shown) that stores a large amount of learning data in which sensing data previously collected from each of the multiple sensor devices 40 is associated with information indicating the inner state of the target user (correct answer label). The learning data acquiring unit 111b stores the acquired learning data in the learning data storage unit 112b.

[0113] In addition, if the learning unit 11b is configured not to include the learning data storage unit 112b, the learning data acquisition unit 111b may output the acquired learning data directly to the learning execution unit 113b described below.

[0114] The learning data storage unit 112b stores the learning data acquired by the learning data acquisition unit 111b. The learning data storage unit 112b accumulates the learning data, for example, by linking it to user identification information that identifies a target user who is the target of inner state determination.

[0115] The learning execution unit 113b acquires the learning data stored in the learning data storage unit 112b, and generates a trained learning model by performing supervised machine learning using the acquired learning data.

[0116] The learning execution unit 113b stores the generated trained learning model in the learning model storage unit 115b. The learning model here includes, for example, information indicating a machine learning algorithm and parameter values.

[0117] The learning model storage unit 115b stores the learned learning model generated by the learning execution unit 113b.

[0118] The learning execution unit 113b is a functional unit realized by a processor (not shown), such as a CPU, included in the discrimination device 10b reading and executing a program pre-stored in a storage medium (not shown) included in the discrimination device 10b. The learning data storage unit 112b and the learning model storage unit 115b include, for example, a storage medium (not shown), such as a semiconductor memory (RAM), an EEPROM, or a ROM, a flash memory (SSD), a magnetic disk (HDD), or an optical disk, or any combination of these storage media. The learning data acquisition unit 111b includes, for example, a communication interface that communicates with an external device via a network 50.

[0119] As shown in FIG. 5, the inference unit 12b includes a sensing data acquisition unit 121b and an inner state determination unit 124b.

[0120] The sensing data acquisition unit 121b acquires sensing data from the sensor device 40. As described above, the sensing data here includes image data captured with a camera of the body of the target user who is the subject of inner state determination, audio data collected with a microphone of the target user's conversation, and vital data measured with a vital sensor on the target user's vital state. In other words, the sensing data acquired by the sensing data acquisition unit 121b is sensing data for inference that includes information on the same items as the learning sensing data that constitutes the learning data acquired by the learning data acquisition unit 111b described above. Note that the sensing data acquisition unit 121b and the learning data acquisition unit 111b may be configured as an integrated hardware device.

[0121] The sensing data acquisition unit 121b outputs the acquired sensing data to the internal state determination unit 124b. Note that the sensing data acquisition unit 121b may store learning data in the learning data storage unit 112b of the learning unit 11, the learning data correlating the acquired sensing data with information indicating an internal state identified by the internal state determination unit 124b, which will be described later, using the sensing data. In other words, the sensing data acquired by the sensing data acquisition unit 121b may be used not only for inference but also as learning data.

[0122] When the sensing data acquired by the sensing data acquisition unit 121b is used as learning data, the sensing data may be used in the next supervised machine learning. Alternatively, each time the internal state is identified by the internal state determination unit 124b, the acquired sensing data may be associated with information indicating the internal state to generate learning data, and the learning execution unit 113b may then execute supervised machine learning.

[0123] The inner state determination unit 124b acquires the sensing data output from the sensing data acquisition unit 121b. The inner state determination unit 124b also acquires the trained learning model stored in the learning model storage unit 115b of the learning unit 11b. The inner state determination unit 124b determines the inner state of the target user based on the acquired sensing data and the trained learning model. The inner state determination unit 124b outputs information indicating the inner state of the target user to the determination result output unit 13.

[0124] Specifically, for example, the inner state determination unit 124b inputs the acquired sensing data into a trained learning model, acquires information indicating the inner state output by the trained learning model, and identifies the inner state of the target user. The inner state determination unit 124b outputs the information indicating the identified inner state of the target user to the determination result output unit 13.

[0125] The inner surface state determination unit 124b is a functional unit realized by a processor (not shown), such as a CPU, included in the determination device 10b reading and executing a program stored in advance in a storage medium (not shown) included in the determination device 10b. The sensing data acquisition unit 121b includes a communication interface that communicates with the sensor device 40 via the network 50, for example.

[0126] The determination result output unit 13 acquires information indicating the inner state of the target user output from the inner state determination unit 124 b. The information acquired by the determination result output unit 13 from the inner state determination unit 124 b includes, for example, user identification information that identifies the target user. The determination result output unit 13 transmits the acquired determination result information, including the information indicating the inner state of the target user and the user identification information, to the service providing device 20 via the network 50.

[0127] The determination result output unit 13 may transmit instruction information for instructing the service providing device 20 to the service providing device 20. As described above, the instruction information here refers to information that instructs the service providing device 20 to provide a service related to the target user based on the user identification information, with service content and timing according to the inner state of the target user.

[0128] The service providing device 20 provides a service related to the target user based on the user identification information in accordance with the received determination result information or instruction information. As described above, the service related to the target user here refers to a service that is provided by controlling the content and timing of provision according to the inner state of the target user. Specific examples of services related to the target user will be described in detail in the embodiments described below (especially the fifth and sixth embodiments).

[0129] The determination result output unit 13 includes a communication interface that connects to the service providing device 20 via the network 50, for example.

[0130] The determination result output unit 13 may be a device that displays information indicating the inner state of the target user. In this case, the determination result output unit 13 is configured to include a display device such as a liquid crystal display, an organic EL display, or a CRT. In this case, for example, an operator of the service providing device 20 refers to the information indicating the inner state of the target user displayed on the determination result output unit 13 and manually inputs the information into the service providing device 20. As a result, the provision of a service is executed in accordance with the determined inner state of the target user.

[0131] [Second Operation Example of Discrimination Device] Hereinafter, an example of the operation of the discrimination device 10b during supervised machine learning learning and an example of the operation during inference of the inner state of the target user will be described.

[0132] 6 is a flowchart showing the operation of the discrimination device 10b during learning according to the first embodiment of the present invention. The operation of the discrimination device 10b shown in the flowchart of FIG. 6 is started when training data (i.e., teacher data) including correct labels is input to the training data acquisition unit 111b from, for example, an external device (not shown).

[0133] The learning data acquisition unit 111b of the learning unit 11b acquires learning data including correct labels (step S101). As described above, the learning data here is, for example, data in which sensing data collected by the sensor device 40 is associated with information indicating the inner state of the target user (i.e., correct labels). The learning data storage unit 112b stores the learning data acquired by the learning data acquisition unit 111b (step S102).

[0134] The learning execution unit 113b acquires the learning data stored in the learning data storage unit 112b, and performs supervised machine learning using the acquired learning data (step S103).

[0135] The learning model storage unit 115b stores the trained learning model generated by the learning execution unit 113b as a result of the supervised machine learning (step S104). This completes the learning operation of the discrimination device 10b shown in the flowchart of FIG.

[0136] 7 is a flowchart showing the operation of the discrimination device 10b during inference according to the first embodiment of the present invention. The operation of the discrimination device 10b shown in the flowchart of FIG. 7 is started, for example, when sensing data output from a certain sensor device 40 is input to the sensing data acquisition unit 121b.

[0137] The sensing data acquisition unit 121b of the inference unit 12b acquires sensing data from the sensor device 40 (step S105). As described above, the sensing data here refers to sensing data such as image data captured with a camera of the body of a target user who is the subject of inner state determination, audio data collected with a microphone of the target user's conversation, or vital data measured with a vital sensor on the target user's vital state. The sensing data acquisition unit 121b outputs the acquired sensing data to the inner state determination unit 124b.

[0138] The inner state determination unit 124b acquires the sensing data output from the sensing data acquisition unit 121b. The inner state determination unit 124b also acquires a trained learning model stored in the learning model storage unit 115b of the learning unit 11b (step S106). The inner state determination unit 124b inputs the acquired sensing data into the trained learning model, and identifies the inner state of the target user based on information output from the learning model (step S107). The inner state determination unit 124b outputs information indicating the identified inner state of the target user as a determination result to the determination result output unit 13.

[0139] The determination result output unit 13 acquires information indicating the inner state of the target user output from the inner state determination unit 124b. Furthermore, the information acquired by the determination result output unit 13 from the inner state determination unit 124b includes user identification information. The determination result output unit 13 outputs the acquired determination result information, including the information indicating the inner state of the target user and the user identification information, to the service providing device 20 via the network 50 (step S108). This completes the inference operation of the determination device 10b shown in the flowchart of FIG. 7.

[0140] [Third Configuration Example of Discrimination Device] The configuration of a discrimination device 10c that uses rule-based AI will be described in detail below. Fig. 8 is a block diagram showing the functional configuration of the discrimination device 10c according to the first embodiment of the present invention. As shown in Fig. 8, the discrimination device 10c includes a discrimination rule registration unit 11c, an inference unit 12c, and a discrimination result output unit 13.

[0141] The discrimination rule registration unit 11c stores discrimination rules input by, for example, a system operator, etc. The discrimination rules here refer to, for example, predetermined rules that determine what state the inner state of the target user is when a certain value of the sensing data is taken.

[0142] As shown in FIG. 8, the discrimination rule registration unit 11c includes a discrimination rule input unit 111c and a discrimination rule storage unit 115c.

[0143] The discrimination rule input unit 111c receives input of a discrimination rule from, for example, a system operator, etc. The discrimination rule input unit 111c stores the input discrimination rule in the discrimination rule storage unit 115c.

[0144] The discrimination rule storage unit 115c stores the discrimination rule acquired by the discrimination rule input unit 111c.

[0145] The discrimination rule storage unit 115c is configured to include, for example, a semiconductor memory such as RAM, EEPROM, and ROM, a flash memory such as SSD, a magnetic disk such as HDD, an optical disk, or other storage medium (not shown), or any combination of these storage media. The discrimination rule input unit 111c is configured to include, for example, a communication interface that communicates with an external device via the network 50.

[0146] As shown in FIG. 8, the inference unit 12c includes a sensing data acquisition unit 121c and an inner state determination unit 124c.

[0147] The sensing data acquisition unit 121c acquires sensing data from the sensor device 40. As described above, the sensing data here includes image data captured by a camera of the body of the target user who is the subject of inner state determination, audio data collected by a microphone of the target user's conversation, and vital data measured by a vital sensor on the target user's vital state. Note that the sensing data may include multiple types of data from among the image data, audio data, and vital data. The sensing data acquisition unit 121c outputs the acquired sensing data to the inner state determination unit 124c.

[0148] The inner state determination unit 124c acquires the sensing data output from the sensing data acquisition unit 121b. The inner state determination unit 124c refers to each determination rule stored in the determination rule storage unit 115c of the determination rule registration unit 11c, and identifies a determination rule whose condition is satisfied by the value of the acquired sensing data. The inner state determination unit 124c identifies the inner state defined by the identified determination rule as the inner state of the target user to be determined. The inner state determination unit 124c outputs information indicating the inner state of the target user to the determination result output unit 13.

[0149] The inner surface state determination unit 124c is a functional unit realized by a processor (not shown), such as a CPU, included in the determination device 10c reading and executing a program stored in advance in a storage medium (not shown) included in the determination device 10c. The sensing data acquisition unit 121c includes a communication interface that communicates with the sensor device 40 via the network 50, for example.

[0150] The determination result output unit 13 acquires information indicating the inner state of the target user output from the inner state determination unit 124c. The information acquired by the determination result output unit 13 from the inner state determination unit 124c includes, for example, user identification information that identifies the target user. The determination result output unit 13 transmits the acquired determination result information, including the information indicating the inner state of the target user and the user identification information, to the service providing device 20 via the network 50.

[0151] The determination result output unit 13 may transmit instruction information for instructing the service providing device 20 to the service providing device 20. As described above, the instruction information here refers to information that instructs the service providing device 20 to provide a service related to the target user based on the user identification information, with service content and timing according to the inner state of the target user.

[0152] The service providing device 20 provides a service related to the target user based on the user identification information in accordance with the received determination result information or instruction information. As described above, the service related to the target user here refers to a service that is provided by controlling the content and timing of provision according to the inner state of the target user. Specific examples of services related to the target user will be described in detail in the embodiments described below (especially the fifth and sixth embodiments).

[0153] The determination result output unit 13 includes a communication interface that connects to the service providing device 20 via the network 50, for example.

[0154] The determination result output unit 13 may be a device that displays information indicating the inner state of the target user. In this case, the determination result output unit 13 is configured to include a display device such as a liquid crystal display, an organic EL display, or a CRT. In this case, for example, an operator of the service providing device 20 refers to the information indicating the inner state of the target user displayed on the determination result output unit 13 and manually inputs the information into the service providing device 20. As a result, the provision of a service is executed in accordance with the determined inner state of the target user.

[0155] [Third Operation Example of the Discrimination Device] Hereinafter, an example of the operation of the discrimination device 10c when registering a discrimination rule and an example of the operation when inferring the inner state of the target user will be described.

[0156] 9 is a flowchart showing the operation of the discrimination device 10c according to the first embodiment of the present invention when registering a discrimination rule. The operation of the discrimination device 10c shown in the flowchart of FIG. 9 is started when a discrimination rule is input to the discrimination rule registration unit 11c from, for example, an external device (not shown).

[0157] The discrimination rule input unit 111c of the discrimination rule registration unit 11c accepts input of a discrimination rule by, for example, a system operator or the like (step S201). As described above, the discrimination rule here is, for example, a rule that defines what state the inner state of the target user is determined to be when what value of the sensing data is.

[0158] The discrimination rule input unit 111c stores the input discrimination rule in the discrimination rule storage unit 115c. The discrimination rule storage unit 115c stores the discrimination rule acquired by the discrimination rule input unit 111c (step S202). This completes the operation of the discrimination device 10c when registering a discrimination rule, as shown in the flowchart of FIG. 9.

[0159] 10 is a flowchart showing the operation of the discrimination device 10c during inference according to the first embodiment of the present invention. The operation of the discrimination device 10c shown in the flowchart of FIG. 10 is started, for example, when sensing data output from a certain sensor device 40 is input to the sensing data acquisition unit 121c.

[0160] The sensing data acquisition unit 121c of the inference unit 12c acquires sensing data from the sensor device 40 (step S203). As described above, the sensing data here refers to sensing data such as image data captured with a camera of the body of the target user who is the subject of inner state determination, audio data collected with a microphone of the target user's conversation, or vital data measured with a vital sensor on the target user's vital state. The sensing data acquisition unit 121c outputs the acquired sensing data to the inner state determination unit 124c.

[0161] The inner state determination unit 124b acquires the sensing data output from the sensing data acquisition unit 121c. The inner state determination unit 124c refers to each determination rule stored in the determination rule storage unit 115c of the determination rule registration unit 11c (step S204) and identifies a determination rule whose condition is satisfied by the value of the acquired sensing data (step S205). The inner state determination unit 124c identifies the inner state defined by the identified determination rule as the inner state of the target user to be determined. The inner state determination unit 124c outputs information indicating the inner state of the target user to the determination result output unit 13.

[0162] The determination result output unit 13 acquires information indicating the inner state of the target user output from the inner state determination unit 124c. Furthermore, the information acquired by the determination result output unit 13 from the inner state determination unit 124c includes user identification information. The determination result output unit 13 outputs the acquired determination result information, including the information indicating the inner state of the target user and the user identification information, to the service providing device 20 via the network 50 (step S206). This completes the inference operation of the determination device 10b shown in the flowchart of FIG. 10 .

[0163] [Authentication Function] In order to always provide appropriate services at appropriate times to target users, etc., it is desirable that the above-mentioned sensing data be continuously acquired and that the latest sensing data be always available (anytime sensing). In addition, for example, as in this embodiment, especially when the discrimination device 10 (10a to 10c) is a server that provides cloud-type services to an unspecified number of target users, it is required to have an authentication function that prevents sensing data of different target users from being mixed together within the server and prevents information leakage.

[0164] Therefore, for example, as described above, since the discrimination device 10 in this embodiment can use sensing data indicating the vital status of the target user, this sensing data may also be configured to be utilized in target user authentication.

[0165] For example, if sensing data indicating the target user's fingerprint can be obtained from the sensor device 40, the discrimination device 10 may store this sensing data and perform fingerprint authentication to authenticate the target user by matching the fingerprint. Furthermore, if sensing data indicating the target user's heartbeat can be obtained from the sensor device 40, the discrimination device 10 may store this sensing data and perform electrocardiogram authentication to authenticate the target user by matching an electrocardiogram. Furthermore, if sensing data indicating the target user's iris can be obtained from the sensor device 40, the discrimination device 10 may store this sensing data and perform iris authentication to authenticate the target user by matching the iris.

[0166] In addition, sensing data such as pupillary response, pupillary reflex, and electroencephalogram can also be used in both the user authentication and the determination of the internal state of the target user. Note that any conventional technology can be used as a method for verifying sensing data in the above user authentication, and for example, a verification method that performs wavelet transform, short-time Fourier transform, or the like on waveform data obtained from these sensing data can be used.

[0167] In the discrimination system 1 of the present embodiment, the type of sensor used to discriminate the inner state of the target user is not limited. In the discrimination system 1 of the present embodiment, any sensor can be used as long as it can objectively quantify the state of the target user.

[0168] As described above, the discrimination device 10 (10a to 10c) according to the first embodiment of the present invention is configured to discriminate the inner state of a target user, who is the subject of inner state discrimination, using sensing data such as image data captured by a camera of the body of the target user, audio data collected by a microphone of the target user's conversation, or vital data measured by a vital sensor on the target user's vital state. The discrimination device 10 is also configured to discriminate the inner state of the target user based on a learning model trained by unsupervised machine learning, a learning model trained by supervised machine learning, or a rule base, and the sensing data.

[0169] With this configuration, the discrimination system 1 according to the first embodiment of the present invention can more accurately discriminate the inner state of the target user even when there is a discrepancy between the outward emotional expression and the inner state. As a result, the discrimination system 1 can suppress a decrease in discrimination accuracy that may occur due to a discrepancy between the outward emotional expression and the inner state when discriminating a person's inner state through sensing.

[0170] Furthermore, the discrimination device 10a according to the first embodiment of the present invention discriminates the inner state of a target user using unsupervised machine learning, and therefore does not need to prepare a large amount of training data. In other words, there is no need to assign correct labels to a large amount of data. Therefore, the discrimination device 10a according to the first embodiment can more easily obtain ideal learning results without spending time or money, compared to, for example, the discrimination device 10b that discriminates the inner state of a target user using supervised machine learning.

[0171] Furthermore, the discrimination device 10b according to the first embodiment of the present invention discriminates the inner state of the target user using supervised machine learning, which learns using teacher data to which correct labels have been assigned. Therefore, the discrimination device 10b according to the first embodiment can discriminate the inner state of the target user more accurately with higher learning accuracy than, for example, the discrimination device 10a, which discriminates the inner state of the target user using unsupervised machine learning.

[0172] Furthermore, the discrimination device 10c according to the first embodiment of the present invention discriminates the inner state of the target user using rule-based AI. Therefore, compared to the discrimination devices 10a and 10b that discriminate the inner state of the target user using machine learning, the discrimination device 10c according to the first embodiment can save time and training data required for learning, and can obtain highly interpretable discrimination results.

[0173] Second Embodiment A second embodiment of the present invention will now be described.

[0174] [Configuration of the Discrimination System] The configuration of a discrimination system according to a second embodiment of the present invention will be described below. The overall configuration of the discrimination system according to the second embodiment is the same as that of the discrimination system 1 according to the first embodiment shown in FIG. 1, except that the discrimination devices 10 (10a to 10c) are replaced with discrimination device 10d. Therefore, descriptions of the service providing device 20, sensor device 40, network 50, etc. will be omitted here. The discrimination system according to the second embodiment, like the discrimination system 1 according to the first embodiment shown in FIG. 1, is an information system that discriminates the inner state of a target user using discrimination device 10d and performs information processing using the discrimination results.

[0175] [Configuration Example of Discrimination Device] The configuration of a discrimination device 10d that uses supervised machine learning AI will be described in detail below. Fig. 11 is a block diagram showing the functional configuration of the discrimination device 10d according to a second embodiment of the present invention. As shown in Fig. 11, the discrimination device 10d includes a learning unit 11d, an inference unit 12d, and a discrimination result output unit 13.

[0176] The learning unit 11d acquires learning data and performs supervised machine learning to generate a trained learning model that enables output of information indicating a determination result of the inner state of the target user who is the subject of observation by the sensor device 40 in response to input of sensing data transmitted from the sensor device 40. Note that any method such as DNN, support vector machine, or linear regression can be used as the supervised machine learning method.

[0177] In the first embodiment described above, the discrimination device 10b using supervised machine learning AI is configured to prepare learning data in advance, in which correct labels are assigned to sensing data, and to perform learning in advance using the learning data. In contrast, the discrimination device 10d in the second embodiment described below detects changes in the values ​​of sensing data, and when a change is detected, confirms (queries) the target user about their inner state. The discrimination device 10d then generates learning data to which information indicating the confirmed inner state is assigned as a correct label, and further performs supervised machine learning. This enables the discrimination device 10d to discriminate even new inner states that arise in the target user.

[0178] As shown in FIG. 11, the learning unit 11d is composed of a sensing data acquisition unit 111d, a change detection unit 116, a sensing data storage unit 117, an inner surface state confirmation unit 118, a learning data storage unit 112d, a learning execution unit 113d, and a learning model storage unit 115d.

[0179] The sensing data acquisition unit 111d acquires various types of sensing data transmitted from the sensor devices 40 via the network 50. The sensing data acquired by the sensing data acquisition unit 111d becomes part of the data used as learning data in the learning execution unit 113d, which will be described later. The sensing data acquisition unit 111d outputs the acquired sensing data to the change detection unit 116.

[0180] As described above, the sensing data acquisition unit 111d acquires sensing data such as image data captured by a camera of the body of the target user, who is the subject of the internal state determination, audio data collected by a microphone of the target user's conversation, or vital data measured by a vital sensor on the target user's vital state.

[0181] The sensing data may include multiple types of data among the image data, voice data, and vital data. The sensing data acquisition unit 111d may acquire any items of sensing data, but it is considered that the more items of sensing data are acquired, the higher the accuracy of determining the inner state of the target user.

[0182] The change detection unit 116 acquires the sensing data output from the sensing data acquisition unit 111 d and checks whether data similar to past sensing data regarding the target user corresponding to the acquired sensing data is stored in the sensing data storage unit 117 (described later).

[0183] To perform the above processing, the sensing data must include user identification information, such as a user ID (identifier), that identifies the target user. In this embodiment, as described below, the change detection unit 116 is configured to detect time-series changes in the feature amounts of the sensing data for each target user. However, if a unique target user is determined for each sensor device 40 (i.e., if one sensor device is configured to observe only one specific target user), the time-series changes in the feature amounts of the sensing data may be detected for each sensor device 40 (e.g., for each terminal ID). In this case, the sensing data must include identification information, such as a terminal ID, that identifies the sensor device 40.

[0184] If past sensing data is not stored in the sensing data storage unit 117 (i.e., if this is the first time sensing data has been acquired), the change detection unit 116 outputs the acquired sensing data to the inner surface condition confirmation unit 118 and stores the acquired sensing data in the sensing data storage unit 117.

[0185] If past sensing data is stored in the sensing data storage unit 117 (i.e., if this is not the first time that sensing data has been acquired), the change detection unit 116 compares the past sensing data with the acquired sensing data. If the comparison results in a time-series change in the feature amount indicated by the sensing data, the change detection unit 116 outputs the acquired sensing data to the inner state confirmation unit 118 and overwrites the past sensing data stored in the sensing data storage unit 117 with the acquired sensing data.

[0186] If the comparison result does not detect a time-series change in the feature amount indicated by the sensing data, the change detection unit 116 discards the acquired sensing data. Note that the change detection unit 116 may determine that there has been a change in the sensing data if the difference between the feature amount indicated by the past sensing data and the feature amount indicated by the acquired sensing data exceeds a predetermined threshold.

[0187] The inner state confirmation unit 118 acquires the sensing data output from the change detection unit 116. The inner state confirmation unit 118 inquires of a user terminal (not shown) corresponding to the acquired sensing data about the current inner state of the target user, for example, via the network 50 or the like.

[0188] The user terminal (not shown) may be a terminal device carried by a user, such as a smartphone or a wearable device. The user terminal (not shown) and the sensor device 40 may be the same device.

[0189] The inquiry method is arbitrary, but may be, for example, an inquiry method that allows the target user to select one of a "stressed state" and a "relaxed state." Alternatively, the inquiry method may be, for example, an inquiry method that allows the target user to select the inner state that is closest to the current inner state from a plurality of predetermined inner states (for example, from the above-mentioned five-level inner state).

[0190] The inner state confirmation unit 118 acquires information indicating a response to an inquiry about the current inner state of the target user, transmitted from a user terminal (not shown). The inner state confirmation unit 118 associates the sensing data output from the change detection unit 116 with information indicating the inner state of the target user based on the response transmitted from the sensor device 40. This generates learning data (i.e., teacher data) including correct labels. The inner state confirmation unit 118 stores the generated learning data including the correct labels in the learning data storage unit 112d. The learning data storage unit 112d stores the learning data including the correct labels.

[0191] By repeating this process, new training data including correct labels is accumulated in the training data storage unit 112d every time a change occurs in the sensing data. As described above, the training data including correct labels is data in which sensing data is associated with information indicating the inner state of the target user.

[0192] The learning execution unit 113d acquires learning data including correct labels stored in the learning data storage unit 112d. Note that the learning execution unit 113d may acquire the learning data when a predetermined amount of learning data has accumulated in the learning data storage unit 112d. The learning execution unit 113d generates a trained learning model by performing supervised machine learning using the acquired learning data. In other words, the learning model generated here is a learning model that outputs a discrimination result of the inner state of the target user in response to input sensing data.

[0193] The learning execution unit 113d stores the generated trained learning model in the learning model storage unit 115d. The learning model here includes, for example, information indicating the machine learning algorithm and parameter values.

[0194] The learning model storage unit 115d stores the learned learning model generated by the learning execution unit 113d.

[0195] The change detection unit 116, the inner surface state confirmation unit 118, and the learning execution unit 113d are functional units realized by a processor (not shown), such as a CPU, included in the discrimination device 10d reading and executing a program pre-stored in a storage medium (not shown) included in the discrimination device 10d. The sensing data storage unit 117, the learning data storage unit 112d, and the learning model storage unit 115d are configured to include, for example, a storage medium (not shown), such as a semiconductor memory (RAM), an EEPROM, or a ROM, a flash memory (SSD), a magnetic disk (HDD), or an optical disk, or any combination of these storage media. The sensing data acquisition unit 111d and the inner surface state confirmation unit 118 are configured to include a communication interface that communicates with an external device via, for example, a network 50.

[0196] As shown in FIG. 11, the inference unit 12d includes a sensing data acquisition unit 121d and an inner state determination unit 124d.

[0197] The sensing data acquisition unit 121d acquires various sensing data transmitted from the sensor devices 40 via the network 50. The items of sensing data acquired by the sensing data acquisition unit 121d are, for example, the same as the items of sensing data acquired by the sensing data acquisition unit 111d of the learning unit 11d. Note that the sensing data acquisition unit 121d and the sensing data acquisition unit 111d may be configured as an integrated device in terms of hardware. The sensing data acquisition unit 121d outputs the acquired sensing data to the inner surface state determination unit 124d.

[0198] The inner state determination unit 124d acquires the sensing data output from the sensing data acquisition unit 121d. The inner state determination unit 124d also acquires the trained learning model stored in the learning model storage unit 115d of the learning unit 11d. The inner state determination unit 124d identifies the current inner state of the target user based on the acquired sensing data and the trained learning model. The inner state determination unit 124d outputs information indicating the identified current inner state of the target user as a determination result to the determination result output unit 13.

[0199] The inner surface state determination unit 124d is a functional unit realized by a processor (not shown), such as a CPU, included in the determination device 10d reading and executing a program stored in advance in a storage medium (not shown) included in the determination device 10d. The sensing data acquisition unit 121d includes a communication interface that communicates with the sensor device 40 via the network 50, for example.

[0200] The determination result output unit 13 acquires information indicating the inner state of the target user output from the inner state determination unit 124d. The information acquired by the determination result output unit 13 from the inner state determination unit 124d includes, for example, user identification information for identifying the target user. The determination result output unit 13 transmits the acquired determination result information, including the information indicating the inner state of the target user and the user identification information, to the service providing device 20 via the network 50.

[0201] The determination result output unit 13 may transmit instruction information for instructing the service providing device 20 to the service providing device 20. As described above, the instruction information here refers to information for instructing the service providing device 20 to provide a service related to the target user based on the user identification information, with service content and timing according to the inner state of the target user.

[0202] [Example of Operation of Discrimination Device] Hereinafter, an example of operation of the discrimination device 10d during supervised machine learning learning and an example of operation during inference of the inner state of the target user will be described.

[0203] 12 is a flowchart showing the operation of the discrimination device 10d during learning according to the second embodiment of the present invention. The operation of the discrimination device 10d shown in the flowchart of FIG. 12 is started when sensing data is input from the sensor device 40 to the sensing data acquisition unit 111d, for example.

[0204] The sensing data acquisition unit 111d of the learning unit 11d acquires various sensing data transmitted from the sensor devices 40 via the network 50 (step S301). For example, as described above, the sensing data acquisition unit 111d acquires sensing data such as image data captured with a camera of the body of a target user who is the subject of inner state determination, audio data collected with a microphone of the target user's conversation, or vital data measured with a vital sensor on the target user's vital state. The sensing data acquisition unit 111d outputs the acquired sensing data to the change detection unit 116.

[0205] The change detection unit 116 acquires the sensing data output from the sensing data acquisition unit 111d and checks whether past sensing data related to the target user corresponding to the acquired sensing data is stored in the sensing data storage unit 117 (step S302).

[0206] If past sensing data is not stored in the sensing data storage unit 117 (step S303, NO), the change detection unit 116 outputs the acquired sensing data to the inner surface state confirmation unit 118 and stores the acquired sensing data in the sensing data storage unit 117 (step S304).

[0207] If past sensing data is stored in the sensing data storage unit 117 (step S303: YES), the change detection unit 116 compares the past sensing data with the acquired sensing data (step S305).

[0208] If the comparison results in a time series change in the feature amount indicated by the sensing data (YES in step S306), the change detection unit 116 outputs the acquired sensing data to the inner surface state confirmation unit 118 and overwrites the previous sensing data stored in the sensing data storage unit 117 with the acquired sensing data (step S307).On the other hand, if the comparison results in no change in the sensing data (NO in step S306), the change detection unit 116 discards the acquired sensing data (step S308).

[0209] The inner state confirmation unit 118 acquires the sensing data output from the change detection unit 116. The inner state confirmation unit 118 inquires of a user terminal (not shown) corresponding to the acquired sensing data about the current inner state of the target user, for example, via the network 50 (step S309).

[0210] The inner state confirmation unit 118 acquires information indicating a response to an inquiry about the current inner state of the target user, transmitted from a user terminal (not shown). The inner state confirmation unit 118 associates the sensing data output from the change detection unit 116 with information indicating the inner state of the target user based on the response transmitted from the sensor device 40. This generates learning data (i.e., teacher data) including correct labels (step S310). The inner state confirmation unit 118 stores the generated learning data including the correct labels in the learning data storage unit 112d (step S311).

[0211] The learning unit 11d repeats the above-described processes from step S301 to step S311 until a predetermined amount of learning data is accumulated. When the predetermined amount of learning data has been accumulated (step S312: YES), the learning execution unit 113d performs supervised machine learning using the learning data including the correct labels stored in the learning data storage unit 112d to generate a trained learning model (step S313).

[0212] The learning execution unit 113d stores the generated trained learning model in the learning model storage unit 115d (step S314). This completes the learning operation of the discrimination device 10d shown in the flowchart of FIG.

[0213] 13 is a flowchart showing the operation of the discrimination device 10d during inference according to the second embodiment of the present invention. The operation of the discrimination device 10d shown in the flowchart of FIG. 13 is started, for example, when sensing data is input from the sensor device 40 to the sensing data acquisition unit 121d.

[0214] The sensing data acquisition unit 121d of the inference unit 12d acquires various sensing data transmitted from the sensor device 40 via the network 50 (step S315). The sensing data acquisition unit 121d outputs the acquired sensing data to the inner state determination unit 124d.

[0215] The inner state determination unit 124d acquires the sensing data output from the sensing data acquisition unit 121d. The inner state determination unit 124d also acquires the trained learning model stored in the learning model storage unit 115d of the learning unit 11d (step S316). The inner state determination unit 124d determines the current inner state of the target user based on the acquired sensing data and the trained learning model (step S317). The inner state determination unit 124d outputs information indicating the current inner state of the target user as a determination result to the determination result output unit 13.

[0216] The determination result output unit 13 acquires information indicating the inner state of the target user output from the inner state determination unit 124. The information acquired by the determination result output unit 13 from the inner state determination unit 124 includes user identification information. The determination result output unit 13 transmits the acquired determination result information including the information indicating the inner state of the target user and the user identification information to the service providing device 20 via the network 50 (step S318). This completes the operation of the determination device 10d during inference shown in the flowchart of FIG. 13.

[0217] As described above, the discrimination device 10d according to the second embodiment of the present invention is configured to discriminate the inner state of a target user, who is the subject of discrimination of the inner state, using sensing data such as image data obtained by capturing an image of the body of the target user with a camera, audio data obtained by collecting the target user's conversational voice with a microphone, or vital data obtained by measuring the target user's vital state with a vital sensor. The discrimination device 10d is also configured to discriminate the inner state of the target user based on a learning model trained by supervised machine learning and the sensing data.

[0218] With this configuration, the discrimination system according to the second embodiment of the present invention can more accurately discriminate the inner state of the target user even when there is a discrepancy between the outward emotional expression and the inner state. As a result, the discrimination system can suppress a decrease in discrimination accuracy that may occur due to a discrepancy between the outward emotional expression and the inner state when discriminating a person's inner state through sensing.

[0219] Furthermore, the discrimination device 10d according to the second embodiment of the present invention discriminates the inner state of the target user using supervised machine learning, which learns using teacher data to which correct labels have been assigned. Therefore, the discrimination device 10d according to the second embodiment can discriminate the inner state of the target user more accurately with higher learning accuracy than, for example, the discrimination device 10a according to the first embodiment, which discriminates the inner state of the target user using unsupervised machine learning.

[0220] Furthermore, the discrimination device 10d according to the second embodiment of the present invention is configured to detect changes in the values ​​of the sensing data and, when a change is detected, to confirm (inquire about) the inner state of the target user. The discrimination device 10d is then configured to generate learning data to which information indicating the inner state, which is the confirmation result, is assigned as a correct answer label, and to further perform supervised machine learning. With such a configuration, the discrimination device 10d can also discriminate new inner states that arise in the target user.

[0221] Third Embodiment A third embodiment of the present invention will now be described.

[0222] [Configuration of the Discrimination System] The configuration of a discrimination system according to a third embodiment of the present invention will be described below. The overall configuration of the discrimination system according to the third embodiment is the same as that of the discrimination system 1 according to the first embodiment shown in FIG. 1, except that the discrimination devices 10 (10a to 10c) are replaced with discrimination device 10e. Therefore, descriptions of the service providing device 20, sensor device 40, network 50, etc. will be omitted here. The discrimination system according to the third embodiment, like the discrimination system 1 according to the first embodiment shown in FIG. 1, is an information system that discriminates the inner state of a target user using discrimination device 10e and performs information processing using the discrimination results.

[0223] [Configuration of the Discrimination Device] The configuration of a discrimination device 10e that uses supervised machine learning AI will be described in detail below. Fig. 14 is a block diagram showing the functional configuration of the discrimination device 10e according to a third embodiment of the present invention. As shown in Fig. 14, the discrimination device 10e includes a learning unit 11e, an inference unit 12e, and a discrimination result output unit 13.

[0224] The learning unit 11e acquires learning data and performs supervised machine learning to generate a trained learning model that enables output of information indicating a determination result of the inner state of the target user who is the subject of observation by the sensor device 40 in response to input sensing data transmitted from the sensor device 40. Note that any method such as DNN, support vector machine, or linear regression can be used as the supervised machine learning method.

[0225] The discrimination device 10d in the third embodiment described above was configured to detect a change in the value of sensing data, and when a change was detected, to obtain a correct label by confirming (inquiring about) the inner state of the target user, thereby generating teacher data. In contrast, the generation of teacher data by the learning unit 11e in the third embodiment described below is triggered by the discrimination device 10e acquiring information indicating the current inner state of the target user from a user terminal (not shown).

[0226] That is, in the third embodiment, the execution of learning is triggered when the target user voluntarily provides the discrimination device 10e with information indicating an inner state corresponding to a correct label in the training data. In this way, the configuration of the learning unit 11e in the third embodiment is different from the configuration of the learning unit 11d in the second embodiment, in which the execution of learning is triggered by detecting a change in the value of sensing data corresponding to an example problem in the training data.

[0227] As shown in FIG. 14, the learning unit 11e includes an inner state acquisition unit 118e, a learning data storage unit 112e, a learning execution unit 113e, and a learning model storage unit 115e.

[0228] The inner state acquisition unit 118e acquires information indicating the inner state of the target user at the current time transmitted from a user terminal (not shown) via the network 50. Note that the information indicating the inner state of the target user at the current time is generated, for example, by the target user determining their own inner state at the current time and voluntarily inputting the information into the user terminal (not shown). Note that the user terminal (not shown) and the sensor device 40 may be the same device.

[0229] The inner state acquisition unit 118e outputs user identification information of the target user who inputs information indicating the current inner state to the sensing data acquisition unit 111e. Therefore, the inner state acquisition unit 118e needs to be able to identify the target user who inputs information indicating the current inner state. For example, when a communication connection with a user terminal (not shown) into which information indicating the inner state has been input is established, the inner state acquisition unit 118e may acquire user identification information corresponding to the user terminal (not shown) from the user terminal (not shown).

[0230] The inner state acquisition unit 118e associates the acquired information indicating the inner state of the target user at the current time with the user identification information and stores them in the learning data storage unit 112d. The learning data storage unit 112d stores the information indicating the inner state of the target user in association with the user identification information.

[0231] The sensing data acquisition unit 111e acquires the user identification information output from the inner state acquisition unit 118e. The sensing data acquisition unit 111e transmits request information requesting the transmission of sensing data to the sensor device 40 that observes the user identified by the acquired user information. When the sensor device 40 acquires the request information transmitted from the sensing data acquisition unit 111e, it transmits the sensing data to the sensing data acquisition unit 111e.

[0232] The sensing data acquisition unit 111e acquires, via the network 50, sensing data returned from the sensor device 40 in response to the transmission of request information. The sensing data acquisition unit 111e acquires sensing data such as, for example, image data of the body of a target user who is the subject of inner state determination, captured by a camera, audio data of the target user's conversation collected by a microphone, or vital data measuring the vital state of the target user by a vital sensor.

[0233] The sensing data may include multiple types of data among the image data, voice data, and vital data. The sensing data acquisition unit 111d may acquire any items of sensing data, but it is considered that the more items of sensing data are acquired, the higher the accuracy of determining the inner state of the target user.

[0234] The sensing data acquisition unit 111e stores the acquired sensing data in the learning data storage unit 112e. At this time, the sensing data acquisition unit 111e reads information indicating an inner state associated with user identification information that is the same as user identification information that identifies the target user observed by the sensor device 40 that is the sender of the sensing data, stored in the learning data storage unit 112e. The sensing data acquisition unit 111e then associates the read information indicating the inner state with the acquired sensing data and stores the result in the learning data storage unit 112e. As a result, teacher data is generated by associating information indicating the current inner state (i.e., correct label) of the same target user (i.e., user identification information that matches) with the sensing data.

[0235] Note that, if the learning unit 11e is configured to periodically acquire sensing data from the sensor device 40 at sufficiently short intervals, it is not necessary to request the sensor device 40 to transmit sensing data again in response to acquisition of information indicating the inner state. That is, in this case, for example, when acquiring information indicating the inner state, the learning unit 11e may generate teacher data by associating the information indicating the inner state with the most recently acquired sensing data, and store the teacher data in the learning data storage unit 112e.

[0236] The training data storage unit 112e stores training data including correct labels. By repeating this process, training data including correct labels is accumulated in the training data storage unit 112d. As described above, the training data including correct labels is training data in which sensing data is associated with information indicating the inner state of the target user.

[0237] The learning execution unit 113e acquires learning data including correct labels stored in the learning data storage unit 112e. The learning execution unit 113e may acquire the learning data when a predetermined amount of learning data has been accumulated in the learning data storage unit 112d. The learning execution unit 113e generates a trained learning model by performing supervised machine learning using the acquired learning data. In other words, the generated learning model outputs a discrimination result of the inner state of the target user in response to input sensing data.

[0238] The learning execution unit 113e stores the generated trained learning model in the learning model storage unit 115e. The learning model here includes, for example, information indicating a machine learning algorithm and parameter values.

[0239] The learning model storage unit 115e stores the learned learning model generated by the learning execution unit 113e.

[0240] The learning execution unit 113e is a functional unit realized by a processor (not shown), such as a CPU, included in the discrimination device 10e reading and executing a program pre-stored in a storage medium (not shown) included in the discrimination device 10e. The learning data storage unit 112e and the learning model storage unit 115e include, for example, a storage medium (not shown), such as a semiconductor memory (RAM), an EEPROM, or a ROM, a flash memory (SSD), a magnetic disk (HDD), or an optical disk, or any combination of these storage media. The inner surface state acquisition unit 118e and the sensing data acquisition unit 111e include, for example, a communication interface for communication with an external device via a network 50.

[0241] As shown in FIG. 14, the inference unit 12 e includes a sensing data acquisition unit 121 e, an inner state determination unit 124 e, and an inner state confirmation unit 126 .

[0242] The sensing data acquisition unit 121e acquires various sensing data transmitted from the sensor devices 40 via the network 50. The items of sensing data acquired by the sensing data acquisition unit 121e are, for example, the same as the items of sensing data acquired by the sensing data acquisition unit 111e of the learning unit 11e. Note that the sensing data acquisition unit 121e and the sensing data acquisition unit 111e may be configured as an integrated device in terms of hardware. The sensing data acquisition unit 121e outputs the acquired sensing data to the inner surface state determination unit 124e.

[0243] The inner state determination unit 124e acquires the sensing data output from the sensing data acquisition unit 121e. The inner state determination unit 124e also acquires the trained learning model stored in the learning model storage unit 115e of the learning unit 11e. The inner state determination unit 124e determines the current inner state of the target user based on the acquired sensing data and the trained learning model.

[0244] The inference unit 12e may further perform the following process to confirm whether the inner state determined by the inner state determination unit 124e is actually correct as the current inner state of the target user. This confirmation process does not necessarily need to be performed all the time, and may be performed, for example, as needed or periodically, for the purpose of maintaining or improving the accuracy of determination of the target user's inner state.

[0245] The inner state determination unit 124e outputs the acquired sensing data and information indicating the determined inner state of the target user to the inner state confirmation unit 126.

[0246] The inner state confirmation unit 126 acquires the sensing data and information indicating the inner state of the target user output from the inner state determination unit 124e. When the inner state confirmation unit 126 acquires this information from the inner state determination unit 124e, it inquires about the current inner state of the target user from the user terminal (not shown) of the corresponding target user, for example, via the network 50. Therefore, the sensing data needs to include user identification information that identifies the target user. Note that the inquiry method is arbitrary, but may be, for example, an inquiry method in which the target user is prompted to select an arbitrary inner state from a plurality of inner state options and information indicating the selected inner state is returned as a response.

[0247] The inner state confirmation unit 126 acquires information indicating the current inner state of the target user, which is returned from the user terminal (not shown) as a response to the above inquiry. The inner state confirmation unit 126 compares the inner state of the target user based on the information acquired from the inner state determination unit 124e (i.e., the estimated inner state) with the inner state of the target user acquired from the user terminal (not shown) (i.e., the correct inner state). If the above two inner states of the target user match (i.e., if the determination result by the inner state determination unit 124d is correct), the inner state confirmation unit 126 outputs information indicating that the determination was correct to the inner state determination unit 124e.

[0248] On the other hand, if the inner states of the two target users do not match (i.e., if the determination result by the inner state determination unit 124e is incorrect), the inner state confirmation unit 126 stores data that associates information indicating the current inner state of the target user obtained from the sensor device 40 (i.e., the correct label) with the sensing data obtained from the inner state determination unit 124e as new teacher data in the learning data storage unit 112e of the learning unit 11e.

[0249] Then, if the inner states of the two target users do not match (i.e., if the determination result by the inner state determination unit 124e is incorrect), the inner state confirmation unit 126 outputs information indicating the current inner state of the target user obtained from the sensor device 40 (i.e., a correct label) to the inner state determination unit 124e.

[0250] In this way, the discrimination device 10e may further include a configuration for, after discriminating the inner state of the target user, inquiring about the target user's actual inner state to confirm whether the discrimination result was correct. If the discrimination result is incorrect, the discrimination device 10e may also include a configuration for adding, as new training data, learning data in which the confirmation result obtained from the target user is used as a correct answer label. This enables the discrimination device 10e to further improve the learning accuracy of supervised machine learning by the learning execution unit 113e.

[0251] When the inner state determination unit 124e obtains information indicating that the determination result is correct from the inner state confirmation unit 126, or when the target user is not inquired about their inner state, the inner state determination unit 124e outputs information indicating the current inner state of the target user as the determination result to the determination result output unit 13.

[0252] On the other hand, when the internal state determination unit 124e obtains information indicating the current internal state of the target user obtained from the sensor device 40 (i.e., a correct label) from the internal state confirmation unit 126 (i.e., when the determination result is incorrect), it outputs the information indicating the current internal state of the target user obtained from the sensor device 40 to the determination result output unit 13.

[0253] The inner surface state determination unit 124e and the inner surface state confirmation unit 126 are functional units that are realized when a processor (not shown), such as a CPU, included in the discrimination device 10e reads and executes a program that is pre-stored in a storage medium (not shown) included in the discrimination device 10e. The sensing data acquisition unit 121e and the inner surface state confirmation unit 126 are configured to include a communication interface that communicates with the sensor device 40 via the network 50, for example.

[0254] The determination result output unit 13 acquires information indicating the inner state of the target user output from the inner state determination unit 124e. The information acquired by the determination result output unit 13 from the inner state determination unit 124e includes, for example, user identification information for identifying the target user. The determination result output unit 13 transmits the acquired determination result information, including the information indicating the inner state of the target user and the user identification information, to the service providing device 20 via the network 50.

[0255] The determination result output unit 13 may transmit instruction information for instructing the service providing device 20 to the service providing device 20. As described above, the instruction information here refers to information for instructing the service providing device 20 to provide a service related to the target user based on the user identification information, with service content and timing according to the inner state of the target user.

[0256] [Example of Operation of Discrimination Device] Hereinafter, an example of operation of the discrimination device 10e during supervised machine learning learning and an example of operation during inference of the inner state of the target user will be described.

[0257] 15 is a flowchart showing the operation of the discrimination device 10e during learning according to the third embodiment of the present invention. The operation of the discrimination device 10e shown in the flowchart of FIG. 15 is started when information indicating the current inner state of the target user is input to the inner state acquisition unit 118e from, for example, a user terminal (not shown).

[0258] The inner state acquisition unit 118e of the learning unit 11e acquires information indicating the inner state of the target user at the current time transmitted from the sensor device 40 via the network 50 (step S401).

[0259] The inner state acquisition unit 118e outputs the user identification information of the target user who transmitted the information indicating the current inner state to the sensing data acquisition unit 111e. The inner state acquisition unit 118e also associates the acquired information indicating the current inner state of the target user with the user identification information of the target user, and stores the associated information in the learning data storage unit 112e. The learning data storage unit 112e stores the information indicating the inner state of the target user with the user identification information (step S402).

[0260] The sensing data acquisition unit 111e acquires user identification information identifying the target user output from the inner state acquisition unit 118e. The sensing data acquisition unit 111e transmits request information requesting transmission of sensing data to a user terminal (not shown) of the user identified by the acquired user identification information (step S403). When the sensor device 40 acquires the request information transmitted from the sensing data acquisition unit 111e, it transmits sensing data to the sensing data acquisition unit 111e.

[0261] The sensing data acquiring unit 111e acquires, via the network 50, sensing data returned from a user terminal (not shown) in response to the transmission of the request information (step S404). The sensing data acquiring unit 111e acquires sensing data such as, for example, image data of the body of a target user who is the subject of inner state determination, captured by a camera, audio data of the target user's conversation collected by a microphone, or vital data measured by a vital sensor on the vital state of the target user.

[0262] The sensing data acquisition unit 111e stores the acquired sensing data in the learning data storage unit 112e. At this time, the sensing data acquisition unit 111e reads out information indicating an inner state associated with user identification information identical to user identification information identifying a target user observed by the sensor device 40 that is the sender of the sensing data, stored in the learning data storage unit 112e. The sensing data acquisition unit 111e then associates the read information indicating the inner state with the acquired sensing data and stores the information in the learning data storage unit 112e (step S405). This associates the sensing data with information indicating the current inner state (i.e., the correct label) of the same target user (i.e., the user identification information matches), thereby generating training data. The learning data storage unit 112e stores the learning data including the correct label.

[0263] The learning execution unit 113e acquires learning data including the correct label stored in the learning data storage unit 112e. Note that the learning execution unit 113e may acquire the learning data when a predetermined amount of learning data has been accumulated in the learning data storage unit 112e. The learning execution unit 113e generates a trained learning model by performing supervised machine learning using the acquired learning data (step S406).

[0264] The learning execution unit 113e stores the trained learning model generated as a result of the supervised machine learning in the learning model storage unit 115e. The learning model storage unit 115e stores the trained learning model generated by the learning execution unit 113e (step S407). This completes the learning operation of the discrimination device 10e shown in the flowchart of FIG. 15.

[0265] 16 is a flowchart showing the operation of the discrimination device 10e during inference according to the third embodiment of the present invention. The operation of the discrimination device 10e shown in the flowchart of FIG. 16 is started, for example, when sensing data from a certain sensor device 40 is input to the sensing data acquisition unit 121e.

[0266] The sensing data acquisition unit 121e of the inference unit 12e acquires various sensing data transmitted from the sensor device 40 via the network 50 (step S408). The sensing data items acquired by the sensing data acquisition unit 121e are, for example, the same as the sensing data items acquired by the sensing data acquisition unit 111e of the learning unit 11e. The sensing data acquisition unit 121e outputs the acquired sensing data to the inner state determination unit 124e.

[0267] The inner state determination unit 124e acquires the sensing data output from the sensing data acquisition unit 121e. The inner state determination unit 124e also acquires the trained learning model stored in the learning model storage unit 115e of the learning unit 11e (step S409). The inner state determination unit 124e determines the current inner state of the target user based on the acquired sensing data and the trained learning model (step S410).

[0268] The inner state determination unit 124e outputs the acquired sensing data and information indicating the determined inner state of the target user to the inner state confirmation unit 126. The inner state confirmation unit 126 acquires the sensing data and the information indicating the inner state of the target user output from the inner state determination unit 124e. When acquiring this information from the inner state determination unit 124e, the inner state confirmation unit 126 inquires about the current inner state of the target user via, for example, the network 50, etc. (step S411).

[0269] The inner state confirmation unit 126 acquires information indicating the current inner state of the target user, which is returned from the user terminal (not shown) as a response to the inquiry (step S412). The inner state confirmation unit 126 compares the inner state of the target user based on the information acquired from the inner state determination unit 124e (i.e., the estimated inner state) with the inner state of the target user acquired from the user terminal (not shown) (i.e., the correct inner state) (step S413).

[0270] If the inner states of the two target users match (step S414: YES), the inner state confirmation unit 126 outputs information indicating that the determination result of the target user's inner state at the current time was correct to the inner state determination unit 124e. When the inner state determination unit 124e obtains information indicating that the determination result was correct from the inner state confirmation unit 126, it outputs information indicating the target user's inner state at the current time as the determination result to the determination result output unit 13.

[0271] The determination result output unit 13 acquires information indicating the inner state of the target user based on the result of the above determination, output from the inner state determination unit 124e. The information acquired by the determination result output unit 13 from the inner state determination unit 124e includes the user identification information of the target user. The determination result output unit 13 outputs the acquired determination result information, including the information indicating the inner state of the target user and the user identification information, to the service providing device 20 via the network 50 (step S415).

[0272] On the other hand, if the inner states of the two target users do not match (step S414, NO), the inner state confirmation unit 126 associates information indicating the current inner state of the target user (i.e., the correct label) obtained from the user terminal (not shown) with the sensing data obtained from the inner state determination unit 124e to create new teacher data, and adds it to the existing teacher data stored in the learning data storage unit 112e of the learning unit 11e (step S416).

[0273] Furthermore, if the inner states of the two target users do not match (step S414, NO), the inner state confirmation unit 126 outputs information indicating the current inner state of the target user obtained from the user terminal (not shown) (i.e., the correct label) to the inner state determination unit 124e. If the inner state determination unit 124e obtains information indicating the current inner state of the target user obtained from the user terminal (not shown) (i.e., the correct label) from the inner state confirmation unit 126 (i.e., if the determination result is incorrect), the inner state determination unit 124e outputs the information indicating the current inner state of the target user obtained from the user terminal (not shown) to the determination result output unit 13.

[0274] The determination result output unit 13 acquires information indicating the inner state of the target user based on the result of the inquiry, output from the inner state determination unit 124e. The information acquired by the determination result output unit 13 from the inner state determination unit 124e includes the user identification information of the target user. The determination result output unit 13e outputs the acquired determination result information, including the information indicating the inner state of the target user and the user identification information, to the service providing device 20 via the network 50 (step S417). This completes the inference operation of the determination device 10e shown in the flowchart of FIG. 16.

[0275] As described above, the discrimination device 10e according to the third embodiment of the present invention is configured to discriminate the inner state of a target user by using sensing data such as image data of the body of the target user, who is the subject of discrimination of the inner state, captured by a camera, audio data of the target user's conversation collected by a microphone, or vital data of the target user's vital state measured by a vital sensor. The discrimination device 10e is also configured to discriminate the inner state of the target user based on a learning model trained by supervised machine learning and the sensing data.

[0276] With this configuration, the discrimination system according to the third embodiment of the present invention can more accurately discriminate the inner state of the target user even when there is a discrepancy between the outward emotional expression and the inner state. As a result, the discrimination system can suppress a decrease in discrimination accuracy that may occur due to a discrepancy between the outward emotional expression and the inner state when discriminating a person's inner state through sensing.

[0277] Furthermore, the discrimination device 10e according to the third embodiment of the present invention discriminates the inner state of the target user using supervised machine learning, which learns using teacher data to which correct labels have been assigned. Therefore, the discrimination device 10e according to the third embodiment can discriminate the inner state of the target user more accurately with higher learning accuracy than, for example, the discrimination device 10a according to the first embodiment, which discriminates the inner state of the target user using unsupervised machine learning.

[0278] Furthermore, the discrimination device 10e according to the third embodiment of the present invention is configured to generate training data triggered by acquisition of information indicating the current inner state of the target user from a user terminal (not shown) and perform supervised machine learning. With such a configuration, the discrimination device 10e can perform learning based on training data in which the current inner state voluntarily declared by the target user is used as a correct answer label, and can generate a learning model with higher accuracy in discriminating the inner state of the target user than the discrimination device 10a according to the first embodiment, etc.

[0279] Furthermore, the discrimination device 10e according to the third embodiment of the present invention is configured to inquire about the actual current inner state of the target user and evaluate whether the current inner state of the target user determined using the learning model is correct. By being provided with such a configuration, the discrimination device 10e can output discrimination result information including information indicating a more accurately determined inner state of the target user to the service providing device 20, compared to the discrimination device 10a according to the first embodiment, etc. Furthermore, by being provided with such a configuration, the discrimination device 10e can generate new training data even during inference, thereby generating a larger amount of training data.

[0280] <Fourth Embodiment> Depending on the situation at the time, etc., there may be cases where a user does not want some of their behaviors to be observed (i.e., when they want to specify specific items of sensing data so that they are not used for machine learning, etc.), where they want their inner state to be determined using only some of the sensing data (i.e., when they want to specify specific items of sensing data and perform inference), or where they want their inner state to be determined by weighting each item of sensing data. Below, we will explain in order the configurations of three types of determination systems that have configurations that can solve these three problems, respectively.

[0281] The overall configuration of the discrimination system according to the fourth embodiment of the present invention is the same as the overall configuration of the discrimination system 1 according to the first embodiment shown in Fig. 1, except that the discrimination devices 10 (10a to 10c) are replaced with discrimination devices 10 (10f to 10h). Therefore, a description of the service providing device 20, the sensor device 40, the network 50, etc. will be omitted here.

[0282] The following describes the configuration of the discrimination device 10f that addresses the first problem mentioned above, which allows a target user to specify sensing data of a specific item so that it is not used for machine learning or the like.

[0283] [Configuration of the discrimination device] The configuration of the discrimination device 10f will be described in detail below. Fig. 17 is a block diagram showing the functional configuration of the discrimination device 10f according to the fourth embodiment of the present invention. As shown in Fig. 17, the discrimination device 10f includes a learning unit 11f, an inference unit 12f, and a discrimination result output unit 13.

[0284] The learning unit 11f acquires learning data and performs supervised machine learning to generate a trained learning model that enables output of information indicating a determination result of the inner state of the target user observed by the sensor device 40 in response to input of sensing data transmitted from the sensor device 40. Note that any method such as DNN, support vector machine, or linear regression can be used as the supervised machine learning method.

[0285] Learning by the learning unit 11f is initiated when the target user designates in advance the items of sensing data that are permitted to be used and when information indicating the target user's current inner state is acquired from the sensor device 40. That is, in learning by the learning unit 11f, the target user designates in advance the items of sensing data that can be used for learning, and the target user's inner state is determined using a learning model that has been trained using sensing data of only some of the permitted items. In this respect, the configuration of the learning unit 11f differs from the configuration of the learning unit 11 in each of the above-described embodiments, which performs learning using sensing data of all items acquired from the sensor device 40.

[0286] As shown in FIG. 17, the learning unit 11f includes a sensing data designation unit 119, an inner state acquisition unit 118f, a learning data storage unit 112f, a learning execution unit 113f, and a learning model storage unit 115f.

[0287] The sensing data designation unit 119 acquires information indicating sensing data items that can be used for machine learning performed by the discrimination device 10f, for example, from a user terminal (not shown) via the network 50. The usable sensing data items are designated, for example, by the target user operating the user terminal (not shown). Note that any method can be used to designate the usable sensing data items, and, for example, a designation method in which the target user selects items that are permitted to be used from a predetermined plurality of sensing data items may be used.

[0288] The sensing data designation unit 119 associates information indicating the acquired items of usable sensing data with user identification information that identifies the target user who transmitted the information, and stores the information in the learning data storage unit 112f.

[0289] The learning data storage unit 112f stores information in which information indicating the items of sensing data output from the sensing data designation unit 119 is associated with user identification information of the target user.

[0290] The inner state acquisition unit 118f acquires information indicating the inner state of the target user at the current time transmitted from a user terminal (not shown) via the network 50. Note that the information indicating the inner state of the target user at the current time is generated, for example, by the target user determining his or her own inner state at the current time and inputting it into the user terminal (not shown).

[0291] The inner state acquisition unit 118f outputs user identification information for identifying a user who inputs information indicating the current inner state to the sensing data acquisition unit 111f. Therefore, the inner state acquisition unit 118f needs to be able to identify the target user who transmitted the information indicating the current inner state of the target user. For example, the inner state acquisition unit 118f may acquire user identification information of the target user who owns a user terminal (not shown) when a communication connection with the user terminal (not shown) or the sensor device 40 is established, or may acquire user identification information of the target user who owns the user terminal (not shown) when acquiring information indicating the current inner state of the target user.

[0292] Furthermore, the inner state acquisition unit 118f associates the acquired information indicating the inner state of the target user at the current time with user identification information identifying the target user, and stores the associated information in the learning data storage unit 112f. The learning data storage unit 112f stores the information indicating the inner state of the target user with the user identification information identifying the target user. This allows the learning data storage unit 112f to store the user identification information identifying the target user, information indicating the items of usable sensing data, and information indicating the inner state of the target user at the current time, in association with each other.

[0293] The sensing data acquisition unit 111f acquires the user identification information output from the inner state acquisition unit 118f. The sensing data acquisition unit 111f transmits request information requesting the transmission of sensing data to the sensor device 40 that observes the target user identified by the acquired user identification information. When the sensor device 40 acquires the request information transmitted from the sensing data acquisition unit 111f, it transmits the sensing data to the sensing data acquisition unit 111f.

[0294] The sensing data acquisition unit 111f acquires, via the network 50, sensing data returned from the sensor device 40 in response to the transmission of request information. The sensing data acquisition unit 111f acquires sensing data such as, for example, image data of the body of the target user who is the subject of inner state determination, captured with a camera, audio data of the target user's conversation collected with a microphone, or vital data measuring the target user's vital state with a vital sensor. Note that the sensing data acquisition unit 111f may acquire any items of sensing data, but it is considered that the more items of sensing data are obtained, the higher the accuracy of determining the target user's inner state.

[0295] The sensing data acquiring unit 111f stores the acquired sensing data in the learning data storage unit 112f. At this time, the sensing data acquiring unit 111f identifies user identification information that identifies the target user observed by the sensor device 40 that is the source of the sensing data. Then, the sensing data acquiring unit 111f associates the acquired sensing data with information indicating an inner state associated with the same user identification information as the identified user identification information, and stores the associated information in the learning data storage unit 112f. This generates training data in which the current inner state (i.e., the correct label) for the same target user is associated with the sensing data.

[0296] At this time, the sensing data acquisition unit 111f also reads, from the learning data storage unit 112f, information indicating usable sensing data items associated with the same identification information as the user identification information that identifies the target user observed by the sensor device 40 that is the source of the sensing data. The sensing data acquisition unit 111f performs a process of deleting sensing data items that are not usable sensing data items from the sensing data corresponding to the target user stored in the learning data storage unit 112f. This enables the learning unit 11f to avoid using sensing data items that the target user has not permitted to use in machine learning.

[0297] Note that, if the learning unit 11f is configured to periodically acquire sensing data from the sensor device 40 at sufficiently short intervals, the sensing data acquisition unit 111f does not need to request the sensor device 40 to transmit sensing data again in response to acquisition of information indicating the inner state. That is, in this case, for example, when acquiring information indicating the inner state, the learning unit 11f may store data in which the information indicating the inner state is associated with the most recently acquired sensing data as teacher data in the learning data storage unit 112f.

[0298] The training data storage unit 112f stores training data including correct labels. By repeating this process, training data including correct labels is accumulated in the training data storage unit 112f. As described above, the training data including correct labels is training data in which sensing data is associated with information indicating the inner state of the target user.

[0299] The learning execution unit 113f acquires learning data including correct labels stored in the learning data storage unit 112f. Note that the learning execution unit 113f may acquire the learning data when a predetermined amount of learning data has accumulated in the learning data storage unit 112f. The learning execution unit 113f generates a trained learning model by performing supervised machine learning using the acquired learning data. In other words, the learning model generated here is a learning model that outputs a discrimination result of the inner state of the target user in response to input sensing data.

[0300] The learning execution unit 113f stores the generated trained learning model in the learning model storage unit 115f. The learning model here includes, for example, information indicating the machine learning algorithm and parameter values.

[0301] The learning model storage unit 115f stores the learned learning model generated by the learning execution unit 113f.

[0302] The learning execution unit 113f is a functional unit realized by a processor (not shown), such as a CPU, included in the discrimination device 10f reading and executing a program pre-stored in a storage medium (not shown) included in the discrimination device 10f. The learning data storage unit 112f and the learning model storage unit 115f include, for example, a storage medium (not shown), such as a semiconductor memory (RAM), an EEPROM, or a ROM, a flash memory (SSD), a magnetic disk (HDD), or an optical disk, or any combination of these storage media. The sensing data designation unit 119, the inner surface state acquisition unit 118f, and the sensing data acquisition unit 111f include, for example, a communication interface that communicates with an external device via a network 50.

[0303] As shown in FIG. 17, the inference unit 12f includes a sensing data acquisition unit 121f, an inner state determination unit 124f, and an inner state confirmation unit 126f.

[0304] The configurations of the sensing data acquisition unit 121f, the inner state determination unit 124f, and the inner state confirmation unit 126f of the inference unit 12f are similar to the configurations of the sensing data acquisition unit 121e, the inner state determination unit 124e, and the inner state confirmation unit 126 of the inference unit 12e in the third embodiment described above, and therefore will not be described here.

[0305] The configuration of the discrimination device 10f described above is the same as the configuration of the discrimination device 10e in the third embodiment except that a configuration that allows the target user to specify items of sensing data that can be used for machine learning is added. However, the configuration is not limited to this, and it is also possible to add a configuration that allows the target user to specify items of sensing data that can be used for machine learning to the configuration of any of the discrimination devices in the other embodiments described above.

[0306] [Operation of the discrimination device] An example of the operation of the discrimination device 10f during supervised machine learning learning will be described below. Note that the example of the operation of the discrimination device 10f during inference is similar to the example of the operation of the discrimination device 10e in the third embodiment shown in Figure 16, and therefore description thereof will be omitted.

[0307] 18 is a flowchart showing the operation of the discrimination device 10 f during learning according to the fourth embodiment of the present invention. The operation of the discrimination device 10 f shown in the flowchart of FIG. 18 is started when information indicating sensing data items that can be used for machine learning performed by the discrimination device 10 f is input to the sensing data designation unit 119 from, for example, a user terminal (not shown).

[0308] The sensing data designation unit 119 of the learning unit 11e acquires information indicating sensing data items that can be used for machine learning performed by the discrimination device 10f, for example, from a user terminal (not shown) of the target user via the network 50 (step S501). The usable sensing data items are designated, for example, by the target user operating the user terminal (not shown).

[0309] The sensing data specifying unit 119 associates information indicating the acquired usable sensing data items with user identification information identifying the user who owns the user terminal (not shown) that transmitted the information, and stores the associated information in the learning data storage unit 112f. The learning data storage unit 112f stores information in which the information indicating the sensing data items output from the sensing data specifying unit 119 is associated with the user identification information identifying the target user (step S502).

[0310] The inner state acquisition unit 118f acquires information indicating the inner state of the target user at the current time transmitted from a user terminal (not shown) via the network 50 (step S503). The inner state acquisition unit 118e outputs user identification information identifying the user who owns the user terminal (not shown), which is the source of the information indicating the inner state of the target user at the current time, to the sensing data acquisition unit 111e.

[0311] Furthermore, the inner state acquisition unit 118f associates the acquired information indicating the inner state of the target user at the current time with user identification information identifying the target user and stores them in the learning data storage unit 112e. The learning data storage unit 112e stores the information indicating the inner state of the target user and the user identification information identifying the target user in association with each other (step S504). This allows the learning data storage unit 112f to store the user identification information identifying the target user, information indicating the items of usable sensing data, and information indicating the inner state of the target user at the current time in association with each other.

[0312] The sensing data acquisition unit 111f acquires the user identification information output from the inner state acquisition unit 118f. The sensing data acquisition unit 111f transmits request information requesting the transmission of sensing data to the sensor device 40 that observes the target user identified by the acquired user identification information (step S505). When the sensor device 40 acquires the request information transmitted from the sensing data acquisition unit 111f, it transmits the sensing data to the sensing data acquisition unit 111f.

[0313] The sensing data acquiring unit 111f acquires, via the network 50, the sensing data returned from the sensor device 40 in response to the transmission of the request information (step S506). The sensing data acquiring unit 111f acquires sensing data such as, for example, image data of the body of the target user who is the subject of the inner state determination, captured by a camera, audio data of the target user's conversation, collected by a microphone, or vital data measuring the vital state of the target user by a vital sensor.

[0314] The sensing data acquiring unit 111f stores the acquired sensing data in the learning data storage unit 112f. At this time, the sensing data acquiring unit 111f identifies user identification information that identifies the target user observed by the sensor device 40 that is the source of the sensing data. Then, the sensing data acquiring unit 111f associates the acquired sensing data with information indicating an inner state associated with the same user identification information as the identified user identification information, and stores the associated information in the learning data storage unit 112f (step S507). As a result, training data is generated in which the current inner state (i.e., the correct label) is associated with sensing data for the same target user.

[0315] At this time, the sensing data acquisition unit 111f also identifies user identification information that identifies the user observed by the sensor device 40 that is the source of the sensing data. Then, the sensing data acquisition unit 111f reads information indicating usable sensing data items associated with the same user identification information as the identified user identification information from the learning data storage unit 112f. The sensing data acquisition unit 111f performs a process of deleting sensing data items that are not usable sensing data items from the sensing data corresponding to the above user identification information stored in the learning data storage unit 112f (step S508). This allows the learning unit 11f to avoid using sensing data items whose use is not permitted by the target user in machine learning.

[0316] The training data storage unit 112f stores training data including correct labels. The learning execution unit 113f acquires the training data including correct labels stored in the training data storage unit 112f. Note that the learning execution unit 113f may acquire the training data when a predetermined amount of training data has been accumulated in the training data storage unit 112f. The learning execution unit 113f generates a trained training model by performing supervised machine learning using the acquired training data (step S509).

[0317] The learning execution unit 113f stores the trained learning model generated as a result of the supervised machine learning in the learning model storage unit 115f. The learning model storage unit 115f stores the trained learning model generated by the learning execution unit 113f (step S510). This completes the learning operation of the discrimination device 10f shown in the flowchart of FIG. 18.

[0318] Below, we will explain the configuration of the discrimination device 10g, which addresses the second problem mentioned above, by allowing the target user to specify sensing data for specific items and having the internal state discriminated using only the sensing data for the specified items.

[0319] [Configuration of the discrimination device] The configuration of the discrimination device 10g will be described in detail below. Fig. 19 is a block diagram showing the functional configuration of the discrimination device 10g according to the fourth embodiment of the present invention. As shown in Fig. 19, the discrimination device 10g includes a learning unit 11g, an inference unit 12g, and a discrimination result output unit 13.

[0320] As shown in FIG. 19, the learning unit 11g includes an inner state acquiring unit 118g, a sensing data acquiring unit 111g, a learning data storage unit 112g, a learning execution unit 113g, and a learning model storage unit 115g.

[0321] The configurations of the internal state acquisition unit 118g, sensing data acquisition unit 111g, learning data storage unit 112g, learning execution unit 113g, and learning model storage unit 115g of the learning unit 11g are similar to the configurations of the internal state acquisition unit 118e, sensing data acquisition unit 111e, learning data storage unit 112e, learning execution unit 113e, and learning model storage unit 115e of the learning unit 11e in the third embodiment shown in Figure 14 above, and therefore will not be described again.

[0322] As shown in FIG. 19, the inference unit 12g includes a sensing data designation unit 127, a designation item storage unit 128, a sensing data acquisition unit 121g, and an inner state determination unit 124g.

[0323] The sensing data designation unit 127 acquires information indicating sensing data items that can be used for inference performed by the discrimination device 10g, for example, from a user terminal (not shown) owned by the target user via the network 50. The usable sensing data items are designated, for example, by the target user operating the user terminal (not shown). Note that any method can be used to designate the usable sensing data items, and, for example, a designation method in which the target user selects items that are permitted to be used from a predetermined number of sensing data items may be used.

[0324] For example, if the target user intends to use specific information preferentially, the target user can arbitrarily specify a combination of information for inference by specifying usable sensing data items. In the present embodiment, the target user himself / herself specifies the sensing data items usable for inference, but this is not limited to such a configuration, and the discrimination device 10g may automatically specify the same. For example, the sensing data specification unit 127 may be configured to identify sensing data items with a relatively large range of variation in feature values ​​from among the sensing data items transmitted from the target user's user terminal (not shown), and automatically specify the identified items as sensing data items usable for inference.

[0325] The sensing data designation unit 127 associates information indicating the items of acquired usable sensing data with user identification information that identifies the user who owns the user terminal (not shown) that sent the information, and stores the information in the designated item memory unit 128.

[0326] The specified item memory unit 128 stores information indicating the items of sensing data output from the sensing data specification unit 127 in association with user identification information that identifies the target user observed by the sensor device 40 that transmitted the information.

[0327] The sensing data acquiring unit 121g acquires various types of sensing data transmitted from the sensor devices 40 via the network 50. The items of sensing data acquired by the sensing data acquiring unit 121g are, for example, the same as the items of sensing data acquired by the sensing data acquiring unit 111g of the learning unit 11g. Note that the sensing data acquiring unit 121g and the sensing data acquiring unit 111g may be configured as an integrated device in terms of hardware.

[0328] At this time, the sensing data acquisition unit 121g identifies user identification information that is the same as the user identification information that identifies the user observed by the sensor device 40 that is the source of the sensing data. The sensing data acquisition unit 121g reads information indicating the items of usable sensing data associated with the identified user identification information from the specified item storage unit 128. The sensing data acquisition unit 121g performs a process of deleting sensing data of items that are not usable sensing data items from the acquired sensing data. This allows the inference unit 12g to avoid using sensing data of items that the target user is not permitted to use for inference.

[0329] The sensing data acquisition unit 121g outputs sensing data consisting of only usable items to the inner surface state determination unit 124g.

[0330] The inner state determination unit 124g acquires sensing data consisting only of usable items output from the sensing data acquisition unit 121g. The inner state determination unit 124g also acquires a trained learning model stored in the learning model storage unit 115g of the learning unit 11g. The inner state determination unit 124g identifies the current inner state of the target user based on the sensing data consisting only of usable items and the trained learning model. The inner state determination unit 124g outputs information indicating the identified current inner state of the target user to the determination result output unit 13 as a determination result.

[0331] The sensing data acquisition unit 121g and the inner surface state determination unit 124g are functional units realized by a processor (not shown), such as a CPU, included in the determination device 10g reading and executing a program pre-stored in a storage medium (not shown) included in the determination device 10g. The specified item storage unit 128 includes, for example, a storage medium (not shown), such as a semiconductor memory (RAM), an EEPROM, or a ROM, a flash memory (SSD), a magnetic disk (HDD), or an optical disk, or any combination of these storage media. The sensing data acquisition unit 121g and the sensing data designation unit 127 include, for example, a communication interface that communicates with the sensor device 40 via the network 50.

[0332] The configuration of the discrimination device 10g described above is the same as the configuration of the discrimination device 10e in the third embodiment except that a configuration is added that allows the target user to specify items of sensing data that can be used during inference. However, the configuration is not limited to this, and it is also possible to add a configuration that allows the target user to specify items of sensing data that can be used during inference to the configuration of any of the discrimination devices in the other embodiments described above.

[0333] [Operation of the discrimination device] An example of the operation of the discrimination device 10g during supervised machine learning inference will be described below. Note that the example of the operation of the discrimination device 10g during learning is similar to the example of the operation of the discrimination device 10e in the third embodiment shown in Figure 15, and therefore description thereof will be omitted.

[0334] 20 is a flowchart showing the operation of the discrimination device 10g during inference according to the fourth embodiment of the present invention. The operation of the discrimination device 10g shown in the flowchart of Fig. 20 is started when information indicating sensing data items that can be used for inference performed by the discrimination device 10g is input to the sensing data designation unit 127 from, for example, a user terminal (not shown).

[0335] The sensing data designation unit 127 of the inference unit 12g acquires information indicating sensing data items that can be used for inference performed by the discrimination device 10g, for example, from the target user or a user terminal (not shown) owned by the target user, via the network 50 (step S601). The usable sensing data items are designated, for example, by the target user operating the user terminal (not shown).

[0336] The sensing data designation unit 127 associates information indicating the acquired items of usable sensing data with user identification information identifying the target user who owns the user terminal (not shown) that transmitted the information, and stores the associated information in the designated item storage unit 128. The designated item storage unit 128 stores information in which the information indicating the items of sensing data output from the sensing data designation unit 127 is associated with the user identification information identifying the target user (step S602).

[0337] The sensing data acquiring unit 121g acquires various sensing data transmitted from the sensor device 40 via the network 50 (step S603). The items of sensing data acquired by the sensing data acquiring unit 121g are the same as the items of sensing data acquired by the sensing data acquiring unit 111g of the learning unit 11g, for example.

[0338] At this time, the sensing data acquisition unit 121g identifies user identification information that identifies the user observed by the sensor device 40 that is the source of the sensing data. Then, the sensing data acquisition unit 121g reads information indicating usable sensing data items associated with the same user identification information as the identified user identification information from the specified item storage unit 128. The sensing data acquisition unit 121g performs a process of deleting sensing data items that are not usable sensing data items from the acquired sensing data (step S604). The sensing data acquisition unit 121g outputs sensing data consisting only of usable items to the inner state determination unit 124g.

[0339] The inner state determination unit 124g acquires sensing data consisting only of usable items output from the sensing data acquisition unit 121g. The inner state determination unit 124g also acquires a trained learning model stored in the learning model storage unit 115g of the learning unit 11g (step S605). The inner state determination unit 124g determines the current inner state of the target user based on the acquired sensing data consisting only of usable items and the trained learning model (step S606). The inner state determination unit 124g outputs information indicating the current inner state of the target user, which is the result of the above determination, to the determination result output unit 13.

[0340] The determination result output unit 13 acquires information indicating the inner state of the target user based on the above determination result output from the inner state determination unit 124g. The information acquired by the determination result output unit 13 from the inner state determination unit 124g includes the user identification information of the target user. The determination result output unit 13 transmits instruction information including the acquired information indicating the inner state of the target user and the user identification information identifying the target user to the service providing device 20 via the network 50 (step S607). This completes the inference operation of the determination device 10g shown in the flowchart of FIG. 20.

[0341] The following describes the configuration of the determination device 10h that addresses the third problem mentioned above, which allows the target user to weight each item of sensing data and determine the inner state using the weighted sensing data.

[0342] [Configuration of the discrimination device] The configuration of the discrimination device 10h will be described in detail below. Fig. 21 is a block diagram showing the functional configuration of the discrimination device 10h according to the fourth embodiment of the present invention. As shown in Fig. 21, the discrimination device 10h includes a learning unit 11h, an inference unit 12h, and a discrimination result output unit 13.

[0343] As shown in FIG. 21, the learning unit 11h includes an inner state acquiring unit 118h, a sensing data acquiring unit 111h, a learning data storage unit 112h, a learning execution unit 113h, and a learning model storage unit 115h.

[0344] The configurations of the internal state acquisition unit 118h, sensing data acquisition unit 111h, learning data storage unit 112h, learning execution unit 113h, and learning model storage unit 115h of the learning unit 11h are similar to the configurations of the internal state acquisition unit 118e, sensing data acquisition unit 111e, learning data storage unit 112e, learning execution unit 113e, and learning model storage unit 115e of the learning unit 11e in the third embodiment shown in Figure 14 above, and therefore will not be described again.

[0345] As shown in FIG. 21, the inference unit 12h includes a weighting specification unit 129, a weighting information storage unit 128h, a sensing data acquisition unit 121h, and an inner state determination unit 124h.

[0346] The weighting specification unit 129 acquires information (hereinafter referred to as "weighting information"), in which a weighting value to be assigned to each piece of sensing data used for inference by the inference unit 12h is specified for each item, from, for example, a user terminal (not shown) owned by the target user via the network 50. The weighting value to be assigned to each piece of sensing data is specified, for example, by the target user operating the user terminal (not shown). Note that the method for specifying the weighting value to be assigned to each piece of sensing data is arbitrary, and may be, for example, a specification method in which the target user inputs a numerical value representing the weight for each item of sensing data.

[0347] For example, if the target user wants to receive a notification based on the determined internal state with more weight given to the health state or activity level, the target user can specify that the weight value of the sensing data related to vital signs be relatively large. In this way, the target user can specify that a specific sensing data item be given more weight in the inference by the discrimination device 10h.

[0348] In this embodiment, the target user himself / herself specifies the weight value for each item of sensing data, but the present invention is not limited to this configuration, and the discrimination device 10h may automatically specify this. For example, the weighting specification unit 129 may specify, among the items of sensing data transmitted from the sensor device 40, an item of sensing data having a relatively large fluctuation range of the feature amount, and automatically specify a larger weight value for the specified item.

[0349] The weighting specification unit 129 associates information indicating the weight value for each item of the acquired sensing data with user identification information that identifies the target user who owns the user terminal (not shown) that transmitted the information, and stores the information in the weighting information storage unit 128h.

[0350] The weighting information storage unit 128h stores information that corresponds the weight value for each item of sensing data output from the weighting designation unit 129 with user identification information that identifies the target user who owns the user terminal (not shown) that transmitted the information.

[0351] The sensing data acquisition unit 121h acquires various types of sensing data transmitted from the sensor devices 40 via the network 50. The items of sensing data acquired by the sensing data acquisition unit 121h are, for example, the same as the items of sensing data acquired by the sensing data acquisition unit 111h of the learning unit 11h. Note that the sensing data acquisition unit 121h and the sensing data acquisition unit 111h may be configured as an integrated device in terms of hardware.

[0352] At this time, the sensing data acquiring unit 121h identifies the user identification information that identifies the target user observed by the sensor device 40 that is the source of the sensing data. Then, the sensing data acquiring unit 121h reads out, from the weighting information storage unit 128h, the weight values ​​for each item of sensing data associated with the same user identification information as the identified user identification information. The sensing data acquiring unit 121h performs a process of weighting each of the acquired sensing data with the weight value specified for each item.

[0353] Any method can be used to weight the sensing data, but for example, the sensing data acquisition unit 121h may adjust the number of sensing data items for each item so that the number is proportional to the weight value for each item.

[0354] The sensing data acquisition unit 121h outputs the weighted sensing data to the inner surface state determination unit 124h.

[0355] The inner state determination unit 124h acquires the sensing data output from the sensing data acquisition unit 121h. The inner state determination unit 124h also acquires the trained learning model stored in the learning model storage unit 115h of the learning unit 11h. The inner state determination unit 124h identifies the current inner state of the target user based on the acquired sensing data and the trained learning model. The inner state determination unit 124h outputs information indicating the identified current inner state of the target user to the determination result output unit 13 as a determination result.

[0356] The sensing data acquisition unit 121h and the inner surface state determination unit 124f are functional units realized by a processor (not shown), such as a CPU, included in the determination device 10h reading and executing a program pre-stored in a storage medium (not shown) included in the determination device 10h. The weighting information storage unit 128h includes, for example, a storage medium (not shown), such as a semiconductor memory (RAM), an EEPROM, or a ROM, a flash memory (SSD), a magnetic disk (HDD), or an optical disk, or any combination of these storage media. The sensing data acquisition unit 121h and the weighting designation unit 129 include, for example, a communication interface that communicates with the sensor device 40 via the network 50.

[0357] The configuration of the discrimination device 10h described above is the same as the configuration of the discrimination device 10e in the third embodiment except that a configuration is added that allows the target user to specify a weight value to be assigned to each item of sensing data during inference. However, the configuration is not limited to this, and it is also possible to add a configuration that allows the target user to specify a weight value to be assigned to each item of sensing data during inference to the configuration of any of the discrimination devices in the other embodiments described above.

[0358] [Operation of the discrimination device] An example of the operation of the discrimination device 10h during supervised machine learning inference will be described below. Note that the example of the operation of the discrimination device 10h during learning is similar to the example of the operation of the discrimination device 10e in the third embodiment shown in Figure 15, and therefore description thereof will be omitted.

[0359] 22 is a flowchart showing the operation of the discrimination device 10h during inference according to the fourth embodiment of the present invention. The operation of the discrimination device 10h shown in the flowchart of FIG. 22 is started when weighting information is input to the weighting designation unit 129 from, for example, a user terminal (not shown).

[0360] The weighting specification unit 129 of the inference unit 12h acquires weighting information from, for example, a user terminal (not shown) held by the target user via the network 50 (step S701). The weighting value to be assigned to each piece of sensing data included in the weighting information is specified, for example, by the target user operating the user terminal (not shown).

[0361] The weighting specification unit 129 associates the acquired weighting information with user identification information that identifies the user who owns the user terminal (not shown) that transmitted the weighting information, and stores the associated weighting information in the weighting information storage unit 128h. The weighting information storage unit 128h stores information in which the weighting information output from the weighting specification unit 129 is associated with the user identification information that identifies the target user (step S702).

[0362] The sensing data acquiring unit 121h acquires various sensing data transmitted from the sensor device 40 via the network 50 (step S703). The items of sensing data acquired by the sensing data acquiring unit 121h are the same as the items of sensing data acquired by the sensing data acquiring unit 111h of the learning unit 11h, for example.

[0363] At this time, the sensing data acquisition unit 121h identifies user identification information that identifies the target user observed by the sensor device 40 that is the source of the sensing data. Then, the sensing data acquisition unit 121h reads weighting information associated with the same user identification information as the identified user identification information from the weighting information storage unit 128h. The sensing data acquisition unit 121h performs a process of weighting each of the acquired sensing data with a weight value specified for each item (step S704). The sensing data acquisition unit 121h outputs the weighted sensing data to the inner state determination unit 124h.

[0364] The inner state determination unit 124h acquires the sensing data output from the sensing data acquisition unit 121h. The inner state determination unit 124h also acquires the trained learning model stored in the learning model storage unit 115h of the learning unit 11h (step S705). The inner state determination unit 124h determines the current inner state of the target user based on the acquired sensing data and the trained learning model (step S706). The inner state determination unit 124h outputs information indicating the current inner state of the target user, which is the result of the above determination, to the determination result output unit 13.

[0365] The determination result output unit 13 acquires information indicating the inner state of the target user based on the above determination result output from the inner state determination unit 124h. Furthermore, the information acquired by the determination result output unit 13 from the inner state determination unit 124h includes user identification information identifying the target user observed by the sensor device 40. The determination result output unit 13 transmits the acquired determination result information, including the information indicating the inner state of the target user and the user identification information identifying the target user, to the service providing device 20 via the network 50 (step S707). This completes the inference operation of the determination device 10h shown in the flowchart of FIG. 22 .

[0366] As described above, the discrimination device 10 (10f to 10h) according to the fourth embodiment of the present invention is configured to discriminate the inner state of a target user by using sensing data such as image data of the body of the target user, who is the subject of discrimination of the inner state, captured by a camera, audio data of the target user's conversation collected by a microphone, or vital data of the target user's vital state measured by a vital sensor. The discrimination device 10 (10f to 10h) is also configured to discriminate the inner state of the target user based on a learning model trained by supervised machine learning and the sensing data.

[0367] With this configuration, the discrimination system according to the fourth embodiment of the present invention can more accurately discriminate the inner state of the target user even when there is a discrepancy between the outward emotional expression and the inner state. As a result, the discrimination system can suppress a decrease in discrimination accuracy that may occur due to a discrepancy between the outward emotional expression and the inner state when discriminating a person's inner state through sensing.

[0368] Furthermore, the discrimination devices 10 (10f to 10h) according to the fourth embodiment of the present invention discriminate the inner state of the target user using supervised machine learning, which learns using teacher data to which correct labels have been assigned. Therefore, the discrimination devices 10 (10f to 10h) according to the fourth embodiment can achieve higher learning accuracy and more accurate discrimination of the inner state of the target user than, for example, the discrimination device 10a according to the first embodiment, which discriminates the inner state of the target user using unsupervised machine learning.

[0369] Furthermore, the discrimination device 10 f according to the fourth embodiment of the present invention has a configuration that allows a target user to specify items of sensing data that can be used for machine learning. With this configuration, the discrimination device 10 f can control the provision of services by the service providing device 20 so that services are provided taking into consideration, for example, the wishes and privacy of the target user.

[0370] Furthermore, the discrimination device 10f according to the fourth embodiment of the present invention is configured to generate teacher data triggered by acquiring information indicating the current inner state of the target user from the target user's user terminal (not shown) and perform supervised machine learning. With such a configuration, the discrimination device 10f performs learning based on teacher data in which the current inner state voluntarily declared by the target user is used as a correct answer label, and therefore can generate a learning model with higher accuracy in discriminating the inner state of the target user than the discrimination device 10a according to the first embodiment, etc.

[0371] Furthermore, as described above, the discrimination device 10f according to the fourth embodiment of the present invention is configured to inquire about the actual current inner state of the target user and evaluate whether the current inner state of the target user determined using the learning model is correct. By being provided with such a configuration, the discrimination device 10f can transmit to the service providing device 20 discrimination result information including information indicating a more accurately identified inner state of the target user, compared to the discrimination device 10a according to the first embodiment, etc. Furthermore, by being provided with such a configuration, the discrimination device 10f can generate training data even during inference, thereby generating a larger amount of training data.

[0372] Furthermore, the discrimination device 10g according to the fourth embodiment of the present invention is configured to allow the target user to specify sensing data items that can be used for inference. This configuration allows the target user to specify sensing data items that can be used when the target user intends to use specific information preferentially, and the discrimination device 10g can arbitrarily change the combination of information used for inference in accordance with the specification. Furthermore, this configuration allows the discrimination device 10g to control the provision of services by the service providing device 20 so that services are provided while taking into consideration the target user's wishes and privacy.

[0373] Furthermore, the discrimination device 10h according to the fourth embodiment of the present invention is configured to allow the target user to specify a weight value to be assigned to each piece of sensing data during inference. This configuration allows the target user to specify that specific sensing data items should be given more consideration, and the discrimination device 10h can arbitrarily change the degree to which information is utilized for inference in accordance with the specification. Furthermore, this configuration allows the discrimination device 10h to control the provision of services by the service providing device 20 so that service provision notification is provided taking into consideration the target user's wishes, health condition, activity level, etc.

[0374] In the first to fourth embodiments described above, examples of the configuration of the determination system have been described, mainly relating to the determination of the inner state of the target user by the determination devices 10, 10b to 10h. In contrast, in the fifth and sixth embodiments described below, examples of the configuration of the determination system have been described, mainly relating to the provision of services by the service providing device 20. As described above, the service providing device 20 is a device that provides various services related to the target user in accordance with the results of the determination of the inner state of the target user by the determination devices 10, 10b to 10h.

[0375] Specifically, the fifth embodiment described below is an example in which the configuration of the determination system in each of the above-described embodiments is applied to a chat system, and the sixth embodiment described below is an example in which the configuration of the determination system in each of the above-described embodiments is applied to an interview system.

[0376] Fifth Embodiment A fifth embodiment of the present invention will now be described.

[0377] [Configuration of Chat System] The configuration of a chat system 1i according to a fifth embodiment of the present invention will be described below. The chat system 1i is an information system that uses a discrimination device 10i to discriminate the inner state of a target user and a chat server 20i to provide a chat service that utilizes the discrimination result.

[0378] Fig. 23 is a diagram showing the overall configuration of a chat system 1i according to the fifth embodiment of the present invention. As shown in Fig. 23, the chat system 1i includes a discrimination device 10i, a chat server 20i, a plurality of terminal devices 40i, and a network 50. Note that the number of terminal devices 40i may be one.

[0379] The chat system 1i is an example of a discrimination system in the first to fourth embodiments, the chat server 20i is an example of a service providing device 20 in the first to fourth embodiments, and the terminal device 40i is an example of a sensor device 40 in the first to fourth embodiments.

[0380] The discrimination device 10i includes an information processing device such as a general-purpose computer. The discrimination device 10i is connected to a network 50. The discrimination device 10i can communicate with the chat server 20i and each of the plurality of terminal devices 40i via the network 50.

[0381] The determination device 10i is a device that determines the inner state of a target user using information collected by sensing by the terminal device 40i and provides the determination result to the chat server 20i. As described above, the target user is a person whose inner state is to be determined. The determination device 10i determines the inner state of the target user using AI.

[0382] The determination device 10i is a server device that provides a cloud-type service in cooperation with the chat server 20i via the network 50. The cloud-type service referred to here is a chat service in which the internal state of a target user who owns the terminal device 40i is determined on the server (i.e., the chat server 20i) side on the network 50, and messages are transmitted between users while controlling notifications to the target user who is the destination of the message according to the determination result.

[0383] The discrimination device 10i acquires sensing data obtained by sensing using various sensors provided in the terminal device 40i. The discrimination device 10i discriminates the inner state of the target user who is the subject of sensing based on the acquired sensing data. The discrimination of the inner state here means discriminating whether the target user is in a relaxed state or a stressful state. However, the discrimination of the inner state performed by the discrimination device 10i is not limited to discriminating whether the target user is in a relaxed state, and may also discriminate other types of inner state.

[0384] For example, the discrimination device 10i stores information indicating a plurality of predefined inner states, and outputs one of the pieces of information indicating the plurality of inner states as the discrimination result. As described above, for example, if the predefined inner states are classified into two categories, the two inner states are a "comfortable state" and a "tight state." Also, for example, if the predefined inner states are classified into five categories, the five inner states are a "comfortable state," a "slightly comfortable state," a "neutral state," a "slightly tight state," and a "tight state." The discrimination device 10i outputs information indicating one of these inner states as the discrimination result.

[0385] The discrimination device 10i outputs information indicating the discrimination result to the chat server 20i. Note that the configuration of the discrimination device 10i may be, for example, any of the configurations of the discrimination devices 10, 10b to 10h in the first to fourth embodiments. Note that the discrimination device 10i and the chat server 20i may be integrated into one device.

[0386] The chat server 20i includes an information processing device such as a general-purpose computer. The chat server 20i is connected to a network 50. The chat server 20i can communicate with the discrimination device 10i and each of the multiple terminal devices 40i via the network 50.

[0387] The chat server 20i provides a chat service that controls notifications in accordance with the determination result based on the acquired information. As described above, the chat server 20i is a server device that provides a cloud-based service via the network 50 in cooperation with the determination device 10i.

[0388] The chat server 20i pre-stores information associating the inner state of the target user with a notification control method. The chat server 20i acquires, from the discrimination device 10i, discrimination result information including information indicating the inner state of the target user and user identification information. The chat server 20i identifies the target user identified by the user identification information included in the acquired discrimination result information, and identifies the terminal device 40i owned by the target user. The chat server 20i then provides a chat service in accordance with the notification control method associated with the inner state based on the information included in the acquired discrimination result information. The configuration of the chat server 20i will be described in detail later.

[0389] The network 50 is an information communication network that communicatively connects the discrimination device 10i, the chat server 20i, and each of the multiple terminal devices 40i with one another. For example, the network 50 is configured to include the Internet, a LAN, etc. Also, for example, a wireless communication base station (not shown) is connected to the network 50. For example, the terminal device 40i, which is a wireless communication terminal, is connected to the network 50 via the wireless communication base station.

[0390] The terminal device 40i is equipped with a communication function and various sensors that observe the behavior of the target user to determine the inner state of the target user. The terminal device 40i is a user terminal owned by the target user, such as a smartphone, tablet terminal, or laptop computer, on which a chat application that receives the chat service provided by the chat server 20i is installed.

[0391] As described above, the various sensors included in the terminal device 40i include, for example, a camera, a microphone, or a vital sign measurement sensor. Note that the device equipped with the various sensors for observing the target user and the user terminal on which the chat application is installed may be separate devices.

[0392] [Configuration of Chat Server] The configuration of the chat server 20i will be described in more detail below. Fig. 24 is a block diagram showing the functional configuration of the chat server 20i in the fifth embodiment of the present invention. As shown in Fig. 24, the chat server 20i is configured to include an internal state acquisition unit 201, an internal state storage unit 202, a message receiving unit 203, an internal state identification unit 204, a message analysis unit 205, an importance determination unit 206, a notification rule designation unit 207, a notification rule storage unit 208, a notification method determination unit 209, a notification control unit 210, and a message transfer unit 211.

[0393] The inner state acquisition unit 201 acquires the determination result information output from the determination device 10i. As described above, the determination result information includes information indicating the inner state of the target user and user identification information. The inner state acquisition unit 201 stores the acquired determination result information in the inner state storage unit 202.

[0394] The inner state storage unit 202 stores the determination result information for each user identification information. Therefore, if there are multiple pieces of determination result information containing the same user identification information, only the most recent determination result information is left, and the other pieces of determination result information are deleted (overwritten). In other words, the inner state storage unit 202 stores information indicating the latest inner state for each target user.

[0395] The message receiving unit 203 receives a chat message sent from a terminal device 40i. The terminal device 40i here refers to a terminal device 40i that is the sender of a chat message and that has sent the chat message to another terminal device 40i (the terminal device 40i that is the destination of the chat message) using a chat application installed on the terminal device 40i. The chat message is assigned user identification information that identifies the target user who owns the destination terminal device 40i. Note that the chat message may also include importance information that indicates the importance of the chat message. The importance information is information that is set by the user who sent the chat message using the terminal device 40i.

[0396] The message receiving unit 203 outputs the user identification information attached to the chat message to the inner state identifying unit 204. The message receiving unit 203 also outputs the chat message to the message analyzing unit 205. If importance information is attached to the chat message, the message receiving unit 203 also outputs the importance information to the message analyzing unit 205.

[0397] The inner state identification unit 204 acquires the user identification information output from the message receiving unit 203. The inner state identification unit 204 identifies discrimination result information including user identification information identical to the acquired user identification information from among the plurality of pieces of discrimination result information stored in the inner state storage unit 202. The inner state identification unit 204 outputs information indicating the inner state of the target user, which is included in the identified discrimination result information, to the notification method determination unit 209.

[0398] The message analysis unit 205 acquires the chat message output from the message receiving unit 203. The message analysis unit 205 performs an analysis process on the chat message to enable the importance determination unit 206, which will be described later, to determine the importance of the chat message. The message analysis unit 205 outputs information indicating the analysis result to the importance determination unit 206. The message analysis unit 205 determines, for example, whether the chat message contains the words "urgent" or "urgent," and outputs the result to the importance determination unit 206 as the analysis result.

[0399] If importance information is attached to a chat message, it is possible to omit the analysis process performed by the message analysis unit 205. That is, the importance of a chat message may be explicitly designated manually by the sender user, or may be automatically determined by the analysis process of the chat message performed by the chat server 20i.

[0400] The analysis process of chat messages by the message analysis unit 205 may be performed using text mining, AI, etc. For text mining, for example, Janome, a morphological analysis library dedicated to the programming language Python, may be used.

[0401] The importance determination unit 206 acquires information indicating the analysis result output from the message analysis unit 205. The importance determination unit 206 determines the importance of the chat message based on the acquired information indicating the analysis result. For example, if the analysis result indicates that the chat message contains the words "urgent" or "urgent," the importance determination unit 206 determines that the importance of the chat message is high. On the other hand, if the analysis result indicates that the chat message does not contain the words "urgent" or "urgent," the importance determination unit 206 determines that the importance of the chat message is low. The importance determination unit 206 assigns importance information indicating the importance to the chat message and outputs the chat message to the notification method determination unit 209.

[0402] The notification method determination unit 209 acquires information indicating the inner state of the target user output from the inner state identification unit 204. The notification method determination unit 209 also acquires chat messages to which importance information has been assigned output from the importance determination unit 206. The notification method determination unit 209 determines a notification method for the chat message in the destination terminal device 40i, based on the acquired information indicating the inner state of the target user and the importance information assigned to the acquired chat message.

[0403] The notification method for a chat message is determined, for example, according to a predetermined notification rule. As shown in FIG. 24 , the chat server 20i may further include a notification rule designation unit 207 and a notification rule storage unit 208. The notification rule designation unit 207 receives information indicating the notification rule designated by the target user from the terminal device 40i. The notification rule designation unit 207 stores the information indicating the notification rule in the notification rule storage unit 208.

[0404] The notification rule storage unit 208 stores information indicating the notification rule for each target user (i.e., for each piece of user identification information) output from the notification rule specification unit 207. Note that the information indicating the notification rule is provided with user identification information that identifies the target user who specified the notification rule.

[0405] For example, the notification rule may be a rule that, if the importance of a chat message is low and the target user is in a stressful state, the terminal device 40i of the target user is not notified of the receipt of the chat message. Alternatively, the notification rule may be a rule that chat messages are not received at all. Furthermore, for example, the notification rule may be a rule that, if the importance of a chat message is high, the terminal device 40i of the target user is notified of the receipt of a chat message regardless of the target user's internal state. By performing notifications according to such rules, the target user can receive notifications only about the receipt of chat messages of high importance when he or she is in a stressful state.

[0406] The notification method determination unit 209 outputs information indicating the determined notification method and the chat message to the notification control unit 210. The notification control unit 210 acquires the information indicating the notification method and the chat message output from the notification method determination unit 209. The notification control unit 210 causes the message transfer unit 211 to transfer the chat message in accordance with the notification method based on the acquired information. The message transfer unit 211 transfers the chat message to the destination terminal device 40i under the control of the notification control unit 210.

[0407] The internal state identification unit 204, the message analysis unit 205, the importance determination unit 206, the notification method determination unit 209, and the notification control unit 210 are functional units realized by a processor (not shown), such as a CPU, included in the chat server 20i reading and executing a program pre-stored in a storage medium (not shown) included in the chat server 20i. The internal state storage unit 202 and the notification rule storage unit 208 are configured to include, for example, a storage medium (not shown), such as a semiconductor memory (RAM), an EEPROM, or a ROM, a flash memory (SSD), a magnetic disk (HDD), or an optical disk, or any combination of these storage media. The internal state acquisition unit 201, the message receiving unit 203, the notification rule designation unit 207, and the message transfer unit 211 are configured to include a communication interface that communicates with the discrimination device 10i or the terminal device 40i via, for example, the network 50.

[0408] [Operation of Chat Server] An example of the operation of the chat server 20i will be described below. Fig. 25 is a flowchart showing the operation of the chat server 20i in the fifth embodiment of the present invention. The operation of the chat server 20i shown in the flowchart of Fig. 25 is started when the chat server 20i receives, for example, discrimination result information transmitted from the discrimination device 10i.

[0409] The inner surface state acquisition unit 201 acquires the discrimination result information output from the discrimination device 10i (step S801). The inner surface state acquisition unit 201 stores the acquired discrimination result information in the inner surface state storage unit 202.

[0410] Next, the message receiving unit 203 receives the chat message transmitted from the terminal device 40i (step S802). The message receiving unit 203 outputs the user identification information attached to the chat message to the inner state identifying unit 204. The message receiving unit 203 also outputs the chat message to the message analyzing unit 205.

[0411] Next, the inner state identification unit 204 acquires the user identification information output from the message receiving unit 203. The inner state identification unit 204 identifies discrimination result information including user identification information identical to the acquired user identification information from among the plurality of pieces of discrimination result information stored in the inner state storage unit 202. The inner state identification unit 204 identifies information indicating the inner state of the target user included in the identified discrimination result information, and outputs the information to the notification method determination unit 209 (step S803).

[0412] Next, the message analysis unit 205 acquires the chat message output from the message receiving unit 203. The message analysis unit 205 performs an analysis process on the chat message (step S804). The message analysis unit 205 outputs information indicating the analysis result to the importance determination unit 206. As described above, the message analysis unit 205 determines, for example, whether the chat message contains the wording "urgent" or "urgent," and outputs the result to the importance determination unit 206 as the analysis result.

[0413] Next, the importance determination unit 206 acquires information indicating the analysis result output from the message analysis unit 205. The importance determination unit 206 determines the importance of the chat message based on the acquired information indicating the analysis result (step S805). As described above, for example, if the analysis result indicates that the chat message contains the word "urgent" or "urgent," the importance determination unit 206 determines that the importance of the chat message is high.

[0414] Next, the notification method determination unit 209 acquires information indicating the inner state of the target user output from the inner state identification unit 204. The notification method determination unit 209 also acquires the chat message to which importance information has been assigned output from the importance determination unit 206. The notification method determination unit 209 determines a notification method for the chat message in the destination terminal device 40i based on the acquired information indicating the inner state of the target user and the importance information assigned to the acquired chat message (step S806).

[0415] Next, the notification method determination unit 209 outputs information indicating the determined notification method and the chat message to the notification control unit 210. The notification control unit 210 acquires the information indicating the notification method and the chat message output from the notification method determination unit 209. The notification control unit 210 causes the message transfer unit 211 to transfer the chat message in accordance with the notification method based on the acquired information (step S807). The message transfer unit 211 transfers the chat message to the destination terminal device 40i under the control of the notification control unit 210. This completes the operation of the chat server 20i shown in the flowchart of FIG. 25.

[0416] As described above, the discrimination device 10i of the chat system 1i in the fifth embodiment of the present invention is configured to discriminate the inner state of a target user who is the subject of inner state discrimination using sensing data such as image data captured by a camera of the body of the target user, audio data collected by a microphone of the target user's conversation, or vital data measured by a vital sensor on the target user's vital state. The discrimination device 10i is also configured to discriminate the inner state of the target user based on a learning model trained by unsupervised machine learning, a learning model trained by supervised machine learning, or a rule base, and the sensing data.

[0417] With this configuration, the chat system 1i according to the fifth embodiment of the present invention can determine the inner state of the target user with higher accuracy even when there is a discrepancy between the outward emotional expression and the inner state. As a result, the chat system 1i can suppress a decrease in the accuracy of determination of a person's inner state by sensing, which may occur due to a discrepancy between the outward emotional expression and the inner state.

[0418] As described above, the chat server 20i of the chat system 1i according to the fifth embodiment of the present invention is configured to provide a chat service by utilizing information indicating the inner state of the target user determined by the determination device 10i, thereby enabling the chat server 20i to transfer a chat message to the target user in an appropriate notification method and timing according to the inner state of the target user.

[0419] As described above, the chat server 20i of the chat system 1i according to the fifth embodiment of the present invention is configured to analyze chat messages and determine the importance of the chat messages, thereby enabling the chat server 20i to transfer chat messages to the target user in a more appropriate notification method and timing that takes into account not only the inner state of the target user but also the importance of the chat message.

[0420] For example, it is generally considered that the target user is likely to be in a state of being stressed during business hours, in a meeting, during a call, etc. In contrast, according to the chat system 1i of the fifth embodiment, in such cases, the chat server 20i can perform notification control so as not to notify the target user of unimportant chat messages.

[0421] Sixth Embodiment A sixth embodiment of the present invention will now be described.

[0422] [Configuration of Interview System] The configuration of an interview system 1j in a sixth embodiment of the present invention will be described below. The interview system 1j is an information system that uses a discrimination device 10j to determine the inner state of a target user (interviewee) and provides feedback to the interviewer about the inner state of the interviewee at the time of the interview using an interview server 20j.

[0423] Figure 26 is an overall configuration diagram of an interview system 1j according to a sixth embodiment of the present invention. As shown in Figure 26, the interview system 1j includes a discrimination device 10j, an interview server 20j, an interviewer terminal 30j, an interviewee terminal 40j, and a network 50. The interview system 1j is an example of the discrimination system in the first to fourth embodiments, the interview server 20j is an example of the service providing device 20 in the first to fourth embodiments, and the interviewer terminal 30j and the interviewee terminal 40j are examples of the sensor device 40 in the first to fourth embodiments.

[0424] The discrimination device 10j includes an information processing device such as a general-purpose computer. The discrimination device 10j is connected to a network 50. The discrimination device 10j can communicate with the interview server 20j, the interviewer terminal 30j, and the interviewee terminal 40j via the network 50.

[0425] The discrimination device 10j is a device that discriminates the inner state of a target user (interviewee) using information collected by sensing by the interviewee terminal 40j and provides the discrimination result to the interview server 20j. As described above, the target user is a person whose inner state is to be discriminated. The discrimination device 10j discriminates the inner state of the target user using AI.

[0426] The discrimination device 10j is a server device that provides a cloud-based service in cooperation with the interview server 20j via the network 50. The cloud-based service referred to here is an interview support service that supports interviews by determining the inner state of a target user (interviewee) who possesses the interviewee terminal 40j on the server (i.e., the interview server 20j) on the network 50, generating feedback information that associates the determined inner state with the interview situation at that time, and transmitting the information to the interviewer terminal 30j, which is the terminal of the interviewer.

[0427] The discrimination device 10j acquires sensing data obtained by sensing using various sensors provided in the interviewee terminal 40j. The discrimination device 10j discriminates the inner state of the target user who is the subject of sensing based on the acquired sensing data. The discrimination of the inner state here refers to discriminating whether the target user is in a relaxed state or a stressful state. However, the discrimination of the inner state performed by the discrimination device 10j is not limited to discriminating whether the target user is in a relaxed state, and may also discriminate other types of inner state.

[0428] For example, the discrimination device 10j stores information indicating a plurality of predefined inner states, and outputs one of the pieces of information indicating the plurality of inner states as the discrimination result. As described above, for example, if the predefined inner states are classified into two categories, the two inner states are a "comfortable state" and a "tight state." Also, for example, if the predefined inner states are classified into five categories, the five inner states are a "comfortable state," a "slightly comfortable state," a "neutral state," a "slightly tight state," and a "tight state." The discrimination device 10j outputs information indicating one of these inner states as the discrimination result.

[0429] The discrimination device 10j outputs information indicating the discrimination result to the interview server 20j. Note that the configuration of the discrimination device 10j can be, for example, any of the configurations of the discrimination devices 10, 10b to 10h in the first to fourth embodiments. Note that the discrimination device 10j and the interview server 20j may be integrated into a single device.

[0430] The determination device 10j may use interview data, which will be described later, instead of sensing data to determine the inner state of the target user from the interview data.

[0431] The interview server 20j includes an information processing device such as a general-purpose computer. The interview server 20j is connected to a network 50. The interview server 20j can communicate with the discrimination device 10j, the interviewer terminal 30j, and the interviewee terminal 40j via the network 50.

[0432] The interview server 20j provides an interview support service that associates the discrimination result based on the acquired information with the current interview situation and provides information to the interviewee. As mentioned above, the interview server 20j is a server device that provides cloud-type services via the network 50 in cooperation with the discrimination device 10j. The configuration of the interview server 20j will be described in detail later.

[0433] The network 50 is an information and communication network that interconnects the discrimination device 10j, the interview server 20j, the interviewer terminal 30j, and the interviewee terminal 40j for communication. For example, the network 50 may include the Internet, a LAN, etc. Also, for example, a wireless communication base station (not shown) is connected to the network 50. For example, the interviewer terminal 30j and the interviewee terminal 40j, which are wireless communication terminals, are connected to the network 50 via the wireless communication base station.

[0434] The interviewer terminal 30j has a communication function and is a user terminal owned by a user (interviewer), such as a smartphone, tablet terminal, or laptop computer, on which an interview application (e.g., a web conference application) that receives the interview service and interview support service provided by the interview server 20j is installed.

[0435] The interviewee terminal 40j is equipped with a communication function and various sensors that observe the behavior of the target user to determine the inner state of the target user. The interviewee terminal 40j is a user terminal owned by the target user (interviewee), such as a smartphone, tablet terminal, or laptop computer, on which an interview application (e.g., a web conferencing application) that receives the interview service provided by the interview server 20j is installed.

[0436] As described above, the various sensors included in the interviewee terminal 40j include, for example, a camera, a microphone, or a vital sign measurement sensor. Note that the device equipped with the various sensors for observing the target user and the user terminal on which the interview application is installed may be separate devices.

[0437] [Configuration of the Interview Server] The configuration of the interview server 20j will be described in more detail below. Figure 27 is a block diagram showing the functional configuration of the interview server 20j in the sixth embodiment of the present invention. As shown in Figure 27, the interview server 20j includes an interview application 221, an inner state acquisition unit 222, an inner state storage unit 223, an interview data acquisition unit 224, a timing unit 225, a text conversion unit 226, a feedback information generation unit 227, a feedback information storage unit 228, and a feedback information transmission unit 229.

[0438] The interview application 221 is a server application that realizes an online interview between the interviewer terminal 30j and the interviewee terminal 40j. The interview application 221 outputs interview data, which is, for example, a video file containing information indicating the video, audio, and time of the interview, to the interview data acquisition unit 224.

[0439] The inner state acquisition unit 222 acquires the discrimination result information output from the discrimination device 10j. Furthermore, the clock unit 225 outputs information indicating the current time in response to a request. The inner state acquisition unit 222 makes a request to the clock unit 225 at the timing of acquiring the discrimination result information, and acquires the information indicating the time. The inner state acquisition unit 222 associates the discrimination result information with information indicating the time and stores them in the inner state storage unit 223. This associates the inner state of the target user with the time.

[0440] The interview data acquisition unit 224 acquires interview data from the interview application 221. The interview data acquisition unit 224 also makes a request to the timing unit to acquire information indicating the time. The interview data acquisition unit 224 acquires information stored in the inner state storage unit 223, in which discrimination result information is associated with information indicating the time. The interview data acquisition unit 224 links information indicating the inner state of the target user included in the discrimination result information with the interview data at the same time. This associates the inner state of the target user with interview data indicating the status of the interview at that time. The interview data acquisition unit 224 outputs the information indicating the inner state of the target user and the interview data, which are associated at the same time, to the text conversion unit 226.

[0441] The text conversion unit 226 acquires the interview data and information indicating the inner state of the target user that are linked at the same time and output from the interview data acquisition unit 224. The text conversion unit 226 converts the voice included in the interview data into text data. The text conversion unit 226 outputs information in which the converted text data, the interview data, and the information indicating the inner state are associated with each other to the feedback information generation unit 227.

[0442] The interviewee may be allowed to select whether or not to convert the voice data into text data.

[0443] The feedback information generation unit 227 acquires information output from the text conversion unit 226, in which the text data, interview data, and information indicating the internal state are associated with each other. Based on the acquired information, the feedback information generation unit 227 generates feedback information for providing the interviewer with feedback on the interview situation. For example, the feedback information is information summarizing the interview situation (video, conversational audio, conversational text, etc.) at the time when the interviewee is determined to be in a specific internal state during the interview. The feedback information generation unit 227 stores the generated feedback information in the feedback information storage unit 228.

[0444] The feedback information transmitting unit 229 reads out the feedback information stored in the feedback information storage unit 228. The feedback information transmitting unit 229 transmits the read feedback information via the network 50 to the interviewer terminal 30j.

[0445] The feedback information sending unit 229 may send the feedback information to the interviewer terminal 30j in real time while the interview is taking place. For example, the feedback information sending unit 229 may send the feedback information to the interviewer terminal 30j each time the determination result of the inner state of the target user (interviewee) changes. Alternatively, for example, the feedback information sending unit 229 may send the feedback information to the interviewer terminal 30j all at once after the interview.

[0446] The interview application 221, interview data acquisition unit 224, timing unit 225, text conversion unit 226, and feedback information generation unit 227 are functional units realized by a processor (not shown), such as a CPU, provided in the interview server 20j reading and executing programs pre-stored in a storage medium (not shown) provided in the interview server 20j. The internal state storage unit 223 and feedback information storage unit 228 are configured to include, for example, semiconductor memory such as RAM, EEPROM, and ROM, flash memory such as SSD, magnetic disk such as HDD, optical disk, or other storage medium (not shown), or any combination of these storage media. The internal state acquisition unit 222, feedback information transmission unit 229, and interview application 221 are configured to include a communication interface that communicates with the discrimination device 10j, interviewer terminal 30j, or interviewee terminal 40j via, for example, a network 50.

[0447] [Operation of the Interview Server] An example of the operation of the interview server 20j will be described below. Figure 28 is a flowchart showing the operation of the interview server 20j in the sixth embodiment of the present invention. The operation of the interview server 20j shown in the flowchart of Figure 28 is started in response to the start of an online interview service by the interview application 221, for example.

[0448] The inner state acquisition unit 222 acquires the determination result information output from the determination device 10j (step S901). Furthermore, the clock unit 225 outputs information indicating the current time in response to a request. The inner state acquisition unit 222 makes a request to the clock unit 225 at the timing of acquiring the determination result information, and acquires information indicating the time (step S902). The inner state acquisition unit 222 associates the determination result information with information indicating the time and stores the association in the inner state storage unit 223. This associates the inner state of the target user with the time (step S903).

[0449] Next, the interview data acquisition unit 224 acquires interview data from the interview application 221 (step S904). The interview data acquisition unit 224 also makes a request to the clock unit to acquire information indicating the time. The interview data acquisition unit 224 acquires information stored in the inner state storage unit 223, in which the discrimination result information is associated with information indicating the time. The interview data acquisition unit 224 associates information indicating the inner state of the target user included in the discrimination result information with the interview data at the same time. This associates the inner state of the target user with the interview data indicating the status of the interview at that time (step S905). The interview data acquisition unit 224 outputs the information indicating the inner state of the target user and the interview data, which are associated with the same time, to the text conversion unit 226.

[0450] Next, the text conversion unit 226 acquires the interview data and information indicating the inner state of the target user that are linked at the same time and output from the interview data acquisition unit 224. The text conversion unit 226 converts the voice included in the interview data into text data (step S906). The text conversion unit 226 outputs information in which the converted text data, the interview data, and the information indicating the inner state are associated with each other to the feedback information generation unit 227.

[0451] Next, the feedback information generation unit 227 acquires information output from the text conversion unit 226, in which the text data, interview data, and information indicating the internal state are associated with each other. Based on the acquired information, the feedback information generation unit 227 generates feedback information for providing the interviewer with feedback on the interview situation (step S907). As described above, for example, the feedback information is information summarizing the interview situation (video, conversational audio, conversational text, etc.) at the time when the interviewee is determined to be in a specific internal state during the interview. The feedback information generation unit 227 stores the generated feedback information in the feedback information storage unit 228.

[0452] Next, the feedback information transmitting unit 229 reads out the feedback information stored in the feedback information storage unit 228. The feedback information transmitting unit 229 transmits the read out feedback information to the interviewer terminal 30j via the network 50 (step S908). This completes the operation of the interview server 20j shown in the flowchart of FIG.

[0453] As described above, the discrimination device 10j of the interview system 1j in the sixth embodiment of the present invention is configured to discriminate the inner state of a target user who is the subject (interviewee) of inner state discrimination using sensing data such as image data captured by a camera of the body of the target user, audio data collected by a microphone of the target user's conversation, or vital data measured by a vital sensor on the target user's vital state. Furthermore, the discrimination device 10j is configured to discriminate the inner state of the target user based on a learning model trained by unsupervised machine learning, a learning model trained by supervised machine learning, or a rule base, and the sensing data.

[0454] With this configuration, the interview system 1j according to the sixth embodiment of the present invention can determine the inner state of the target user (interviewee) with higher accuracy even when there is a discrepancy between the outward emotional expression and the inner state. As a result, the interview system 1j can suppress a decrease in the accuracy of determination that may occur due to a discrepancy between the outward emotional expression and the inner state when determining a person's inner state through sensing.

[0455] Furthermore, as described above, the interview server 20j of the interview system 1j in the sixth embodiment of the present invention is configured to provide an interview support service that utilizes information indicating the inner state of the target user determined by the discrimination device 10j to provide feedback information to the interviewer. This allows the interview server 20j to provide the interviewer with information that associates the situation at the time of the interview with the inner state of the target user, thereby enabling a better interview to be realized.

[0456] According to the above-described embodiment, the discrimination device includes an acquisition unit, a learning unit, and a discrimination unit. For example, the discrimination device is the discrimination devices 10a to 10b, 10d to 10h in the embodiment, the acquisition unit is the learning data acquisition unit 111, 111b and the sensing data acquisition unit 111d to 111h, 121, 121b, 121d to 121h in the embodiment, the learning unit is the learning execution unit 113, 113b, 113d to 113h in the embodiment, and the discrimination unit is the inner state discrimination unit 124, 124b, 124d to 124h in the embodiment.

[0457] The acquisition unit acquires sensing data indicating at least one of the target user's movements and vital signs, which change depending on the target user's internal state. The learning unit performs machine learning using the learning sensing data and information indicating the target user's internal state, and generates a learning model that outputs information indicating the internal state in response to input sensing data. The discrimination unit discriminates the target user's internal state based on the learning model and the sensing data for inference.

[0458] In the above-described determination device, the determination unit may determine whether or not the target user is in a relaxed state.

[0459] The discrimination device may further include a designation unit. For example, the designation unit is the sensing data designation unit 119, 127 and the weighting designation unit 129 in the embodiment. The designation unit designates at least one of the items of the sensing data for learning to be used for machine learning, the items of the sensing data for inference to be used for discriminating the inner state, and the weighting for each item.

[0460] In the above-described embodiments, the discrimination device 10, 10b-10j, chat server 20i, and interview server 20j may be implemented in part or in whole by a computer. In this case, a program for implementing the functions may be recorded on a computer-readable recording medium, and the program may be loaded and executed by a computer system. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as floppy disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, and media that store programs for a fixed period of time, such as volatile memory within the computer systems that serve as the server or client. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in a computer system, or may be one that is realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0461] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the present invention that do not deviate from the gist of the present invention.

[0462] 1...Discrimination system, 1i...Chat system, 1j...Interview system, 10, 10a to 10j...Discrimination device, 11, 11b, 11d to 11h...Learning unit, 11c...Discrimination rule registration unit, 12, 12b to 12h...Inference unit, 13, 13e...Discrimination result output unit, 20...Service providing device, 20i...Chat server, 20j...Interview server, 30j...Interviewer terminal, 40...Sensor device, 40i...Terminal device, 40j...Interviewee terminal, 50...Network, 111, 1 11b... learning data acquisition unit, 111c... discrimination rule input unit, 111d to 111h... sensing data acquisition unit, 112, 112b to 112h... learning data storage unit, 113, 113b to 113h... learning execution unit, 114... inner surface state registration unit, 115, 115b, 115d to 115h... learning model storage unit, 115c... discrimination rule storage unit, 116... change detection unit, 117... sensing data storage unit, 118... inner surface state confirmation unit, 118e to 118h... inner surface state Acquisition unit, 119... sensing data designation unit, 121, 121b to 121h... sensing data acquisition unit, 124, 124b to 124h... inner state determination unit, 126, 126f... inner state confirmation unit, 127... sensing data designation unit, 128... designation item storage unit, 128h... weighting information storage unit, 129... weighting designation unit, 201... inner state acquisition unit, 202... inner state storage unit, 203... message reception unit, 204... inner state identification unit, 205... message Analysis unit, 206... Importance determination unit, 207... Notification rule designation unit, 208... Notification rule storage unit, 209... Notification method determination unit, 210... Notification control unit, 211... Message transfer unit, 221... Interview application, 222... Internal state acquisition unit, 223... Internal state storage unit, 224... Interview data acquisition unit, 225... Timekeeping unit, 226... Text conversion unit, 227... Feedback information generation unit, 228... Feedback information storage unit, 229... Feedback information transmission unit

Claims

1. A discrimination device comprising: an acquisition unit that acquires sensing data indicating at least one of a target user's movements and vital state, which change depending on the inner state of the target user; a learning unit that performs machine learning using the sensing data for learning and information indicating the inner state of the target user, and generates a learning model that outputs information indicating the inner state in response to input of the sensing data; and a discrimination unit that discriminates the inner state of the target user based on the learning model and the sensing data for inference.

2. The discrimination device according to claim 1, further comprising a designation unit that designates at least one of the items of the sensing data for learning to be used for the machine learning, the items of the sensing data for inference to be used for discriminating the inner state, and a weighting for each of the items.

3. A discrimination method by a computer, comprising: an acquisition step of acquiring sensing data indicating at least one of the target user's movements and vital state, which change depending on the inner state of the target user; a learning step of performing machine learning using the sensing data for learning and information indicating the inner state of the target user to generate a learning model that outputs information indicating the inner state in response to input of the sensing data; and a discrimination step of discriminating the inner state of the target user based on the learning model and the sensing data for inference.

4. A program for causing a computer to execute the following steps: an acquisition step of acquiring sensing data indicating at least one of the target user's movements and vital state, which change depending on the target user's inner state; a learning step of performing machine learning using the sensing data for learning and information indicating the target user's inner state, to generate a learning model that outputs information indicating the inner state in response to input of the sensing data; and a determination step of determining the inner state of the target user based on the learning model and the sensing data for inference.

Images