System, storage device, and storage program

Abstract

Provided is a mechanism for efficiently acquiring information for accommodating individual differences among users. This system comprises: a first information acquisition unit that acquires pieces of first information pertaining to users; a second information acquisition unit that acquires pieces of second information pertaining to the users; a classification unit that classifies, when feature amounts are extracted from the acquired pieces of the first information, the pieces of the first information from which the feature amounts have been extracted into a plurality of groups by grouping together first information in which the extracted feature amounts are similar to each other; an association unit that associates, when psychological state information which pertains to the psychological state of the users is identified on the basis of second information allocated in a predetermined time range among the acquired pieces of the second information, the identified psychological state information with the first information classified into a group related to the identified psychological state information; and a storage unit that stores the first information associated with the psychological state information.

Description

System, storage device and storage program

[0001] The present disclosure relates to a system, a storage device, and a storage program.

[0002] 2. Description of the Related Art There is known a technique for capturing an image of a user's face using an imaging device or the like, and estimating the emotion of the user based on feature amounts extracted from the captured image data.

[0003] On the other hand, facial expressions vary from person to person, and even if feature amounts extracted from image data of multiple users are similar to each other, it does not necessarily mean that the emotions of the multiple users are the same. For example, in the case of a user who naturally has droopy eyes or a frowning face, estimating the emotion using the extracted feature amounts may result in an incorrect estimation.

[0004] In response to this, a technology has been proposed that absorbs individual differences in users' facial expressions. Specifically, one technology is one in which an analyst analyzes extracted features and converts the features according to individual differences in users' facial expressions, thereby customizing the estimation of a user's emotion for each user. According to this technology, the user's emotion is estimated based on the converted features, so that individual differences in users' facial expressions can be absorbed, and highly accurate emotion estimation can be achieved.

[0005] Japanese Patent Application Laid-Open No. 2021-56591

[0006] However, customizing the system for each user places a heavy workload on the analyst, resulting in inefficiency. Additionally, there is also the problem that the actual emotions of the user may not necessarily match the emotions that the analyst reads from the image data.

[0007] One aspect of the present invention is to provide a mechanism for efficiently acquiring appropriate information to accommodate individual differences among users.

[0008] According to one aspect, the system includes: a first information acquisition unit that acquires first information about a user; a second information acquisition unit that acquires second information about the user; a classification unit that, when features are extracted from the acquired first information, classifies the first information from which the extracted features are extracted into multiple groups for each piece of first information whose extracted features are similar to each other; a linking unit that, when psychological state information that is information about the psychological state of the user is identified based on second information within a predetermined time range from the acquired second information, links the identified psychological state information to first information classified into a group related to the identified psychological state information; and a storage unit that stores the first information to which the psychological state information is linked.

[0009] It is possible to provide a mechanism for efficiently acquiring information that absorbs individual differences among users.

[0010] FIG. 1 is a diagram illustrating an application example of a multimodal system. FIG. 2 is a diagram illustrating an example of a system configuration of a multimodal system. FIG. 3 is a diagram illustrating an example of a hardware configuration of a storage device. FIG. 4 is a diagram illustrating an example of a functional configuration of a storage device. FIG. 5 is a diagram illustrating a specific example of processing by a determination unit of the storage device. FIG. 6 is a diagram illustrating a specific example of processing by a feature extraction unit of the storage device. FIG. 7 is a diagram illustrating a specific example of processing by a similarity calculation unit of the storage device. FIG. 8 is a diagram illustrating a specific example of processing by a classification unit of the storage device. FIG. 9 is a diagram illustrating a specific example of processing by a psychological state information identification unit of the storage device. FIG. 10 is a diagram illustrating a specific example of processing by a linking unit of the storage device. FIG. 11 is a first diagram illustrating a specific example of processing by an offset processing unit of the storage device. FIG. 12 is a second diagram illustrating a specific example of processing by an offset processing unit of the storage device. FIG. 13 is a third diagram illustrating a specific example of processing by an offset processing unit of the storage device. FIG. 14 is a flowchart illustrating the flow of storage processing by the storage device. FIG. 15 is a diagram illustrating an example of the functional configuration of a learning device. FIG. 16 is a diagram illustrating a specific example of processing by a learning data generation unit of the learning device. Fig. 17 is a diagram showing a specific example of processing by a learning unit of a learning device. Fig. 18 is a flowchart showing the flow of learning processing by the learning device. Fig. 19 is a diagram showing an example of the functional configuration of an estimation device. Fig. 20 is a diagram showing a specific example of processing by a feature extraction unit of an estimation device. Fig. 21 is a diagram showing a specific example of processing by an offset processing unit of an estimation device. Fig. 22 is a first diagram showing a specific example of processing by an estimation unit of an estimation device. Fig. 23 is a flowchart showing the flow of estimation processing by the estimation device. Fig. 24 is a second diagram showing a specific example of processing by the estimation unit of an estimation device.

[0011] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] [First embodiment] <Application example of multimodal system> First, an application example of a multimodal system according to the first embodiment will be described. The multimodal system according to the first embodiment is a system that provides a mechanism for efficiently acquiring appropriate information for absorbing individual differences in a user's facial expression, and that estimates psychological state information (e.g., various emotions) that is information related to the psychological state of a user by combining: image data obtained by capturing an image of the user's face with an imaging device, and biometric information obtained by measuring the user's biometric information with a sensor.

[0013] Fig. 1 is a diagram showing an application example of a multimodal system. As shown in Fig. 1, the multimodal system 110 is mounted on, for example, a robot. Alternatively, the multimodal system 110 may be mounted on, for example, an information terminal. Alternatively, the multimodal system 110 may be mounted on, for example, a game device or the like.

[0014] Regardless of the hardware on which the multimodal system 110 is implemented, in the accumulation process, the multimodal system 110 acquires: image data obtained by photographing the face of the user 120 during communication; and biometric information obtained by measuring the biometric information of the user 120 during communication. Note that the multimodal system 110 is assumed to have an interactive function for communicating with the user 120. Alternatively, the multimodal system 110 is assumed to have an interactive function for the user 120 to communicate with another user (not shown) via the multimodal system 110.

[0015] In the accumulation process, the multimodal system 110 extracts features from the acquired image data of the user 120 and identifies psychological state information of the user 120 from the acquired biometric information of the user 120. The multimodal system 110 also accumulates the features and the psychological state information, and calculates information for absorbing individual differences in the user's facial expression (offset data used when performing offset processing on the features). Furthermore, the multimodal system 110 performs offset processing on the features using the information for absorbing individual differences in the user's facial expression (offset data), thereby calculating the feature after offset processing.

[0016] Next, in the learning process, the multimodal system 110 generates learning data including the feature amounts after the offset process and the psychological state information. The multimodal system 110 also uses the generated learning data to train a model, thereby generating a trained model that absorbs individual differences in the facial expressions of users.

[0017] Next, in the estimation process, the multimodal system 110 extracts features from image data obtained by photographing the face of the user 120 during communication. Furthermore, in the estimation process, the multimodal system 110 performs offset processing on the extracted features using the offset data, thereby acquiring features after the offset processing. Furthermore, in the estimation process, the multimodal system 110 inputs the features after the offset processing into the trained model, thereby estimating psychological state information of the user 120.

[0018] In this way, the multimodal system 110 can efficiently acquire information (offset data) for absorbing individual differences in the facial expressions of the user 120 by automatically performing a storage process that stores image data linked to psychological state information. At this time, the multimodal system 110 identifies the psychological state information of the user 120 from the biometric information, and therefore can acquire appropriate offset data using highly accurate psychological state information. In other words, the multimodal system 110 can efficiently acquire appropriate information (offset data) for absorbing individual differences in the facial expressions of the user.

[0019] Furthermore, in the learning process, the multimodal system 110 generates a trained model that absorbs individual differences in the user's facial expression by learning the model using the feature amounts after the offset process. As a result, the multimodal system 110 can estimate psychological state information that absorbs individual differences in the user's facial expression in the estimation process.

[0020] <System Configuration of Multimodal System> Next, a description will be given of the system configuration of the multimodal system 110. Fig. 2 is a diagram showing an example of the system configuration of the multimodal system.

[0021] As shown in FIG. 2 , the multimodal system 110 includes an imaging device 210 , a sensor 220 , an interactive device 230 , a storage device 240 , a learning device 250 , and an estimation device 260 .

[0022] The imaging device 210 is a device that captures an image of the face of the user 120 and generates image data. In the first embodiment, the imaging device 210 captures an image of the face of the user 120 using light with a wavelength of 360 nm or more and 2500 nm or less, for example.

[0023] The sensor 220 is a sensor that measures biometric information of the user 120. In the first embodiment, the sensor 220 measures the biometric information of the user 120 using, for example, electromagnetic waves with a frequency of 300 MHz or more and 300 GHz or less. Note that the biometric information of the user 120 measured by the sensor 220 includes information that can be used to identify the psychological state information of the user 120, such as heart rate, pulse wave, and body temperature.

[0024] The interactive device 230 is a device for communicating with the user 120 or for realizing communication between the user 120 and other users.

[0025] When the interactive device 230 is a device that communicates with the user 120, the interactive device 230 includes, for example, a voice input device, a voice recognition device, a voice output device, a display device, etc. In this case, the voice output device and the display device generate and output voice and a display image according to the psychological state information of the user 120 during communication, which is estimated by the estimation device 260, for example.

[0026] On the other hand, when the interactive device 230 is a device for realizing communication between the user 120 and another user, the interactive device 230 includes, for example, a voice input device, a communication device, a voice output device, a display device, etc. In this case, the communication device is a device that communicates with a device used by another user (a device having an interactive function). Specifically, the communication device transmits the following information acquired by the voice input device to the device used by the other user: voice data of the user 120; image data of the user's face captured by an imaging device; and psychological state information of the user 120 during communication estimated by the estimation device 260. In addition, the communication device receives the following information from the device used by the other user and notifies the voice output device and the display device.

[0027] The storage device 240 acquires image data of the face of the user 120 captured by the imaging device 210 and biometric information of the user 120 measured by the sensor 220. The storage device 240 also excludes from the acquired image data any image data that is not suitable for generating learning data from the classification targets, and then extracts features from the image data excluding the excluded image data. The storage device 240 also classifies the image data into multiple groups so that image data with similar extracted features are grouped together.

[0028] Furthermore, the storage device 240 identifies psychological state information of the user 120 during communication based on the acquired biometric information, and stores the identified psychological state information by linking it to image data classified into a related group.

[0029] The storage device 240 also selects a group in which the identified psychological state information is associated with a neutral emotion, and calculates offset data using the feature amounts of the image data classified into the selected group. As described above, the offset data is information for absorbing individual differences in the facial expressions of the user 120. The storage device 240 also uses the calculated offset data to perform offset processing on the feature amounts of the image data classified into each group, and notifies the learning device 250 of the feature amounts after the offset processing and the psychological state information.

[0030] The learning device 250 generates training data including the feature amounts after offset processing and the psychological state information notified by the storage device 240. The learning device 250 also uses the generated training data to train a model, thereby generating a trained model that absorbs individual differences in facial expressions of the user 120. Furthermore, the learning device 250 notifies the estimation device 260 of the generated trained model and the offset data.

[0031] The estimation device 260 extracts features from image data obtained by photographing the face of the user 120 during communication, and performs offset processing on the extracted features using offset data to obtain features after offset processing. The estimation device 260 also inputs the features after offset processing into a trained model to estimate psychological state information of the user 120 and output the information to the interactive device 230.

[0032] 2, the interactive device 230, the storage device 240, the learning device 250, and the estimation device 260 have been described as being realized by separate devices. However, in the multimodal system 110, the interactive device 230, the storage device 240, the learning device 250, and the estimation device 260 may be configured as an integrated device. Alternatively, any two or three of the interactive device 230, the storage device 240, the learning device 250, and the estimation device 260 may be configured as an integrated device.

[0033] <Hardware Configuration of Storage Device> Next, we will explain the hardware configuration of each device (interactive device 230, storage device 240, learning device 250, and estimation device 260) that makes up the multimodal system 110. Note that the interactive device 230, storage device 240, learning device 250, and estimation device 260 have similar hardware configurations, so here we will explain the hardware configuration of the storage device 240 as a representative.

[0034] 3 is a diagram showing an example of the hardware configuration of a storage device. As shown in Fig. 3, the storage device 240 includes a processor 301, a memory 302, an auxiliary storage device 303, an I / F (Interface) device 304, a communication device 305, and a drive device 306. The hardware components of the storage device 240 are connected to each other via a bus 307.

[0035] The processor 301 has various computing devices such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 301 reads various programs (for example, stored programs, etc.) into the memory 302 and executes them.

[0036] The memory 302 has a main storage device such as a read-only memory (ROM) or a random access memory (RAM). The processor 301 and the memory 302 form a so-called computer, and the processor 301 executes various programs read onto the memory 302, thereby enabling the computer to realize various functions.

[0037] The auxiliary storage device 303 stores various programs and various data used when the processor 301 executes the various programs.

[0038] When the multimodal system 110 is mounted on a robot, the I / F device 304 is a device for connecting the display device 310 and operation device 320, which are examples of external devices, with the storage device 240. The I / F device 304 accepts operations on the storage device 240 by an administrator (not shown) of the multimodal system 110 via the operation device 320. The I / F device 304 also outputs the results of the storage process by the storage device 240 and displays them to the administrator of the multimodal system 110 via the display device 310.

[0039] The communication device 305 is a communication device for communicating with other devices that make up the multimodal system 110 (if the multimodal system 110 is mounted on a robot, the imaging device 210, the sensor 220, and the learning device 250).

[0040] The drive device 306 is a device for loading a recording medium 330. The recording medium 330 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk. The recording medium 330 may also include semiconductor memory that records information electrically, such as a ROM or flash memory.

[0041] The various programs to be installed in the auxiliary storage device 303 are installed, for example, by setting the distributed recording medium 330 in the drive device 306 and reading the various programs recorded on the recording medium 330 by the drive device 306. Alternatively, the various programs to be installed in the auxiliary storage device 303 may be installed by being downloaded from a network (not shown) via the communication device 305.

[0042] <Details of Functional Configuration of Storage Device> Next, the functional configuration of the storage device 240 will be described in detail. Fig. 4 is a diagram showing an example of the functional configuration of the storage device. As described above, a storage program is installed in the storage device 240, and by executing this program, the storage device 240 functions as follows, as shown in Fig. 4: an image data acquisition unit 401, a determination unit 402, a feature extraction unit 403, a similarity calculation unit 404, and a classification unit 405; a biometric information acquisition unit 411, a psychological state information identification unit 412, and a linking unit 413; and an offset processing unit 414.

[0043] The image data acquisition unit 401 acquires image data captured by the imaging device 210 and notifies the determination unit 402 of the acquired image data.

[0044] The determination unit 402 determines image data that is not appropriate for generating learning data from the image data notified by the image data acquisition unit 401. Specifically, the determination unit 402 determines: - whether or not the notified image data contains movement of the user 120; - the distance to the user 120 contained in the notified image data; - whether or not the face of the user 120 is detected in the notified image data; and - the relative angle between the orientation of the face of the user 120 (facing forward from the user 120) contained in the notified image data and the imaging direction of the imaging device 210 that captured the user 120. Here, the "distance to the user 120" refers to, for example, the distance from the imaging device 210 that captures the user 120 to the user 120.

[0045] If any of the results of these determinations satisfy a predetermined condition, the determination unit 402 excludes the image data from the classification target. Specifically, if the determination unit 402 determines that the user 120 is moving as a result of determining whether the user 120 is moving, the determination unit 402 excludes the image data from the classification target. Furthermore, if the determination unit 402 determines that the distance to the user 120 is greater than or equal to a predetermined distance as a result of determining whether the face of the user 120 has been detected, the determination unit 402 excludes the image data from the classification target. Furthermore, if the determination unit 402 determines that the face of the user 120 has not been detected as a result of determining whether the face of the user 120 has been detected, the determination unit 402 excludes the image data from the classification target. Furthermore, if the determination unit 402 determines that the relative angle between the front direction of the user 120 and the shooting direction of the imaging device 210 that captured the user 120 is greater than or equal to a predetermined relative angle as a result of determining whether the relative angle is greater than or equal to a predetermined relative angle, the determination unit 402 excludes the image data from the classification target.

[0046] The determination unit 402 notifies the feature extraction unit 403 of the image data notified from the image data acquisition unit 401 , excluding the image data excluded from the classification target.

[0047] The feature extraction unit 403 has a feature extraction model, and uses the feature extraction model to extract features of the facial expression of the user 120 from the image data notified by the determination unit 402. The features of the facial expression of the user 120 extracted using the feature extraction model include, for example, the angle of the eyebrows of the user 120, the state of the wrinkles between the eyebrows of the user 120, the angle of the mouth of the user 120, etc.

[0048] The similarity calculation unit 404 calculates the similarity of the feature amounts between the image data for each image data whose feature amounts have been extracted by the feature amount extraction unit 403. The similarity calculation unit 404 calculates the similarity of the feature amounts between the image data, for example, by calculating the Euclidean distance between the feature amount vectors of the image data.

[0049] The classification unit 405 classifies each image data into a plurality of groups so that image data whose similarity calculated by the similarity calculation unit 404 is equal to or greater than a predetermined threshold value are grouped together. The classification unit 405 also notifies the linking unit 413 of each image data classified into each group.

[0050] The biometric information acquisition section 411 acquires biometric information of the user 120 measured by the sensor 220 and notifies the psychological state information identification section 412 of the acquired biometric information.

[0051] The psychological state information identification unit 412 has a psychological state information identification model, and by using the psychological state information identification model, identifies the psychological state information of the user 120 from the biometric information notified by the biometric information acquisition unit 411. In addition, the psychological state information identification unit 412 notifies the linking unit 413 of the identified psychological state information.

[0052] The linking unit 413 links the psychological state information identified by the psychological state information identification unit 412 to image data classified into a group related to the identified psychological state information. For example, when the psychological state information is identified based on biological information within a predetermined time range, the group related to the identified psychological state information refers to a group into which image data captured within the predetermined time range is classified.

[0053] Furthermore, the linking unit 413 stores the feature amount of the image data linked with the psychological state information in the storage unit 420 together with the psychological state information.

[0054] The offset processing unit 414 calculates offset data using feature quantities associated with neutral emotions as psychological state information, among the feature quantities stored in the storage unit 420. The offset processing unit 414 also performs offset processing on the feature quantities stored in the storage unit 420, using the calculated offset data. Furthermore, the offset processing unit 414 notifies the learning device 250 of the feature quantities and psychological state information after the offset processing.

[0055] <Details of processing by each part of the storage device> Next, details of processing by each part of the storage device 240 (here, the judgment unit 402, feature extraction unit 403, similarity calculation unit 404, classification unit 405, psychological state information identification unit 412, linking unit 413, and offset processing unit 414) will be described.

[0056] (1) Details of Processing by Determination Unit 402 Fig. 5 is a diagram showing a specific example of processing by the determination unit of the storage device. As shown in Fig. 5, the determination unit 402 further includes a movement determination unit 501, a distance determination unit 502, a face detection feasibility determination unit 503, a relative angle determination unit 504, and an image data exclusion unit 505.

[0057] The movement determination unit 501 calculates the three-dimensional position coordinates of each part of the user 120 (e.g., head, shoulders, chest, etc.) included in the image data notified from the image data acquisition unit 401. The movement determination unit 501 also calculates the difference value of the three-dimensional position coordinates of each part between image data adjacent in the time axis direction. Furthermore, if the movement determination unit 501 determines that the calculated difference value is equal to or greater than a predetermined threshold, it notifies the image data exclusion unit 505 of each of the adjacent image data as image data to be excluded from classification.

[0058] The distance determination unit 502 calculates the three-dimensional position coordinates of the head of the user 120 included in the image data notified from the image data acquisition unit 401. The distance determination unit 502 also calculates the distance from the image capture device 210 based on the calculated three-dimensional position coordinates. Furthermore, when the distance determination unit 502 determines that the calculated distance is equal to or greater than a predetermined threshold, it notifies the image data exclusion unit 505 that the image data is image data to be excluded from classification targets.

[0059] The face detection feasibility determination unit 503 performs image recognition processing on the image data notified from the image data acquisition unit 401. If the face of the user 120 cannot be detected as a result of the image recognition processing, the face detection feasibility determination unit 503 notifies the image data exclusion unit 505 that the image data is image data to be excluded from classification targets.

[0060] The relative angle determination unit 504 performs image recognition processing on the image data notified by the image data acquisition unit 401 to detect the face of the user 120 and calculate the direction of the detected face of the user 120 (the direction facing the user 120). The relative angle determination unit 504 also uses information related to the shooting direction to calculate the relative angle between the calculated direction of the face of the user 120 and the shooting direction. Note that the relative angle determination unit 504 is assumed to previously store information related to the shooting direction when the imaging device 210 captured an image of the user 120. Furthermore, if the relative angle determination unit 504 determines that the calculated relative angle is equal to or greater than a predetermined threshold, it notifies the image data exclusion unit 505 that the image data is image data to be excluded from classification targets.

[0061] The image data exclusion unit 505 excludes image data that has been notified by any of the motion determination unit 501 to the relative angle determination unit 504 as image data to be excluded, and notifies the feature extraction unit 403 of the image data excluding the excluded image data.

[0062] (2) Details of Processing by Feature Extraction Unit 403 Fig. 6 is a diagram showing a specific example of processing by the feature extraction unit of the storage device. As shown in Fig. 6, the feature extraction unit 403 has a feature extraction model 610, and extracts features (numeric values ​​between 0 and 1) by inputting each piece of image data notified by the determination unit 402 to the feature extraction model 610.

[0063] 6 shows a state in which 1024 features are extracted by the feature extraction unit 403 inputting the first image data (image data 1) of the image data notified from the determination unit 402 into the feature extraction model 610. Similarly, the example of Fig. 6 shows a state in which 1024 features are extracted by the feature extraction unit 403 inputting the nth image data (image data n) of the image data notified from the determination unit 402 into the feature extraction model 610.

[0064] 6 shows a case where a dimensionless numerical value (a numerical value between 0 and 1) is extracted as a feature, but a dimensional index (for example, xx degrees in the case of eyebrow angle) may be extracted as a feature. Alternatively, when a dimensionless numerical value (a numerical value between 0 and 1) is extracted, "present / absent" may be extracted as a feature by comparing whether it is greater or smaller than a predetermined threshold.

[0065] (3) Details of Processing by Similarity Calculation Unit 404 Fig. 7 is a diagram showing a specific example of processing by the similarity calculation unit of the storage device. As shown in Fig. 7, the similarity calculation unit 404 identifies feature vectors of each piece of image data (image data 1, image data 2, ..., image data n) from which features have been extracted by the feature extraction unit 403. The example in Fig. 7 shows how feature vector 1 is identified as the feature vector of image data 1, and feature vector 2 is identified as the feature vector of image data 2. The example in Fig. 7 also shows how feature vector n is identified as the feature vector of image data n.

[0066] As shown in FIG. 7, the similarity calculation unit 404 calculates the similarity by calculating the Euclidean distance between the feature vectors of each piece of image data (image data 1, image data 2, . . . image data n).

[0067] (4) Details of Processing by Classification Unit 405 Fig. 8 is a diagram showing a specific example of processing by the classification unit of the storage device. The classification unit 405 acquires each Euclidean distance between the feature vectors of each piece of image data (image data 1, image data 2, ..., image data n) calculated by the similarity calculation unit 404. The classification unit 405 also classifies image data for which each acquired Euclidean distance is equal to or less than a predetermined threshold into the same group as image data with similar features (image data with high similarity).

[0068] The example in FIG. 8 shows how each image data (image data 1, image data 2, . . . image data n) is classified into seven groups, group 1 to group 7.

[0069] (5) Details of Processing by the Psychological State Information Identification Unit 412 Fig. 9 is a diagram showing a specific example of processing by the psychological state information identification unit of the storage device. In Fig. 9, reference numeral 901 denotes an example of biometric information measured by the sensor 220 when each image data classified into each group is captured by the imaging device 210.

[0070] As described above, the psychological state information identification unit 412 has a psychological state information identification model 910, and by inputting the biometric information indicated by the symbol 901 into the psychological state information identification model 910, the psychological state information identification unit 412 identifies the psychological state information of the user 120 in each time range.

[0071] The example of Fig. 9 shows how the psychological state information of the user 120 in the time range 921 is identified as emotion A based on biometric information in the time range 921. Similarly, the example of Fig. 9 shows how the psychological state information of the user 120 in the time range 922 is identified as neutral based on biometric information in the time range 922. Similarly, the example of Fig. 9 shows how the psychological state information of the user 120 in the time range 923 is identified as emotion B based on biometric information in the time range 923. Similarly, the example of Fig. 9 shows how the psychological state information of the user 120 in the time range 924 is identified as emotion C based on biometric information in the time range 924. Similarly, the example of Fig. 9 shows how the psychological state information of the user 120 in the time range 925 is identified as emotion D based on biometric information in the time range 925. Similarly, the example of Fig. 9 shows how the psychological state information of the user 120 in the time range 926 is identified as emotion E based on biometric information in the time range 926. Similarly, the example of FIG. 9 shows how the psychological state information of the user 120 in a time range 927 is identified as emotion F based on the biological information in the time range 927 .

[0072] Emotions A to F include, for example, anger, disgust, fear, joy, sadness, surprise, etc. A neutral emotion refers to a state that does not belong to a specific emotion (i.e., a neutral state without any emotion).

[0073] (6) Details of Processing by the Linking Unit 413 FIG. 10 is a diagram showing a specific example of processing by the linking unit of the storage device. As shown in FIG. 10 , the linking unit 413 links the psychological state information identified by the psychological state information identification unit 412 to image data classified into a group related to the identified psychological state information. The example of FIG. 10 shows how emotion A is linked to image data classified into group 1 and how neutral emotion is linked to image data classified into group 2. The example of FIG. 10 also shows how emotion B is linked to image data classified into group 3, emotion C is linked to image data classified into group 4, and emotion D is linked to image data classified into group 5. Furthermore, the example of FIG. 10 shows how emotion E is linked to image data classified into group 6 and emotion F is linked to image data classified into group 7.

[0074] (7) Details of Processing by Offset Processing Unit 414 FIGS. 11 to 13 are first to third diagrams showing specific examples of processing by the offset processing unit of the storage device.

[0075] 11, the offset processing unit 414 selects image data classified into a group associated with a neutral emotion as psychological state information. The example in Fig. 11 shows how the offset processing unit 414 selects image data 5 to 10 classified into group 2.

[0076] 12, the offset processing unit 414 calculates offset data based on the feature amount extracted from the selected image data 5 to 10. In the example of FIG. 12, the offset data is calculated based on the feature amount of "eyebrow angle" of the image data 5 to 10. ｏｆｆｓｅｔ 12 shows how the offset data (values ​​between 0 and 1) is calculated based on the feature amount of "brow wrinkles" of image data 5 to 10. ｏｆｆｓｅｔ 12 shows how offset data=Mθ (a value between 0 and 1) is calculated based on the feature amount of "mouth angle" of image data 5 to 10. ｏｆｆｓｅｔ(a value between 0 and 1) is calculated.

[0077] The calculation method shown in FIG. 12 is an example of a method for calculating offset data by the offset processing unit 414, and the method for calculating offset data by the offset processing unit 414 is not limited to the calculation method shown in FIG.

[0078] For example, when a dimensional index (for example, xx degrees in the case of eyebrow angle) is extracted as a feature, a dimensional index may also be calculated for the offset data.

[0079] 13, the offset processing unit 414 uses the calculated offset data to perform offset processing on the feature values ​​extracted from each image data (image data 1, image data 2, ..., image data n), and calculates the feature values ​​after the offset processing. In the example of FIG. 13, for image data 1 associated with emotion A, the following is calculated: EBθ as the eyebrow angle after the offset processing １ -EBθ ｏｆｆｓｅｔ is calculated, and BEBθ is calculated as the wrinkle between the eyebrows after the offset processing. １ -BEBθ ｏｆｆｓｅｔ is calculated, and... As the mouth angle after offset processing, Mθ １ -Mθ ｏｆｆｓｅｔ is calculated, and the figure shows how it is done.

[0080] Similarly, in the example of FIG. 13, for image data n associated with emotion F, the following is set: ｎ -EBθ ｏｆｆｓｅｔ is calculated, and BEBθ is calculated as the wrinkle between the eyebrows after the offset processing. ｎ -BEBθ ｏｆｆｓｅｔ is calculated, and... As the mouth angle after offset processing, Mθ ｎ -Mθ ｏｆｆｓｅｔ is calculated, and the figure shows how it is done.

[0081] <Storage Process Flow> Next, a description will be given of the flow of storage process by the storage device 240. Fig. 14 is a flowchart showing the flow of storage process by the storage device.

[0082] In step S 1401 , the storage device 240 acquires image data from the image capture device 210 .

[0083] In step S1402, the storage device 240 evaluates the acquired image data and excludes image data that is not suitable for generating learning data.

[0084] In step S1403, the storage device 240 extracts features from the image data excluding the excluded image data.

[0085] In step S1404, the storage device 240 calculates the similarity of the feature amounts for each image data from which the feature amounts have been extracted.

[0086] In step S1405, the storage device 240 classifies each image data into a plurality of groups so that image data with similar features are grouped together.

[0087] In step S 1406 , the storage device 240 acquires the biometric information from the sensor 220 .

[0088] In step S1407, the storage device 240 identifies psychological state information based on the biological information.

[0089] In step S1408, the storage device 240 links the identified psychological state information to image data classified into a group related to the psychological state information.

[0090] In step S1409, the storage device 240 stores the feature amount extracted from the image data and the psychological state information linked to the image data.

[0091] In step S1410, the storage device 240 calculates offset data using the feature amount extracted from the image data associated with the neutral emotion, and performs offset processing on the stored feature amount using the calculated offset data.

[0092] In step S1411, the storage device 240 notifies the learning device of the feature amount after the offset processing and the psychological state information.

[0093] <Functional Configuration of Learning Device> Next, the functional configuration of the learning device 250 will be described. Fig. 15 is a diagram showing an example of the functional configuration of the learning device. A learning program is installed in the learning device 250, and by executing this program, the learning device 250 functions as a learning data generation unit 1510 and a learning unit 1520, as shown in Fig. 15.

[0094] The learning data generating unit 1510 acquires the feature amounts after offset processing and the psychological state information from the storage device 240, and generates learning data.

[0095] The learning unit 1520 generates a trained model by learning the model using the training data generated by the training data generation unit 1510. The learning unit 1520 also notifies the estimation device 260 of the generated trained model and offset data.

[0096] <Details of Processing by Each Unit of the Learning Device> Next, details of processing by each unit of the learning device 250 (learning data generation unit 1510, learning unit 1520) will be described.

[0097] 16 is a diagram showing a specific example of processing by the learning data generation unit 1510. As described above, the learning data generation unit 1510 acquires the feature amounts after offset processing and psychological state information, and generates learning data.

[0098] 16, training data 1600_1 to 1600_n indicate the training data for image data 1 to image data n, respectively, generated by training data generation unit 1510. As shown in Fig. 16, training data 1600_1 to 1600_n include information items such as "feature identifier," "feature item," "input data," and "correct answer data."

[0099] "Feature identifier" stores an identifier for identifying a feature extracted from image data. "Feature item" stores an item of a feature extracted from image data. "Input data" stores the feature after offset processing notified from the storage device 240. "Correct answer data" stores psychological state information notified from the storage device 240.

[0100] 17 is a diagram showing a specific example of processing by the learning unit of the learning device. As shown in FIG. 17, the learning unit 1520 of the learning device 250 further includes a model 1710 and a comparison / modification unit 1720.

[0101] The model 1710 outputs psychological state information when the feature values ​​after offset processing stored in the "input data" of the learning data are input.

[0102] When psychological state information stored in the "correct answer data" of the learning data is input, the comparison and modification unit 1720 calculates the error between this information and the psychological state information output from the model 1710, and updates the model parameters of the model 1710 based on the calculated error. In this way, the model 1710 is trained, and a trained model is generated.

[0103] <Learning Process Flow by Learning Device> Next, a description will be given of the learning process flow by the learning device 250. Fig. 18 is a flowchart showing the learning process flow by the learning device.

[0104] In step S1801, the learning device 250 acquires the feature amount after the offset processing and the psychological state information from the storage device 240.

[0105] In step S1802, the learning device 250 generates learning data including the acquired feature amounts after offset processing and psychological state information.

[0106] In step S1803, the learning device 250 uses the generated learning data to learn a model and generate a trained model.

[0107] In step S1804, the learning device 250 notifies the estimating device 260 of the generated trained model together with the offset data.

[0108] <Functional Configuration of Estimation Device> Next, the functional configuration of the estimation device 260 will be described. Fig. 19 is a diagram showing an example of the functional configuration of the estimation device. An estimation program is installed in the estimation device 260, and by executing the estimation program, the estimation device 260 functions as an image data acquisition unit 1910, a feature extraction unit 1920, and an offset processing unit 1930, as shown in Fig. 19. The estimation device 260 also functions as an estimation unit 1940 and an estimation result output unit 1950.

[0109] The image data acquisition unit 1910 acquires image data captured by the image capture device 210 and notifies the feature extraction unit 1920 of the acquired image data.

[0110] The feature extraction unit 1920 has a feature extraction model, and uses the feature extraction model to extract features of the facial expression of the user 120 from the image data notified by the image data acquisition unit 1910 .

[0111] The offset processing unit 1930 performs offset processing on the feature extracted by the feature extraction unit 1920 using the offset data notified by the learning device 250 .

[0112] The estimation unit 1940 estimates psychological state information of the user 120 by inputting the feature amount after offset processing calculated by the offset processing unit 1930 into the trained model notified by the training device 250.

[0113] The estimation result output unit 1950 notifies the interactive device 230 of the mental state information estimated by the estimation unit 1940 .

[0114] <Details of Processing by Each Unit of Estimation Device> Next, details of processing by each unit of the estimation device 260 (here, the feature extraction unit 1920, the offset processing unit 1930, and the estimation unit 1940) will be described.

[0115] (1) Details of Processing by the Feature Extraction Unit 1920 Fig. 20 is a diagram showing a specific example of processing by the feature extraction unit of the estimation apparatus. As shown in Fig. 20, the feature extraction unit 1920 has a feature extraction model 610, and extracts features by inputting each piece of image data notified from the image data acquisition unit 1910 to the feature extraction model 610.

[0116] 20 shows a state in which 1024 features are extracted by the feature extraction unit 1920 inputting the (m+1)th image data (image data m+1) out of the image data notified by the image data acquisition unit 1910 into the feature extraction model 610. Similarly, the example of Fig. 20 shows a state in which 1024 features are extracted by the feature extraction unit 1920 inputting the (m+n)th image data (image data m+n) out of the image data notified by the image data acquisition unit 1910 into the feature extraction model 610.

[0117] (2) Details of Processing by the Offset Processing Unit 1930 Fig. 21 is a diagram showing a specific example of processing by the offset processing unit of the estimation apparatus. As shown in Fig. 21, the offset processing unit 1930 performs offset processing using offset data on the feature amounts of image data m+1 to m+n calculated by the feature amount extraction unit 1920, thereby calculating feature amounts after the offset processing.

[0118] In FIG. 21, reference numerals 2100_m+1 to 2100_m+n indicate the state in which the feature amounts after the offset processing of image data m+1 to image data m+n are calculated, respectively.

[0119] (3) Details of Processing by the Estimation Unit 1940 Fig. 22 is a first diagram illustrating a specific example of processing by the estimation unit of the estimation device. As shown in Fig. 22, the estimation unit 1940 has a trained model 2210, and estimates psychological state information by inputting the feature amounts after offset processing calculated by the offset processing unit 1930 to the trained model 2210. Specifically, the estimation unit 1940 estimates one of emotions A to F or a neutral emotion as the psychological state information.

[0120] <Flow of Estimation Process by Estimation Device> Next, we will explain the flow of estimation process by the estimation device 260. Fig. 23 is a flowchart showing the flow of estimation process by the estimation device.

[0121] In step S2301, the estimation device 260 acquires image data from the imaging device 210 and extracts feature amounts.

[0122] In step S2302, the estimating device 260 performs offset processing on the extracted feature amounts and calculates the feature amounts after the offset processing.

[0123] In step S2303, the estimation device 260 estimates the psychological state information of the user 120 by inputting the feature amounts after the offset processing into the trained model.

[0124] In step S2304, the estimating device 260 notifies the interactive device 230 of the estimated psychological state information.

[0125] <Summary> As is clear from the above description, the multimodal system 110 according to the first embodiment includes: an image data acquisition unit 401 that acquires image data of the user 120; a biometric information acquisition unit 411 that acquires biometric information of the user 120 during image capture; a classification unit 405 that, when feature amounts are extracted from each of the acquired image data, classifies the image data from which the feature amounts have been extracted into multiple groups for each of the image data having similar extracted feature amounts; a linking unit 413 that, when psychological state information of the user 120 is identified based on biometric information within a predetermined time range from the acquired biometric information, links the identified psychological state information to image data classified into a group related to the identified psychological state information; and a storage unit 420 that stores image data linked with the psychological state information.

[0126] In this way, the multimodal system 110 according to the first embodiment automatically performs the accumulation process of accumulating image data linked to psychological state information, thereby making it possible to efficiently acquire offset data, which is information for absorbing individual differences in the facial expressions of the user 120.

[0127] Furthermore, the multimodal system 110 according to the first embodiment identifies psychological state information of the user 120 from biometric information. As a result, according to the first embodiment, it is possible to acquire appropriate offset data using highly accurate psychological state information.

[0128] That is, according to the first embodiment, it is possible to provide a mechanism for efficiently acquiring appropriate information for absorbing individual differences in the facial expressions of users.

[0129] Second Embodiment In the first embodiment, the estimation device 260 is configured to use feature amounts extracted from image data when estimating psychological state information. However, the method by which the estimation device 260 estimates psychological state information is not limited to this. For example, the estimation device 260 may be configured to estimate the psychological state information of the user 120 by further taking into account psychological state information identified from biometric information measured by a sensor. The second embodiment will be described below, focusing on the differences from the first embodiment.

[0130] Fig. 24 is a second diagram showing a specific example of processing by the estimation unit of the estimation device. The difference from the estimation unit 1940 of the estimation device 260 described using Fig. 22 is that the estimation unit 2400 of the estimation device 260 shown in Fig. 24 includes a psychological state information identification unit 2410 and an integration unit 2420 in addition to the learned unit 2210.

[0131] The psychological state information identification section 2410 has a psychological state information identification model, and identifies the psychological state information of the user 120 from the biological information 2430 using the psychological state information identification model.

[0132] The integration unit 2420 estimates the psychological state information of the user 120 based on the psychological state information estimated by the trained model 2210 and the psychological state information identified by the psychological state information identification unit 2410.

[0133] In this way, by configuring the system to estimate the psychological state information of user 120 using both the features extracted from the image data and the biometric information, the accuracy of estimation of the psychological state information can be improved compared to when only the features extracted from the image data are used.

[0134] [Third Embodiment] The multimodal system 110 according to the first embodiment is configured to acquire: image data obtained by capturing an image of the face of the user 120 using an imaging device; and biometric information obtained by measuring the biometric information of the user 120 using a sensor. However, the information used by the multimodal system 110 when estimating the psychological state information of the user 120 is not limited to a combination of image data and biometric information. For example, any other combination may be used as long as it combines first information about the user 120 with second information about the user 120 that is different from the first information. Examples of the first information and second information about the user 120 include information about physical changes in the user 120 and information about physiological changes in the user 120.

[0135] For example, the information may be a combination of information regarding physical changes in user 120 (first information regarding user 120) and information regarding physiological changes in user 120 (second information regarding user 120), or a combination of information regarding physical changes in user 120 (first information regarding user 120) and information regarding physical changes different from the first information regarding user 120 (second information regarding user 120), or a combination of information regarding physiological changes in user 120 (first information regarding user 120) and information regarding physiological changes different from the first information regarding user 120 (second information regarding user 120).

[0136] The information related to the physical changes of the user 120 referred to here includes, for example: the facial expression, gaze or eye movement, blinking, and pupils of the user 120, which are measured using an imaging device, an electro-oculometer, etc. the posture and behavior of the user 120, which are measured using an imaging device, an acceleration sensor, etc. the voice, linguistic content, and acoustic features (fundamental frequency, formant, intensity, fluctuation, speed, voice quality, intonation) of the user 120, which are measured using a voice input device, etc. When the information related to the physical changes of the user 120 is used as the first information related to the user 120, a first information acquisition unit that acquires information related to the physical changes of the user 120 is arranged in the storage device 240 instead of the image data acquisition unit 401.

[0137] Furthermore, the information related to physiological changes of the user 120 includes, for example: the electroencephalogram and cerebral blood flow of the user 120 measured using an electroencephalograph, a near-infrared spectroscopy (NIRS) system, etc. the electrocardiogram of the user 120 measured using an electrocardiograph, etc. the pulse wave of the user 120 measured using a photoplethysmography (PPG) device, a remote photoplethysmography (RPGG) device, an electromagnetic wave sensor, etc. the amount of sweat and components of sweat of the user 120 measured using a sweat meter, a patch sensor, etc. the state of muscle tension measured using an electromyometer, etc. When the information related to physiological changes of the user 120 is used as the second information related to the user 120, the storage device 240 is provided with a second information acquisition unit that acquires information related to physiological changes of the user 120 instead of the biological information acquisition unit 411.

[0138] In this way, the multimodal system 110 may be configured to estimate psychological state information of the user 120 by arbitrarily combining any of the information contained in the first information with any of the information contained in the second information.

[0139] [Other Embodiments] In the first embodiment, a case has been described in which offset processing is performed on the user 120 as a target. However, the target is not limited to one person, and offset processing may be performed on each of multiple targets including the user 120. In this case, offset data may be calculated for each target and stored in the estimation device 260. In this case, the estimation device 260 is configured to identify the user, read out offset data corresponding to the identified user, and perform offset processing.

[0140] Furthermore, in the first embodiment described above, the determination unit 402 is configured to determine image data to be excluded by determining: - whether or not there is movement of the user 120 contained in the notified image data; - the distance to the user 120 contained in the notified image data; - whether or not the face of the user 120 is detected in the notified image data; and - the relative angle between the direction of the face of the user 120 contained in the notified image data (the front direction of the user 120) and the shooting direction when the user 120 was photographed. However, the method by which the determination unit 402 determines image data to be excluded is not limited to this.

[0141] In the first embodiment, the feature extraction model 610 is described as extracting 1024 feature quantities. However, the number of feature quantities extracted by the feature extraction model 610 is not limited to 1024.

[0142] Furthermore, in the first embodiment, the similarity calculation unit 404 calculates the similarity by calculating the Euclidean distance between the feature vectors of each image data, but the method of calculating the similarity is not limited to this.

[0143] Furthermore, in the first embodiment, the image data is classified into seven groups, but the number of groups to be classified is not limited to seven.

[0144] Furthermore, in the first embodiment, seven types of emotions are exemplified as psychological state information, but the emotions included in the psychological state information are not limited to seven types.

[0145] Furthermore, although the first embodiment does not mention a method for calculating offset data, the offset data may be calculated, for example, by calculating an average value of feature amounts extracted from each image data associated with a neutral emotion.

[0146] The present invention is not limited to the configurations described in the above embodiments, but may be combined with other elements, etc. These aspects can be changed without departing from the spirit of the present invention, and can be appropriately determined depending on the application form.

[0147] This application claims priority based on Japanese Patent Application No. 2023-166347, filed on September 27, 2023, the entire contents of which are incorporated herein by reference.

[0148] 110: Multimodal system 210: Imaging device 220: Sensor 230: Interactive device 240: Storage device 250: Learning device 260: Estimation device 401: Image data acquisition unit 402: Determination unit 403: Feature extraction unit 404: Similarity calculation unit 405: Classification unit 411: Biometric information acquisition unit 412: Psychological state information identification unit 413: Linking unit 414: Offset processing unit 501: Movement determination unit 502: Distance determination unit 503: Face detection feasibility determination unit 504: Relative angle determination unit 505: Image data exclusion unit 610: Feature extraction model 910: Psychological state information identification model 1510: Learning data generation unit 1520: Learning unit 1710: Model 1910 : Image data acquisition unit 1920 : Feature extraction unit 1930 : Offset processing unit 1940 : Estimation unit 1950 : Estimation result output unit 2210 : Trained model 2400 : Estimation unit 2410 : Mental state information identification unit 2420 : Integration unit

Claims

1. A system having a first information acquisition unit that acquires first information related to a user; a second information acquisition unit that acquires second information related to a user; a classification unit that, when features are extracted from the acquired first information, classifies the first information from which the features are extracted into a plurality of groups for each piece of first information having similar features; a linking unit that, when psychological state information that is information related to the psychological state of the user is identified based on second information within a predetermined time range from the acquired second information, links the identified psychological state information to first information classified into a group related to the identified psychological state information; and a storage unit that stores the first information to which the psychological state information is linked.

2. The system according to claim 1, wherein the first information is image data of the user, and the second information is biometric information of the user during the photographing process.

3. The system according to claim 2, further comprising an imaging device that captures an image of the user, the imaging device capturing an image of the user using light having a wavelength of 360 nm or more and 2500 nm or less.

4. The system according to claim 2, further comprising a sensor for measuring the user's biometric information, the sensor measuring the user's biometric information using electromagnetic waves with a frequency of 300 MHz or more and 300 GHz or less.

5. The system of claim 2, wherein the psychological state information is identified by inputting biological information for a predetermined time range into a psychological state information identification model.

6. The system described in claim 2, further comprising a determination unit that determines, based on the acquired image data, whether or not the user is moving, the distance to the user, whether or not the user's face is detected, or the relative angle of the user's forward direction relative to the shooting direction when the user was photographed.

7. The system described in claim 6, wherein image data whose judgment results satisfy predetermined conditions are excluded from the classification target, and the classification unit classifies the image data, excluding the image data excluded from the classification target, into multiple groups for each image data whose extracted features are similar to each other when features are extracted from each of the acquired image data, excluding the image data excluded from the classification target.

8. The system according to claim 2, further comprising a determination unit that determines, based on the acquired image data, whether or not the user's face has been detected, and the relative angle of the user's forward direction relative to the shooting direction when the user was photographed, or determines, based on the acquired image data, whether or not the user has moved, and the distance to the user.

9. The system described in claim 8, wherein image data for which the results of a determination as to whether or not the user's face has been detected and the relative angle of the user's forward direction relative to the shooting direction when the user was photographed satisfy predetermined conditions is excluded from the classification target, and the classification unit classifies the image data, excluding the image data excluded from the classification target, into a plurality of groups for each image data for which the extracted features are similar to each other when features are extracted from each of the acquired image data, excluding the image data excluded from the classification target.

10. The system described in claim 8, wherein image data in which the results of the determination regarding the presence or absence of the user's movement and the distance to the user satisfy predetermined conditions are excluded from the classification target, and the classification unit classifies the image data, excluding the image data excluded from the classification target, into multiple groups for each image data in which the extracted features are similar to each other when features are extracted from each of the acquired image data, excluding the image data excluded from the classification target.

11. The system described in claim 2, further comprising a feature extraction model that extracts features from each of the acquired image data, wherein when features are extracted from each of the acquired image data by the feature extraction model, the classification unit classifies the image data from which the features are extracted into a plurality of groups for each piece of image data in which the extracted features are similar to each other.

12. The system described in claim 11, further comprising a similarity calculation unit that calculates a Euclidean distance between feature vectors of each image data as the similarity of the extracted features, and the classification unit classifies the image data from which the features have been extracted into a plurality of groups for each image data for which the Euclidean distance calculated by the similarity calculation unit is equal to or less than a predetermined threshold when features are extracted from each of the acquired image data.

13. The system of claim 2, wherein the group related to the identified psychological state information is a group in which image data taken in a time range corresponding to the biometric information in the specified time range is classified.

14. The system described in claim 2, further comprising: a calculation unit that calculates offset data based on features extracted from image data to which psychological state information indicating a neutral emotion is linked, among the accumulated image data to which the psychological state information is linked; and an offset processing unit that performs offset processing using the offset data on the features extracted from the image data to which the psychological state information is linked.

15. The system described in claim 14, further comprising a learning unit that performs model training using training data including the features after the offset processing and the psychological state information, and generates a trained model.

16. The system described in claim 15, further comprising an estimation unit that performs offset processing using the offset data on features extracted from image data photographing a user, and when the features after the offset processing are calculated, estimates psychological state information of the user by inputting the features after the offset processing into the trained model.

17. A storage device having: a first information acquisition unit that acquires first information related to a user; a second information acquisition unit that acquires second information related to a user; a classification unit that, when features are extracted from the acquired first information, classifies the first information from which the extracted features are extracted into a plurality of groups for each piece of first information that is similar to each other; a linking unit that, when psychological state information that is information related to the psychological state of the user is identified based on second information within a predetermined time range from the acquired second information, links the identified psychological state information to first information classified into a group related to the identified psychological state information; and a storage unit that stores the first information to which the psychological state information is linked.

18. A storage program for causing a computer to execute the following steps: acquiring first information about a user; acquiring second information about a user; when features are extracted from the acquired first information, classifying the first information from which the features are extracted into a plurality of groups for each piece of first information whose extracted features are similar to each other; when psychological state information that is information about the psychological state of the user is identified based on second information within a predetermined time range from the acquired second information, linking the identified psychological state information to first information classified into a group related to the identified psychological state information; and storing the first information to which the psychological state information is linked.

Images