Estimation emotion processing device, method for producing the same, program for estimation emotion processing, and recording medium

Abstract

Provided is an estimated emotion processing device that performs data processing by using estimated emotions, said device realizing: a configuration in which the degree of design freedom of hardware resources is prevented from decreasing and emotion estimation accuracy can be improved; and a configuration in which if the levels of emotion estimation accuracy are about the same, the degree of design freedom of hardware resources can be improved. An estimated emotion processing device 1 has a control unit 12 that outputs output data including estimated emotion information and other information. The estimated emotion information includes estimated emotion color information, in which: the difference in emotional intensity is expressed in gradation of shades of color; and color data is included that represents an estimated emotion, in a color reference such that different types of emotions are expressed in gradation of color tone. The other information includes influential factor indirect visual information, which is visual information that indirectly indicates an emotion. Output data includes composite visual indirect emotion information configured in a data format where both the estimated emotion color information and the influential factor indirect visual information are simultaneously displayed.

Description

Technical Field

[0001] This invention relates to an estimated emotion processing device that associates and outputs estimated emotion information indicating an estimated emotion and other information related to the estimated emotion.

Background Art

[0002] Numerous estimated emotion processing devices have been proposed that perform data processing using an estimated emotion estimated based on various data obtained from an emotion estimation target person who is the target of emotion estimation.

[0003] For example, Patent Document 1 discloses a device that performs emotion analysis based on biological information and records an emotion parameter indicating the degree of emotion obtained as a result in association with information content data. Further, Patent Document 2 discloses a device that represents the emotional state of a live viewer watching content as a reaction of the live viewer at a specific moment on the device of an online viewer who is watching the content online.

[0004] Further, Patent Document 3 discloses a device that allows other users to relive the emotion of a user at the time of content recording by reproducing biological information and content. Also, Patent Document 4 discloses a device that controls the driving of a ride in an amusement park based on the state of a passenger such as the emotional state or attention of the passenger.

[0005] Further, Patent Document 5 discloses a device that estimates the emotion and cognitive ability of a worker based on the biological measurement data and movement measurement data of the worker and learning data, and estimates the production ability of the worker based on the estimation result and relational expression. Also, Patent Document 6 discloses a device that evaluates tire performance using a signal of biological information (such as electroencephalogram) that can be obtained from a passenger of a vehicle equipped with an evaluation target tire.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2005-124909 [Patent Document 2] U.S. Patent No. 9854410 [Patent Document 3] Japanese Patent Publication No. 2017-211862 [Patent Document 4] U.S. Patent No. 9943769 [Patent Document 5] Japanese Patent Publication No. 2018-142258 [Patent Document 6] Japanese Patent Publication No. 2011-158402 [Overview of the project] [Problems that the invention aims to solve]

[0007] Each of the aforementioned patent documents discloses an estimated emotion processing device that uses the estimated emotions of an emotion target to record, live view, play content, drive control, manage, or evaluate. When using such an estimated emotion processing device, when managing the estimated emotion processing device, or when evaluating the performance of the estimated emotion processing device, there are situations in which multiple estimated emotions estimated based on data obtained from one emotion target are compared, and / or situations in which estimated emotions of different emotion target individuals are compared.

[0008] In order to popularize the aforementioned emotion estimation device, it is necessary to achieve an accuracy in emotion estimation that is acceptable not only to the person whose emotions are being estimated, but also to the users of the emotion estimation device, the decision-makers who evaluate the emotion estimation device and decide whether to adopt or introduce it, and the administrators who manage and operate the emotion estimation device, among other stakeholders involved with the emotion estimation device, in each of the situations in which estimated emotions are compared as described above.

[0009] To improve the accuracy of emotion estimation, one could consider increasing the types or amount of data, such as biometric information, used for emotion estimation. While this can improve the accuracy of emotion estimation, increasing the types or amount of data used for emotion estimation reduces the design flexibility of the hardware resources in the emotion estimation processing unit.

[0010] The object of the present invention is to realize a configuration for an estimated emotion processing device that performs data processing using estimated emotions, which can improve the accuracy of emotion estimation while suppressing a decrease in the design flexibility of hardware resources, or, if the level of emotion estimation accuracy is about the same, can realize a configuration which can improve the design flexibility of hardware resources. [Means for solving the problem]

[0011] The inventors of the present invention have investigated a configuration for an estimated emotion processing device that performs data processing using estimated emotions, which can improve the accuracy of emotion estimation while suppressing a decrease in the design flexibility of hardware resources, or, if the level of emotion estimation accuracy is the same, which can realize a configuration that can improve the design flexibility of hardware resources.

[0012] The inventors first attempted to investigate the accuracy of emotion estimation. As a result of their investigation, the inventors found that the accuracy of emotion estimation is determined by the degree of agreement between the estimated emotion, estimated using, for example, biosignals, movements, facial features, and expressions, and the cognitive emotion recognized by the person's brain. Furthermore, the inventors found that this degree of agreement improves the sense of satisfaction not only for the person whose emotion is being estimated, but also for users of the emotion estimation device, decision-makers who evaluate the emotion estimation device and decide whether to adopt or introduce it, and administrators who manage and operate the emotion estimation device, and other stakeholders involved with the emotion estimation device.

[0013] Furthermore, in further investigations into improving the accuracy of estimated emotions, the inventors found that the discrepancy between estimated emotions and actual emotions can be suppressed by devising a way to present information that influences the cognitive emotions of the individuals concerned, rather than by increasing the types or amount of data. This devising method of presenting information that influences the cognitive emotions of the individuals concerned is, for example, a method of presenting information based on the technical idea of ​​improving the accuracy of emotion estimation by combining multiple pieces of information in a way that takes advantage of human ambiguity, thereby increasing the sense of satisfaction when comparing the combined pieces of information.

[0014] Specifically, by devising a color standard that is ambiguous when viewed in isolation but clear when viewed in comparison, and by simultaneously displaying ambiguous colors with other ambiguous visual information, the degree of certainty can be increased compared to numerical representation, although some ambiguity remains, and differences can be clearly recognized when compared. Therefore, even if the type and amount of data used for emotion estimation are the same, the sense of satisfaction of the stakeholders increases, and the accuracy of the estimated emotion can be improved. Thus, the accuracy of emotion estimation can be improved while suppressing a decrease in the design flexibility of hardware resources in emotion estimation processing. Furthermore, if the level of emotion estimation accuracy does not change, the design flexibility of hardware resources in emotion estimation processing can be improved.

[0015] Furthermore, the inventors have realized that by using the estimated emotions of the subject in data processing, the data processing using estimated emotions is not limited to the uses of the configurations described in the aforementioned patent documents, but can be used for a variety of purposes. For example, the configuration of Patent Document 1 may be a device that stores estimated emotions in association with other data as well as information content data; the configuration of Patent Document 2 may display the emotional states of various live viewers on a device or the like; and the configuration of Patent Document 3 may play various content so that the user can relive their emotions. Also, for example, the configuration of Patent Document 4 may control the driving of moving objects other than amusement park rides based on the emotional state of passengers; the configuration of Patent Document 5 may perform management other than that of workers based on estimated emotions; and the configuration of Patent Document 6 may be used for evaluations other than tire performance.

[0016] Considering the above points, the inventors have come up with the following configuration.

[0017] An estimated emotion processing device according to one embodiment of the present invention includes an information content recording device that records estimated emotions and information content estimated based on various data obtained from an emotion estimation target whose emotions are estimated; a live event providing device that provides a live event so that the estimated emotions of live viewers are conveyed to online viewers; a content playback device that plays back the estimated emotions and content; a vehicle control device that controls the attraction vehicles according to the estimated emotions of passengers; an employee management device that manages employees using the estimated emotions; or an evaluation device that evaluates objects that affect the five senses of the emotion estimation target using the estimated emotions. The device acquires estimated emotion information indicating the estimated emotions and other information that is different from the estimated emotions but related to the estimated emotions, stores the acquired estimated emotion information and the other information in a storage unit in association with each other, and outputs output data including the estimated emotion information and the other information stored in the storage unit. The estimated emotion information acquired by the control unit includes estimated emotion color information, in which differences in emotion intensity and differences in emotion type are represented by different types of gradients, with differences in emotion intensity represented by gradients of color intensity that change in the radial direction of a virtual circle, and differences in emotion type represented by gradients of color tone that change in the circumferential direction of the virtual circle, while differences in emotion type are represented by numerical information, word information, and phrase information that directly indicate the estimated emotion, so as to be less ambiguous than when the estimated emotion is represented numerically. Other information acquired by the control unit and related to the estimated emotion includes information about factors related to the emotion estimation subject and that influence the data used to estimate the emotion of the emotion estimation subject, such as the place the emotion estimation subject was in, the environment they were in, what they saw, what they touched, what they ate, what they smelled, what they heard, who they were with or talking to, and who were around the emotion estimation subject, while including indirect visual information of influencing factors, which is visual information that indirectly indicates emotion, but does not include numerical information, word information, and phrase information that directly indicates emotion.The output data output from the control unit includes composite visual indirect emotion information configured in a data format in which both the estimated emotion color information and the indirect visual influence information are displayed simultaneously, such that the ambiguity of the color data of the estimated emotion color information in the color standard, in which differences in the intensity of the emotion and differences in the type of emotion are expressed by a gradient, is compensated for by the indirect visual influence information, or the indirect visual influence information is compensated for by the ambiguity of the color data of the estimated emotion color information.

[0018] The estimated emotion color information includes color data representing estimated emotions in a color standard where differences in emotion intensity and emotion type are represented by different types of gradients. The difference in emotion intensity is represented by a gradient of color shades that changes in the radial direction of a virtual circle, and the difference in emotion type is represented by a gradient of color tone that changes in the circumferential direction of the virtual circle. However, it does not include numerical information, word information, or phrase information that directly indicate the estimated emotion, so that the degree of ambiguity is less than when the estimated emotion is represented numerically. Therefore, when a person visually acquires and perceives the estimated emotion color information, the cognitive emotion can be made more ambiguous when viewed in isolation compared to when it is represented numerically, while the differences in cognitive emotions can be clarified when viewed in comparison. Moreover, by configuring the estimated emotion color information as described above, the above effects can be obtained with a small amount of data. It should be noted that cognitive emotions are more strongly influenced by directly indicating information than by ambiguous information. Therefore, it is important that the estimated emotion color information does not include numerical information, word information, or phrase information that directly indicates the estimated emotion.

[0019] The indirect visual information of influencing factors includes information regarding factors that are related to the emotion-estimation target person, such as the place where the person was, the environment the person was in, the things the person saw, the things the person touched, the things the person ate, the smells the person smelled, the sounds the person heard, the people the person was with or had conversations with, and the people around the emotion-estimation target person, and that affect the data used for estimating the emotion of the emotion-estimation target person. On the other hand, it does not include numerical information, word information, and clause information that directly indicate emotion, but is visual information that indirectly indicates emotion. Therefore, when a person visually obtains and recognizes the indirect visual information of influencing factors, the cognitive emotion can be made more ambiguous than when expressed numerically. Moreover, by configuring the indirect visual information of influencing factors as described above, the above-described operational effects can be obtained without increasing the amount of data. [[ID=!]]

[0020] The output data includes composite visual indirect emotion information configured in a data format in which both the estimated emotion color information and the indirect visual information of influencing factors are displayed simultaneously, such that the ambiguity of the color data of the estimated emotion color information in the color standard in which the difference in the intensity of the emotion and the difference in the type of emotion are expressed by gradation is compensated by the indirect visual information of influencing factors, or such that the indirect visual information of influencing factors is compensated by the ambiguity of the color data of the estimated emotion color information.

[0021] [[ID=!]] When a person obtains the output data, the person collects information necessary for judging emotion from the output data. Neither the estimated emotion color information nor the indirect visual information of influencing factors included in the output data contains information regarding direct emotion. However, the output data contains a plurality of pieces of information that are ambiguous but indirectly indicate emotion. Therefore, a person recognizes emotion using those pieces of information. Since emotion is a personal subjective matter and there are individual differences, it is easy for a deviation to occur between the estimated emotion and the personal recognition. In contrast, by showing a plurality of pieces of information that are ambiguous regarding emotion as described above, it is推测 that the human brain estimates emotion closer to the personal subjective. Therefore, by estimating emotion using a plurality of pieces of ambiguous information, the accuracy of emotion estimation can be improved.

[0022] Of course, such effects cannot be obtained simply by combining and displaying a plurality of visual informations. That is, it is because the accuracy improvement of emotion estimation using the interpretation of the human brain cannot be realized unless a plurality of visual informations that are ambiguous with respect to the emotion to be estimated are used, such as the output data including both the estimated emotion color information and the influencing factor indirect visual information.

[0023] Moreover, generally, when expressing the estimated emotion in color, efforts are made to improve the accuracy of the color expression so that the estimated emotion can be more accurately visualized by the color. That is, when expressing the estimated emotion in color, it is expressed so that the estimated emotion can be intuitively understood from the color. On the other hand, in the above-described configuration, by further combining the ambiguous visual information that indirectly indicates the emotion with the ambiguous information expressing the estimated emotion in color, the accuracy improvement of emotion estimation using the interpretation of the human brain is realized. Such a combination of ambiguous visual informations that indirectly indicate the emotion would not be attempted by those skilled in the art.

[0024] Also, in the above-described configuration, although ambiguity remains when the cognitive emotion when a person visually acquires and recognizes the composite visual indirect emotion information is expressed numerically, the certainty can be increased. As a result, the emotion estimation accuracy when the composite visual indirect emotion information is viewed alone can be improved. Also, due to the clarification effect during comparison in the above-described estimated emotion color information, the emotion estimation accuracy can also be improved when the composite visual indirect emotion information is viewed in comparison.

[0025] Moreover, since it is not necessary to significantly increase the data type or data amount to improve the estimation accuracy, a decrease in the design freedom of the hardware resources in the estimated emotion processing device can be suppressed. Also, when the emotion estimation accuracy is at the same level, the design freedom of the hardware resources in the estimated emotion processing device can be improved.

[0026] Therefore, in an estimated emotion processing device that performs data processing using estimated emotions, it is possible to realize a configuration that can improve the accuracy of emotion estimation while suppressing a decrease in the design flexibility of hardware resources, or, if the level of emotion estimation accuracy is about the same, it is possible to realize a configuration that can improve the design flexibility of hardware resources.

[0027] From another perspective, the estimated emotion processing device of the present invention preferably includes the following configuration: The estimated emotion color information includes a plurality of color data. The composite visual indirect emotion information is configured in a data format in which the plurality of color data included in the estimated emotion color information and the influencing factor indirect visual information are displayed simultaneously so that the plurality of color data included in the estimated emotion color information are compared.

[0028] This allows for the comparison of multiple color data included in the estimated emotion color information, clarifying the differences in perceived emotions when a person visually acquires and recognizes the multiple color data. Therefore, by comparing the multiple color information with the indirect visual information of influencing factors, the human brain can more easily recognize the information, thereby increasing the accuracy of the estimated emotion and enhancing the sense of satisfaction. This improves the accuracy of emotion estimation. Moreover, since this can be achieved simply by comparing the multiple color data with the indirect visual information of influencing factors, it can be implemented with a small amount of data. Therefore, it can suppress an increase in the load on hardware resources. Note that the indirect visual information of influencing factors may be singular or plural.

[0029] Therefore, in an estimated emotion processing device that performs data processing using estimated emotions, it is possible to realize a configuration that can improve the accuracy of emotion estimation while suppressing a decrease in the design flexibility of hardware resources, or, if the level of emotion estimation accuracy is about the same, it is possible to realize a configuration that can improve the design flexibility of hardware resources.

[0030] From another perspective, the emotion estimation processing device of the present invention preferably includes the following configuration: The plurality of color data includes at least one of the color data of different emotion estimation subjects, the color data of different locations, or the color data of a time series. The composite visual indirect emotion information is configured in a data format in which the influencing factor indirect visual information and the plurality of color data are displayed simultaneously so as to be able to recognize at least one of the differences between different emotion estimation subjects in the plurality of color data, the differences at different locations in the plurality of color data, or the time series changes in the plurality of color data.

[0031] By simultaneously displaying multiple color data and indirect visual information of influencing factors, it becomes possible to recognize at least one of the following: differences between different emotion estimation subjects in multiple color data, differences at different locations in multiple color data, or time-series changes in multiple color data. In this way, by comparing multiple color data, the accuracy of the estimated emotion can be increased, and the sense of satisfaction can be enhanced. This improves the accuracy of emotion estimation. Moreover, since this can be achieved simply by comparing the multiple color data and indirect visual information of influencing factors, it can be implemented with a small amount of data. Therefore, the increase in the load on hardware resources can be suppressed.

[0032] Therefore, in an estimated emotion processing device that performs data processing using estimated emotions, it is possible to realize a configuration that can improve the accuracy of emotion estimation while suppressing a decrease in the design flexibility of hardware resources, or, if the level of emotion estimation accuracy is about the same, it is possible to realize a configuration that can improve the design flexibility of hardware resources.

[0033] From another perspective, the estimated emotion processing device of the present invention preferably includes the following configuration: The estimated emotion information includes gradient color plane information, which includes at least a portion of the color standard, in which the difference in the intensity of the emotion and the type of emotion are represented by a gradient. The composite visual indirect emotion information is configured in a data format in which the gradient color plane information, the estimated emotion color information, and the influential factor indirect visual information are displayed simultaneously.

[0034] This makes it easy to visually identify the position of the displayed estimated emotion color information within the gradient color plane information. In this way, by displaying the estimated emotion color information, which is ambiguous when viewed alone compared to when it is expressed numerically, together with the gradient color plane information, the accuracy of the estimated emotion can be increased, leading to greater satisfaction. This improves the accuracy of emotion estimation.

[0035] Furthermore, by displaying indirect visual information of influencing factors in addition to the estimated emotion color information and the gradient color plane information, the accuracy of the estimated emotion displayed by the estimated emotion color information can be further enhanced, thereby increasing the sense of satisfaction.

[0036] Moreover, since this can be achieved simply by simultaneously displaying the gradient color information, the estimated emotional color information, and the indirect visual information of the influencing factors, it can be implemented while suppressing an increase in the amount of data. Therefore, an increase in the load on hardware resources can be suppressed.

[0037] Therefore, in an estimated emotion processing device that performs data processing using estimated emotions, it is possible to realize a configuration that can improve the accuracy of emotion estimation while suppressing a decrease in the design flexibility of hardware resources, or, if the level of emotion estimation accuracy is about the same, it is possible to realize a configuration that can improve the design flexibility of hardware resources.

[0038] From another perspective, the estimated emotion processing device of the present invention preferably includes the following configuration: The output data includes the estimated emotion color information displayed overlaid on the color reference.

[0039] This allows estimated emotional color information to be displayed overlaid on the color standard. Therefore, compared to when the estimated emotional color information is presented alone, it becomes easier for the human brain to recognize, thereby increasing the accuracy of the estimated emotion and enhancing the sense of satisfaction. This improves the accuracy of emotion estimation.

[0040] Furthermore, since this can be achieved simply by overlaying the estimated emotion color information onto the aforementioned color standard, an increase in data volume can be suppressed. Therefore, an increase in the load on hardware resources can be suppressed.

[0041] Therefore, in an estimated emotion processing device that performs data processing using estimated emotions, it is possible to realize a configuration that can improve the accuracy of emotion estimation while suppressing a decrease in the design flexibility of hardware resources, or, if the level of accuracy of emotion estimation is about the same, it is possible to realize a configuration that can improve the design flexibility of hardware resources.

[0042] From another perspective, the estimated emotion processing device of the present invention preferably includes the following configuration: The indirect visual information of the influencing factors includes at least one of video information or still image information that displays a plurality of pieces of information. The composite visual indirect emotion information is configured in a data format in which the estimated emotion color information and at least one of the video information or still image information are displayed simultaneously so that the estimated emotion color information and at least one of the video information or still image information are recognized as different information, or so that the estimated emotion color information and at least one of the video information or still image information are fused together.

[0043] This allows for the display of estimated emotion color information included in the estimated emotion color information, as well as at least one of the video information or still image information included in the indirect visual information of influencing factors. Therefore, because it is easier for the human brain to perceive, the cognitive emotion when a person visually acquires and perceives the composite visual indirect emotion information can be made more accurate, although some ambiguity remains compared to when it is expressed numerically. As a result, the accuracy of emotion estimation when viewing the composite visual indirect emotion information alone can be improved. Moreover, since the video information and the still image information each contain multiple pieces of information, displaying them simultaneously with the estimated emotion color information can further enhance the accuracy of the estimated emotion and increase the sense of satisfaction among those involved.

[0044] Furthermore, since it only displays estimated emotion color information and at least one of video information or still image information, it can suppress the increase in the amount of data processed by the estimated emotion processing unit. Therefore, it can suppress the increase in the load on hardware resources.

[0045] The aforementioned information includes, for example, information about the subject, and information about the weather, time, and location in the image.

[0046] Therefore, in an estimated emotion processing device that performs data processing using estimated emotions, it is possible to realize a configuration that can improve the accuracy of emotion estimation while suppressing a decrease in the design flexibility of hardware resources, or, if the level of emotion estimation accuracy is about the same, it is possible to realize a configuration that can improve the design flexibility of hardware resources.

[0047] From another perspective, the estimated emotion processing device of the present invention preferably includes the following configuration: The estimated emotion processing device has a display unit capable of displaying information and a communication unit capable of communicating information with an external device by wired or wireless means using electrical signals. The control unit outputs the composite visual indirect emotion information to at least one of the storage unit, the display unit, or the communication unit.

[0048] As a result, the control unit of the estimated emotion processing device can output the composite visual indirect emotion information to at least one of the storage unit, display unit, or communication unit so as to simultaneously display the estimated emotion color information and the indirect visual information of the influencing factors. Therefore, the composite visual indirect emotion information can be stored in the storage unit, displayed on the display unit, or transmitted to other devices via the communication unit. Furthermore, by making the composite visual indirect emotion information outputtable as described above, it becomes possible to share information with other users via a network or the like.

[0049] Furthermore, the control unit, storage unit, display unit, and communication unit may be provided in a mobile terminal. In this case, the control unit of the mobile terminal outputs the composite visual indirect emotion information to the storage unit or display unit of the mobile terminal.

[0050] Furthermore, the control unit and the communication unit may be provided in a mobile terminal, and the storage unit and the display unit may be provided in a server capable of communicating with the mobile terminal. In this case, the control unit of the mobile terminal outputs the composite visual indirect emotion information to the storage unit or display unit of the server. Alternatively, either the storage unit or the display unit may be provided in the mobile terminal. In this case, the control unit of the mobile terminal outputs the composite visual indirect emotion information to the other of the storage unit or display unit provided in the server.

[0051] Alternatively, the control unit and the communication unit may be located in a server capable of communicating with a mobile terminal, while the storage unit and the display unit are located in the mobile terminal. In this case, the control unit of the server outputs the composite visual indirect emotion information to the storage unit or display unit of the mobile terminal. Alternatively, either the storage unit or the display unit may be located in the server. In this case, the control unit of the server outputs the composite visual indirect emotion information to the other of the storage unit or display unit located in the mobile terminal.

[0052] Alternatively, a tablet device or similar device may be used instead of the aforementioned mobile device.

[0053] This allows for the implementation of an emotion estimation processing device using hardware resources with diverse configurations without affecting the accuracy of emotion estimation. Therefore, it improves the design flexibility of hardware resources.

[0054] Therefore, in an estimated emotion processing device that performs data processing using estimated emotions, it is possible to realize a configuration that improves the design flexibility of hardware resources without reducing the level of accuracy in emotion estimation.

[0055] From another perspective, the estimated emotion processing device of the present invention preferably includes the following configuration. The control unit acquires character emotion information which includes numerical information, word information, and phrase information that directly indicate emotion and is different from the composite visual indirect emotion information, and outputs composite emotion information as output data which is configured in a data format that simultaneously displays both the character emotion information and the composite visual indirect emotion information.

[0056] This allows textual emotion information, including numerical, word, and phrase information that directly express emotions, to be displayed simultaneously with composite visual indirect emotion information. Therefore, the textual emotion information and the composite visual indirect emotion information can complement each other, increasing the accuracy of estimated emotions and enhancing the sense of satisfaction. Consequently, the accuracy of emotion estimation can be improved. Moreover, since only the textual emotion information is added, the increase in hardware resource load can be suppressed.

[0057] Therefore, in an estimated emotion processing device that performs data processing using estimated emotions, it is possible to realize a configuration that can improve the accuracy of emotion estimation while suppressing a decrease in the design flexibility of hardware resources, or, if the level of emotion estimation accuracy is about the same, it is possible to realize a configuration that can improve the design flexibility of hardware resources.

[0058] A method for producing an estimated emotion processing device according to one embodiment of the present invention involves installing an estimated emotion processing program into a computer, which includes an information content recording device that records estimated emotions and information content estimated based on various data obtained from an emotion estimation target whose emotions are estimated; a live event providing device that provides a live event so that the estimated emotions of live viewers are conveyed to online viewers; a content playback device that plays back the estimated emotions and content; a vehicle control device that controls attraction vehicles according to the estimated emotions of passengers; an employee management device that manages employees using the estimated emotions; and an evaluation device that evaluates objects that affect the five senses of an emotion estimation target using the estimated emotions. The program involves installing an estimated emotion processing program into a computer that obtains estimated emotion information indicating the estimated emotions and other information that is different from the estimated emotions but related to the estimated emotions, stores the obtained estimated emotion information and the other information in a storage unit in association with the storage unit, and outputs output data including the estimated emotion information and the other information stored in the storage unit. The estimated emotion information includes estimated emotion color information, which is visual information that indirectly indicates emotion, and does not include numerical information, word information, or phrase information that directly indicates emotion, while including color data that represents the estimated emotion in a color standard where the difference in the intensity of emotion is represented by a gradient of color shades that changes in the radial direction of a virtual circle, and the difference in the type of emotion is represented by a gradient of color tones that changes in the circumferential direction of the virtual circle, so that the difference in the intensity of emotion and the difference in the type of emotion are represented by different types of gradients, while including estimated emotion color information that indirectly indicates emotion, so that the degree is more ambiguous than when the estimated emotion is represented numerically. Other information related to the estimated emotion includes information about factors that are related to the person whose emotion is estimated and that affect the data used to estimate the emotion of the person whose emotion is estimated, such as the place the person was, the environment they were in, what they saw, what they touched, what they ate, what they smelled, what they heard, the people they were with or talking to, and the people around the person whose emotion is estimated, while including indirect visual information that indirectly indicates emotion, which is visual information that indirectly indicates emotion, and does not include numerical information, word information, or phrase information that directly indicates emotion.The output data includes composite visual indirect emotion information configured in a data format in which both the estimated emotion color information and the indirect visual influence information are displayed simultaneously, such that the ambiguity of the color data of the estimated emotion color information in the color standard, where differences in the intensity of the emotion and differences in the type of emotion are represented by a gradient, is compensated for by the indirect visual influence information, or the indirect visual influence information is compensated for by the ambiguity of the color data of the estimated emotion color information. The process includes installing the estimated emotion processing program into the computer to obtain an estimated emotion processing device that outputs the output data including the estimated emotion information and the other information.

[0059] By installing a program for processing estimated emotions on a computer, an estimated emotion processing device can be obtained that outputs output data containing estimated emotion information and other information.

[0060] An estimated emotion processing program according to one embodiment of the present invention is an estimated emotion processing program that causes a computer to acquire estimated emotion information indicating the estimated emotion and other information that is different from the estimated emotion but related to the estimated emotion, store the acquired estimated emotion information and the other information in a storage unit in association with the acquired estimated emotion information and the other information stored in the storage unit, and output output data including the estimated emotion information and the other information stored in the storage unit. The program is used in applications such as an information content recording device that records estimated emotions and information content estimated based on various data obtained from an emotion estimation target whose emotions are estimated, a live event providing device that provides a live event so that the estimated emotions of live viewers are conveyed to online viewers, a content playback device that plays back the estimated emotions and content, a vehicle control device that controls the attraction vehicles according to the estimated emotions of passengers, an employee management device that manages employees using the estimated emotions, and an evaluation device that evaluates objects that affect the five senses of an emotion estimation target using the estimated emotions. The estimated emotion information includes estimated emotion color information, which is visual information that indirectly indicates emotion, and does not include numerical information, word information, or phrase information that directly indicates emotion, while including color data that represents the estimated emotion in a color standard where the difference in the intensity of emotion is represented by a gradient of color shades that changes in the radial direction of a virtual circle, and the difference in the type of emotion is represented by a gradient of color tones that changes in the circumferential direction of the virtual circle, so that the difference in the intensity of emotion and the difference in the type of emotion are represented by different types of gradients, while including estimated emotion color information that indirectly indicates emotion, so that the degree is more ambiguous than when the estimated emotion is represented numerically. Other information related to the estimated emotion includes information about factors that are related to the person whose emotion is estimated and that affect the data used to estimate the emotion of the person whose emotion is estimated, such as the place the person was, the environment they were in, what they saw, what they touched, what they ate, what they smelled, what they heard, the people they were with or talking to, and the people around the person whose emotion is estimated, while including indirect visual information that indirectly indicates emotion, which is visual information that indirectly indicates emotion, and does not include numerical information, word information, or phrase information that directly indicates emotion.The output data includes composite visual indirect emotion information configured in a data format in which both the estimated emotion color information and the indirect visual influence information are displayed simultaneously, such that the ambiguity of the color data of the estimated emotion color information in the color standard, where the difference in the intensity of the emotion and the difference in the type of emotion are represented by a gradient, is compensated for by the indirect visual influence information, or the indirect visual influence information is compensated for by the ambiguity of the color data of the estimated emotion color information.

[0061] The recording medium according to one embodiment of the present invention is a recording medium that records an estimated emotion processing program that causes a computer to acquire estimated emotion information indicating the estimated emotion and other information that is different from the estimated emotion but related to the estimated emotion, store the acquired estimated emotion information and the other information in a storage unit in association with the storage unit, and output output data including the estimated emotion information and the other information stored in the storage unit, and is readable by the computer. The recording medium is used in applications such as an information content recording device that records estimated emotion and information content estimated based on various data acquired from an emotion estimation target whose emotions are estimated, a live event providing device that provides a live event so that the estimated emotion of a live viewer is conveyed to an online viewer, a content playback device that plays back the estimated emotion and content, a vehicle control device that controls the attraction vehicle according to the estimated emotion of a passenger, an employee management device that manages employees using the estimated emotion, and an evaluation device that evaluates an object that affects the five senses of an emotion estimation target using the estimated emotion. The estimated emotion information includes estimated emotion color information, which is visual information that indirectly indicates emotion, and does not include numerical information, word information, or phrase information that directly indicates emotion, while including color data that represents the estimated emotion in a color standard where the difference in the intensity of emotion is represented by a gradient of color shades that changes in the radial direction of a virtual circle, and the difference in the type of emotion is represented by a gradient of color tones that changes in the circumferential direction of the virtual circle, so that the difference in the intensity of emotion and the difference in the type of emotion are represented by different types of gradients, while including estimated emotion color information that indirectly indicates emotion, so that the degree is more ambiguous than when the estimated emotion is represented numerically. Other information related to the estimated emotion includes information about factors that are related to the person whose emotion is estimated and that affect the data used to estimate the emotion of the person whose emotion is estimated, such as the place the person was, the environment they were in, what they saw, what they touched, what they ate, what they smelled, what they heard, the people they were with or talking to, and the people around the person whose emotion is estimated, while including indirect visual information that indirectly indicates emotion, which is visual information that indirectly indicates emotion, and does not include numerical information, word information, or phrase information that directly indicates emotion.The output data includes composite visual indirect emotion information configured in a data format in which both the estimated emotion color information and the indirect visual influence information are displayed simultaneously, such that the ambiguity of the color data of the estimated emotion color information in the color standard, where the difference in the intensity of the emotion and the difference in the type of emotion are represented by a gradient, is compensated for by the indirect visual influence information, or the indirect visual influence information is compensated for by the ambiguity of the color data of the estimated emotion color information.

[0062] The technical terms used herein are for the purpose of defining only specific embodiments and are not intended to limit the invention.

[0063] As used herein, "and / or" includes all combinations of one or more relatedly listed components.

[0064] In this specification, the use of “including,” “comprising,” or “having,” and variations thereof, identifies the presence of described features, processes, operations, elements, components, and / or equivalents thereof, but may include one or more of the steps, operations, elements, components, and / or groups thereof.

[0065] In this specification, “attached,” “connected,” “joined,” and / or their equivalents are used in a broad sense and include both “direct and indirect” attachments, connections, and combinations. Furthermore, “connected” and “joined” are not limited to physical or mechanical connections or combinations, but may include direct or indirect electrical connections or combinations.

[0066] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meanings as those generally understood by those skilled in the art to which this invention pertains.

[0067] Terms as defined in commonly used dictionaries should be interpreted to have meanings consistent with their meanings in the context of the relevant technology and this disclosure, and not to be interpreted in an ideal or overly formal sense unless expressly defined herein.

[0068] It is understood that several techniques and processes are disclosed in this description of the present invention. Each of these has its own individual benefit and may be used in conjunction with one or more, or possibly all, of the other disclosed techniques.

[0069] Therefore, for clarity, the description of this invention refrains from unnecessarily repeating all possible combinations of individual steps. However, this specification and the claims should be read with the understanding that all such combinations are within the scope of the invention.

[0070] This specification describes embodiments of the estimated emotion processing device, the production method thereof, the estimated emotion processing program, and the recording medium according to the present invention.

[0071] The following description includes numerous specific examples to provide a complete understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be carried out without these specific examples.

[0072] Therefore, the following disclosures should be considered illustrative examples of the present invention and are not intended to limit the present invention to any specific embodiments shown in the following drawings or description.

[0073] [Storage] In this specification, "storage unit" means a storage device or storage medium capable of storing data. The storage unit includes not only storage devices or storage mediums capable of temporary storage, but also recording media that cannot overwrite data. The storage unit includes, for example, memory inside the processor such as registers or cache memory, volatile memory, storage devices such as hard disks and solid-state drives, and recording media such as USBs.

[0074] [Emotional intensity] In this specification, emotional intensity is a representation of the intensity of an emotion in stages. The emotional intensity may be expressed numerically, or by letters, words, or phrases. The emotional intensity refers, for example, to the magnitude of emotional arousal and emotional valence, respectively, on a two-dimensional plane represented using two evaluation indices: emotional arousal and emotional valence.

[0075] [Types of emotions] In this specification, the types of emotions include, for example, joy, anger, sadness, pleasure, excitement, relief, boredom, and tension. The types of emotions may be, for example, those represented in Russell's Circle of Emotions model, which is represented on a two-dimensional plane using two evaluation indices: emotional arousal and emotional valence, or they may be those represented by other emotion models.

[0076] [output] In this specification, "output" includes not only outputting data to an external device, but also displaying data on a display unit or the like.

[0077] [Radial direction of the virtual circle] In this specification, the radial direction of a virtual circle means, for example, the radial direction of a virtual circle centered at the intersection of two evaluation indices, emotional arousal and emotional valence, in Russell's Circle of Emotion model. The radial direction of the virtual circle may also mean the radial direction of a virtual circle centered at the intersection of two evaluation indices on a two-dimensional plane represented using two evaluation indices related to emotion.

[0078] [Circumferential direction of a virtual circle] In this specification, the circumferential direction of a virtual circle means, for example, the circumferential direction of a virtual circle centered at the intersection of two evaluation indices, emotional arousal and emotional valence, in Russell's Circle of Emotion model. The circumferential direction of the virtual circle may also mean the circumferential direction of a virtual circle centered at the intersection of two evaluation indices on a two-dimensional plane represented using two evaluation indices related to emotion.

[0079] [Other information related to estimated emotions] In this specification, other information related to estimated emotion includes, for example, information about the location where the emotion was estimated, environmental information (sound, temperature, humidity, time), things the person being estimated saw, touched, ate, smelled, heard, people the person being estimated was with, people they talked to, and people in their vicinity. Other information related to estimated emotion is, for example, information that is directly related to the person being estimated when the data used for estimation is obtained from the person being estimated. Other information related to estimated emotion is, for example, information about factors obtained from the person being estimated that influence the data used for estimation. Other information related to estimated emotion is, for example, information separate from the information about the estimated emotion.

[0080] [Color gradient] In this specification, a gradient of color intensity means a continuous and gradual change in color intensity, where the color intensity changes smoothly without abrupt changes. For example, if the boundary between the color intensity can be visually distinguished, it does not constitute a gradient of color intensity in this invention.

[0081] [Color gradient] In this specification, a color gradient means a continuous and gradual change in color tone, where the color tone changes smoothly without abrupt changes. For example, if the boundary between color tones can be visually distinguished, it does not constitute a color gradient in this invention.

[0082] [Estimated emotional information] In this specification, estimated emotion refers to the emotion of a subject whose emotions are estimated, based on their biometric information, movements, facial features, and expressions. Estimated emotion information refers to information relating to the estimated emotion. That is, estimated emotion information refers to information relating to the estimated emotion of the subject. Emotion refers to a state in which the subject is feeling joy, anger, sadness, etc. The emotions include states in which the subject is feeling happy, relaxed, angry, and sad. The estimated emotion information includes information relating to the probability of the estimated emotion of the subject. The estimated emotion information may also include information relating to the degree of the estimated emotion of the subject. One piece of estimated emotion information may contain multiple emotions. Estimated emotion information is created, for example, by a control unit located in a mobile terminal held by the subject or on a server.

[0083] [Color standard] In this specification, a color standard is a standard used when indicating estimated emotions by color, where the relationship between emotion and color is expressed by a gradient of color intensity and a gradient of hue. More specifically, the color standard is such that differences in the intensity of emotion and differences in the type of emotion are expressed by different types of gradients, where the difference in the intensity of emotion is expressed by a gradient of color intensity that changes in the radial direction of a virtual circle, and the difference in the type of emotion is expressed by a gradient of hue that changes in the circumferential direction of the virtual circle.

[0084] Furthermore, the color standards may include not only gradients of color intensity and gradients of hue, but also overlaid on these gradients the representation of boundaries indicating the boundaries of emotions, areas representing each emotion, or textual information related to emotions.

[0085] [Estimated emotional color information] In this specification, estimated emotion color information is information that includes color data representing an estimated emotion in a color standard, but does not include numerical information, word information, or phrase information that directly indicates the estimated emotion, to the extent that the degree of the estimated emotion is more ambiguous than when the estimated emotion is expressed numerically.

[0086] [Numerical information, word information, and phrase information that directly indicate estimated emotions] In this specification, numerical information that directly indicates an estimated emotion refers to information in which the estimated emotion is expressed numerically. Word information that directly indicates an estimated emotion refers to information in which the estimated emotion is expressed as a word. Sentence information that directly indicates an estimated emotion refers to information in which the estimated emotion is expressed as a sentence. Thus, directly indicating an estimated emotion means directly indicating the estimated emotion itself, and does not include indicating the estimated emotion through indirect or ambiguous expressions.

[0087] For example, the legend included in the color criteria, which shows differences in the direction of different types of emotions, consists of word and phrase information, but it does not directly indicate the estimated emotion.

[0088] Numerical, word, and phrase information that directly indicates estimated emotion means, for example, information that expresses emotion using coordinate data, information that expresses emotion using numbers or formulas, information about words or phrases that express emotion, and information about words or phrases that indicate the degree of emotion. Information about words or phrases that represent color, information about words or phrases that represent hue, and information about words or phrases that represent gradient are information that indirectly indicates estimated emotion and are not included in numerical, word, and phrase information that directly indicates estimated emotion.

[0089] In addition to the numerical, word, and phrase information mentioned above, information that directly indicates estimated emotions may also include information that represents emotions graphically, information that represents emotions audibly, and information that represents emotions visually.

[0090] [Factors that influence the data used for sentiment estimation] In this specification, factors that influence the data used for emotion estimation mean factors that directly or indirectly influence the data (e.g., biometric information) used for emotion estimation of the person being estimated. These factors include all items that influence emotion estimation of the person being estimated, such as the place the person was in, the environment they were in, what they saw, what they touched, what they ate, what they smelled, what they heard, who they were with or talking to, and who was around the person being estimated. The data used for emotion estimation of the person being estimated is not limited to biometric information, but may also include the person's movements, facial features, and expressions.

[0091] [Indirect visual information on influencing factors] In this specification, indirect visual information of influencing factors is visual information that indirectly indicates emotion, while including information about factors related to the person whose emotion is estimated and influencing the data used to estimate the emotion of the said person, but without including numerical information, word information, and phrase information that directly indicate emotion. Indirect visual information of influencing factors includes, for example, information related to the actions of the person whose emotion is estimated, including information about places visited (travel destinations, shops, etc.), address, workplace (office, etc.), tourist spots visited (castles, the sea, etc.), vehicles used (trains, airplanes, etc.), food eaten (ramen, eel, etc.), drinks consumed (juice, tea, etc.), smells smelled, sounds heard, temperature felt, atmosphere felt on the skin, vibrations felt, people with whom one was, people spoken to, scenery seen (sea, mountains, etc.), surrounding environment, objects touched, events attended (e.g., live concert, etc.). Furthermore, indirect visual information of influencing factors may include information about adjectives such as beautiful, bright, and light, as long as it does not include information that directly indicates emotion.

[0092] Furthermore, information that directly expresses emotions such as joy, happiness, comfort, and sadness is not included in indirect visual information that influences perception.

[0093] [Video or still image information displaying multiple pieces of information] In this specification, video information or still image information displaying multiple pieces of information means video information or still image information that contains and displays multiple pieces of information. Video information or still image information displaying multiple pieces of information is information that is only perceived when the viewer's brain interprets multiple pieces of information. Video information or still image information displaying multiple pieces of information is more ambiguous than when it is expressed numerically. Therefore, individual differences are likely to occur in the degree of perception.

[0094] Video or still image information displaying multiple pieces of information includes cases where the pieces of information are displayed side by side, where at least a portion of the pieces of information are displayed overlapping, and where at least a portion of the pieces of information are displayed enclosed in a frame.

[0095] [Color data] In this specification, color data refers to a part of a color standard in which differences in the intensity of emotion are represented by a gradient of color shades that change in the radial direction of a virtual circle, and differences in the type of emotion are represented by a gradient of color tones that change in the circumferential direction of the virtual circle, and also refers to information about the color when estimated emotions are represented by different types of gradients in which differences in the intensity of emotion and differences in the type of emotion are expressed.

[0096] [Estimated emotional color information is fused with at least one of video or still image information.] In this specification, "merging estimated emotion color information with at least one of video information or still image information" means combining estimated emotion color information with at least one of video information or still image information to create a single piece of information. In such merging, for example, estimated emotion color information may be incorporated into a part of at least one of the video information or still image information to generate new information. Alternatively, in such merging, for example, information that displays at least one of the video information or still image information may be generated based on the estimated emotion color information.

[0097] [Text reading information] In this specification, textual sentiment information includes numerical information, word information, and phrase information that directly indicate emotions, and is different from composite visual indirect sentiment information. Textual sentiment information includes textual information such as comments, messages, and text. The textual sentiment information may also be textual information displayed on social networking services, web pages, emails, books, etc. [Effects of the Invention]

[0098] According to one embodiment of the present invention, in an estimated emotion processing device that processes data using estimated emotions, it is possible to realize a configuration that can improve the accuracy of emotion estimation while suppressing a decrease in the design flexibility of hardware resources, or a configuration that can improve the design flexibility of hardware resources when the level of emotion estimation accuracy is about the same. [Brief explanation of the drawing]

[0099] [Figure 1] Figure 1 is a diagram showing the schematic configuration of an estimated emotion processing device according to Embodiment 1 of the present invention. [Figure 2] Figure 2 shows an example of a color standard. [Figure 3] Figure 3 is a schematic diagram showing the output data. [Figure 4] Figure 4 shows an example of displaying estimated emotion color information and indirect visual information of influencing factors, output by the estimated emotion processing device according to Embodiment 2, on a display unit. [Figure 5] Figure 5 shows an example of simultaneously displaying indirect visual information of influencing factors and multiple color data on the display unit. [Figure 6] Figure 6 shows a schematic configuration of the estimated emotion processing device according to Embodiment 3. [Figure 7] Figure 7 shows an example of displaying gradient color information, estimated emotion color information, and indirect visual information of influencing factors, which are output by the estimated emotion processing device according to Embodiment 3, on a display unit. [Figure 8] Figure 8 shows a schematic configuration of the estimated emotion processing device according to Embodiment 4. [Figure 9] Figure 9 shows a schematic configuration of the estimated emotion processing device according to Embodiment 5. [Figure 10] Figure 10 shows an example of displaying textual emotion information and composite visual indirect emotion information output by the estimated emotion processing device according to Embodiment 5 on a display unit. [Modes for carrying out the invention]

[0100] The following describes each embodiment with reference to the drawings. Note that the dimensions of the components in each drawing do not faithfully represent the actual dimensions of the components or the dimensional ratios of each component.

[0101] [Embodiment 1] Figure 1 shows a schematic configuration of the estimated emotion processing device 1 according to Embodiment 1 of the present invention. The estimated emotion processing device 1 is a device that processes data using estimated emotion information that indicates the estimated emotion of a person whose emotion is to be estimated (hereinafter also referred to as estimated emotion). The estimated emotion processing device 1 may be realized by installing a program having the function of an estimated emotion processing device on, for example, a mobile terminal, tablet terminal, server, computer, etc. (hereinafter collectively referred to as a computer). Alternatively, the estimated emotion processing device 1 may be a dedicated device. In other words, the estimated emotion processing device 1 can be any device that can perform data processing using estimated emotion information that indicates estimated emotion. A computer is a device that has arithmetic processing functions such as a CPU (Central Processing Unit) and is capable of performing arithmetic processing.

[0102] The estimated emotion processing device 1 is a device that processes data using estimated emotions, such as an information content recording device that records estimated emotions and information content estimated based on various data obtained from an emotion estimation target whose emotions are estimated; a live event providing device that provides a live event so that the estimated emotions of live viewers are conveyed to online viewers; a content playback device that plays the estimated emotions and content; a vehicle control device that controls the attraction vehicles according to the estimated emotions of passengers; an employee management device that manages employees using the estimated emotions; or an evaluation device that evaluates objects that affect the five senses of the emotion estimation target using the estimated emotions.

[0103] Specifically, the estimated emotion processing device 1 acquires estimated emotion information D1 relating to the estimated emotion of the person whose emotion is to be estimated, and other information D2 that is different from the estimated emotion but related to the estimated emotion, and outputs the acquired estimated emotion information D1 and other information D2 in association. The estimated emotion information D1 is acquired from a device that estimates the emotion of the person whose emotion is to be estimated. The other information D2 is acquired in association with the estimated emotion information D1 from, for example, an imaging device, a recording device such as a driving recorder, a GPS device, an external storage device, an information management device, etc.

[0104] The estimated emotion is, for example, an emotion estimated based on the biological information of the person being estimated. The estimated emotion includes the type of emotion (joy, anger, sadness, etc.) and the intensity (degree) of the emotion. The estimated emotion may also be an emotion estimated based on information other than biological information. Biological information refers to various physiological and anatomical information emitted by the subject or participant. Biological information means all life activities that occur in the body of the subject or participant. Biological information includes, for example, information on the subject's heart rate, heart sounds, heart waveform, heart cycle, changes in heart rate, blood pressure, pulse wave, triaxial acceleration, body surface temperature, electroencephalogram, respiratory rate, pupil state, electromyography, blood components, exhalation, exhalation volume, exhalation components, etc. The biological information is measured by wearable sensors worn by the subject, such as heart rate sensors, acceleration sensors, and temperature sensors.

[0105] The estimated emotion information D1 is information indicating the estimated emotion, and for example, includes estimated emotion color information d1 which includes color data da that represents the estimated emotion in the color standard described later. The color data da is data that represents the estimated emotion in a color standard in which the difference in the intensity of the emotion is expressed by a gradient of color shades that changes in the radial direction of the virtual circle P described later, and the difference in the type of emotion is expressed by a gradient of color tones that changes in the circumferential direction of the virtual circle P, so that the difference in the intensity of the emotion and the difference in the type of emotion are expressed by different types of gradients. The color data da may be color data that is displayed alone, or it may be data that displays the position of the color in the color standard. The estimated emotion color information d1 does not include numerical information, word information, and phrase information that directly indicate the estimated emotion so that the degree is more ambiguous than when the estimated emotion is expressed numerically.

[0106] Figure 2 shows an example of the color standard. As shown in Figure 2, the color standard, for example in Russell's circular emotion model, represents the intensity of emotion by a gradient of color shades and the type of emotion by a gradient of hue. The color standard may also represent the intensity of emotion by a gradient of color shades and the type of emotion by a gradient of hue in other emotion models.

[0107] Specifically, in the aforementioned color standard, the intensity of emotion increases as you move radially outward from the intersection of the two evaluation indices, emotional arousal and emotional valence, on the virtual circle P, and therefore the color becomes darker. Also, in the aforementioned color standard, the type of emotion changes along the circumferential direction of the virtual circle P, and therefore the hue changes. For example, in the aforementioned color standard shown in Figure 2, emotion is blue, relief is green, boredom is yellow, and tension is red. In the aforementioned color standard, as each type of emotion changes along the virtual circle P, the color changes continuously and smoothly due to a gradient of hue.

[0108] Based on the above, in the aforementioned color standard, the intersection of the two evaluation indices, emotional arousal and emotional valence, is white. As one moves radially outward from the intersection of the virtual circle P, one of the colors—blue, green, yellow, or red—becomes darker, and the color changes continuously and smoothly in a gradient of tones around the virtual circle P. Therefore, in the aforementioned color standard, even if the intensity and type of emotion are different, the boundaries are not clear and are ambiguous. In addition, the color standard may include overlays of gradients of color intensity and gradients of tones, as well as boundary lines indicating the boundaries of emotions, areas indicating each emotion, or textual information related to emotions.

[0109] In the color standard shown in Figure 2, the color data da corresponding to the estimated emotion is the color data representing the estimated emotion. In Figure 2, an example of the position of the color data da in the color standard is indicated by the symbol X. The estimated emotion information D1 may contain information about the position of the color data da representing the estimated emotion in the color standard shown in Figure 2, or it may contain information about the intensity and hue of the color data da representing the estimated emotion. In the example shown in Figure 2, the position of the color data da in the color standard is indicated by a ring, but it may also be indicated by markers of other shapes such as squares or circles.

[0110] Other information D2 that is different from and related to the estimated emotion includes indirect visual influencing information d2, which includes information about factors related to the person whose emotion was estimated, such as the place they were in, the environment they were in, what they saw, what they touched, what they ate, what they smelled, what they heard, who they were with or talking to, and who were around them. Indirect visual influencing information d2 does not include numerical information, word information, and phrase information that directly indicate the emotion of the person whose emotion was estimated, but is visual information that indirectly indicates the emotion of the person whose emotion was estimated. The visual information is information obtained by sight. Indirect visual influencing information may include information about adjectives, such as beautiful, bright, and light, as long as it does not include information that directly indicates an emotion.

[0111] The estimated emotion processing device 1 comprises a storage unit 11 and a control unit 12.

[0112] The memory unit 11 stores the acquired estimated emotion information D1 in association with other information D2. The memory unit 11 includes not only storage devices or storage media capable of temporarily storing data, but also recording media that cannot be overwritten, as long as they are capable of storing data.

[0113] The control unit 12 outputs the estimated emotion color information d1 contained in the estimated emotion information D1 stored in the memory unit 11, and the indirect visual information d2 of the influencing factors contained in other information D2 stored in the memory unit 11, to be displayed simultaneously. Specifically, the control unit 12 generates and outputs output data D0 that includes the estimated emotion color information d1 and the indirect visual information d2 of the influencing factors, and displays both information simultaneously.

[0114] Figure 3 schematically shows the output data D0. The output data D0 has a data format that simultaneously displays estimated emotion color information d1 and indirect visual information of influencing factors d2, as shown in Figure 3.

[0115] The output data D0 includes composite visual indirect emotion information d0, which is configured in a data format in which both the estimated emotion color information d1 and the indirect visual information d2 are displayed simultaneously, such that the ambiguity of the color data of the estimated emotion color information d1 in the color standard, in which the differences in the intensity of the emotion and the differences in the types of the emotion are represented by a gradient, is compensated for by the indirect visual information d2 of the influencing factors, or that the indirect visual information d2 of the influencing factors is compensated for by the ambiguity of the estimated emotion color information d1.

[0116] By outputting output data D0 having the configuration described above, the control unit 12 outputs the ambiguity of the color data da of the estimated emotion color information d1 in the color standard, where the difference in the intensity of the emotion and the difference in the type of emotion are expressed by a gradient, and the indirect visual information d2 of the influencing factors mutually complement each other, thereby increasing the accuracy of the estimated emotion.

[0117] The estimated emotion color information d1 is included in a color standard in which the difference in emotion intensity is represented by a gradient of color shades that changes in the radial direction of a virtual circle P, and the difference in emotion type is represented by a gradient of color tone that changes in the circumferential direction of the virtual circle P. Furthermore, the difference in emotion intensity and the difference in emotion type are represented by different types of gradients, which do not include numerical information, word information, or phrase information that directly indicate the estimated emotion, so as to be less ambiguous than when expressed numerically. As a result, when a person visually acquires and perceives the estimated emotion color information d1, the cognitive emotion can be made more ambiguous when viewed in isolation compared to when it is expressed numerically, while the differences in cognitive emotions can be made clearer when viewed in comparison. Moreover, by configuring the estimated emotion color information d1 as described above, the above effects can be obtained with a small amount of data.

[0118] The indirect visual information d2 of the influencing factors is visual information that indirectly indicates emotion, and is included in the information of factors that are related to the person whose emotion is estimated and that influence the data used for emotion estimation, such as the place the person whose emotion is estimated was, the environment they were in, what they saw, what they touched, what they ate, what they smelled, what they heard, the people they were with or talking to, and the people around the person whose emotion is estimated. It does not include numerical information, word information, or phrase information that directly indicate emotion. Therefore, the cognitive emotion when a person visually acquires and perceives the influencing factor indirect visual information d2 can be made more ambiguous compared to when it is expressed numerically. Moreover, by configuring the influencing factor indirect visual information d2 as described above, the above effects can be obtained without increasing the amount of data.

[0119] The output data D0 includes composite visual indirect emotion information d0, which is configured in a data format in which both the estimated emotion color information d1 and the indirect visual information d2 are displayed simultaneously, such that the ambiguity of the color data da of the estimated emotion color information d1 included in the color standard, where both the difference in the intensity of the emotion and the difference in the type of emotion are expressed by a gradient, is compensated for by the indirect visual information d2 of the influencing factor, or the indirect visual information d2 of the influencing factor is compensated for by the ambiguity of the color data da of the estimated emotion color information d1. Therefore, when a person visually acquires and perceives the composite visual indirect emotion information d0, the perceived emotion, while still having some ambiguity compared to when it is expressed numerically, can be made more certain. As a result, the accuracy of emotion estimation when viewing the composite visual indirect emotion information d0 can be improved. In addition, since there is no need to significantly increase the type or amount of data, the decrease in the design flexibility of the hardware resources in the estimated emotion processing device 1 can be suppressed. Furthermore, when the emotion estimation accuracy is at the same level, the design flexibility of the hardware resources in the estimated emotion processing device 1 can be improved.

[0120] Therefore, in the estimated emotion processing device 1 that performs data processing using estimated emotions, it is possible to realize a configuration that can improve the accuracy of emotion estimation while suppressing a decrease in the design flexibility of hardware resources, or, if the level of emotion estimation accuracy is about the same, it is possible to realize a configuration that can improve the design flexibility of hardware resources.

[0121] The estimated emotion processing device 1 may be implemented, for example, by installing an estimated emotion processing program having the functions of an estimated emotion processing device on a computer. In this case, the estimated emotion processing program causes the computer to function as the control unit 12 of the estimated emotion processing device 1. Alternatively, a method for producing the estimated emotion processing device 1 can be realized by installing the estimated emotion processing program on the computer and obtaining an estimated emotion processing device 1 that outputs output data D0 including estimated emotion information D1 and other information D2.

[0122] The estimated emotion processing program according to this embodiment is an estimated emotion processing program that causes a computer to acquire estimated emotion information indicating the estimated emotion and other information that is different from the estimated emotion but related to the estimated emotion, store the acquired estimated emotion information and the other information in a storage unit in association with the acquired estimated emotion information and the other information stored in the storage unit, and output output data including the estimated emotion information and the other information stored in the storage unit. Examples include an information content recording device that records estimated emotions and information content estimated based on various data obtained from an emotion estimation target whose emotions are estimated, a live event providing device that provides a live event so that the estimated emotions of live viewers are conveyed to online viewers, a content playback device that plays back the estimated emotions and content, a vehicle control device that controls the attraction vehicles according to the estimated emotions of passengers, an employee management device that manages employees using the estimated emotions, and an evaluation device that evaluates objects that affect the five senses of the emotion estimation target using the estimated emotions. The estimated emotion information includes estimated emotion color information, which is visual information that indirectly indicates emotion, and does not include numerical information, word information, or phrase information that directly indicates emotion, while including color data that represents the estimated emotion in a color standard where the difference in the intensity of emotion is represented by a gradient of color shades that changes in the radial direction of a virtual circle, and the difference in the type of emotion is represented by a gradient of color tones that changes in the circumferential direction of the virtual circle, so that the difference in the intensity of emotion and the difference in the type of emotion are represented by different types of gradients, while including estimated emotion color information that indirectly indicates emotion, so that the degree is more ambiguous than when the estimated emotion is represented numerically. Other information related to the estimated emotion includes information about factors that are related to the person whose emotion is estimated and that affect the data used to estimate the emotion of the person whose emotion is estimated, such as the place the person was, the environment they were in, what they saw, what they touched, what they ate, what they smelled, what they heard, the people they were with or talking to, and the people around the person whose emotion is estimated, while including indirect visual information that indirectly indicates emotion, which is visual information that indirectly indicates emotion, and does not include numerical information, word information, or phrase information that directly indicates emotion.The output data includes composite visual indirect emotion information configured in a data format in which both the estimated emotion color information and the indirect visual influence information are displayed simultaneously, such that the ambiguity of the color data of the estimated emotion color information in the color standard, where the difference in the intensity of the emotion and the difference in the type of emotion are represented by a gradient, is compensated for by the indirect visual influence information, or the indirect visual influence information is compensated for by the ambiguity of the color data of the estimated emotion color information.

[0123] A method for producing an estimated emotion processing device according to one embodiment of the present invention involves installing an estimated emotion processing program into a computer, which includes an information content recording device that records estimated emotions and information content estimated based on various data obtained from an emotion estimation target whose emotions are estimated; a live event providing device that provides a live event so that the estimated emotions of live viewers are conveyed to online viewers; a content playback device that plays back the estimated emotions and content; a vehicle control device that controls attraction vehicles according to the estimated emotions of passengers; an employee management device that manages employees using the estimated emotions; and an evaluation device that evaluates objects that affect the five senses of an emotion estimation target using the estimated emotions. The program involves installing an estimated emotion processing program into a computer that obtains estimated emotion information indicating the estimated emotions and other information that is different from the estimated emotions but related to the estimated emotions, stores the obtained estimated emotion information and the other information in a storage unit in association with the storage unit, and outputs output data including the estimated emotion information and the other information stored in the storage unit. The estimated emotion information includes estimated emotion color information, which is visual information that indirectly indicates emotion, and does not include numerical information, word information, or phrase information that directly indicates emotion, while including color data that represents the estimated emotion in a color standard where the difference in the intensity of emotion is represented by a gradient of color shades that changes in the radial direction of a virtual circle, and the difference in the type of emotion is represented by a gradient of color tones that changes in the circumferential direction of the virtual circle, so that the difference in the intensity of emotion and the difference in the type of emotion are represented by different types of gradients, while including estimated emotion color information that indirectly indicates emotion, so that the degree is more ambiguous than when the estimated emotion is represented numerically. Other information related to the estimated emotion includes information about factors that are related to the person whose emotion is estimated and that affect the data used to estimate the emotion of the person whose emotion is estimated, such as the place the person was, the environment they were in, what they saw, what they touched, what they ate, what they smelled, what they heard, the people they were with or talking to, and the people around the person whose emotion is estimated, while including indirect visual information that indirectly indicates emotion, which is visual information that indirectly indicates emotion, and does not include numerical information, word information, or phrase information that directly indicates emotion.The output data includes composite visual indirect emotion information configured in a data format in which both the estimated emotion color information and the indirect visual influence information are displayed simultaneously, such that the ambiguity of the color data of the estimated emotion color information in the color standard, where differences in the intensity of the emotion and differences in the type of emotion are represented by a gradient, is compensated for by the indirect visual influence information, or the indirect visual influence information is compensated for by the ambiguity of the color data of the estimated emotion color information. The process includes installing the estimated emotion processing program into the computer to obtain an estimated emotion processing device that outputs the output data including the estimated emotion information and the other information.

[0124] The program for processing estimated emotions may be recorded on a recording medium that the computer can read. The recording medium includes magnetic recording media, optical recording media, magneto-optical recording media, or semiconductor recording media. The recording medium may be writable or rewritable, or it may be a read-only recording medium.

[0125] The recording medium according to this embodiment is a recording medium that records an estimated emotion processing program that causes a computer to acquire estimated emotion information indicating the estimated emotion and other information that is different from the estimated emotion but related to the estimated emotion, store the acquired estimated emotion information and the other information in a storage unit in association with the storage unit, and output output data including the estimated emotion information and the other information stored in the storage unit, and is readable by the computer. Examples include an information content recording device that records estimated emotion and information content estimated based on various data acquired from an emotion estimation target whose emotions are estimated, a live event providing device that provides a live event so that the estimated emotion of a live viewer is conveyed to an online viewer, a content playback device that plays back the estimated emotion and content, a vehicle control device that controls the attraction vehicle according to the estimated emotion of a passenger, an employee management device that manages employees using the estimated emotion, and an evaluation device that evaluates an object that affects the five senses of the emotion estimation target using the estimated emotion. The estimated emotion information includes estimated emotion color information, which is visual information that indirectly indicates emotion, and does not include numerical information, word information, or phrase information that directly indicates emotion, while including color data that represents the estimated emotion in a color standard where the difference in the intensity of emotion is represented by a gradient of color shades that changes in the radial direction of a virtual circle, and the difference in the type of emotion is represented by a gradient of color tones that changes in the circumferential direction of the virtual circle, so that the difference in the intensity of emotion and the difference in the type of emotion are represented by different types of gradients, while including estimated emotion color information that indirectly indicates emotion, so that the degree is more ambiguous than when the estimated emotion is represented numerically. Other information related to the estimated emotion includes information about factors that are related to the person whose emotion is estimated and that affect the data used to estimate the emotion of the person whose emotion is estimated, such as the place the person was, the environment they were in, what they saw, what they touched, what they ate, what they smelled, what they heard, the people they were with or talking to, and the people around the person whose emotion is estimated, while including indirect visual information that indirectly indicates emotion, which is visual information that indirectly indicates emotion, and does not include numerical information, word information, or phrase information that directly indicates emotion.The output data includes composite visual indirect emotion information configured in a data format in which both the estimated emotion color information and the indirect visual influence information are displayed simultaneously, such that the ambiguity of the color data of the estimated emotion color information in the color standard, where the difference in the intensity of the emotion and the difference in the type of emotion are represented by a gradient, is compensated for by the indirect visual influence information, or the indirect visual influence information is compensated for by the ambiguity of the color data of the estimated emotion color information.

[0126] [Embodiment 2] Figure 4 shows an example of displaying estimated emotion color information d1 and indirect visual information d2 of influencing factors, output by the estimated emotion processing device according to Embodiment 2, on the display unit 20. This embodiment differs from Embodiment 1 in that the estimated emotion color information d1 and indirect visual information d2 of influencing factors are displayed simultaneously. In the following, components similar to those in Embodiment 1 are denoted by the same reference numerals, and only the parts that differ from Embodiment 1 will be described.

[0127] In this embodiment, the estimated emotion color information d1 includes estimated emotion color plane information that displays a color included in the color standard. The influential factor indirect visual information d2 includes at least one of video information or still image information that displays multiple pieces of information. As a result, the estimated emotion processing device can display the estimated emotion color information d1, which includes the estimated emotion color plane information, and the influential factor indirect visual information d2, which includes at least one of the video information or still image information, on the display unit 20, as shown in Figure 4.

[0128] Furthermore, in this embodiment, the composite visual indirect emotion information d0 is configured in a data format in which the estimated emotion color information d1 and at least one of the video information or the still image information are displayed simultaneously, so that the estimated emotion color information d1 and at least one of the video information or the still image information are recognized as different information, or so that the estimated emotion color information d1 and at least one of the video information or the still image information are fused together. As a result, the estimated emotion processing device can simultaneously display the estimated emotion color information d1, which includes the estimated emotion color plane information, and the influential factor indirect visual information d2, which includes at least one of the video information or the still image information, on the display unit 20.

[0129] The composite visual indirect emotion information d0 has a data format in which estimated emotion color information d1 and influential factor indirect visual information d2 are displayed side by side in one direction on the display unit 20. The estimated emotion color information d1 and influential factor indirect visual information d2 may be displayed side by side in the vertical or horizontal direction on the display unit 20, or side by side in the diagonal direction, or partially overlap. At least one of the estimated emotion color information d1 and influential factor indirect visual information d2 may be enclosed by a frame. The estimated emotion color information d1 and influential factor indirect visual information d2 may be fused, for example, so that the color of a part of a photograph is changed to match the color of the estimated emotion color information d1. The composite visual indirect emotion information d0 may have a data format in which the display size of the estimated emotion color information d1 and the display size of the influential factor indirect visual information d2 on the display unit 20 are the same, or it may have a data format in which one of the information has a larger display size.

[0130] The display unit 20 includes, for example, a display screen of a mobile device, a display, or a display device such as a projector. The display unit 20 is capable of displaying an image on a screen or other surface.

[0131] The estimated emotion color surface information includes information to display color data da, which is included in the color standard and represents the estimated emotion, on a surface. Specifically, the estimated emotion color surface information includes information to display color data da on a surface having a predetermined shape, such as a circular or polygonal shape. By including the estimated emotion color information d1 in the estimated emotion color surface information, the estimated emotion processing device can display color data da that represents the estimated emotion.

[0132] The video information and the still image information each contain a plurality of pieces of information related to estimated emotions. The plurality of pieces of information include, for example, information about the place where the person whose emotions are estimated was, the environment in which they were (time, season, brightness, etc.), the route they traveled, what they saw, what they touched, what they ate, who they were with or talking to, and who was around the person whose emotions are estimated. The video information and the still image information each contain at least one of image data, graphic data, figure data, animation data, or text data. The video information and the still image information are information that can be displayed on a surface.

[0133] With the above configuration, it is possible to display the estimated emotion color information contained in the estimated emotion color information d1, as well as the video information or still image information contained in the indirect visual information d2 that influences the information. Therefore, the cognitive emotion when a person visually acquires and perceives the composite visual indirect emotion information d0 can be made more accurate, although some ambiguity remains compared to when it is expressed numerically. As a result, the accuracy of emotion estimation when viewing the composite visual indirect emotion information d0 alone can be improved. Moreover, since at least one of the video information or the still image information contains multiple pieces of information, displaying them simultaneously with the estimated emotion color information can further increase the accuracy of the estimated emotion and enhance the sense of satisfaction of those involved. This further improves the accuracy of emotion estimation.

[0134] Therefore, in an estimated emotion processing device that performs data processing using estimated emotions, it is possible to realize a configuration that can improve the accuracy of emotion estimation while suppressing a decrease in the design flexibility of hardware resources, or, if the level of emotion estimation accuracy is about the same, it is possible to realize a configuration that can improve the design flexibility of hardware resources.

[0135] As shown in Figure 5, the estimated emotion color information d1 may include multiple color data da. Figure 5 is a diagram showing an example in which the influential factor indirect visual information d2 and multiple color data da are simultaneously displayed on the display unit 20. As shown in Figure 5, the estimated emotion processing device causes the display unit 20 to simultaneously display the influential factor indirect visual information d2 and multiple color data da included in the estimated emotion color information d1 so that the multiple color data da included in the estimated emotion color information d1 can be compared.

[0136] The control unit of the estimated emotion processing device generates and outputs composite visual indirect emotion information d0, which has a data format in which the influencing factor indirect visual information d2 and the multiple color data da included in the estimated emotion color information d1 are displayed simultaneously, so that the multiple color data da included in the estimated emotion color information d1 are compared.

[0137] Multiple color data da may be displayed with some of the color data da partially overlapping, or with all of the color data da partially overlapping. The influencing factor indirect visual information d2 and multiple color data da may be displayed with partial overlapping.

[0138] The display size of multiple color data da may be the same. The display size of some of the color data da may be different. The display size of the influential indirect visual information d2 and the display size of multiple color data da may be the same or different. The shapes of the influential indirect visual information d2 and the multiple color data da displayed on the display unit 20 may be the same or different. The shapes of the influential indirect visual information d2 and the multiple color data da displayed on the display unit 20 may be any shape, such as a circle or a polygon.

[0139] Multiple color data da include at least one of the color data of different emotion estimation subjects, color data of different locations, or time-series color data. Therefore, the control unit of the estimation emotion processing device generates and outputs composite visual indirect emotion information d0 having a data format in which the influencing factor indirect visual information d2 and the multiple color data da are displayed simultaneously, so as to recognize at least one of the differences between different emotion estimation subjects in the multiple color data da, the differences at different locations in the multiple color data da, or the time-series changes in the multiple color data da.

[0140] The color data at different locations may, for example, be displayed on a map or displayed side by side. The time-series color data may be displayed in chronological order for a predetermined period. Multiple color data da may all be displayed side by side, or some of the color data may be selectively displayed side by means of a selection switch (not shown), for example.

[0141] Furthermore, multiple color data sets da may include, for example, color data representing estimated emotions at different times, or color data representing estimated emotions in different environments. In other words, multiple color data sets da only need to include color data representing different estimated emotions.

[0142] As described above, the composite visual indirect emotion information d0 is a data format in which multiple color data da included in the influential factor indirect visual information d2 and the estimated emotion color information d1 are displayed simultaneously. Therefore, it is possible to compare and view the multiple color data da included in the estimated emotion color information d1, and to clarify the differences in perceived emotions when a person visually acquires and perceives multiple color data da. Thus, by comparing and viewing multiple color data da with the influential factor indirect visual information d2, it is easier for the human brain to perceive, and the accuracy of emotion estimation can be improved. Moreover, since this can be achieved simply by comparing multiple color data da with the influential factor indirect visual information d2, it can be achieved with a small amount of data. Therefore, it is possible to suppress an increase in the load on hardware resources.

[0143] Therefore, in an estimated emotion processing device that performs data processing using estimated emotions, it is possible to realize a configuration that can improve the accuracy of emotion estimation while suppressing a decrease in the design flexibility of hardware resources, or, if the level of emotion estimation accuracy is about the same, it is possible to realize a configuration that can improve the design flexibility of hardware resources.

[0144] [Embodiment 3] Figure 6 shows an example of displaying the gradient color plane information d3, estimated emotion color information d1, and indirect visual influence factor information d2 output by the estimated emotion processing device 300 according to Embodiment 3 on the display unit 20. In this embodiment, the configuration of the output data D0 output from the estimated emotion processing device 300 differs from that of Embodiments 1 and 2. In the following description, components similar to those in Embodiment 2 are denoted by the same reference numerals, and only the parts that differ from Embodiments 1 and 2 will be described.

[0145] In this embodiment, the estimated emotion information acquired by the control unit 312 includes gradient color surface information d3, which includes at least a portion of a color standard in which differences in emotion intensity and emotion type are expressed by a gradient. The gradient color surface information d3 may include all of the information regarding the color standard.

[0146] In this embodiment, as shown in Figure 6, the estimated emotion processing device 300 has a storage unit 11 and a control unit 312. The control unit 312 generates and outputs composite visual indirect emotion information d0, which is composed of a data format that simultaneously displays gradient color plane information d3, estimated emotion color information d1, and influential factor indirect visual information d2. As a result, as shown in Figure 7, the estimated emotion processing device 300 can simultaneously display the estimated emotion color information d1, the influential factor indirect visual information d2, and the gradient color plane information d3 on the display unit 20.

[0147] The composite visual indirect emotion information d0 may have a data format in which estimated emotion color information d1, influential factor indirect visual information d2, and gradient color plane information d3 are displayed in one direction on the display unit 20, or it may have a data format in which two of the estimated emotion color information d1, influential factor indirect visual information d2, and gradient color plane information d3 are displayed in one direction, and the remaining one piece of information is displayed in a direction different from the direction in which the other two pieces of information are displayed.

[0148] For example, the estimated emotion color information d1 and the gradient color plane information d3 may be displayed side by side in the left-right direction on the display unit 20, while the influential factor indirect visual information d2 may be displayed side by side vertically relative to the other two pieces of information on the display unit 20. Alternatively, the estimated emotion color information d1 and the influential factor indirect visual information d2 may be displayed side by side in the left-right direction on the display unit 20, while the gradient color plane information d3 may be displayed side by side vertically relative to the other two pieces of information on the display unit 20. Alternatively, the influential factor indirect visual information d2 and the gradient color plane information d3 may be displayed side by side in the left-right direction on the display unit 20, while the estimated emotion color information d1 may be displayed side by side vertically relative to the other two pieces of information.

[0149] Furthermore, some or all of the estimated emotion color information d1, indirect visual information of influencing factors d2, and gradient color plane information d3 may be displayed on the display unit 20 in a diagonal arrangement or overlapping arrangement.

[0150] The display size of the estimated emotion color information d1, the indirect visual information of influencing factors d2, and the gradient color area information d3 in the display unit 20 may be the same, or the display size of at least one of the three pieces of information may be different. The display shapes of the estimated emotion color information d1, the indirect visual information of influencing factors d2, and the gradient color area information d3 displayed in the display unit 20 may be the same, or they may be partially different. The display shapes of the estimated emotion color information d1, the indirect visual information of influencing factors d2, and the gradient color area information d3 displayed in the display unit 20 may be any shape, such as a circle or a polygon.

[0151] This makes it easy to see the position of the color data da contained in the displayed estimated emotion color information d1 within the gradient color field information d3, which represents at least a portion of the color standard. In this way, by displaying the estimated emotion color information d1, which is ambiguous when viewed alone compared to when it is expressed numerically, together with the gradient color field information d3, the accuracy of the estimated emotion can be increased, and the sense of satisfaction can be enhanced.

[0152] Furthermore, by displaying indirect visual information d2, an influencing factor, in addition to estimated emotion color information d1 and gradient color plane information d3, the accuracy of the estimated emotion displayed by the estimated emotion color information d1 can be further enhanced, thereby increasing the sense of satisfaction.

[0153] Moreover, since this can be achieved simply by simultaneously displaying gradient color information d3, estimated emotion color information d1, and indirect visual information d2 as influencing factors, it is possible to improve the accuracy of emotion estimation while suppressing an increase in the amount of data.

[0154] Therefore, in an estimated emotion processing device that performs data processing using estimated emotions, it is possible to realize a configuration that can improve the accuracy of emotion estimation while suppressing a decrease in the design flexibility of hardware resources, or, if the level of emotion estimation accuracy is about the same, it is possible to realize a configuration that can improve the design flexibility of hardware resources.

[0155] [Embodiment 4] Figure 8 shows a schematic configuration of the estimated emotion processing device 400 according to Embodiment 4. This embodiment differs from the configuration of Embodiment 1 in that the estimated emotion processing device 400 has a display unit 20 and a communication unit 30. In the following, components similar to those in Embodiment 1 are denoted by the same reference numerals, and only the parts that differ from Embodiment 1 will be described.

[0156] As shown in Figure 8, the estimated emotion processing device 400 includes a storage unit 11, a control unit 412, a display unit 20, and a communication unit 30. The control unit 412 outputs output data D0, which includes composite visual indirect emotion information d0, configured in a data format in which both estimated emotion color information d1 and indirect visual information d2 of influencing factors are displayed simultaneously, to at least one of the storage unit 11, the display unit 20, and the communication unit 30.

[0157] The communication unit 30 is a device capable of communicating output data D0 with external equipment via wired or wireless means using electrical signals. The communication unit 30 includes, for example, a communication module, communication equipment, or communication device.

[0158] As a result, the control unit 412 of the estimated emotion processing device 400 can output the composite visual indirect emotion information d0 included in the output data D0 to at least one of the storage unit 11, the display unit 20, or the communication unit 30 so as to simultaneously display the estimated emotion color information d1 and the influential factor indirect visual information d2. Therefore, the composite visual indirect emotion information d0 can be stored in the storage unit 11, displayed in the display unit 20, or transmitted to other devices via the communication unit 30. Furthermore, by making the composite visual indirect emotion information d0 outputtable as described above, it becomes possible to share information with other users via a network or the like.

[0159] Furthermore, by simultaneously displaying estimated emotion color information d1 and indirect visual influencing factor information d2, the cognitive emotion when a person visually acquires and perceives the composite visual indirect emotion information d0 can be expressed numerically, although some ambiguity remains, but the accuracy can be increased. As a result, the accuracy of emotion estimation when viewing the composite visual indirect emotion information d0 can be improved. In addition, since there is no need to significantly increase the types or amount of data, the decrease in the design flexibility of the hardware resources in the emotion estimation processing device can be suppressed. Moreover, if the emotion estimation accuracy is at the same level, the design flexibility of the hardware resources in the emotion estimation processing device can be improved.

[0160] Therefore, in the estimated emotion processing device 400 that performs data processing using estimated emotions, it is possible to realize a configuration that can improve the accuracy of emotion estimation while suppressing a decrease in the design flexibility of hardware resources, or, if the level of emotion estimation accuracy is about the same, it is possible to realize a configuration that can improve the design flexibility of hardware resources.

[0161] [Embodiment 5] Figure 9 shows a schematic configuration of the estimated emotion processing device 500 according to Embodiment 5. In this embodiment, the information output from the estimated emotion processing device is different from that of Embodiment 1. In the following description, components similar to those in Embodiment 1 are denoted by the same reference numerals, and only the parts that differ from Embodiment 1 will be described.

[0162] As shown in Figure 9, the estimated emotion processing device 500 has a storage unit 11 and a control unit 512.

[0163] The control unit 512 acquires character emotion information d4, which includes numerical information, word information, and phrase information that directly indicate emotion, and is different from the composite visual indirect emotion information d0. The control unit 512 stores the acquired character emotion information d4 in the storage unit 11. The character emotion information d4 is character information that allows for the direct recognition of emotion. The character emotion information d4 may be acquired by input to the control unit 512, or the control unit 512 may acquire it from other input devices or storage devices, etc.

[0164] The control unit 512 generates and outputs not only composite visual indirect emotion information d0, which is composed of a data format in which both estimated emotion color information d1 and influential factor indirect visual information d2 are displayed simultaneously, but also composite emotion information composed of a data format in which character emotion information d4 is displayed simultaneously.

[0165] Figure 10 shows an example of displaying the composite visual indirect emotion information d0 and textual emotion information d4 output by the estimated emotion processing device 500 on the display unit 20. As shown in Figure 10, by displaying not only the composite visual indirect emotion information d0 but also the textual emotion information d4, the accuracy of the estimated emotion in the estimated emotion color information d1 included in the composite visual indirect emotion information d0 can be further increased. This makes it possible to further increase the degree of agreement between the estimated emotion and the cognitive emotion recognized by the brain.

[0166] Therefore, it is possible to improve the sense of satisfaction not only for the person whose emotions are estimated, but also for those involved with the emotion estimation processing device 500. Moreover, since the discrepancy between the estimated emotion and the perceived emotion can be suppressed without increasing the type or amount of data, the accuracy of emotion estimation can be improved while suppressing a decrease in the design flexibility of hardware resources in emotion estimation processing. Furthermore, if the level of emotion estimation accuracy does not change, the design flexibility of hardware resources in emotion estimation processing can be improved.

[0167] Furthermore, the composite estimated emotion information may be configured in a data format in which the character emotion information d4 is superimposed on at least one of the estimated emotion color information d1 and the influencing factor indirect visual information d2 of the composite visual indirect emotion information d0.

[0168] [Other embodiments] Although embodiments of the present invention have been described above, the embodiments described above are merely examples for carrying out the present invention. Therefore, the present invention is not limited to the embodiments described above, and it is possible to carry out the present invention by appropriately modifying the embodiments described above without departing from the spirit of the invention. For example, the configurations of each of the embodiments described above can be applied to the configurations of other embodiments.

[0169] In the above embodiment 1, the estimated emotion processing device 1 acquires estimated emotion information D1 and other information D2. However, the estimated emotion processing device may also estimate the emotions of the person whose emotions are to be estimated and use the estimation result as estimated emotion information. Furthermore, the estimated emotion processing device may acquire other information itself, rather than acquiring other information from other devices. That is, the estimated emotion processing device may have an imaging device, a recording device such as a driving recorder, a GPS device, a storage device, an information management device, etc.

[0170] In the above embodiment 1, the estimated emotion processing device 1 expresses estimated emotion using color data included in a color standard created based on Russell's Circle of Emotion model. However, the estimated emotion processing device may express estimated emotion using color data included in a color standard created based on other emotion models.

[0171] In the first embodiment described above, the color criterion is blue for emotion, green for relief, yellow for boredom, and red for tension. However, different types of emotions may be represented by colors other than those described above in the color criterion.

[0172] In the first embodiment described above, the color criterion is white at the intersection of the two evaluation indices, emotional arousal and emotional valence. As one moves radially outward from the intersection of the virtual circle P, one of the colors—blue, green, yellow, or red—becomes darker, and the color changes continuously and smoothly in a gradient of tones around the circumferential direction of the virtual circle P. However, in the color criterion, the color may become lighter as one moves radially outward from the intersection of the virtual circle, or other colors not mentioned above may become darker in the circumferential direction of the virtual circle.

[0173] In the above embodiment 2, the estimated emotion processing device generates and outputs composite visual indirect emotion information having a data format that simultaneously displays indirect visual information d2 of influencing factors and a plurality of color data da. However, the estimated emotion processing device may generate and output composite visual indirect emotion information having a data format that simultaneously displays color data and a plurality of indirect visual information of influencing factors, or it may generate and output composite visual indirect emotion information having a data format that simultaneously displays a plurality of color data and a plurality of indirect visual information of influencing factors.

[0174] In the above embodiment 4, the estimated emotion processing device 400 includes a storage unit 11, a control unit 412, a display unit 20, and a communication unit 30. However, the estimated emotion processing device may have only one of the display unit or the communication unit, or it may not have either the display unit or the communication unit.

[0175] In Embodiment 5, the estimated emotion processing device 500 generates and outputs not only composite visual indirect emotion information d0, which is composed of a data format in which both estimated emotion color information d1 and influential factor indirect visual information d2 are displayed simultaneously, but also composite estimated emotion information composed of a data format in which character emotion information d4 is displayed simultaneously. However, the estimated emotion processing device may generate and output composite estimated emotion information composed of a data format in which, in addition to estimated emotion color information, influential factor indirect visual information, and character emotion information, gradient color plane information as shown in Embodiment 3 is displayed. Alternatively, the estimated emotion processing device may generate and output composite visual indirect emotion information composed of a data format in which character emotion information and estimated emotion color information or influential factor indirect visual information are displayed simultaneously. [Industrial applicability]

[0176] This invention can be used in an estimated emotion processing device that processes data using estimated emotions. [Explanation of Symbols]

[0177] 1, 300, 400, 500 Estimated Emotion Processing Unit 11 Storage section 12, 312, 412, 512 Control Unit 20 Display section 30 Communications Department P Virtual Yen X Color data position D1 Estimated Sentiment Information D2 Other information D0 Output Data da color data d0 Composite visual indirect emotional information d1 Estimated emotion color information d2 Influence factor indirect visual information d3 gradient color information d4 Textual Sentiment Information

Claims

1. An information content recording device that records estimated emotions and information content estimated based on various data obtained from an emotion estimation target whose emotions are estimated; a live event provision device that provides a live event so that the estimated emotions of live viewers are conveyed to online viewers; a content playback device that plays the estimated emotions and content; a vehicle control device that controls the attraction's vehicles according to the estimated emotions of passengers; an employee management device that manages employees using the estimated emotions; or an evaluation device that evaluates objects that affect the five senses of the emotion estimation target using the estimated emotions, etc. An estimated emotion processing device having a control unit that acquires estimated emotion information indicating the estimated emotion and other information that is different from the estimated emotion but related to the estimated emotion, stores the acquired estimated emotion information and the other information in a storage unit in association with each other, and outputs output data including the estimated emotion information and the other information stored in the storage unit, The estimated emotion information obtained by the control unit is, The color standard includes color data representing the estimated emotion, such that the difference in the intensity of the emotion and the difference in the type of emotion are represented by gradients of different types with indistinct boundaries, where the difference in the intensity of the emotion is represented by a gradient of light and dark shades that changes in the radial direction of a virtual circle with indistinct boundaries, and the difference in the type of emotion is represented by a gradient of tone that changes in the circumferential direction of the virtual circle with indistinct boundaries, while including estimated emotion color information that does not include numerical information, word information, and phrase information that directly indicate the estimated emotion, so that the degree of ambiguity is greater than when the estimated emotion is represented by gradient boundaries and numerical values. The aforementioned color standard is, In the gradient of color intensity, the color intensity changes continuously and gradually, and in the gradient of tone, the tone changes continuously and gradually, so that even if the intensity and type of emotion are different, the boundaries are not clear and are ambiguous, and when the estimated emotion color information is viewed alone, the perception of the estimated emotion becomes ambiguous compared to when it is expressed numerically. Other information obtained by the control unit and related to the estimated emotion is, The aforementioned subject of emotion estimation includes information about factors related to the subject of emotion estimation, such as the place where the subject was, the environment in which they were, what they saw, what they touched, what they ate, what they smelled, what they heard, who they were with or talking to, and who were around the subject of emotion estimation, as well as factors that influence the data used to estimate the subject's emotions. However, it does not include numerical information, word information, and phrase information that directly indicate emotions, but includes indirect visual information, which is visual information that indirectly indicates emotions. The output data output from the control unit is, The differences in the intensity of the emotions and the differences in the types of emotions are represented by gradients with unclear boundaries, and the ambiguity of the color data in the estimated emotion color information in the color standard is compensated for by the indirect visual information of the influencing factors, or the indirect visual information of the influencing factors is compensated for by the ambiguity of the color data in the estimated emotion color information. Furthermore, by simultaneously displaying both the estimated emotion color information, which is configured to be ambiguous in the color standard by including the color data, and the indirect visual information of the influencing factors, which does not include information that directly indicates emotion but indirectly indicates emotion, the accuracy of the estimated emotion is increased compared to when only one of them is displayed. The data includes composite visual indirect emotion information, which is composed of a data format in which both the estimated emotion color information and the indirect visual information of the influencing factors are displayed simultaneously. Estimation emotion processing device.

2. In the estimated emotion processing device according to claim 1, The estimated emotional color information is, Including multiple color data, The aforementioned composite visual indirect emotional information is, The data format is configured such that the multiple color data included in the estimated emotion color information and the indirect visual information of the influencing factors are displayed simultaneously, so that the multiple color data included in the estimated emotion color information can be compared. Estimation emotion processing device.

3. In the estimated emotion processing device according to claim 2, The aforementioned multiple color data are The color data includes at least one of the color data of different emotion estimation subjects, the color data of different locations, or the color data of time series. The aforementioned composite visual indirect emotional information is, The data format is configured such that the indirect visual information of the influencing factors and the multiple color data are displayed simultaneously, making it possible to recognize at least one of the differences between different emotion estimation targets in the multiple color data, differences at different locations in the multiple color data, or time-series changes in the multiple color data. Estimation emotion processing device.

4. In the estimated emotion processing device according to any one of claims 1 to 3, The estimated sentiment information is, The differences in the intensity of the aforementioned emotions and the differences in the types of the aforementioned emotions include gradient color plane information that includes at least a portion of the color standard represented by gradients with indistinct boundaries, The aforementioned composite visual indirect emotional information is, The data format is configured to simultaneously display the gradient color information, the estimated emotional color information, and the indirect visual information of the influencing factors. Estimation emotion processing device.

5. In the estimated emotion processing device according to any one of claims 1 to 3, The output data includes the estimated emotion color information displayed overlaid on the color reference. Estimation emotion processing device.

6. In the estimated emotion processing device according to any one of claims 1 to 3, The aforementioned indirect visual information of influencing factors is, Includes at least one of video information or still image information that displays multiple pieces of information, The aforementioned composite visual indirect emotional information is, The estimated emotion color information and at least one of the video information or the still image information are displayed simultaneously in a data format such that they are recognized as different information, or so that they are fused together. Estimation emotion processing device.

7. In the estimated emotion processing device according to any one of claims 1 to 3, A display unit capable of displaying information, It has a communication unit that can communicate information with external devices via electrical signals, either wired or wirelessly. The control unit, The storage unit, the display unit, or the communication unit outputs the composite visual indirect emotion information to at least one of them. Estimation emotion processing device.

8. In the estimated emotion processing device according to any one of claims 1 to 3, The control unit, The system acquires textual emotion information, which includes numerical information, word information, and phrase information that directly indicate emotions, and is different from the composite visual indirect emotion information. It then outputs composite emotion information, which is composed of both the textual emotion information and the composite visual indirect emotion information in a data format that displays them simultaneously, as the output data. Estimation emotion processing device.

9. An information content recording device that records estimated emotions and information content estimated based on various data obtained from an emotion estimation target whose emotions are estimated; a live event provision device that provides a live event so that the estimated emotions of live viewers are conveyed to online viewers; a content playback device that plays the estimated emotions and content; a vehicle control device that controls the attraction's vehicles according to the estimated emotions of passengers; an employee management device that manages employees using the estimated emotions; an evaluation device that evaluates objects that affect the five senses of the emotion estimation target using the estimated emotions, etc. A method for producing an estimated emotion processing device, comprising: installing an estimated emotion processing program on a computer that acquires estimated emotion information indicating the estimated emotion and other information that is different from the estimated emotion but related to the estimated emotion; stores the acquired estimated emotion information and the other information in a memory unit in association with each other; and outputs output data including the estimated emotion information and the other information stored in the memory unit; The estimated sentiment information is, The color standard includes color data representing the estimated emotion, such that the difference in the intensity of the emotion and the difference in the type of emotion are represented by gradients of different types with indistinct boundaries, where the difference in the intensity of the emotion is represented by a gradient of light and dark shades that changes in the radial direction of a virtual circle with indistinct boundaries, and the difference in the type of emotion is represented by a gradient of tone that changes in the circumferential direction of the virtual circle with indistinct boundaries, while including estimated emotion color information that does not include numerical information, word information, and phrase information that directly indicate the estimated emotion, so that the degree of ambiguity is greater than when the estimated emotion is represented by gradient boundaries and numerical values. The aforementioned color standard is, In the gradient of color intensity, the color intensity changes continuously and gradually, and in the gradient of tone, the tone changes continuously and gradually, so that even if the intensity and type of emotion are different, the boundaries are not clear and are ambiguous, and when the estimated emotion color information is viewed alone, the perception of the estimated emotion becomes ambiguous compared to when it is expressed numerically. Other information related to the aforementioned estimated emotions is, The aforementioned subject of emotion estimation includes information about factors related to the subject of emotion estimation, such as the place where the subject was, the environment in which they were, what they saw, what they touched, what they ate, what they smelled, what they heard, who they were with or talking to, and who were around the subject of emotion estimation, as well as factors that influence the data used to estimate the subject's emotions. However, it does not include numerical information, word information, and phrase information that directly indicate emotions, but includes indirect visual information, which is visual information that indirectly indicates emotions. The output data mentioned above is: The differences in the intensity of the emotions and the differences in the types of emotions are represented by gradients with unclear boundaries, and the ambiguity of the color data in the estimated emotion color information in the color standard is compensated for by the indirect visual information of the influencing factors, or the indirect visual information of the influencing factors is compensated for by the ambiguity of the color data in the estimated emotion color information. Furthermore, by simultaneously displaying both the estimated emotion color information, which is configured to be ambiguous in the color standard by including the color data, and the indirect visual information of the influencing factors, which does not include information that directly indicates emotion but indirectly indicates emotion, the accuracy of the estimated emotion is increased compared to when only one of them is displayed. The data includes composite visual indirect emotion information, which is composed of a data format in which both the estimated emotion color information and the indirect visual information of the influencing factors are displayed simultaneously. The process includes the step of installing the estimated emotion processing program into the computer to obtain an estimated emotion processing device that outputs output data including the estimated emotion information and other information. A method for producing an estimated emotion processing device.

10. An information content recording device that records estimated emotions and information content estimated based on various data obtained from an emotion estimation target whose emotions are estimated; a live event provision device that provides a live event so that the estimated emotions of live viewers are conveyed to online viewers; a content playback device that plays the estimated emotions and content; a vehicle control device that controls the attraction's vehicles according to the estimated emotions of passengers; an employee management device that manages employees using the estimated emotions; an evaluation device that evaluates objects that affect the five senses of the emotion estimation target using the estimated emotions, etc. A program for processing estimated emotions, which causes a computer to acquire estimated emotion information indicating the estimated emotion and other information that is different from the estimated emotion but related to the estimated emotion, to associate the acquired estimated emotion information with the other information and store it in a memory unit, and to output output data including the estimated emotion information and the other information stored in the memory unit, The estimated sentiment information is, The color standard includes color data representing the estimated emotion, such that the difference in the intensity of the emotion and the difference in the type of emotion are represented by gradients of different types with indistinct boundaries, where the difference in the intensity of the emotion is represented by a gradient of light and dark shades that changes in the radial direction of a virtual circle with indistinct boundaries, and the difference in the type of emotion is represented by a gradient of tone that changes in the circumferential direction of the virtual circle with indistinct boundaries, while including estimated emotion color information that does not include numerical information, word information, and phrase information that directly indicate the estimated emotion, so that the degree of ambiguity is greater than when the estimated emotion is represented by gradient boundaries and numerical values. The aforementioned color standard is, In the gradient of color intensity, the color intensity changes continuously and gradually, and in the gradient of tone, the tone changes continuously and gradually, so that even if the intensity and type of emotion are different, the boundaries are not clear and are ambiguous, and when the estimated emotion color information is viewed alone, the perception of the estimated emotion becomes ambiguous compared to when it is expressed numerically. Other information related to the aforementioned estimated emotions is, The aforementioned subject of emotion estimation includes information about factors related to the subject of emotion estimation, such as the place where the subject was, the environment in which they were, what they saw, what they touched, what they ate, what they smelled, what they heard, who they were with or talking to, and who were around the subject of emotion estimation, as well as factors that influence the data used to estimate the subject's emotions. However, it does not include numerical information, word information, and phrase information that directly indicate emotions, but includes indirect visual information, which is visual information that indirectly indicates emotions. The output data mentioned above is: The differences in the intensity of the emotions and the differences in the types of emotions are represented by gradients with unclear boundaries, and the ambiguity of the color data in the estimated emotion color information in the color standard is compensated for by the indirect visual information of the influencing factors, or the indirect visual information of the influencing factors is compensated for by the ambiguity of the color data in the estimated emotion color information. Furthermore, by simultaneously displaying both the estimated emotion color information, which is configured to be ambiguous in the color standard by including the color data, and the indirect visual information of the influencing factors, which does not include information that directly indicates emotion but indirectly indicates emotion, the accuracy of the estimated emotion is increased compared to when only one of them is displayed. The data includes composite visual indirect emotion information, which is composed of a data format in which both the estimated emotion color information and the indirect visual information of the influencing factors are displayed simultaneously. A program for processing estimated emotions.

11. An information content recording device that records estimated emotions and information content estimated based on various data obtained from an emotion estimation target whose emotions are estimated; a live event provision device that provides a live event so that the estimated emotions of live viewers are conveyed to online viewers; a content playback device that plays the estimated emotions and content; a vehicle control device that controls the attraction's vehicles according to the estimated emotions of passengers; an employee management device that manages employees using the estimated emotions; an evaluation device that evaluates objects that affect the five senses of the emotion estimation target using the estimated emotions, etc. A recording medium readable by the computer records a program for processing estimated emotions that causes the computer to acquire estimated emotion information indicating the estimated emotion and other information that is different from the estimated emotion but related to the estimated emotion, to associate the acquired estimated emotion information with the other information and store it in a storage unit, and to output output data including the estimated emotion information and the other information stored in the storage unit, The estimated sentiment information is, The color standard includes color data representing the estimated emotion, such that the difference in the intensity of the emotion and the difference in the type of emotion are represented by gradients of different types with indistinct boundaries, where the difference in the intensity of the emotion is represented by a gradient of light and dark shades that changes in the radial direction of a virtual circle with indistinct boundaries, and the difference in the type of emotion is represented by a gradient of tone that changes in the circumferential direction of the virtual circle with indistinct boundaries, while including estimated emotion color information that does not include numerical information, word information, and phrase information that directly indicate the estimated emotion, so that the degree of ambiguity is greater than when the estimated emotion is represented by gradient boundaries and numerical values. The aforementioned color standard is, In the gradient of color intensity, the color intensity changes continuously and gradually, and in the gradient of tone, the tone changes continuously and gradually, so that even if the intensity and type of emotion are different, the boundaries are not clear and are ambiguous, and when the estimated emotion color information is viewed alone, the perception of the estimated emotion becomes ambiguous compared to when it is expressed numerically. Other information related to the aforementioned estimated emotions is, The aforementioned subject of emotion estimation includes information about factors related to the subject of emotion estimation, such as the place where the subject was, the environment in which they were, what they saw, what they touched, what they ate, what they smelled, what they heard, who they were with or talking to, and who were around the subject of emotion estimation, as well as factors that influence the data used to estimate the subject's emotions. However, it does not include numerical information, word information, and phrase information that directly indicate emotions, but includes indirect visual information, which is visual information that indirectly indicates emotions. The output data mentioned above is: The differences in the intensity of the emotions and the differences in the types of emotions are represented by gradients with unclear boundaries, and the ambiguity of the color data in the estimated emotion color information in the color standard is compensated for by the indirect visual information of the influencing factors, or the indirect visual information of the influencing factors is compensated for by the ambiguity of the color data in the estimated emotion color information. Furthermore, by simultaneously displaying both the estimated emotion color information, which is configured to be ambiguous in the color standard by including the color data, and the indirect visual information of the influencing factors, which does not include information that directly indicates emotion but indirectly indicates emotion, the accuracy of the estimated emotion is increased compared to when only one of them is displayed. The data includes composite visual indirect emotion information, which is composed of a data format in which both the estimated emotion color information and the indirect visual information of the influencing factors are displayed simultaneously. Recording medium.

Images