Information processing system and method, information processing server, and storage medium

Abstract

An information processing system and method, an information processing server, and a storage medium can constitute an avatar that more realistically reflects a user's state. The information processing system includes a terminal device that acquires vital data including a user's heart rate variability and transmits the acquired vital data in real time via a communication network, and an information processing server that processes the vital data transmitted from the terminal device. The information processing server stores the vital data and, based on the heart rate variability, infers a user's state, generates display data for displaying an avatar reflecting the inference result, and, in a case where a transmission request for the display data is received via the communication network, transmits the display data to a request source of the request.

Description

[0001] This application is a divisional application of patent application No. 201880038038.4 filed on June 7, 2018, entitled "Information Processing System", the entire contents of which are incorporated herein by reference.

[0002] Cross-references to related applications

[0003] This application is based on Japanese Patent Application No. 2017-112845, filed on June 7, 2017, the contents of which are incorporated herein by reference. Technical Field

[0004] This invention relates to an information processing system for collecting and visualizing information related to living organisms. Background Technology

[0005] In recent years, technologies have been known to visualize information by using various sensors to collect information related to the user as a biological entity and then reflecting and displaying that information in an avatar. Here, an avatar refers to "a role that appears on the internet in place of the user" ("Basic Knowledge of Modern Terminology 2017", Jiyu Kokuminsha, p. 1231).

[0006] For example, Patent Document 1 discloses a technique for measuring emotional data from internet-enabled applications. Specifically, based on physiological data such as electrodermal activity (EDA), accelerometer readings, and skin temperature, or facial expressions and head posture observed from a webcam, the technique infers the user's mental state when interacting with rendered images of websites, videos, etc., and establishes a correlation between this mental state information and the rendered images. Furthermore, Patent Document 1 uses a visual representation, such as an avatar, to display this mental state information.

[0007] Patent document 2 discloses the following technology: detecting the potential at multiple locations on a user's head, or the acceleration or angular velocity of the head, inferring head movement and / or facial expression based on the detection results, and embodying the inferred expression and head movement together and displaying it on a display.

[0008] Existing technical documents

[0009] Patent documents

[0010] Patent Document 1: Japanese Patent Publication No. 2014-504460

[0011] Patent Document 2: Japanese Patent Application Publication No. 2016-126500. Summary of the Invention

[0012] In either of Patent Documents 1 and 2, the avatar is used as a means of communication by displaying the user's mental state and facial expressions. However, due to the development of telecommunications and the emergence of various applications in recent years, it is foreseeable that the methods of using avatars will become more diverse and sophisticated. Therefore, it is desired not only to depict simple mental states and facial expressions such as happiness and sadness, but also to create avatars that more realistically reflect the user's state.

[0013] Therefore, the present invention was made in view of such circumstances, and one of its objectives is to provide an information processing system that can constitute an embodiment that more realistically reflects the user's state.

[0014] To address the aforementioned issues, an information processing system according to one aspect of the present invention comprises: a vital data acquisition unit that acquires vital data including at least a user's heart rate; a terminal device that transmits the vital data acquired by the vital data acquisition unit in real time via a communication network; and an information processing server that processes the vital data transmitted from the terminal device, the information processing server comprising: a storage unit that stores the vital data; a user state inference unit that infers the user's state in real time based at least on the heart rate and a heart rate variability calculated based on the heart rate; an avatar data generation unit that generates display data for displaying an avatar that at least reflects the inference result based on the user state inference unit; and a communication interface that, upon receiving a request to transmit the display data via the communication network, transmits the display data to the requesting source.

[0015] In the aforementioned information processing system, the user state inference unit may infer the user's mental state, and the avatar data generation unit may generate display data for the avatar that reflects the user's mental state.

[0016] In the aforementioned information processing system, the aforementioned avatar data generation unit may generate display data that changes the expression or posture of the aforementioned avatar based on the mental state of the aforementioned user.

[0017] In the aforementioned information processing system, the aforementioned avatar data generation unit may generate display data that changes the color or display range of the avatar's aura based on the user's mental state.

[0018] In the aforementioned information processing system, the user status inference unit may infer the user's health status, and the avatar data generation unit may generate display data for the avatar that reflects the user's health status.

[0019] In the aforementioned information processing system, the aforementioned avatar data generation unit may generate display data that causes the color of the aforementioned avatar to change locally based on the health status of the aforementioned user.

[0020] In the aforementioned information processing system, the aforementioned avatar data generation unit may generate display data that changes the color or display range of the avatar's halo based on the user's health status.

[0021] In the aforementioned information processing system, the user state inference unit may infer the user's activity state, and the avatar data generation unit may generate display data for the avatar that reflects the user's activity state.

[0022] In the aforementioned information processing system, the aforementioned avatar data generation unit may generate display data that changes the shape of the aforementioned avatar based on the user's activity status.

[0023] In the aforementioned information processing system, the display data may be three-dimensional data that includes information related to the interior of the aforementioned avatar, and display data for displaying the interior of the aforementioned avatar may be generated and sent to the aforementioned request source based on the sending request for the display data from the aforementioned request source.

[0024] In the aforementioned information processing system, the life data acquisition unit may also acquire life data of different types than the heart rate, the information processing server may also have a correlation analysis unit, the correlation analysis unit may analyze the correlation between multiple different types of life data, and the user status inference unit may infer the user's status based on the analysis results of the correlation analysis unit.

[0025] According to this disclosure, vital data including at least the user's heart rate is acquired, and the user's state is inferred in real time based on the vital data. Display data reflecting the inference results is generated, thus forming an avatar that more realistically reflects the user's state. Attached Figure Description

[0026] Figure 1 This is a system block diagram that schematically illustrates an example of an information processing system according to an embodiment of the present invention.

[0027] Figure 2 It is shown in general terms. Figure 1 The diagram shows an example of the structure of a user terminal.

[0028] Figure 3 It is shown in general terms. Figure 1 The diagram shows an example of the structure of an information processing server.

[0029] Figure 4 Is Figure 3 The diagram shown illustrates an example of the information stored in the user management database within the storage unit.

[0030] Figure 5 Is Figure 3 The diagram shown illustrates an example of the information stored in the user information database within the storage unit.

[0031] Figure 6 Is Figure 3 The diagram shown illustrates an example of the information stored in the database containing the relevant information stored in the storage unit.

[0032] Figure 7 It means Figure 3 The flowchart shown illustrates the process of constructing the correlation information database performed by the correlation analysis department.

[0033] Figure 8 Is Figure 1 The diagram shows the sequence of information collection and processing performed in the information processing system.

[0034] Figure 9 Is Figure 1 The diagram shows the sequence of display processes performed by the avatar in the information processing system.

[0035] Figure 10 This is a schematic diagram representing an example of an avatar.

[0036] Figure 11 This is a schematic diagram representing other examples of incarnations.

[0037] Figure 12 It is used for Figure 1 The diagram illustrates an example of the application of an SNS in an information processing system.

[0038] Figure 13 It is used for Figure 1 The diagram illustrates an example of the application of an SNS in an information processing system. Detailed Implementation

[0039] The embodiments of the present invention will now be described in detail. Furthermore, the following embodiments are examples for explaining the present invention and are not intended to limit the invention to these embodiments only. In addition, various modifications can be made to the present invention without departing from its spirit. Moreover, those skilled in the art can employ embodiments in which the elements described below are replaced with equivalent elements, and these embodiments are also included within the scope of the present invention.

[0040] (1) Structure of the implementation method

[0041] Figure 1 This is a system block diagram that schematically illustrates an example of an information processing system according to an embodiment of the present invention. For example... Figure 1As shown, the information processing system 1 includes: a life data collection unit 10 for collecting users' biological information, i.e., life data; a user terminal 20; and an information processing server 30. The user terminal 20 and the information processing server 30 are connected via a communication network N (but not limited thereto).

[0042] Network N is a communication network consisting of the Internet, LAN, leased lines, telephone lines, corporate intranets, mobile communication networks, Bluetooth (Bluetooth: a registered trademark), WiFi (Wireless Fidelity), other communication lines, and combinations thereof, whether wired or wireless.

[0043] The vital signs data collection unit 10 includes multiple devices installed on or around the user's body to monitor the user's health and collect vital signs data. Specifically, in addition to a heart rate monitor 11 that measures the user's heart rate, the vital signs data collection unit 10 includes a pulse oximeter 12, a blood pressure monitor 13, a thermometer 14, a network camera 15 that captures the user's face and body movements, and a surface myoelectric sensor 16 that measures the user's muscle movements. Each device can be installed individually or in multiple units. For example, by installing multiple pulse oximeters 12 at different locations on the user's body, measurement accuracy can be improved. Furthermore, a microphone for collecting the user's voice, a pedometer, etc., can also be included in the vital signs data collection unit 10.

[0044] Figure 2 This is a system block diagram schematically illustrating an example of the structure of a user terminal 20 of an information processing system according to an embodiment of the present invention. The user terminal 20 can utilize any terminal device capable of exchanging data with other communication devices via a communication network, such as a tablet terminal, personal computer (PC), laptop PC, smartphone, mobile phone, or portable information terminal (PDA). In this embodiment, by installing and executing a dedicated application on the tablet terminal, the tablet terminal can be used as the user terminal 20.

[0045] The user terminal 20 includes a communication interface 21, an input unit 22, a display unit 23, a camera unit 24, a signal input / output unit 25, a storage unit 26, and a processor 27.

[0046] Communication interface 21 is a hardware module used to connect user terminal 20 to communication network N and to communicate with other terminals on communication network N. Communication interface 21 is, for example, a modem such as an ISDN modem, ADSL modem, cable modem, optical modem, or soft modem.

[0047] Input unit 22 includes various operation buttons, touch panels, and other input devices. Display unit 23 is, for example, an LCD or OLED display. Imaging unit 24 is a camera built into the tablet terminal.

[0048] The signal input / output unit 25 is an interface for connecting external devices to the user terminal 20 via wireless communication based on wired (cable) or Bluetooth (Bluetooth: registered trademark) standards, and for transmitting and receiving signals between the external devices. In this embodiment, each device included in the life data collection unit 10 is connected to the user terminal 20 via the signal input / output unit 25.

[0049] Storage unit 26 is a logical device provided by the storage area of ​​the physical device, and stores the operating system program, drivers, various data, etc. used for processing the user terminal 20. Here, the physical device is a recording medium that can be read by a computer, such as a semiconductor memory. Examples of drivers include communication interface drivers for controlling communication interface 21, input device drivers for controlling input unit 22, display device drivers for controlling display unit 23, imaging device drivers for controlling imaging unit 24, and various drivers for controlling external devices connected to signal input / output unit 25.

[0050] In addition to the various programs and data mentioned above, the storage unit 26 also stores a dedicated application program 261 that is executed by the processor 27 and performs prescribed actions in cooperation with the information processing server 30. Examples of the application program 261 include, for instance, an application program for processing life data collected by the life data collection unit 10 (life information processing application), an application program for SNS (social networking service) (SNS application), and an application program for managing the user's health (health management application).

[0051] The processor 27 consists of an arithmetic logic unit (CPU, etc.) that handles arithmetic operations, logical operations, bit operations, etc., and various registers. It centrally controls various parts of the user terminal 20 by executing various programs stored in the storage unit 26. These registers include, for example, a program counter, a data register, a command register, and general-purpose registers. Furthermore, the processor 27 reads the application program 261 and functions as an application execution unit 271 for life information processing, SNS, health management, etc.

[0052] Such a user terminal 20 preferably receives various life data output from the life data collection unit 10 and sends it to the information processing server 30 continuously and in real time via the communication network N.

[0053] Furthermore, in this embodiment, the life data collection unit 10 is connected to the user terminal 20, and the life data is sent to the information processing server 30 via the user terminal 20. However, it is also possible to configure communication functions for each life data collection unit 10, and to pre-register the identification code (ID) of each life data collection unit 10 with the information processing server 30, so that life data can be sent directly from each life data collection unit 10 to the information processing server 30.

[0054] in addition, Figure 1 The figures show a life data collection unit 10 and a user terminal 20, but are not limited to this. That is, it is also possible to connect two or more user terminals 20, each connected to the life data collection unit 10, to the communication network N, and to access the information processing server 30 simultaneously from each user terminal 20.

[0055] Figure 3 This is a system block diagram schematically illustrating an example of the structure of an information processing server in an information processing system according to an embodiment of the present invention. The information processing server 30 is a server device that accumulates life data sent from the user terminal 20 (or the life data collection unit 10), infers the user's status in real time based on the accumulated life data, and visualizes the user's status according to requests from the user terminal 20, and provides this service to the user terminal 20. The information processing server 30 is, for example, composed of a high-performance mainframe computer, and performs server functions by executing a predetermined server program within this mainframe computer. Furthermore, the information processing server 30 does not necessarily require a single computer; it can be composed of multiple computers distributed across a communication network N.

[0056] The information processing server 30 has a communication interface 31, a storage unit 32 and a processor 33.

[0057] Communication interface 31 is a hardware module that connects to communication network N and is used to communicate with other terminals on communication network N. Specifically, communication interface 31 is, for example, a modem such as an ISDN modem, ADSL modem, cable modem, optical modem, or soft modem.

[0058] The storage unit 32 is a logical device provided by a storage area of ​​a physical device consisting of a recording medium that can be read by a computer, such as a disk drive or semiconductor memory (ROM, RAM, etc.). The storage unit 32 can be constructed by mapping multiple physical devices to one logical device, or by mapping one physical device to multiple logical devices. The storage unit 32 stores various programs, including operating system programs and drivers, as well as various data used in the execution of these programs. Specifically, the storage unit 32 stores the information processing program 321 executed by the processor 33, the user management database 322, the user information database 323, and the related information database 324.

[0059] Information processing program 321 is a program executed by processor 33 to accumulate user's vital data and visualize and provide the user's status (mental state, health status, activity status, etc.) based on the accumulated vital data.

[0060] Figure 4 This is a schematic diagram illustrating information stored in the user management database 322. The user management database 322 stores user account information, including user ID, username, and password, as well as information used to manage access restrictions. Access restrictions are used to limit the scope of information disclosed to other users when other users request access to information related to that user. Access restrictions can be set by the user in a tiered manner, ranging from "fully public (no access restrictions)" to "not public except to the user," based on the user's relationship with other users.

[0061] Figure 5 This is a schematic diagram illustrating the information stored in the user information database 323. The user information database 323 stores basic user information D1, such as the user's date of birth, height, weight, and blood type, D2, and user status information D3 representing the user's status inferred based on the life data D2, according to each user ID.

[0062] The vital data D2 includes raw data directly acquired through the vital data collection unit 10 and secondary data acquired from the raw data. The raw data includes heart rate, pulse rate, blood pressure, body temperature, facial or scalp or body muscle movements, eye or pupil movements, and speech. The secondary data includes: heart rate variability calculated from heart rate; facial expressions or body posture calculated from facial or scalp or body muscle movements; diaphragm movements or spinal extension calculated from abdominal or back muscle movements; eye movement variability calculated from eye movements; and changes in tone of voice (volume, pitch, speed, etc.). This secondary data can also be calculated by the user terminal 20 and sent to the information processing server 30, or it can be calculated by the information processing server 30 itself.

[0063] Furthermore, since the life data D2 is sent from the user terminal 20 in real time and stored in the information processing server 30, the information processing server 30 can also sequentially delete the life data D2 that has been received for a specified period (e.g., several years). Alternatively, in this case, user status information obtained from the life data of the deleted object can be saved.

[0064] User status information D3 includes information representing emotions (joy, anger, sorrow, happiness), stress levels, health levels, uncomfortable body parts, and activity statuses such as "asleep," "awake," "eating," and "exercising." This information can be represented numerically, textually (or symbolically), or by a combination of numerical and textual (or symbolic) information.

[0065] Figure 6 This is a schematic diagram illustrating information stored in the correlation information database 324. The correlation information database 324 stores information (correlation information) that establishes a link between life data and the user's state (mental state, health state, activity state). As an example of correlation information, several different types of life data are listed (…). Figure 6 The table shows the relationship between data A and B and the user's current state. Figure 6 The diagram shows that when a certain life data (data A) has a level of "5" and the other life data (data B) has a level of "4", the user's status is "X1".

[0066] Here, heart rate, as a vital sign, varies depending on physical condition (e.g., normal body temperature or fever), mental state (e.g., calmness or tension / excitement), and activity level (e.g., rest or exercise). On the other hand, a certain degree of fluctuation in heart rate intervals is normal; however, if there is physical or mental stress, or decreased autonomic nervous system function, the variation in heart rate intervals will decrease. Furthermore, in Traditional Chinese Medicine, heart rate and its variability are also used to assess mental state (emotional state, stress level) and health status (functional level of organs, etc.).

[0067] Therefore, by measuring the rate of change in heart rate and heart rate intervals (heart rate variability), it is possible to infer a user's mental state, health status, and activity level to some extent. Furthermore, by combining this data with other vital signs data (blood pressure, body temperature, eye movements captured by a camera, facial muscle movements (expressions), changes in voice tone, body muscle movements (movement), diaphragmatic movements, spinal extension, etc.), the number of inferable aspects of a user's state can be increased, and the accuracy of these inferences can be improved. For example, in addition to mental and health information, it is also possible to infer activity levels such as being asleep or awake.

[0068] Here, the information used to infer a user's state does not need to be limited to vital data; inferences based on vital data can also be used. That is, other types of user states can be inferred from inferences based on vital data and the vital data itself. For example, a user's stress level and facial muscle movements inferred from heart rate and / or heart rate variability can provide a more detailed inference of their mental state.

[0069] The correlation information database 324 stores correlation information used when inferring a user's state in more than one way. Furthermore, the correlation information does not necessarily need to be in table form; it can also store functions with multiple types of life data as variables, inference results based on life data, and functions with life data as variables.

[0070] The relevance information stored in the relevance information database 324 can also be generated in advance based on external information, or it can be generated based on the life data accumulated in the information processing server 30. Moreover, the relevance information generated in advance based on external information can also be updated based on the life data accumulated in the information processing server 30.

[0071] The processor 33 consists of an arithmetic logic unit (CPU, etc.) for handling arithmetic operations, logical operations, bit operations, etc., and various registers. It centrally controls the various parts of the information processing server 30 by executing various programs stored in the storage unit 32. These registers include, for example, a program counter, a data register, a command register, and general-purpose registers. Furthermore, the processor 33, by executing the information processing program 321, cooperates with the user terminal 20 to implement the prescribed information processing functions.

[0072] The functional units implemented by the processor 33 executing the information processing program 321 include the authentication management unit 331, the user information management unit 332, the user status inference unit 333, the avatar data generation unit 334, and the correlation analysis unit 335.

[0073] The authentication management department 331 authenticates the user terminal 20 when it accesses the information processing server 30. Specifically, when the user terminal 20 requests access, the authentication management department 331 requests the user terminal 20 to enter its user ID and password, and authenticates whether to allow the user terminal 20 to access the server by referring to the user management database 322.

[0074] The user information management department 332 manages the user information database 323 based on the information sent from the user terminal 20.

[0075] The user status inference unit 333 infers the user's status based on the life data accumulated in the life data D2 and the correlation information database 324.

[0076] The avatar data generation unit 334 generates an avatar, a persona displayed on the Internet space, as a clone of the user, and generates display data (hereinafter referred to as avatar data) that reflects the user's life data and the inference result (user's state) based on the user state inference unit 333 onto the avatar. There are no particular limitations on the types of life data reflected on the avatar and the user's state, or on the method of displaying the avatar. Examples of avatar display will be described later.

[0077] Here, life data is sent from user terminal 20 in real time, and the inferred user's state is constantly changing, so animated display of the avatar is preferred. Alternatively, the avatar data generation unit 334 can generate three-dimensional data, including information representing the avatar's interior, as avatar data, and generate display data for viewing the avatar's state from the inside (e.g., inside the digestive tract) and display data for cross-sections, each time according to a request from user terminal 20.

[0078] The correlation analysis unit 335 analyzes the correlation between life data (input data) sent from the user terminal 20 and the correlation between life data (input data) and the inference results (output data) based on the user state inference unit 333, thereby constructing a database of correlation information that links life data with the user's state.

[0079] Figure 7 This is a flowchart representing the construction process of the correlation information database performed by the correlation analysis unit 335.

[0080] First, in step S10, the correlation analysis unit 335 acquires first information and one or more second pieces of information that have been pre-associated with the user's state. Here, as described above, it is known that heart rate variability is correlated with the user's stress level; therefore, a pre-association between heart rate variability and stress level can be established, and heart rate variability can be used as the first information in step S10. Additionally, as the second information, data other than heart rate variability, such as the movement of specific body parts like the eyeballs, changes in voice tone, the bulging pattern of the diaphragm, and spinal extension, are acquired. The second information can also consist of two or more different pieces of information.

[0081] In the next step S11, the correlation analysis unit 335 analyzes the correlation between the first information and the second information. In the example above, the correlation between heart rate variability and eye movement, the correlation between heart rate variability and changes in tone of voice, and the correlation between heart rate variability and diaphragmatic movement are analyzed.

[0082] In the next step S12, the correlation analysis unit 335 determines whether the correlation between the first information and the second information is strong. For example, if the correlation coefficient between the two is above a predetermined value, it is determined to be a strong correlation; if the correlation coefficient is below the predetermined value, it is determined to be a weak correlation.

[0083] If the correlation between the first piece of information and the second piece of information is weak (step S12: yes), the correlation analysis unit 335 ends the processing.

[0084] On the other hand, if the correlation between the first and second pieces of information is strong (step S12: Yes), the correlation analysis unit 335 establishes a correlation between the second piece of information and the user's state, which has been previously associated with the first piece of information, based on the correlation analysis results (step S13). Specifically, a table is generated that associates the second piece of information with the user's state. Alternatively, a function can be generated that uses the second piece of information as an input variable and the user's state as an output value. Thus, the user's state can be directly inferred based on the second piece of information. In the example above, the user's stress level can be inferred from the user's eye movement data without considering heart rate fluctuations. Alternatively, the user's emotions can be inferred from changes in tone of voice. Furthermore, the user's level of tension can be inferred from the bulging of the diaphragm and the extension of the spine. The correlation analysis unit 335 stores the correlation information between the second piece of information and the user's state obtained in this way in the correlation information database 324 (see [reference]). Figure 3 Subsequently, the correlation analysis section 335 completed the processing.

[0085] As the first piece of information, any information that is associated with the user's state, other than heart rate variability, can be used. For example, if eye movement is temporarily associated with stress level in step S13 above, then eye movement can also be used as new first information in step S10. In this case, other vital signs data are also used as second information in step S11, and the correlation with the new first information is analyzed, thereby establishing a correlation between stress level via eye movement and the other vital signs data.

[0086] Furthermore, the objects of correlation analysis are not limited to the correlation between vital data. Analysis can also be performed on the correlation between data arbitrarily input by the user and vital data, as well as the correlation between data inferred from vital data and vital data. Specifically, examples include having users input their date of birth, time and place of birth, blood type, DNA type, and their self-evaluation of vital data, and then correlating this input data with vital data (such as heart rate variability).

[0087] In this way, by analyzing the correlations between accumulated vital data and between vital data and other data, it is possible to infer more diverse aspects of a user's state (mental state, health state, activity level) and improve the accuracy of these inferences. Furthermore, by reflecting this inferred user state in an avatar, an avatar more closely reflecting the user's current situation can be displayed. Moreover, by accumulating the results of such correlation analysis, it is also possible to infer diseases that the user is unaware of.

[0088] Furthermore, by accumulating analysis results from a large number of users (e.g., hundreds to tens of thousands) over a certain period (e.g., one year to several years), it is also possible to grasp general trends related to user status. For example, it is also possible to extract trends such as users born in month B from region A being more prone to disease X.

[0089] The analysis results from the correlation analysis unit 335 are stored in the correlation information database 324. The correlation information stored by the correlation analysis unit 335 can be used solely for inferring the status of the user used in the analysis. Alternatively, the generalizable correlation information can be used to infer the status of other users.

[0090] (2) Actions of the implementation method

[0091] Figure 8 This is a sequence diagram of information collection and processing performed in the information processing system 1 according to an embodiment of the present invention.

[0092] If user terminal 20 requests access to information processing server 30 (step S101), then information processing server 30 requests user ID and password, or new user registration, from user terminal 20 (step S201).

[0093] If user terminal 20 sends user registration information for new user registration (step S102), then information processing server 30 issues user ID and password, and regenerates user information (step S202).

[0094] If the user ID and password are sent from the user terminal 20 (step S103) and the authentication is successful in the information processing server 30 (step S203), then the user terminal 20 becomes logged in and becomes a state in which life data can be stored from the user terminal 20 in the information processing server 30.

[0095] If data is sent from user terminal 20 to life data collection unit 10 (refer to...) Figure 1 If the collected life data is received (step S104), the information processing server 30 receives the life data and stores it in the user information database 323 (step S204). Next, the information processing server 30 infers the user's state (mental state, health state, activity state) based on the accumulated life data (step S205), and generates avatar data to display an avatar reflecting the user's state (step S206).

[0096] Figure 9 This is a sequence diagram of the avatar display processing performed in the information processing system 1 according to an embodiment of the present invention.

[0097] If user terminal 20 requests access to information processing server 30 (step S111), then information processing server 30 requests user ID and password from user terminal 20 (step S211).

[0098] If the user ID and password are sent from the user terminal 20 (step S112) and authentication is successful in the information processing server 30 (step S212), then the user terminal 20 becomes logged in. Alternatively, steps S112, S211, and S212 are omitted if the user terminal 20 remains logged in.

[0099] If user terminal 20 requests avatar data for a specific user from information processing server 30 (step S113), information processing server 30 refers to user management database 322 to confirm the access restrictions of the user who requested the avatar data (step S213). Furthermore, within the scope of the access restrictions, the requested avatar data is sent to user terminal 20 (step S214). For example, if the access restriction is set to "not disclosed to anyone other than the user," information processing server 30 will not send the avatar data to users other than the user who requested the avatar data.

[0100] User terminal 20 displays the avatar on the screen based on the received avatar data (step S114).

[0101] Figure 10 This is a schematic diagram illustrating an example of an avatar display, showing an avatar A1 simulating the entire human body. Alternatively, a heart model a11 could be overlaid on avatar A1, for example, so that the heart model a11 beats in sync with the user's heart rate. Furthermore, the overall color of avatar A1 can change according to the user's body temperature. Alternatively, the inferred mental state of the user (e.g., emotions, stress level) can be reflected in the facial expression and color (complexion) of avatar A1's face a12. Moreover, the inferred health status of the user (e.g., level of health to discomfort, stress level) can be reflected in the halo (backlight) a13 of avatar A1. As an example, the healthier the user, the larger the display area of ​​halo a13, or the color of halo a13 can change according to the stress level. Additionally, the color of the portion of avatar A1 corresponding to the user's uncomfortable area can change according to the degree of discomfort. As an example, if the user's shoulder is severely stiff, the brightness of the shoulder portion a14 of avatar A1 can be reduced to indicate poor blood circulation. Additionally, the shape of avatar A1 can be changed to match the user's activity state. As an example, based on the surface myoelectric sensor 16 (see reference...) Figure 1 The user's muscle movements are obtained, causing the avatar A1 to change its posture.

[0102] Alternatively, the avatar data generation unit 334 may change the form of avatar A1 based on a request sent from the user terminal 20. For example, the display unit 23 of the user terminal 20 may display a slider a15 that can be moved by operating the input unit 22. If the slider a15 moves, information indicating the position of the slider a15 is sent to the information processing server 30. The avatar data generation unit 334 generates avatar data reflecting past life data based on the position of the slider a15 and sends it to the user terminal 20. Thus, the user terminal 20 displays an avatar A1 reflecting life data from the period desired by the user. In this way, past avatars A1 can also be displayed, allowing the user to confirm changes in health status over time.

[0103] Alternatively, by performing a prescribed operation (e.g., a click operation) on the avatar A1 displayed on the display unit 23 of the user terminal 20, information indicating the selected area is sent to the information processing server 30. The avatar data generation unit 334 generates avatar data representing the interior (e.g., organs) of the selected area and sends it to the user terminal 20. Thus, the user terminal 20 displays the avatar A1 with the desired internal area exposed. The method of displaying the internal area can also be by displaying a cross-section of the avatar A1, or by inserting a small camera inside the avatar A1 and displaying images captured by that camera.

[0104] Figure 11 This is a schematic diagram representing other display examples of avatars, showing an avatar A2 that simulates the user's head. Within this avatar A2, there are areas representing the user's emotions (emotion area a21), areas representing the activity state of the right brain (right brain area a22), and areas representing the activity state of the left brain (left brain area a23). Based on vital data and the inferred state of the user, the size, color, etc. of each area are changed.

[0105] As explained above, according to this embodiment, vital data including at least the user's heart rate is acquired in real time, and avatar data is generated based on the user's state inferred in real time from the vital data, thus enabling the creation of an avatar that truly reflects the user's state.

[0106] Furthermore, according to this embodiment, the correlation between multiple life data and the correlation between the inference result of the user's state and the life data are analyzed, and the user's state is further inferred based on their correlation. Therefore, it is possible to increase the items that can be inferred about the user's state and improve the inference accuracy.

[0107] Information processing systems like this 1, which use user biometric data to create avatars, can be utilized in various applications. As an example, by combining information processing system 1 with SNS (social networking systems) such as "Facebook" and "LinkedIn," avatars can be used as user profiles.

[0108] Figure 12 as well as Figure 13 This diagram illustrates an application example of the SNS in Information Processing System 1. For example, as shown... Figure 12As shown, in a certain social networking service (SNS), user A is associated with users B, C, and D as "friends." Additionally, user B is also associated with users E and F as "friends," in addition to user A. The mental state, health state, activity state, or aura values ​​of these users' avatars are quantified, and statistical values ​​are calculated based on the users' friendship relationships. For example, the aura values ​​for users A through F are 2, 3, 8, 5, 6, and 4, respectively. In this case, as... Figure 13 As shown, the total aura value of User A's friends (Users B, C, and D) is 3 + 8 + 5 = 16, with an average of approximately 5.3. On the other hand, the total aura value of User B's friends (Users A, E, and F) is 2 + 6 + 4 = 12, with an average of approximately 4. Therefore, it can be concluded that User A has more friends with higher aura values ​​than User B.

[0109] Furthermore, by changing the numerical values ​​used in the analysis, for example, analysis could be performed on user A to determine if they have many friends with unstable personalities or unhealthy traits. Alternatively, a comparison could be made between user A's professional and personal networks. In addition to numerical values ​​such as aura, mental state, health status, and activity level, vital signs data (such as heart rate) can also be used as numerical values ​​for the analysis. Furthermore, while the above explanation uses the average as the statistical value, the median, most frequent value, etc., can also be used.

[0110] Furthermore, it could also be, in addition to through Figure 12 Beyond the online friendships illustrated, the system also acquires aura values ​​and life data related to followers and commenters on posted messages, displaying them in the form of points and rankings. This allows for the analysis of trends among followers, such as user A having a high stress level or user B having a high health level but a low activity level.

[0111] Additionally, as another example, information processing systems can be used on recruitment websites and dating websites. In this case, the information of members (job applicants, dating applicants) can be presented to companies and potential partners.

[0112] Additionally, as another example, information processing systems can be used in gaming websites. For instance, avatars can be pitted against each other in competitive games.

[0113] As another example, information processing system 1 can also be used in health management applications. In this case, even when the user is in a remote location, indicators of the user's health status, such as pulse and body temperature, can be obtained from the avatar displayed on the user terminal 20. Furthermore, organs can be displayed with the avatar's abdomen open, or the avatar's digestive tract can be displayed from the inside. Additionally, usage methods such as allowing the user to observe the avatar's overall posture and correct their own posture can also be provided.

[0114] In the above embodiments and variations, humans are used as users of the information processing system 1, but animals such as pets and livestock can also be used as users. That is, a life data collection unit is installed on dogs, cats, etc., and an avatar of the animal is generated based on the collected life data. In this case, veterinarians can also observe the avatar for medical treatment.

[0115] The embodiments described above are merely examples for illustrating the present invention and are not intended to limit the invention to these embodiments. Furthermore, the present invention can be modified in various ways without departing from its spirit. For example, those skilled in the art can substitute the resources (hardware resources or software resources) described in the embodiments with equivalents, and such substitutions are also included within the scope of the present invention.

[0116] Explanation of reference numerals in the attached figures

[0117] 1...Information processing system; 10...Life data collection unit; 11...Heart rate monitor; 13...Blood pressure monitor; 14...Thermometer; 15...Network camera; 16...Surface myoelectric potential sensor; 20...User terminal; 21...Communication interface; 22...Input unit; 23...Display unit; 24...Image capture unit; 25...Signal input / output unit; 26...Storage unit; 27...Processor; 30...Information processing server; 31...Communication interface; 32...Storage unit; 33...Processor; 261...Application program; 271...Application program execution unit; 321...Information processing program; 322...User management database; 323...User information database; 324...Relevance information database; 331...Authentication management unit; 332...User information management unit; 333...User status inference unit; 334...Avatar data generation unit; 335...Relevance analysis unit.

Claims

1. An information processing system, characterized in that, have: The vital data acquisition unit acquires the user's vital data. A terminal device that transmits life data acquired by the life data acquisition unit via a communication network; and An information processing server processes the life data sent from the terminal device. The information processing server has: Storage unit, which stores the life data; The user status inference unit analyzes the correlation between the first life data, which includes the rate of change of heart rate and heart rate interval (i.e., the rate of change of heart rate), and the second life data, which is of a different type from the first life data, and infers the user's status based on the analysis results using a pre-stored table that includes the level of the first life data, the level of the second life data, and the user's status corresponding to the level of the first life data and the second life data together. An avatar data generation unit generates display data, which is used to display an avatar reflecting the user's state as inferred by the user state inference unit; as well as A communication interface that, upon receiving a request to send display data via a communication network, sends the display data to the requesting source. The information processing system is able to display an avatar reflecting the user's status in the request source.

2. An information processing system, characterized in that, have: The vital data acquisition unit acquires the user's vital data. A terminal device that transmits life data acquired by the life data acquisition unit via a communication network; and An information processing server processes the life data sent from the terminal device. The information processing server has: Storage unit, which stores the life data; The user status inference unit analyzes the correlation between first life data, which includes the rate of change of heart rate and heart rate interval (i.e., heart rate variability), and second life data of a different type from the first life data. Based on the analysis results, the user status is inferred. The user status inference unit determines whether the correlation coefficient between the first life data and the second life data is above a predetermined value. If the correlation coefficient is above the predetermined value, the second life data is associated with the user's status, which has been previously associated with the first life data, thereby directly inferring the user's status based on the second life data. An avatar data generation unit generates display data, which is used to display an avatar reflecting the user's life data and the user's state inferred by the user state inference unit; as well as A communication interface that, upon receiving a request to send display data via a communication network, sends the display data to the requesting source. The information processing system is able to display an avatar reflecting the user's state in the request source.

3. An information processing server, characterized in that, have: The storage unit stores first life data received from the terminal device via the communication network, including the user's heart rate and the rate of change of heart rate intervals, i.e., the heart rate variability. The user status inference unit analyzes the correlation between the first vital data, including the heart rate and the heart rate variability, and the second vital data, which is of a different type from the first vital data. Based on the analysis results, it uses a pre-stored table that includes the level of the first vital data, the level of the second vital data, and the user status corresponding to the level of the first vital data and the second vital data together to infer the user's status. An avatar data generation unit generates display data, which is used to display an avatar reflecting the user's state as inferred by the user state inference unit; as well as A communication interface that, upon receiving a request to send display data via a communication network, sends the display data to the requesting source. The information processing server can display an avatar reflecting the user's status in the request source.

4. The information processing server according to claim 3, characterized in that, The avatar data generation unit generates display data for the avatar that corresponds to the information input from the request source and reflects past life data. The communication interface sends data reflecting the past life data of the avatar to the request source, thereby enabling the time-series changes of the user's state to be confirmed in the request source.

5. The information processing server according to claim 3, characterized in that, The user state inference unit infers at least one state of the user, selected from the group consisting of the user's mental state, the user's health state, and the user's activity state. The avatar data generation unit generates display data for the avatar that reflects at least one of the user's states.

6. The information processing server according to claim 3, characterized in that, The display data is three-dimensional data that includes information related to the internal structure of the avatar. Based on the request to send display data from the request source, display data for displaying the interior of the avatar is generated and sent to the request source.

7. An information processing server, characterized in that, have: The storage unit stores first life data received from the terminal device via the communication network, including the user's heart rate and the rate of change of heart rate intervals, i.e., the heart rate variability. The user status inference unit analyzes the correlation between the first life data and the second life data of a different type from the first life data, and infers the user's status based on the analysis results. The user status inference unit determines whether the correlation coefficient between the first life data and the second life data is above a predetermined value. If the correlation coefficient is above the predetermined value, the second life data is associated with the user's status that has been previously associated with the first life data, thereby directly inferring the user's status based on the second life data. An avatar data generation unit generates display data, which is used to display an avatar reflecting the user's life data and the user's state inferred by the user state inference unit; as well as A communication interface that, upon receiving a request to send display data via a communication network, sends the display data to the requesting source. The information processing server displays an avatar reflecting the user's state in the request source.

8. An information processing method, comprising an information processing server having a processor and a storage unit, characterized in that, The processor has the following functions: First life data, including the user's heart rate and the rate of change of heart rate intervals, i.e., heart rate variability, is received from the terminal device via a communication network. The received heart rate and heart rate variability are stored in the storage unit; The correlation between first life data, including the heart rate and the heart rate variability, and second life data of a different type from the first life data is analyzed. Based on the analysis results, the user's status is inferred using a pre-stored table that includes the level of the first life data, the level of the second life data, and the user's status corresponding to the level of the first life data and the second life data together. Generate display data for displaying an avatar that reflects the inferred state of the user; Upon receiving a request to send display data via a communication network, the display data is sent to the requesting source. The information processing method can display an avatar reflecting the user's state in the request source.

9. A storage medium, characterized in that, It stores programs for causing the computer to perform the following steps: First life data, including the user's heart rate and the rate of change of heart rate intervals, i.e., heart rate variability, is received from the terminal device via a communication network. The received heart rate and heart rate variability are stored in the storage unit; The correlation between first life data, including the heart rate and the heart rate variability, and second life data of a different type from the first life data is analyzed. Based on the analysis results, the user's status is inferred using a pre-stored table that includes the level of the first life data, the level of the second life data, and the user's status corresponding to the level of the first life data and the second life data together. Generate display data for displaying an avatar that reflects the inferred state of the user; as well as Upon receiving a request to send display data via a communication network, the display data is sent to the requesting source, whereby the program displays an avatar reflecting the user's state in the requesting source.

Images