Information processing device, information processing method, and information processing program

Abstract

This information processing device comprises: a biological information acquisition unit that acquires biological information pertaining to a user; an image generation unit that generates, from the acquired biological information, a first image in a format in which the biological information can be concealed; and a state determination unit that determines the state of the user from the generated first image using a learning model that has been caused to learn the relationship between the state of a person and a second image that is generated from biological information pertaining to the person and is in a format in which the biological information can be concealed.

Description

Information processing device, information processing method, and information processing program

[0001] The present disclosure relates to a technique for determining a user's condition from the user's biological information.

[0002] For example, Patent Document 1 discloses a discomfort estimation device that receives biometric information of an occupant, determines the driving state of the vehicle, selects a learned model corresponding to the determined driving state from learned models for each driving state created by machine learning the relationship between the occupant's biometric information and the occupant's discomfort level for each driving state of the vehicle, and estimates the occupant's discomfort level using the selected learned model and the received biometric information.

[0003] However, with the above-mentioned conventional technology, it is difficult to accurately determine the user's condition while protecting privacy and to perform appropriate intervention control in accordance with the determined user's condition, and further improvements are needed.

[0004] Japanese Patent Application Laid-Open No. 2023-407

[0005] The present disclosure has been made to solve the above problems, and aims to provide a technology that can accurately determine a user's condition while protecting privacy and can perform appropriate intervention control according to the determined user's condition.

[0006] The information processing device according to the present disclosure includes an acquisition unit that acquires biometric information of a user, a generation unit that generates a first image in a format that allows the biometric information to be concealed from the acquired biometric information, and a determination unit that determines the state of the user from the generated first image using a learning model that has learned the relationship between a second image in a format that allows the biometric information to be concealed, generated from the biometric information of the person, and the state of the person.

[0007] According to the present disclosure, it is possible to accurately determine a user's state while protecting privacy, and to perform appropriate intervention control in accordance with the determined user's state.

[0008] 1 is a diagram illustrating a configuration of a content playback system according to the first embodiment. FIG. 2 is a diagram illustrating an example of LFHF data and a first image in which the LFHF data is visualized in the first embodiment. FIG. 3 is a flowchart illustrating content playback processing by an information processing device according to the first embodiment of the present disclosure. FIG. 4 is a first flowchart illustrating content playback processing by an information processing device according to a modified example of the first embodiment of the present disclosure. FIG. 5 is a diagram illustrating a configuration of a content playback system according to the second embodiment. FIG. 6 is a diagram illustrating a face image and an example of a first image in which only a portion of a face feature is cut out from the face image in the second embodiment. FIG. 7 is a diagram illustrating a face image and an example of a first image generated by converting the face image into a binary image in the first modified example of the second embodiment. FIG. 8 is a flowchart illustrating content playback processing by an information processing device according to the second embodiment of the present disclosure. FIG. 9 is a first flowchart illustrating content playback processing by an information processing device according to the second modified example of the second embodiment of the present disclosure. FIG. 10 is a second flowchart illustrating content playback processing by an information processing device according to the second modified example of the second embodiment of the present disclosure.

[0009] (Foundation of the Present Disclosure) Conventionally, a user's state has been estimated from biometric information.

[0010] The discomfort estimation device shown in Patent Document 1 above receives biometric information representing heart rate, electrocardiogram, respiration, body temperature, or sweating, selects a trained model corresponding to the driving state of the occupant, and estimates the degree of motion sickness using the selected trained model and the received biometric information.

[0011] However, in the above-mentioned Patent Document 1, unprocessed biometric information is input into the trained model, so there is a risk that the biometric information may be leaked to the outside, and privacy may not be protected.

[0012] In order to solve the above problems, the following techniques are disclosed.

[0013] (1) An information processing device according to one aspect of the present disclosure includes an acquisition unit that acquires biometric information of a user, a generation unit that generates a first image in a format that allows the biometric information to be concealed from the acquired biometric information, and a determination unit that determines the state of the user from the generated first image using a learning model that has learned the relationship between a second image in a format that allows the biometric information to be concealed, generated from the biometric information of the person, and the state of the person.

[0014] According to this configuration, a first image in a format that allows the biometric information to be concealed is generated from the acquired biometric information, and the user's condition is determined from the first image using a learning model. Here, since the first image is in a format that allows the biometric information, which is personal information, to be concealed, it is difficult to identify the original biometric information from the first image. Therefore, it is possible to accurately determine the user's condition while protecting privacy, and to perform appropriate intervention control according to the determined user's condition.

[0015] (2) In the information processing device described in (1) above, the biometric information may include heart rate data relating to heart rate, and the generation unit may calculate time series data of LFHF values, which are stress indicators, as LFHF data based on the heart rate data, and convert a power spectrum obtained by frequency characteristic analysis of the LFHF data into the first image.

[0016] According to this configuration, the first image, which visualizes the time series data of the LFHF value, which is a stress index, is used in the judgment process, making it difficult to identify the original heart rate data from the first image, thereby protecting the user's privacy.

[0017] (3) In the information processing device described in (2) above, the first image may represent the frequency components of the LFHF data in the power spectrum and the time change of the LFHF value using a plurality of colors.

[0018] According to this configuration, the first image, which represents the frequency components of the LFHF data in the power spectrum and the time changes of the LFHF values ​​using a plurality of colors, is used in the determination process, thereby making it possible to determine the user's condition with high accuracy.

[0019] (4) In the information processing device described in (1) above, the biometric information includes a facial image of the user, and the generation unit may generate the first image by processing of cutting out only a portion of the facial features from the facial image, converting the facial image into a binary image, converting the facial image into a binary image and extracting multiple feature points of the facial features, or reducing the resolution of the facial image.

[0020] According to this configuration, the first image is generated by a process of cutting out only a portion of the facial features from the facial image, a process of converting the facial image into a binary image, a process of converting the facial image into a binary image and extracting multiple feature points of the facial features, or a process of reducing the resolution of the facial image, so that it is difficult to identify the original facial image from the first image, and the user's privacy can be protected.

[0021] (5) In the information processing device according to any one of (1) to (4) above, the determination unit may determine whether the user is in a tense state or a relaxed state.

[0022] With this configuration, it is possible to determine whether the user is in a tense state or a relaxed state.

[0023] (6) In the information processing device according to any one of (1) to (4) above, the determination unit may determine whether the user is in a sleeping state.

[0024] According to this configuration, it is possible to determine whether the user is in a sleeping state.

[0025] (7) In the information processing device according to any one of (1) to (6) above, the acquisition unit may periodically acquire the biometric information.

[0026] According to this configuration, biological information is acquired periodically, so that the user's condition can be determined in real time.

[0027] (8) In the information processing device according to any one of (1) to (7) above, the acquisition unit may acquire the biometric information from a current time up to a predetermined time period ago.

[0028] According to this configuration, a first image reflecting the user's most recent biometric information is generated, and the user's condition is determined from the generated first image, so that the user's most recent condition can be determined.

[0029] (9) In the information processing device described in any one of (1) to (8) above, the user may be a passenger of a moving body.

[0030] According to this configuration, the state of a user riding on a moving object can be determined.

[0031] (10) The information processing device according to any one of (1) to (9) above may further include a playback unit that plays back content according to the state of the user.

[0032] According to this configuration, content is played back according to the state of the user, so that the state of the user to whom the content is provided can be changed or maintained.

[0033] (11) In the information processing device described in (10) above, when it is determined that the user's state is tense, the playback unit may play a first video content to guide the user from the tense state to a relaxed state.

[0034] According to this configuration, when it is determined that the user's state is tense, a first video content is played to guide the user from the tense state to a relaxed state, thereby changing the user's state from the tense state to a relaxed state.

[0035] (12) In the information processing device described in (10) above, if it is determined that the user's state is not tense, the playback unit may play a second video content to induce the user from a relaxed state to a sleep state.

[0036] According to this configuration, if it is determined that the user's state is not tense, a second video content is played to guide the user from a relaxed state to a sleeping state, thereby changing the user's state from a relaxed state to a sleeping state.

[0037] Furthermore, the present disclosure can be realized not only as an information processing device having the above-described characteristic configuration, but also as an information processing method that executes characteristic processing corresponding to the characteristic configuration of the information processing device. Furthermore, the present disclosure can also be realized as a computer program that causes a computer to execute characteristic processing included in such an information processing method. Therefore, the same effects as those of the above-described information processing device can be achieved in the following other aspects.

[0038] (13) An information processing method according to another aspect of the present disclosure is an information processing method executed by a computer, including acquiring biometric information of a user, generating a first image from the acquired biometric information in a format that allows the biometric information to be concealed, and determining the state of the user from the generated first image using a learning model that has learned the relationship between a second image generated from the person's biometric information in a format that allows the biometric information to be concealed and the state of the person.

[0039] (14) An information processing program according to another aspect of the present disclosure causes a computer to acquire biometric information of a user, generate a first image from the acquired biometric information in a format that allows the biometric information to be kept confidential, and determine the state of the user from the generated first image using a learning model that has learned the relationship between a second image generated from the person's biometric information in a format that allows the biometric information to be kept confidential and the state of the person.

[0040] (15) A non-transitory computer-readable recording medium according to another aspect of the present disclosure records an information processing program, which causes a computer to acquire biometric information of a user, generate a first image from the acquired biometric information in a format that allows the biometric information to be concealed, and determine the state of the user from the generated first image using a learning model that has learned the relationship between a second image generated from the person's biometric information in a format that allows the biometric information to be concealed and the state of the person.

[0041] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components. Furthermore, in all embodiments, the respective contents can be combined.

[0042] First Embodiment FIG. 1 is a diagram showing the configuration of a content reproduction system 10 according to a first embodiment.

[0043] The content reproduction system 10 shown in FIG. 1 includes a sensor 1, an information processing device 2, and a display 3.

[0044] The sensor 1 measures biometric information of a user. The biometric information includes heartbeat data related to the heartbeat. The heartbeat data includes heart rate data. The sensor 1 is, for example, a heart rate monitor that measures the user's heart rate over a predetermined period and outputs the heart rate data. The sensor 1 transmits the biometric information including the heart rate data to the information processing device 2. The sensor 1 is connected to the information processing device 2 via a network so that they can communicate with each other. The network is, for example, the Internet.

[0045] The sensor 1 may be provided in a wearable device such as a smartwatch. The wearable device may transmit the measured biological information directly to the information processing device 2, or may transmit the measured biological information to the information processing device 2 via a mobile terminal such as a smartphone.

[0046] Sensor 1 may also be a millimeter-wave radar configured to detect heartbeats without contact. Millimeter-wave radar detects minute vibrations on the order of micrometers in the millimeter wave band without contact. Sensor 1 may also be a Doppler sensor configured to capture weak radio waves reflected from the human body and measure pulse waves without contact.

[0047] The sensor 1 may store the biological information for a predetermined period, and upon receiving a request to transmit the biological information from the information processing device 2, transmit the biological information for the predetermined period to the information processing device 2.

[0048] The information processing device 2 includes at least a computer system including, for example, a control program, a processing circuit such as a processor or logic circuit that executes the control program, and a recording device such as an internal memory or an accessible external memory that stores the control program. Note that the information processing device 2 may be realized, for example, by hardware implementation using the processing circuit, or by execution of a software program stored in the memory by the processing circuit or distributed from an external server, or by a combination of these hardware and software implementations.

[0049] The information processing device 2 is installed in a vehicle in which a user rides. The vehicle is, for example, a vehicle that performs autonomous driving using artificial intelligence technology. The information processing device 2 may also be a server.

[0050] The information processing device 2 includes a biometric information acquisition unit 21 , an image generation unit 22 , a state determination unit 26 , a content storage unit 27 , a content selection unit 28 , and a content playback unit 29 .

[0051] The biometric information acquisition unit 21 acquires biometric information of the user. The biometric information includes heart rate data. The biometric information acquisition unit 21 receives the biometric information transmitted by the sensor 1. The biometric information acquisition unit 21 synchronizes with an application of the sensor 1 and acquires the biometric information from the sensor 1.

[0052] The biometric information acquisition unit 21 acquires the user's biometric information from the current time up to a predetermined period of time ago. The user may be a passenger of the mobile body. If the user is a passenger of the mobile body, the biometric information acquisition unit 21 acquires biometric information from the time the user boarded the mobile body up to a predetermined period of time ago. The predetermined period of time is, for example, 30 minutes. When the mobile body is powered on, the biometric information acquisition unit 21 may send a request to send biometric information to the sensor 1.

[0053] The biometric information acquiring unit 21 periodically acquires biometric information. The biometric information acquiring unit 21 may periodically receive the biometric information transmitted by the sensor 1.

[0054] The image generating unit 22 generates a first image in a format that can conceal the biometric information from the biometric information acquired by the biometric information acquiring unit 21. The image generating unit 22 calculates LFHF data, which is a stress index, based on heart rate data related to the heart rate, and converts a power spectrum obtained by frequency characteristic analysis of the LFHF data into the first image.

[0055] The image generation unit 22 includes an LFHF processing unit 23 , a spectrum conversion unit 24 , and an image conversion unit 25 .

[0056] The LFHF processing unit 23 converts the heart rate data into heart rate interval variability time series data (RRI time series data) and calculates time series data of LFHF values, which are stress indexes, as LFHF data based on the heart rate interval variability time series data. The LFHF data is time series data of LFHF values, which are the ratio between the LF component and the HF component of the power spectrum obtained by frequency analysis of the heart rate variability time series data.

[0057] The LFHF value is a value used to evaluate autonomic nervous function. Heartbeats include multiple waveforms: P waves, Q waves, R waves, S waves, and T waves. The RR interval represents the interval between R waves, which are the largest peak waves among the multiple waveforms. The RR interval is synonymous with the heartbeat and is known to fluctuate rather than always be constant. The RR interval time series data is heart rate variability time series data. The LFHF processing unit 23 obtains a power spectrum by frequency analysis of the heart rate variability time series data. This power spectrum contains low frequency LF (low frequency) components and high frequency HF (high frequency) components. The LF component is related to blood pressure regulation and is influenced by both the sympathetic and parasympathetic nervous systems. The HF component is related to respiratory fluctuations and is influenced by the parasympathetic nervous system. The LFHF processing unit 23 calculates the ratio between the LF component and the HF component to calculate an LFHF value, which is an index of the sympathetic nerve activity, that is, a stress index.

[0058] The LF component is, for example, a value in the frequency band of 0.05 Hz to 0.15 Hz, and the HF component is, for example, a value in the frequency band of 0.15 Hz to 0.40 Hz. When the user is in a relaxed state (relaxed state), the HF component is larger relative to the LF component, and the LFHF value is smaller. Conversely, when the user is in a tense state (stressed state), the LF component is larger relative to the HF component, and the LFHF value is larger.

[0059] The heart rate data may be electrocardiogram data. The LFHF processing unit 23 may convert the electrocardiogram data into heart rate interval variability time series data and calculate, as LFHF data, time series data of the LFHF value, which is a stress index, based on the heart rate interval variability time series data.

[0060] The heart rate data may also be heart rate variability (HRV (Heart Rate Variability)) data. Heart rate variability is a phenomenon in which the intervals between heartbeats periodically fluctuate. Heart rate variability is regulated by the autonomic nervous system and synchronizes with fluctuations in respiration and blood pressure. Heart rate variability is used as an index for evaluating stress levels or the state of the autonomic nervous system. Heart rate variability reflects the activity state of the autonomic nervous system and indicates how well the sympathetic and parasympathetic nervous systems are balanced. Generally, the higher the heart rate variability value, the more dominant the parasympathetic nervous system is, indicating a state of relaxation of the mind and body. On the other hand, the lower the heart rate variability value, the more dominant the sympathetic nervous system is, indicating a state of accumulated stress or fatigue. The LFHF processing unit 23 may calculate time series data of the LFHF value, which is a stress index, as LFHF data by frequency analyzing the heart rate variability data.

[0061] The spectrum transform unit 24 transforms the LFHF data in the time domain into a power spectrum in the frequency domain. The spectrum transform unit 24 performs a Fourier transform on the LFHF data to transform the waveform of the LFHF data into a power spectrum that is obtained by decomposing the waveform into frequency components.

[0062] The image conversion unit 25 converts the power spectrum converted by the spectrum conversion unit 24 into a first image.

[0063] FIG. 2 is a diagram showing an example of LFHF data and a first image in which the LFHF data is visualized in the first embodiment.

[0064] In the LFHF data shown in Fig. 2, the horizontal axis represents time and the vertical axis represents the LFHF value. In the first image shown in Fig. 2, the horizontal axis represents time, the vertical axis represents frequency, and color represents the magnitude of the LFHF value. Note that color is expressed in color.

[0065] The image conversion unit 25 generates a first image based on continuous power spectra contained in the LFHF data for a predetermined period. The image conversion unit 25 generates the first image by chronologically arranging the power spectra converted by the spectrum conversion unit 24. The first image represents the frequency components of the LFHF data in the power spectrum and the temporal changes in the LFHF values ​​using multiple colors.

[0066] The state determination unit 26 determines the state of the user from the first image generated by the image generation unit 22 using a learning model that has learned the relationship between the second image, generated from the person's biometric information, in a format that allows the biometric information to be concealed, and the person's state. The state determination unit 26 determines whether the user's state is a tense state or a relaxed state. A tense state can also be said to be a stress state in which the user is under stress.

[0067] Whether the user is in a tense state or a relaxed state can be determined based on the tension level output from the learning model. The tension level indicates the degree to which a person is tense. Here, the state determination unit 26 inputs a first image into the learning model and determines whether the tension level output from the learning model is equal to or greater than a threshold. The tension level is expressed as a numerical value between 1.00 and 0. The threshold is, for example, 0.50. If the user is in a tense state, the tension level output from the learning model approaches 1.00, and if the user is in a relaxed state, the tension level output from the learning model approaches 0. If the tension level output from the learning model is equal to or greater than the threshold, the state determination unit 26 determines that the user is in a tense state. Furthermore, if the tension level output from the learning model is less than the threshold, the state determination unit 26 determines that the user is in a relaxed state.

[0068] The learning model is created by machine learning using, as training data, a tension state image obtained by visualizing first LFHF data based on a power spectrum obtained by frequency characteristic analysis of the first LFHF data of a person in a tension state, and a relaxed state image obtained by visualizing second LFHF data based on a power spectrum obtained by frequency characteristic analysis of the second LFHF data of a person in a relaxed state. A label indicating that the person's state is tension is assigned to the tension state image, and a label indicating that the person's state is relaxed is assigned to the relaxed state image. Machine learning of the learning model is performed so that when a tension state image is input, the learning model outputs a tension level value of "1" indicating that the person's state is tension, and when a relaxed state image is input, the learning model outputs a tension level value of "0" indicating that the person's state is relaxed. A plurality of second images and a plurality of third images are used to train the learning model.

[0069] The LFHF processing unit 23 may extract 30 pieces of heartbeat interval variability time series data every minute from the 30-minute heartbeat interval variability time series data and calculate 30 pieces of LFHF data based on the extracted 30 pieces of heartbeat interval variability time series data. The spectrum conversion unit 24 may convert the 30 pieces of LFHF data into 30 power spectra. The image conversion unit 25 may convert the 30 power spectra converted by the spectrum conversion unit 24 into 30 first images. The state determination unit 26 may input each of the 30 first images into a learning model and determine whether the average of the 30 tension levels output from the learning model is equal to or greater than a threshold. Although the LFHF processing unit 23 extracts heartbeat interval variability time series data every minute from the 30-minute heartbeat interval variability time series data, it may also extract heartbeat interval variability time series data every 30 seconds from the 30-minute heartbeat interval variability time series data. The LFHF processing unit 23 may extract heart rate interval variability time series data for each predetermined period from the heart rate interval variability time series data for the predetermined period.

[0070] The content storage unit 27 stores a plurality of video contents in advance, including relaxation video contents having a relaxing effect, sleep video contents having a sleep-inducing effect, and awakening video contents having an awakening effect.

[0071] The content playback system 10 guides the user, who is a passenger of the vehicle, into a state corresponding to a plurality of phases. The plurality of phases can include a relax phase, a sleep phase following the relax phase, and a wakefulness phase following the sleep phase. The relax phase is a phase that guides the user from a state of tension to a relaxed state. The sleep phase is a phase that guides the user into a sleep state. The wakefulness phase is a phase that guides the user into a wakefulness state. By performing the relax phase and the sleep phase in this order, the user can get a good night's sleep. Furthermore, by performing the sleep phase and the wakefulness phase in this order, the user can be woken up in a refreshed state. This allows the user to be active after arriving at their destination.

[0072] In the relaxation phase, relaxation video content is played to guide the user from a tense state to a relaxed state, in the sleep phase, sleep video content is played to guide the user from a relaxed state to a sleeping state, and in the wake phase, wake video content is played to guide the user from a sleeping state to a waking state. Note that the relaxation video content is an example of the first video content, and the sleep video content is an example of the second video content.

[0073] The content selection unit 28 selects video content corresponding to the user's state determined by the state determination unit 26 from among the plurality of video contents stored in the content storage unit 27. When the state determination unit 26 determines that the user's state is a tense state, the content selection unit 28 selects relaxation video content for guiding the user from the tense state to a relaxed state. Furthermore, when the state determination unit 26 determines that the user's state is a relaxed state, the content selection unit 28 selects sleep video content for guiding the user from the relaxed state to a sleep state and wakefulness video content for guiding the user from the sleep state to a wakefulness state.

[0074] If the state determination unit 26 determines that the user is in a tense state, the content selection unit 28 may select a plurality of relaxation video contents.

[0075] The playback time of one video content is, for example, 10 to 15 minutes. If the playback time of one video content is short, there is a risk that the user's heart rate will not be affected even if the user watches that video content. Therefore, it is preferable that the playback time of one video content is set to a length that will affect the user's heart rate. If the playback time of one video content is short, the content selection unit 28 may select multiple relaxing video contents. For example, if the playback time of one video content is five minutes, the content selection unit 28 may select two to three video contents.

[0076] The content playback unit 29 plays back content according to the user's status. The content playback unit 29 plays back the video content selected by the content selection unit 28. The content playback unit 29 outputs the played back video content to the display 3.

[0077] When the state determination unit 26 determines that the user's state is a tense state, the content playback unit 29 plays back the relaxation video content selected by the content selection unit 28. When the state determination unit 26 determines that the user's state is a relaxed state, the content playback unit 29 sequentially plays back the sleep video content and the wakefulness video content selected by the content selection unit 28.

[0078] Furthermore, when the state determination unit 26 determines that the user is in a relaxed state, the content selection unit 28 may select the relaxation video content, the sleep video content, and the wakefulness video content. In this case, the content playback unit 29 may sequentially play back the relaxation video content, the sleep video content, and the wakefulness video content selected by the content selection unit 28.

[0079] Furthermore, when the state determination unit 26 determines that the user's state is a relaxed state, the content selection unit 28 may select only the awakening video content without selecting the sleep video content. In this case, the content playback unit 29 may play only the awakening video content selected by the content selection unit 28. For example, if the estimated arrival time of the mobile object is known, the video content may not be played from the current time until a predetermined time before the estimated arrival time, and the awakening video content may be played from a predetermined time before the estimated arrival time. Furthermore, the user's awakening time may be set by the user. The content playback unit 29 may play the awakening video content at the set awakening time.

[0080] Furthermore, after playback of one piece of relaxation video content has ended, the bioinformation acquisition unit 21 may acquire heart rate variability time series data for the period during which the piece of relaxation video content was being played. The image generation unit 22 may then calculate LFHF data based on the acquired heart rate variability time series data and convert the power spectrum obtained by frequency characteristic analysis of the LFHF data into a first image. The state determination unit 26 may then use a learning model to determine whether the user's state is tense or relaxed from the first image generated by the image generation unit 22. If the user's state is again determined to be tense, the content selection unit 28 may select a piece of relaxation video content different from the previously selected piece of relaxation video content. The acquisition of heart rate variability time series data, generation of the first image, determination of the user's state, selection of video content, and playback of video content may be repeated until the user's state is determined to be relaxed.

[0081] Furthermore, after the playback of one piece of relaxation video content has finished, if the state determination unit 26 again determines that the user's state is tense, the state determination unit 26 may compare the tension level currently output from the learning model with the tension level previously output from the learning model. If the tension level currently output is equal to or greater than the tension level previously output, the content selection unit 28 may select relaxation video content that has a greater relaxing effect than the previously selected relaxation video content.

[0082] Furthermore, the playback times for the sleep video content and the wake-up video content may be determined in advance, or if the estimated arrival time of the mobile object is known, they may be determined based on the estimated arrival time. For example, the sleep video content may be played from the current time until a predetermined time before the estimated arrival time, and the wake-up video content may be played from a predetermined time before the estimated arrival time. The content selection unit 28 may select the sleep video content and the wake-up video content based on the playback times of the sleep video content and the wake-up video content, respectively. The user's wake-up time may also be set by the user. For example, the sleep video content may be played from the current time until the wake-up time, and the wake-up video content may be played from the wake-up time.

[0083] The display 3 is, for example, a liquid crystal display, and is installed inside the vehicle. The display 3 is installed in a position visible to the user, who is a passenger of the vehicle. The display 3 displays video content played by the content playback unit 29. If the user's state is determined to be a tense state, the display 3 displays relaxation video content. The user is induced from the tense state to a relaxed state by watching the relaxation video content displayed on the display 3. Furthermore, if the user's state is determined to be a relaxed state, the display 3 sequentially displays sleep video content and wakefulness video content. The user is induced from the relaxed state to a sleep state by watching the sleep video content displayed on the display 3. Furthermore, the user is induced from the sleep state to a wakefulness state by watching the wakefulness video content displayed on the display 3.

[0084] The display 3 may be virtual reality (VR) goggles, augmented reality (AR) goggles, a smartphone display, or a tablet computer display. The content playback system 10 may also include a projector that projects video content onto a screen instead of the display 3. For example, the projector may be installed on the ceiling or a seat inside the vehicle.

[0085] Next, a content playback process performed by the information processing device 2 according to the first embodiment of the present disclosure will be described.

[0086] FIG. 3 is a flowchart illustrating a content playback process performed by the information processing device 2 according to the first embodiment of the present disclosure.

[0087] First, in step S1, the biometric information acquisition unit 21 acquires biometric information from the current time up to a predetermined time period ago. The biometric information is the user's heart rate data.

[0088] Next, in step S2, the LFHF processing unit 23 calculates LFHF data based on the heart rate data. The LFHF processing unit 23 converts the heart rate data into heart rate variability time series data. The LFHF processing unit 23 then calculates time series data of the LFHF value, which is a stress index, as LFHF data based on the heart rate variability time series data.

[0089] Next, in step S3, the spectrum transform unit 24 transforms the LFHF data in the time domain into a power spectrum in the frequency domain.

[0090] Next, in step S4, the image conversion unit 25 converts the power spectrum converted by the spectrum conversion unit 24 into a first image.

[0091] Next, in step S5, the state determination unit 26 inputs the first image into the trained learning model, determines whether the tension level output from the learning model is equal to or greater than a threshold, and determines whether the user's state is tense based on the determination result. The tension level is expressed as a value between 1.00 and 0. When the user's state is tense, the tension level approaches 1.00, and when the user's state is relaxed, the tension level approaches 0. The first threshold is, for example, 0.50. When the tension level output from the learning model is equal to or greater than the threshold, the state determination unit 26 determines that the user's state is tense. Furthermore, when the tension level output from the learning model is less than the threshold, the state determination unit 26 determines that the user's state is not tense, i.e., that the user's state is relaxed.

[0092] Here, if it is determined that the user's state is tense (YES in step S5), in step S6, the content selection unit 28 selects a relaxing video content from the multiple video contents stored in the content storage unit 27 to guide the user from a tense state to a relaxed state.

[0093] Next, in step S7, the content playback unit 29 plays back the relaxing video content selected by the content selection unit 28. The content playback unit 29 outputs the played back relaxing video content to the display 3. The display 3 displays the relaxing video content played back by the content playback unit 29.

[0094] Next, in step S8, the content playback unit 29 determines whether the playback of the relaxing video content has ended. If it is determined that the playback of the relaxing video content has not ended (NO in step S8), the determination process of step S8 is performed. The determination process of step S8 is repeated until the playback of the relaxing video content has ended.

[0095] On the other hand, if it is determined that the playback of the relaxation video content has ended (YES in step S8), in step S9, the biometric information acquisition unit 21 acquires biometric information during the period when the relaxation video content was being played. Thereafter, the process returns to step S2, and steps S2 to S9 are repeated until the user's state changes from a tense state to a relaxed state.

[0096] On the other hand, if it is determined that the user's state is not tense, i.e., if it is determined that the user's state is relaxed (NO in step S5), in step S10, the content selection unit 28 selects, from the multiple video contents stored in the content storage unit 27, sleep video content for guiding the user from a relaxed state to a sleep state and wakefulness video content for guiding the user from a sleep state to an wakefulness state.

[0097] Next, in step S11, the content playback unit 29 sequentially plays back the sleep video content and wakefulness video content selected by the content selection unit 28. The content playback unit 29 sequentially outputs the played back sleep video content and wakefulness video content to the display 3. The display 3 sequentially displays the sleep video content and wakefulness video content played back by the content playback unit 29.

[0098] According to the first embodiment, a first image in a format that allows the biometric information to be concealed is generated from the acquired biometric information, and a user's condition is determined from the first image using a learning model. Here, since the first image is in a format that allows the biometric information, which is personal information, to be concealed, it is difficult to identify the original biometric information from the first image. Therefore, it is possible to accurately determine the user's condition while protecting privacy, and to perform appropriate intervention control according to the determined user's condition.

[0099] In the first embodiment, video content is played back according to the state of the user, but the present disclosure is not particularly limited to this, and audio content may be played back according to the state of the user.

[0100] In this case, the content reproduction system 10 may further include a speaker. The content storage unit 27 may store a plurality of sound contents in advance. The plurality of sound contents may include relaxing sound content having a relaxing effect, sleep sound content having a sleep-inducing effect, and awakening sound content having an awakening effect. The content selection unit 28 may select sound content corresponding to the user's state determined by the state determination unit 26 from the plurality of sound contents stored in the content storage unit 27. When the state determination unit 26 determines that the user's state is a tense state, the content selection unit 28 may select relaxing sound content for guiding the user from the tense state to a relaxed state. Furthermore, when the state determination unit 26 determines that the user's state is a relaxed state, the content selection unit 28 may select sleep sound content for guiding the user from the relaxed state to a sleep state and awakening sound content for guiding the user from the sleep state to a wakeful state.

[0101] Then, the content playback unit 29 may play back audio content according to the state of the user. The content playback unit 29 may play back audio content selected by the content selection unit 28. The content playback unit 29 may output the played audio content to a speaker. When the state determination unit 26 determines that the user's state is a tense state, the content playback unit 29 may play back relaxing audio content selected by the content selection unit 28. Furthermore, when the state determination unit 26 determines that the user's state is a relaxed state, the content playback unit 29 may sequentially play back sleeping audio content and awake audio content selected by the content selection unit 28.

[0102] The content reproducing unit 29 may reproduce both the video content and the audio content, or may reproduce either the video content or the audio content.

[0103] The content playback system 10 may further include a lighting device, and the information processing device 2 may further include a lighting control unit that controls the illuminance of the lighting device. The lighting control unit may control the dimming pattern of the lighting device depending on the user's state determined by the state determination unit 26. When the state determination unit 26 determines that the user's state is a tense state, the lighting control unit may control the lighting device to illuminate with a dimming pattern that induces the user from a tense state to a relaxed state. When the state determination unit 26 determines that the user's state is a relaxed state and sleep video content is being played, the lighting control unit may control the lighting device to illuminate with a dimming pattern that induces the user from a relaxed state to a sleeping state. When the state determination unit 26 determines that the user's state is a relaxed state and wakefulness video content is being played, the lighting control unit may control the lighting device to illuminate with a dimming pattern that induces the user from a sleeping state to a wakefulness state. Note that each dimming pattern is predetermined.

[0104] The content playback system 10 may further include a diffuser that emits a fragrance component, and the information processing device 2 may further include a fragrance control unit that controls the fragrance component emitted from the diffuser. The fragrance control unit may control the fragrance component emitted from the diffuser depending on the user's state determined by the state determination unit 26. When the state determination unit 26 determines that the user's state is tense, the fragrance control unit may control the diffuser to emit a fragrance component for guiding the user from a tense state to a relaxed state. When the state determination unit 26 determines that the user's state is relaxed and the awakening video content is being played, the fragrance control unit may control the diffuser to emit a fragrance component for guiding the user from a sleep state to an awakening state. Note that each fragrance component is predetermined.

[0105] Furthermore, in this embodiment 1, if the user's state is determined to be a relaxed state, sleep video content and wakefulness video content are played sequentially, but the present disclosure is not particularly limited to this, and in a modified example of this embodiment 1, if the user's state is determined to be a relaxed state, it may be determined whether the user's state is a sleeping state or not.

[0106] Fig. 4 is a first flowchart illustrating a content playback process by the information processing device 2 according to the modified example of the first embodiment of the present disclosure, and Fig. 5 is a second flowchart illustrating a content playback process by the information processing device 2 according to the modified example of the first embodiment of the present disclosure. Note that the modified example of the first embodiment will be described using the information processing device 2 according to the first embodiment.

[0107] The processes in steps S21 to S29 are the same as those in steps S1 to S9 shown in FIG. 3, and therefore will not be described here.

[0108] If it is determined that the user's state is not tense, i.e., if it is determined that the user's state is relaxed (NO in step S25), in step S30, the content selection unit 28 selects sleep video content from the multiple video contents stored in the content storage unit 27 to guide the user from the relaxed state to a sleep state.

[0109] Next, in step S31, content playback unit 29 plays the sleep video content selected by content selection unit 28. Content playback unit 29 outputs the played sleep video content to display 3. Display 3 displays the sleep video content played by content playback unit 29.

[0110] Next, in step S32, content playback unit 29 determines whether playback of the sleep video content has ended. If it is determined that playback of the sleep video content has not ended (NO in step S32), the determination process of step S32 is performed. The determination process of step S32 is repeated until playback of the sleep video content has ended.

[0111] On the other hand, if it is determined that the playback of the sleep video content has ended (YES in step S32), then in step S33, the biological information acquirer 21 acquires biological information during the period in which the sleep video content was being played.

[0112] The processing in steps S34 to S36 is the same as the processing in steps S2 to S4 shown in FIG. 3, and therefore a description thereof will be omitted.

[0113] Next, in step S37, the state determination unit 26 inputs the first image into the trained learning model, determines whether the tension level output from the learning model is equal to or less than a second threshold value that is lower than the first threshold value, and determines whether the user is in a sleeping state based on the determination result. The second threshold value is, for example, 0.20. If the tension level output from the learning model is equal to or less than the second threshold value, the state determination unit 26 determines that the user is in a sleeping state. Furthermore, if the tension level output from the learning model is greater than the second threshold value, the state determination unit 26 determines that the user is not in a sleeping state, i.e., the user is not yet asleep or is in a light sleep state.

[0114] If the level of tension decreases further in the relaxed state, it can be determined that the user is asleep or in a deep sleep state. Therefore, if the user is not asleep, sleep video content is played to guide the user from the relaxed state to the asleep state, and if the user's state has transitioned to the asleep state, wake-up video content is played to guide the user from the asleep state to the wake-up state.

[0115] If it is determined that the user is not in a sleeping state (NO in step S37), in step S38, the state determination unit 26 determines whether the tension level output from the learning model is equal to or greater than a first threshold, and determines whether the user is in a tense state based on the determination result. If the tension level output from the learning model is equal to or greater than the first threshold, the state determination unit 26 determines that the user is in a tense state. Furthermore, if the tension level output from the learning model is less than the first threshold, the state determination unit 26 determines that the user is not in a tense state, i.e., that the user is in a relaxed state.

[0116] If it is determined that the user is in a tense state (YES in step S38), the process returns to step S26.

[0117] In determining the level of tension in step S38, the state determination unit 26 may use a third threshold as a threshold other than the first threshold. The third threshold is preferably smaller than the first threshold and larger than the second threshold (first threshold > third threshold > second threshold). In addition, in step S26, the content selection unit 28 may select different video content depending on whether the process branches from step S25 or step S38.

[0118] The state of tension rather than sleep after playback of the sleep video content has ended is considered to be a temporary (sudden) wake-up state during the sleep phase, caused by, for example, jerking, changing body position, or sudden movement or vibration of the moving object. In other words, since the state of tension after playback of the sleep video content has ended is a state different from a normal state of tension, a third threshold different from the first threshold may be used, and relaxing video content different from normal relaxing video content may be played. Relaxing video content different from normal relaxing video content may be, for example, video that induces a relaxed state, but with quieter sound than normal, video with less movement than normal, or video that does not disturb sleep.

[0119] On the other hand, if it is determined that the user's state is not tense, i.e., if it is determined that the user's state is relaxed (NO in step S38), in step S39, the content selection unit 28 selects sleep video content from the multiple video contents stored in the content storage unit 27 to guide the user from the relaxed state to a sleep state.

[0120] Next, in step S40, content playback unit 29 plays back the sleep video content selected by content selection unit 28. Content playback unit 29 outputs the played back sleep video content to display 3. Display 3 displays the sleep video content played back by content playback unit 29. Thereafter, the process returns to step S32, and steps S32 to S40 are repeated until the user's state changes from the relaxed state to the sleeping state.

[0121] On the other hand, if it is determined that the user's state is a sleeping state (YES in step S37), in step S41, the content selection unit 28 selects, from among the multiple video contents stored in the content storage unit 27, an awakening video content for guiding the user from a sleeping state to an awakening state.

[0122] Next, in step S42, the content playback unit 29 plays the awakening video content selected by the content selection unit 28. The content playback unit 29 outputs the played awakening video content to the display 3. The display 3 displays the awakening video content played by the content playback unit 29.

[0123] The content playback unit 29 may determine whether the current time is the awakening time at which the user is to be awakened. The awakening time is a time a predetermined time before the scheduled arrival time of the mobile object. If it is determined that the current time is the awakening time, the content playback unit 29 may play the awakening video content selected by the content selection unit 28. On the other hand, if it is determined that the current time is not the awakening time, the content playback unit 29 may not play the awakening video content selected by the content selection unit 28. In this case, the content playback unit 29 may repeatedly perform the process of determining whether the current time is the awakening time until the current time becomes the awakening time.

[0124] In this way, in the modification of the first embodiment, if the user is in a relaxed state and also in a sleeping state, awakening video content for guiding the user from the sleeping state to the awake state is played, and if the user is in a relaxed state and also not in a sleeping state, sleeping video content for guiding the user from the relaxed state to the sleeping state is played. Therefore, it is possible to promote sleep in the user and to wake up the sleeping user comfortably.

[0125] Second Embodiment In the first embodiment, the biometric information is heartbeat data relating to heartbeats, but in the second embodiment, the biometric information is a facial image of the user.

[0126] FIG. 6 is a diagram showing the configuration of a content reproduction system 10A according to the second embodiment.

[0127] 6 includes a sensor 1A, an information processing device 2A, and a display 3. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and description thereof will be omitted.

[0128] The sensor 1A measures biometric information of a user. The biometric information includes a facial image of the user. The sensor 1A is, for example, a camera, and captures a facial image of the user for a predetermined period of time and outputs the facial image. The sensor 1A transmits the biometric information including the facial image to the information processing device 2A.

[0129] The sensor 1A is installed in a vehicle in which a user is riding. The sensor 1A is connected to the information processing device 2A by wire or wirelessly so that a captured facial image can be input to the information processing device 2A. The sensor 1A may be communicably connected to the information processing device 2A via a network. The sensor 1A may be a camera of a smartphone or a tablet computer.

[0130] When the sensor 1A receives a request to transmit biological information from the information processing device 2A, the sensor 1A may transmit the biological information to the information processing device 2A.

[0131] The information processing device 2A includes at least a computer system including, for example, a control program, a processing circuit such as a processor or logic circuit that executes the control program, and a recording device such as an internal memory or an accessible external memory that stores the control program. Note that the information processing device 2A may be realized, for example, by hardware implementation using the processing circuit, or by execution of a software program stored in the memory by the processing circuit or distributed from an external server, or by a combination of these hardware and software implementations.

[0132] The information processing device 2A is installed inside a vehicle. The information processing device 2A may be a server.

[0133] The information processing device 2A includes a biometric information acquisition unit 21A, an image generation unit 22A, a state determination unit 26A, a content storage unit 27, a content selection unit 28, and a content playback unit 29.

[0134] The biometric information acquisition unit 21A acquires biometric information of a user. The biometric information includes a facial image. The biometric information acquisition unit 21A receives the biometric information transmitted by the sensor 1A. The biometric information acquisition unit 21A synchronizes with an application of the sensor 1A and acquires the biometric information from the sensor 1A.

[0135] The biometric information acquisition unit 21A acquires current biometric information. The user may be a passenger of a moving body. If the user is a passenger of a moving body, the biometric information acquisition unit 21A acquires biometric information at the time the user boarded the moving body. When the power of the moving body is turned on, the biometric information acquisition unit 21A may transmit a request to send biometric information to the sensor 1A.

[0136] The image generation unit 22A generates a first image in a format that can conceal the biometric information from the biometric information acquired by the biometric information acquisition unit 21 A. The image generation unit 22A generates the first image by cutting out only some of the facial features from the face image.

[0137] The image generating unit 22A includes an image converting unit 25A.

[0138] The image conversion unit 25A converts the facial image into a first image consisting of only some of the facial features by cutting out only some of the facial features from the facial image.

[0139] FIG. 7 is a diagram showing an example of a face image and a first image obtained by cutting out only some of the face features from the face image in the second embodiment.

[0140] The image conversion unit 25A recognizes each facial feature from the facial image 201. For example, the image conversion unit 25A recognizes the eyebrows, eyes, and mouth from the facial image 201. The image conversion unit 25A then cuts out only the recognized eyebrows, eyes, and mouth from the facial image 201, and generates a first image 202 consisting only of the eyebrows, eyes, and mouth. Because the first image 202 consists of only some of the user's facial features, it is difficult to recognize the user's face, and the user's privacy can be protected.

[0141] The state determination unit 26A determines the user's state from the first image generated by the image generation unit 22A using a learning model that has learned the relationship between the second image, generated from the person's biometric information and in a format that allows the biometric information to be concealed, and the person's state. The state determination unit 26A determines whether the user's state is tense or relaxed. A tense state can also be said to be a stress state in which the user is under stress. The state determination unit 26A estimates the user's facial expression from the first image using facial expression sensing technology, and determines whether the user's state is tense or relaxed from the estimated user's facial expression.

[0142] Whether the user is in a tense state or a relaxed state can be determined from the level of tension output from the learning model. The level of tension indicates the degree to which a person is tense. When a first image is input, the learning model extracts multiple feature points of facial features from the first image. The learning model then calculates feature amounts for the extracted multiple feature points and calculates the level of tension from the calculated feature amounts. The feature amounts are, for example, the distance between each feature point. The distance between multiple feature points changes depending on a person's facial expression. The state determination unit 26A calculates the level of tension by using the distance between multiple feature points as the feature amount, and determines whether the user is in a tense state or a relaxed state based on the calculated level of tension.

[0143] Here, the state determination unit 26A inputs the first image into the learning model and determines whether the tension level output from the learning model is equal to or greater than a threshold. The tension level is expressed as a numerical value between 1.00 and 0. The threshold is, for example, 0.50. If the user is in a tense state, the tension level output from the learning model approaches 1.00, and if the user is in a relaxed state, the tension level output from the learning model approaches 0. If the tension level output from the learning model is equal to or greater than the threshold, the state determination unit 26A determines that the user is in a tense state. Furthermore, if the tension level output from the learning model is less than the threshold, the state determination unit 26A determines that the user is in a relaxed state.

[0144] The learning model is created by machine learning using, as training data, a tension state image obtained by cutting out only a portion of a facial feature from a first face image of a person in a tension state, and a relaxed state image obtained by cutting out only a portion of a facial feature from a second face image of a person in a relaxed state. A label indicating that the person's state is tension is assigned to the tension state image, and a label indicating that the person's state is relaxed is assigned to the relaxed state image. Machine learning of the learning model is performed so that when a tension state image is input, the learning model outputs a tension level value of "1" indicating that the person's state is tension, and when a relaxed state image is input, the learning model outputs a tension level value of "0" indicating that the person's state is relaxed. A plurality of second images and a plurality of third images are used to train the learning model.

[0145] The image conversion unit 25A may extract 30 facial images (still images) every minute from the facial motion image from the current time up to 30 minutes ago, and convert the extracted 30 facial images into 30 first images consisting of only some of the facial features. The state determination unit 26A may input each of the 30 first images into a learning model and determine whether the average of the 30 tension levels output from the learning model is equal to or greater than a threshold. While the image conversion unit 25A extracts a facial image every minute from the facial motion image over 30 minutes, it may also extract a facial image every 30 seconds from the 30 minutes. The image conversion unit 25A may extract a facial image every predetermined time from the facial motion image over a predetermined period.

[0146] The image conversion unit 25A may also generate the first image by converting a facial image into a binary image. The image conversion unit 25A may convert a color facial image into a first image consisting only of white and black. In this case, when the first image is input, the learning model may extract multiple feature points of facial features (e.g., eyebrows, eyes, and mouth) from the first image. The learning model may then calculate feature amounts of the extracted multiple feature points and calculate the level of tension from the calculated feature amounts. Because the first image is a binary image, it is difficult to recognize the user's face, and the user's privacy can be protected.

[0147] Furthermore, the image conversion unit 25A may convert the facial image into a binary image and generate the first image by processing to extract a plurality of feature points of the facial features.

[0148] FIG. 8 shows an example of a face image and a first image generated by converting the face image into a binary image and extracting multiple feature points of the face parts in variant example 1 of embodiment 2 of the present invention.

[0149] The image conversion unit 25A may convert the color facial image 201 into a first image 203 consisting only of black and white. In this case, the image conversion unit 25A may detect the contours of a person from the color facial image 201 and extract multiple feature points 204 of facial features (e.g., eyebrows, eyes, nose, mouth, and contour) from the first image 203. The image conversion unit 25A may then generate the first image 203 in which an area corresponding to the background is represented in black, an area corresponding to the person is represented in white, and the multiple feature points 204 are represented as dots. When the first image 203 is input, the learning model may calculate feature quantities of the multiple feature points 204 and calculate a level of tension from the calculated feature quantities. Because the first image 203 is a binary image and is composed of multiple feature points, it is difficult to recognize the user's face, thereby protecting the user's privacy.

[0150] The image conversion unit 25A may also generate the first image by reducing the resolution of the facial image. The image conversion unit 25A converts the facial image into a first image having a resolution lower than that of the facial image. The resolution of the first image is lower than that of the facial image, and is a resolution that allows facial feature points to be extracted but does not allow the user's face to be recognized. In this case, when the first image is input, the learning model may extract multiple feature points of facial features (e.g., eyebrows, eyes, and mouth) from the first image. The learning model may then calculate feature quantities of the extracted multiple feature points and calculate the level of tension from the calculated feature quantities. Because the resolution of the first image is lower than that of the original facial image, it is difficult to recognize the user's face, and the user's privacy can be protected.

[0151] Next, a content playback process performed by the information processing device 2A according to the second embodiment of the present disclosure will be described.

[0152] FIG. 9 is a flowchart illustrating a content playback process performed by information processing device 2A according to the second embodiment of the present disclosure.

[0153] First, in step S51, the biometric information acquisition unit 21A acquires current biometric information, which is a facial image of the user.

[0154] Next, in step S52, the image conversion unit 25A converts the face image into a first image consisting of only some of the facial features by cutting out only some of the facial features from the face image.

[0155] Next, in step S53, the state determination unit 26A inputs the first image into the trained learning model, determines whether the tension level output from the learning model is equal to or greater than a threshold, and determines whether the user is in a tense state based on the determination result. The tension level is expressed as a value between 1.00 and 0. When the user is in a tense state, the tension level approaches 1.00, and when the user is in a relaxed state, the tension level approaches 0. The first threshold is, for example, 0.50. When the tension level output from the learning model is equal to or greater than the threshold, the state determination unit 26A determines that the user is in a tense state. Furthermore, when the tension level output from the learning model is less than the threshold, the state determination unit 26A determines that the user is not in a tense state, i.e., that the user is in a relaxed state.

[0156] The processes in steps S54 to S56 are the same as those in steps S6 to S8 shown in FIG. 3, and therefore will not be described here.

[0157] If it is determined that the playback of the relaxation video content has ended (YES in step S56), in step S57, the biometric information acquisition unit 21A acquires current biometric information immediately after the playback of the relaxation video content has ended. The biometric information is a facial image. Then, the process returns to step S52, and steps S52 to S57 are repeated until the user's state changes from a tense state to a relaxed state.

[0158] In step S57, the biometric information acquisition unit 21A may acquire, as biometric information, facial motion images captured while the relaxation video content was being played. In this case, the image conversion unit 25A may extract multiple facial images (still images) every minute from the facial motion images captured while the relaxation video content was being played and convert the extracted multiple facial images into multiple first images consisting of only some of the facial features. The state determination unit 26A may input each of the multiple first images into a learning model and determine whether the average of the multiple tension levels output from the learning model is equal to or greater than a threshold.

[0159] The processing in steps S58 and S59 is the same as the processing in steps S10 and S11 shown in FIG. 3, and therefore a description thereof will be omitted.

[0160] According to the second embodiment, a first image in a format that allows biometric information to be concealed is generated from an acquired facial image, and a learning model is used to determine the user's state from the first image. Here, since the first image is in a format that allows the facial image, which is personal information, to be concealed, it is difficult to identify the original facial image from the first image. Therefore, it is possible to accurately determine the user's state while protecting privacy, and to perform appropriate intervention control according to the determined user's state.

[0161] In the second embodiment, sound content may be played back in accordance with the state of the user. In this case, the content playback unit 29 may play back both the video content and the sound content, or may play back either the video content or the sound content. In the second embodiment, lighting equipment may be controlled in accordance with the state of the user, or a diffuser may be controlled in accordance with the state of the user.

[0162] Furthermore, in this embodiment 2, if the user's state is determined to be a relaxed state, sleep video content and wakefulness video content are played sequentially, but the present disclosure is not particularly limited to this, and in variant 2 of this embodiment 2, similar to the variant of embodiment 1, if the user's state is determined to be a relaxed state, it may be determined whether the user's state is a sleeping state or not.

[0163] Fig. 10 is a first flowchart illustrating a content playback process by an information processing device 2A according to Modification 2 of Embodiment 2 of the present disclosure, and Fig. 11 is a second flowchart illustrating a content playback process by an information processing device 2A according to Modification 2 of Embodiment 2 of the present disclosure. Note that Modification 2 of Embodiment 2 will be described using the information processing device 2A of Embodiment 2.

[0164] The processing in steps S71 to S77 is the same as the processing in steps S51 to S57 shown in FIG. 9, and therefore a description thereof will be omitted.

[0165] If it is determined that the user's state is not tense, i.e., if it is determined that the user's state is relaxed (NO in step S73), in step S78, the content selection unit 28 selects sleep video content from the multiple video contents stored in the content storage unit 27 to guide the user from the relaxed state to a sleep state.

[0166] Next, in step S79, content playback unit 29 plays the sleep video content selected by content selection unit 28. Content playback unit 29 outputs the played sleep video content to display 3. Display 3 displays the sleep video content played by content playback unit 29.

[0167] Next, in step S80, content playback unit 29 determines whether playback of the sleep video content has ended. If it is determined that playback of the sleep video content has not ended (NO in step S80), the determination process of step S80 is performed. The determination process of step S80 is repeated until playback of the sleep video content has ended.

[0168] On the other hand, if it is determined that the playback of the sleep video content has ended (YES in step S80), in step S81, the biometric information acquirer 21A acquires current biometric information immediately after the playback of the sleep video content has ended. The biometric information is a face image.

[0169] The process of step S82 is the same as the process of step S52 shown in FIG. 9, and therefore a description thereof will be omitted.

[0170] Next, in step S83, the state determination unit 26A inputs the first image into the trained learning model, determines whether the tension level output from the learning model is equal to or less than a second threshold value that is lower than the first threshold value, and determines whether the user is in a sleeping state based on the determination result. The second threshold value is, for example, 0.20. If the tension level output from the learning model is equal to or less than the second threshold value, the state determination unit 26A determines that the user is in a sleeping state. Furthermore, if the tension level output from the learning model is greater than the second threshold value, the state determination unit 26A determines that the user is not in a sleeping state, i.e., the user is not yet asleep or is in a light sleep state.

[0171] In step S81, biometric information acquisition unit 21A may acquire, as biometric information, facial motion images captured while the sleep video content was being played. In this case, image conversion unit 25A may extract multiple facial images (still images) every minute from the facial motion images captured while the sleep video content was being played and convert the extracted multiple facial images into multiple first images consisting of only some of the facial features. State determination unit 26A may input each of the multiple first images into a learning model and determine whether the average of multiple tension levels output from the learning model is equal to or greater than a threshold.

[0172] If it is determined that the user is not in a sleeping state (NO in step S83), in step S84, the state determination unit 26A determines whether the tension level output from the learning model is equal to or greater than a first threshold, and determines whether the user is in a tense state based on the determination result. If the tension level output from the learning model is equal to or greater than the first threshold, the state determination unit 26A determines that the user is in a tense state. Furthermore, if the tension level output from the learning model is less than the first threshold, the state determination unit 26A determines that the user is not in a tense state, i.e., that the user is in a relaxed state.

[0173] If it is determined that the user is in a tense state (YES in step S84), the process returns to step S74.

[0174] On the other hand, if it is determined that the user's state is not tense, i.e., if it is determined that the user's state is relaxed (NO in step S84), in step S85, the content selection unit 28 selects sleep video content from the multiple video contents stored in the content storage unit 27 to guide the user from the relaxed state to a sleep state.

[0175] Next, in step S86, content playback unit 29 plays back the sleep video content selected by content selection unit 28. Content playback unit 29 outputs the played back sleep video content to display 3. Display 3 displays the sleep video content played back by content playback unit 29. Thereafter, the process returns to step S80, and steps S80 to S86 are repeated until the user's state changes from the relaxed state to the sleeping state.

[0176] On the other hand, if it is determined that the user's state is a sleeping state (YES in step S83), in step S87, the content selection unit 28 selects, from among the multiple video contents stored in the content storage unit 27, an awakening video content for guiding the user from a sleeping state to an awakening state.

[0177] The processing in steps S87 and S88 is the same as the processing in steps S41 and S42 shown in FIG. 5, and therefore a description thereof will be omitted.

[0178] In this way, in the second modification of the second embodiment, if the user is in a relaxed state and also in a sleeping state, awakening video content for guiding the user from the sleeping state to the awake state is played, and if the user is in a relaxed state and also not in a sleeping state, sleeping video content for guiding the user from the relaxed state to the sleeping state is played. Therefore, it is possible to promote sleep in the user and to wake up a sleeping user comfortably.

[0179] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for that component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. Furthermore, the program may be executed by another independent computer system by recording the program on a recording medium and transferring it, or by transferring the program via a network.

[0180] Some or all of the functions of the device according to the embodiment of the present disclosure are typically realized as an LSI (Large Scale Integration), which is an integrated circuit. These may be individually integrated into a single chip, or some or all of them may be integrated into a single chip. Furthermore, the integrated circuit is not limited to an LSI, and may be realized using a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array), which can be programmed after LSI manufacturing, or a reconfigurable processor, which can reconfigure the connections and settings of circuit cells within an LSI.

[0181] Furthermore, some or all of the functions of the device according to the embodiment of the present disclosure may be realized by a processor such as a CPU executing a program.

[0182] Furthermore, all the numbers used above are merely examples to specifically explain the present disclosure, and the present disclosure is not limited to the numbers used as examples.

[0183] The order in which the steps are executed in the above flowchart is merely an example for specifically explaining the present disclosure, and other orders may be used as long as similar effects are obtained. Also, some of the steps may be executed simultaneously (in parallel) with other steps.

[0184] The technology disclosed herein is useful as a technology for determining a user's condition from the user's biometric information, as it can accurately determine the user's condition while protecting privacy and perform appropriate intervention control according to the determined user's condition.

Claims

1. An information processing apparatus comprising: an acquisition unit that acquires biometric information of a user; a generation unit that generates a first image in a form in which the biometric information can be anonymized from the acquired biometric information; and a determination unit that determines the state of the user from the generated first image using a learning model that has learned the relationship between a second image in a form in which the biometric information can be anonymized generated from the biometric information of a person and the state of the person.

2. The biometric information includes heartbeat data related to the heartbeat. The generation unit calculates time-series data of an LFHF value, which is a stress index, as LFHF data based on the heartbeat data, and converts a power spectrum obtained by frequency characteristic analysis of the LFHF data into the first image. The information processing apparatus according to claim 1.

3. In the first image, the frequency component of the LFHF data in the power spectrum and the temporal change of the LFHF value are represented by a plurality of colors. The information processing apparatus according to claim 2.

4. The biometric information includes a face image of the user. The generation unit generates the first image by performing a process of cutting out only a part of the face parts from the face image, a process of converting the face image into a binary image, a process of converting the face image into a binary image and extracting a plurality of feature points of the face parts, or a process of reducing the resolution of the face image. The information processing apparatus according to claim 1.

5. The determination unit determines whether the state of the user is a tense state or a relaxed state. The information processing apparatus according to any one of claims 1 to 4.

6. The determination unit determines whether the state of the user is a sleeping state. The information processing apparatus according to any one of claims 1 to 4.

7. The acquisition unit periodically acquires the biometric information. The information processing apparatus according to any one of claims 1 to 4.

8. The acquisition unit acquires the biometric information from a predetermined period before the current time. The information processing apparatus according to any one of claims 1 to 4.

9. The user is a passenger of a moving body. The information processing apparatus according to any one of claims 1 to 4.

10. Further, the information processing apparatus according to any one of claims 1 to 4 includes a reproduction unit that reproduces content according to the state of the user.

11. The information processing apparatus according to claim 10, wherein when it is determined that the state of the user is a tense state, the playback unit plays back first video content for inducing the user from the tense state to a relaxed state.

12. The information processing apparatus according to claim 10, wherein when it is determined that the state of the user is not a tense state, the playback unit plays back second video content for inducing the user from the relaxed state to a sleeping state.

13. An information processing method executed by a computer, the method including: obtaining biometric information of a user; generating a first image in a form capable of concealing the biometric information from the obtained biometric information; and determining the state of the user from the generated first image using a learning model that has learned a relationship between a second image in a form capable of concealing the biometric information generated from the biometric information of a person and the state of the person.

14. An information processing program for causing a computer to function so as to obtain biometric information of a user, generate a first image in a form capable of concealing the biometric information from the obtained biometric information, and determine the state of the user from the generated first image using a learning model that has learned a relationship between a second image in a form capable of concealing the biometric information generated from the biometric information of a person and the state of the person.

Images