Information processing method, information processing device, and moving body

Abstract

To provide an information processing method capable of obtaining accurate biometric data from a user.SOLUTION: The information processing method includes the following steps: acquiring biometric data corresponding to changes in the intensity of reflected light from a user during a first period during which the video content is being played; acquiring the feature amount that changes over time in the first period, which is a color feature of the video content; and correcting the biometric data based on the features.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to techniques for acquiring biometric data from images. [Background technology]

[0002] Conventionally, there are known techniques for acquiring biometric data based on changes in the intensity of light reflected from a user. For example, Patent Document 1 discloses a technique for measuring a pulse wave by photographing a user to acquire successive color images, extracting the luminance values ​​of the G (green) component of the pixels in the successive color images, and calculating the difference in luminance values ​​between the successive color images. Furthermore, Patent Document 2 discloses a technique for capturing an image of a living body to be measured for a pulse wave, calculating the difference between the R signal and the G signal extracted from the image data of the captured image, and generating a correction signal that reduces common-mode noise between the two signals. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2016 / 006027 [Patent Document 2] Japanese Patent Publication No. 2020-157102 Summary of the Invention [Problem to be solved by the invention]

[0004] If the biometric data is acquired while the user is watching video content, there is a possibility that the acquired biometric data will be inaccurate. Specifically, when the user is watching video content using a display member such as a display, light from the display (hereinafter referred to as display light) irradiates the user's skin, and the intensity of the light reflected from the user fluctuates due to the influence of this display light. As a result, there is a possibility that the acquired biometric data will be inaccurate.

[0005] An object of the present disclosure is to provide a technology that can acquire accurate biometric data. [Means for solving the problem]

[0006] An information processing method in one aspect of the present disclosure is an information processing method in a computer, which includes acquiring biometric data corresponding to changes in the intensity of reflected light from a user during a first period in which video content is being played, acquiring features related to the color of the video content, the features changing over time during the first period, and correcting the biometric data based on the features. [Effects of the Invention]

[0007] According to this configuration, accurate biometric data can be obtained from the user. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an overview of an information processing system according to a first embodiment. [Figure 2] 1 is a block diagram showing an electrical configuration of an information processing system according to a first embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a data configuration of feature amount data. [Figure 4] 4 is a flowchart showing a flow of processing in the information processing system according to the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a data configuration of user image data. [Figure 6] FIG. 10 is a diagram showing how the scene of video content transitions from scene 1 to scene 2. [Figure 7] FIG. 1 is a diagram illustrating a use case of an information processing system. [Figure 8] FIG. 10 is a block diagram showing the electrical configuration of an information processing system according to a second embodiment. [Figure 9] 10 is a flowchart showing a flow of processing in an information processing system according to a second embodiment. [Figure 10] FIG. 2 is a diagram illustrating an example of a data configuration of feature amount data. [Figure 11] 10A and 10B are diagrams for explaining the processing of a user position detection unit; [Figure 12] 10 is a diagram showing how the scene of the video content transitions from scene 1 to the first, second, and third examples of scene 2. FIG. [Figure 13] 10 is a table summarizing the average G level, the G center of gravity position, the user's face position, and the irradiation coefficient in the first to third examples. [Figure 14] 10 is a graph showing the correlation between the position of the user's face, the G center of gravity, and the irradiation coefficient. [Figure 15] FIG. 10 is a diagram showing how the scene of the video content transitions from scene 1 to a first example of scene 2. [Figure 16] FIG. 10 is a diagram showing how the scene of the video content transitions from scene 1 to a second example of scene 2. [Figure 17] FIG. 10 is a diagram showing how the scene of the video content transitions from scene 1 to scene 2, a third example. [Figure 18] FIG. 4 is a diagram illustrating an example of a data configuration of pulse signal data. [Figure 19] FIG. 11 is a diagram illustrating an example of a configuration of an information processing system according to a third embodiment. [Figure 20] FIG. 10 is a diagram illustrating an example of a configuration of an information processing system according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] (Findings underlying this disclosure) As mentioned above, if the display light from the display illuminates the user's skin, inaccurate biometric data may be obtained.

[0010] For example, consider a case where a pulse wave is detected based on the brightness value of the G component (hereinafter referred to as the G component value) of an image captured by photographing a user. When video content depicting a forest or the like is projected onto a display, the user's skin is illuminated green by the display light. If an image is captured at this time, the skin illuminated green will appear in the image. If the G component value is extracted from this image, an excessive G component value may be extracted, which could result in an inaccurate pulse wave being obtained.

[0011] As a result of detailed investigation into the above-mentioned problem, the inventors have discovered that biometric data acquired from a user can be corrected by treating information about the color of video content as a feature, and have arrived at the following aspects of the present disclosure.

[0012] (1) An information processing method in one aspect of the present disclosure is an information processing method in a computer, which includes acquiring biometric data corresponding to changes in the intensity of reflected light from a user during a first period in which video content is being played, acquiring features related to the color of the video content, the features changing over time during the first period, and correcting the biometric data based on the features.

[0013] According to this configuration, the biometric data is corrected based on the color feature of the video content, which can suppress the influence of the color of the video content and reduce noise contained in the biometric data, thereby making it possible to obtain accurate biometric data from a user watching the video content.

[0014] (2) In the information processing method described in (1) above, acquiring the feature may include acquiring feature data in which the feature is associated with time, and the biometric data may be corrected based on the feature data.

[0015] According to this configuration, feature data in which time and feature are associated with each other is acquired, which makes it easy to identify what feature values ​​were included in video content played at what time, thereby enabling efficient correction of biometric data.

[0016] (3) In the information processing method described in (1) or (2) above, the biometric data is data corresponding to the change in intensity of the reflected light of a specific wavelength corresponding to at least one wavelength band of blue, green, and red, and the biometric data may be corrected based on the feature of the video content of the color corresponding to the specific wavelength.

[0017] According to this configuration, the biometric data is data corresponding to a specific wavelength corresponding to at least one wavelength band of blue, green, or red, and the biometric data is corrected by a feature amount related to the color corresponding to the specific wavelength, thereby enabling more accurate biometric data to be acquired compared to a case where the biometric data is corrected by a feature amount related to a color not corresponding to the specific wavelength.

[0018] (4) In the information processing method described in any one of (1) to (3) above, correcting the biometric data may be performed during the first period.

[0019] In this configuration, the biometric data is corrected during the first period during which the video content is being played, so the corrected biometric data can be obtained more quickly than when the biometric data is corrected after the video content has finished playing. The above configuration is particularly useful in cases where the user is short on time and the corrected biometric data needs to be presented quickly.

[0020] (5) In the information processing method described in any one of (1) to (4) above, correcting the biometric data may be performed in a second period after the first period.

[0021] When biometric data is corrected during playback of video content, playback of the video content, acquisition of feature amounts, and correction of the biometric data must be performed simultaneously, which may result in an excessive processing load on the computer. In contrast, with the above configuration, the biometric data is corrected after playback of the video content has finished, thereby preventing the processing load from becoming excessive.

[0022] (6) In the information processing method described in any one of (1) to (5) above, the method may further include acquiring position information indicating a relative positional relationship between the position of a display element onto which the video content is projected and the position of a user viewing the video content, and correcting the biometric data may include correcting the biometric data based on the position information and the feature.

[0023] The effect of the display light on the biometric data may vary depending on the relative positional relationship between the position of the display member and the position of the user's face. Therefore, if the biometric data is corrected without considering the relative positional relationship, inappropriate correction may be performed. In contrast, with the above configuration, the biometric data is corrected using position information indicating the relative positional relationship between the position of the display member and the position of the user's face. Therefore, more accurate biometric data can be obtained compared to when the biometric data is corrected without considering the relative positional relationship.

[0024] (7) In the information processing method described in (6) above, the display member may be arranged inside the vehicle, and the position information may be generated based on a camera arranged inside the vehicle or a sensor for detecting seating provided on a seat of the vehicle.

[0025] According to this configuration, the display member is disposed inside the vehicle, and the user's location information is acquired by a camera or a sensor disposed inside the vehicle, so that biometric data can be suitably acquired from the user watching video content inside the vehicle.

[0026] (8) In the information processing method described in (6) above, the position information may include information indicating a relative positional relationship between the position of the center of gravity of the color in the video content and the position of the user watching the video content.

[0027] According to this configuration, biometric data is corrected based on the relative positional relationship between the position of the center of gravity of the color in the video content and the position of the user viewing the video content, thereby making it possible to obtain more accurate biometric data.

[0028] (9) In the information processing method described in any one of (1) to (8) above, the biological data may indicate a pulse wave of the user.

[0029] According to this configuration, an accurate pulse wave can be obtained.

[0030] (10) In the information processing method described in any one of (1) to (9) above, the method may further include acquiring display light data generated by having a photodetector detect return light from a second position different from a first position where the user is present and where display light resulting from the video content is irradiated, and the biometric data may be corrected based on the feature estimated based on the display light data.

[0031] According to this configuration, the feature amount is estimated based on the display light data acquired in real time as the video content is projected onto the display device, which eliminates the need to prepare data that associates time with the feature amount.

[0032] (11) In the information processing method described in (10) above, the photodetector may be a camera, and the camera may detect the returned light and the reflected light and output an image signal including pixels corresponding to the returned light and pixels corresponding to the reflected light.

[0033] According to this configuration, the camera detects reflected light and returned light, so it is possible to eliminate the need to provide a separate device for detecting returned light.

[0034] (12) In the information processing method described in (11) above, the biometric data may be generated based on pixels corresponding to the reflected light, and the display light data may be generated based on pixels corresponding to the returned light.

[0035] According to this configuration, the biometric data and the display light data can be easily generated.

[0036] (13) An information processing method according to another aspect of the present disclosure includes acquiring biometric data corresponding to a change in intensity of reflected light from a user during a first period in which illumination light whose color changes over time is irradiated, and correcting the biometric data based on the change over time in a characteristic amount related to the color of the illumination light.

[0037] According to this configuration, the biometric data is corrected based on the color feature of the illumination light, which can suppress the influence of the illumination light and reduce noise contained in the biometric data, thereby making it possible to obtain accurate biometric data from a user illuminated by the illumination light.

[0038] (14) In yet another aspect of the present disclosure, an information processing device includes a first acquisition unit that acquires biometric data corresponding to changes in the intensity of reflected light from a user during a first period in which video content is being played, a second acquisition unit that acquires features related to the color of the video content, the features changing over time during the first period, and a correction unit that corrects the biometric data based on the features.

[0039] According to this configuration, the same effects as those of the information processing method described above in (1) can be obtained.

[0040] (15) In yet another aspect of the present disclosure, a moving body includes a display element that displays video content played during a first period, a photodetector that detects reflected light from a user, and a processing device, wherein the processing device generates biometric data corresponding to changes in the intensity of the reflected light, acquires a feature related to the color of the video content, the feature changing over time during the first period, and corrects the biometric data based on the feature.

[0041] According to this configuration, the same effects as those of the information processing method described above in (1) can be obtained.

[0042] (16) In yet another aspect of the present disclosure, an information processing program causes a computer to function as an information processing device, and causes the computer to execute the following process: acquire biometric data corresponding to changes in the intensity of reflected light from a user during a first period in which video content is being played; acquire features related to the color of the video content, which features change over time during the first period; and correct the biometric data based on the features.

[0043] According to this configuration, the same effects as those of the information processing method described above in (1) can be obtained.

[0044] The present disclosure can also be realized as a control system operated by such an information processing program. Needless to say, such a computer program can be distributed on a non-transitory computer-readable recording medium such as a CD-ROM or via a communication network such as the Internet.

[0045] Note that the embodiments described below each illustrate a specific example of the present disclosure. The numerical values, shapes, components, steps, step orders, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in an independent claim that represents a top-level concept are described as optional components. Furthermore, in all embodiments, the respective contents can be combined.

[0046] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0047] (Embodiment 1) Fig. 1 is a diagram showing an overview of an information processing system 1 to which the present disclosure is applied. As shown in Fig. 1, the information processing system 1 includes a display 2 (an example of a display member), a camera 3, and a lighting device 4. In the information processing system 1, a user watches video content projected on the display 2. At this time, the user is illuminated by light (hereinafter referred to as display light L) emitted from the display 2. The camera 3 captures an image of the user watching the video content.

[0048] The display 2 is configured by a device including pixels. The display 2 is configured by, for example, an organic EL panel or a liquid crystal panel. However, the configuration of the display 2 is not limited to this, and the display 2 may be configured by, for example, a screen. In this case, the information processing system 1 may further include a projector for projecting video content onto the screen.

[0049] Video content is projected onto the display 2. The video content is, for example, content such as environmental video, and stimulates the user visually and aurally to encourage relaxation, naps, or refreshment. In the first embodiment, video content in which 60 images are displayed continuously per second is projected onto the display 2. That is, video content of 60 fps (frames per second) is projected onto the display 2. Hereinafter, the images (frames) that make up the video content are referred to as content images.

[0050] The camera 3 acquires a color image with a resolution of, for example, 1920 pixels horizontally by 1080 pixels vertically. Each pixel has a luminance value (pixel value) that indicates the intensity of the R (red), G (green), and B (blue) components. Hereinafter, the luminance value of the R component will be referred to as the R component value, the luminance value of the G component will be referred to as the G component value, and the luminance value of the B component will be referred to as the B component value. These component values ​​are expressed, for example, in 256 gradations, with 0 being the lowest and 255 being the highest.

[0051] The camera 3 according to the first embodiment can continuously generate 60 images per second and acquire them as a video. That is, the camera 3 can acquire a video at 60 fps. Hereinafter, a video acquired by the camera 3 capturing an image of a user will be referred to as a user video, and an image (frame) constituting the user video will be referred to as a user image. The camera 3 captures, for example, the face of the user.

[0052] Fig. 2 is a block diagram showing the electrical configuration of the information processing system 1. As shown in Fig. 2, the information processing system 1 further includes a display control device 20 and a camera control device 50. In the first embodiment, the display 2, the display control device 20, the camera 3, and the camera control device 50 are connected to each other so as to be able to communicate with each other via a network (not shown). The network is, for example, a local area network.

[0053] The display control device 20 includes a storage unit 30 and a control circuit 40. The storage unit 30 is configured as a non-volatile rewritable storage device such as a hard disk drive or a solid state drive.

[0054] The storage unit 30 stores feature amount data 31 that associates feature amounts of video content with time. The feature amounts are related to the color of the video content and change as time passes during a first period in which the video content is played. In the first embodiment, the feature amount is the average value of at least one of the B component value, G component value, and R component value of each pixel in a content image that constitutes the video content. In particular, in the first embodiment, the feature amount is the average value of the G component values ​​of all pixels in the content image (hereinafter referred to as the average G level). The average G level is expressed in 256 gradations, with 0 being the lowest and 255 being the highest.

[0055] FIG. 3 is a diagram showing an example of the data configuration of the feature data 31. As shown in FIG. 3, the feature data 31 has a day column C1, a time column C2, a frame number column C3, and a feature column C4. The day column C1 stores the date on which video content was projected onto the display 2. If video content is not being projected, the day column C1 does not need to store a date. In this case, the date when the video content was projected may be recorded. The time column C2 stores the time when a content image constituting the video content was projected onto the display 2. If a content image is not being projected onto the display 2, the time column C2 does not need to store a time; the time may be recorded as soon as the content image is projected. The frame number column C3 stores a frame number assigned to each content image to distinguish them from one another. The feature column C4 stores the average G level for each content image.

[0056] Returning to FIG. 2, the storage unit 30 stores data for projecting video content onto the display 2 (hereinafter referred to as video content data 32).

[0057] The control circuit 40 of the display control device 20 includes a processor (CPU) (not shown) and a memory 70 (not shown) such as a ROM and a RAM. The processor includes a data control unit 41. The data control unit 41 may be realized by the processor executing a program stored in the memory 70, or may be realized by a dedicated electric circuit.

[0058] The data control unit 41 reads out the video content data 32 stored in the storage unit 30 and transmits this data to the display 2. Upon receiving the video content data 32, the display 2 projects the video content and presents the video content to the user.

[0059] Furthermore, in response to a data request from the camera control device 50, the data control unit 41 transmits the feature amount data 31 stored in the storage unit 30 to the camera control device 50.

[0060] The camera control device 50 is configured as a computer including a processor 60 (CPU) and a memory 70 such as a ROM and a RAM. The processor 60 includes an image processing unit 61 (an example of a first acquisition unit) and a data correction unit 62 (an example of a second acquisition unit and correction unit). The image processing unit 61 and the data correction unit 62 may be realized by the processor 60 executing a program stored in the memory 70, or may be realized by a dedicated electric circuit.

[0061] The image processing unit 61 acquires biometric data corresponding to changes in the intensity of reflected light from the user during a first period in which the video content is being played. The biometric data is data corresponding to changes in the intensity of reflected light of a specific wavelength corresponding to at least one wavelength band of blue, green, and red. The biometric data according to the first embodiment indicates a pulse wave.

[0062] Reflected light refers to light that falls into at least one of the following categories (1) to (3): (1) Light originating from a light source included in the information processing system 1; (2) Light originating from ambient light, including sunlight and light from indoor lighting, i.e., light that is returned when ambient light is irradiated onto the user; and (3) Scattered light that is scattered inside the skin, in addition to light reflected from the surface of the user's skin.

[0063] The image processing unit 61 according to the first embodiment acquires biometric data (data indicating a pulse wave) based on a user image. Note that techniques for acquiring data indicating a user's pulse wave based on a user image are disclosed in the above-mentioned Patent Documents 1 and 2, among others. As an example, the image processing unit 61 according to the first embodiment detects a pulse wave by extracting a G component value from the user image. In detail, the image processing unit 61 extracts the G component value from each of a plurality of consecutive user images in a user video and acquires this as a pulse signal. Next, the image processing unit 61 calculates the difference in the pulse signal between consecutive user images (the difference in the G component value). Then, the image processing unit 61 accumulates the calculated differences in the pulse signal between consecutive user images. This makes it possible to acquire data in which the amount of change in the pulse signal is sampled at a sampling period corresponding to the frame rate of the user video, i.e., biometric data indicating a pulse wave.

[0064] The storage unit 30 may store user images and user videos captured by a camera during playback of video content, or pulse signals analyzed from these. The image processing unit 61 may then extract these data stored in the storage unit 30 for the biometric data correction process described below. Such actions also fall under the category of "acquiring biometric data." In other words, "acquiring biometric data" includes not only acquiring a pulse signal in real time, but also reading out data previously stored in the storage unit 30, etc.

[0065] The data correcting unit 62 acquires the feature amount and corrects the biometric data based on the feature amount. In detail, the data correcting unit 62 acquires the feature amount data 31 (FIG. 3) from the display control device 20 and corrects the biometric data based on the average G level stored in column C4 of the feature amount of the feature amount data 31. In the first embodiment, the biometric data correction process is performed during a first period in which the video content is being played.

[0066] The processing of the information processing system 1 configured as above will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of processing of the information processing system 1 according to the first embodiment. The processing shown in Fig. 4 starts, for example, when the display control device 20 and the camera control device 50 receive a control signal indicating an instruction to start measuring biological data. This control signal is input, for example, by a user operating a user terminal (such as a smartphone or laptop computer). Alternatively, the control signal may be transmitted to the display control device 20 and the camera control device 50 from an external server (not shown).

[0067] In step S1, the control software for starting up the camera control device 50 is started.

[0068] In step S101, control software for starting up the display control device 20 is started.

[0069] In step S2, the image processing unit 61 transmits a signal to the camera 3 to command it to capture a user image. Upon receiving this signal, the camera 3 captures an image of the user's face and captures the user image. The camera 3 then transmits the user image to the image processing unit 61.

[0070] Furthermore, in step S2, the image processing unit 61 transmits a user image database 71 relating to the user images to the image processing unit 61. FIG. 5 is a diagram showing an example of the data configuration of the user image database 71. As shown in FIG. 5, the user image database 71 has a day column C11, a time column C12, and a user image number column C13. The day column C11 stores the date on which the user image was acquired. The time column C12 stores the time on which the user image was acquired. The user image number column C13 stores a user image number assigned to each user image in order to distinguish the user images from one another.

[0071] In step S102, the data control unit 41 acquires the feature amount data 31 (FIG. 3) from the storage unit 30. As described above, the feature amount data 31 records the average G level of each content image.

[0072] In step S3, the image processing unit 61 extracts a pulse signal. First, the image processing unit 61 performs a predetermined image recognition process, such as template matching, on the user image to identify the user's face region. Then, the image processing unit 61 extracts G component values ​​from each pixel included in the face region and calculates a representative value. The representative value is represented, for example, by the average of the G component values. The image processing unit 61 treats this representative value as a pulse signal. Note that the representative value may be represented by the median or mode of the G component values.

[0073] In step S103, the data control unit 41 acquires the video content data 32 from the storage unit 30.

[0074] In step S 4 , the data correcting unit 62 transmits a signal requesting the feature amount data 31 to the data control unit 41 .

[0075] In step S104, the data control unit 41 checks whether a signal requesting feature amount data 31 has been received from the data correction unit 62, and proceeds to step S105. If the signal has been received (YES in step S105), the data control unit 41 proceeds to step S106. On the other hand, if the signal has not been received (NO in step S105), the data control unit 41 proceeds to step S107.

[0076] In step S 106 , the data control unit 41 transmits the feature amount data 31 to the data correction unit 62 .

[0077] In step S107, the data control unit 41 transmits the video content data 32 to the display 2. Upon receiving the video content data 32, the display 2 presents the video content to the user.

[0078] In step S5, the data correction unit 62 checks whether the feature amount data 31 has been received, and proceeds to step S6. If the feature amount data 31 has been received (YES in step S6), the data correction unit 62 proceeds to step S7. On the other hand, if the feature amount data 31 has not been received (NO in step S6), the data correction unit 62 proceeds to step S8.

[0079] In step S7, the data correcting unit 62 performs a process of correcting the pulse signal. The process of the data correcting unit 62 will be described in detail below with reference to FIG.

[0080] 6 is a diagram showing how a scene transitions from scene 1 to scene 2 in video content. As shown in FIG. 6, in scene 1, a black content image (hereinafter referred to as first content image IM1) is projected onto the display 2, and in scene 2, a content image showing a forest (hereinafter referred to as second content image IM2) is projected onto the display 2. In the following, it is assumed that the average G level of the first content image IM1 is 0, and the average G level of the second content image IM2 is 100.

[0081] Graph GR1 in FIG. 6 shows the change in the average G level when the scene of the video content transitions from scene 1 to scene 2. Point TM on graph GR1 represents the point in time when the scene transitions from scene 1 to scene 2. As shown in graph GR1, the average G level increases by 100 as the scene changes from scene 1 to scene 2.

[0082] Graph GR2 in FIG. 6 shows the user's pulse wave when the scene transitioned from scene 1 to scene 2. Point TM on graph GR2 represents the time point when the scene transitioned from scene 1 to scene 2. Also, average line m1 on graph GR2 represents the average magnitude of the pulse signal in scene 1, and average line m2 represents the average magnitude of the pulse signal in scene 2. As shown in graph GR2, the difference value between average line m2 and average line m1 is 100. This indicates that the pulse signal, i.e., the representative value of the G component value extracted from the user image, increased by an average of 100 as the scene transitioned from scene 1 to scene 2. This sudden increase in the pulse signal is thought to be caused by the user being illuminated by display light L (FIG. 1). Specifically, this is thought to be caused by the pulse signal being acquired based on the G component value of the user image when a content image with an average G level at least greater than 0, such as second content image IM2, is projected onto display 2 and the user's skin is illuminated by display light L containing green wavelengths.

[0083] Therefore, the data correcting unit 62 according to the first embodiment performs a correction process to subtract the average G level from the pulse signal. Specifically, the process subtracts the average G component values ​​(average G level) of all pixels constituting the content image displayed on the display 2 from the average G component values ​​(pulse signal) of pixels constituting the area in the user image where the user's face appears. Graph GR3 in FIG. 6 shows a pulse wave based on the pulse signal from which the average G level has been subtracted. That is, graph GR3 shows a pulse wave obtained by performing the above-described integration process of the difference between the pulse signals based on the pulse signal from which the fluctuation component caused by the display light L being superimposed on the user image has been canceled. In this way, the data correcting unit 62 performs a process to subtract the average G level from the pulse signal, thereby enabling an accurate pulse wave to be obtained.

[0084] Returning to FIG. 4, in step S7, the data correction unit 62 first identifies a content image (hereinafter referred to as a corresponding image) that was projected on the display 2 when the user image in step S2 was captured by the camera 3. Specifically, the corresponding image is identified by referring to the user image database 71 (FIG. 5) and the feature amount data 31 (FIG. 3). As an example, assume that a user image having information stored in row R1 in FIG. 5 was captured in step S2. In this case, the data correction unit 62 references the time column C12 in the user image database 71 to determine that the user image was captured by the camera 3 at 14:18:19.339. Next, the data correction unit 62 references the time column C2 in the feature amount data 31 in FIG. 3 to identify a content image that was displayed on the display 2 at the time closest to the time the target image was captured. In the example shown in FIG. 3, the content image with frame number 3 was projected on the display 2 at 14:18:19.339. In this case, the data correction unit 62 identifies the content image with frame number 3 as the corresponding image. Next, having identified the corresponding image, the data correction unit 62 references column C4 of the feature amount in FIG. 3 and determines that the average G level of the corresponding image is 120. Then, the data correction unit 62 performs a process of subtracting the average G level of the corresponding image from the pulse signal calculated by the image processing unit 61 in step S3.

[0085] In order to perform the process of identifying corresponding images in step S7, the data correction unit 62 may perform synchronization processing. In the synchronization processing, the user image number assigned to each user image is associated with the frame number assigned to each content image. This synchronization processing may be performed, for example, based on the time when each user image was captured by the camera 3 and the time when each content image was played back. Performing this synchronization processing makes it easier to identify corresponding images.

[0086] In step S8, the data correction unit 62 outputs the corrected pulse signal. For example, the data correction unit 62 outputs the corrected pulse signal to the memory 70 of the camera control device 50. Note that if the camera control device 50 is connected to an external server via a network such as the Internet, the data correction unit 62 may output the corrected pulse signal to the external server.

[0087] If it is determined in step S6 that the feature amount data 31 has not been received (NO in step S6) and the process proceeds to step S8, the data correction unit 62 assumes that there is no fluctuation in the pulse signal due to the display light L and does not perform the correction process for the pulse signal. In other words, the process of step S7 is not executed. Then, in step S8, the pulse signal that has not been subjected to the correction process is output.

[0088] In step S9, the data correction unit 62 determines whether or not a control signal indicating an instruction to end measurement of biological data has been received. If the instruction has been received (YES in step S9), the process ends. On the other hand, if the instruction has not been received (NO in step S9), the process returns to step S2.

[0089] In step S108, the data control unit 41 of the display control device 20 determines whether or not a control signal indicating an instruction to end measurement of biological data has been received. If the instruction has been received (YES in step S108), the process ends. On the other hand, if the instruction has not been received (NO in step S108), the process returns to step S102.

[0090] In the first embodiment, the processes of steps S2 to S8 and steps S102 to S107 are executed at a sampling period based on the frame rate of the user video. That is, in the first embodiment, the pulse signal is corrected every time a user image is acquired. Then, the difference between the pulse signals (pulse signals after the correction process) between consecutive user images is calculated, and this difference is integrated to acquire biometric data indicating an accurate pulse wave.

[0091] According to the information processing system 1 described above, the biometric data is corrected based on the feature amount related to the color of the video content, so that the influence of the display light L can be suppressed and the noise contained in the biometric data can be reduced. As a result, accurate biometric data can be acquired from the user watching the video content.

[0092] 7 is a diagram showing a use case of the information processing system 1. As shown in FIG. 7, the information processing system 1 according to the present disclosure can be used in an environment in which a display 2 is installed between a front seat 11 and a rear seat 12 of an automobile 10 (an example of a vehicle), a user (not shown) is seated in the rear seat 12, and video content is projected onto the display 2 to be presented to the user. In this environment, the intensity of light reflected from the user's skin surface may fluctuate due to the influence of display light L from a display placed in front of the user, but accurate biometric data can be obtained by utilizing the present disclosure.

[0093] Examples of the automobile 10 include a passenger car, a taxi, and a bus. The front seats 11 are seats including the driver's seat of the automobile 10. The rear seats 12 are seats located behind the front seats 11 (to the right in FIG. 7) and do not include the driver's seat. In this use case, the camera 3 is installed, for example, on the ceiling of the automobile 10. The display control device 20 is configured, for example, by an ECU that is standard equipment on the automobile 10. The camera control device 50 is placed, for example, inside a console box of the automobile 10. The camera control device 50 may be configured by the above-mentioned ECU. In other words, the ECU may perform the role of both the display control device 20 and the camera control device 50.

[0094] 7, the information processing system 1 may be used when acquiring biometric data from a user watching video content while on a train, bullet train, ship, airplane, etc. In addition, the information processing system 1 may be used when acquiring biometric data from a user watching video content in their own room, a hospital room, etc.

[0095] The present disclosure can employ the following modifications.

[0096] (Variation 1-1) In the first embodiment, an example has been described in which the image processing unit 61 extracts a representative value of the G component value from the user image and uses this representative value as the pulse signal. However, the image processing unit 61 may extract a representative value of the R component value from the user image and use this representative value as the pulse signal. In this case, the average R level of each content image may be stored in the feature column C4 of the feature data 31 shown in FIG. 3. The average R level refers to the average value of the R component values ​​of all pixels in the content image.

[0097] Alternatively, the image processing unit 61 may extract a representative value of the B component value from the user image and use this representative value as the pulse signal. In this case, the feature column C4 may store the average B level of each content image. The average B level refers to the average value of the B component values ​​of all pixels in the content image.

[0098] (Variation 1-2) The image processing unit 61 may extract two types of component values ​​from the RGB component values ​​and acquire a pulse signal based on the difference between these component values. For example, the image processing unit 61 may calculate the difference between the R component value and the G component value and acquire a pulse signal in which common-mode noise between the two components has been reduced.

[0099] The feature column C4 of the feature data 31 according to this modification only needs to store the average G level of each content image and the average R level of each content image.

[0100] In this modification, the image processing unit 61 extracts a representative value of the R component value and a representative value of the G component value from the user image. Then, the data correction unit 62 identifies the content image (corresponding image) that was projected on the display 2 when the user image was acquired. Next, the data correction unit 62 references the feature amount data 31 and acquires the average G level and average R level of the corresponding image. Next, the data correction unit 62 executes a process of subtracting the average R level from the representative value of the R component value and a process of subtracting the average G level from the representative value of the G component value. Then, the data correction unit 62 calculates the difference between the representative value of the R component value from which the average R level has been subtracted and the representative value of the G component value from which the average G level has been subtracted, and acquires a pulse signal.

[0101] According to the above configuration, a process is performed in which the average R level is subtracted from the representative value of the R component value, and a process is performed in which the average G level is subtracted from the representative value of the G component value, thereby making it possible to obtain G component values ​​and R component values ​​in which the influence of the display light L is suppressed. Then, by calculating the difference between these G component values ​​and R component values ​​and obtaining a pulse signal in which common-mode noise is reduced, it is possible to obtain a more accurate pulse signal.

[0102] (Variation 1-3) In the first embodiment, an example has been described in which the camera 3 captures an image of the user's face, but there are no particular limitations on the part of the body that the camera 3 captures, as long as it is an area where the user's skin is exposed and is irradiated with the display light L. For example, the camera 3 may capture an image of the user's fingers, the back of the hand, the neck, etc.

[0103] (Variation 1-4) In the first embodiment, an example has been described in which the camera 3 acquires user video at 60 fps and video content is projected on the display 2 at 60 fps. However, the frame rate of the user video and the frame rate of the video content can be changed as appropriate. For example, the camera 3 may acquire user video at 30 fps and video content may be projected on the display 2 at 60 fps. Alternatively, the camera 3 may acquire user video at 60 fps and video content may be projected on the display 2 at 24 fps. In either case, the processes of steps S2 to S8 and steps S102 to S107 may be performed at a sampling period based on the frame rate of the user video.

[0104] (Variation 1-5) If it is determined in step S6 of FIG. 4 that the feature data 31 has not been received, the data correction unit 62 may estimate a feature (e.g., an average G level). For example, assume that the processes of steps S2 to S9 are repeated multiple times, and the data correction unit 62 requests feature data for each of the nth content image, the n+1th content image, and the n+2th content image. Furthermore, assume that only the feature data for the n+1th content image has not been received. In this case, the data correction unit 62 may estimate the feature data of the n+1th content image based on the feature data of the nth content image (e.g., an average G level) and the feature data of the n+2th content image. In other words, the feature data may be complemented based on the feature data of the preceding and following content images. The data correction unit 62 may then correct the biometric data based on the estimated feature data.

[0105] (Variation 1-6) The biometric data correction process may be executed in a second period after the first period. That is, the biometric data correction process may be executed after the playback of the video content is completed.

[0106] (Embodiment 2) An information processing system 1A according to a second embodiment will be described. In the first embodiment, an example has been described in which an average G level based on all pixels of a content image is used as a feature. However, depending on the position of the user's face, the amount of display light L actually irradiated on the user may vary, which may result in an error in pulse correction. Therefore, the information processing system 1A according to the second embodiment distinguishes the content image projected on the display 2 according to its display position. Then, an illumination coefficient is calculated based on the display position and the position of the user's face, and the average G level is corrected by the illumination coefficient, thereby dealing with fluctuations in the amount of display light L. Note that in the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and description thereof will be omitted.

[0107] Fig. 8 is a block diagram showing the electrical configuration of an information processing system 1A according to the second embodiment. As shown in Fig. 8, the processor 60 of the camera control device 50 according to the second embodiment further includes a user position detection unit 63. The processor 60 also includes a data correction unit 62A. The display 2 according to the second embodiment has a resolution of 1920 x 1080 pixels and employs Lambertian light distribution as a light distribution model. In the second embodiment, the camera 3 captures an image of the user's face.

[0108] The user position detection unit 63 acquires position information indicating the relative positional relationship between the position of the display 2 and the position of the user viewing the video content. Specifically, the user position detection unit 63 recognizes the user's face based on the user image acquired by the camera 3, and maps the position of the user's face into a three-dimensional coordinate system based on the resolution of the display 2. The three-dimensional coordinate system based on the resolution of the display 2 includes an X axis corresponding to the horizontal width of 1920 pixels of the display 2, a Y axis corresponding to the vertical width of 1080 pixels, and a Z axis perpendicular to both the X axis and the Y axis. In this coordinate system, the origin (0,0,0) is set at, for example, the lower left corner of the display 2. The coordinates of the position of the user's face in this coordinate system correspond to the above-mentioned position information.

[0109] The information processing system 1A may further include a sensor for acquiring user position information. For example, a seat pressure sensor or a temperature sensor may be provided on the seat where the user sits. In this case, the seat where the user sits may be identified using these sensors, and the coordinates associated with the seat may be acquired as position information. In this way, the position of the user's face can be identified with higher accuracy.

[0110] 7, the distance from the display 2 to the rear seat 12 is constant, and therefore, the Z-axis coordinate of the user's face may be determined based on this distance. In this way, the processing load on the user position detection unit 63 is reduced compared to when the user position detection unit 63 determines the Z-axis coordinate of the user's face each time based on the user image.

[0111] The data correcting unit 62A according to the second embodiment calculates an irradiation coefficient based on the position of the user's face and the G center of gravity of the content image, which will be described later. The data correcting unit 62A also calculates a correction value by multiplying the average G level of the content image by the irradiation coefficient. The data correcting unit 62A then corrects the pulse signal by subtracting the correction value from the pulse signal. Details of the processing by the data correcting unit 62A will be described later.

[0112] The processing of the information processing system 1A configured as above will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the flow of processing of the information processing system 1A according to embodiment 2. The processing shown in Fig. 9 starts, for example, when the display control device 20 and the camera control device 50 receive a control signal indicating an instruction to start measuring biological data.

[0113] The processes shown in steps S21, S22, and S201 are similar to the processes related to steps S1, S2, and S101 in FIG. 4, and therefore will not be described further.

[0114] In step S202, the data control unit 41 of the display control device 20 acquires feature amount data 31A from the memory 70. FIG. 10 is a diagram illustrating an example of the data configuration of feature amount data 31A according to the second embodiment. As illustrated in FIG. 10, the feature amount data 31A includes a G centroid column C25 in addition to a day column C21 to a feature amount column C24, which store information similar to the day column C1 to the feature amount column C4 described in the first embodiment. The G centroid (an example of a color centroid in video content) refers to the pixel with the highest G level in the content image. As illustrated in FIG. 10, the G centroid column C25 stores the coordinates of the G centroid in the above-mentioned three-dimensional coordinate system. Note that the Z-axis coordinates of the G centroids are all 0, and therefore are not illustrated. Although not illustrated in detail, if the display 2 includes multiple regions, the G centroid column C25 may store the coordinates of the G centroid of each region.

[0115] The processes of steps S203 to S208 in Fig. 9 are similar to the processes of steps S103 to S108 in Fig. 4, and therefore their explanations will be omitted. Also, the processes of steps S22 and S23 are similar to the processes of steps S2 and S3 in Fig. 4, and therefore their explanations will be omitted.

[0116] In step S24, the user position detection unit 63 detects the position of the user's face based on the user image. Specifically, the user position detection unit 63 performs a predetermined image recognition process on the user image to recognize the user's face included in the user image. Then, the user position detection unit 63 maps a reference point P of the user's face onto the above-mentioned three-dimensional coordinate system. The reference point P refers to, for example, the user's between the eyebrows, nose, or top of the head. In this example, the reference point P is assumed to be mapped to coordinates (480, 540, 960) in the above-mentioned three-dimensional coordinate system.

[0117] Furthermore, in step S24, the user position detection unit 63 acquires position information indicating the relative positional relationship between the position of the G center of gravity and the position of the user viewing the video content. FIG. 11 is a diagram for explaining the processing of the user position detection unit 63 in step S24. As shown in FIG. 11, the display 2 includes a first area A and a second area B. In this example, the first area A is the left half of the display 2. The second area B is the right half of the display 2. The user position detection unit 63 first acquires the coordinates of a first center of gravity CE1, which is the G center of gravity of the first area A. The coordinates of the first center of gravity CE1 can be acquired, for example, by referring to column C25 of the G center of gravity in FIG. 10. Next, the user position detection unit 63 acquires the coordinates of a second center of gravity CE2, which is the G center of gravity of the second area B.

[0118] Then, the user position detection unit 63 calculates a first angle. The first angle is the angle formed by a first reference line L1 (FIG. 11) that is parallel to the Z-axis direction of the three-dimensional coordinate system and passes through the first center of gravity CE1, and a first virtual line V1 (FIG. 11) that connects the first center of gravity CE1 and a reference point P on the user's face. The first angle can be calculated mainly based on the X-coordinate and Z-coordinate of the reference point P on the user's face and the X-coordinate and Z-coordinate of the first center of gravity CE1. In the example shown in FIG. 11, the first angle is 0 degrees. The user position detection unit 63 also calculates a second angle. The second angle is the angle formed by a second reference line L2 that is parallel to the Z-axis direction and passes through the second center of gravity CE2, and a second virtual line V2 that connects the second center of gravity CE2 and the reference point P. The second angle can be calculated mainly based on the X-coordinate and Z-coordinate of the reference point P and the X-coordinate and Z-coordinate of the second center of gravity CE2. In the example shown in FIG. 11, the second angle is 45 degrees. These first and second angles correspond to position information indicating the relative positional relationship between the position of the G center of gravity and the position of the user viewing the video content. By acquiring the first and second angles, the approximate positional relationship between the position of the user's face and each area of ​​the display 2 is determined. That is, the user position detection unit 63 determines that the user's face is located in front of the first area A and at about 45 degrees with respect to the second area B. Note that the user position detection unit 63 determines that the user's face is located in front of the first area A when the first angle is, for example, 0 degrees ± 5 degrees. Furthermore, when the second angle is, for example, 45 degrees ± 5 degrees, it determines that the user's face is located at about a 45-degree angle with respect to the second area B.

[0119] Furthermore, when the display 2 is, for example, a vertically long display, the user position detection unit 63 may calculate a predetermined angle using the Y coordinate of the reference point P of the user's face and the Y coordinate of the G center of gravity of the first area A and the second area B, and grasp the positional relationship between the user and the display 2 based on this angle.

[0120] Furthermore, the user position detection unit 63 may acquire information indicating the relative positional relationship between the center of the first region A and the user's face, and the relative positional relationship between the center of the second region B and the user's face. The position of the center of the first region A can be found, for example, based on the X coordinates and Y coordinates of each of the pixels that make up the edge portions (e.g., four corners) of the first region A. The position of the center of the second region B can also be found using a similar method.

[0121] The processing from step S25 to step S27 is the same as the processing from step S4 to step S6 in FIG. 4, and therefore a description thereof will be omitted.

[0122] In step S28, the data corrector 62A calculates the irradiation coefficients. Hereinafter, the calculation method of the irradiation coefficients will be described with reference to first to third examples.

[0123] FIG. 12 is a diagram showing how a scene of a video content transitions from scene 1 to first, second, and third examples of scene 2. In the first example, in scene 2, a content image IM101 is projected onto the display 2, in which the average value of the G component values ​​of all pixels constituting the first region A (hereinafter referred to as the average G level of the first region A) is 255 and the average value of the G component values ​​of all pixels constituting the second region B (hereinafter referred to as the average G level of the second region B) is 0. Note that point CE in the figure indicates the G center of gravity of the content image IM101. The same applies to content images IM102 and IM103, which will be described later. In the second example, in scene 2, a content image IM102 in which the average G level of the first region A and the second region B is 127 is projected onto the display 2. In the third example, in scene 2, a content image IM103 in which the average G level of the first region A is 0 and the average G level of the second region B is 127 is projected onto the display 2. In the first to third examples, the position of the face of the user watching the video content is assumed to be constant. Specifically, as detected by the user position detection unit 63 in step S24, the user's face is assumed to be located in front of the first area A and at an angle of about 45 degrees to the second area B.

[0124] The data corrector 62A calculates an illumination coefficient based on the light distribution characteristics of the display 2, the relative positional relationship between the position of the user's face and the position of the G center of gravity, and the ratio to the maximum average G level in a predetermined area of ​​the display 2. The illumination coefficient is found by calculating the following formula (1).

[0125] C=g1×k1+g2×k2...Equation (1) In the above formula (1), C represents the illumination coefficient. g1 represents the ratio (%) of the average G level of the first region A to the highest average G level. k1 represents the ratio (decimal) of the amount of display light L irradiated from the first region A to the user. g2 represents the ratio (%) of the average G level of the second region B to the highest average G level. k2 represents the ratio (decimal) of the amount of display light L irradiated from the first region A to the user.

[0126] For example, in the first example, the average G level of the first region A is 255. As described above, the highest average G level is 255. Therefore, the ratio of the average G level of the first region A to the highest average G level is 100%. Therefore, g1 in the above formula (1) is 100. Furthermore, in the first example, the user's face is positioned directly in front of the first region A. In the display 2 that employs a Lambertian light distribution, 100% of light is irradiated in the front direction. Therefore, k1 in formula (1) is 1. Furthermore, in the first example, the average G level of the second region B is 0, and therefore the ratio of the average G level of the second region B to the highest average G level is 0%. Therefore, g2 in formula (1) is 0. Furthermore, as described above, the user's face is positioned at approximately 45 degrees to the second region B, and approximately 70% of light is irradiated in the 45-degree direction in the Lambertian light distribution. Therefore, k2 is 0.7. From the above, the data corrector 62A calculates "C=100×1+0×0.7" to obtain "C=100." That is, in the first example, the irradiation coefficient is 100.

[0127] Similarly, in the second example, the average G level of the first region A is 127, and the ratio of the average G level of the first region A to the maximum average G level is approximately 50%. Therefore, g1 in the above formula (1) is 50. Furthermore, the user's face is located directly in front of the first region A. Therefore, k1 in formula (1) is 1. Furthermore, in the second example, the average G level of the second region B is 127, and the ratio of the average G level of the second region B to the maximum average G level is approximately 50%. Therefore, g2 in formula (1) is 50. Furthermore, since the user's face is located at approximately 45 degrees with respect to the second region B, k2 in formula (1) is 0.7. From the above, the data correction unit 62A calculates "C = 50 × 1 + 50 × 0.7" to obtain "C = 85". In other words, in the second example, the illumination coefficient is 85.

[0128] Similarly, in the third example, the average G level of the first area A is 0, and the ratio of the average G level of the first area A to the maximum average G level is 0%. Therefore, g1 in the above formula (1) is 0. Furthermore, since the user's face is located directly in front of the first area A, k1 in formula (1) is 1. Furthermore, in the third example, the average G level of the second area B is 255, and the ratio of the average G level of the second area B to the maximum average G level is 100%. Therefore, g2 in formula (1) is 100. Furthermore, since the user's face is located at approximately 45 degrees with respect to the second area B, k2 in formula (1) is 0.7. From the above, the data correction unit 62A calculates "C = 0 × 1 + 100 × 0.7" to obtain "C = 70". In other words, in the third example, the illumination coefficient is 70.

[0129] If the display 2 employs a light distribution model other than the Lambertian light distribution, the data correction unit 62A may calculate the amount of display light L using a calculation method according to the model, and obtain the values ​​of k1 and k2 in the above formula (1).

[0130] Furthermore, although not shown, the display 2 may have a third region in addition to the first region A and the second region B. In this case, the term "g3+k3" is added to the above formula (1). g3 indicates the ratio (%) of the average G level of the third region to the highest average G level. k3 indicates the ratio (decimal) of the amount of display light L irradiated from the third region to the user. A similar method can be used when the display 2 has a fourth region and a fifth region.

[0131] Fig. 13 is a table summarizing the average G level, G center of gravity position, user's face position, and irradiation coefficient in Examples 1 to 3. Note that the average G level shown in Fig. 13 refers to the average G level based on all pixels of the content image displayed on display 2 in scene 2. Also, the G center of gravity position refers to the coordinates of the G center of gravity of the content image projected onto display 2 in scene 2. The user's face position refers to the coordinates of reference point P of the user's face in the above-mentioned three-dimensional coordinate system.

[0132] 13, in all of the first, second, and third examples, the average G level is 127. That is, in the first example, the average G level of the first area A is 255 and the average G level of the second area B is 0, but when the average G level is calculated based on the G levels of all pixels in the content image, the average G level becomes 127. Similarly, in the second and third examples, when the average G level is calculated based on all pixels in the content image, the average G level becomes 127.

[0133] 13, the coordinates of the G center of gravity in the first example are (480, 540), the coordinates of the G center of gravity in the second example are (960, 540), and the coordinates of the G center of gravity in the third example are (1440, 540). As described above, the coordinates of the G center of gravity are stored in the feature amount data 31A (FIG. 10). Note that the Z coordinates of the G centers of gravity are all 0, and therefore are not shown in the figure.

[0134] The coordinates of the reference point P of the user's face in the first to third examples are all (480, 540). That is, the user's face is located in front of the first area A and at an angle of about 45 degrees to the second area B. The Z coordinate of the reference point P is, for example, 960.

[0135] In the first example, the exposure coefficient is 100, in the second example, the exposure coefficient is 85, and in the third example, the exposure coefficient is 70.

[0136] FIG. 14 is a graph showing the correlation between the position of the user's face, the G center of gravity, and the illumination coefficient. The horizontal axis of FIG. 14 represents the X coordinate of the G center of gravity of the content image, and the vertical axis represents the magnitude of the illumination coefficient. Line S1 in the figure represents the correlation between the X coordinate of the G center of gravity and the magnitude of the illumination coefficient when the X coordinate of the user's face position is 480. Line S2 in the figure represents the correlation between the X coordinate of the G center of gravity and the magnitude of the illumination coefficient when the X coordinate of the user's face is 960. Line S3 in the figure represents the correlation between the X coordinate of the G center of gravity and the magnitude of the illumination coefficient when the X coordinate of the user's face is 1440. As shown by lines S1 to S3, the influence of light radiation from the image changes depending on the offset value between the X coordinate of the user's face and the X coordinate of the G center of gravity, so the illumination coefficient increases or decreases.

[0137] Although detailed illustration is omitted, for example, when the X coordinate of the user's face is located at 720, which is between 480 and 960, the magnitude of the illumination coefficient takes a value intermediate between the two. Therefore, when the X coordinate of the user's face is located at 720, a line is formed that transits midway between line S1 and line S2. A graph such as that shown in FIG. 14 is stored in memory 70 of, for example, camera control device 50.

[0138] 9, in step S29, the data correction unit 62A performs processing to correct the pulse signal. The processing of the data correction unit 62A in step S29 will be described below with reference to Figures 15 to 17. In the following description, for convenience, it is assumed that the average G level of the content image displayed in scene 2 in the first to third examples is 100.

[0139] FIG. 15 is a diagram illustrating a transition of a scene in a video content from scene 1 to the first example of scene 2. Graph GR4 in FIG. 15 illustrates the pulse wave of a user when the scene transitions from scene 1 to the first example of scene 2. That is, the graph illustrates the fluctuation of the pulse signal when the scene transitions from scene 1 to the first example of scene 2. The average line m3 in graph GR4 represents the average magnitude of the pulse signal in scene 1, and the average line m4 represents the average magnitude of the pulse signal in scene 2. As shown in graph GR4, the difference value between average lines m3 and m4 is 100. This indicates that the pulse signal increased by 100 on average as the scene transitioned from scene 1 to scene 2. Graph GR5 in FIG. 15 illustrates a pulse wave based on the pulse signal from which the average G level has been subtracted. Graph GR6 in FIG. 15 illustrates a pulse wave based on the pulse signal when a correction value obtained by multiplying the average G level by an irradiation coefficient is subtracted from the pulse signal.

[0140] FIG. 16 is a diagram showing how the scene of video content transitions from scene 1 to a second example of scene 2. Graph GR7 in FIG. 16 shows the pulse wave when the scene transitions from scene 1 to scene 2. Average line m5 in graph GR7 represents the average magnitude of the pulse signal in scene 1, and average line m6 represents the average magnitude of the pulse signal in scene 2. As shown in graph GR8, the difference value between average line m5 and average line m6 is 85. Graph GR8 in FIG. 16 also shows the pulse wave based on the pulse signal from which the average G level has been subtracted. Graph GR9 in FIG. 16 also shows the pulse wave based on the pulse signal when a correction value has been subtracted from the pulse signal.

[0141] FIG. 17 is a diagram showing how the scene of video content transitions from scene 1 to scene 2, a third example. Graph GR10 in FIG. 17 shows the pulse wave when the scene transitions from scene 1 to scene 2. Average line m7 in graph GR10 represents the average magnitude of the pulse signal in scene 1, and average line m8 represents the average magnitude of the pulse signal in scene 2. The difference value between average line m7 and average line m8 is 70. Graph GR11 in FIG. 17 shows the pulse wave based on the pulse signal from which the average G level has been subtracted. Graph GR12 in FIG. 17 shows the pulse wave based on the pulse signal when a process of subtracting a correction value from the pulse signal has been performed.

[0142] As shown in graph GR8 of FIG. 16 , when the average G level is subtracted from the pulse signal in the second example, the pulse signal in scene 2 decreases by 15 on average compared to the pulse signal in scene 1. This difference of 15 is thought to be an error resulting from subtracting the average G level of the content image (=100) from the pulse signal, which increased by 85 on average in scene 2. Furthermore, as shown in graph GR11 of FIG. 17 , when the average G level is subtracted from the pulse signal in the third example, the pulse signal in scene 2 decreases by 30 on average compared to the pulse signal in scene 1. As in the second example, this difference of 30 is thought to be an error resulting from subtracting the average G level of the content image (=100) from the pulse signal, which increased by 70 on average in scene 2. Thus, errors may occur when the average G level is uniformly subtracted from the pulse signal without considering that the amount of display light L actually irradiated varies depending on the position of the user's face.

[0143] Therefore, the data correcting unit 62A according to the second embodiment calculates a correction value by multiplying the average G level by an irradiation coefficient. Specifically, the correction value is calculated by multiplying the average G level by a value obtained by dividing the irradiation coefficient by 100. In the second example, since the irradiation coefficient is 85, the data correcting unit 62A multiplies the average G level (=100) by a value obtained by dividing the irradiation coefficient by 100 (=0.85) to calculate a correction value of 85. The data correcting unit 62A then performs a process of subtracting the correction value (=85) from the pulse signal. In this way, a pulse signal with reduced error can be acquired. Then, by performing the above correction process each time a pulse signal is acquired, a pulse wave with reduced error can be acquired, as shown in graph GR9 in FIG. 16.

[0144] Similarly, in the third example, the data corrector 62A multiplies the average G level (=100) by the irradiation coefficient divided by 100 (=0.7) to calculate a correction value of 70. Then, the data corrector 62A subtracts the correction value (=70) from the pulse signal. In this way, a pulse wave with reduced error can be obtained, as shown in graph GR12 in FIG. 17.

[0145] Similarly, in the first example, the data corrector 62A multiplies the average G level (=100) by the irradiation coefficient divided by 100 (=1) to calculate a correction value of 100. Then, the data corrector 62A performs a process of subtracting the correction value (=100) from the pulse signal. Note that in the first example, the correction value and the average G level are the same value, so similar pulse waves are acquired in graphs GR5 and GR6 in FIG. 15.

[0146] Furthermore, the data corrector 62A may further multiply the above-mentioned correction value by a set constant. Then, the correction value multiplied by the set constant may be subtracted from the pulse signal. The set constant may be determined taking into consideration the dynamic range of the pulse signal and the dynamic range of the average G level.

[0147] FIG. 18 is a diagram showing an example of the data configuration of the pulse signal data 72. The pulse signal data 72 is generated by the user position detection unit 63 detecting the position of the user's face, the image processing unit 61 acquiring the pulse signal before correction, and the data correction unit 62A calculating an irradiation coefficient and performing correction processing on the pulse signal to acquire the corrected pulse signal. As shown in FIG. 18, the pulse signal data 72 has a column C31 for day, a column C32 for time, a column C33 for user position, a column C34 for irradiation coefficient, a column C35 for the pulse signal before correction, and a column C36 for the pulse signal after correction. The column C31 for day stores the date on which the corrected pulse signal was acquired. The column C32 for time stores the time on which the corrected pulse signal was acquired. The column C33 for user position stores the coordinates of the position of the user's face when the corrected pulse signal was acquired. The column C34 for irradiation coefficient stores the irradiation coefficient calculated to acquire the corrected pulse signal. Column C35 of the pulse signal before correction stores the pulse signal acquired by the image processing unit 61 before correction processing by the data correction unit 62A. Column C36 of the pulse signal after correction stores the pulse signal after correction processing by the data correction unit 62A.

[0148] 9, in step S30, data correction unit 62A outputs a pulse signal. For example, data correction unit 62A outputs pulse signal data 72 shown in FIG. 18 to memory 70 of camera control device 50. Note that if camera control device 50 is connected to an external server via a network such as the Internet, data correction unit 62A may output pulse signal data 72 to the external server.

[0149] In step S31, data correction unit 62A determines whether or not a control signal indicating an instruction to end measurement of biological data has been received. The process of step S31 is the same as the process described in step S9 of FIG. 4, and therefore detailed description thereof will be omitted.

[0150] According to the information processing system 1A configured as described above, biometric data is corrected using position information indicating the relative positional relationship between the position of the display 2 and the position of the user's face. Therefore, more accurate biometric data can be obtained compared to when biometric data is corrected without taking the above-mentioned relative positional relationship into consideration.

[0151] (Embodiment 3) An information processing system 1B according to a third embodiment will be described. In the third embodiment, a detection area AR is set around the user's face, and feature amounts are acquired based on return light RL acquired from the detection area AR. In the third embodiment, the same components as those in the first embodiment are given the same reference numerals, and descriptions thereof will be omitted. Furthermore, since the information processing system 1B according to the third embodiment also uses the display control device 20 and camera control device 50 shown in FIG. 2, illustrations and detailed descriptions will be omitted, and FIG. 2 will be used instead as necessary for description.

[0152] FIG. 19 is a diagram illustrating an example of the configuration of an information processing system 1B. As illustrated in FIG. 19, the information processing system 1B includes a display 2, a camera 3 (an example of a photodetector), and an illumination device 4. In the information processing system 1B, a detection area AR is set at a second position that is different from a first position where a user is present and is irradiated with display light L. In the example illustrated in FIG. 19, the detection area AR is set behind the user. The detection area AR is an area that reflects the display light L. In addition, a white board having a constant spectral reflectance across the visible light wavelength range is installed in the detection area AR according to the third embodiment, for example.

[0153] The camera 3 according to the third embodiment captures an image of the user with the area around the user's face and at least a part of the detection area AR included in its angle of view. Specifically, the camera 3 captures an image of the user with the range 300 in FIG. 19 included in its angle of view. As a result, the camera 3 detects return light RL from the detection area AR and reflected light from the user. The camera 3 then acquires an image signal including pixels corresponding to the return light RL and pixels corresponding to the reflected light. The camera 3 transmits this image signal to the image processing unit 61.

[0154] The image processing unit 61 according to the third embodiment generates biometric data based on pixels corresponding to the reflected light of the image signal. Specifically, a G component value is calculated from the pixels corresponding to the reflected light, and this is used as a pulse signal. More specifically, the G component values ​​of the pixels constituting the area in which the user's skin is captured are acquired, a representative value of these G component values ​​is calculated, and this representative value is used as the pulse signal. Note that instead of the representative value of the G component value, a representative value of the R component value or the B component value may be used as the pulse signal.

[0155] Furthermore, the image processing unit 61 generates display light data based on pixels corresponding to the return light RL in the image signal. The display light data includes B, G, and R component values ​​of the pixels corresponding to the return light RL.

[0156] The data correcting unit 62 estimates an average G level (an example of a feature amount) of the content image based on the G component value included in the display light data. Then, the data correcting unit 62 corrects the pulse signal based on the estimated average G level. The pulse signal correction process is the same as that described in the first embodiment, and therefore will not be described again.

[0157] As described above, in the third embodiment, the average G level (an example of a feature) of the content image is estimated based on the light reflected from the detection area AR, and the pulse signal is corrected using this estimated average G level. With this configuration, the average G level is acquired in real time, so it is possible to omit the work of preparing in advance the feature data 31 in which time and the average G level are associated as described in FIG. 3 of the first embodiment. The information processing system 1B according to the third embodiment is particularly useful when the feature data 31 described in the first embodiment has not been prepared or when the work of preparing the feature data 31 is complicated.

[0158] Furthermore, in the third embodiment, a white board with a constant spectral reflectance over the visible light wavelength range is placed in the detection area AR, so that the camera 3 can detect the returned light RL with high accuracy.

[0159] The present disclosure can employ the following modifications.

[0160] (Variation 3-1) The return light RL does not necessarily have to be detected by the camera 3, and may be detected by a device different from the camera 3. For example, the information processing system 1C may further include a spectroscope (an example of a photodetector). The return light RL may then be detected by this spectroscope.

[0161] (Fourth embodiment) An information processing system 1C according to embodiment 4 will be described. The information processing system 1C corrects the pulse signal based on a feature amount related to the color of light emitted from a lighting device 4C. In embodiment 4, the same components as those in embodiment 1 are denoted by the same reference numerals, and description thereof will be omitted.

[0162] 20 is a diagram showing the configuration of an information processing system 1C according to Embodiment 4. The information processing system 1C includes a camera 3 and a lighting device 4C.

[0163] The lighting device 4C has a light source whose color changes over time in order to stimulate human psychology and emotions or to create a spatial effect.

[0164] The memory 70 of the camera control device 50 according to the fourth embodiment stores feature data (not shown) in which time and feature values ​​are associated with each other. For example, the feature data stores a correspondence relationship between information indicating the amount of light from the lighting device 4C, information indicating the color temperature of the light source (an example of a feature value), and information indicating the cycle at which the color temperature changes. That is, the feature data stores information indicating the time, intensity, and color of light irradiated by the lighting device 4C on the user.

[0165] The image processing unit 61 of the camera control device 50 according to the fourth embodiment acquires biometric data corresponding to changes in the intensity of reflected light from the user during a first period in which the user is illuminated by the illumination light from the illumination device 4C. Specifically, the image processing unit 61 acquires a user image captured during the first period, and acquires a representative value of the G component value in the user image as a pulse signal.

[0166] The data corrector 62 according to the fourth embodiment refers to the time when the user image was captured and the information indicating the cycle of color temperature change, and determines the intensity and color of light that the user was exposed to when the user image was captured. Then, the data corrector 62 calculates a predetermined correction value based on the information indicating the light intensity and the information indicating the color temperature, and corrects the pulse signal by subtracting the correction value from the pulse signal.

[0167] According to the information processing system 1C configured as described above, the biometric data is corrected based on the feature amount related to the color of the illumination light, so that the influence of the illumination light can be suppressed and the noise contained in the biometric data can be reduced, thereby making it possible to acquire accurate biometric data from a user illuminated by the illumination light.

[0168] (Embodiment 5) A moving body according to the fifth embodiment will be described. The moving body is, for example, a standard automobile. However, the moving body may be a large vehicle such as a bus. The moving body may also be an airplane, a train, or the like.

[0169] The mobile object includes a display member that displays video content played during a first period, a photodetector that detects reflected light from the user, and a processing device. The processing device generates biometric data corresponding to changes in the intensity of the reflected light, acquires feature quantities related to the color of the video content that change over time during the first period, and corrects the biometric data based on the feature quantities. The display member includes, for example, the display 2 described in the first embodiment. The photodetector includes, for example, the camera 3 described in the fourth embodiment. The processing device includes, for example, the display control device 20 and the camera control device 50 described in the first embodiment. [Industrial Applicability]

[0170] The technology of the present disclosure is useful in the technical field of measuring biological data based on images. [Explanation of symbols]

[0171] 1: Information processing system 2: Display 3: Camera 4: Lighting equipment 10: Automobiles 20: Display control device 31: Feature data 40: Control circuit 41: Data control section 50: Camera control device 60: Processor 61: Image processing section 62: Data correction section 63: User position detection unit AR: Detection area

Claims

1. An information processing method in a computer, comprising: acquiring biometric data corresponding to a change in intensity of reflected light from the user during a first period in which the video content is being played; acquiring a feature amount related to a color of the video content, the feature amount changing with the passage of time during the first period; correcting the biometric data based on the feature amount. Information processing methods.

2. acquiring the feature amount includes acquiring feature amount data in which the feature amount is associated with time; the biometric data is corrected based on the feature amount data; The information processing method according to claim 1 .

3. the biological data is data corresponding to a change in intensity of the reflected light of a specific wavelength corresponding to at least one wavelength band of blue, green, and red, the biological data is corrected based on the feature amount of the video content having a color corresponding to the specific wavelength; 3. The information processing method according to claim 1 or 2.

4. correcting the biological data is performed during the first period.

3. The information processing method according to claim 1 or 2.

5. correcting the biological data is performed in a second period after the first period; 3. The information processing method according to claim 1 or 2.

6. acquiring position information indicating a relative positional relationship between a position of a display member onto which the video content is projected and a position of a user viewing the video content; correcting the biometric data includes correcting the biometric data based on the position information and the feature amount.

3. The information processing method according to claim 1 or 2.

7. The display member is disposed inside the vehicle, The position information is generated based on a camera disposed inside the vehicle or a sensor provided on a seat of the vehicle for detecting an occupant. The information processing method according to claim 6.

8. the position information includes information indicating a relative positional relationship between a position of a center of gravity of a color in the video content and a position of a user viewing the video content; The information processing method according to claim 6.

9. The biological data indicates a pulse wave of the user.

3. The information processing method according to claim 1.

10. acquiring display light data generated by detecting return light from a second position, which is different from a first position where the user is present, and which is irradiated with display light resulting from the video content, using a photodetector; the biometric data is corrected based on the feature amount estimated based on the display light data. The information processing method according to claim 1 .

11. the photodetector is a camera; The camera is detecting the returned light and the reflected light; The information processing method according to claim 10 , further comprising outputting an image signal including pixels corresponding to the returned light and pixels corresponding to the reflected light.

12. the biometric data is generated based on pixels corresponding to the reflected light; the display light data is generated based on pixels corresponding to the returned light; The information processing method according to claim 11.

13. acquiring biometric data according to a change in intensity of reflected light from the user during a first period during which illumination light whose color changes with time is irradiated; correcting the biometric data based on a time change in a feature amount related to the color of the illumination light. Information processing methods.

14. a first acquisition unit that acquires biometric data corresponding to a change in intensity of reflected light from the user during a first period in which the video content is being played; a second acquisition unit that acquires a feature amount related to a color of the video content, the feature amount changing with the passage of time in the first period; a correction unit that corrects the biometric data based on the feature amount, Information processing device.

15. a display member that displays video content to be played during a first period; a photodetector for detecting reflected light from the user; a processing device, The processing device includes: generating biometric data corresponding to the change in intensity of the reflected light; acquiring a feature amount related to the color of the video content, the feature amount changing with the passage of time during the first period; correcting the biometric data based on the feature amount; Mobile object.

Images