Learning device, biological data estimation device, learning method, and biological data estimation method

Abstract

A learning device (100) comprises a luminance data acquisition unit (110) that acquires time-series luminance data based on time-series imaging data, a true value data acquisition unit (120) that acquires time-series true value data using time-series biological reference information based on a biological signal, a time difference correction unit (130) that corrects a time difference between the time-series luminance data and the time-series true value data, and a model training unit (140) that constructs a learning model that outputs biological data from the time-series luminance data using the time-series true value data and the time-series luminance data output by the time difference correction unit.

Description

Learning device, biological data estimation device, learning method, and biological data estimation method

[0001] The disclosed technology relates to a technology for constructing a learning model for estimating biometric data.

[0002] Among the techniques for estimating biometric data, there is a method for estimating biometric data such as pulse waves from minute changes in luminance on the face surface using a facial image sequence (time-series images) of a subject, as described in Patent Literature 1. In Patent Literature 1, a machine learning model is used as an estimator, and the relationship between the facial image sequence and pulse waves is learned to construct a model so as to eliminate an error between an estimated value by the machine learning model and a correct value calculated based on the chrominance signal of the image.

[0003] International Publication No. 2019 / 152983 (WO2019 / 152983)

[0004] However, the chrominance signal of an image may vary significantly depending on the environment outside the device, such as the state of the object itself included in the image (for example, the state of the subject's face). (Hereinafter, the "environment outside the device" will also be referred to as the "external environment.") Therefore, the technology described in Patent Document 1 has a problem in that when a learning model is constructed using a correct answer value calculated based on the chrominance signal of an image, it is easily affected by the external environment, and there is a high possibility that the accuracy of the learning model will be reduced.

[0005] The present disclosure is intended to solve the above-mentioned problems, and aims to improve the accuracy of learning models used to estimate biometric data compared to conventional methods.

[0006] The learning device of the present disclosure includes a luminance data acquisition unit that acquires time-series luminance data based on time-series imaging data, a true value data acquisition unit that acquires time-series true value data using time-series biological reference information based on biological signals, a time difference correction unit that corrects the time difference between the time-series luminance data and the time-series true value data, and a model learning unit that constructs a learning model that outputs biological data from the time-series luminance data using the time-series luminance data and the time-series true value data output by the time difference correction unit.

[0007] According to the present disclosure, it is possible to achieve an effect of improving the accuracy of a learning model used to estimate biometric data compared to conventional techniques.

[0008] FIG. 1 is a diagram illustrating an example of a basic configuration of a learning device according to the present disclosure. FIG. 2 is a diagram illustrating an example of a learning device according to the present disclosure and a source of data used by the learning device. FIG. 3 is a diagram illustrating an example of a learning device according to the present disclosure and a source of data used by the learning device. FIG. 4A is a diagram illustrating time-series luminance data and time-series true value data used by the learning device according to the present disclosure. FIG. 4B is a diagram illustrating an example of time difference correction in the learning device according to the present disclosure. FIG. 4C is a diagram illustrating an example of a case in which time-series luminance data and time-series true value data are in opposite phase. FIG. 5 is a diagram illustrating an example of a data set including time-series luminance data and time-series true value data used by the learning device according to the present disclosure. FIG. 6 is a flowchart illustrating an example of processing by the learning device according to the present disclosure. FIG. 7 is a diagram illustrating an example of a configuration in which the learning device according to the present disclosure is applied to a biological data estimation device. FIG. 8 is a flowchart illustrating an example of processing by the biological data estimation device. FIG. 9 is a flowchart illustrating a detailed example of time difference correction processing in processing by a learning device according to a second embodiment of the present disclosure. FIG. 10 is a diagram illustrating time difference correction according to the second embodiment. FIG. 11 is a flowchart showing a detailed example of a time difference correction process in the processing of the learning device according to the third embodiment of the present disclosure. FIG. 12 is a flowchart showing a detailed example of a time difference correction process in the processing of the learning device according to the fourth embodiment of the present disclosure. FIG. 13 is a flowchart showing a detailed example of a time difference calculation process in the processing of the learning device according to the fifth embodiment of the present disclosure. FIG. 14 is a flowchart showing a detailed example of a time difference calculation process in the processing of the learning device according to the sixth embodiment of the present disclosure. FIG. 15 is a flowchart showing a detailed example of a time difference calculation process in the processing of the learning device according to the seventh embodiment of the present disclosure. FIG. 16 is a diagram showing a first example of a hardware configuration for realizing functions according to the configuration of the present disclosure. FIG. 17 is a diagram showing a second example of a hardware configuration for realizing functions according to the configuration of the present disclosure.

[0009] In order to explain the present disclosure in more detail, embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0010] First Embodiment In the first embodiment, a basic form of the present disclosure will be described.

[0011] An example configuration of a learning device according to a first embodiment of the present disclosure will be described. FIG. 1 is a diagram illustrating an example basic configuration of the learning device 100 of the present disclosure. FIG. 2 is a diagram illustrating an example of the learning device 100 of the present disclosure and a source of data used by the learning device 100. FIG. 3 is a diagram illustrating an example of the learning device 100 of the present disclosure and a source of data used by the learning device 100. The learning device 100 constructs a learning model for estimating biometric data. The learning device 100 constructs the learning model by learning using time-series luminance data based on time-series imaging data and time-series true value data acquired using time-series biometric reference information based on biometric signals. Here, if learning is simply performed using time-series luminance data based on time-series imaging data and time-series true value data acquired using time-series biometric reference information based on biometric signals, a time difference will occur between the reference point of the correct value (true value) and the face (for example, a difference in pulse wave propagation time if the biometric data is a pulse wave), and this difference may vary due to individual differences, etc. As a result, the learning model is likely unable to stably learn the original input-output relationship, leading to a decrease in the estimation accuracy of the learning model. Therefore, the learning device 100 of the present disclosure corrects the time difference and constructs the learning model using the corrected data. The learning device 100 shown in each of Figures 1, 2, and 3 includes a luminance data acquisition unit 110, a true value data acquisition unit 120, a time difference correction unit 130, and a model learning unit 140.

[0012] The luminance data acquisition unit 110 acquires time-series luminance data based on time-series imaging data. The imaging data is a series of frames “Im(t ｉThe image data includes information such as time. The luminance data acquisition unit 110 acquires the luminance data of the frame Im(t) included in the image data in the same manner as known processes such as skin region detection, measurement region setting, and pulse wave source signal extraction. ｉ ) (i=1, 2, 3, . . . ) ｌ ) (i=1, 2, 3, ...) is acquired. ｌ ) (i=1, 2, 3, ...) contains information such as time that can be synchronized with the time-series true value data described later. Here, there are two types of luminance data: luminance data that is not used for evaluation of the learning model described later, and luminance data that is used for evaluation of the learning model. Hereinafter, when distinguishing between the two types of luminance data, they will be referred to as "time-series luminance data" and "time-series luminance data for evaluation", respectively. Note that the time-series luminance data L(t ｌ ) there is always one or more pieces of time-series true value data, which will be described later.

[0013] When configured as shown in FIG. 2, the luminance data acquisition unit 110 acquires time-series luminance data L(t ｌ Specifically, the luminance data acquisition unit 110 acquires the imaging data output from the imaging device 200, and uses the imaging data to acquire the time-series luminance data L(t ｌ ) to obtain the

[0014] When configured as shown in FIG. 3, the luminance data acquisition unit 110 acquires the time-series luminance data L(t ｌ ) to obtain the

[0015] When constructing a learning model for estimating biometric data of a vehicle occupant (e.g., a driver), the luminance data acquisition unit 110 acquires time-series luminance data L(t ｌ ) to obtain the

[0016] The true value data acquisition unit 120 acquires time-series biological reference information P(t ｐ) to obtain the time series true value data B(t ｂ ) is acquired. The biological signal is a signal obtained in time series by contact measurement of a living body, such as a pulse wave, a continuous blood pressure waveform, or an electrocardiogram waveform. The biological reference information is a continuous value "P(t ｐ That is, when the biological signal is a measurement result of a pulse wave, the biological reference information P(t ｐ ) indicates a pulse wave. That is, when the biological signal is a measurement result of blood pressure, the biological reference information P(t ｐ ) indicates a continuous blood pressure waveform. That is, when the biosignal is a measurement result of an electrocardiogram waveform, the bioreference information P(t ｐ ) indicates an electrocardiogram waveform. The true value data acquisition unit 120 acquires the biological reference information P(t ｐ ) to frame Im(t ｉ ) for each time series luminance data L(t ｌ ) and the same subject and time. This value is used as the time-series true value data B(t ｂ ) is defined as the time series true value data B(t ｂ ) is time-series luminance data L(t ｌ ) and contains information that allows it to be synchronized with the

[0017] Here, as shown in FIG. 2, when the learning device 100 is configured to be able to receive a signal output from the sensor device 300, the true value data acquisition unit 120 acquires time-series true value data B(t ｂ Specifically, the true value data acquiring unit 120 acquires the time-series biological reference information P(t ｐ ) and obtain the biometric reference information P(t ｐ ) to obtain the time series true value data B(t ｂ ) to obtain the

[0018] As shown in FIG. 3, when the learning device 100 is configured to refer to information stored in the database 400, the true value data acquisition unit 120 acquires the time-series true value data B(t ｂ ) to obtain the

[0019] The time difference correction unit 130 calculates the time series luminance data L(t ｌ ) and the time series true value data B(t ｂ 4A shows the time series luminance data L(t ｌ ) and time series true value data B(t ｂ 4A and 4B are diagrams illustrating an image of time difference correction in the learning device 100 of the present disclosure. ｐ 4A shows an example in which the true value data B(t ｂ ) and luminance data L(t ｌ ) and use it for learning, the time difference Δt needs to be corrected. By correcting the time difference Δt, the time difference correction unit 130 obtains the true value data B(t ｂ ) is the corrected true value data B(t ｂｆ ), and the corrected time-series luminance data L(t ｌｆ ) and time series true value data B(t ｂｆ ) is output. Here, the corrected time series luminance data L(t ｌｆ ) and the corrected time series true value data B(t ｂｆ ) are in an antiphase relationship. FIG. 4C is a diagram showing an image of a case where the time-series luminance data and the time-series true value data are in antiphase. As shown in FIG. 4B, when the biometric reference information P(t ｐ ) are, for example, pulse waves (PPG), they are in phase. However, as shown in FIG. 4C, ｐ ) is, for example, time-series true value data of a continuous blood pressure waveform, the corrected time-series luminance data L(tｌｆ ) and the corrected time series true value data B(t ｂｆ ) can be assumed to have an antiphase relationship. Therefore, the biometric reference information P(t ｐ ), the time difference correction unit 130 is configured to perform correction taking into account the phase difference. For example, if the time difference correction unit 130 performs time difference correction using cross-correlation (see embodiment 5 described later), the time difference correction unit 130 performs correction for the number N frames from the beginning of the sequence a of cross-correlation coefficients. ｆ Based on the above-described concept, the time difference correction unit 130 calculates the time series luminance data L(t ｌ ) and time series true value data B(t ｂ ), the time series luminance data L(t ｌ ) and time series true value data B(t ｂ The time difference correction unit 130 corrects the time series luminance data L(t ｌｆ ) and time series true value data B(t ｂｆ ) to the model learning unit 140.

[0020] The model learning unit 140 calculates the time series luminance data L(t ｌｆ ) and time series true value data B(t ｂｆ ) to obtain the time series luminance data L(t ｌ ) to construct a learning model M that outputs biological data. ｌｆ ) and time series true value data B(t ｂｆ ) and calculates the time-series luminance data L(t ｌ ) and outputs the model M (learning model). The model M is generated by estimating the biometric data from the time-series luminance data L(t ｌ ) is used as input data, and biometric data E(t ｅ) as output data. The learning method is not particularly specified. For example, the learning method learned from the following document can be used. Document "Turnip: Time-Series U-Net With Recurrence For Nir Imaging Ppg (signalprocessingsociety.org)"

[0021] In addition to the above components, the learning device 100 also includes a control unit (not shown), a memory unit (not shown), and a communication unit (not shown). The control unit (not shown) controls the entire learning device 100 and each of its components. The control unit (not shown), for example, starts up the learning device 100 in response to external commands. The control unit (not shown) also controls the state of the learning device 100 (operating state = startup, shutdown, sleep, etc.). The memory unit (not shown) stores various data used by the learning device 100. For example, the memory unit (not shown) stores output (output data) from each component of the learning device 100 and outputs data requested by each component to the requesting component. The communication unit (not shown) communicates with external devices. For example, communication is performed between the learning device 100 and a peripheral device (e.g., an imaging device, a sensor device, a database, or an output device). For example, if the learning device 100 and the peripheral device are not connected via a wire, the communication unit (not shown) has the function of communicating between the learning device 100 and the peripheral device. The communication unit (not shown) has a function of communicating with an external device. The control unit (not shown), the storage unit (not shown), and the communication unit (not shown) are the same in other configurations described later.

[0022] Here, the learning device 100 uses the time-series luminance data L(t ｌ ) and time series true value data B(t ｂ) for a plurality of evaluation subjects, and may be used for model learning. The evaluation subjects are users who can be the evaluation subjects. For example, in the case of a learning model for estimating biometric data of a driver of a mobile body, the evaluation subjects are users who can be the driver. For example, combinations of time-series luminance data and time-series true value data are prepared in advance for each evaluation subject. The database 400 shown in FIG. 3 stores time-series luminance data L(t ｌ ) and time series true value data B(t ｂ The learning device 100 stores the time-series luminance data L(t ｌ ) and time series true value data B(t ｂ ) used by the learning device 100 of the present disclosure to construct a learning model. ｌ ) and time series true value data B(t ｂ 5 ) is an image of a data set consisting of time-series luminance data L(t ). In FIG. 5 , "time-series luminance data" is simply written as "luminance data", and "time-series true value data" is simply written as "true value data". The database shown in FIG. 3 holds a data set as shown in the image in FIG. 5 , for example. The data set shown in FIG. 5 is a data set consisting of time-series luminance data L(t ｌ ) ("411" shown in FIG. 5) and the time-series true value data B(t ｂ ) (shown as "412" in FIG. 5) １ and pair data 410A ２ The time-series luminance data L(t ｌ ) and time series true value data B(t ｂ ) and pair data 410B １ and pair data 410B ２ The time-series luminance data L(t ｌ ) and time series true value data B(t ｂ ) and pair data 410C １ and pair data 410C ２The data set shown in FIG. 5 shows two sets of paired data for each of three people, but is not limited to this example and may include paired data for two or four or more people, and may also include two or more sets of paired data per person. When configured in this way, the learning device 100 stores the time-series luminance data L(t ｌ ) and time series true value data B(t ｂ ) are acquired in order for each combination, and for each combination, processing is performed by the brightness data acquisition unit 110, the true value data acquisition unit 120, the time difference correction unit 130, and the model learning unit 140.

[0023] A description will be given of an example of processing by the learning device 100. Fig. 6 is a flowchart showing an example of processing by the learning device 100 of the present disclosure. For example, upon receiving a learning start command from an external device, the learning device 100 starts the processing shown in Fig. 6.

[0024] The learning device 100 executes a luminance data acquisition process (step ST100). In the luminance data acquisition process, the luminance data acquisition unit 110 of the learning device 100 acquires time-series luminance data L(t ｌ ) to obtain the

[0025] The learning device 100 executes a true value data acquisition process (step ST200). In the true value data acquisition process, the true value data acquisition unit 120 of the learning device 100 acquires time-series true value data B(t ｂ ) to obtain the

[0026] The learning device 100 executes a time difference correction process (step ST300). In the time difference correction process, the time difference correction unit 130 of the learning device 100 calculates the time series luminance data L(t ｌ ) and the time series true value data B(t ｂ The time difference correction unit 130 corrects the time difference Δt between the time series luminance data L(t ｌ ) and time series true value data B(t ｂ The time difference correction unit 130 then receives the time series luminance data L(t ｌ) and time series true value data B(t ｂ ) based on the time series luminance data L(t ｌ ) and time series true value data B(t ｂ Specifically, the time difference correction unit 130 checks the relative speed and slowness of the time-series luminance data L(t ｌ ) and time series true value data B(t ｂ ), one of which is determined to be the delayed data and the other to be the advanced data. The time difference correction unit 130 then determines that the delayed data (time-series luminance data L(t ｌ ), or time series true value data B(t ｂ )) a time difference Δt is added to the time t, and data corresponding to the time difference Δt is removed from the beginning, and the data that is advanced in time (time-series luminance data L(t ｌ ), or time series true value data B(t ｂ The time difference correction unit 130 removes data corresponding to the time difference Δt from the end of the time series luminance data L(t ｌｆ ), and time series true value data B(t ｂｆ ) to the model learning unit 140. Here, “t” is the time t ｌ Or time t ｂ After the time difference correction, the time when the time series luminance data is ｌｆ and the time series true value data is ｂｆ For example, if the biometric reference information is delayed, the time t ｌｆ is "t ｌ ”, the time t of the corrected true value data ｂｆ is "t ｂ +Δt" (see FIG. 10 described later). Note that in the present disclosure, the method for checking the relative speed-delay relationship between the imaging data and the biological reference information is not particularly limited, but the speed-delay relationship itself may be, for example, a speed-delay relationship such as that described in embodiment 2, embodiment 3, or embodiment 4 described later.

[0027] The learning device 100 executes a model learning process (step ST400). In the model learning process, the model learning unit 140 of the learning device 100 uses the time-series luminance data and the time-series true value data output by the time difference correction unit to construct a learning model that outputs biological data from the time-series luminance data.

[0028] After executing the model learning process (step ST400), the learning device 100 then proceeds to an end determination process (step ST500). In the end determination process, a control unit (not shown) of the learning device 100 determines whether to end the processing of the learning device 100. The control unit (not shown) determines whether to end the processing of the learning device 100, for example, in accordance with an external end command or an execution program. If the control unit (not shown) determines not to end the processing of the learning device 100 (step ST500 "NO"), the learning device 100 proceeds to the processing of step ST100 and repeats the processing from step ST100. If the control unit (not shown) determines to end the processing of the learning device 100 (step ST500 "YES"), the learning device 100 ends the processing.

[0029] Next, a configuration example of a biological data estimation device including a learning device will be described. FIG. 7 is a diagram showing a configuration example when the learning device of the present disclosure is applied to a biological data estimation device. In the description here, in order to distinguish from other configuration examples, the reference numerals of some components will be denoted with "A", such as biological data estimation device 10A and learning device 100A. The biological data estimation device 10A captures a space including a skin region of a subject at a predetermined frame rate Fr, and generates a series of frames Im(t ｉ ) and continuous values ​​P(t) representing the subject's biological signals (e.g., pulse wave, blood pressure waveform, electrocardiogram waveform) acquired at predetermined time intervals Ts at the same time as the imaging information. ｐ ) and the biometric data estimation device 10A receives the biometric reference information. ｆ For each particular series of frames Im(t ｉ -N f +1) to Im(tｉ ) the estimated result E(t ｅ ) as a continuous value E(t ｉ -N ｆ +1) to E(t ｉ ) is output. ｉ " indicates the frame number assigned to each frame. ｐ " indicates an input number assigned to a continuous value that is input biometric reference information. ｉ "," "t ｐ " and "t ｅ " is an integer equal to or greater than 1. The number of frames N ｆ is an integer equal to or greater than 2. The number of subjects included in the imaging data and the biological reference information is equal to or greater than 1. The imaging data may be any color information (for example, RGB color or monochrome).

[0030] The biological data estimation device 10A includes a luminance data acquisition unit 110A, a true value data acquisition unit 120, a time difference correction unit 130A, a model learning unit 140, and a biological data estimation unit 150. The luminance data acquisition unit 110A, the true value data acquisition unit 120, the time difference correction unit 130A, and the model learning unit 140 constitute a learning device 100A.

[0031] The true value data acquisition unit 120, the time difference correction unit 130A, or the model learning unit 140 has the same functions as the true value data acquisition unit 120, the time difference correction unit 130, or the model learning unit 140 already described.

[0032] In addition to the functions of the luminance data acquisition unit 110 already described, the luminance data acquisition unit 110A outputs the acquired time-series luminance data to the biological data estimation unit 150. When the biological data estimation device 10A executes the process of constructing a learning model, the luminance data acquisition unit 110A outputs the acquired time-series luminance data L(t ｌ ) to the time difference correction unit 130A, and when the biological data estimation device 10A executes the process of estimating biological data, the acquired time series luminance data L(t ｌ ) (evaluation time-series luminance data) to the biological data estimation unit 150.

[0033] The biological data estimation unit 150 uses the learning model M constructed by the model learning unit 140 and the time-series luminance data L(t ｌ ) (evaluation time-series luminance data), based on the biological data E(t ｅ The biological data estimation unit 150 estimates the time-series luminance data L(t ｌ ) and inputs the time-series luminance data L(t ｌ ) to estimate biometric data, and the estimated biometric data E(t ｅ ) is output.

[0034] When constructing a learning model for estimating biometric data of a vehicle occupant (e.g., a driver), the luminance data acquisition unit 110A acquires time-series luminance data L(t ｌ The biometric data estimation unit 150 estimates the biometric data of the occupant (driver).

[0035] An example of processing by the biological data estimation device 10A will be described. The processing by which a configuration corresponding to the learning device 100A constructs a learning model among the processing by the biological data estimation device 10A is similar to the processing by the learning device 100 already described, and therefore a detailed description thereof will be omitted here. The processing by which a configuration corresponding to the learning device 100A constructs a learning model among the processing by the biological data estimation device 10A is included as part of a biological data estimation method.

[0036] Fig. 8 is a flowchart showing an example of processing by the biological data estimation device 10 (10A). The processing shown in Fig. 8 is a part of a biological data estimation method by the biological data estimation device 10 (10A). For example, the biological data estimation device 10 (10A) starts the processing shown in Fig. 8 when it receives an external command to estimate generated data. Alternatively, in the case where the biological data estimation device 10 (10A) estimates the biological data of an occupant of a mobile body, for example, the processing shown in Fig. 8 starts when the power source of the mobile body starts.

[0037] The biological data estimation device 10 (10A) first executes a luminance data acquisition process (step ST1100). In the luminance data acquisition process, the luminance data acquisition unit 110A of the biological data estimation device 10 (10A) acquires time-series luminance data L(t ｌ The luminance data acquisition unit 110A acquires the acquired time-series luminance data L(t ｌ ) to the biological data estimation unit 150.

[0038] The biological data estimation device 10 (10A) then executes a biological data estimation process (step ST1200). In the biological data estimation process, the biological data estimation unit 150 of the biological data estimation device 10 (10A) uses the learning model M constructed by the model learning unit 140 and the time-series luminance data L(t ｌ ), based on the biometric data E(t ｅ ) is estimated.

[0039] In the biometric data estimation device 10 (10A), the biometric data estimation unit 150 outputs biometric data E(t ｅ ), the process then proceeds to an end determination process (step ST1300). In the end determination process, a control unit (not shown) of the biological data estimation device 10 (10A) determines whether to end the processing of the biological data estimation device. The control unit (not shown) determines whether to end the processing of the biological data estimation device 10 (10A) in accordance with, for example, an external end command or an execution program. If the control unit (not shown) determines that the processing of the biological data estimation device 10 (10A) is not to end (step ST1300 "NO"), the process proceeds to step ST1100, and the process is repeated from step ST1100. If the control unit (not shown) determines that the processing of the biological data estimation device 10 (10A) is to end (step ST1300 "YES"), the biological data estimation device 10 (10A) ends the processing.

[0040] As described above, the learning device or biometric data estimation device of the present disclosure corrects the time difference between the luminance data and true value data used to construct a learning model used for biometric data estimation to eliminate variations in time difference between data, such as individual differences, that are contained in the data used to construct the learning model. This allows the model to learn the original input-output relationship, thereby improving the accuracy of biometric data estimation. Furthermore, it is possible to construct a robust learning model that is less susceptible to external factors.

[0041] In this embodiment, the following configuration is disclosed: a learning device including: a luminance data acquisition unit that acquires time-series luminance data based on time-series imaging data; a true-value data acquisition unit that acquires time-series true-value data using time-series biological reference information based on a biological signal; a time-difference correction unit that corrects the time difference between the time-series luminance data and the time-series true-value data; and a model learning unit that constructs a learning model that outputs biological data from the time-series luminance data using the time-series luminance data and the time-series true-value data output by the time-difference correction unit. This provides an effect of providing a learning device that enables the accuracy of a learning model used to estimate biological data to be improved compared to conventional methods.

[0042] In this embodiment, the following configuration is disclosed: a biometric data estimation device including: a luminance data acquisition unit that acquires time-series luminance data based on time-series imaging data; a true-value data acquisition unit that acquires time-series true-value data using time-series biometric reference information based on a biometric signal; a time-difference correction unit that corrects a time difference between the time-series luminance data and the time-series true-value data; a model learning unit that constructs a learning model that outputs biometric data from the time-series luminance data using the time-series luminance data and the time-series true-value data output by the time-difference correction unit; and a biometric data estimation unit that estimates biometric data based on the learning model constructed by the model learning unit and the time-series luminance data acquired by the luminance data acquisition unit. This provides an advantage of providing a biometric data estimation device that enables improved accuracy of the learning model used to estimate biometric data compared to conventional devices. The present disclosure also provides an advantage of providing a biometric data estimation device that enables improved accuracy of estimated biometric data compared to conventional devices.

[0043] In this embodiment, the following configuration is disclosed: a learning method executed by a learning device, the learning method comprising: a step in which a luminance data acquisition unit of the learning device acquires time-series luminance data based on time-series imaging data; a step in which a true value data acquisition unit of the learning device acquires time-series true value data using time-series biological reference information based on biological signals; a step in which a time difference correction unit of the learning device corrects the time difference between the time-series luminance data and the time-series true value data; and a step in which a model learning unit of the learning device constructs a learning model that outputs biological data from the time-series luminance data, using the time-series luminance data and the time-series true value data output by the time difference correction unit. This provides an effect of providing a learning method that enables the accuracy of a learning model used to estimate biological data to be improved compared to conventional methods.

[0044] In this embodiment, the following configuration is disclosed: a biometric data estimation method executed by a biometric data estimation device, the biometric data estimation method comprising: a luminance data acquisition unit of the learning device acquiring time-series luminance data based on time-series imaging data; a true value data acquisition unit of the learning device acquiring time-series true value data using time-series biometric reference information based on a biometric signal; a time difference correction unit of the learning device correcting a time difference between the time-series luminance data and the time-series true value data; a model learning unit of the learning device constructing a learning model that outputs biometric data from the time-series luminance data, using the time-series luminance data and the time-series true value data output by the time difference correction unit; and a biometric data estimation unit of the learning device estimating biometric data based on the learning model constructed by the model learning unit and the time-series luminance data acquired by the luminance data acquisition unit. This advantageously provides a biometric data estimation method that enables improved accuracy of the learning model used to estimate biometric data compared to conventional methods. The present disclosure also has the effect of providing a biometric data estimation method that enables the accuracy of estimated biometric data to be improved compared to conventional methods.

[0045] This embodiment further discloses the following configuration: a learning device, wherein the biological reference information is a pulse wave. This provides an advantage of providing a learning device that can improve the accuracy of a learning model used to estimate a pulse wave, which is biological data, compared to conventional methods. Furthermore, by applying the configuration to the learning method, the present disclosure provides the same advantage as the above.

[0046] This embodiment further discloses the following configuration: a learning device characterized in that the biological reference information is a continuous blood pressure waveform. This provides an advantage of providing a learning device that enables the accuracy of a learning model used to estimate blood pressure, which is biological data, to be improved compared to conventional methods. Furthermore, by applying the above configuration to the above learning method, the present disclosure provides the same advantage as the above.

[0047] This embodiment further discloses the following configuration: A learning device, wherein the biological reference information is an electrocardiogram waveform. As a result, the present disclosure further provides an effect of being able to provide a learning device that enables improvement in the accuracy of a learning model used to estimate an electrocardiogram waveform, which is biological data, compared to conventional methods.

[0048] This embodiment further discloses the following configuration: A learning device comprising: a learning device that sequentially acquires combinations of time-series luminance data and time-series true value data for each of a plurality of people, and executes processing for each combination by the luminance data acquisition unit, the true value data acquisition unit, the time difference correction unit, and the model learning unit. This provides an advantage of providing a learning device that can improve the accuracy of a learning model used to estimate biometric data compared to conventional methods. Furthermore, the present disclosure provides the same advantage as the above by applying the above configuration to the above learning method.

[0049] This embodiment further discloses the following configuration: A biological data estimation device, wherein the biological reference information is a pulse wave. As a result, the present disclosure further achieves an effect of providing a biological data estimation device that enables the accuracy of a learning model used to estimate a pulse wave, which is biological data, to be improved compared to conventional devices. The present disclosure also achieves an effect of providing a biological data estimation device that enables the accuracy of estimating a pulse wave, which is biological data, to be improved compared to conventional devices. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the generated data estimation method.

[0050] This embodiment further discloses the following configuration: A biological data estimation device, wherein the biological reference information is a continuous blood pressure waveform. As a result, the present disclosure further provides an effect of providing a biological data estimation device that enables the accuracy of a learning model used to estimate blood pressure, which is biological data, to be improved compared to conventional devices. The present disclosure also provides an effect of providing a biological data estimation device that enables the accuracy of estimating blood pressure, which is biological data, to be improved compared to conventional devices. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the generated data estimation method.

[0051] This embodiment further discloses the following configuration: A biological data estimation device, wherein the biological reference information is an electrocardiogram waveform. As a result, the present disclosure further provides an effect of providing a biological data estimation device that enables the accuracy of a learning model used to estimate an electrocardiogram waveform, which is biological data, to be improved compared to conventional devices. The present disclosure also provides an effect of providing a biological data estimation device that enables the accuracy of estimating an electrocardiogram waveform, which is biological data, to be improved compared to conventional devices. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the generated data estimation method.

[0052] This embodiment further discloses the following configuration: the biometric data estimation device, wherein the luminance data acquisition unit acquires time-series luminance data using image data of a driver of a moving body captured in time series, and the biometric data estimation unit estimates the biometric data of the driver. This provides an advantage of providing a biometric data estimation device that enables improved accuracy in estimating the biometric data of a driver of a moving body compared to conventional devices. Furthermore, when applied to an occupant monitoring device that monitors occupants, the present disclosure also provides an advantage of, for example, improving the accuracy in detecting an abnormality in the driver compared to conventional devices. Furthermore, the present disclosure provides the same advantage as the above-described advantage by applying the above configuration to the generated data estimation method.

[0053] This embodiment further discloses the following configuration: A biometric data estimation device comprising: a combination of time-series luminance data and time-series true value data for each of a plurality of people is acquired in sequence for each combination; and for each combination, processing is performed by the luminance data acquisition unit, the true value data acquisition unit, the time difference correction unit, and the model learning unit. This provides an advantage of providing a biometric data estimation device that enables improved accuracy of a learning model used to estimate biometric data compared to conventional devices. Furthermore, the present disclosure provides the same advantage as the above by applying the above configuration to the generated data estimation method.

[0054] Embodiment 2. Embodiment 2 is a more detailed configuration example (first example) of the configuration related to the time difference correction of Embodiment 1. In Embodiment 2, among the components related to Embodiment 2, components similar to the components related to Embodiment 1 already described are denoted by the same component names and the same or similar reference numerals, and duplicate descriptions are omitted as appropriate.

[0055] A configuration example according to embodiment 2 will be described. In this description, in order to distinguish from configuration examples of other embodiments, the reference numerals of some components will be denoted with "B", such as a biological data estimation device 10B and a learning device 100B. The biological data estimation device 10B is configured to include a luminance data acquisition unit 110A, a true value data acquisition unit 120, a time difference correction unit 130B, a model learning unit 140, and a biological data estimation unit 150. The luminance data acquisition unit 110A, the true value data acquisition unit 120, the time difference correction unit 130B, and the model learning unit 140 constitute the learning device 100B.

[0056] The time difference correction unit 130B calculates the time series luminance data L(t ｌ ) and the time-series true value data B(t ｂ ) and the time series luminance data L(t ｌ ) and the time series true value data B(t ｂ ) and corrects the time difference Δt between the time series luminance data L(t ｌ ) based on the time series true value data B(t ｂ ) is corrected.

[0057] A description will be given of example processes of a learning device and a biological data estimation device according to a second embodiment of the present disclosure. Processes similar to those already described will be omitted here to avoid duplication. FIG. 9 is a flowchart showing a detailed example of a time difference correction process in the process of the learning device 100B according to the second embodiment of the present disclosure. FIG. 10 is a diagram illustrating time difference correction according to the second embodiment. For example, the time difference correction unit 130B starts the time difference correction process when the luminance data acquisition process of step ST100 and the true value data acquisition process of step ST200 already described are executed and time-series luminance data and time-series true value data are acquired.

[0058] The time difference correction unit 130B executes a time difference calculation process (step ST320). The time difference correction unit 130B calculates the time difference based on the time series luminance data L(t ｌ) and the time-series true value data B(t ｂ ) is compared with the time-series luminance data L(t ｌ ) and time series true value data B(t ｂ ) based on the time series luminance data L(t ｌ ) as the reference time series true value data B(t ｂ ) and calculate the time difference Δt.

[0059] The time difference correction unit 130B executes a process of correcting the true value data based on the luminance data (step ST322). ｌ ) based on the time series true value data B(t ｂ The time difference correction unit 130B corrects, for example, the time series true value data B(t ｂ The time difference correction unit 130B performs a process of correcting the delayed data (time series true value data B(t ｂ )), the time difference Δt is added to the time t, and only the data D corresponding to the time difference Δt is removed from the beginning, and the data that is advanced in time (time-series luminance data L(t ｌ )), data D corresponding to the time difference Δt is removed from the end. By this process, the time difference correction unit 130B removes the time series true value data B(t ｂｆ ) and time series luminance data L(t ｌｆ The time difference correction unit 130B calculates the corrected time series true value data B(t ｂｆ ) and time series luminance data L(t ｌｆ ) to the model learning unit 140. In this case, the time-series luminance data L(t ｌｆ ) is the time series luminance data L(t ｌ ) is the same data as

[0060] The time difference correction unit 130B calculates the corrected time series true value data B(tｂｆ ) and time series luminance data L(t ｌｆ ) to the model learning unit 140, the time difference correction process ends.

[0061] In this embodiment, the following configuration is disclosed: The learning device is characterized in that the time difference correction unit compares the time-series luminance data acquired by the luminance data acquisition unit with the time-series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time-series luminance data and the time-series true value data, and corrects the time-series true value data based on the time-series luminance data. This further provides an advantage that the processing load can be reduced because it is only necessary to perform correction processing on the time-series true value data based on the time-series luminance data. Furthermore, the present disclosure provides an advantage similar to the above-described advantage by applying the above configuration to the above-described learning method.

[0062] This embodiment further discloses the following configuration: the biological data estimation device, wherein the time difference correction unit compares the time-series luminance data acquired by the luminance data acquisition unit with the time-series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time-series luminance data and the time-series true value data, and corrects the time-series true value data based on the time-series luminance data. This further provides an advantage that the processing load can be reduced because it is only necessary to perform correction processing on the time-series true value data based on the time-series luminance data. Furthermore, the present disclosure provides an advantage similar to the above-described advantage by applying the above configuration to the generated data estimation method.

[0063] Embodiment 3. Embodiment 3 is a more detailed configuration example (second example) of the configuration related to the time difference correction of Embodiment 1. In Embodiment 3, among the components related to Embodiment 3, components similar to the components related to Embodiment 1 or Embodiment 2 already described are designated by the same component names and the same or similar reference numerals, and duplicate descriptions are omitted as appropriate.

[0064] A configuration example according to the third embodiment will be described. In the description here, in order to distinguish from configuration examples of other embodiments, the reference numerals of some components will be denoted with "C", such as a biological data estimation device 10C and a learning device 100C. The biological data estimation device 10C is configured to include a luminance data acquisition unit 110A, a true value data acquisition unit 120, a time difference correction unit 130C, a model learning unit 140, and a biological data estimation unit 150. The luminance data acquisition unit 110A, the true value data acquisition unit 120, the time difference correction unit 130C, and the model learning unit 140 constitute the learning device 100C.

[0065] The time difference correction unit 130C calculates the time series luminance data L(t ｌ ) and the time-series true value data B(t ｂ ) and the time series luminance data L(t ｌ ) and the time series true value data B(t ｂ ) and corrects the time difference Δt between the time series true value data B(t ｂ ) based on the time series luminance data L(t ｌ ) is corrected.

[0066] An example of processing of a learning device and a biological data estimation device according to a third embodiment of the present disclosure will be described. Processing similar to processing already described will be omitted here to avoid redundancy. FIG. 11 is a flowchart showing a detailed example of time difference correction processing in the processing of the learning device 100C according to the third embodiment of the present disclosure. The time difference correction unit 130C, for example, executes the luminance data acquisition processing of step ST100 and the true value data acquisition processing of step ST200 already described, and obtains the time-series luminance data L(t ｌ ) and time series true value data B(t ｂ ) is acquired, the time difference correction process starts.

[0067] The time difference correction unit 130C executes a time difference calculation process (step ST320). The time difference correction unit 130C calculates the time difference based on the time series luminance data L(t ｌ) and the time-series true value data B(t ｂ ) is compared with the time-series luminance data L(t ｌ ) and time series true value data B(t ｂ ) based on the time series true value data B(t ｂ ) as the reference, the time series luminance data L(t ｌ ) and calculate the time difference Δt.

[0068] The time difference correction unit 130C executes a process of correcting the luminance data based on the true value data (step ST332). ｂ ) based on the time series luminance data L(t ｌ The time difference correction unit 130C corrects, for example, the time series luminance data L(t ｌ The time difference correction unit 130C performs a process of correcting the delayed data (time-series luminance data L(t ｌ )), the time difference Δt is added to the time t, and only the data D corresponding to the time difference Δt is removed from the beginning, and the data that is advanced in time (time series true value data B(t ｂ )), data D corresponding to the time difference Δt is removed from the end. By this process, the time difference correction unit 130C removes the time series true value data B(t ｂｆ ) and time series luminance data L(t ｌｆ The time difference correction unit 130C calculates the corrected time series true value data B(t ｂｆ ) and time series luminance data L(t ｌｆ ) to the model learning unit 140. In this case, the corrected time-series true value data B(t ｂｆ ) is the time series true value data B(t ｂ ) is the same data as

[0069] The time difference correction unit 130C calculates the corrected time series true value data B(tｂｆ ) and time series luminance data L(t ｌｆ ) to the model learning unit 140, the time difference correction process ends.

[0070] This embodiment further discloses the following configuration: The learning device is characterized in that the time difference correction unit compares the time-series luminance data acquired by the luminance data acquisition unit with the time-series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time-series luminance data and the time-series true value data, and corrects the time-series luminance data based on the time-series true value data. This further provides an advantage that the processing load can be reduced because it is only necessary to perform correction processing on the time-series luminance data based on the time-series true value data. Furthermore, the present disclosure further provides an advantage that the accuracy of the learning model can be improved because the time-series true value data that is likely to be closest to the actual value is used as the reference. Furthermore, the present disclosure provides an advantage similar to the above-described advantage by applying the above configuration to the generated data estimation device, the learning method, or the generated data estimation method.

[0071] This embodiment further discloses the following configuration: The time difference correction unit compares the time-series luminance data acquired by the luminance data acquisition unit with the time-series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time-series luminance data and the time-series true value data, correcting the time-series luminance data based on the time-series true value data. This further reduces the processing load because the time-series luminance data only needs to be corrected based on the time-series true value data. Furthermore, the present disclosure uses time-series true value data that is likely to be closer to the actual value as a reference, thereby improving the accuracy of the learning model. Furthermore, the present disclosure achieves the same effect as the above by applying the above configuration to the generated data estimation device, the learning method, or the generated data estimation method.

[0072] Embodiment 4. Embodiment 4 is a more detailed configuration example (third example) of the configuration related to the time difference correction of Embodiment 1. In Embodiment 4, among the components related to Embodiment 4, components similar to the components related to Embodiment 1, Embodiment 2, or Embodiment 3 already described will be designated by the same component names and the same or similar reference numerals, and duplicate explanations will be omitted as appropriate.

[0073] A configuration example according to embodiment 4 will be described. In this description, in order to distinguish from configuration examples of other embodiments, the reference numerals of some components will be denoted with "D", such as a biological data estimation device 10D and a learning device 100D. The biological data estimation device 10D is configured to include a luminance data acquisition unit 110A, a true value data acquisition unit 120, a time difference correction unit 130D, a model learning unit 140, and a biological data estimation unit 150. The luminance data acquisition unit 110A, the true value data acquisition unit 120, the time difference correction unit 130D, and the model learning unit 140 constitute the learning device 100D.

[0074] The time difference correction unit 130D calculates the time series luminance data L(t ｌ ) and the time-series true value data B(t ｂ ) and the time series luminance data L(t ｌ ) and the time series true value data B(t ｂ ) and corrects the time difference Δt between the time series true value data B(t ｂ ) and the time series luminance data L(t ｌ ) are corrected respectively.

[0075] An example of processing of the learning device and biological data estimation device according to the fourth embodiment of the present disclosure will be described. Processing similar to processing already described will be omitted here to avoid redundancy. FIG. 12 is a flowchart showing a detailed example of time difference correction processing in the processing of the learning device according to the fourth embodiment of the present disclosure. The time difference correction unit 130D, for example, executes the luminance data acquisition processing of step ST100 and the true value data acquisition processing of step ST200 already described, and obtains the time-series luminance data L(t ｌ ) and time series true value data B(t ｂ ) is acquired, the time difference correction process starts.

[0076] The time difference correction unit 130D executes a time difference calculation process (step ST320). ｌ ) and the time-series true value data B(t ｂ The time difference correction unit 130C compares the time series luminance data L(t ｌ ) and time series true value data B(t ｂ ) is used to calculate the time difference Δt that occurs due to the difference in the positions of the imaging data and the biological reference information obtained from the same subject at the same time.

[0077] The time difference correction unit 130D executes a process of correcting the true value data and the luminance data (step ST342). ｂ ) and the time series luminance data L(t ｌ For example, the time difference correction unit 130D corrects the time series true value data B(t ｂ ) and the time series luminance data L(t ｌ The time difference correction unit 130D corrects the data that is determined to be ahead in time (time-series luminance data L(t)) so that it is delayed by the time difference Δt / 2, and corrects the data that is determined to be behind in time so that it is advanced by the time difference Δt / 2. ｌ ), or time series true value data B(t ｂ) that is determined to be delayed), the time difference Δt is added to the time t, and only data D corresponding to the time difference Δt / 2 is removed from the beginning, and the data that is advanced in time (time series true value data B(t ｂ ), or time-series luminance data L(t ｌ ) that is determined to be ahead in time), data D corresponding to the time difference Δt / 2 is removed from the end. ｂｆ ) and time series luminance data L(t ｌｆ ) is calculated. The time difference Δt / 2, which is the amount of time to be corrected to delay or advance, is an example, and it is sufficient if the time difference Δt becomes 0 after correction. After performing the correction process, the time difference correction unit 130D calculates the corrected time series true value data B(t ｂｆ ) and the corrected time-series luminance data L(t ｌｆ ) to the model learning unit 140.

[0078] The time difference correction unit 130D calculates the corrected time series true value data B(t ｂｆ ) and the corrected time-series luminance data L(t ｌｆ ) to the model learning unit 140, the time difference correction process ends.

[0079] This embodiment further discloses the following configuration: the learning device, wherein the time difference correction unit compares the time-series luminance data acquired by the luminance data acquisition unit with the time-series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time-series luminance data and the time-series true value data, and corrects the time-series true value data and the time-series luminance data, respectively. This further provides an advantage that correcting both the time-series true value data and the time-series luminance data can reduce errors caused by correction compared to correcting only one of the data. Furthermore, the present disclosure provides an advantage similar to the above-described advantage by applying the above configuration to the above-described learning method.

[0080] This embodiment further discloses the following configuration: the biological data estimation device, wherein the time difference correction unit compares the time-series luminance data acquired by the luminance data acquisition unit with the time-series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time-series luminance data and the time-series true value data, and corrects the time-series true value data and the time-series luminance data, respectively. This further provides an advantage that, by correcting both the time-series true value data and the time-series luminance data, it is possible to suppress the occurrence of errors due to correction, compared to correcting only one of the data. Furthermore, by applying the above configuration to the generated data estimation method, the present disclosure provides the same advantage as the above.

[0081] Embodiment 5. In embodiment 5, an example (first example) of detailed processing for calculating a time difference in a time difference correction unit will be described. In embodiment 5, among the components according to embodiment 5, components similar to those according to embodiment 1, embodiment 2, embodiment 3, or embodiment 4 already described will be designated by the same component names and the same or similar reference numerals, and duplicate descriptions will be omitted as appropriate.

[0082] A configuration example according to embodiment 5 will be described. In this description, in order to distinguish from configuration examples of other embodiments, the reference numerals of some components will be denoted with an "E", such as a biological data estimation device 10E and a learning device 100E. The biological data estimation device 10E is configured to include a luminance data acquisition unit 110A, a true value data acquisition unit 120, a time difference correction unit 130E, a model learning unit 140, and a biological data estimation unit 150. The luminance data acquisition unit 110A, the true value data acquisition unit 120, the time difference correction unit 130E, and the model learning unit 140 constitute the learning device 100E.

[0083] The time difference correction unit 130E calculates the time series luminance data L(t ｌ ) and the time series true value data B(t ｂThe time difference correction unit 130E calculates the time difference Δt when correcting the time difference between the time difference Δt and the time difference Δt. The time difference correction unit 130E calculates the time difference Δt using cross-correlation.

[0084] An example of processing of the learning device and the biological data estimation device according to the fifth embodiment of the present disclosure will be described. Processing similar to the processing already described will be omitted here to avoid redundancy. FIG. 13 is a flowchart showing a detailed example of the time difference calculation processing in the processing of the learning device according to the fifth embodiment of the present disclosure. The time difference correction unit 130E calculates time-series luminance data L(t ｌ ) and time series true value data B(t ｂ ) is acquired, the time difference calculation process starts.

[0085] The time difference correction unit 130E executes a process of calculating the time difference Δt using cross-correlation (step ST325). Specifically, the time difference correction unit 130E first calculates the time series luminance data L(t ｌ ) and time series true value data B(t ｂ ), the cross-correlation coefficient series a = {a １ , a ２ , a ３ , ..., a ｍ Next, the time difference correction unit 130E obtains the time series luminance data L(t ｌ ) or time series true value data B(t ｂ ) to d = {d １ , d ２ , d ３ , ..., d ｎ}, a discrete Fourier transform F is applied to "d" to find the peak frequency k in the frequency power spectrum P. ｐ Calculate the peak frequency k ｐ The number of frames N for one period (for example, one pulse in the case of a pulse wave) is calculated by multiplying the inverse of ｆ Then, the time difference correction unit 130E obtains the number of frames N from the beginning in the sequence a of cross-correlation coefficients. ｆThe time difference between the positions where the maximum value is reached within a minute is obtained as the time difference Δt. The following are equations (1), (2), (3), (4), and (5) corresponding to the above-described processing. "f" indicates the sequence of data to be Fourier transformed. "F" indicates the discrete Fourier transform. "W ｈａｎｎｉｎｇ " indicates a Hanning window. "k" indicates a frequency. "N" indicates the number of data. "j" indicates a complex number. "m", "n", and "N" are each integers equal to or greater than 1. "k ｌｌ " and "k ｕｌ " is the time series luminance data L(t ｌ ) or time series true value data B(t ｂ ) is an arbitrary constant that changes depending on the frequency to be estimated for biological data. In the argmax function, if multiple elements are obtained, the first element is returned. Note that the corrected time series luminance data L(t ｌｆ ) and the corrected time series true value data B(t ｂｆ ) are in an antiphase relationship (for example, when the biological reference information indicates a continuous blood pressure waveform), in the above-mentioned cross-correlation coefficient sequence a, ｆ The correspondence can be achieved by obtaining the time difference at the position where the "minimum value" is reached within the minute as the time difference Δt.

[0086] After calculating the time difference Δt, the time difference correction unit 130E ends the time difference calculation process. The time difference correction unit 130E then proceeds to, for example, the process of step ST322, the process of step ST332, or the process of step ST342, which have already been described.

[0087] As a result, the present disclosure further provides the time-series luminance data L(t ｌ ) and time series true value data B(t ｂ Furthermore, the present disclosure can provide a configuration for calculating a time difference Δt between the input signal and the output signal using cross-correlation. Furthermore, by applying the above configuration to the learning device, the generated data estimation device, the learning method, or the generated data estimation method, the same effect as the above effect can be achieved.

[0088] Embodiment 6. In embodiment 6, an example (second example) of detailed processing for calculating the time difference in the time difference correction unit will be described. In embodiment 6, among the components according to embodiment 6, components similar to those according to embodiment 1, embodiment 2, embodiment 3, embodiment 4, or embodiment 5 already described will be designated by the same component names and the same or similar reference numerals, and duplicate descriptions will be omitted as appropriate.

[0089] A configuration example according to embodiment 6 will be described. In this description, in order to distinguish from configuration examples of other embodiments, the reference numerals of some components will be denoted with "F", such as a biological data estimation device 10F and a learning device 100F. The biological data estimation device 10F is configured to include a luminance data acquisition unit 110A, a true value data acquisition unit 120, a time difference correction unit 130F, a model learning unit 140, and a biological data estimation unit 150. The luminance data acquisition unit 110A, the true value data acquisition unit 120, the time difference correction unit 130F, and the model learning unit 140 constitute the learning device 100F.

[0090] The time difference correction unit 130F calculates the time series luminance data L(t ｌ The time difference correction unit 130F calculates the time difference Δt when correcting the time difference between the time-series true value data and the time-series true value data. The time difference correction unit 130F calculates the time difference Δt using quadrature detection.

[0091] An example of processing of the learning device and biological data estimation device according to the sixth embodiment of the present disclosure will be described. Processing similar to processing already described will be omitted here to avoid redundancy. FIG. 14 is a flowchart showing a detailed example of the time difference calculation processing in the processing of the learning device according to the sixth embodiment of the present disclosure. The time difference correction unit 130F, for example, executes the luminance data acquisition processing of step ST100 and the true value data acquisition processing of step ST200 already described, and calculates the time series luminance data L(t ｌ ) and time series true value data B(t ｂ ) is acquired, the time difference calculation process starts.

[0092] The time difference correction unit 130F executes a process of calculating the time difference Δt using quadrature detection (step ST326). Specifically, “x” and “y” are the time series luminance data L(t ｌ ) or time series true value data B(t ｂ ), and "H(x)" and "H(y)" are defined as the Hilbert transforms of "x" and "y", respectively. The time difference correction unit 130F obtains the time difference Δt by, for example, the following equation (6): Δt=tan －１ (H(y) / y)-tan －１ (H(x) / x) (6)

[0093] After calculating the time difference Δt, the time difference correction unit 130F ends the time difference calculation process. The time difference correction unit 130F then proceeds to, for example, the process of step ST322, the process of step ST332, or the process of step ST342, which have already been described.

[0094] As a result, the present disclosure further provides the time-series luminance data L(t ｌ ) and time series true value data B(t ｂ Furthermore, the present disclosure can provide a configuration for calculating a time difference Δt between the input signal and the output signal using quadrature detection. Furthermore, by applying the above configuration to the learning device, the generated data estimation device, the learning method, or the generated data estimation method, the same effect as the above effect can be achieved.

[0095] Embodiment 7. In embodiment 7, an example (third example) of detailed processing for calculating the time difference in the time difference correction unit will be described. In embodiment 7, among the components according to embodiment 6, components similar to the components according to embodiment 1, embodiment 2, embodiment 3, embodiment 4, embodiment 5, or embodiment 6 already described will be designated by the same component names and the same or similar reference numerals, and duplicate descriptions will be omitted as appropriate.

[0096] A configuration example according to embodiment 7 will be described. In this description, in order to distinguish from configuration examples of other embodiments, the reference numerals of some components will be denoted with "G", such as a biological data estimation device 10G and a learning device 100G. The biological data estimation device 10G is configured to include a luminance data acquisition unit 110A, a true value data acquisition unit 120, a time difference correction unit 130G, a model learning unit 140, and a biological data estimation unit 150. The luminance data acquisition unit 110A, the true value data acquisition unit 120, the time difference correction unit 130G, and the model learning unit 140 constitute the learning device 100G.

[0097] The time difference correction unit 130G calculates the time series luminance data L(t ｌ ) and the time series true value data B(t ｂ ) and calculates the time difference Δt. The time difference correction unit 130G calculates the time difference Δt using peak detection. Specifically, the time difference correction unit 130G first calculates the time difference Δt from the time series luminance data L(t ｌ ) and time series true value data B(t ｂ ) and obtain a series of peak positions. The time difference correction unit 130G then associates the elements of the two series of peak positions obtained above so that they form pairs. The time difference correction unit 130G then obtains the average value of the series of differences in the values ​​of the corresponding elements as the time difference Δt. Note that the peak detection method may be realized using any known technology and is not limited to the method described above.

[0098] An example of processing of the learning device and biological data estimation device according to the seventh embodiment of the present disclosure will be described. Processing similar to processing already described will be omitted here to avoid redundancy. FIG. 15 is a flowchart showing a detailed example of the time difference calculation processing in the processing of the learning device according to the seventh embodiment of the present disclosure. The time difference correction unit 130G, for example, executes the luminance data acquisition processing of step ST100 and the true value data acquisition processing of step ST200 already described, and calculates the time series luminance data L(t ｌ ) and time series true value data B(t ｂ) is acquired, the time difference calculation process starts.

[0099] The time difference correction unit 130G executes a process of calculating the time difference using peak detection (step ST327). ｌ ) and time series true value data B(t ｂ ) and obtain a series of peak positions. The time difference correction unit 130G then associates the elements of the two series of peak positions obtained above so that they form pairs. The time difference correction unit 130G then obtains the average value of the series of differences in the values ​​of the corresponding elements as the time difference Δt.

[0100] After calculating the time difference Δt, the time difference correction unit 130G ends the time difference calculation process. The time difference correction unit 130G then proceeds to, for example, the process of step ST322, the process of step ST332, or the process of step ST342, which have already been described.

[0101] As a result, the present disclosure further provides the time-series luminance data L(t ｌ ) and time series true value data B(t ｂ ) is the time difference Δt between the time series luminance data L(t ｌ ) and time series true value data B(t ｂ ) A configuration can be provided that uses each peak to calculate.

[0102] Here, a hardware configuration for realizing the functions of the present disclosure will be described. FIG. 16 is a diagram illustrating a first example of a hardware configuration for realizing the functions of the configuration of the present disclosure. FIG. 17 is a diagram illustrating a second example of a hardware configuration for realizing the functions of the configuration of the present disclosure. The biological data estimation device 10 (10A, 10B, 10C, 10D, 10E, 10F, 10G) and the learning device 100 (100A, 100B, 100C, 100D, 100E, 100F, 100G) of the present disclosure are each realized by hardware such as that shown in FIG. 16 or FIG. 17.

[0103] As shown in Fig. 16, each of the biological data estimation devices 10 (10A, 10B, 10C, 10D, 10E, 10F, and 10G) and the learning devices 100 (100A, 100B, 100C, 100D, 100E, 100F, and 100G) is configured, for example, with a processor 10001, a memory 10002, an input / output interface 10003, and a communication circuit 10004. The processor 10001 and the memory 10002 are, for example, installed in a computer. The memory 10002 stores programs that cause the computer to function as the luminance data acquisition units 110, 110A, the true value data acquisition unit 120, the time difference correction units 130 (130A, 130B, 130C, 130E, 130F, 130G), the model learning unit 140, the biometric data estimation unit 150, and a control unit (not shown). The processor 10001 reads and executes the programs stored in the memory 10002, thereby realizing the functions of the luminance data acquisition units 110, 110A, the true value data acquisition unit 120, the time difference correction units 130 (130A, 130B, 130C, 130E, 130F, 130G), the model learning unit 140, the biometric data estimation unit 150, and a control unit (not shown). Furthermore, the memory 10002 or another memory (not shown) implements the database 400 and a storage unit (not shown). Furthermore, the communication circuit 10004 implements a communication unit (not shown).

[0104] The processor 10001 is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a microprocessor, a microcontroller, or a DSP (Digital Signal Processor). The memory 10002 may be a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable Read Only Memory) or flash memory, or a magnetic disk such as a hard disk or flexible disk, or an optical disk such as a CD (Compact Disc) or DVD (Digital Versatile Disc), or a magneto-optical disk. The processor 10001 and the memory 10002 or the communication circuit 10004 are connected in a state capable of transmitting data to each other. The processor 10001, memory 10002, and communication circuit 10004 are connected via an input / output interface 10003 so as to be capable of transmitting data to and from other hardware.

[0105] Alternatively, the functions of the brightness data acquisition units 110, 110A, true value data acquisition unit 120, time difference correction units 130 (130A, 130B, 130C, 130E, 130F, 130G), model learning unit 140, biometric data estimation unit 150, and a control unit (not shown) in the biometric data estimation device 10 (10A, 10B, 10C, 10D, 10E, 10F, 10G) and the learning devices 100, 100A, 100B, 100C, 100D, 100E, 100F, 100G may be realized by a dedicated processing circuit 20001, as shown in FIG. 17 .

[0106] The processing circuit 20001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field-Programmable Gate Array), a SoC (System-on-a-Chip), or a system LSI (Large-Scale Integration), etc. In addition, a storage unit (not shown) is realized by the memory 20002 or another memory (not shown). The memory 20002 may be a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable Read Only Memory), or flash memory, or a magnetic disk such as a hard disk or flexible disk, or an optical disk such as a CD (Compact Disc) or DVD (Digital Versatile Disc), or a magneto-optical disk. The communication circuit 20004 implements a communication unit (not shown). The processing circuit 20001 and the memory 20002 or the communication circuit 20004 are connected in a state where they can transmit data to each other. In addition, the processing circuit 20001, the memory 20002, and the communication circuit 20004 are connected in a state where they can transmit data to other hardware via the input / output interface 20003.In addition, in the biometric data estimation device 10 (10A, 10B, 10C, 10D, 10E, 10F, 10G) and the learning device 100, 100A, 100B, 100C, 100D, 100E, 100F, 100G, the functions of the luminance data acquisition unit 110, 110A, true value data acquisition unit 120, time difference correction unit 130 (130A, 130B, 130C, 130E, 130F, 130G), model learning unit 140, biometric data estimation unit 150, and a control unit (not shown) may be realized by separate processing circuits, or may be realized collectively by a processing circuit.

[0107] Alternatively, in the biometric data estimation device 10 (10A, 10B, 10C, 10D, 10E, 10F, 10G) and the learning device 100, 100A, 100B, 100C, 100D, 100E, 100F, 100G, some functions of the luminance data acquisition unit 110, 110A, true value data acquisition unit 120, time difference correction unit 130 (130A, 130B, 130C, 130E, 130F, 130G), model learning unit 140, biometric data estimation unit 150, and a control unit (not shown) may be realized by the processor 10001 and memory 10002, and the remaining functions may be realized by the processing circuit 20001.

[0108] It should be noted that, within the scope of this disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0109] The present disclosure can improve the accuracy of the learning model used to estimate biometric data compared to conventional methods, and is therefore suitable for use in biometric data estimation devices that estimate the biometric data of an individual being evaluated, such as a driver of a mobile vehicle.

[0110] 10 (10A, 10B, 10C, 10D, 10E, 10F, 10G) Biometric data estimation device, 20 Output device, 100 (100A, 100B, 100C, 100D, 100E, 100F, 100G) Learning device, 110, 110A Brightness data acquisition unit, 120 True value data acquisition unit, 130 (130A, 130B, 130C, 130E, 130F, 130G) Time difference correction unit, 140 Model learning unit, 150 Biometric data estimation unit, 200 Imaging device, 300 Sensor device, 400 Database, 410A １ , 410A ２ , 410B １ , 410B ２ , 410C １ , 410C ２ Pair data (data set), 411 luminance data, 412 true value data, 10001 processor, 10002 memory, 10003 input / output interface, 10004 communication circuit, 20001 processing circuit, 20002 memory, 20003 input / output interface, 20004 communication circuit.

Claims

1. A learning device comprising: a luminance data acquisition unit that acquires time-series luminance data based on time-series imaging data; a true-value data acquisition unit that acquires time-series true-value data using time-series biological reference information based on a biological signal; a time difference correction unit that corrects the time difference between the time-series luminance data and the time-series true-value data; and a model learning unit that constructs a learning model that outputs biological data from the time-series luminance data, using the time-series luminance data and the time-series true-value data output by the time difference correction unit.

2. The learning device according to claim 1, wherein the time difference correction unit compares the time series luminance data acquired by the luminance data acquisition unit with the time series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time series luminance data and the time series true value data, and corrects the time series true value data based on the time series luminance data.

3. The learning device according to claim 1, wherein the time difference correction unit compares the time series luminance data acquired by the luminance data acquisition unit with the time series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time series luminance data and the time series true value data, and corrects the time series luminance data based on the time series true value data.

4. The learning device according to claim 1, wherein the time difference correction unit compares the time series luminance data acquired by the luminance data acquisition unit with the time series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time series luminance data and the time series true value data, and corrects the time series true value data and the time series luminance data, respectively.

5. The learning device according to any one of claims 1 to 4, wherein the biological reference information is a pulse wave.

6. The learning device according to any one of claims 1 to 4, wherein the biological reference information is a continuous blood pressure waveform.

7. The learning device according to any one of claims 1 to 4, wherein the biological reference information is an electrocardiogram waveform.

8. A learning device as claimed in any one of claims 1 to 4, which acquires combinations of time-series luminance data and time-series true value data for each of a plurality of persons in sequence for each combination, and executes processing for each combination by the luminance data acquisition unit, the true value data acquisition unit, the time difference correction unit, and the model learning unit.

9. A biological data estimation device comprising: a luminance data acquisition unit that acquires time-series luminance data based on time-series imaging data; a true value data acquisition unit that acquires time-series true value data using time-series biological reference information based on a biological signal; a time difference correction unit that corrects a time difference between the time-series luminance data and the time-series true value data; a model learning unit that constructs a learning model that outputs biological data from the time-series luminance data using the time-series luminance data and the time-series true value data output by the time difference correction unit; and a biological data estimation unit that estimates biological data based on the learning model constructed by the model learning unit and the time-series luminance data acquired by the luminance data acquisition unit.

10. The biological data estimation device according to claim 9, wherein the time difference correction unit compares the time series luminance data acquired by the luminance data acquisition unit with the time series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time series luminance data and the time series true value data, and corrects the time series true value data based on the time series luminance data.

11. The biological data estimation device according to claim 9, wherein the time difference correction unit compares the time series luminance data acquired by the luminance data acquisition unit with the time series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time series luminance data and the time series true value data, and corrects the time series luminance data based on the time series true value data.

12. The biological data estimation device according to claim 9, wherein the time difference correction unit compares the time series luminance data acquired by the luminance data acquisition unit with the time series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time series luminance data and the time series true value data, and corrects the time series true value data and the time series luminance data, respectively.

13. The biological data estimation device according to any one of claims 9 to 12, wherein the biological reference information is a pulse wave.

14. The biological data estimation device according to any one of claims 9 to 12, wherein the biological reference information is a continuous blood pressure waveform.

15. The biological data estimation device according to any one of claims 9 to 12, wherein the biological reference information is an electrocardiogram waveform.

16. A biometric data estimation device as described in any one of claims 9 to 12, wherein the luminance data acquisition unit acquires time-series luminance data using image data obtained by imaging a driver of a moving body in a time series, and the biometric data estimation unit estimates the biometric data of the driver.

17. The biometric data estimation device according to claim 13, wherein the luminance data acquisition unit acquires time-series luminance data using image data obtained by imaging a driver of a moving body in a time series, and the biometric data estimation unit estimates the biometric data of the driver.

18. The biometric data estimation device according to claim 14, wherein the luminance data acquisition unit acquires time-series luminance data using imaging data obtained by imaging a driver of a moving body in a time series manner, and the biometric data estimation unit estimates the biometric data of the driver.

19. The biometric data estimation device according to claim 15, wherein the luminance data acquisition unit acquires time-series luminance data using image data obtained by imaging a driver of a moving body in a time series, and the biometric data estimation unit estimates the biometric data of the driver.

20. A biometric data estimation device as described in any one of claims 9 to 12, wherein a combination of time-series luminance data and time-series true value data for each of a plurality of persons is acquired in sequence for each combination, and processing is performed for each combination by the luminance data acquisition unit, the true value data acquisition unit, the time difference correction unit, and the model learning unit.

21. A learning method executed by a learning device, comprising: a step in which a luminance data acquisition unit of the learning device acquires time-series luminance data based on time-series imaging data; a step in which a true value data acquisition unit of the learning device acquires time-series true value data using time-series biological reference information based on biological signals; a step in which a time difference correction unit of the learning device corrects the time difference between the time-series luminance data and the time-series true value data; and a step in which a model learning unit of the learning device constructs a learning model that outputs biological data from the time-series luminance data, using the time-series luminance data and the time-series true value data output by the time difference correction unit.

22. A biometric data estimation method executed by a biometric data estimation device, comprising: a step in which a luminance data acquisition unit of the biometric data estimation device acquires time-series luminance data based on time-series imaging data; a step in which a true value data acquisition unit of the biometric data estimation device acquires time-series true value data using time-series biometric reference information based on a biometric signal; a step in which a time difference correction unit of the biometric data estimation device corrects the time difference between the time-series luminance data and the time-series true value data; a step in which a model learning unit of the biometric data estimation device constructs a learning model that outputs biometric data from the time-series luminance data, using the time-series luminance data and the time-series true value data output by the time difference correction unit; and a step in which the biometric data estimation unit of the biometric data estimation device estimates biometric data based on the learning model constructed by the model learning unit and the time-series luminance data acquired by the luminance data acquisition unit.

Images