Learning device, biological data estimation device, learning method, and biological data estimation method

Abstract

To improve accuracy of a learning model used for estimation of biological data more than before.SOLUTION: A learning device comprises: a brightness data acquisition unit which acquires time-series brightness data based on time-series captured data; a true value data acquisition unit which acquires time-series true value data by using time-series biological reference information based on a biological signal; a time difference correction unit which corrects time difference between the time-series brightness data and the time-series true value data; and a model learning unit which constructs a learning model which outputs biological data from the time-series brightness data, by using the time-series brightness data and the time-series true value data outputted by the time difference correction unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The disclosed technology relates to a technique for constructing a learning model for estimating biological data.

Background Art

[0002] Among the techniques for estimating biological data, for example, as described in Patent Document 1, there is a method of using a series of face images (time-series images) of a subject to estimate biological data such as a pulse wave from minute luminance changes on the face surface. In Patent Document 1, a machine learning model is used in an estimator, and the relationship between the face image series and the pulse wave is learned so as to eliminate the error between the estimated value by the machine learning model and the correct value calculated based on the chrominance signal of the image, and model construction is performed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the chrominance signal of an image may vary greatly depending on the environment outside the device such as the state of the object itself included in the image (for example, the orientation of the subject's face) (hereinafter, the "environment outside the device" is also referred to as the "external environment"). Therefore, in the technique described in Patent Document 1, when constructing a learning model using the correct value calculated based on the chrominance signal of the image, there is a problem that it is easily affected by the external environment and the accuracy of the learning model is likely to be low.

[0005] The present disclosure aims to solve the above problems and improve the accuracy of the learning model used for estimating biological data as compared with the prior art.

Means for Solving the Problem

[0006] The learning device of the present disclosure includes a luminance data acquisition unit that acquires time-series luminance data based on time-series imaging data, a true value data acquisition unit that acquires time-series true value data using time-series biometric reference information based on a biometric signal, a time difference correction unit that corrects the time difference between the time-series luminance data and the time-series true value data, and a model learning unit that constructs a learning model that outputs biometric data from the time-series luminance data using the time-series luminance data and the time-series true value data output by the time difference correction unit. It is equipped with the above components.

Advantages of the Invention

[0007] According to the present disclosure, there is an effect that the accuracy of the learning model used for estimating biometric data can be improved compared to the prior art.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, in order to explain the present disclosure in more detail, embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0010] Embodiment 1. In Embodiment 1, the basic form of the present disclosure will be described.

[0011] A configuration example of the learning device according to Embodiment 1 of the present disclosure will be described. FIG. 1 is a diagram showing a basic configuration example of the learning device 100 of the present disclosure. FIG. 2 is a diagram showing an example of the acquisition source of the data used by the learning device 100 and the learning device 100 of the present disclosure. FIG. 3 is a diagram showing an example of the acquisition source of the data used by the learning device 100 and the learning device 100 of the present disclosure. The learning device 100 constructs a learning model for estimating biological data. The learning device 100 constructs a learning model by learning using time-series luminance data based on time-series imaging data and time-series true value data obtained using time-series biological reference information based on biological signals. Here, when simply trying to perform learning using time-series luminance data based on time-series imaging data and time-series true value data obtained using time-series biological reference information based on biological signals, a time difference (for example, when the biological data is a pulse wave, the difference in pulse wave propagation time) occurs between the reference location of the correct value (true value) and the face, and this difference may vary due to individual differences and the like. As a result, there is a high possibility that the learning model cannot stably learn the original input-output relationship, leading to a decrease in the estimation accuracy by the learning model. Therefore, the learning device 100 of the present disclosure performs correction related to the time difference and constructs a learning model using the corrected data. The learning device 100 shown in each of FIGS. 1, 2, and 3 includes a luminance data acquisition unit 110, a true value data acquisition unit 120, a time difference correction unit 130, and a model learning unit 140.

[0012] The luminance data acquisition unit 110 acquires time-series luminance data based on time-series imaging data. The imaging data is a series of frames "Im(t i )" (i = 1, 2, 3, ···) that represents captured images of a space that includes the skin area of a subject (the "subject" will also be described as the "evaluation target person" or the "user" in the following explanations), captured at a predetermined frame rate Fr. The imaging data includes information such as time, for example. The luminance data acquisition unit 110 acquires luminance data L(t i )(i = 1, 2, 3, ···) indicating the brightness change of the skin based on the frames Im(t l )(i = 1, 2, 3, ···) included in the aforementioned imaging data, in the same manner as processes such as skin area detection, measurement area setting, and extraction of the pulse wave source signal, which are known techniques, for example. The luminance data L(t l )(i = 1, 2, 3, ···) includes information such as time that can be synchronized with time-series true value data described later. Here, there are two types of luminance data: luminance data not used for evaluating the learning model described later and luminance data used for evaluating the learning model. Hereinafter, when distinguishing each luminance data, they will be described as "time-series luminance data" and "time-series luminance data for evaluation", respectively. Note that it is assumed that there is always one or more pieces of time-series true value data described later corresponding to the time-series luminance data L(t l ) used for learning.

[0013] When configured as shown in FIG. 2, the luminance data acquisition unit 110 acquires time-series luminance data L(t l ) based on the imaging data captured by the imaging device 200. Specifically, the luminance data acquisition unit 110 acquires the imaging data output from the imaging device 200 and uses the imaging data to acquire time-series luminance data L(t l ).

[0014] When configured as shown in FIG. 3, the luminance data acquisition unit 110 acquires time-series luminance data L(t l ) stored in the database 400.

[0015] When constructing a learning model for estimating the biological data of an occupant (e.g., a driver) of a moving body, the luminance data acquisition unit 110 uses imaging data obtained by imaging the occupant (driver) of the moving body in time series to obtain time series luminance data L(t l ).

[0016] The true value data acquisition unit 120 uses time series biological reference information P(t p ) based on a biological signal to obtain time series true value data B(t b ). A biological signal is a signal obtained by contact measurement of a living body in time series, and is, for example, a pulse wave, a continuous blood pressure waveform, or an electrocardiogram waveform. Biological reference information is a continuous value "P(t p )" representing a biological signal (e.g., a pulse wave, a blood pressure waveform, or an electrocardiogram waveform) of a subject obtained at a predetermined time interval "Ts" at the same time as the imaging data. That is, when the biological signal is a measurement result of a pulse wave, the biological reference information P(t p ) indicates a pulse wave. That is, when the biological signal is a measurement result of blood pressure, the biological reference information P(t p ) indicates a continuous blood pressure waveform. That is, when the biological signal is a measurement result of an electrocardiogram waveform, the biological reference information P(t p ) indicates an electrocardiogram waveform. The true value data acquisition unit 120 obtains a continuous value that is the same subject and at the same time as the time series luminance data L(t p ) for each frame Im(t i ) from the biological reference information P(t l ). This value is defined as time series true value data B(t b ). The time series true value data B(t b ) includes information that can be synchronized with the time series luminance data L(t l ), such as time, for example.

[0017] Here, as shown in FIG. 2, when the learning device 100 is configured to receive the signal output from the sensor device 300, the true value data acquisition unit 120 generates time-series true value data B(t b ) based on the biological signal output by the sensor device 300. Specifically, the true value data acquisition unit 120 acquires time-series biological reference information P(t p ) based on the biological signal output from the sensor device 300, and uses the biological reference information P(t p ) to acquire time-series true value data B(t b ).

[0018] As shown in FIG. 3, when the learning device 100 is configured to refer to the information stored in the database 400, the true value data acquisition unit 120 acquires the time-series true value data B(t b ) stored in the database 400.

[0019] The time difference correction unit 130 corrects the time difference Δt between the time-series luminance data L(t l ) and the time-series true value data B(t b ). FIG. 4A is a diagram for explaining the time-series luminance data L(t l ) and the time-series true value data B(t b ) used by the learning device 100 of the present disclosure. FIG. 4B is a diagram showing an image of time difference correction in the learning device 100 of the present disclosure. FIGS. 4A and 4B show an example in which the biological reference information P(t p ) is a pulse wave (PPG: photoplethysmography). As shown in FIG. 4A, due to the difference in the measurement environment (external environment which is the environment outside the device) of the biological reference information based on the biological signal, when acquiring the true value data B(t b ) and the luminance data L(t l ) based on the biological signal for use in learning, correction of the time difference Δt is required. The time difference correction unit 130 corrects the time difference Δt, and as shown in FIG. 4B, corrects the true value data B(t b ) before correction to the true value data B(tbf ) is corrected to obtain the time-series luminance data L(t lf ) and the time-series true value data B(t bf ) after the correction process, and outputs them. Here, an example corresponding to the case where the corrected time-series luminance data L(t lf ) and the corrected time-series true value data B(t bf ) have an inverse phase relationship will be described. FIG. 4C is a diagram showing an image when the time-series luminance data and the time-series true value data have an inverse phase. As shown in FIG. 4B, when the biological reference information P(t p ) is, for example, a pulse wave (PPG), they are in the same phase. However, as shown in FIG. 4C, when the biological reference information P(t p ) is, for example, the time-series true value data of a continuous blood pressure waveform, it can be assumed that the corrected time-series luminance data L(t lf ) and the corrected time-series true value data B(t bf ) have an inverse phase relationship. Therefore, when the biological reference information P(t p ) is such that an inverse phase relationship is formed, the time difference correction unit 130 is configured to perform correction taking into account the phase difference. For example, when the time difference correction unit 130 performs time difference correction using cross-correlation (see Embodiment 5 described later), the time difference correction unit 130 obtains the time difference Δt as the time difference at the position where the minimum value is taken within the number of frames N f from the head in the series a of cross-correlation coefficients. Based on the above concept, the time difference correction unit 130 performs correction related to the time difference Δt on the time-series luminance data L(t l ) and the time-series true value data B(t b ). That is, the time difference correction unit 130 corrects the time difference Δt between the time-series luminance data L(t l ) and the time-series true value data B(t b ). The time difference correction unit 130 outputs the corrected time-series luminance data L(t lf ) and the time-series true value data B(t bf ) after performing the correction related to the time difference Δt to the model learning unit 140.

[0020] The model learning unit 140 uses the time-series luminance data L(t lf ) and the time-series true value data B(t bf ) to construct a learning model M that outputs biological data from the time-series luminance data L(t l ). The model learning unit 140 receives the time-series luminance data L(t lf ) and the time-series true value data B(t bf ), constructs a model M (learning model) that estimates biological data from the time-series luminance data L(t l ) using the time-series true value data as teacher data, and outputs the model M (learning model). The model M is a learning model trained to use the time-series luminance data L(t l ) as input data and output biological data E(t e ). Note that the learning method is not particularly specified. For example, a learning method learned from the following literature can be used. Literature: "Turnip: Time-Series U-Net With Recurrence For Nir Imaging Ppg (signalprocessingsociety.org)"

[0021] In addition to the above configuration, the learning device 100 includes a control unit (not shown), a storage unit (not shown), and a communication unit (not shown). The control unit (not shown) controls the entire learning device 100 and each component. The control unit (not shown) activates the learning device 100 according to a command from the outside, for example. Also, the control unit (not shown) controls the state of the learning device 100 (operating states such as startup, shutdown, and sleep). The storage unit (not shown) stores each data used in the learning device 100. The storage unit (not shown) stores, for example, the output (output data) by each component in the learning device 100, and outputs the data requested for each component to the component that requested it. A communication unit (not shown) communicates with an external device. For example, it communicates between the learning device 100 and a peripheral device (for example, at least one of an imaging device, a sensor device, a database, or an output device). For example, when the learning device 100 and the peripheral device are not connected by wire, the communication unit (not shown) has a function of communicating between the learning device 100 and the peripheral device. Also, the communication unit (not shown) has a function of communicating with an external device. The control unit (not shown), the storage unit (not shown), and the communication unit (not shown) are the same in the configurations of other embodiments described later.

[0022] Here, the learning device 100 may be configured to acquire combinations of time-series luminance data L(t l ) and time-series true value data B(t b ) for a plurality of evaluation subjects and use them for model learning. An evaluation subject is a user who can be an evaluation target. For example, in the case of a learning model for estimating biometric data of a driver of a moving body, it is a user who can be a driver. For example, combinations of time-series luminance data and time-series true value data are prepared in advance for each of the same evaluation subjects. The database 400 shown in FIG. 3 stores combinations of time-series luminance data L(t l ) and time-series true value data B(t b ) for each of the same evaluation subjects. The learning device 100 acquires time-series luminance data L(t l ) and time-series true value data B(t b ) for each combination from the database 400 and learns to construct a learning model. FIG. 5 is a diagram showing an image of a data set including time-series luminance data L(t l ) and time-series true value data B(t b ) used by the learning device 100 of the present disclosure. In FIG. 5, "time-series luminance data" is simply described as "luminance data", and "time-series true value data" is simply described as "true value data". The database shown in FIG. 3 holds a dataset as shown in the image of FIG. 5, for example. The dataset shown in FIG. 5 holds paired data 410A1 and paired data 410A2, which are combinations of the time-series luminance data L(t l )(“411” shown in FIG. 5) of user A and the time-series true value data B(t b )(“412” shown in FIG. 5), holds paired data 410B1 and paired data 410B2, which are combinations of the time-series luminance data L(t l ) of user B and the time-series true value data B(t b ), and holds paired data 410C1 and paired data 410C2, which are combinations of the time-series luminance data L(t l ) of user C and the time-series true value data B(t b ). The dataset shown in FIG. 5 shows two sets of paired data for each of the three people, but is not limited to this example, and may include paired data for two people or four or more people, or may include two or more sets of paired data per person. When configured in this way, the learning device 100 sequentially acquires combinations of the time-series luminance data L(t l ) and the time-series true value data B(t b ) for each person, and for each combination, executes processing by the luminance data acquisition unit 110, the true value data acquisition unit 120, the time difference correction unit 130, and the model learning unit 140.

[0023] An example of the processing of the learning device 100 will be described. FIG. 6 is a flowchart showing an example of the processing of the learning device 100 of the present disclosure. When the learning device 100 receives a learning start command from the outside, for example, it starts the processing shown in FIG. 6.

[0024] The learning device 100 executes luminance data acquisition processing (step ST100). In the luminance data acquisition processing, the luminance data acquisition unit 110 of the learning device 100 acquires time-series luminance data L(t l) is acquired.

[0025] The learning device 100 executes true value data acquisition processing (step ST200). In the true value data acquisition processing, the true value data acquisition unit 120 of the learning device 100 uses time-series biological reference information based on biological signals to obtain time-series true value data B(t b ) is acquired.

[0026] The learning device 100 executes time difference correction processing (step ST300). In the time difference correction processing, the time difference correction unit 130 of the learning device 100 corrects the time difference Δt between the time-series luminance data L(t l ) and the time-series true value data B(t b ). The time difference correction unit 130 receives the time-series luminance data L(t l ) and the time-series true value data B(t b ). The time difference correction unit 130 then calculates the time difference Δt generated by the difference in the positions of the acquisition locations of each of the imaging data and the biological reference information for the same subject and at the same time based on the time-series luminance data L(t l ) and the time-series true value data B(t b ). The time difference correction unit 130 then checks the relative delay relationship between the time-series luminance data L(t l ) and the time-series true value data B(t b ). Specifically, the time difference correction unit 130 determines which of the time-series luminance data L(t l ) and the time-series true value data B(t b ) is the data on the delayed side and which is the data on the advancing side. The time difference correction unit 130 then adds the time difference Δt to the time t in the data with the delayed time (either the time-series luminance data L(t l ), or the time-series true value data B(t b )) and removes data by the time difference Δt from the beginning, and for the data with the advancing time (the time-series luminance data L(t l ), or the time-series true value data B(tb ) In either one of the data), data removal is performed from the end by a time difference Δt. The time difference correction unit 130 outputs the time-series luminance data L(t lf ), and the time-series true value data B(t bf ) to the model learning unit 140. Here, "t" is the time t l or the time t b , and the time after time difference correction is such that the time-series luminance data is at time t lf , and the time-series true value data is at time t bf . For example, when the biological reference information is delayed information, the time t of the time-series luminance data after correction lf is "t l ", and the time t of the true value data after correction bf is "t b + Δt" (see FIG. 10 described later). Note that in the present disclosure, the method for confirming the relative delay relationship between the imaging data and the biological reference information is not particularly limited, but the delay relationship itself can be, for example, the delay relationship as in Embodiment 2, Embodiment 3, or Embodiment 4 described later.

[0027] The learning device 100 executes model learning processing (step ST400). In the model learning processing, the model learning unit 140 of the learning device 100 constructs a learning model that outputs biological data from the time-series luminance data using the time-series luminance data and the time-series true value data output by the time difference correction unit.

[0028] After executing the model learning processing (step ST400), the learning device 100 then proceeds to the end determination processing (step ST500). In the end determination processing, a control unit (not shown) of the learning device 100 determines whether to end the processing of the learning device 100. The control unit (not shown) determines whether to end the processing of the learning device 100, for example, according to an end command from the outside or an execution program. When the control unit (not shown) determines that the processing of the learning device 100 is not completed (step ST500 “NO”), the process proceeds to the process of step ST100, and the repetitive process is performed from the process of step ST100. When the control unit (not shown) determines that the processing of the learning device 100 is completed (step ST500 “YES”), the learning device 100 ends the processing.

[0029] Next, a configuration example of the biological data estimation device including the learning device will be described. FIG. 7 is a diagram showing a configuration example when the learning device of the present disclosure is applied to a biological data estimation device. In the description here, in order to distinguish from the configuration examples of other forms, the description using the symbols of some component parts with “A” attached, such as the biological data estimation device 10A and the learning device 100A, will be used for the description. The biological data estimation device 10A receives imaging information that is a video composed of a series of frames Im(t i ) that images a space including the skin area of the subject at a predetermined frame rate Fr, and biological reference information that is continuous values P(t p ) representing the biological signals (for example, pulse wave, blood pressure waveform, electrocardiogram waveform) of the subject acquired at a predetermined time interval Ts at the same time as the imaging information. Then, the biological data estimation device 10A outputs continuous values E(t f -N i +1) to E(t f +1) to E(t i ) as the estimation results E(t e ) of the biological signals corresponding to a certain series of frames Im(t i -N f +1) to Im(t i ) every certain number of frames N. Here, “t i ” indicates the frame number assigned to each frame. “t p ” indicates the input number assigned to the continuous value that is the input biological reference information. “t i ”, “t p ”, and “te is an integer of 1 or more. Number of frames N f is an integer of 2 or more. It is assumed that the number of subjects included in the imaging data and the biological reference information is 1 or more. The imaging data may be any color information (for example, RGB color or monochrome).

[0030] The biological data estimation device 10A includes a luminance data acquisition unit 110A, a true value data acquisition unit 120, a time difference correction unit 130A, a model learning unit 140, and a biological data estimation unit 150. The luminance data acquisition unit 110A, the true value data acquisition unit 120, the time difference correction unit 130A, and the model learning unit 140 constitute the learning device 100A.

[0031] The true value data acquisition unit 120, the time difference correction unit 130A, or the model learning unit 140 has the same functions as the true value data acquisition unit 120, the time difference correction unit 130, or the model learning unit 140 already described.

[0032] In addition to the functions of the luminance data acquisition unit 110 already described, the luminance data acquisition unit 110A outputs the acquired time-series luminance data to the biological data estimation unit 150. When the biological data estimation device 10A executes the process of constructing a learning model, the luminance data acquisition unit 110A outputs the acquired time-series luminance data L(t l ) to the time difference correction unit 130A, and when the biological data estimation device 10A executes the process of estimating biological data, the acquired time-series luminance data L(t l )(time-series luminance data for evaluation) is output to the biological data estimation unit 150.

[0033] The biological data estimation unit 150 estimates biological data E(t l ) based on the learning model M constructed by the model learning unit 140 and the time-series luminance data L(t e ) (time-series luminance data for evaluation) acquired by the luminance data acquisition unit 110A. The biological data estimation unit 150 receives the model M constructed by the model learning unit 140 and the time-series luminance data L(t l ), inputs the time-series luminance data L(t l ) for evaluation into the aforementioned model M, estimates the biological data, and outputs the biological data E(t e ) which is the estimation result.

[0034] When constructing a learning model for estimating the biological data of an occupant (e.g., a driver) of a moving body, the luminance data acquisition unit 110A acquires the time-series luminance data L(t l ) using the imaging data obtained by imaging the occupant (driver) of the moving body in time series. The biological data estimation unit 150 estimates the biological data of the occupant (driver).

[0035] An example of the processing of the biological data estimation device 10A will be described. Among the processes of the biological data estimation device 10A, the process corresponding to the learning device 100A for constructing the learning model is the same as the process of the learning device 100 already described, so a detailed description is omitted here. Among the processes of the biological data estimation device 10A, the process corresponding to the learning device 100A for constructing the learning model is included in a part of the biological data estimation method.

[0036] FIG. 8 is a flowchart showing an example of the processing of the biological data estimation device 10(10A). The process shown in FIG. 8 is a part of the biological data estimation method by the biological data estimation device 10(10A). When the biological data estimation device 10(10A) receives a command to estimate generated data from the outside, for example, it starts the process shown in FIG. 8. Or, when the biological data estimation device 10(10A) is for estimating the biological data of an occupant of a moving body, for example, when the power source of the moving body starts, it starts the process shown in FIG. 8.

[0037] The biological data estimation device 10(10A) first executes luminance data acquisition processing (step ST1100). In the luminance data acquisition process, the luminance data acquisition unit 110A of the biological data estimation device 10 (10A) obtains time-series luminance data L(t l ) based on time-series imaging data. The luminance data acquisition unit 110A outputs the acquired time-series luminance data L(t l ) to the biological data estimation unit 150.

[0038] The biological data estimation device 10 (10A) then executes a biological data estimation process (step ST1200). In the biological data estimation process, the biological data estimation unit 150 of the biological data estimation device 10 (10A) estimates biological data E(t l ) based on the learning model M constructed by the model learning unit 140 and the time-series luminance data L(t e ) acquired by the luminance data acquisition unit 110A.

[0039] When the biological data estimation device 10 (10A) outputs the biological data E(t e ) as the estimation result by the biological data estimation unit 150, it then proceeds to an end determination process (step ST1300). In the end determination process, a control unit (not shown) of the biological data estimation device 10 (10A) determines whether to end the processing of the biological data estimation device. The control unit (not shown) determines, for example, whether to end the processing of the biological data estimation device 10 (10A) according to an end command from the outside or an execution program. If the control unit (not shown) determines not to end the processing of the biological data estimation device 10 (10A) (step ST1300 “NO”), the process proceeds to the process of step ST1100, and repetitive processing is performed from the process of step ST1100. If the control unit (not shown) determines to end the processing of the biological data estimation device 10 (10A) (step ST1300 “YES”), the biological data estimation device 10 (10A) ends the processing.

[0040] As described above, according to the learning device or the biological data estimation device of the present disclosure, regarding the learning model used for biological data estimation, in order to exclude the variation in the time difference for each piece of data such as individual differences included in the data for learning model construction, by correcting the time difference between the luminance data and the true value data used for model construction, it leads to the model learning the original input-output relationship, and the biological data estimation accuracy can be improved. In addition, a robust learning model that is less affected by external factors can be constructed.

[0041] In the present embodiment, the following configuration is disclosed. A luminance data acquisition unit that acquires time-series luminance data based on time-series imaging data, A true value data acquisition unit that acquires time-series true value data using time-series biological reference information based on a biological signal, A time difference correction unit that corrects the time difference between the time-series luminance data and the time-series true value data, A model learning unit that constructs a learning model that outputs biological data from the time-series luminance data using the time-series luminance data and the time-series true value data output by the time difference correction unit, A learning device comprising: Thereby, the present disclosure can provide a learning device that can improve the accuracy of the learning model used for biological data estimation as compared with the prior art.

[0042] In the present embodiment, the following configuration is disclosed. A luminance data acquisition unit that acquires time-series luminance data based on time-series imaging data, A true value data acquisition unit that acquires time-series true value data using time-series biological reference information based on a biological signal, A time difference correction unit that corrects the time difference between the time-series luminance data and the time-series true value data, A model learning unit that constructs a learning model that outputs biological data from time-series luminance data using the time-series luminance data and time-series true value data output by the time difference correction unit; A biological data estimation unit that estimates biological data based on the learning model constructed by the model learning unit and the time-series luminance data acquired by the luminance data acquisition unit; A biological data estimation device comprising: As a result, the present disclosure can provide a biological data estimation device that can improve the accuracy of a learning model used for estimating biological data as compared with the prior art. In addition, the present disclosure can provide a biological data estimation device that can improve the accuracy of the biological data to be estimated as compared with the prior art.

[0043] In the present embodiment, the following configuration is disclosed. A learning method executed by a learning device, A step of acquiring, by a luminance data acquisition unit of the learning device, time-series luminance data based on time-series imaging data; A step of acquiring, by a true value data acquisition unit of the learning device, time-series true value data using time-series biological reference information based on a biological signal; A step of correcting, by a time difference correction unit of the learning device, a time difference between the time-series luminance data and the time-series true value data; A step of constructing, by a model learning unit of the learning device, a learning model that outputs biological data from time-series luminance data using the time-series luminance data and time-series true value data output by the time difference correction unit; A learning method comprising: As a result, the present disclosure can provide a learning method that can improve the accuracy of a learning model used for estimating biological data as compared with the prior art.

[0044] In this embodiment, the following configuration is disclosed. A biological data estimation method executed by a biological data estimation device, a step in which a luminance data acquisition unit of the learning device acquires time-series luminance data based on time-series imaging data; a step in which a true value data acquisition unit of the learning device acquires time-series true value data using time-series biological reference information based on a biological signal; a step in which a time difference correction unit of the learning device corrects a time difference between the time-series luminance data and the time-series true value data; a step in which a model learning unit of the learning device constructs a learning model that outputs biological data from time-series luminance data using the time-series luminance data and time-series true value data output by the time difference correction unit; a step in which a biological data estimation unit of the learning device estimates biological data based on the learning model constructed by the model learning unit and the time-series luminance data acquired by the luminance data acquisition unit; A biological data estimation method comprising the above. As a result, the present disclosure can provide a biological data estimation method that can improve the accuracy of a learning model used for estimating biological data as compared with the conventional method. In addition, the present disclosure can provide a biological data estimation method that can improve the accuracy of the biological data to be estimated as compared with the conventional method.

[0045] In this embodiment, further, the following configuration is disclosed. The biological reference information is a pulse wave, characterizing the learning device. As a result, the present disclosure can further provide a learning device that can improve the accuracy of a learning model used for estimating a pulse wave, which is biological data, as compared with the conventional method. Furthermore, by applying the above configuration to the above learning method, the present disclosure achieves the same effect as the above effect.

[0046] In the present embodiment, the following configuration is further disclosed. The biological reference information is a continuous blood pressure waveform, characterizing the learning device. Thereby, the present disclosure can further provide a learning device that can improve the accuracy of a learning model used for estimating blood pressure, which is biological data, as compared with the prior art. Furthermore, by applying the above configuration to the above learning method, the present disclosure achieves the same effect as the above effect.

[0047] In the present embodiment, the following configuration is further disclosed. The biological reference information is an electrocardiogram waveform, characterizing the learning device. Thereby, the present disclosure can further provide a learning device that can improve the accuracy of a learning model used for estimating the waveform of an electrocardiogram, which is biological data, as compared with the prior art.

[0048] In the present embodiment, the following configuration is further disclosed. Acquire in order, for each combination, the combination of time-series luminance data and time-series true value data for each of a plurality of persons, and for each combination, execute processing by the luminance data acquisition unit, the true value data acquisition unit, the time difference correction unit, and the model learning unit. characterizing the learning device. Thereby, the present disclosure can further provide a learning device that can improve the accuracy of a learning model used for estimating biological data as compared with the prior art. Furthermore, by applying the above configuration to the above learning method, the present disclosure achieves the same effect as the above effect.

[0049] In the present embodiment, the following configuration is further disclosed. The biological reference information is a pulse wave. A biological data estimation device characterized by this. Thereby, the present disclosure can provide a biological data estimation device that can further improve the accuracy of a learning model used for estimating a pulse wave, which is biological data, as compared with the conventional art. In addition, the present disclosure can provide a biological data estimation device that can further improve the accuracy of estimating a pulse wave, which is biological data, as compared with the conventional art. Furthermore, by applying the above configuration to the above generation data estimation method, the present disclosure achieves the same effect as the above effect.

[0050] In the present embodiment, the following configuration is further disclosed. The biological reference information is a continuous blood pressure waveform. A biological data estimation device characterized by this. Thereby, the present disclosure can provide a biological data estimation device that can further improve the accuracy of a learning model used for estimating blood pressure, which is biological data, as compared with the conventional art. In addition, the present disclosure can provide a biological data estimation device that can further improve the accuracy of estimating blood pressure, which is biological data, as compared with the conventional art. Furthermore, by applying the above configuration to the above generation data estimation method, the present disclosure achieves the same effect as the above effect.

[0051] In the present embodiment, the following configuration is further disclosed. The biological reference information is an electrocardiogram waveform, characterizing a biological data estimation device. Thereby, the present disclosure can further provide a biological data estimation device capable of improving the accuracy of a learning model used for estimating the waveform of an electrocardiogram, which is biological data, as compared with the conventional art, and has the effect of achieving such an effect. In addition, the present disclosure can further provide a biological data estimation device capable of improving the accuracy of estimating the waveform of an electrocardiogram, which is biological data, as compared with the conventional art, and has the effect of achieving such an effect. Furthermore, the present disclosure applies the above configuration to the above-described generated data estimation method, thereby achieving the same effect as the above effect.

[0052] In the present embodiment, the following configuration is further disclosed. The luminance data acquisition unit acquires time-series luminance data using imaging data obtained by imaging the driver of the moving body in time series, The biological data estimation unit estimates the biological data of the driver, characterizing a biological data estimation device. Thereby, the present disclosure can further provide a biological data estimation device capable of improving the accuracy of estimating the biological data of the driver of the moving body as compared with the conventional art, and has the effect of achieving such an effect. In addition, the present disclosure can further provide an effect that, when applied to a passenger monitoring device for monitoring a passenger, for example, the accuracy of detecting an abnormality of the driver can be improved as compared with the conventional art. Furthermore, the present disclosure applies the above configuration to the above-described generated data estimation method, thereby achieving the same effect as the above effect.

[0053] In the present embodiment, the following configuration is further disclosed. For each combination of time-series luminance data and time-series true value data for a plurality of persons, the combinations are sequentially acquired, and for each combination, processing by the luminance data acquisition unit, the true value data acquisition unit, the time difference correction unit, and the model learning unit is executed. A biological data estimation device characterized by the above. Accordingly, the present disclosure can provide a biological data estimation device that can further improve the accuracy of the learning model used for estimating biological data as compared with the prior art. Furthermore, the present disclosure has the same effect as the above effect by applying the above configuration to the generated data estimation method.

[0054] Embodiment 2. Embodiment 2 is a form of a more detailed configuration example (first example) of the configuration related to the time difference correction in Embodiment 1. In Embodiment 2, among the components related to Embodiment 2, for components having the same configuration as those related to Embodiment 1 already described, the same component names and the same or similar reference numerals are used, and duplicate descriptions are appropriately omitted.

[0055] A configuration example according to Embodiment 2 will be described. In the description here, in order to distinguish from the configuration examples of other forms, the description will be made using notations in which "B" is added to the reference numerals of some components such as the biological data estimation device 10B and the learning device 100B. The biological data estimation device 10B includes a luminance data acquisition unit 110A, a true value data acquisition unit 120, a time difference correction unit 130B, a model learning unit 140, and a biological data estimation unit 150. The luminance data acquisition unit 110A, the true value data acquisition unit 120, the time difference correction unit 130B, and the model learning unit 140 constitute the learning device 100B.

[0056] The time difference correction unit 130B corrects the time difference of the time-series luminance data L(t) acquired by the luminance data acquisition unit 110A l) and the time-series true value data B(t) acquired by the true value data acquisition unit 120 b ) and compares the time-series luminance data L(t l ) with the time-series true value data B(t b ) to correct the time difference Δt therebetween, and corrects the time-series true value data B(t l ) based on the time-series luminance data L(t b ).

[0057] A processing example of the learning device and the biological data estimation device according to Embodiment 2 of the present disclosure will be described. For the processes similar to those already described, the description will be omitted here because it would be redundant. FIG. 9 is a flowchart showing a detailed example of the time difference correction process in the process of the learning device 100B according to Embodiment 2 of the present disclosure. FIG. 10 is a diagram for explaining the time difference correction according to Embodiment 2. The time difference correction unit 130B starts the time difference correction process, for example, when the luminance data acquisition process in step ST100 and the true value data acquisition process in step ST200 already described are executed to acquire the time-series luminance data and the time-series true value data.

[0058] The time difference correction unit 130B executes a time difference calculation process (step ST320). The time difference correction unit 130B compares the time-series luminance data L(t l ) acquired by the luminance data acquisition unit 110A with the time-series true value data B(t b ) acquired by the true value data acquisition unit 120. The time difference correction unit 130B then calculates the time difference Δt generated by the difference in the positions of the information acquisition locations of each of the imaging data and the biological reference information for the same subject and at the same time based on the time-series luminance data L(t l ) and the time-series true value data B(t b ). The time difference correction unit 130B, for example, when the time-series luminance data L(t l ) is used as a reference, the time-series true value data B(t bCalculate the time difference Δt of ().

[0059] The time difference correction unit 130B executes a process of correcting the true value data based on the luminance data (step ST322). The time difference correction unit 130B is based on the time-series luminance data L(t l ) to correct the time-series true value data B(t b ). The time difference correction unit 130B performs a process of correcting, for example, the time-series true value data B(t b ) so as to shift it by the time difference Δt along the time axis. The time difference correction unit 130B adds the time difference Δt to the time t in the data with a later time (time-series true value data B(t b )) and removes the data D for the time difference Δt from the beginning, and in the data with an earlier time (time-series luminance data L(t l )) removes the data D for the time difference Δt from the end. By such a process, the time difference correction unit 130B calculates the time-series true value data B(t bf ) and the time-series luminance data L(t lf ) having the effective time-series length T shown in FIG. 10. The time difference correction unit 130B outputs the time-series true value data B(t bf ) and the time-series luminance data L(t lf ) after the correction process to the model learning unit 140. Note that, in this case, the time-series luminance data L(t lf ) after the correction process is the same data as the time-series luminance data L(t l ).

[0060] When the time difference correction unit 130B outputs the time-series true value data B(t bf ) and the time-series luminance data L(t lf ) after the correction process to the model learning unit 140, the time difference correction process ends.

[0061] In the present embodiment, the following configuration is disclosed. The time difference correction unit is The time difference between the time-series luminance data acquired by the luminance data acquisition unit and the time-series true value data acquired by the true value data acquisition unit is corrected by comparing them, and corrects the time-series true value data based on the time-series luminance data, A learning device characterized by the above. As a result, the present disclosure further has an effect that the processing load can be reduced because it is only necessary to correct the time-series true value data based on the time-series luminance data. Furthermore, the present disclosure has the same effect as the above effect by applying the above configuration to the above learning method.

[0062] In the present embodiment, the following configuration is further disclosed. The time difference correction unit corrects the time difference between the time-series luminance data acquired by the luminance data acquisition unit and the time-series true value data acquired by the true value data acquisition unit by comparing them, and corrects the time-series true value data based on the time-series luminance data, A biological data estimation device characterized by the above. As a result, the present disclosure further has an effect that the processing load can be reduced because it is only necessary to correct the time-series true value data based on the time-series luminance data. Furthermore, the present disclosure has the same effect as the above effect by applying the above configuration to the above generated data estimation method.

[0063] Embodiment 3. Embodiment 3 is a form of a more detailed configuration example (second example) of the configuration related to the time difference correction of Embodiment 1. In Embodiment 3, among the components according to Embodiment 3, for components that are the same as those in Embodiment 1 or Embodiment 2 that have already been described, the same component names and the same or similar reference numerals are used, and overlapping descriptions are appropriately omitted.

[0064] A configuration example according to Embodiment 3 will be described. In the description here, for the purpose of distinguishing from configuration examples of other forms, descriptions using reference numerals with "C" attached to some component names such as the biological data estimation device 10C and the learning device 100C will be used for the description. The biological data estimation device 10C includes a luminance data acquisition unit 110A, a true value data acquisition unit 120, a time difference correction unit 130C, a model learning unit 140, and a biological data estimation unit 150. The luminance data acquisition unit 110A, the true value data acquisition unit 120, the time difference correction unit 130C, and the model learning unit 140 constitute the learning device 100C.

[0065] The time difference correction unit 130C compares the time-series luminance data L(t l ) acquired by the luminance data acquisition unit 110A with the time-series true value data B(t b ) acquired by the true value data acquisition unit 120, and corrects the time difference Δt between the time-series luminance data L(t l ) and the time-series true value data B(t b ), and corrects the time-series luminance data L(t b ) based on the time-series true value data B(t l ).

[0066] A processing example of the learning device and the biological data estimation device according to Embodiment 3 of the present disclosure will be described. For processes that are the same as those already described, the description is repeated, so the description here is omitted. FIG. 11 is a flowchart showing a detailed example of the time difference correction process in the process of the learning device 100C according to Embodiment 3 of the present disclosure. The time difference correction unit 130C starts the time difference correction process, for example, when the luminance data acquisition process in step ST100 and the true value data acquisition process in step ST200, which have already been described, are executed, and time-series luminance data L(t l ) and time-series true value data B(t b ) are acquired.

[0067] The time difference correction unit 130C executes a time difference calculation process (step ST320). The time difference correction unit 130C compares the time-series luminance data L(t l ) acquired by the luminance data acquisition unit 110A with the time-series true value data B(t b ) acquired by the true value data acquisition unit 120. Next, the time difference correction unit 130C calculates a time difference Δt generated by the difference in the positions of the information acquisition locations of each of the imaging data and the biological reference information for the same subject and at the same time based on the time-series luminance data L(t l ) and the time-series true value data B(t b ). The time difference correction unit 130C calculates, for example, the time difference Δt of the time-series luminance data L(t b ) with respect to the time-series true value data B(t l ) as a reference.

[0068] The time difference correction unit 130C executes a process of correcting the luminance data based on the true value data (step ST332). Specifically, the time difference correction unit 130C corrects the time-series luminance data L(t b ) based on the time-series true value data B(t l ). The time difference correction unit 130C performs a process of correcting, for example, the time-series luminance data L(t l ) so as to shift it by the amount of the time difference Δt along the time axis. The time difference correction unit 130C adds the time difference Δt to the time t in the data with a later time (time-series luminance data L(t l )) and removes data D corresponding to the amount of the time difference Δt from the beginning, and for the data with an earlier time (time-series true value data B(t b)) performs data removal for only the data D corresponding to the time difference Δt from the end. Through such processing, the time difference correction unit 130C obtains the time-series true value data B(t bf ) and the time-series luminance data L(t lf ). The time difference correction unit 130C outputs the time-series true value data B(t bf ) and the time-series luminance data L(t lf ) after the correction process to the model learning unit 140. In this case, the time-series true value data B(t bf ) after the correction process is the same data as the time-series true value data B(t b ).

[0069] When the time difference correction unit 130C outputs the time-series true value data B(t bf ) and the time-series luminance data L(t lf ) after the correction process to the model learning unit 140, the time difference correction process ends.

[0070] In the present embodiment, further, the following configuration is disclosed. The time difference correction unit compares the time-series luminance data acquired by the luminance data acquisition unit with the time-series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time-series luminance data and the time-series true value data, corrects the time-series luminance data based on the time-series true value data, and is characterized by a learning device. Thereby, the present disclosure further has an effect that the processing load can be reduced because it is only necessary to correct the time-series luminance data based on the time-series true value data. In addition, the present disclosure further has an effect that the accuracy of the learning model can be improved because it is based on the time-series true value data that is likely to be close to the actual value. Furthermore, by applying the above configuration to the above-described generated data estimation device, the above learning method, or the above generated data estimation method, the same effects as the above effects can be achieved.

[0071] In the present embodiment, the following configuration is further disclosed. The time difference correction unit compares the time-series luminance data acquired by the luminance data acquisition unit with the time-series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time-series luminance data and the time-series true value data, corrects the time-series luminance data based on the time-series true value data, A biological data estimation device characterized by the above. Thereby, the present disclosure only needs to perform correction processing on the time-series luminance data based on the time-series true value data, so that the processing load can be reduced. In addition, since the present disclosure is based on time-series true value data that is likely to be close to the actual value, it is possible to improve the accuracy of the learning model. Furthermore, by applying the above configuration to the above-described generated data estimation device, the above learning method, or the above generated data estimation method, the same effects as the above effects can be achieved.

[0072] Embodiment 4. Embodiment 4 is a form of a more detailed configuration example (third example) of the configuration related to the time difference correction in Embodiment 1. In Embodiment 4, among the components according to Embodiment 4, for the components that are the same as the components according to Embodiment 1, Embodiment 2, or Embodiment 3 that have already been described, the same component names and the same or similar reference numerals are used, and the overlapping descriptions are appropriately omitted.

[0073] A configuration example according to Embodiment 4 will be described. In the following description, for the purpose of distinguishing from other forms of configuration examples, the description will be given using the notation of attaching "D" to the reference numerals of some components such as the biological data estimation device 10D and the learning device 100D. The biological data estimation device 10D includes a luminance data acquisition unit 110A, a true value data acquisition unit 120, a time difference correction unit 130D, a model learning unit 140, and a biological data estimation unit 150. The luminance data acquisition unit 110A, the true value data acquisition unit 120, the time difference correction unit 130D, and the model learning unit 140 constitute the learning device 100D.

[0074] The time difference correction unit 130D compares the time-series luminance data L(t l ) acquired by the luminance data acquisition unit 110A with the time-series true value data B(t b ) acquired by the true value data acquisition unit 120, and corrects the time difference Δt between the time-series luminance data L(t l ) and the time-series true value data B(t b ), and corrects the time-series true value data B(t b ) and the time-series luminance data L(t l ) respectively.

[0075] The processing examples of the learning device and the biological data estimation device according to the fourth embodiment of the present disclosure will be described. For the processing similar to the processing already described, the description is omitted here because the description is redundant. FIG. 12 is a flowchart showing a detailed example of the time difference correction process in the processing of the learning device according to the fourth embodiment of the present disclosure. The time difference correction unit 130D, for example, executes the luminance data acquisition process of step ST100 and the true value data acquisition process of step ST200 already described, and when the time-series luminance data L(t l ) and the time-series true value data B(t b ) are acquired, the time difference correction process is started.

[0076] The time difference correction unit 130D executes a time difference calculation process (step ST320). The time difference correction unit 130D compares the time-series luminance data L(t l ) acquired by the luminance data acquisition unit 110A with the time-series true value data B(t b ) acquired by the true value data acquisition unit 120. The time difference correction unit 130C calculates a time difference Δt generated by the difference in the positions of the information acquisition locations of each of the captured data and the biological reference information for the same subject and at the same time based on the time-series luminance data L(t l ) and the time-series true value data B(t b ).

[0077] The time difference correction unit 130D executes a process of correcting the true value data and the luminance data (step ST342). The time difference correction unit 130D corrects each of the time-series true value data B(t b ) and the time-series luminance data L(t l ). For example, the time difference correction unit 130D corrects the data determined to be temporally advanced among the time-series true value data B(t b ) and the time-series luminance data L(t l ) to be delayed by Δt / 2, and corrects the data determined to be temporally delayed to be advanced by Δt / 2. The time difference correction unit 130D adds the time difference Δt to the time t in the data (either the time-series luminance data L(t l ), or the data determined to be temporally delayed among the time-series true value data B(t b )), removes the data D for the fraction of Δt / 2 from the beginning, and for the data (either the time-series true value data B(t b ), or the data determined to be temporally advanced among the time-series luminance data L(t l )) that is temporally advanced, removes the data D for the fraction of Δt / 2 from the end. By such processing, the time difference correction unit 130D calculates the time-series true value data B(t bf ) and the time-series luminance data L(t lf ) with an effective time-series length T. The time difference Δt / 2, which is the amount of time for correction to delay and the amount of time for correction to advance, is an example, and it is sufficient that the time difference Δt becomes 0 by the correction. After performing the correction process, the time difference correction unit 130D outputs the corrected time-series true value data B(t bf ) and the corrected time-series luminance data L(t lf ) to the model learning unit 140.

[0078] After the time difference correction unit 130D outputs the corrected time-series true value data B(t bf ) and the corrected time-series luminance data L(t lf ) to the model learning unit 140, the time difference correction process ends.

[0079] In the present embodiment, further, the following configuration is disclosed. The time difference correction unit compares the time-series luminance data acquired by the luminance data acquisition unit with the time-series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time-series luminance data and the time-series true value data, corrects each of the time-series true value data and the time-series luminance data, and is characterized by a learning device. Thereby, the present disclosure further has an effect that by correcting both the time-series true value data and the time-series luminance data, it is possible to suppress the occurrence of an error due to correction as compared with the case of correcting only one of the data. Furthermore, the present disclosure applies the above configuration to the above learning method, and has the same effect as the above effect.

[0080] In the present embodiment, further, the following configuration is disclosed. The time difference correction unit The time difference between the time-series luminance data acquired by the luminance data acquisition unit and the time-series true value data acquired by the true value data acquisition unit is corrected by comparing them, and the time-series true value data and the time-series luminance data are each corrected, A biological data estimation device characterized by this. Thereby, the present disclosure further suppresses the occurrence of errors due to correction by correcting both the time-series true value data and the time-series luminance data, as compared with the case of correcting only one of the data, and has the effect of Furthermore, the present disclosure applies the above configuration to the above-described generated data estimation method, thereby achieving the same effect as the above effect.

[0081] Embodiment 5. Embodiment 5 describes an example of a detailed process (first example) for calculating the time difference in the time difference correction unit. In Embodiment 5, among the components according to Embodiment 5, for components similar to those according to Embodiment 1, Embodiment 2, Embodiment 3, or Embodiment 4 that have already been described, the same component names and the same or similar reference numerals are used, and duplicate descriptions are appropriately omitted.

[0082] A configuration example according to Embodiment 5 will be described. In the description here, in order to distinguish from the configuration examples of other embodiments, the description uses the notation of attaching "E" to the reference numerals of some components, such as the biological data estimation device 10E and the learning device 100E. The biological data estimation device 10E includes a luminance data acquisition unit 110A, a true value data acquisition unit 120, a time difference correction unit 130E, a model learning unit 140, and a biological data estimation unit 150. The luminance data acquisition unit 110A, the true value data acquisition unit 120, the time difference correction unit 130E, and the model learning unit 140 constitute the learning device 100E.

[0083] The time difference correction unit 130E calculates a time difference Δt when correcting the time difference between the time-series luminance data L(t l ) and the time-series true value data B(t b ). The time difference correction unit 130E calculates the time difference Δt using cross-correlation.

[0084] A processing example of the learning device and the biological data estimation device according to Embodiment 5 of the present disclosure will be described. For the processes similar to the processes already described, the description will be omitted here because it would be redundant. FIG. 13 is a flowchart showing a detailed example of the time difference calculation process in the process of the learning device according to Embodiment 5 of the present disclosure. The time difference correction unit 130E, for example, executes the luminance data acquisition process in step ST100 and the true value data acquisition process in step ST200 that have already been described, and when the time-series luminance data L(t l ) and the time-series true value data B(t b ) are acquired, it starts the time difference calculation process.

[0085] The time difference correction unit 130E executes a process of calculating the time difference Δt using cross-correlation (step ST325). Specifically, the time difference correction unit 130E first obtains a series a = {a1, a2, a3, ···, a l} of cross-correlation coefficients for the time-series luminance data L(t b ) and the time-series true value data B(t m ). Next, when the time-series luminance data L(t l ) or the time-series true value data B(t b ) is set as d = {d1, d2, d3, ···, d n}, the time difference correction unit 130E applies the discrete Fourier transform F to "d" to obtain the peak frequency k p in the frequency power spectrum P, and multiplies the reciprocal of the peak frequency k p by the frame rate Fr to obtain the number of frames N f for one period (for example, for a pulse wave, it is the number of frames for one pulse beat). Then, in the series a of the cross-correlation coefficients, the time difference correction unit 130E obtains, as the time difference Δt, the time difference at the position that takes the maximum value within the number of frames N from the head. f The following equations (1), (2), (3), (4), and (5) correspond to the above-described processing. The following shows equations (1), (2), (3), (4), and (5) corresponding to the above-described processing. TIFF2025096730000002.tif63166 "f" represents the series of data to be Fourier-transformed. "F" represents the discrete Fourier transform. "W hanning " represents the Hanning window. "k" represents the frequency. "N" represents the number of data. "j" represents a complex number. "m", "n", and "N" are each integers of 1 or more. "k ll " and "k ul " are arbitrary constants that are changed according to the frequency targeted for biological data estimation in the time-series luminance data L(t l ) or the time-series true value data B(t b ). In the argmax function, when a plurality of elements are obtained, the first element shall be returned. Note that when the biological reference information is such that the corrected time-series luminance data L(t lf ) and the corrected time-series true value data B(t bf ) are in an inverse phase relationship (for example, when the biological reference information shows a continuous blood pressure waveform), in the series a of the cross-correlation coefficients described above, the time difference at the position that takes the "minimum value" within the number of frames N from the head is obtained as the time difference Δt, so that they can be made to correspond. f By doing so, they can be made to correspond.

[0086] When the time difference correction unit 130E calculates the time difference Δt, it ends the time difference calculation process. The time difference correction unit 130E then proceeds to, for example, the process of step ST322, the process of step ST332, or the process of step ST342, which have already been described.

[0087] Accordingly, the present disclosure can further provide a configuration for calculating the time difference Δt between the time-series luminance data L(t l ) and the time-series true value data B(t b ) using cross-correlation. Furthermore, by applying the above configuration to the above learning device, the above generated data estimation device, the above learning method, or the above generated data estimation method, an effect similar to the above effect can be achieved.

[0088] Embodiment 6. Embodiment 6 describes an example form (second example form) of the detailed process for calculating the time difference in the time difference correction unit. In Embodiment 6, among the components according to Embodiment 6, for components similar to those according to Embodiment 1, Embodiment 2, Embodiment 3, Embodiment 4, or Embodiment 5 that have already been described, the same component names and the same or similar reference numerals are used, and duplicate descriptions are appropriately omitted.

[0089] A configuration example according to Embodiment 6 will be described. In the description here, for the purpose of distinguishing from the configuration examples of other forms, descriptions using reference numerals with "F" attached to some component names, such as the biological data estimation device 10F and the learning device 100F, will be used for the description. The biological data estimation device 10F includes a luminance data acquisition unit 110A, a true value data acquisition unit 120, a time difference correction unit 130F, a model learning unit 140, and a biological data estimation unit 150. The luminance data acquisition unit 110A, the true value data acquisition unit 120, the time difference correction unit 130F, and the model learning unit 140 constitute the learning device 100F.

[0090] The time difference correction unit 130F is configured to calculate the time difference between the time-series luminance data L(t lWhen correcting the time difference between the and the time series true value data, the time difference Δt is calculated.

[0091] A processing example of the learning device and the biological data estimation device according to Embodiment 6 of the present disclosure will be described. For the processing similar to the processing already described, the description will be omitted here because it overlaps. FIG. 14 is a flowchart showing a detailed example of the time difference calculation process in the processing of the learning device according to Embodiment 6 of the present disclosure. The time difference correction unit 130F, for example, executes the luminance data acquisition process in step ST100 and the true value data acquisition process in step ST200 already described, and the time series luminance data L(t l ) and the time series true value data B(t b ) are acquired, and then the time difference calculation process is started.

[0092] The time difference correction unit 130F executes a process of calculating the time difference Δt using quadrature detection (step ST326). Specifically, when it is defined that "x" and "y" represent the time series luminance data L(t l ) or the time series true value data B(t b ), and "H(x)" and "H(y)" represent the Hilbert transforms of "x" and "y" respectively, the time difference correction unit 130F acquires the time difference Δt by, for example, the following formula (6). Δt = tan -1 (H(y) / y) - tan -1 (H(x) / x) (6)

[0093] When the time difference correction unit 130F calculates the time difference Δt, the time difference calculation process ends. The time difference correction unit 130F then proceeds to, for example, the process of step ST322, the process of step ST332, or the process of step ST342 already described.

[0094] As a result, the present disclosure further provides a configuration for calculating a time difference Δt between the time-series luminance data L(t l ) and the time-series true value data B(t b ) using quadrature detection. Furthermore, by applying the above configuration to the above learning device, the above generated data estimation device, the above learning method, or the above generated data estimation method, an effect similar to the above effect can be achieved.

[0095] Embodiment 7. Embodiment 7 describes an example of a detailed process (the third example) for calculating the time difference in the time difference correction unit. In Embodiment 7, among the components according to Embodiment 6, for components similar to those according to Embodiment 1, Embodiment 2, Embodiment 3, Embodiment 4, Embodiment 5, or Embodiment 6 that have already been described, the same component names and the same or similar reference numerals are used, and duplicate descriptions are appropriately omitted.

[0096] A configuration example according to Embodiment 7 will be described. In the description here, for the purpose of distinguishing from the configuration examples of other forms, descriptions are made using notations with "G" appended to the reference numerals of some components, such as the biological data estimation device 10G and the learning device 100G. The biological data estimation device 10G includes a luminance data acquisition unit 110A, a true value data acquisition unit 120, a time difference correction unit 130G, a model learning unit 140, and a biological data estimation unit 150. The luminance data acquisition unit 110A, the true value data acquisition unit 120, the time difference correction unit 130G, and the model learning unit 140 constitute the learning device 100G.

[0097] When correcting the time difference between the time-series luminance data L(t l ) and the time-series true value data B(t b ), the time difference correction unit 130G calculates the time difference Δt. The time difference correction unit 130G calculates the time difference Δt using peak detection. Specifically, the time difference correction unit 130G first performs peak detection on the time-series luminance data L(t l ) and the time-series true value data B(t b ) respectively, and obtains a series of peak positions. Next, the time difference correction unit 130G performs association on the two series of peak positions obtained previously so that the elements between the series are paired. Then, the time difference correction unit 130G obtains the average value of the series of differences between the values of the corresponding elements as the time difference Δt. Note that the method of peak detection may be realized by using known techniques and is not limited to the method described above.

[0098] A processing example of the learning device and the biological data estimation device according to Embodiment 7 of the present disclosure will be described. For the processing similar to the processing already described, the description will be omitted here because the description would be redundant. FIG. 15 is a flowchart showing a detailed example of the time difference calculation process in the processing of the learning device according to Embodiment 7 of the present disclosure. For example, when the luminance data acquisition process in step ST100 and the true value data acquisition process in step ST200 already described are executed and the time-series luminance data L(t l ) and the time-series true value data B(t b ) are obtained, the time difference correction unit 130G starts the time difference calculation process.

[0099] The time difference correction unit 130G executes a process of calculating the time difference using peak detection (step ST327). The time difference correction unit 130G first performs peak detection on the time-series luminance data L(t l ) and the time-series true value data B(t b ) respectively, and obtains a series of peak positions. Next, the time difference correction unit 130G performs association on the two series of peak positions obtained previously so that the elements between the series are paired. Then, the time difference correction unit 130G obtains the average value of the series of differences between the values of the corresponding elements as the time difference Δt.

[0100] When the time difference correction unit 130G calculates the time difference Δt, it ends the time difference calculation process. Next, the time difference correction unit 130G proceeds to, for example, the process of step ST322, the process of step ST332, or the process of step ST342, which have already been described.

[0101] Accordingly, the present disclosure further provides a configuration for calculating the time difference Δt between the time-series luminance data L(t l ) and the time-series true value data B(t b ) using the peaks of the time-series luminance data L(t l ) and the time-series true value data B(t b ), respectively.

[0102] Here, a hardware configuration for realizing the functions of the present disclosure will be described. FIG. 16 is a diagram showing a first example of a hardware configuration for realizing the functions according to the configuration of the present disclosure. FIG. 17 is a diagram showing a second example of a hardware configuration for realizing the functions according to the configuration of the present disclosure. The biological data estimation device 10 (10A, 10B, 10C, 10D, 10E, 10F, 10G) and the learning device 100 (100A, 100B, 100C, 100D, 100E, 100F, 100G) of the present disclosure are each realized by hardware as shown in FIG. 16 or FIG. 17.

[0103] The biological data estimation device 10 (10A, 10B, 10C, 10D, 10E, 10F, 10G) and the learning device 100 (100A, 100B, 100C, 100D, 100E, 100F, 100G) are each composed of, for example, a processor 10001, a memory 10002, an input / output interface 10003, and a communication circuit 10004, as shown in FIG. 16. The processor 10001 and the memory 10002 are, for example, those installed in a computer. The memory 10002 stores a program for causing the computer to function as a luminance data acquisition unit 110, 110A, a true value data acquisition unit 120, a time difference correction unit 130 (130A, 130B, 130C, 130E, 130F, 130G), a model learning unit 140, a biological data estimation unit 150, and a control unit (not shown). By reading and executing the program stored in the memory 10002 by the processor 10001, the functions of the luminance data acquisition unit 110, 110A, the true value data acquisition unit 120, the time difference correction unit 130 (130A, 130B, 130C, 130E, 130F, 130G), the model learning unit 140, the biological data estimation unit 150, and the control unit (not shown) are realized. Also, a database 400 and a storage unit (not shown) are realized by the memory 10002 or another memory (not shown). Also, a communication unit (not shown) is realized by the communication circuit 10004.

[0104] The processor 10001 is, for example, one using a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a microprocessor, a microcontroller, or a DSP (Digital Signal Processor). The memory 10002 may be a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable Read Only Memory), or a flash memory, or may be a magnetic disk such as a hard disk or a flexible disk, or may be an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or may be a magneto-optical disk. The processor 10001, the memory 10002, or the communication circuit 10004 is connected in a state where they can transmit data to each other. Also, the processor 10001, the memory 10002, and the communication circuit 10004 are connected in a state where they can transmit data to and from other hardware via the input / output interface 10003.

[0105] Alternatively, in the biological data estimation devices 10 (10A, 10B, 10C, 10D, 10E, 10F, 10G) and the learning devices 100, 100A, 100B, 100C, 100D, 100E, 100F, 100G, the functions of the luminance data acquisition units 110, 110A, the true value data acquisition unit 120, the time difference correction units 130 (130A, 130B, 130C, 130E, 130F, 130G), the model learning unit 140, the biological data estimation unit 150, and a control unit (not shown) may be realized by a dedicated processing circuit 20001 as shown in FIG. 17.

[0106] The processing circuit 20001 is, for example, a single circuit, a composite circuit, a programmed processor, a parallel-programmed processor, an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field-Programmable Gate Array), a SoC (System-on-a-Chip), or a system LSI (Large-Scale Integration), etc. Also, a storage unit (not shown) is realized by the memory 20002 or another memory (not shown). The memory 20002 may be a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable Read Only Memory), or flash memory, or may be a magnetic disk such as a hard disk or a flexible disk, or may be an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or may be a magneto-optical disk. Also, the communication circuit 20004 realizes a communication unit (not shown). The processing circuit 20001 is connected to the memory 20002 or the communication circuit 20004 in a state where data can be transmitted mutually. Also, the processing circuit 20001, the memory 20002, and the communication circuit 20004 are connected in a state where data can be transmitted mutually with other hardware via the input / output interface 20003. Note that the functions of the luminance data acquisition units 110, 110A, the true value data acquisition unit 120, the time difference correction units 130 (130A, 130B, 130C, 130E, 130F, 130G), the model learning unit 140, the biological data estimation unit 150, and the control unit (not shown) in the biological data estimation devices 10 (10A, 10B, 10C, 10D, 10E, 10F, 10G) and the learning devices 100, 100A, 100B, 100C, 100D, 100E, 100F, 100G may be realized by separate processing circuits or may be realized collectively by a processing circuit.

[0107] Alternatively, some of the functions of the biological data estimation devices 10 (10A, 10B, 10C, 10D, 10E, 10F, 10G), and the learning devices 100, 100A, 100B, 100C, 100D, 100E, 100F, 100G, such as the luminance data acquisition units 110, 110A, the true value data acquisition unit 120, the time difference correction units 130 (130A, 130B, 130C, 130E, 130F, 130G), the model learning unit 140, the biological data estimation unit 150, and a control unit (not shown) may be realized by the processor 10001 and the memory 10002, and the remaining functions may be realized by the processing circuit 20001.

[0108] Note that within the scope of this disclosure, any combination of the embodiments, any modification of any component of each embodiment, or any omission of any component of each embodiment is possible.

[0109] Since this disclosure can improve the accuracy of the learning model used for estimating biological data compared to the prior art, it is suitable for use in, for example, a biological data estimation device that estimates the biological data of an evaluation target person such as a driver of a moving body.

Description of Reference Numerals

[0110] 10 (10A, 10B, 10C, 10D, 10E, 10F, 10G) biological data estimation device, 20 output device, 100 (100A, 100B, 100C, 100D, 100E, 100F, 100G) learning device, 110, 110A luminance data acquisition unit, 120 true value data acquisition unit, 130 (130A, 130B, 130C, 130E, 130F, 130G) time difference correction unit, 140 model learning unit, 150 biological data estimation unit, 200 imaging device, 300 sensor device, 400 database, 410A1, 410A2, 410B1, 410B2, 410C1, 410C2 pair data (data set), 411 luminance data, 412 true value data, 10001 processor, 10002 memory, 10003 input / output interface, 10004 communication circuit, 20001 processing circuit, 20002 memory, 20003 input / output interface, 20004 communication circuit.

Claims

1. A luminance data acquisition unit that acquires time-series luminance data based on time-series imaging data, A true value data acquisition unit that acquires time-series true value data using time-series biological reference information based on a biological signal, A time difference correction unit that corrects the time difference between the time-series luminance data and the time-series true value data, A model learning unit that constructs a learning model that outputs biological data from the time-series luminance data using the time-series luminance data and the time-series true value data output by the time difference correction unit, A learning device comprising the same.

2. The time difference correction unit, compares the time-series luminance data acquired by the luminance data acquisition unit with the time-series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time-series luminance data and the time-series true value data, corrects the time-series true value data based on the time-series luminance data, The learning device according to claim 1.

3. The time difference correction unit, compares the time-series luminance data acquired by the luminance data acquisition unit with the time-series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time-series luminance data and the time-series true value data, corrects the time-series luminance data based on the time-series true value data, The learning device according to claim 1.

4. The time difference correction unit, compares the time-series luminance data acquired by the luminance data acquisition unit with the time-series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time-series luminance data and the time-series true value data, corrects the time-series true value data and the time-series luminance data respectively, The learning device according to claim 1.

5. The biological reference information is a pulse wave, The learning device according to any one of claims 1 to 4.

6. The biological reference information is a continuous blood pressure waveform, The learning device according to any one of claims 1 to 4.

7. The biological reference information is an electrocardiogram waveform, The learning device according to any one of claims 1 to 4.

8. Acquire in order, for each combination, the combination of the time-series luminance data and the time-series true value data of each of a plurality of persons, and for each combination, execute the processing by the luminance data acquisition unit, the true value data acquisition unit, the time difference correction unit, and the model learning unit, The learning device according to any one of claims 1 to 4.

9. A luminance data acquisition unit that acquires time-series luminance data based on time-series imaging data; A true value data acquisition unit that acquires time-series true value data using time-series biological reference information based on a biological signal; A time difference correction unit that corrects the time difference between the time-series luminance data and the time-series true value data; A model learning unit that constructs a learning model that outputs biological data from the time-series luminance data using the time-series luminance data and the time-series true value data output by the time difference correction unit; A biological data estimation unit that estimates biological data based on the learning model constructed by the model learning unit and the time-series luminance data acquired by the luminance data acquisition unit; A biological data estimation device comprising:

10. The time difference correction unit: Compares the time-series luminance data acquired by the luminance data acquisition unit with the time-series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time-series luminance data and the time-series true value data, Corrects the time-series true value data based on the time-series luminance data; The biological data estimation device according to claim 9.

11. The time difference correction unit: Compares the time-series luminance data acquired by the luminance data acquisition unit with the time-series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time-series luminance data and the time-series true value data, Corrects the time-series luminance data based on the time-series true value data; The biological data estimation device according to claim 9.

12. The time difference correction unit: Compares the time-series luminance data acquired by the luminance data acquisition unit with the time-series true value data acquired by the true value data acquisition unit, and corrects the time difference between the time-series luminance data and the time-series true value data, Corrects the time-series true value data and the time-series luminance data respectively; The biological data estimation device according to claim 9.

13. The biological reference information is a pulse wave. The biological data estimation device according to any one of claims 9 to 12.

14. The biological reference information is a continuous blood pressure waveform. The biological data estimation device according to any one of claims 9 to 12.

15. The biological reference information is an electrocardiogram waveform. The biological data estimation device according to any one of claims 9 to 12.

16. The luminance data acquisition unit acquires time-series luminance data using imaging data obtained by imaging the driver of the moving body over time, The biological data estimation unit estimates the biological data of the driver, The biological data estimation device according to any one of claims 9 to 12.

17. The luminance data acquisition unit acquires time-series luminance data using imaging data obtained by imaging the driver of the moving body over time, The biological data estimation unit estimates the biological data of the driver, The biological data estimation device according to claim 13.

18. The luminance data acquisition unit acquires time-series luminance data using imaging data obtained by imaging the driver of the moving body over time, The biological data estimation unit estimates the biological data of the driver, The biological data estimation device according to claim 14.

19. The luminance data acquisition unit acquires time-series luminance data using imaging data obtained by imaging the driver of the moving body over time, The biological data estimation unit estimates the biological data of the driver, The biological data estimation device according to claim 15.

20. Combinations of time-series luminance data and time-series true value data for each of a plurality of persons are sequentially acquired, and for each combination, processing by the luminance data acquisition unit, the true value data acquisition unit, the time difference correction unit, and the model learning unit is executed. The biological data estimation device according to any one of claims 9 to 12.

21. A learning method executed by a learning device, A step of the luminance data acquisition unit of the learning device acquiring time-series luminance data based on time-series imaging data, A step of the true value data acquisition unit of the learning device acquiring time-series true value data using time-series biological reference information based on a biological signal, A step of the time difference correction unit of the learning device correcting the time difference between the time-series luminance data and the time-series true value data, A step of the model learning unit of the learning device constructing a learning model that outputs biological data from the time-series luminance data using the time-series luminance data and the time-series true value data output by the time difference correction unit, A learning method comprising:

22. A biological data estimation method executed by a biological data estimation device, A step of the luminance data acquisition unit of the biological data estimation device acquiring time-series luminance data based on time-series imaging data, The step in which the true value data acquisition unit of the biological data estimation device acquires time-series true value data using time-series biological reference information based on a biological signal; The step in which the time difference correction unit of the biological data estimation device corrects the time difference between the time-series luminance data and the time-series true value data; The step in which the model learning unit of the biological data estimation device constructs a learning model that outputs biological data from the time-series luminance data using the time-series luminance data and the time-series true value data output by the time difference correction unit; The step in which the biological data estimation unit of the biological data estimation device estimates biological data based on the learning model constructed by the model learning unit and the time-series luminance data acquired by the luminance data acquisition unit; A biological data estimation method comprising the above steps.

Images