Model generation device, biometric data estimation device, model generation system, biometric data estimation system, and model generation program

Abstract

To provide a model generation device that generates a machine learning model that can accurately acquire first biometric data of a subject while suppressing the influence of noise, even in a case of acquiring the first biometric data of the subject in scenes where events that may cause noise are likely to occur.SOLUTION: A model generation device includes: a photographed image acquisition unit (11) that acquires a photographed image including the skin region of a test subject; a luminance data extraction unit (12) that extracts luminance data from the photographed image; a biometric reference data acquisition unit (13) that acquires biometric reference data as second biometric data of the test subject, measured using a device whose measurement result is less susceptible to noise that may occur in environments where first biometric data can be obtained; a data processing unit (14) that generates learning waveform data simulating the waveform of the first biometric data from the biometric reference data; and a model learning unit (15) that generates a machine learning model which receives luminance data as input and outputs the first biometric data, using the luminance data and the learning waveform data as learning data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a model generation device, a biological data estimation device, a model generation system, a biological data estimation system, and a model generation program.

Background Art

[0002] Conventionally, a technique for obtaining human biological data such as a pulse wave based on a captured image of a human face and a learned model in machine learning (hereinafter referred to as a "machine learning model") is known (for example, Patent Document 1). As an application example of such a technique, in recent years, for example, in a vehicle, a technique for obtaining passenger biological data based on a captured image of a passenger's face and a machine learning model has been developed. The obtained biological data is used, for example, to detect the occurrence of physical abnormalities in the passenger.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to accurately obtain the biological data of a person (hereinafter referred to as a "subject") for whom biological data is to be estimated using a machine learning model, the machine learning model is generated based on learning data collected in a scene where biological data is to be obtained, and it is required that the biological data as teacher data included in the learning data has little noise. Hereinafter, the biological data of the subject obtained using the machine learning model is also referred to as "first biological data", and the biological data as teacher data is also referred to as "second biological data". However, in a scene where an attempt is made to obtain first biological data, events that cause noise, such as vibrations or external light, may occur, and the second biological data collected may contain a lot of noise, making it difficult to generate a machine learning model for accurately obtaining the first biological data. For example, for collecting the second biological data, it is conceivable to use a pulse wave sensor such as a pulse oximeter. However, the second biological data collected by the pulse wave sensor is affected by noise such as vibrations or external light. Therefore, even if a machine learning model is generated using the second biological data collected by the pulse wave sensor as teacher data, it may not be a machine learning model that can accurately obtain the first biological data.

[0005] The present disclosure has been made to solve the above problems, and is a model generation device for generating a machine learning model for obtaining first biological data of a subject. Even when obtaining the first biological data of the subject in a scene where an event that causes noise may occur, it is an object to obtain a model generation device that can generate a machine learning model capable of suppressing the influence of noise and accurately obtaining the first biological data of the subject.

Means for Solving the Problems

[0006] The model generation device according to the present disclosure is a model generation device that generates a machine learning model for obtaining first biological data of a subject, including an imaging image acquisition unit that acquires an imaging image including the skin area of a test performer, which is an image of the test performer; a luminance data extraction unit that extracts luminance data indicating the luminance change in the skin area of the test performer from the imaging image acquired by the imaging image acquisition unit; a biological reference data acquisition unit that acquires biological reference data, which is second biological data of the test performer measured by a device in which the influence of noise that may occur in an environment where the first biological data can be obtained from the measurement result is less likely to appear; a data processing unit that generates learning waveform data imitating the waveform of first biological data of a type different from the type of the second biological data measured as the biological reference data, which is teacher data when learning the machine learning model, from the biological reference data acquired by the biological reference data acquisition unit; and a model learning unit that generates a machine learning model that takes the luminance data as an input and outputs the first biological data, using the luminance data extracted by the luminance data extraction unit and the learning waveform data generated by the data processing unit as learning data.

Effect of the Invention

[0007] According to the present disclosure, it is possible to provide a machine learning model that can suppress the influence of noise and accurately obtain the first biological data of a subject even when obtaining the first biological data of the subject in a scene where an event that causes noise may occur.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

MODE FOR CARRYING OUT THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1. FIG. 1 is a diagram showing a configuration example of a model generation device 10 according to Embodiment 1. The model generation device 10 according to Embodiment 1 is connected to an imaging device 20, a device 30, and a biological data estimation device 40. The model generation device 10 and the biological data estimation device 40 constitute a biological data estimation system 500.

[0010] The biological data estimation device 40 according to Embodiment 1 uses a model (hereinafter referred to as a "machine learning model") that has been trained to output biological data with the luminance data obtained in advance based on the captured image as input, based on a captured image (hereinafter referred to as an "estimation captured image") captured by the imaging device 20 of a person, and estimates the biological data of the person. In Embodiment 1, a person whose biological data is estimated by the biological data estimation device 40 is referred to as a "subject". Also, the biological data of the subject estimated by the biological data estimation device 40 is also referred to as "first biological data". Note that the luminance data obtained based on the captured image will be described later. The biological data estimation device 40 acquires an estimation captured image composed of a series of frames Im(k) captured at a predetermined frame rate Fr in at least a range (hereinafter referred to as a "skin existence range") where a skin region, which is a region including the skin of the subject, should exist. Here, k indicates the frame number assigned to each frame. For example, the frame given at the next timing of the frame Im(k) is the frame Im(k + 1). In Embodiment 1, the skin region is a region corresponding to the face of the subject. Note that this is merely an example, and the skin region may be a region other than the face of the subject. For example, the skin region may be a region corresponding to a part belonging to the face, such as the eyes, eyebrows, nose, mouth, forehead, cheeks, or jaw, of the subject. Also, the skin region may be a region corresponding to a body part other than the face, such as the head, shoulders, hands, neck, or feet, of the subject. The skin region may be a plurality of regions.

[0011] Then, the biological data estimation device 40 estimates the first biological data of the subject based on a series of frames Im(k-Tp+1) to Im(k) and the machine learning model every certain number of frames Tp, and outputs the biological data estimation result P(t), which is the estimated first biological data. Specifically, the biological data estimation device 40 estimates the first biological data of the subject by inputting luminance data, which is a signal based on the luminance change of the subject's skin area in a series of frames Im(k-Tp+1) to Im(k), into the machine learning model to obtain biological data.

[0012] Here, t indicates the output number assigned for each certain number of frames Tp. For example, the biological data estimation result given at the next timing of the biological data estimation result P(t) is the biological data estimation result P(t+1). The frame number k and the output number t are integers of 1 or more. The number of frames Tp is an integer of 2 or more.

[0013] Note that the number of subjects, who are the people included in the imaging image for estimation, may be one or more than one. For the sake of simplicity of explanation, in the following Embodiment 1, it is described that the number of subjects included in the imaging image for estimation is one.

[0014] The model generation device 10 according to Embodiment 1 performs learning in machine learning before the biological data estimation device 40 estimates the first biological data of the subject, and generates a machine learning model that takes luminance data as an input and outputs the first biological data, which is used when the biological data estimation device 40 estimates the first biological data of the subject. The model generation device 10 generates a machine learning model based on a captured image (hereinafter referred to as a "test captured image") captured by the imaging device 20 of a person and biometric data of the person (hereinafter referred to as "biometric reference data") measured by the device 30. In Embodiment 1, the person who is the imaging target of the test captured image used when the model generation device 10 generates a machine learning model and who is the source of the biometric reference data is referred to as a "test subject". Also, the biometric data (i.e., biometric reference data) of the test subject that the model generation device 10 acquires from the device 30 is also referred to as "second biometric data". The type of the first biometric data is different from the type of the biometric reference data which is the second biometric data.

[0015] The model generation device 10 acquires a test captured image composed of a series of frames Im(k) that captures at least the skin presence range of the test subject at a predetermined frame rate Fr. Then, the model generation device 10 generates a machine learning model based on a series of frames Im(k - Tp + 1) to Im(k) for a specific number of frames Tp and biometric reference data Br(n - Tq + 1) to Br(n) of a specific number of data Tq acquired during a specific period. Here, n indicates the data number assigned to each biometric reference data. For example, the biometric reference data given at the timing next to the biometric reference data Br(n) is the biometric reference data Br(n + 1). Note that a specific period during which biometric reference data of a specific number of data Tq is acquired is, for example, a period of the same length as the period during which frames for a specific number of frames Tp are acquired. Specifically, the model generation device 10 generates a machine learning model based on luminance data based on the luminance change of the skin area of the test subject in a series of frames Im(k - Tp + 1) to Im(k) and a series of biometric reference data Br(n - Tq + 1) to Br(n) for a specific number of data Tq acquired during a specific period.

[0016] Here, the imaging device 20 and the device 30 will be described. The imaging device 20 images a subject or a test subject. The imaging device 20 is installed so as to be able to image the skin presence areas of the subject and the test performer in the imaging target area. The imaging device 20 includes an imaging unit (not shown) and an illumination unit (not shown). The illumination unit is composed of, for example, LEDs (Light Emitting Diodes). The illumination unit irradiates light onto the imaging range of the imaging unit. The imaging unit images the imaging range irradiated with light when the illumination unit emits light. Note that one or a plurality of illumination units may be provided in the imaging device 20. The imaging device 20 outputs an imaging image (test imaging image or imaging image for estimation) to the model generation device 10 or the biological data estimation device 40.

[0017] The device 30 is a device that measures biological reference data, which is the second biological data of the test performer, and is a device in which the influence of noise that may occur in an environment where the first biological data is obtained in the measurement result, specifically, noise due to vibration, external light, etc., is hardly manifested. In Embodiment 1, the device 30 assumes an electrocardiograph such as a monitor electrocardiograph, an induction electrocardiograph, an exercise stress electrocardiograph, a Holter electrocardiograph, a portable electrocardiograph, a loop implanted electrocardiograph, etc. That is, the biological reference data assumes electrocardiogram data. For example, the test performer attaches an electrocardiograph to the surface of the body and then performs a test to obtain biological reference data under a situation assumed to be the same as when the biological data estimation device 40 estimates the first biological data of the subject. The electrocardiograph measures the electrocardiogram data when the test performer performs the test. Note that in Embodiment 1, the device 30 may be any device in which the influence of noise due to vibration, external light, etc., that may occur in an environment where the first biological data is obtained in the measurement result is hardly manifested, but at least not a pulse wave sensor. Pulse wave sensors are prone to noise due to vibration, external light, etc., that may occur in an environment where the first biological data is obtained in the measurement result. The device 30 outputs the biological reference data to the model generation device 10.

[0018] In the following Embodiment 1, the subject is the driver of the vehicle, and the test performer is a developer or the like. Also, in the following Embodiment 1, the first biological data estimated by the biological data estimation device 40 is pulse wave data. That is, the biological data estimation device 40 estimates the pulse wave data of the driver of the vehicle, and the model generation device 10 generates a machine learning model that takes luminance data as input and outputs pulse wave data based on the luminance data and electrocardiogram data.

[0019] For example, a developer or the like who is a test performer attaches a device 30, that is, an electrocardiograph here, to the body and performs a test drive of the vehicle before the vehicle is shipped. The imaging device 20 images the developer or the like during the test drive of the vehicle and outputs the captured test imaging image to the model generation device 10. Also, the device 30 outputs the biological reference data of the developer or the like, that is, the electrocardiogram data measured during the test drive of the vehicle, to the model generation device 10. The model generation device 10 generates a machine learning model based on the luminance data and electrocardiogram data based on the test imaging image. After the vehicle is shipped, the biological data estimation device 40 estimates the pulse wave data of the driver of the vehicle based on the estimation imaging image captured by the imaging device 20 of the driver of the vehicle during the running of the vehicle and the machine learning model created by the model generation device 10. In the following Embodiment 1, the imaging device 20 is mounted on the vehicle and images the passengers of the vehicle. The imaging target area of the imaging device 20 is the interior of the vehicle, and the imaging device 20 is installed in the interior of the vehicle so as to be able to image the range where the skin of the passenger exists.

[0020] A configuration example of the model generation device 10 and the biological data estimation device 40 according to Embodiment 1 will be described. First, a configuration example of the model generation device 10 will be described. In Embodiment 1, the model generation device 10 is mounted on a local device (not shown) such as a notebook PC (Personal Computer) used by a developer or the like. As shown in FIG. 1, the model generation device 10 includes an imaging image acquisition unit 11, a luminance data extraction unit 12, a biological reference data acquisition unit 13, a data processing unit 14, a model learning unit 15, and a model storage unit 16. The luminance data extraction unit 12 includes a skin area detection unit 121, a measurement area setting unit 122, and a signal extraction unit 123. The data processing unit 14 includes a first data processing unit 141 and a second data processing unit 142.

[0021] The imaging image acquisition unit 11 acquires a test imaging image obtained by imaging a test subject. More specifically, the imaging image acquisition unit 11 acquires a test imaging image obtained by imaging a developer or the like who is performing a test drive of a vehicle by an imaging device 20 mounted on the vehicle. The imaging image acquisition unit 11 outputs the acquired test imaging image to the luminance data extraction unit 12.

[0022] The luminance data extraction unit 12 extracts luminance data indicating the luminance change in the skin area of the test subject from the test imaging image acquired by the imaging image acquisition unit 11. The luminance data extraction unit 12 will be described in detail.

[0023] First, the skin area detection unit 121 of the luminance data extraction unit 12 detects the skin area of the test subject from the frame Im(k) included in the test imaging image acquired by the imaging image acquisition unit 11. The skin area detection unit 121 may detect the skin area using known means. For example, the skin area detection unit 121 can detect the skin area using a cascade-type face detector using Haar-like features. The skin area detection unit 121 generates skin area information S(k) indicating the detected skin area. The skin area information S(k) can include information indicating the presence or absence of detection of the skin area and information indicating the position and size of the detected skin area on the test imaging image. In Embodiment 1, it is assumed that the skin area is represented by a rectangular area on the test imaging image, and the skin area information S(k) includes information indicating the position and size of the rectangular area on the test imaging image. Specifically, when the skin area corresponds to the area of the face of the test subject, the skin area information S(k) indicates, for example, the presence or absence of detection of the face of the test subject, the center coordinates Fc (Fcx, Fcy) of the rectangle surrounding the face of the test subject on the test imaging image, and the width Fcw and height Fch of this rectangle. The presence or absence of detection of the face of the test subject is represented, for example, by "1" when detected and "0" when not detected. Also, the center coordinates of the rectangle surrounding the face are expressed in the coordinate system of the frame Im(k). Note that the upper left of the frame Im(k) is the origin, the right direction of the frame Im(k) is the positive direction of the x-axis, and the downward direction of the frame Im(k) is the positive direction of the y-axis. The skin area detection unit 121 outputs the generated skin area information S(k) to the measurement area setting unit 122 of the luminance data extraction unit 12.

[0024] Next, based on the frame Im(k) of the test imaging image acquired by the imaging image acquisition unit 11 and the skin area information S(k) output by the skin area detection unit 121, the measurement area setting unit 122 sets a plurality of measurement areas for extracting luminance data, which is a signal indicating a luminance change, in the image area corresponding to the skin area indicated by the skin area information S(k) on the frame Im(k). Note that the measurement area setting unit 122 may acquire the test imaging image acquired by the imaging image acquisition unit 11 via the skin area detection unit 121. When the measurement area setting unit 122 sets a plurality of measurement areas, it generates measurement area information R(k) indicating the set plurality of measurement areas. The measurement area information R(k) includes information indicating the positions and sizes of the Rn (positive integer) measurement areas on the test imaging image. Each measurement area is denoted as ri(k) (i = 1, 2, ···, Rn). In Embodiment 1, the measurement area ri(k) is a quadrilateral, and the position and size of the measurement area ri(k) are the coordinate values of the four vertices of the quadrilateral on the test imaging image.

[0025] Here, FIGS. 2A, 2B, and 2C are diagrams for explaining an example of a method for setting measurement areas by the measurement area setting unit 122 in the model generation device 10 according to Embodiment 1. Using FIG. 2, an example of a method for the measurement area setting unit 122 to set a plurality of measurement areas will be described. First, as shown in FIGS. 2A and 2B, in the skin area sr indicated by the skin area information S(k), the measurement area setting unit 122 detects Ln (positive integer) landmarks of facial organs such as the outer corner of the eye, the inner corner of the eye, the nose, and the mouth. In FIGS. 2A and 2B, the landmarks are indicated by circles. The measurement area setting unit 122 sets a vector storing the coordinate values of the detected landmarks as L(k). Note that the measurement area setting unit 122 may detect facial organs using known means such as using a model called Constrained Local Model (CLM).

[0026] Next, the measurement area setting unit 122 sets the vertex coordinates of the quadrilateral of the measurement area ri(k) based on the detected landmarks. For example, the measurement area setting unit 122 sets the vertex coordinates of a quadrilateral as shown in FIG. 2C and sets Rn measurement areas ri(k).

[0027] Taking an example where the measurement area setting unit 122 sets the measurement area ri(k) in a portion corresponding to the cheek of the skin area sr as an example, the measurement area setting unit 122 selects the landmark LA1 of the facial contour and the landmark LA2 of the nose. The measurement area setting unit 122 may first select the landmark LA2 of the nose and then select the landmark LA1 of the facial contour closest to the landmark LA2 of the nose. Then, the measurement area setting unit 122 sets auxiliary landmarks a1, a2, and a3 so as to divide the line segment between the landmark LA1 and the landmark LA2 into four equal parts. Similarly, the measurement area setting unit 122 selects the landmark LB1 of the facial contour and the landmark LB2 of the nose. Also, the measurement area setting unit 122 sets auxiliary landmarks b1, b2, and b3 so as to divide the line segment between the landmark LB1 and the landmark LB2 into four equal parts. Note that the landmarks LB1 and LB2 may be selected from, for example, the facial contour or the nose landmark adjacent to the landmarks LA1 and LA2, respectively. The measurement area setting unit 122 sets a quadrilateral area surrounded by the auxiliary landmarks a1, b1, b2, and a2 as one measurement area R1. The auxiliary landmarks a1, b1, b2, and a2 respectively have vertex coordinates corresponding to the measurement area R1. Similarly, the measurement area setting unit 122 sets one measurement area R2 surrounded by the auxiliary landmarks a2, b2, b3, and a3 and the vertex coordinates of the measurement area R2.

[0028] Here, an example of setting the measurement area ri(k) in the corresponding part of the cheek has been described. However, the measurement area setting unit 122 similarly sets the measurement area ri(k) and the vertex coordinates of the measurement area ri(k) for other parts of the cheek and the skin area sr corresponding to the chin. Although not shown in FIG. 2C, the measurement area setting unit 122 may set the measurement area ri(k) in the part corresponding to the forehead or the part corresponding to the tip of the nose of the skin area sr of the test subject.

[0029] The measurement area setting unit 122 outputs the generated measurement area information R(k) to the signal extraction unit 123 of the luminance data extraction unit 12.

[0030] Next, based on the frame Im(k) of the test captured image acquired by the imaging image acquisition unit 11 and the measurement area information R(k) output from the measurement area setting unit 122, the signal extraction unit 123 extracts, from each of the plurality of measurement areas ri(k) indicated by the measurement area information R(k) on the frame Im(k), a signal indicating the luminance change during a predetermined period, in other words, during the period corresponding to the number of frames Tp, as luminance data. The signal extraction unit 123 may acquire the test captured image acquired by the imaging image acquisition unit 11 via the skin area detection unit 121 and the measurement area setting unit 122. When the signal extraction unit 123 extracts the luminance data, it generates luminance change information W(t) indicating the extracted luminance data.

[0031] The luminance change information W(t) includes information indicating the luminance data wi(t) extracted from the measurement region ri(k). The luminance data wi(t) is time-series data over Tp minutes. For example, it is extracted based on the frames Im(k-Tp+1), Im(k-Tp+2), ···, Im(k) of the past Tp minutes and the measurement region information R(k-Tp+1), R(k-Tp+2), ···, R(k). When extracting the luminance data wi(t), the signal extraction unit 123 calculates the luminance feature amount Gi(j) (j = k-Tp+1, k-Tp+2, ···, k) of each measurement region ri(k) for each frame Im(k) of the test imaging image. The luminance feature amount Gi(j) is a value calculated based on the luminance value on the frame Im(j) of the test imaging image for each measurement region ri(j). The luminance feature amount Gi(j) is, for example, the average or variance of the luminance values of the pixels included in the measurement region ri(j). In Embodiment 1, as an example, the luminance feature amount Gi(j) is the average of the luminance values of the pixels included in the measurement region ri(j). The signal extraction unit 123 arranges the calculated Gi(j) in time series for each frame Im(k) of the test imaging image to obtain the luminance data wi(t). That is, the signal extraction unit 123 sets the luminance data wi(t) = [Gi(k-Tp+1), Gi(k-Tp+2), ···, Gi(k)].

[0032] The signal extraction unit 123 generates the luminance change information W(t) by combining the luminance data wi(t) in each measurement region ri(k). The signal extraction unit 123 outputs the generated luminance change information W(t) to the model learning unit 15.

[0033] Note that the luminance data wi(t) includes various noise components in addition to the aforementioned pulse wave components and the movement components of the test subject (here, the face). Examples of the noise components include noise due to element defects of the imaging device 20. In order to remove these noise components, it is desirable to perform filter processing as preprocessing on the luminance data wi(t). For example, the signal extraction unit 123 performs the filter processing. In the filtering process, the luminance data wi(t) is processed using, for example, a low-pass filter, a high-pass filter, or a band-pass filter. In the following description, it will be described as being subjected to a band-pass filter. As the band-pass filter, for example, a Butterworth filter or the like can be used. As the cut-off frequencies of the band-pass filter, for example, it is desirable that the lower cut-off frequency is 0.5 Hz and the higher cut-off frequency is 5.0 Hz.

[0034] The biological reference data acquisition unit 13 acquires biological reference data, which is the second biological data of the test subject measured by the device 30. The biological reference data acquisition unit 13 outputs the acquired biological reference data to the data processing unit 14.

[0035] The data processing unit 14 generates data (hereinafter referred to as "learning waveform data") that mimics the waveform of the first biological data from the biological reference data acquired by the biological reference data acquisition unit 13. Here, the data processing unit 14 generates learning waveform data that mimics the waveform of the pulse wave data from the electrocardiogram data. As described above, the type of the first biological data is different from the type of the biological reference data, which is the second biological data. That is, the data processing unit 14 generates learning waveform data that mimics the waveform of the first biological data of a type different from the type of the second biological data measured as the biological reference data by the device 30. The data processing unit 14 will be described in detail.

[0036] First, the first data processing unit 141 of the data processing unit 14 detects the peaks in the biological reference data Br(n) from the biological reference data Br(n), and generates biological reference data Br(n) (hereinafter referred to as "peak-added biological reference data Brp(n)") with the detected peaks added to the biological reference data Br(n). The peak-added biological reference data Brp(n) is time-series data for Tq minutes. For example, it is generated based on the biological reference data Br(n - Tq + 1), Br(n - Tq + 2), ···, Br(n) for the past Tq minutes. The first data processing unit 141 arranges the biological reference data Br(n - Tq + 1), Br(n - Tq + 2), ···, Br(n) for the past Tq minutes in time series and detects peaks in the period corresponding to the number of data Tq.

[0037] The first data processing unit 141 may detect peaks in the biological reference data Br using a known peak detection method. Here, an example of the peak detection method will be described with reference to the drawings. FIG. 3A and FIG. 3B are diagrams for explaining an example of the peak detection method by the first data processing unit 141 in Embodiment 1. For example, assume that the first data processing unit 141 detects a peak in a waveform (indicated by "W" in FIG. 3A) as shown in FIG. 3A. The first data processing unit 141 determines that it is a peak when the following (Condition 1) and (Condition 2) are satisfied. However, for the first peak, the first data processing unit 141 detects the peak using only Condition (1). (Condition 1) The waveform is in a "convex" shape In the example of FIG. 3A, "y(t) - y(t - Ta)>θa and y(t) - y(t + Tb)>θb" (Condition 2) A certain amount of time has elapsed since the previously detected peak In the example of FIG. 3A, "t - tb>Tc"

[0038] Note that "tb" in the above (Condition 2) indicates the position (frame number) of the previous peak.

[0039] Also, "Ta" in the above (Condition 1) indicates how many frames before to refer to the value when determining "convex". For example, "6 (frames)" is set in "Ta" in advance. For example, in the case of an average pulse rate of 60 beats per minute, since one "convex" appears per second, it is assumed to refer to the value 0.20 seconds before corresponding to 1 / 5 cycle. (For example, it may be assumed to refer to the value 1 / 2 cycle before, but there is a possibility that a valley will appear in between.) Then, the value obtained by replacing the assumed time (0.20 seconds in the above example) with the number of frames according to the frame rate is set in "Ta". For example, if the frame rate is 30 fps, 0.20 seconds before means 6 frames before.

[0040] Also, "Tb" in the above (Condition 1) indicates how many frames after to refer to the value when determining "convex". Similar to "Ta", for "Tb", after assuming how many seconds after to refer to the value, the value replaced with the number of frames according to the frame rate is set. For example, in "Tb", "6 (frames)" is set in advance in the same way as the above "Ta".

[0041] Also, "θa" and "θb" in the above (Condition 1) are thresholds for finding a "convex" with a certain height. For example, the first data processing unit 141 determines a standard for the height of the "convex" from the maximum value and the minimum value within Td seconds before and after t. The first data processing unit 141 may calculate "θa" and "θb" by, for example, the following formula (1). That is, the first data processing unit 141 may determine "θa" and "θb" so as to find a mountain having a height of 1 / 4 or more of (the maximum value - the minimum value). (See FIG. 3B. For the sake of convenience of explanation, the waveform shown in FIG. 3B is a waveform with a different shape from the waveform shown in FIG. 3A.) θa = θb = (max - min) / 4 ···(1)

[0042] In addition, "Tc" in the above (Condition 2) indicates the frame number that should have elapsed since the previous peak.

[0043] Note that the peak detection method as described above is just an example, and the first data processing unit 141 may detect peaks using other known techniques. Also, the peak detection method and parameters as described above are just examples, and they may be variably applied according to the subject's heart rate, etc. For example, the same value may be set for "Ta" and "Tb" as described above, or different values may be set for "Ta" and "Tb". Also, in the example as described above, it was assumed that peaks that are "convex" upward are detected, but this is just an example, and peaks that are "convex" downward may be detected instead.

[0044] The first data processing unit 141 assigns data in which the detected peaks in the past Tq minutes of the biological reference data Br(n - Tq + 1), Br(n - Tq + 2), ···, Br(n) are known to obtain the peak-added biological reference data Brp(t). Then, the first data processing unit 141 outputs the generated peak-added biological reference data Brp(t) to the second data processing unit 142.

[0045] The second data processing unit 142 generates learning waveform data WT(t) based on the peaks detected by the first data processing unit 141 for the peak-added biological reference data Brp(t) generated by the first data processing unit 141. The second data processing unit 142 generates the learning waveform data WT(t) such that the error between the interval between peaks in the learning waveform data WT(t) and the interval between peaks in the peak-added biological reference data Brp(t) is within a preset threshold value (hereinafter referred to as the "interval determination threshold value"). The interval determination threshold value is set in advance by developers or the like. The developers or the like set the allowable range of error that can be regarded as the intervals between peaks in the learning waveform data WT(t) and the intervals between peaks in the peak-added biological reference data Brp(t) being the same as the interval determination threshold value.

[0046] For example, when the second data processing unit 142 sets each peak so that the error from the interval between peaks in the peak-added biological reference data Brp(t) is within the interval determination threshold, for the set peaks, it generates a waveform imitating the waveform of the first biological data by smoothly connecting adjacent peaks to each other, and sets this as the learning waveform data WT(t). Note that in Embodiment 1, "smoothly connecting the peaks" means "connecting the peaks so as to form a smooth curve".

[0047] Also, for example, when the second data processing unit 142 sets each peak so that the error from the interval between peaks in the peak-added biological reference data Brp(t) is within the interval determination threshold, for the set peaks, it generates a waveform imitating the waveform of the first biological data by connecting them with Gaussian distribution curves centered on each peak, and sets this as the learning waveform data WT(t).

[0048] For the generated learning waveform data WT(t), the second data processing unit 142 may remove components in frequency bands not included in the first biological data by filtering in order to make it not include information in unnecessary frequency bands. The second data processing unit 142 outputs the generated learning waveform data WT(t) to the model learning unit 15.

[0049] Here, FIGS. 4 and 5 are diagrams for explaining an example of the flow from when the data processing unit 14 detects peaks from the biological reference data Br(n) for the past Tq minutes and generates the learning waveform data WT(t) in the model generation device 10 according to Embodiment 1.

[0050] FIG. 4 shows the process from when the first data processing unit 141 of the data processing unit 14 detects peaks in the time-series biological reference data Br(n) of the past Tq minutes measured by the device 30 (that is, the electrocardiogram data of the past Tq minutes measured by an electrocardiograph here (see the left figure in FIG. 4)) to generate the peak-added biological reference data Brp(t) (see the middle figure in FIG. 4), and then the second data processing unit 142 of the data processing unit 14 generates the learning waveform data WT(t) by smoothly connecting adjacent peaks for the peaks set such that the error from the interval between peaks in the peak-added biological reference data Brp(t) is within the interval determination threshold (see the right figure in FIG. 4).

[0051] FIG. 5 shows the process from when the first data processing unit 141 of the data processing unit 14 detects peaks in the time-series biological reference data Br(n) of the past Tq minutes measured by the device 30 to generate the peak-added biological reference data Brp(t), and then the second data processing unit 142 of the data processing unit 14 generates the learning waveform data WT(t) by connecting each peak with a Gaussian distribution curve centered on the peak for the peaks set such that the error from the interval between peaks in the peak-added biological reference data Brp(t) is within the interval determination threshold. In FIG. 5, for simplicity of explanation, since the process from when the first data processing unit 141 detects peaks in the time-series biological reference data Br(n) of the past Tq minutes measured by the device 30 to generate the peak-added biological reference data Brp(t) is as shown in the left figure and the middle figure of FIG. 4, the illustration is omitted.

[0052] The model learning unit 15 generates a machine learning model that takes luminance data as input and outputs first biological data, using the luminance data extracted by the luminance data extraction unit 12, more specifically, the luminance change information W(t) generated by the signal extraction unit 123 and the learning waveform data WT(t) generated by the data processing unit 14 as learning data. Note that in the learning data, the learning waveform data WT(t) is teacher data. The model learning unit 15 learns a machine learning model so that the difference between the first biological data as an estimated result and the learning waveform data WT(t) is reduced, using the learning waveform data WT(t) which is teacher data as a template. The model learning unit 15 may learn by a known learning method to generate a machine learning model. For example, the model learning unit 15 generates a machine learning model by using the technique described in the following Reference 1. <Reference 1> Comas, Armand; Marks, Tim; Mansour, Hassan; Lohit, Suhas; Ma, Yechi; Liu, Xiaoming, ”TURNIP: TIME-SERIES U-NET WITH RECURRENCE FOR NIR IMAGING PPG”, TR2021-099 September 10, 2021

[0053] The model learning unit 15 stores the generated machine learning model in the model storage unit 16.

[0054] The model storage unit 16 stores the machine learning model generated by the model learning unit 15. In FIG. 1, the model storage unit 16 is shown as being provided in the model generation device 10, but this is merely an example. The model storage unit 16 may be provided, for example, at a location external to the model generation device 10 and accessible by the model generation device 10 and the biological data estimation device 40.

[0055] Next, a configuration example of the biological data estimation device 40 will be described. The biological data estimation device 40 is assumed to be mounted on a vehicle, for example. As shown in FIG. 1, the biological data estimation device 40 includes an imaging image acquisition unit 41 for estimation, a luminance data extraction unit 42 for estimation, and an estimation unit 43. The luminance data extraction unit 42 for estimation includes a muscle region detection unit 421 for estimation, a measurement region setting unit 422 for estimation, and a signal extraction unit 423 for estimation.

[0056] The imaging image acquisition unit 41 for estimation acquires an imaging image for estimation of the subject. More specifically, the imaging image acquisition unit 41 for estimation acquires an imaging image for estimation that the imaging device 20 mounted on the vehicle has imaged the driver of the vehicle. The imaging image acquisition unit 41 for estimation outputs the acquired imaging image for estimation to the luminance data extraction unit 42 for estimation.

[0057] The luminance data extraction unit 42 for estimation extracts luminance data indicating the luminance change in the skin area of the subject from the imaging image for estimation acquired by the imaging image acquisition unit 41 for estimation. The luminance data extraction unit 42 for estimation may extract luminance data indicating the luminance change in the skin area of the subject from the imaging image for estimation by the same method as the method in which the luminance data extraction unit 12 in the model generation device 10 extracts luminance data indicating the luminance change in the skin area of the test performer from the test imaging image.

[0058] The skin area detection unit 421 for estimation in the luminance data extraction unit 42 for estimation detects the skin area of the subject from the frame Im(k) included in the imaging image for estimation acquired by the imaging image acquisition unit 41 for estimation. The skin area detection unit 421 for estimation may detect the skin area of the subject from the frame Im(k) included in the imaging image for estimation by the same method as the method in which the skin area detection unit 121 in the model generation device 10 detects the skin area of the test performer from the frame Im(k) included in the test imaging image. Since the method in which the skin area detection unit 121 detects the skin area of the test performer from the frame Im(k) included in the test imaging image has been described, the overlapping description is omitted. The skin area detection unit 421 for estimation outputs the generated skin area information S(k) to the measurement area setting unit 422 for estimation in the luminance data extraction unit 42 for estimation.

[0059] The measurement region setting unit 422 for estimation of the luminance data extraction unit 42 for estimation sets a plurality of measurement regions for extracting luminance data, which is a signal indicating luminance change, in the image region corresponding to the skin region indicated by the skin region information S(k) on the frame Im(k) based on the frame Im(k) of the estimation captured image acquired by the estimation captured image acquisition unit 41 and the skin region information S(k) output by the skin region detection unit 421 for estimation. The measurement region setting unit 422 for estimation may set a plurality of measurement regions in the image region corresponding to the skin region indicated by the skin region information S(k) on the frame Im(k) of the estimation captured image in the same manner as the method in which the measurement region setting unit 122 in the model generation device 10 sets a plurality of measurement regions in the image region corresponding to the skin region indicated by the skin region information S(k) on the frame Im(k) based on the frame Im(k) of the test captured image and the skin region information S(k) output by the skin region detection unit 121. Since the method by which the measurement region setting unit 122 sets the measurement regions has been described, duplicate descriptions are omitted. When the measurement region setting unit 422 for estimation sets the measurement regions, it generates measurement region information R(k) indicating the set plurality of measurement regions and outputs the generated measurement region information R(k) to the signal extraction unit 423 for estimation of the luminance data extraction unit 42 for estimation.

[0060] The signal extraction unit 423 for estimation of the luminance data extraction unit 42 for estimation extracts, as luminance data, a signal indicating luminance change in a predetermined period, in other words, in the period corresponding to the number of frames Tp, from each of the plurality of measurement regions ri(k) indicated by the measurement region information R(k) on the frame Im(k) based on the frame Im(k) of the estimation captured image acquired by the estimation captured image acquisition unit 41 and the measurement region information R(k) output from the measurement region setting unit 422 for estimation. The signal extraction unit for estimation 423 extracts luminance data from each of the plurality of measurement regions ri(k) indicated by the measurement region information R(k) on the frame Im(k) of the imaging image for estimation in the same way as the signal extraction unit 123 in the model generation device 10 extracts luminance data from each of the plurality of measurement regions ri(k) indicated by the measurement region information R(k) on the frame Im(k) of the test imaging image. Since the method by which the signal extraction unit 123 extracts luminance data has already been described, duplicate explanations are omitted. Note that the signal extracted as the luminance data here is the signal that is the basis of the first biological data. The biological data estimation device 40 estimates the first biological data of the subject, here, the pulse wave data, using the luminance data and the machine learning model. The estimation of the subject's pulse wave data is performed by the estimation unit 43. Details of the estimation unit 43 will be described later. The signal extraction unit for estimation 423 generates, as luminance change information W(t), the combined luminance data wi(t) in each measurement region ri(k). The signal extraction unit for estimation 423 outputs the generated luminance change information W(t) to the estimation unit 43.

[0061] The estimation unit 43 estimates the first biological data of the subject, here, the driver of the vehicle, by inputting the luminance data extracted by the luminance data extraction unit 42 for estimation, more specifically, the luminance change information W(t) generated by the luminance data extraction unit 42 for estimation, into the machine learning model to obtain the first biological data. The estimation unit 43 may acquire the machine learning model from the model storage unit 16 of the model generation device 10. The estimation unit 43 outputs the biological data estimation result P(t), which is the pulse wave data as the estimated first biological data, to an external device (not shown). The pulse wave data is, for example, the time-series data of the subject's pulse wave estimated by the estimation unit 43. The estimation unit 43 may detect the subject's pulse rate (the number of beats per minute) from the pulse wave data and output the subject's pulse rate to an external device, or may output the pulse wave interval (the interval between peaks) to an external device.

[0062] The external device that is the output destination of the biological data estimation result P(t) by the estimation unit 43 is, for example, a state estimation device that estimates the state of the subject. For example, the state estimation device estimates the degree of wakefulness indicating whether the driver is awake or not as the state of the subject, here the driver of the vehicle. When the wakefulness of the driver is decreasing, the state estimation device outputs a warning sound for warning the driver of the decrease in wakefulness via an output device (not shown).

[0063] The operations of the model generation device 10 and the biological data estimation device 40 according to the first embodiment will be described. First, the operation of the model generation device 10 according to the first embodiment will be described. FIG. 6 is a flowchart for explaining the operation of the model generation device 10 according to the first embodiment. When the model generation device 10 receives a start instruction input by a developer or the like from a notebook PC or the like, for example, it executes an operation as shown in the flowchart of FIG. 6. More specifically, for example, a control unit (not shown) of the model generation device 10 receives the start instruction input by a developer or the like, and when receiving the start instruction, starts operations of the imaging image acquisition unit 11, the luminance data extraction unit 12, the biological reference data acquisition unit 13, and the data processing unit 14.

[0064] The imaging image acquisition unit 11 acquires a test imaging image obtained by imaging a test performer (step ST1). More specifically, the imaging image acquisition unit 11 acquires a test imaging image obtained by an imaging device 20 mounted on the vehicle imaging a developer or the like performing a test drive of the vehicle. The imaging image acquisition unit 11 outputs the acquired test imaging image to the luminance data extraction unit 12.

[0065] The luminance data extraction unit 12 performs a process of extracting luminance data indicating a luminance change in the skin region of the test performer from the test imaging image acquired by the imaging image acquisition unit 11 in step ST1 (step ST2).

[0066] The biological reference data acquisition unit 13 acquires biological reference data, which is the second biological data of the test subject measured by the device 30 (step ST3). The biological reference data acquisition unit 13 outputs the acquired biological reference data to the data processing unit 14.

[0067] The data processing unit 14 generates learning waveform data that mimics the waveform of the first biological data, that is, the pulse wave data here, from the biological reference data acquired by the biological reference data acquisition unit 13 in step ST3 (step ST4). Specifically, first, the first data processing unit 141 detects peaks in the biological reference data Br(n) from the biological reference data Br(n), and generates peak-added biological reference data Brp(n). The first data processing unit 141 outputs the generated peak-added biological reference data Brp(t) to the second data processing unit 142. Next, the second data processing unit 142 generates learning waveform data WT(t) based on the peaks detected by the first data processing unit 141 for the peak-added biological reference data Brp(t) generated by the first data processing unit 141. The second data processing unit 142 outputs the generated learning waveform data WT(t) to the model learning unit 15.

[0068] The model learning unit 15 generates a machine learning model that takes the luminance data extracted by the luminance data extraction unit 12 in step ST2, more specifically, the luminance change information W(t) generated by the signal extraction unit 123, and the learning waveform data WT(t) generated by the data processing unit 14 in step ST4 as learning data, takes the luminance data as input, and outputs the first biological data (step ST5). The model learning unit 15 stores the generated machine learning model in the model storage unit 16.

[0069] FIG. 7 is a flowchart for explaining the details of the process of extracting luminance data by the luminance data extraction unit 12 in step ST2 of FIG. 6. The skin area detection unit 121 detects the skin area of the test performer from the frame Im(k) included in the test captured image acquired by the captured image acquisition unit 11 in step ST1 of FIG. 6 (step ST21). The skin area detection unit 121 generates skin area information S(k) indicating the detected skin area. The skin area detection unit 121 outputs the generated skin area information S(k) to the measurement area setting unit 122.

[0070] Based on the frame Im(k) of the test captured image acquired by the captured image acquisition unit 11 in step ST1 of FIG. 6 and the skin area information S(k) output by the skin area detection unit 121 in step ST21, the measurement area setting unit 122 sets a plurality of measurement areas for extracting luminance data, which is a signal indicating a luminance change, in the image area corresponding to the skin area indicated by the skin area information S(k) on the frame Im(k) (step ST22). When setting a plurality of measurement areas, the measurement area setting unit 122 generates measurement area information R(k) indicating the set plurality of measurement areas. The measurement area setting unit 122 outputs the generated measurement area information R(k) to the signal extraction unit 123.

[0071] Based on the frame Im(k) of the test captured image acquired by the captured image acquisition unit 11 in step ST1 of FIG. 6 and the measurement area information R(k) output from the measurement area setting unit 122 in step ST22, the signal extraction unit 123 extracts, as luminance data, a signal indicating a luminance change in a predetermined period, in other words, a period corresponding to the number of frames Tp, from each of the plurality of measurement areas ri(k) indicated by the measurement area information R(k) on the frame Im(k) (step ST23). When extracting the luminance data, the signal extraction unit 123 generates luminance change information W(t) indicating the extracted luminance data. The signal extraction unit 123 outputs the generated luminance change information W(t) to the model learning unit 15.

[0072] Next, the operation of the biological data estimation device 40 according to Embodiment 1 will be described. FIG. 8 is a flowchart for explaining the operation of the biological data estimation device 40 according to the first embodiment. When the power of the vehicle is turned on, for example, the biological data estimation device 40 executes the operation as shown in the flowchart of FIG. 8. The biological data estimation device 40 repeats the operation as shown in the flowchart of FIG. 8 until the power of the vehicle is turned off, for example.

[0073] The imaging image acquisition unit 41 for estimation acquires an imaging image for estimation of the subject (step ST401). More specifically, the imaging image acquisition unit 41 for estimation acquires an imaging image for estimation of the driver of the vehicle imaged by the imaging device 20 mounted on the vehicle. The imaging image acquisition unit 41 for estimation outputs the acquired imaging image for estimation to the luminance data extraction unit 42 for estimation.

[0074] The luminance data extraction unit 42 for estimation performs a process of extracting luminance data indicating a luminance change in the skin area of the subject from the imaging image for estimation acquired by the imaging image acquisition unit 41 in step ST401 (step ST402).

[0075] Specifically, the skin area detection unit 421 for estimation detects the skin area of the subject from the frame Im(k) included in the imaging image for estimation acquired by the imaging image acquisition unit 41 in step ST401, and generates skin area information S(k). The skin area detection unit 421 for estimation outputs the generated skin area information S(k) to the measurement area setting unit 422 for estimation.

[0076] Next, based on the frame Im(k) of the imaging image for estimation acquired by the imaging image acquisition unit 41 in step ST401 and the skin area information S(k) output by the skin area detection unit 421 for estimation, the measurement area setting unit 422 for estimation sets a plurality of measurement areas for extracting luminance data, which is a signal indicating a luminance change, in the image area corresponding to the skin area indicated by the skin area information S(k) on the frame Im(k), and generates measurement area information R(k). The measurement area setting unit 422 for estimation outputs the generated measurement area information R(k) to the signal extraction unit 423 for estimation.

[0077] Then, based on the frame Im(k) of the captured image for estimation obtained by the captured image acquisition unit 41 for estimation in step ST401 and the measurement area information R(k) output from the measurement area setting unit 422 for estimation, the signal extraction unit 423 for estimation extracts, as luminance data, a signal indicating the luminance change in a predetermined period, in other words, in the period corresponding to the number of frames Tp, from each of the plurality of measurement areas ri(k) indicated by the measurement area information R(k) on the frame Im(k). The signal extraction unit 423 for estimation generates luminance change information W(t) indicating the extracted luminance data. The signal extraction unit 423 for estimation outputs the generated luminance change information W(t) to the estimation unit 43.

[0078] The estimation unit 43 estimates the first biological data of the subject, here the driver of the vehicle, by inputting the luminance data extracted by the luminance data extraction unit 42 for estimation in step ST402, more specifically, the luminance change information W(t) generated by the luminance data extraction unit 42 for estimation, into the machine learning model to obtain the first biological data (step ST403). The estimation unit 43 outputs the biological data estimation result P(t), which is the pulse wave data as the estimated first biological data, to an external device.

[0079] As described above, in order to accurately obtain the first biological data of the subject using the machine learning model, the machine learning model is generated based on the learning data collected in the scene where the first biological data is to be obtained, and it is required that the second biological data as the teacher data included in the learning data has little noise. For example, in order to accurately obtain the first biological data of the driver of the vehicle using the machine learning model in the driving scene of the vehicle, the machine learning model is generated based on the learning data collected in the driving scene of the vehicle, and it is required that the second biological data as the teacher data included in the learning data has little noise. This is because the machine learning model is generated based on training data that is obtained under a situation assumed to be the same as the situation where the machine learning model is actually utilized and has little noise in the teacher data, enabling the machine learning model to have good estimation accuracy. When the estimation accuracy of the machine learning model is good, the obtained estimation result, that is, the estimation accuracy of the first biological data, is also good. However, in the scenario of obtaining the first biological data, there is a problem that events that cause noise, such as vibration or external light, may occur, and the collected second biological data may contain a lot of noise. For example, in the driving scenario of a vehicle, events that cause noise, such as vehicle vibration or external light, occur, so the collected second biological data may contain a lot of noise. If the second biological data contains a lot of noise, it is difficult to generate a machine learning model for accurately obtaining the first biological data using the second biological data as teacher data.

[0080] Here, FIG. 9 is a diagram for explaining an example in which the accuracy of the first biological data obtained from the machine learning model generated based on the training data including the teacher data, in other words, the accuracy of the machine learning model, differs depending on the type of teacher data. For example, as shown in FIG. 9A, assume that the second biological data is pulse wave data measured by a pulse wave sensor. The pulse wave waveform indicated by the pulse wave data has a waveform shape similar to the waveform of the luminance data included in the training data, in other words, the data that is the basis of the first biological data to be input and estimated to the machine learning model. In this way, when the pulse wave data having a waveform shape similar to the waveform of the data that is the basis of the first biological data to be estimated is used as teacher data, learning is easy. However, as described above, the second biological data collected by the pulse wave sensor is contaminated with noise caused by vibrations, external light, or the like. For example, when attempting to measure the driver's pulse wave data with a pulse wave sensor in a vehicle during driving, the pulse wave data collected by the pulse wave sensor is contaminated with noise caused by vibrations of the vehicle during driving, external light, or the like. Then, even if a machine learning model is generated using the pulse wave data collected by the pulse wave sensor as teacher data, it may not be a machine learning model that can accurately obtain the first biological data. The pulse wave sensor is prone to noise caused by vibrations, external light, or the like in the measurement results. That is, it can be said that the pulse wave sensor is not suitable as a device 30 for collecting teacher data for generating a machine learning model that is used in an environment where noise is likely to be caused by vibrations, external light, or the like, such as a vehicle during driving.

[0081] On the other hand, the electrocardiogram data collected by the electrocardiograph is less likely to be contaminated with noise caused by vibrations of the vehicle during driving, external light, or the like. That is, the electrocardiograph is less likely to be contaminated with noise caused by vibrations, external light, or the like in the measurement results, and it can be said that it is suitable as a device 30 for collecting teacher data for generating a machine learning model that is used in an environment where noise is likely to be caused by vibrations, external light, or the like, such as a vehicle during driving. However, the electrocardiogram data has a waveform with sharp peaks, and the waveform shape is different from the waveform shape of the luminance data, which is the data that is the basis for estimating the first biological data. Therefore, when using the electrocardiogram data as teacher data in its waveform shape as it is, learning is difficult. FIG. 10 is a diagram for explaining the difference between the waveform shape of the pulse wave data and the waveform shape of the electrocardiogram data. As shown in FIG. 10, the waveform shape of the pulse wave data is similar to the waveform shape of the luminance data, whereas the waveform shape of the electrocardiogram data is not similar to the waveform shape of the luminance data.

[0082] Therefore, as shown in FIG. 9B, the model generation device 10 according to Embodiment 1 acquires electrocardiogram data as biological reference data, which is the second biological data, and employs the electrocardiogram data as teacher data for generating a machine learning model. During learning, the electrocardiogram data is processed into learning waveform data that mimics the waveform of the pulse wave data, that is, a pulse wave-like waveform. Then, the model generation device 10 generates a machine learning model using the learning waveform data as teacher data. As a result, the model generation device 10 can generate a machine learning model that can accurately obtain the first biological data. As a result, the biological data estimation device 40 that estimates the first biological data, in other words, the driver's pulse wave data, using the machine learning model generated by the model generation device 10 can accurately estimate the driver's pulse wave data.

[0083] In the above Embodiment 1, the first biological data is pulse wave data, but this is merely an example. For example, the first biological data may be a blood pressure waveform. In this case, the model generation device 10 generates learning waveform data having a waveform shape that mimics the blood pressure waveform from the biological reference data as the second biological data acquired from the device 30, and generates a machine learning model that takes the learning waveform data as teacher data, inputs luminance data, and outputs a blood pressure waveform. In the biological data estimation device 40, the estimation unit 43 inputs the luminance data extracted by the estimation luminance data extraction unit 42, more specifically, the luminance change information W(t) generated by the estimation luminance data extraction unit 42, into the machine learning model to obtain the first biological data, and estimates the blood pressure waveform of the subject, here the driver of the vehicle, as the first biological data. The estimation unit 43 outputs the biological data estimation result P(t), which is the blood pressure waveform as the estimated first biological data, to an external device. The estimation unit 43 may detect the blood pressure of the subject from the blood pressure waveform and output the blood pressure of the subject to an external device.

[0084] In addition, in the above-described Embodiment 1, the subject was the driver of the vehicle, but this is merely an example. For example, the subject may be a passenger other than the driver of the vehicle, such as a passenger in the passenger seat or a passenger in the rear seat. In this case, the test performer is, for example, a fellow passenger riding in a vehicle that has been test-driven before shipment of the vehicle.

[0085] FIGS. 11A and 11B are diagrams showing an example of the hardware configuration of the model generation device 10 according to Embodiment 1. In Embodiment 1, the functions of the imaging image acquisition unit 11, the skin region detection unit 121, the measurement region setting unit 122, the signal extraction unit 123, the biological reference data acquisition unit 13, the first data processing unit 141, the second data processing unit 142, the model learning unit 15, and a control unit (not shown) are realized by a processing circuit 1001. That is, the model generation device 10 includes a processing circuit 1001 for performing control to generate a machine learning model based on second biological data measured by a device 30 in which the influence of noise associated with driving of the vehicle generated in the vehicle interior is less likely to appear in the measurement result. As shown in FIG. 11A, the processing circuit 1001 may be dedicated hardware, or as shown in FIG. 11B, may be a processor 1004 that executes a program stored in a memory.

[0086] When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0087] When the processing circuit is the processor 1004, the functions of the imaging image acquisition unit 11, the skin area detection unit 121, the measurement area setting unit 122, the signal extraction unit 123, the biological reference data acquisition unit 13, the first data processing unit 141, the second data processing unit 142, the model learning unit 15, and the control unit (not shown) are realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in the memory 1005. The processor 1004 reads and executes the program stored in the memory 1005, thereby executing the functions of the imaging image acquisition unit 11, the skin area detection unit 121, the measurement area setting unit 122, the signal extraction unit 123, the biological reference data acquisition unit 13, the first data processing unit 141, the second data processing unit 142, the model learning unit 15, and the control unit (not shown). That is, the model generation device 10 includes a memory 1005 for storing a program that, when executed by the processor 1004, will result in the execution of steps ST1 to ST5 of FIG. 6 described above. Also, the program stored in the memory 1005 can be said to cause the computer to execute the processing procedures or methods of the imaging image acquisition unit 11, the skin area detection unit 121, the measurement area setting unit 122, the signal extraction unit 123, the biological reference data acquisition unit 13, the first data processing unit 141, the second data processing unit 142, the model learning unit 15, and the control unit (not shown). Here, the memory 1005 is, for example, a non-volatile or volatile semiconductor memory such as a RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc), etc.

[0088] Note that, regarding the functions of the imaging image acquisition unit 11, the skin area detection unit 121, the measurement area setting unit 122, the signal extraction unit 123, the biological reference data acquisition unit 13, the first data processing unit 141, the second data processing unit 142, the model learning unit 15, and the control unit (not shown), part of them may be realized by dedicated hardware and part of them may be realized by software or firmware. For example, regarding the imaging image acquisition unit 11 and the biological reference data acquisition unit 13, their functions are realized by the processing circuit 1001 as dedicated hardware, and regarding the skin area detection unit 121, the measurement area setting unit 122, the signal extraction unit 123, the first data processing unit 141, the second data processing unit 142, the model learning unit 15, and the control unit (not shown), their functions can be realized by the processor 1004 reading and executing the program stored in the memory 1005. The model storage unit 16 is composed of a memory 1005, a RAM, a ROM, or the like. In addition, the model generation device 10 includes input interface devices 1002 and output interface devices 1003 that perform wired communication or wireless communication with devices such as an imaging device 20, a device 30, or a biological data estimation device 40.

[0089] An example of the hardware configuration of the biological data estimation device 40 according to Embodiment 1 is also a configuration as shown in FIGS. 11A and 11B. In Embodiment 1, the functions of the estimation imaging image acquisition unit 41, the estimation skin area detection unit 421, the estimation measurement area setting unit 422, the estimation signal extraction unit 423, and the estimation unit 43 are realized by the processing circuit 1001. That is, the biological data estimation device 40 includes a processing circuit 1001 for performing control to estimate the first biological data of the subject using the machine learning model generated by the model generation device 10.

[0090] When the processing circuit is the processor 1004, the functions of the imaging image acquisition unit 41 for estimation, the muscle region detection unit 421 for estimation, the measurement region setting unit 422 for estimation, the signal extraction unit 423 for estimation, and the estimation unit 43 are realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in the memory 1005. The processor 1004 reads and executes the program stored in the memory 1005, thereby executing the functions of the imaging image acquisition unit 41 for estimation, the muscle region detection unit 421 for estimation, the measurement region setting unit 422 for estimation, the signal extraction unit 423 for estimation, and the estimation unit 43. That is, the biological data estimation device 40 includes a memory 1005 for storing a program that, when executed by the processor 1004, will result in the execution of steps ST401 to ST403 of FIG. 8 described above. Also, the program stored in the memory 1005 can be said to cause a computer to execute the processing procedures or methods of the imaging image acquisition unit 41 for estimation, the muscle region detection unit 421 for estimation, the measurement region setting unit 422 for estimation, the signal extraction unit 423 for estimation, and the estimation unit 43.

[0091] Note that, regarding the functions of the imaging image acquisition unit 41 for estimation, the muscle region detection unit 421 for estimation, the measurement region setting unit 422 for estimation, the signal extraction unit 423 for estimation, and the estimation unit 43, part of them may be realized by dedicated hardware and part by software or firmware. For example, the function of the imaging image acquisition unit 41 for estimation can be realized by the processing circuit 1001 as dedicated hardware, and for the muscle region detection unit 421 for estimation, the measurement region setting unit 422 for estimation, the signal extraction unit 423 for estimation, and the estimation unit 43, the processor 1004 can realize their functions by reading and executing the program stored in the memory 1005. Also, the biological data estimation device 40 includes an input interface device 1002 and an output interface device 1003 that perform wired communication or wireless communication with devices such as the model generation device 10 or the imaging device 20.

[0092] In the above-described Embodiment 1, the model generation device 10 was assumed to be installed in a local device such as a notebook PC used by a developer or the like, but this is merely an example. For example, in the model generation device 10, a part of the processing performed by the data processing unit 14 may be executed on a server. FIG. 12 is a diagram for explaining a configuration example of a system in the case where a part of the processing performed by the data processing unit 14 is executed on a server in Embodiment 1.

[0093] For example, as shown in FIG. 12, among the components of the model generation device 10 described with reference to FIG. 1, the imaging image acquisition unit 11, the luminance data extraction unit 12, the biological reference data acquisition unit 13, and the first data processing unit 141 may be provided in the local device 10a, and the second data processing unit 142, the model learning unit 15, and the model storage unit 16 may be provided in the server 10b. The local device 10a and the server 10b constitute a model generation system 100. The local device 10a and the server 10b are connected via a network. Further, the model generation system 100 and the biological data estimation device 40 constitute a biological data estimation system 500a. Note that each component in the model generation system 100 shown in FIG. 12 is the same as each component in the model generation device 10 shown in FIG. 1, and thus the same reference numerals are given and redundant descriptions are omitted. Also, since the detailed configuration example of the biological data estimation device 40 in the biological data estimation system 500a shown in FIG. 12 is the same as the detailed configuration example of the biological data estimation device 40 in the biological data estimation system 500 shown in FIG. 1, the illustration is omitted and redundant descriptions of each component are also omitted.

[0094] The first data processing unit 141 outputs the generated peak-added biological reference data Brp(t) to the second data processing unit 142 via the network. At this time, the first data processing unit 141 may generate data (hereinafter referred to as "processed peak data") with the amount of information reduced to only the information regarding the peaks for the peak-containing biological reference data Brp(t), and output the processed peak data to the second data processing unit 142. FIG. 13 is a diagram for explaining an example of the content of the processed peak data generated by the first data processing unit 141 in a configuration where a part of the processing performed by the data processing unit 14 is executed on the server 10b. The processed peak data is, for example, data in which the time when the recording of the biological reference data started, in other words, the time when the first biological reference data among the biological reference data of the past Tq minutes was measured is set to 0 ms, and the times corresponding to the peak positions are arranged. In FIG. 13, the horizontal axis represents time [ms]. When the first data processing unit 141 detects the peaks of the biological reference data Br(n - Tq + 1), Br(n - Tq + 2), ···, Br(n) of the past Tq minutes, it extracts the times corresponding to the positions of the detected peaks, arranges them in time series, and generates the processed peak data. By having the first data processing unit 141 generate the processed peak data and output the generated processed peak data to the second data processing unit 142, the amount of information of the data output from the first data processing unit 141 to the second data processing unit 142 can be minimized, and the communication volume can be reduced.

[0095] Also, although processing capabilities for learning are required to generate the machine learning model, as shown in FIG. 12, by adopting a configuration in which the model learning unit 15 is provided in the server 10b, it is possible not to require the local device 10a to be a high-spec PC or the like.

[0096] Note that the configuration of the model generation system 100 as shown in FIG. 12 is merely an example. For example, some or all of the skin area detection unit 121, measurement area setting unit 122, signal extraction unit 123, and first data processing unit 141 may be provided in the server 10b. If there is no need to consider the communication volume, the second data processing unit 142 may be provided in the local device 10a (that is, the data processing unit 14 may be provided in the local device 10a). If there is no need to consider the processing ability, the model learning unit 15 may be provided in the local device 10a.

[0097] In addition, in the above-described first embodiment, the biological data estimation device 40 is assumed to be mounted on a vehicle, but this is merely an example. For example, some or all of the imaging image acquisition unit 41 for estimation, skin area detection unit 421 for estimation, measurement area setting unit 422 for estimation, signal extraction unit 423 for estimation, and estimation unit 43 included in the biological data estimation device 40 may be provided in the server.

[0098] In the above-described first embodiment, the model generation device 10, the biological data estimation device 40, and the model generation system 100 are applied to the driving scene of a vehicle, and the subject is assumed to be an occupant of the vehicle such as a driver of the vehicle. However, this is merely an example, and the model generation device 10, the biological data estimation device 40, and the model generation system 100 can be applied to scenes other than the driving scene of a vehicle. For example, the subject may be a train driver or a conductor, and the biological data estimation device 40 may estimate the first biological data of the train driver or the conductor using a machine learning model based on an imaging image for estimation obtained by the imaging device 20 capturing the train driver or the conductor. The model generation device 10 generates a machine learning model used when the biological data estimation device 40 estimates the first biological data of the train driver or the conductor based on a test imaging image obtained by the imaging device 20 capturing a person and biological reference data, which is the second biological data of the person measured by the device 30. Even in the driving scene of a train, similar to the driving scene of a vehicle, events that cause noise, such as vibrations of the train or external light, occur. The model generation device 10 acquires biological reference data (for example, electrocardiogram data), which is the second biological data, and employs the biological reference data as teaching data for generating a machine learning model. During learning, the biological reference data is made into learning waveform data that mimics the waveform of the first biological data, and by generating a machine learning model using the learning waveform data as teaching data, a machine learning model that can accurately obtain the first biological data can be generated. The biological data estimation device 40 can accurately estimate the first biological data of the train driver or conductor.

[0099] Also, for example, the subject may be a worker in a factory or the like. Here, the worker is assumed to be, for example, a person who operates equipment using a liquid crystal panel. The biological data estimation device 40 may be configured to estimate the first biological data of a worker in a factory or the like using a machine learning model based on an imaging image for estimation captured by the imaging device 20 of the worker in a factory or the like. The model generation device 10 generates a machine learning model used when the biological data estimation device 40 estimates the first biological data of a worker in a factory or the like based on a test imaging image captured by the imaging device 20 of a person and biological reference data, which is the second biological data measured by the device 30, of the person. In the work scene of a worker in a factory or the like, events that cause noise, such as light entering the factory, occur. Also, when a worker operates equipment with a finger or moves the body, there is also a problem that a pulse oximeter or the like attached to the fingertip is not suitable as a device for acquiring the first biological data. The model generation device 10 acquires biometric reference data (for example, electrocardiogram data), which is the second biometric data, and uses the biometric reference data as teaching data for generating a machine learning model. During learning, the biometric reference data is used as learning waveform data that mimics the waveform of the first biometric data. By generating a machine learning model using the learning waveform data as teaching data, a machine learning model that can accurately obtain the first biometric data can be generated. The biometric data estimation device 40 can accurately estimate the first biometric data of workers in a factory or the like.

[0100] Also, for example, the subject is an operator who performs PC operations in a living room or the like, or an e-sports competitor. The biometric data estimation device 40 may estimate the first biometric data of the operator or the competitor using a machine learning model based on the estimation captured image captured by the imaging device 20 of the operator or the competitor. The model generation device 10 generates a machine learning model used when the biometric data estimation device 40 estimates the first biometric data of the operator or the competitor based on the test captured image captured by the imaging device 20 of a person and the biometric reference data, which is the second biometric data measured by the device 30 of the person. In a work scene on a PC or an e-sports competition scene, it is necessary to operate a device with fingers or move the body. In such a scene, when trying to estimate the first biometric data of the operator or the competitor, there is a problem that it is not suitable to use a pulse oximeter or the like attached to the fingertip. The model generation device 10 acquires biometric reference data (for example, electrocardiogram data), which is the second biometric data, and uses the biometric reference data as teaching data for generating a machine learning model. During learning, the biometric reference data is used as learning waveform data that mimics the waveform of the first biometric data. By generating a machine learning model using the learning waveform data as teaching data, a machine learning model that can accurately obtain the first biometric data can be generated. The biometric data estimation device 40 can accurately estimate the first biometric data of the operator or the competitor.

[0101] As described above, the model generation device 10 according to Embodiment 1 includes an imaging image acquisition unit 11 that acquires an imaging image including the skin region of the test subject, which has imaged the test subject; a luminance data extraction unit 12 that extracts luminance data indicating the luminance change in the skin region of the test subject from the imaging image acquired by the imaging image acquisition unit 11; a biological reference data acquisition unit 13 that acquires biological reference data, which is the second biological data of the test subject measured by a device 30 in which the influence of noise that can occur in an environment where the first biological data can be obtained in the measurement result is less likely to appear; a data processing unit 14 that generates learning waveform data that mimics the waveform of the first biological data of a type different from the type of the second biological data measured as the biological reference data, which serves as teacher data when learning a machine learning model, from the biological reference data acquired by the biological reference data acquisition unit 13; and a model learning unit 15 that generates a machine learning model that takes the luminance data as an input and outputs the first biological data, using the luminance data extracted by the luminance data extraction unit 12 and the learning waveform data generated by the data processing unit 14 as learning data. Therefore, even when obtaining the first biological data of the subject in a scene where an event that causes noise can occur, the model generation device 10 can provide a machine learning model that can suppress the influence of noise and accurately obtain the first biological data of the subject.

[0102] Also, in the model generation device 10, the data processing unit 14 detects peaks in the biological reference data and generates learning waveform data based on the detected peaks. Therefore, the model generation device 10 can generate teacher data that is easy to learn when generating a machine learning model.

[0103] More specifically, in the model generation device 10, the data processing unit 14 generates the learning waveform data such that the error between the interval between peaks in the learning waveform data and the interval between peaks in the biological reference data is within an interval determination threshold value. Therefore, the model generation device 10 can generate teacher data that is easy to learn when generating a machine learning model. The model generation device 10 can obtain learning waveform data as biological data with a high peak interval reproducibility as teacher data.

[0104] In the model generation device 10, for the peaks set such that the error from the interval between peaks in the biological reference data is within the interval determination threshold, the data processing unit 14 connects adjacent peaks smoothly to each other, so that the learning waveform data has a waveform imitating the waveform of the first biological data. Therefore, the model generation device 10 can generate teacher data that is easy to learn when generating a machine learning model.

[0105] In the model generation device 10, for the peaks set such that the error from the interval between peaks in the biological reference data is within the interval determination threshold, the data processing unit 14 connects each peak with a Gaussian distribution curve centered on each peak, so that the learning waveform data has a waveform imitating the waveform of the first biological data. Therefore, the model generation device 10 can generate teacher data that is easy to learn when generating a machine learning model.

[0106] Also, the biological data estimation device 40 according to Embodiment 1 is a biological data estimation device 40 that estimates the first biological data of a subject using the machine learning model generated by the model generation device 10, and includes an estimation imaging image acquisition unit 41 that acquires an estimation imaging image including the skin region of the subject, which has imaged the subject, and an estimation luminance data extraction unit 42 that extracts luminance data indicating the luminance change in the skin region of the subject from the estimation imaging image acquired by the estimation imaging image acquisition unit 41, and an estimation unit 43 that estimates the first biological data of the subject by inputting the luminance data extracted by the estimation luminance data extraction unit 42 into the machine learning model to obtain the first biological data. Therefore, even when obtaining the first biological data of the subject in a scene where an event that causes noise may occur, the biological data estimation device 40 can estimate the first data of the subject based on a machine learning model that can suppress the influence of noise and accurately obtain the first biological data of the subject, and can accurately estimate the first biological data of the subject even in a situation where noise is likely to occur due to vibration or external light or the like.

[0107] Further, the model generation system 100 according to Embodiment 1 includes an imaging image acquisition unit 11 that acquires an imaging image including the skin area of the test performer, which has imaged the test performer; a luminance data extraction unit 12 that extracts luminance data indicating the luminance change in the skin area of the test performer from the imaging image acquired by the imaging image acquisition unit 11; a biological reference data acquisition unit 13 that acquires biological reference data, which is the second biological data of the test performer measured by a device 30 in which the influence of noise that can occur in an environment where the first biological data can be obtained in the measurement result is less likely to appear; a data processing unit 14 that generates learning waveform data imitating the waveform of the first biological data of a type different from the type of the second biological data measured as the biological reference data, which is the teacher data when learning the machine learning model, from the biological reference data acquired by the biological reference data acquisition unit 13; and a model learning unit 15 that generates a machine learning model that takes the luminance data as an input and outputs the first biological data, using the luminance data extracted by the luminance data extraction unit 12 and the learning waveform data generated by the data processing unit 14 as learning data. Therefore, even when obtaining the first biological data of the subject in a scene where an event that causes noise can occur, the model generation system 100 can provide a machine learning model capable of suppressing the influence of noise and accurately obtaining the first biological data of the subject.

[0108] In the model generation system 100, a part of the processing performed by the data processing unit 14 may be configured to be executed on the server 10b. In this case, the model generation system 100 can reduce the communication volume in the model generation system 100.

[0109] Specifically, the data processing unit 14 includes a first data processing unit 141 that detects peaks in the biological reference data, and a second data processing unit 142 that generates learning waveform data based on the peaks detected by the first data processing unit 141. In the model generation system 100, the luminance data extraction unit 12, the biological reference data acquisition unit 13, and the first data processing unit 141 are provided in the local device 10a, and the second data processing unit 142 and the model learning unit 15 are provided in the server 10b connected to the local device 10a via a network. The first data processing unit 141 generates processed peak data with the amount of information reduced to only information related to the peaks, and outputs the generated processed peak data to the second data processing unit 142. Thereby, the model generation system 100 can reduce the communication volume between the local device 10a and the server 10b. In addition, the model generation system 100 can be configured to include the model learning unit 15 in the server 10b, so that it is not necessary for the local device 10a to be a high - spec PC or the like.

[0110] Note that any component of the embodiment can be modified or omitted.

[0111] Hereinafter, various aspects of the present disclosure will be summarized as appendices.

[0112] (Appendix 1) A model generation device for generating a machine learning model for obtaining first biological data of a subject, An imaging image acquisition unit that acquires an imaging image including a skin region of the test performer, which is an image of the test performer; A luminance data extraction unit that extracts luminance data indicating a luminance change in the skin region of the test performer from the imaging image acquired by the imaging image acquisition unit; A biological reference data acquisition unit that acquires biological reference data, which is second biological data of the test performer measured by a device in which the influence of noise that may occur in an environment where the first biological data is obtained in the measurement result is hardly apparent; A data processing unit that generates learning waveform data imitating the waveform of the first biological data of a type different from the type of the second biological data measured as the biological reference data, which serves as teacher data when training the machine learning model, from the biological reference data acquired by the biological reference data acquisition unit. A model learning unit that generates a machine learning model that takes the luminance data as input and outputs the first biological data, using the luminance data extracted by the luminance data extraction unit and the learning waveform data generated by the data processing unit as learning data. A model generation device comprising the above. (Appendix 2) The first biological data output by the machine learning model is pulse wave data. The model generation device according to Appendix 1, characterized in that. (Appendix 3) The first biological data output by the machine learning model is a blood pressure waveform. The model generation device according to Appendix 1, characterized in that. (Appendix 4) The data processing unit detects peaks in the biological reference data and generates the learning waveform data based on the detected peaks. The model generation device according to any one of Appendices 1 to 3, characterized in that. (Appendix 5) The data processing unit generates the learning waveform data such that the error between the intervals between peaks in the learning waveform data and the intervals between peaks in the biological reference data is within a threshold for interval determination. The model generation device according to Appendix 4, characterized in that. (Appendix 6) The device is an electrocardiograph, and the biological reference data is electrocardiogram data. The model generation device according to any one of Appendices 1 to 5, characterized in that. (Appendix 7) The data processing unit smooths and connects adjacent peaks among the peaks set such that the error from the interval between peaks in the biological reference data is within the interval determination threshold, thereby making the learning waveform data a waveform imitating the waveform of the first biological data. The model generation device according to supplementary note 5, characterized by the above. (Supplementary note 8) The data processing unit connects the peaks set such that the error from the interval between peaks in the biological reference data is within the interval determination threshold, by connecting them with Gaussian distribution curves centered on each peak, thereby making the learning waveform data a waveform imitating the waveform of the first biological data. The model generation device according to supplementary note 5, characterized by the above. (Supplementary note 9) The subject is a driver of a vehicle. The model generation device according to any one of supplementary notes 1 to 8, characterized by the above. (Supplementary note 10) The luminance data extraction unit includes a skin region detection unit that detects the skin region of the test performer from the captured image, a measurement region setting unit that sets a measurement region for extracting the luminance data in a region corresponding to the skin region on the captured image, and a signal extraction unit that extracts the luminance data based on the luminance change in the measurement region on the captured image. The model generation device according to any one of supplementary notes 1 to 9, characterized by the above. (Supplementary note 11) A biological data estimation device that estimates the first biological data of the subject using the machine learning model generated by the model generation device according to any one of supplementary notes 1 to 10, including an estimation image acquisition unit that acquires an estimation captured image including the skin region of the subject, which is an image of the subject, and an estimation luminance data extraction unit that extracts the luminance data indicating the luminance change in the skin region of the subject from the estimation captured image acquired by the estimation image acquisition unit. An estimation unit that estimates the first biological data of the subject by inputting the luminance data extracted by the luminance data extraction unit for estimation into the machine learning model to obtain the first biological data of the subject A biological data estimation device comprising the above. (Appendix 12) A model generation system for generating a machine learning model for obtaining first biological data of a subject, comprising: An imaging image acquisition unit that acquires an imaging image including a skin region of the test performer, which has imaged the test performer; A luminance data extraction unit that extracts luminance data indicating a luminance change in the skin region of the test performer from the imaging image acquired by the imaging image acquisition unit; A biological reference data acquisition unit that acquires biological reference data, which is second biological data of the test performer measured by a device in which the influence of noise that can occur in an environment where the first biological data is obtained in the measurement result is hardly apparent; A data processing unit that generates learning waveform data that mimics the waveform of the first biological data of a type different from the type of the second biological data measured as the biological reference data, which is the teacher data when learning the machine learning model, from the biological reference data acquired by the biological reference data acquisition unit; A model learning unit that generates the machine learning model that takes the luminance data as an input and outputs the first biological data, using the luminance data extracted by the luminance data extraction unit and the learning waveform data generated by the data processing unit as learning data A model generation system comprising the above. (Appendix 13) Part of the processing performed by the data processing unit is executed on a server The model generation system according to Appendix 12, characterized in that the above is the case. (Appendix 14) The data processing unit A first data processing unit that detects peaks in the biological reference data; A second data processing unit that generates the learning waveform data based on the peaks detected by the first data processing unit, and comprising: The luminance data extraction unit, the biological reference data acquisition unit, and the first data processing unit are provided in a local device, and the second data processing unit and the model learning unit are provided in a server connected to the local device via a network. The first data processing unit generates processed peak data with the amount of information reduced to only the information regarding the peak, and outputs the generated processed peak data to the second data processing unit. A model generation system according to appended claim 12 or appended claim 13, characterized by the above. (Appended claim 15) A model generation system according to any one of appended claims 12 to 14, and A biological data estimation device according to appended claim 11 A biological data estimation system comprising the above. (Appended claim 16) A model generation program for generating a machine learning model for obtaining first biological data of a subject, the program causing a computer to: an imaging image acquisition unit that acquires an imaging image including a skin region of the test performer, the imaging image being an image of the test performer; a luminance data extraction unit that extracts luminance data indicating a luminance change in the skin region of the test performer from the imaging image acquired by the imaging image acquisition unit; a biological reference data acquisition unit that acquires biological reference data, which is second biological data of the test performer measured by a device in which the influence of noise that may occur in an environment where the first biological data is obtained in the measurement result is hardly apparent; a data processing unit that generates learning waveform data simulating a waveform of first biological data of a type different from the type of the second biological data measured as the biological reference data, the learning waveform data being used as teacher data when learning the machine learning model, from the biological reference data acquired by the biological reference data acquisition unit; a model learning unit that generates the machine learning model that takes the luminance data as an input and outputs the first biological data, using the luminance data extracted by the luminance data extraction unit and the learning waveform data generated by the data processing unit as learning data. ​A model generation program for causing it to function as such.

Explanation of symbols

[0113] 10 Model generation device, 11 Imaging image acquisition unit, 12 Luminance data extraction unit, 121 Muscle region detection unit, 122 Measurement region setting unit, 123 Signal extraction unit, 13 Biological reference data acquisition unit, 14 Data processing unit, 141 First data processing unit, 142 Second data processing unit, 15 Model learning unit, 16 Model storage unit, 20 Imaging device, 30 Device, 40 Biological data estimation device, 41 Estimation imaging image acquisition unit, 42 Estimation luminance data extraction unit, 421 Estimation muscle region detection unit, 422 Estimation measurement region setting unit, 423 Estimation signal extraction unit, 43 Estimation unit, 100 Model generation system, 500, 500a Biological data estimation system, 10a Local device, 10b Server, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory.

Claims

1. A model generation device for generating a machine learning model for obtaining first biological data of a subject, an imaging image acquisition unit that acquires an imaging image including a skin region of the test performer, the test performer being imaged; a luminance data extraction unit that extracts luminance data indicating a luminance change in the skin region of the test performer from the imaging image acquired by the imaging image acquisition unit; a biological reference data acquisition unit that acquires biological reference data that is second biological data of the test performer measured by a device in which the influence of noise that may occur in an environment where the first biological data is obtained in the measurement result is unlikely to appear; a data processing unit that generates learning waveform data simulating a waveform of the first biological data of a type different from the type of the second biological data measured as the biological reference data, which is teacher data when learning the machine learning model, from the biological reference data acquired by the biological reference data acquisition unit; a model learning unit that generates the machine learning model using the luminance data extracted by the luminance data extraction unit and the learning waveform data generated by the data processing unit as learning data, and inputs the luminance data and outputs the first biological data. A model generation device comprising:

2. The first biological data output by the machine learning model is pulse wave data The model generation device according to claim 1, characterized in that.

3. The first biological data output by the machine learning model is a blood pressure waveform The model generation device according to claim 1, characterized in that.

4. The data processing unit detects peaks in the biological reference data and generates the learning waveform data based on the detected peaks The model generation device according to claim 1, characterized in that.

5. The data processing unit generates the learning waveform data such that an error between an interval between peaks in the learning waveform data and an interval between peaks in the biological reference data is within an interval determination threshold value. The model generation device according to claim 4, characterized in that.

6. The device is an electrocardiograph, and the biological reference data is electrocardiogram data. The model generation device according to claim 1, characterized in that.

7. For the peaks set such that an error from the interval between peaks in the biological reference data is within the interval determination threshold value, the data processing unit smoothes and connects adjacent peaks to each other to make the learning waveform data a waveform imitating the waveform of the first biological data. The model generation device according to claim 5, characterized in that.

8. For the peaks set such that an error from the interval between peaks in the biological reference data is within the interval determination threshold value, the data processing unit connects each other with Gaussian distribution curves centered on each peak to make the learning waveform data a waveform imitating the waveform of the first biological data. The model generation device according to claim 5, characterized in that.

9. The subject is a driver of a vehicle. The model generation device according to claim 1, characterized in that.

10. The luminance data extraction unit includes a skin region detection unit that detects the skin region of the test performer from the captured image, a measurement region setting unit that sets a measurement region for extracting the luminance data in a region corresponding to the skin region on the captured image, and a signal extraction unit that extracts the luminance data based on the luminance change in the measurement region on the captured image. The model generation device according to claim 1, characterized in that.

11. A biological data estimation device that estimates the first biological data of a subject using the machine learning model generated by the model generation device according to any one of claims 1 to 10, an estimation image acquisition unit that acquires an estimation imaging image including the muscle region of the subject, which is an image of the subject; an estimation luminance data extraction unit that extracts the luminance data indicating the luminance change in the muscle region of the subject from the estimation imaging image acquired by the estimation imaging image acquisition unit; an estimation unit that estimates the first biological data of the subject by inputting the luminance data extracted by the estimation luminance data extraction unit into the machine learning model to obtain the first biological data. A biological data estimation device comprising:

12. A model generation system that generates a machine learning model for obtaining the first biological data of a subject, an imaging image acquisition unit that acquires an imaging image including the muscle region of the test performer, which is an image of the test performer; a luminance data extraction unit that extracts luminance data indicating the luminance change in the muscle region of the test performer from the imaging image acquired by the imaging image acquisition unit; a biological reference data acquisition unit that acquires biological reference data, which is the second biological data of the test performer measured by a device in which the influence of noise that may occur in an environment where the first biological data can be obtained in the measurement result is hardly apparent; a data processing unit that generates learning waveform data imitating the waveform of the first biological data of a type different from the type of the second biological data measured as the biological reference data, which is teacher data when learning the machine learning model, from the biological reference data acquired by the biological reference data acquisition unit; a model learning unit that generates the machine learning model that takes the luminance data as an input and outputs the first biological data, using the luminance data extracted by the luminance data extraction unit and the learning waveform data generated by the data processing unit as learning data. A model generation system comprising:

13. A part of the processing performed by the data processing unit is executed on a server. The model generation system according to claim 12, characterized in that.

14. The data processing unit A first data processing unit that detects peaks in the biological reference data, A second data processing unit that generates the learning waveform data based on the peaks detected by the first data processing unit, and The luminance data extraction unit, the biological reference data acquisition unit, and the first data processing unit are provided in a local device, and the second data processing unit and the model learning unit are provided in a server connected to the local device via a network. The first data processing unit generates processed peak data with the amount of information reduced only to information regarding the peaks, and outputs the generated processed peak data to the second data processing unit. The model generation system according to claim 12 or claim 13, characterized in that.

15. The model generation system according to any one of claims 12 to 14, and The biological data estimation device according to claim 11 A biological data estimation system comprising.

16. A model generation program for generating a machine learning model for obtaining first biological data of a subject, causing a computer to An imaging image acquisition unit that acquires an imaging image including a skin region of the test performer, which has imaged the test performer, A luminance data extraction unit that extracts luminance data indicating a luminance change in the skin region of the test performer from the imaging image acquired by the imaging image acquisition unit, A biological reference data acquisition unit that acquires biological reference data, which is the second biological data of the test subject measured by a device in which the influence of noise that can occur in the environment where the first biological data is obtained in the measurement result is not easily apparent; A data processing unit that generates learning waveform data imitating the waveform of the first biological data of a type different from the type of the second biological data measured as the biological reference data, which is teacher data when learning the machine learning model, from the biological reference data acquired by the biological reference data acquisition unit; A model learning unit that generates the machine learning model that takes the luminance data as an input and outputs the first biological data, using the luminance data extracted by the luminance data extraction unit and the learning waveform data generated by the data processing unit as learning data; A model generation program for causing the above to function.

Images