Learning device, learning method, estimation device, estimation method, and computer program

Abstract

To improve accuracy of SpO2 estimation using a moving image.SOLUTION: A learning device includes a learning unit that acquires a learned model by performing learning processing using a combination of a moving image frame sequence obtained by imaging a first part of a body of a learning target person and a time series of a measurement value of SpO2 obtained by measuring a second part of the body of the learning target person at the same timing as imaging of the first part as teacher data, in which the learning unit performs the learning processing using a value of delay in change between a value of SpO2 in the first part and a value of SpO2 in the second part.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a learning device, a learning method, an estimation device, an estimation method, and a computer program. [Background technology]

[0002] Conventionally, there are techniques for estimating oxygen saturation (SpO2) using video images of the human body surface, such as a person's face, hands, or neck. For example, there is a technique for machine learning an SpO2 estimator that outputs SpO2 from frames of an input video image of the human body surface, as described above. For the input video image, an imaging device such as a general RGB camera, a multispectral camera, or a hyperspectral camera can be used.

[0003] For example, in the technology disclosed in Non-Patent Document 1, a time-series map of AC and DC components is first generated from RGB video images of a face. Then, an oxygen saturation estimation model that estimates oxygen saturation (SpO2) is constructed from the time-series map using a convolutional neural network (CNN). At this time, the oxygen saturation estimation model may be trained using oxygen saturation (SpO2) measured with a pulse oximeter, for example, as training data. In Non-Patent Document 1, SpO2 data measured simultaneously with the RGB video images while holding one's breath for approximately 30 to 60 seconds is used as training data.

[0004] Non-Patent Document 2 discloses a method that uses a ratio of ratios. Specifically, it is as follows: Pulse wave signals are measured using two cameras equipped with 660 nm and 520 nm bandpass filters, and oxygen saturation is estimated using Equation 1. Here, the value shown in Equation 2 is the intensity of the AC and DC components of the signal with wavelength λ. A and B are parameters that minimize the squared error between the correct SpO2 data and R. This method also requires training data to determine the parameters A and B.

[0005]

number

[0006]

number

[0007] To generate training data, SpO2 is typically measured using a measuring device such as a pulse oximeter while video images are captured with a camera. However, the sampling rates of the captured video and the SpO2 measurements taken by the pulse oximeter often differ. For example, the SpO2 rate can be significantly lower, such as 0.25 Hz. This makes it difficult to achieve time synchronization, leading to reduced accuracy.

[0008] Furthermore, it is known that delays occur when the part of the body photographed by the camera (e.g., the face) is far from the part of the body SpO2 is measured (e.g., a finger). Furthermore, Non-Patent Documents 3 and 4 show experimental results that there is a time lag of approximately 15 seconds between SpO2 measurements taken on the forehead and on the fingers. Because it takes time for blood with a change in blood oxygen saturation to reach the peripheral blood vessels, the closer to the periphery, the greater the time lag before the change is reflected as a numerical value. This delay is thought to be a problem that occurs regardless of the part of the body as long as the photographed part is some distance from the actual measurement part. Therefore, in conventional methods, learning was performed without time synchronization between the video images used as training data and the SpO2 measurements, resulting in reduced SpO2 estimation accuracy.

[0009] Non-Patent Documents 1 and 2 do not disclose a method for synchronizing video images with SpO2 measurement values. Rough time synchronization is possible by manually setting a fixed offset. However, because the delay time varies depending on factors such as the subject's constitution and physical condition, and the measurement cycle of the pulse oximeter, precise time synchronization is difficult to achieve with a fixed offset.

[0010] Non-patent literature 5 and 6 are literature related to this issue. For example, Non-patent literature 5 proposes Pysio2Video, which converts biosignals into video images and calculates the alignment between the videos. This technology uses a neural network to extract features from two types of biosignals and performs alignment using DTW (Dynamic Time Warping) to maximize the canonical correlation between the features. Emotion classification was performed by fusing time series data from electrocardiograms (ECG), electromyograms (EMG), and galvanic skin responses (GSR).

[0011] Furthermore, Non-Patent Document 6 employs a method of extracting events (upslope, downslope) from biosignals and calculating alignment using DTW (Dynamic Time Warping). This technology allows for accurate alignment even for biosignals with different sampling rates. The effectiveness of this method has been verified using electrocardiograms (ECG) and photoplethysmograms (PPG). [Prior art documents] [Non-patent literature]

[0012] [Non-Patent Document 1] Akamatsu, Yusuke, Yoshifumi Onishi, and Hitoshi Imaoka. 2022. “Blood Oxygen Saturation Estimation from Facial Video via DC and AC Components of Spatio-Temporal Map.” ICASSP 2023. [Non-patent document 2] L. Kong, et al., “Non-contact detection of oxygen saturation based on visible light imaging device using ambient light,” Opt. Express, 2013. [Non-patent document 3] Koji Nakai, "What nurses need to know about pulse oximeter accuracy by measurement site," Nurse Senka, https: / / knowledge.nurse-senka.jp / 500120. [Non-patent document 4] Nakai et al., "A New Approach to Transcutaneous Arterial Blood Oxygen Saturation (SpO2) Measurement - An Attempt at Earlobe Sensors", Journal of Medical and Electronic Engineering, Vol. 79, No. 8 (2009) [Non-patent document 5] Liu, Yuchi, Yue Yao, Zhengjie Wang, J. Plested, and Tom Gedeon, “Generalized Alignment for Multimodal Physiological Signal Learning.” IEEE International Joint Conference on Neural Network (IJCNN), July, 1-10, 2019. [Non-patent document 6] Yihang Jiang, et al. “EventDTW: An Improved Dynamic Time Warping Algorithm for Aligning Biomedical Signals of Nonuniform Sampling Frequencies, “ Sensors, 2020. Summary of the Invention [Problem to be solved by the invention]

[0013] However, the techniques disclosed in Non-Patent Document 5 and Non-Patent Document 6 did not sufficiently improve accuracy. The technology disclosed in Non-Patent Document 5 uses DTW, but typical DTW requires fixed start and end points for matching. Therefore, time differences of several seconds between the measurement start and end times are not taken into account, which could result in reduced accuracy. Furthermore, the technology disclosed in Non-Patent Document 6 targets the same type of biosignals acquired by two or more different devices and assumes that their increases and decreases are consistent (e.g., when one signal increases, the other also increases). Therefore, it could not be applied to oxygen saturation estimation as is. Furthermore, like the technology disclosed in Non-Patent Document 5, the technology disclosed in Non-Patent Document 6 also does not consider time differences between the measurement start and end times, which could result in reduced accuracy.

[0014] The present invention has been made in view of the above circumstances, and provides a technique for improving the accuracy of SpO2 estimation using moving images. [Means for solving the problem]

[0015] One aspect of the present invention is a learning device that acquires a trained model by performing a learning process using as training data a combination of a sequence of video frames obtained by photographing a first part of the body of a person to be trained and a time series of SpO2 measurement values ​​obtained by measuring a second part of the body of the person to be trained at the same time as the first part is photographed, and the learning unit performs the learning process using a delay value of the change in SpO2 value between the first part and the second part.

[0016] One aspect of the present invention is a learning method that includes a learning step of performing a learning process to acquire a trained model by using, as training data, a combination of a sequence of video frames obtained by photographing a first part of the body of a person to be trained and a time series of SpO2 measurement values ​​obtained by measuring a second part of the body of the person to be trained at the same time as the first part is photographed, and in the learning step, the learning process is performed using a delay value of the change in SpO2 value between the first part and the second part.

[0017] One aspect of the present invention is a computer program for causing a computer to execute a learning method, which includes a learning step of acquiring a trained model by performing learning processing using as training data a combination of a sequence of video frames obtained by photographing a first part of the body of a person to be trained and a time series of SpO2 measurement values ​​obtained by measuring a second part of the body of the person to be trained at the same time as the first part is photographed, and in the learning step, performing learning processing using a delay value of the change in SpO2 value between the first part and the second part.

[0018] One aspect of the present invention is an estimation device that includes an estimation unit that estimates an SpO2 value at a second location of a person to be estimated from a sequence of video frames obtained by photographing a first location of the person to be estimated, using a trained model obtained by performing a learning process using as training data a combination of a sequence of video frames obtained by photographing the first location of the person to be estimated and a time series of SpO2 measurement values ​​obtained by measuring a second location of the person to be estimated at the same time as the first location is photographed, wherein the trained model is a trained model obtained by performing a learning process using a delay value of the change between the SpO2 value at the first location and the SpO2 value at the second location.

[0019] One aspect of the present invention is an estimation method comprising: an estimation step of estimating an SpO2 value at a second location of a person to be estimated from a sequence of video frames obtained by photographing a first location of the person to be estimated, using a trained model obtained by performing a learning process using as training data a combination of a sequence of video frames obtained by photographing a first location of the person to be estimated and a time series of SpO2 measurement values ​​obtained by measuring a second location of the person to be estimated at the same time as the first location is photographed; and the trained model is a trained model obtained by performing a learning process using a delay value of the change between the SpO2 value at the first location and the SpO2 value at the second location.

[0020] One aspect of the present invention is a computer program for causing a computer to execute an estimation method, the method comprising: estimating an SpO2 value at a second location of a person to be estimated from a sequence of video frames obtained by photographing a first location of the person to be estimated, using a trained model obtained by performing a learning process using as training data a combination of a sequence of video frames obtained by photographing a first location of the person to be estimated and a time series of SpO2 measurement values ​​obtained by measuring a second location of the person to be estimated at the same time as the first location is photographed; and the trained model is a trained model obtained by performing a learning process using a delay value of the change in the SpO2 value at the first location and the SpO2 value at the second location. [Effects of the Invention]

[0021] According to the present invention, it is possible to improve the accuracy of SpO2 estimation using moving images. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a diagram illustrating an outline of a learning technique according to an embodiment of the present invention. [Figure 2]FIG. 10 is a diagram showing an outline of the amount of time delay (amount of delayed frames) that occurs between a measurement site and an imaging site. [Figure 3] FIG. 10 is a diagram illustrating an outline of an estimation process using an estimator f_θ after learning. [Figure 4] FIG. 1 is a diagram illustrating an outline of the system configuration of a learning system 100 and an estimation system 200. [Figure 5] 2 is a schematic block diagram showing a specific example of the functional configuration of a learning device 30. FIG. [Figure 6] 10 is a flowchart showing a specific example of processing by the learning device 30. [Figure 7] 2 is a schematic block diagram showing a specific example of the functional configuration of an estimation device 40. FIG. [Figure 8] 10 is a flowchart showing a specific example of processing by the estimation device 40. [Figure 9] FIG. 10 is a diagram illustrating an outline of a delay frame amount initial value estimation process. [Figure 10] FIG. 2 is a diagram illustrating a functional configuration of a learning estimation device 50. [Figure 11] FIG. 2 is a diagram illustrating an outline of an example of the hardware configuration of an information processing device 90 applied to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0023] In the following explanation, subscripts (characters written in relatively small letters to the lower right of another character) may be indicated by adding an underscore to the other character. For example, if B is written in relatively small letters to the lower right of A as a subscript, it may be indicated as "A_B." If multiple characters are used as subscripts, they may be enclosed in parentheses. For example, if B and C are written in relatively small letters to the lower right of A as subscripts, it may be indicated as "A_(B,C)."

[0024] Additionally, superscripts (characters written in relatively small letters to the upper right of another character) may be indicated by placing a hat over the other character. For example, if B is written in relatively small letters as a superscript to the upper right of A, it may be indicated as "A^B". When multiple characters are written as superscripts, they may be enclosed in parentheses. For example, if B and C are written in relatively small letters as superscripts to the upper right of A, it may be indicated as "A^(B,C)".

[0025] When A has B as a subscript and C as a superscript, it can be written as "A_B^C".

[0026] [Summary] 1 is a diagram showing an outline of the learning technique of this embodiment. An imaging device 10 captures a moving image of a predetermined part of a person (hereinafter referred to as "target person") whose SpO2 is to be measured. The predetermined part may be, for example, a part of the head where the skin is exposed, such as the forehead or face, or a part of the upper body where the skin is exposed, such as the neck, throat, nape, shoulders, upper arms, upper limbs, or chest, or may be another part of the body.

[0027] Data from video captured by the imaging device 10 is input to an estimator f_θ, which is a learning model that estimates SpO2 values ​​from the video. The input video (hereinafter referred to as the "input video") is represented by a sequence of frames. In this embodiment, each frame of the input video is represented by x_(i,n). i indicates a sequence. For video data, data obtained from the start of imaging to the end of imaging can be treated as a single sequence. Furthermore, when multiple imaging devices 10 are used to simultaneously image multiple parts of the body, video data obtained from one imaging device 10 can be treated as a single sequence. The meaning of the value indicating the sequence is similar for SpO2 values. If there is only one sequence, the value of i does not need to be used. n indicates a frame index. If a total of N frames are input, n takes a value from 1 to N. This means that a video frame sequence consisting of N frames has been input. In the graph indicated by reference numeral 81, the horizontal axis indicates time, and the vertical axis indicates the SpO2 estimate obtained based on the frames at each time. Alternatively, a DC component frame sequence and an AC component frame sequence may be generated by applying a low-pass filter or a band-pass filter to each frame.

[0028] The SpO2 measurement value measured by the measurement device 20 is denoted by y_(i,m). i indicates the series. If there is only one series, the value of i need not be used. m indicates the timing at which the SpO2 measurement value was sampled. If there are a total of M SpO2 measurement values, m takes a value from 1 to M. For example, M takes an integer value less than or equal to N. However, the value of M may be increased by interpolating the SpO2 measurement values ​​along the time axis, or M may be set to N. In the graph denoted by reference numeral 82, the horizontal axis indicates time, and the vertical axis indicates the SpO2 measurement values ​​sampled at each time.

[0029] The part of the body that is the object of measurement by measuring device 20 (hereinafter referred to as the "measurement part") is different from the part of the body that is the object of imaging by imaging device 10 (hereinafter referred to as the "imaging part"). Therefore, a delay occurs between the SpO2 value at the measurement part and the SpO2 value at the imaging part. That is, there is inherently a time lag (delay) between the measured value of SpO2 measured by measuring device 20 at the measurement part and the measured value of SpO2 measured by measuring device 20 at the imaging part. Therefore, even if the estimator f_θ can accurately estimate the value of SpO2, there will be a discrepancy between the measured value of SpO2 measured at the measurement part by measuring device 20 at the same timing and the estimated value of SpO2.

[0030] FIG. 2 is a diagram showing an outline of the amount of time delay (amount of delayed frames) that occurs between the measurement site and the imaging site. In FIG. 2, the amount of delayed frames is indicated as "Δt." The index of the video frame obtained at the timing closest to the timing at which the measurement value y_m is obtained by the measurement device 20 is indicated as t(m). Similarly, the index of the video frame obtained at the timing closest to the timing at which the measurement value y_(m-1) is obtained by the measurement device 20 is indicated as t(m-1), and the index of the video frame obtained at the timing closest to the timing at which the measurement value y_(m+1) is obtained by the measurement device 20 is indicated as t(m+1). Since the amount of delayed frames is expressed as Δt as described above, the index of the video frame obtained at the timing closest to the timing at which the measurement value y_m is obtained at the imaging site is expressed as t(m)+Δt. In this case, the prediction error is expressed by the following equation 3, and the sum of the prediction errors is expressed by the following equation 4.

[0031]

number

[0032]

number

[0033] The prediction error may be defined as the least square error of the above-mentioned Equation 3, as shown in the following Equation 5.

[0034]

number

[0035] Furthermore, if there are multiple sequences, the amount of delayed frames "Δt_i" is defined for each sequence. This is because it is assumed that the amount of time delay (amount of delayed frames) differs for each sequence. In this case, the sum of prediction errors is expressed by the following equation 6. Note that the amount of delayed frames for each sequence may all be expressed using the same value. In this case, this will affect the processing of the oxygen saturation estimation step at the subsequent stage.

[0036]

number

[0037] The learning process of the estimator f_θ repeatedly executes a process of finding the optimal delay frame amount Δt^* that minimizes the sum of prediction errors shown in Equation 4, and a process of updating the parameter θ of the estimator f_θ so as to minimize the sum of prediction errors shown in Equation 4 when the delay frame amount is the optimal delay frame amount Δt^*.

[0038] Specifically, for example, the two processes expressed by the following equations may be repeatedly executed until a termination condition is met (alternate optimization). The termination condition may be one generally used in machine learning. For example, the termination condition may be that the number of repetitions exceeds a predetermined number, that the prediction error or its change falls below a predetermined threshold, or that the decrease in the estimated generalization error (the error calculated using equation 3 for the validation data) falls below a predetermined threshold. Equation 7 is the process of estimating the amount of delay frames (amount of time delay), and equation 8 is the process of updating parameters.

[0039]

number

[0040]

number

[0041] When the above two processes are repeated, the value shown in the following equation 9 is output.

[0042]

number

[0043] In the processing of Equation 7, the values ​​of y_(i,m) and f_θ(x_(i,t(m)+Δt_i) may be L2 normalized as shown in Equation 10 below.

[0044]

number

[0045] 3 is a diagram showing an outline of estimation processing using the learned estimator f_θ. In the estimation processing, a frame sequence x_n of a video (input video) captured by the imaging device 10 and a frame delay amount Δt are input.

[0046] By inputting a sequence of frames x_n of a video (input video) captured by the imaging device 10 into the estimator f_θ, a time series of estimated SpO2 values ​​f_θ(x_n) is obtained as the estimation result. In this case, it is desirable that the imaging site used during learning and the imaging site used during estimation are the same site. Furthermore, if the target person during learning (hereinafter referred to as the "learning target person") and the target person during estimation (hereinafter referred to as the "estimation target person") are the same person, estimation can be performed with higher accuracy than if they are not.

[0047] If the number of delayed frames is assumed to be the same for all sequences during learning, a single value, Δt^*, is obtained. However, if it is assumed that the number of delayed frames differs for each sequence during learning, the average value of {Δt_i^*}_i, for example, may be used as the number of delayed frames.

[0048] By performing the estimation process, the SpO2 value at the measurement site when the training data for the learning process is obtained is obtained as an estimation result using the following equation 11.

[0049]

number

[0050] If an estimated value of SpO2 at the imaging position of the imaging device 10 is required in the estimation process, the estimated result of SpO2 may be obtained by the following Equation 12.

[0051]

number

[0052] [System Configuration] FIG. 4 is a diagram showing an outline of the system configuration of the learning system 100 and the estimation system 200. The learning system 100 includes an imaging device 10, a measuring device 20, and a learning device 30. The imaging device 10 generates a moving image by capturing an image of a target body part of the person to be studied. Any device capable of generating an image may be used as the imaging device 10. For example, a video camera capturing visible light may be used, a multispectral camera may be used, or a hyperspectral camera may be used. The measuring device 20 measures the SpO2 at a target body part of the person to be studied. The measuring device 20 may be configured using, for example, a pulse oximeter.

[0053] 5 is a schematic block diagram showing a specific example of the functional configuration of learning device 30. Learning device 30 is configured using an information processing device such as a smartphone, tablet, personal computer, dedicated equipment, or server device. Learning device 30 includes an image input unit 31, a measurement data input unit 32, a memory unit 33, and a control unit 34.

[0054] The image input unit 31 is an interface that acquires moving image data of an imaged portion of a person to be studied from another device. In this embodiment, the image input unit 31 acquires moving image data from the imaging device 10. Data may be exchanged between the image input unit 31 and the imaging device 10 using wired communication or wireless communication. If the learning device 30 and the imaging device 10 are configured integrally, the image input unit 31 may input moving image data from a bus.

[0055] However, the image input unit 31 does not need to be limited to this configuration. For example, the image input unit 31 may read moving image data recorded on a recording medium such as a DVD-ROM or a USB memory (Universal Serial Bus Memory). The image input unit 31 may receive moving image data from another information processing device via a network. The image input unit 31 may be configured in any manner as long as it is capable of receiving input of moving image data in which an image of a part of a person to be studied is captured.

[0056] The measurement data input unit 32 is an interface that acquires measurement data of SpO2 at the measurement site of the learning subject from another device. In this embodiment, the measurement data input unit 32 acquires measurement data from the measurement device 20. Data may be exchanged between the measurement data input unit 32 and the measurement device 20 using wired communication or wireless communication. When the learning device 30 and the measurement device 20 are configured integrally, the measurement data input unit 32 may input the measurement data via a bus.

[0057] However, the measurement data input unit 32 does not need to be limited to this configuration. For example, the measurement data input unit 32 may read out measurement data recorded on a recording medium such as a DVD-ROM or USB memory. The measurement data input unit 32 may receive measurement data from another information processing device via a network. The measurement data input unit 32 may be configured in any manner as long as it is capable of receiving input of measurement data obtained by measuring the measurement parts of the person to be studied.

[0058] The storage unit 33 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 33 stores data used by the control unit 34. The storage unit 33 may function as, for example, a teacher data storage unit 331 and a trained model storage unit 332.

[0059] The teacher data storage unit 331 stores teacher data used in the learning process executed by the learning device 30. The teacher data stored in the teacher data storage unit 331 includes video data and measurement data obtained from the same person to be trained at the same time. The video data corresponds to the explanatory variable, and the measurement data corresponds to the target variable. The video data is data input via the image input unit 31 and is video data obtained by capturing images of the target person's imaging region. The measurement data is data input via the measurement data input unit 32. The measurement data is SpO2 measurement data obtained by measuring the target person's measurement region at the same time as the video data is captured. The teacher data stored in the teacher data storage unit 331 may consist of only data obtained from the same person to be trained, or may include data obtained from multiple people to be trained.

[0060] The trained model storage unit 332 stores a trained model obtained by a learning process using the training data stored in the training data storage unit 331. The trained model stored in the trained model storage unit 332 corresponds to, for example, the estimator f_θ shown in FIG.

[0061] The control unit 34 is configured using a processor such as a CPU (Central Processing Unit) and a memory (main storage device). The control unit 34 functions as an information control unit 341 and a learning control unit 342 when the processor executes a program. All or part of the functions of the control unit 34 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a semiconductor storage device (e.g., a solid-state drive (SSD)), as well as storage devices such as a hard disk or semiconductor storage device built into a computer system. The above program may be transmitted via a telecommunications line.

[0062] The control unit 34 may execute, for example, an application installed on its own device (learning device 30). A specific example of such an application is an application provided to the learning device 30 as a dedicated application for the learning system 100. Another specific example of such an application is a web browser application. Such an application may be pre-installed on the learning device 30 or may be downloaded each time a learning process is executed. For example, when implemented as a web browser application, the information processing device may download and execute the application from a device specified by a specific web server (e.g., the web server itself or another server) in response to a connection of the information processing device underlying the learning device 30 to the web server. The control unit 34 operates according to the program of the application being executed.

[0063] The information control unit 341 accepts input of image data via the image input unit 31 and records the input image data in the teacher data storage unit 321. The information control unit 341 accepts input of measurement data via the measurement data input unit 32 and records the input measurement data in the teacher data storage unit 321. The information control unit 341 outputs the trained model recorded in the trained model storage unit 322 to another information processing device or a recording medium.

[0064] The learning control unit 342 executes a learning process to learn the estimator f_θ using, as training data, the data recorded in the training data storage unit 321. The learning process executed by the learning control unit 342 is the learning process described with reference to FIG.

[0065] 6 is a flowchart showing a specific example of processing by the learning device 30. First, the information control unit 341 acquires training data (step S101). The learning control unit 342 repeatedly executes a process of estimating the amount of delayed frames (step S102) and a process of updating parameters (step S103) using the training data until a termination condition is met (step S104-NO). When the termination condition is met (step S104-YES), a trained model including the parameters obtained at that time is recorded in the trained model storage unit 322 (step S105).

[0066] 7 is a schematic block diagram showing a specific example of the functional configuration of the estimation device 40. The estimation device 40 is configured using information devices such as a smartphone, a tablet, a personal computer, or a dedicated device. The estimation device 40 includes an image input unit 41, an output unit 42, a storage unit 43, and a control unit 44.

[0067] The image input unit 41 is an interface that acquires video image data of the imaging region of the person to be estimated from another device. The video input to the image input unit 41 may be video of the same person as the person to be learned in the training data used in the learning process in the learning device 30, or may be video of a different person. If the video is of the same person, SpO2 can be estimated with higher accuracy.

[0068] The image input unit 41 in this embodiment acquires moving image data from the imaging device 10. Data may be exchanged between the image input unit 41 and the imaging device 10 using wired communication or wireless communication. When the estimation device 40 and the imaging device 10 are configured integrally, the image input unit 41 may input moving image data from a bus.

[0069] However, the image input unit 41 does not need to be limited to this configuration. For example, the image input unit 41 may read moving image data recorded on a recording medium such as a DVD-ROM or a USB memory (Universal Serial Bus Memory). The image input unit 41 may receive moving image data from another information processing device via a network. The image input unit 41 may be configured in any manner as long as it is capable of receiving input of moving image data in which an imaged portion of the person to be estimated is captured.

[0070] The output unit 42 outputs information in a form that can be recognized by the user. The output unit 42 may be, for example, an image display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The output unit 42 may be an interface for connecting an image display device to the estimation device 40. In this case, the output unit 42 generates a video signal for displaying image data and outputs the video signal to an image display device connected to the output unit 42. The output unit 42 may be a device for outputting sound, such as a speaker. The output unit 42 may be an interface for connecting an audio output device, such as a speaker or headphones, to the estimation device 40. In this case, the output unit 42 generates an audio signal for reproducing audio data and outputs the audio signal to an audio output device connected to the output unit 42. The output unit 42 may be configured as a touch panel.

[0071] The storage unit 43 is configured using a storage device such as a magnetic hard disk drive, a semiconductor storage device, etc. The storage unit 43 stores data used by the control unit 44.

[0072] The control unit 44 is configured using a processor such as a CPU and a memory. The control unit 44 functions as an information control unit 441 and an estimation unit 442 by the processor executing a program. All or part of the functions of the control unit 44 may be realized using hardware such as an ASIC, PLD, or FPGA. The above program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a semiconductor storage device (e.g., an SSD), as well as storage devices such as a hard disk or semiconductor storage device built into a computer system. The above program may be transmitted via a telecommunications line.

[0073] The information control unit 441 acquires information from another device such as the learning device 30. The information control unit 441 may acquire data of the trained model from the learning device 30, for example, and record it in the trained model storage unit 431. The information control unit 441 may transmit information indicating the estimation result obtained by the estimation unit 442 to another device, or may output the information via the output unit 42.

[0074] The estimation unit 442 performs estimation processing using the learned model stored in the learned model storage unit 431 and the video image data input from the image input unit 41. Through the estimation processing, the SpO2 of the person to be estimated is estimated.

[0075] 8 is a flowchart showing a specific example of processing by the estimation device 40. First, the information control unit 441 acquires video data from the imaging device 10 (step S201). The estimation unit 442 performs estimation processing using the input video data and the trained model (step S202). The estimation unit 442 outputs information indicating the estimation result from the output unit 42 (step S203).

[0076] The estimation system 200 configured in this manner makes it possible to estimate the SpO2 of the person to be estimated with higher accuracy using video of the person to be estimated. Specifically, this is as follows: In the estimation system 200, the learning device 30 performs a learning process that takes into account the amount of delayed frames. This reduces the discrepancy in values ​​due to the time difference (delay) between the video frames used as training data and the measured value of SpO2, making it possible to estimate SpO2 with higher accuracy.

[0077] (Variation) 9 is a diagram showing an outline of the process of estimating the initial value of the amount of delayed frames. In this process, the initial value of the amount of delayed frames is found using prior knowledge between frames x_n (n=1, 2, . . . , N) of the input video sequence and the measured value of SpO2 y_n. The found initial value of the amount of delayed frames may be used in the learning process or estimation process in the learning device 30 described above. Note that the estimation of the initial value of the amount of delayed frames is not limited to that described below, and prior knowledge of the relationship between x_n and y_n may be used.

[0078] The initial value estimation process for the amount of delayed frames uses the frame sequence x_(i,n) of the input video sequence and the measured SpO2 values ​​y_(i,m) as inputs. The time series of measured SpO2 values ​​y_(i,m) can be obtained by performing multiple measurements. Note that the SpO2 values ​​may be interpolated to M=N. First, a specific band (e.g., a band including 660 nm) is extracted from the frame sequence x_(i,n) of the input video sequence. Next, the average brightness value ξ_(i,n) of the observed region of interest is calculated. Near the specific band (e.g., near 660 nm), the difference in absorption rates between hemoglobin and oxygenated hemoglobin is large, making it less susceptible to noise. If the lighting conditions do not fluctuate over time, the oxygen saturation level y_(i,m) and the brightness value ξ_(i,n) have an inverse correlation. Taking advantage of this, the initial value Δt_i^※ of the amount of delayed frames is calculated using the following equation:

[0079]

number

[0080] Here, Corr is the cross-correlation function. The initial value of the amount of delay frames Δt_i^* obtained using this equation (13) is obtained as the output.

[0081] In this embodiment, the learning device 30 and the estimation device 40 are configured as separate devices, but they may also be configured as an integrated device. FIG. 10 is a diagram showing the functional configuration of a learning and estimation device 50 configured in this manner. The image input unit 51 and the measurement data input unit 52 are configured similarly to the image input unit 31 and the measurement data input unit 32, respectively. The teacher data storage unit 531 is configured similarly to the teacher data storage unit 331. The trained model storage unit 532 is configured similarly to the trained model storage unit 431. The information control unit 541 is configured similarly to the information control unit 341 and the information control unit 441. The learning control unit 542 is configured similarly to the learning control unit 342. The estimation unit 543 is configured similarly to the estimation unit 442. The output unit 55 is configured similarly to the output unit 42.

[0082] The learning device 30 may be implemented using a plurality of information processing devices. For example, the learning device 30 may be implemented using a device such as a cloud. For example, in the learning device 30, the memory unit 33 and the control unit 34 may be implemented in different information processing devices. For example, the memory unit 33 of the learning device 30 may be distributed and implemented across a plurality of information processing devices. The estimation device 40 may be implemented using a plurality of information processing devices. For example, the estimation device 40 may be implemented using a device such as a cloud. For example, in the estimation device 40, the memory unit 43 and the control unit 44 may be implemented in different information processing devices. For example, the memory unit 43 of the estimation device 40 may be distributed and implemented across a plurality of information processing devices.

[0083] FIG. 11 is a diagram illustrating an outline of an example hardware configuration of an information processing device 90 applied to this embodiment. The information processing device 90 includes a processor 91, a main memory device 92, a communication interface 93, an auxiliary memory device 94, an input / output interface 95, and an internal bus 96. The processor 91, the main memory device 92, the communication interface 93, the auxiliary memory device 94, and the input / output interface 95 are communicably connected to each other via the internal bus 96. The information processing device 90 may be applied to, for example, the learning device 30, the estimation device 40, and the learning and estimation device 50. In this case, for example, the image input unit 31, the image input unit 41, the image input unit 51, the measurement data input unit 32, and the measurement data input unit 52 may be configured using the input / output interface 95. For example, the memory unit 33, the memory unit 43, and the memory unit 53 may be configured using the auxiliary memory device 94. Furthermore, the control unit 34, the control unit 44, and the control unit 54 may be configured using the processor 91 and the main memory device 92.

[0084] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0085] 100... learning system, 200... estimation system, 10... imaging device, 20... measuring device, 30... learning device, 31... image input unit, 32... measurement data input unit, 33... memory unit, 331... teacher data memory unit, 332... trained model memory unit, 34... control unit, 341... information control unit, 342... learning control unit, 40... estimation device, 41... image input unit, 42... output unit, 43... memory unit, 431... trained model memory unit, 44... control unit, 441... information control unit, 442... estimation unit, 50... learning and estimation device, 51... image input unit, 52... measurement data input unit, 53... memory unit, 531... teacher data memory unit, 532... trained model memory unit, 54... control unit, 541... information control unit, 542... learning control unit 543...estimation unit, 55...output unit

Claims

1. a learning unit that acquires a trained model by performing a learning process using, as training data, a combination of a sequence of video frames obtained by photographing a first part of a body of a training subject and a time series of SpO2 measurement values ​​obtained by measuring a second part of the body of the training subject at the same timing as the photographing of the first part; The learning unit performs a learning process using a delay value of the change between the SpO2 value at the first location and the SpO2 value at the second location.

2. a learning step of acquiring a trained model by performing a learning process using, as training data, a combination of a sequence of video frames obtained by photographing a first part of the body of a training subject and a time series of SpO2 measurement values ​​obtained by measuring a second part of the body of the training subject at the same timing as the photographing of the first part; A learning method in which, in the learning step, learning processing is performed using a delay value of the change between the SpO2 value at the first location and the SpO2 value at the second location.

3. 3. The learning method according to claim 2, wherein the learning step repeatedly executes a first process of determining an optimal delay frame amount that minimizes the sum of prediction errors when the delay frame amount indicating the delay value is Δt, and a second process of updating parameters of an estimator of the trained model so as to minimize the prediction error when the delay frame amount is the optimal delay frame amount.

4. a learning step of acquiring a trained model by performing a learning process using, as training data, a combination of a sequence of video frames obtained by photographing a first part of the body of a training subject and a time series of SpO2 measurement values ​​obtained by measuring a second part of the body of the training subject at the same timing as the photographing of the first part; A computer program for causing a computer to execute a learning method in which, in the learning step, learning processing is performed using a delay value of the change between the SpO2 value at the first location and the SpO2 value at the second location.

5. an estimation unit that estimates an SpO2 value at a second region of a person to be estimated from a video frame sequence obtained by photographing the first region of the body of the person to be estimated, using a trained model obtained by performing a learning process using, as training data, a combination of a video frame sequence obtained by photographing the first region of the body of the person to be estimated and a time series of SpO2 measurement values ​​obtained by measuring the second region of the body of the person to be estimated at the same timing as the photographing of the first region; The trained model is a trained model obtained by performing a training process using a delay value of the change between the SpO2 value at the first location and the SpO2 value at the second location.

6. an estimation step of estimating an SpO2 value at a second region of the person to be estimated from a video frame sequence obtained by photographing the first region of the body of the person to be estimated, using a trained model obtained by performing a learning process using as training data a combination of a video frame sequence obtained by photographing the first region of the body of the person to be estimated and a time series of SpO2 measurement values ​​obtained by measuring the second region of the body of the person to be estimated at the same timing as the photographing of the first region; An estimation method, wherein the trained model is a trained model obtained by performing a training process using a delay value of the change between the SpO2 value at the first location and the SpO2 value at the second location.

7. an estimation step of estimating an SpO2 value at a second region of the person to be estimated from a video frame sequence obtained by photographing the first region of the body of the person to be estimated, using a trained model obtained by performing a learning process using as training data a combination of a video frame sequence obtained by photographing the first region of the body of the person to be estimated and a time series of SpO2 measurement values ​​obtained by measuring the second region of the body of the person to be estimated at the same timing as the photographing of the first region; A computer program for causing a computer to execute an estimation method, wherein the trained model is a trained model obtained by performing a training process using a delay value of the change between the SpO2 value at the first location and the SpO2 value at the second location.

Images