Learning device, learning method, and program

Abstract

A training device 1X primarily comprises a feature information acquisition means 15X and a training means 17X. The feature information acquisition means 15X acquires first feature information, which is spatiotemporal feature information generated from a first video resulting from a first camera imaging a subject, and acquires second feature information, which is spatiotemporal feature information generated from a second video resulting from a second camera imaging the subject. On the basis of the first feature information and the second feature information, the training means 17X trains an inference model for inferring the relationship between feature information and subject condition information. The inference model that has been obtained by machine learning can also be used to support user decision-making or similar.

Description

Learning device, learning method, and storage medium

[0001] The present disclosure relates to the technical fields of a learning device, a learning method, and a storage medium.

[0002] There are technologies for estimating vital signs using biological images captured by a camera. For example, Patent Literature 1 discloses a technology for estimating pulse information from a captured image of a user's face using a trained machine learning model.

[0003] Japanese Patent Application Laid-Open No. 2021-069813

[0004] Generally, training a machine learning model to infer vital signs requires training labels obtained from sensors such as a pulse wave sensor or a respiratory band worn by the subject. Collecting data from these sensors places a burden on both the subject and the data collector.

[0005] In view of the above-mentioned problems, one of the objectives of the present disclosure is to provide a learning device, a learning method, and a storage medium that can suitably execute learning of a machine learning model that makes inferences regarding the state of a subject.

[0006] One aspect of the learning device is a learning device comprising: a feature information acquisition means for acquiring first feature information, which is feature information in time and space generated from a first image of a subject captured by a first camera, and acquiring second feature information, which is feature information generated from a second image of the subject captured by a second camera; and a learning means for learning an inference model that infers the relationship between the feature information and status information of the subject based on the first feature information and the second feature information, wherein the learning means performs the learning by treating the first feature information and the second feature information, which are based on images of the same subject captured during the same period, as a positive pair.

[0007] One aspect of the learning method is a learning method in which a computer acquires first feature information, which is feature information in time and space generated from a first image of a subject captured by a first camera, acquires second feature information, which is feature information generated from a second image of the subject captured by a second camera, learns an inference model that infers a relationship between the feature information and status information of the subject based on the first feature information and the second feature information, and performs the learning by treating the first feature information and the second feature information based on images of the same subject captured during the same period as a positive pair.

[0008] One aspect of the storage medium is a storage medium that stores a program that causes a computer to execute the following process: acquire first feature information, which is feature information in time and space generated from a first video of a subject captured by a first camera; acquire second feature information, which is feature information generated from a second video of the subject captured by a second camera; learn an inference model that infers a relationship between the feature information and status information of the subject based on the first feature information and the second feature information; and perform the learning process by treating the first feature information and the second feature information, which are based on videos of the same subject captured during the same period, as a positive pair.

[0009] One example of the effect of the present disclosure is that it is possible to preferably perform learning of a machine learning model that makes inferences regarding the state of a subject.

[0010] 1 is a schematic configuration of a vital information estimation system in an embodiment. (A) An example of the hardware configuration of a learning device is shown. (B) An example of the hardware configuration of an estimation device is shown. An overview of the process of contrastive learning of a first inference model for a first camera and a second inference model for a second camera using clip footage generated from the first camera and the second camera, respectively is shown. An example of functional blocks of a learning device is shown. An overview diagram of data expansion. An example of functional blocks of an estimation device related to the process of estimating vital information using a learned inference model. An example of a flowchart showing the processing procedure by a learning device. An overview of learning of an inference model in a modified example is shown. An overview of the configuration of a learning device related to a second embodiment is shown. An example of a flowchart showing the processing procedure executed by a learning device in the second embodiment.

[0011] Hereinafter, embodiments of a learning device, a learning method, and a storage medium will be described with reference to the drawings.

[0012] <First Embodiment> (1) Overall Configuration Fig. 1 shows a schematic configuration of a vital information estimation system 100 according to the first embodiment. The vital information estimation system 100 performs processing related to estimation of vital information of a subject captured by a camera. The vital information estimation system 100 mainly includes a learning device 1, a storage device 2, an estimation device 3, and a camera 5. The learning device 1 and the storage device 2, the storage device 2 and the estimation device 3, and the estimation device 3 and the camera 5 each perform data communication via wired or wireless means. Note that these communications may be performed via a network.

[0013] The learning device 1 performs processing related to the learning of a machine learning model used to estimate vital information. The estimated vital information includes information related to any vital sign, such as heart rate, respiration, SpO2, and blood pressure. The learning device 1 performs learning of a machine learning model that serves as an inference engine that makes inferences related to vital information, based on learning data stored in a learning data storage unit 21 of the storage device 2, and stores parameters of the machine learning model obtained by learning in a model information storage unit 22.

[0014] The storage device 2 stores information necessary for the processes executed by the learning device 1 and the storage device 2. Functionally, the storage device 2 includes a learning data storage unit 21 and a model information storage unit 22.

[0015] The learning data storage unit 21 stores learning data, which is data for learning (training) used by the learning device 1 to learn a machine learning model. Here, the learning data includes video data of a human subject. The video data is used for contrastive learning of an inference model, which will be described later. Hereinafter, video data (e.g., video data of a predetermined length) used to generate one batch of input data to a machine learning model will also be referred to as "clip video." Here, the learning data includes clip video generated by multiple cameras, and the clip video generated by multiple cameras includes clip video generated from the multiple cameras by simultaneously capturing images of the same subject.

[0016] The clipped video is not limited to a video of the subject's face, but may be a video of the skin of a hand or any other part of the subject's skin. The clipped video may also be a video of multiple parts of the subject. The subject is not limited to a human, but may also be a non-human animal.

[0017] The model information storage unit 22 stores model information related to an inference model, which is a machine learning model for which learning is performed by the learning device 1. The model information includes parameters of the inference model that are updated by learning performed by the learning device 1. In the present embodiment, as an example, the inference model is assumed to be a camera-dependent model that is learned for each camera (i.e., a model that is learned using input data based on clip video generated from a specific camera). Therefore, there are as many inference models as there are cameras used to shoot the clip video stored in the learning data storage unit 21. The model information for each inference model is stored in the model information storage unit 22, for example, in association with the identification information of the corresponding camera.

[0018] The inference model is a model that, when input data based on a video clip is input to the inference model, outputs an inference result regarding vital information of the subject at the time the video clip was captured. In other words, it is a model that has learned the relationship between the video clip and the vital information of the subject in the video clip. The inference result output by the inference model may be a vital waveform such as a pulse wave or a respiratory waveform, or a vital indicator such as a heart rate or respiratory rate. Here, the architecture of the inference model may be a neural network (including a transformer or any other convolutional neural network), another type of architecture such as a support vector machine, or an architecture that combines these. For example, when an architecture based on a neural network such as a convolutional neural network is used for the inference model, the model information storage unit 22 stores information on various parameters (including hyperparameters) of the inference model, such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weight of each element of each filter.

[0019] The storage device 2 may be an external storage device such as a hard disk connected to or built into the learning device 1 or the estimation device 3, or may be a storage medium such as a flash memory, or may be a server device that performs data communication with the learning device 1 and the estimation device 3. The storage device 2 may also be composed of multiple storage devices, and may have the above-mentioned storage units distributed among them.

[0020] The estimation device 3 estimates vital information of a subject based on video data generated by a camera 5 that captures the subject. In this case, the estimation device 3 extracts model information of an inference model corresponding to the camera 5 from the model information storage unit 22, and constructs a trained inference model corresponding to the camera 5. The estimation device 3 then extracts a clip video from the video data generated by the camera 5, and inputs input data based on the extracted clip video into the trained inference model. The estimation device 3 then acquires vital information based on the inference result output by the inference model.

[0021] The camera 5 is a camera that captures an image of the subject and supplies the generated image data to the estimation device 3. The camera 5 preferably corresponds to any of the cameras used to generate the clip images stored in the training data storage unit 21.

[0022] The configuration of the vital information estimation system 100 shown in FIG. 1 is an example, and various modifications may be made. For example, at least two of the learning device 1, the storage device 2, and the estimation device 3 may be realized by the same device. In another example, the learning device 1 and the estimation device 3 may each be composed of multiple devices. In this case, the multiple devices constituting the learning device 1 and the multiple devices constituting the estimation device 3 exchange information necessary to execute pre-assigned processing between the devices by direct wired or wireless communication or by communication via a network. Furthermore, the vital information estimation system 100 may generate clipped video serving as learning data based on video data output by the camera 5 or the like, and store the generated clipped video in the learning data storage unit 21.

[0023] (2) Hardware Configuration Next, the hardware configurations of the learning device 1 and the estimation device 3 will be described.

[0024] 2A shows an example of the hardware configuration of the learning device 1. The learning device 1 includes, as hardware, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 19.

[0025] The processor 11 executes a program stored in the memory 12 to function as a controller (arithmetic unit) that performs overall control of the learning device 1. The processor 11 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0026] Memory 12 is composed of various types of volatile and non-volatile memory, such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. Memory 12 also stores programs for executing processes performed by learning device 1. Some of the information stored in memory 12 may be stored in an external storage device, such as storage device 2, that can communicate with learning device 1, or may be stored in a storage medium that is detachable from learning device 1. Memory 12 may also store information stored in storage device 2 instead.

[0027] The interface 13 is an interface for electrically connecting the learning device 1 to other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0028] 2(B) shows an example of the hardware configuration of the estimation device 3. The estimation device 3 includes, as hardware, a processor 31, a memory 32, and an interface 33. The processor 31, the memory 32, and the interface 33 are connected via a data bus 30.

[0029] The processor 31 executes a program stored in the memory 32 to function as a controller (arithmetic unit) that performs overall control of the estimation device 3. The processor 31 is, for example, a processor such as a CPU, a GPU, or a TPU. The processor 31 may be composed of multiple processors.

[0030] The memory 32 is composed of various types of volatile and non-volatile memories such as RAM, ROM, and flash memory. The memory 32 also stores programs for executing processes performed by the estimation device 3. Some of the information stored in the memory 32 may be stored in an external storage device such as the storage device 2 that can communicate with the estimation device 3, or may be stored in a storage medium that is detachable from the estimation device 3. The memory 32 may also store information stored in the storage device 2 instead.

[0031] The interface 33 is an interface for electrically connecting the estimation device 3 to other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0032] The hardware configurations of the learning device 1 and the estimation device 3 are not limited to those shown in Figures 2(A) and 2(B). For example, at least one of the learning device 1 and the estimation device 3 may further include a display unit such as a display, an input unit such as a keyboard or a mouse, and a sound output unit such as a speaker.

[0033] (3) Overview of Learning Next, an overview of learning of an inference model by the learning device 1 will be described. The learning device 1 performs contrastive learning of an inference model by assuming that vital information based on clips of the same subject captured simultaneously by two cameras is similar, and that vital information based on clips of different subjects or captured at different times is dissimilar. Specifically, the learning device 1 performs contrastive learning for the inference model for each camera that generates clips, so as to bring inference results based on clips of the same subject captured simultaneously closer together and to move inference results based on clips of different subjects or captured at different times further apart. This allows learning of an inference model for a specific camera without requiring teacher labels.

[0034] A pair of clips of the same subject photographed simultaneously, or a pair of feature information generated based on the clips, or a pair of inference results based on the feature information, are pairs treated as information based on the condition of substantially the same subject (substantially the same vital signs) in contrastive learning (i.e., a pair of positive examples), and hereinafter these pairs will also be referred to as "positive pairs." On the other hand, a pair of clips of different subjects or different photographing periods, or a set of feature information generated based on the clips, or a set of inference results based on the feature information, are pairs treated as information based on the condition of substantially different subjects (substantially different vital signs) in contrastive learning (i.e., a pair of negative examples), and hereinafter these pairs will also be referred to as "negative pairs."

[0035] 3 shows an overview of a process for performing contrastive learning between a first inference model, which is an inference model for the first camera, and a second inference model, which is an inference model for the second camera, using clipped video generated from the first camera and the second camera, respectively. Here, the first inference model and the second inference model are each assumed to be machine learning models that output vital signs waveforms such as pulse waves or respiratory waveforms when a spatiotemporal feature map (also referred to as a "spatiotemporal map") converted from the clipped video is input.

[0036] In Fig. 3, subject X is simultaneously imaged by the first camera and the second camera, whereby clip video CV1 is generated by the first camera and clip video CV2 is generated by the second camera. Furthermore, subject Y is simultaneously imaged by the first camera and the second camera, whereby clip video CV3 is generated by the first camera and clip video CV4 is generated by the second camera. The pair of clip video CV1 and clip video CV2, and the pair of clip video CV3 and clip video CV4, are each a regular pair. Although Fig. 3 shows only two images of subject X and subject Y, in reality, more than two images are used.

[0037] The learning device 1 then converts the video clips CV1 to CV4 into spatiotemporal maps FM1 to FM4, respectively. The spatiotemporal maps FM1 to FM4 are data generated by extracting features from the video clips CV1 to CV4, and are in tensor format that is consistent with the input formats of the first and second inference models. Preferably, the learning device 1 may increase the number of video clip records used in contrastive learning by performing data augmentation on the video clips CV1 to CV4.

[0038] Next, the learning device 1 inputs the spatiotemporal map FM1 corresponding to the clip video CV1 generated by the first camera into the first inference model for the first camera, thereby obtaining a waveform w1 output by the first inference model. Similarly, the learning device 1 inputs the spatiotemporal map FM3 corresponding to the clip video CV3 generated by the first camera into the first inference model for the first camera, thereby obtaining a waveform w3 output by the first inference model. Furthermore, the learning device 1 inputs the spatiotemporal map FM2 corresponding to the clip video CV2 generated by the second camera into the second inference model for the second camera, thereby obtaining a waveform w2 output by the second inference model. Similarly, the learning device 1 inputs the spatiotemporal map FM4 corresponding to the clip video CV4 generated by the second camera into the second inference model for the second camera, thereby obtaining a waveform w4 output by the second inference model.

[0039] The learning device 1 then converts each of the waveforms w1 to w4 into PSDs (Power Spectrum Densities) p1 to p4. The PSD represents frequency characteristics and corresponds to information from which vital signs such as heart rate and respiratory rate can be obtained.

[0040] The learning device 1 then performs contrastive learning of the first inference model and the second inference model so as to approximate PSDp1 and PSDp2, which correspond to the inference results of the first and second inference models based on a correct pair of clip images CV1 and CV2 captured simultaneously of the same subject X. Similarly, the learning device 1 performs contrastive learning of the first and second inference models so as to approximate PSDp3 and PSDp4, which correspond to the inference results of the first and second inference models based on a correct pair of clip images CV3 and CV4 captured simultaneously of the same subject Y.

[0041] Meanwhile, the learning device 1 performs contrastive learning of the first inference model so as to distance PSDp1 and PSDp3, which correspond to the inference results of the first inference model based on clip videos CV1 and CV3 captured during different imaging periods of different subjects. The learning device 1 also performs contrastive learning of the first inference model so as to distance PSDp2 and PSDp4, which correspond to the inference results of the second inference model based on clip videos CV2 and CV4 captured during different imaging periods of different subjects. Similarly, the learning device 1 performs contrastive learning of the first inference model and the second inference model so as to distance PSDp1 and PSDp4, which correspond to the inference results of the first and second inference models based on clip videos CV1 and CV4 captured during different imaging periods of different subjects X and Y. The learning device 1 also performs contrastive learning of the second inference model and the first inference model so as to distance PSDp2 and PSDp3, which correspond to the inference results of the second and first inference models based on clip videos CV2 and CV3 captured during different imaging periods of different subjects X and Y. Although not shown in FIG. 3, clipped images of the same subject captured during different imaging periods are also an example of a negative pair.

[0042] In this case, the learning device 1 sets a loss function that brings positive paired inference results (e.g., PSDp1 and PSDp2) closer together and moves negative paired inference results (e.g., PSDp1 and PSDp3) further apart, and determines the parameters of the first inference model and the second inference model so as to minimize the loss function. Such a parameter determination algorithm may be any learning algorithm used in machine learning, such as gradient descent or backpropagation.

[0043] In this way, the learning device 1 performs contrastive learning of the first and second inference models, regarding vital information inferred based on clips of the same subject captured simultaneously by two cameras as a positive pair, and vital information inferred based on clips of different subjects or captured over different periods as a negative pair. This makes it possible to build an inference model specialized for a specific camera without requiring teacher labels.

[0044] (4) Functional Blocks Figure 4 shows an example of functional blocks of the learning device 1. As shown in Figure 4, the processor 11 of the learning device 1 functionally includes a data expansion unit 14, a conversion unit 15, an inference unit 16, and a learning unit (training unit) 17. In Figure 4, blocks that exchange data are connected by solid lines, but the combination of blocks that exchange data is not limited to that shown in Figure 4. The same applies to other functional block diagrams described below.

[0045] In the following explanation, it is assumed that the learning data storage unit 21 contains a positive pair of clipped images of the same subject captured simultaneously by the first and second cameras, and a negative pair of clipped images of the same subject captured by the first and second cameras with different subjects or different imaging periods. The learning device 1 then determines the parameters of the first inference model corresponding to the first camera and the parameters of the second inference model corresponding to the second camera through contrastive learning.

[0046] The data expansion unit 14 extracts clip images to be used for learning from the learning data storage unit 21 and performs temporal data augmentation on the extracted clip images. As a result, the data expansion unit 14 generates clip images (also referred to as "extended clip images") that represent different vital information from the clip images to which data augmentation is applied (also referred to as "original clip images"). Specifically, the data expansion unit 14 generates the extended clip images by upsampling or downsampling the number of frames of the original clip images. For example, in estimating a heart rate or a respiratory rate, the data expansion unit 14 downsamples the original clip images to generate extended clip images in which the original clip images are sped up (i.e., the heart rate or respiratory rate is increased). Furthermore, the data expansion unit 14 upsamples the original clip images to generate extended clip images in which the original clip images are sped up (i.e., the heart rate or respiratory rate is decreased). The downsampling rate or upsampling rate may be any value. Furthermore, the data extension unit 14 may perform data extension (such as rotation, inversion, clipping, and color conversion) that is commonly used for images, in addition to data extension related to time.

[0047] Preferably, the data expansion unit 14 applies data expansion for the same time to a positive pair of clip images captured simultaneously of the same subject. In this manner, extended clip images generated by applying data expansion for the same time to the original clip images of a positive pair also become a positive pair. Therefore, the data expansion unit 14 generates a positive pair of extended clip images, thereby increasing the number of samples in the positive pair. Furthermore, since a negative pair can be formed from different positive pair of extended clip images, it is possible to increase the number of samples in the negative pair as well as the number of samples in the positive pair. Regarding image data expansion, the data expansion unit 14 may also apply different data expansion to the positive pair of clip images. This is because it may be possible to assume that the vital information is the same even when different data expansion is applied to the positive pair of clip images.

[0048] FIG. 5 is a schematic diagram of data expansion by the data expansion unit 14. In FIG. 5, the same data expansion (here, downsampling at a predetermined downsampling rate) is applied to clip videos CV10 and CV20, each having a frame number "N1" and generated by simultaneously imaging subject X with the first and second cameras. As a result, extended clip videos CV11 and CV21, each having a frame number "N11" (where N11<N1), are generated. The extended clip videos CV11 and CV21 form a positive pair of clip videos. Similarly, the same data expansion (here, upsampling at a predetermined upsampling rate) is applied to clip videos CV30 and CV40, each having a frame number N1 and generated by simultaneously imaging subject Y with the first and second cameras. As a result, extended clip videos CV31 and CV41, each having a frame number "N12" (where N12>N1), are generated. The extended clip video CV31 and the extended clip video CV41 form a correct pair of clip videos.

[0049] Referring again to FIG. 4, the functional block configuration of the learning device 1 will be described.

[0050] The converter 15 converts the clip video (including the original clip video and the extended clip video; the same applies hereinafter) supplied from the data extension unit 14 into a spatiotemporal map, which is a tensor consistent with the input format of the inference model. Here, as an example, it is assumed that the inference model is a two-dimensional convolutional neural network, and the converter 15 converts the clip video into a spatiotemporal map that expresses time and space features in two dimensions. The converter 15 then supplies the converted spatiotemporal map to the inference unit 16.

[0051] Here, a specific example of generating a spatiotemporal map will be described.

[0052] For example, when generating a spatiotemporal map for heart rate estimation, the converter 15 extracts block regions (RGB signals) under the eyes, including the mouth and nose, from the face region of each image in the clip video using an arbitrary image recognition technology, and generates a matrix signal as a spatiotemporal map, with the row direction being the spatial direction and the column direction being the time direction. The spatiotemporal map has three channels, one for RGB. In this case, for example, the matrix signal is obtained by arranging pixel values ​​of multiple block regions extracted from each frame constituting the clip video in the row direction (spatial direction) and arranging time-series information of the pixel values ​​of each block region in the column direction (time direction). Note that, in generating the matrix signal, the converter 15 may also perform processing to reduce the resolution of the block regions. The above-described processing generates a spatiotemporal map of RGB signals based on minute changes in reflected light from the face accompanying the heartbeat cycle, and this spatiotemporal map is suitable for use in pulse wave estimation.

[0053] As another example, when generating a spatiotemporal map for respiration estimation, the converter 15 extracts a region near the chest from the face region of each image in the clip video using any image recognition technology and generates vertical optical flow signals at multiple feature points in the extracted region near the chest. The optical flow signal captures the movement of an object between frames in a video, and corresponds to a signal representing chest movement associated with respiratory movement in this case. Because respiratory movement is vertical movement, a single-channel optical flow signal related to the vertical direction at multiple feature points is extracted. The converter 15 then arranges multiple feature points of the optical flow signals of each frame constituting the clip video in the row direction (spatial direction) and generates a spatiotemporal map in which the time-series information of each feature point is arranged in the column direction (temporal direction). The spatiotemporal map has one channel of optical flow signals. By the above-described processing, a spatiotemporal map of vertical chest movement based on respiratory movement is generated, and this spatiotemporal map is suitable for use in estimating respiration.

[0054] The conversion unit 15 may extract the block areas under the eyes and the area near the chest based on an external input from the user. In this case, the conversion unit 15 may display each image of the clip video on a display unit (not shown) and accept a user input specifying the area to be extracted.

[0055] The generation of the spatiotemporal map is not limited to the above-described specific example. For example, in heart rate estimation, the converter 15 may generate the spatiotemporal map using any one or two channels instead of three channels of RGB signals. In another example, in respiration estimation, the converter 15 may generate the spatiotemporal map using RGB signals instead of optical flow signals. Furthermore, the converter 15 may generate a spatiotemporal map by combining optical flow signals and RGB signals. In yet another example, if a feature extractor has been acquired by training in advance, the converter 15 may input a video clip to the feature extractor to obtain the spatiotemporal map output by the feature extractor. In this case, trained parameters of the feature extractor, etc., are stored in advance in the storage device 2, etc., and the converter 15 inputs a video clip to the feature extractor to obtain the spatiotemporal map output by the feature extractor.

[0056] The inference unit 16 generates an inference result regarding vital information based on the spatiotemporal map supplied from the conversion unit 15 and an inference model (first inference model or second inference model) constructed based on the model information stored in the model information storage unit 22. Here, for a spatiotemporal map based on a clip video generated by a first camera, the inference unit 16 inputs the spatiotemporal map into the first inference model to obtain an inference result output by the first inference model. On the other hand, for a spatiotemporal map based on a clip video generated by a second camera, the inference unit 16 inputs the spatiotemporal map into the second inference model to obtain an inference result output by the second inference model.

[0057] The learning unit 17 performs contrastive learning to update the parameters of the first inference model and the second inference model based on any inference result supplied from the inference unit 16. In this case, the learning unit 17 uses a loss function that brings positive paired inference results closer and moves negative paired inference results farther apart, and determines the parameters of the first inference model and the second inference model so as to minimize the loss function. Such a parameter determination algorithm may be any learning algorithm used in machine learning, such as gradient descent or backpropagation.

[0058] Here, a specific example of the loss function will be described. Here, a case where N (N is an integer equal to or greater than 2) pairs of clip videos are generated will be described. In this case, there are N clip videos of the first camera and N clip videos of the second camera.

[0059] Here, the learning unit 17 calculates, for example, a loss function L shown in the following formula (1): However, the loss function is not limited to formula (1), and any loss function may be used as long as it brings inference results that are positive pairs closer together and moves inference results that are negative pairs farther apart.

[0060] Here, the first term of formula (1) "L ａ ｎ " denotes the loss function for the nth (n=1, . . . , N) clip video of the first camera, and is expressed by the following equation (2). ａ ｎ is a hyperparameter that adjusts the weight of

[0061] where "d" represents a distance function (e.g., mean square error) and "p ａ ｎ "," "p ａ ｋ " indicates the inference result for the nth or kth clip image of the first camera, and "p ｂ ｎ "," "p ｂ ｋ " indicates the inference result for the nth or kth clip image of the second camera, and "τ" indicates the temperature parameter, which is a hyperparameter. ｋ≠ｎ" is a function that returns 1 when k≠n and 0 when k=n. Equation (2) represents the loss related to the inference result based on the clip video of the nth first camera. Specifically, equation (2) represents the cross-entropy loss using the distance between the inference results of the positive pair and the distance between the inference results of the negative pair based on the clip video of the nth first camera. This allows learning to be performed such that the inference results of the positive pair are closer and the inference results of the negative pair are further apart. Note that the inference results of the negative pair are inference results based on clip video generated by the same camera or different cameras where at least one of the subject or the shooting period is different, as described in the example of Figure 3.

[0062] In addition, the second term of formula (1), "L ｂ ｎ " corresponds to the loss function for the clip video of the second camera among the clip videos of the n-th correct pair, and is expressed by the following equation (3). ｂ ｎ is a hyperparameter that adjusts the weight of

[0063] Here, Equation (3) represents the loss related to the inference result based on the n-th clip video of the second camera. Specifically, it represents the cross-entropy loss using the distance between the positive pair of inference results and the distance between the negative pair of inference results based on the n-th clip video of the second camera.

[0064] 6 shows an example of functional blocks of the estimation device 3 related to the process of estimating vital information using an inference model learned by the learning device 1. As shown in FIG. 6, the processor 31 of the estimation device 3 functionally includes an image acquisition unit 34, a conversion unit 35, an inference unit 36, and an output control unit 37. In FIG. 6, blocks between which data is exchanged are connected by solid lines, but the combination of blocks between which data is exchanged is not limited to this. The same applies to the other functional block diagrams described below.

[0065] The video acquisition unit 34 acquires video data capturing an image of the face of a human subject from the camera 5. Here, the camera 5 may be the first camera used for learning, or may be the second camera. The video acquisition unit 34 then extracts a video clip from the acquired video data that has a format that can be converted by the conversion unit 35, and supplies the extracted video clip to the conversion unit 35.

[0066] The conversion unit 35 generates a spatiotemporal map by converting the video clip supplied from the video acquisition unit 34. The processing of the conversion unit 35 is the same as the processing of the conversion unit 15. The conversion unit 35 supplies the generated spatiotemporal map to the inference unit 36.

[0067] The inference unit 36 ​​generates an inference result for vital information based on the model information of the inference model corresponding to the camera 5 and the spatiotemporal map supplied from the conversion unit 35. In this case, the inference unit 36 ​​reads out the learned parameters of the inference model corresponding to the camera 5 from the model information storage unit 22, and acquires the inference result for vital information output by the inference model by inputting the spatiotemporal map into the inference model to which the learned parameters have been applied. Note that if the data output by the inference model is a waveform such as a pulse wave or a respiratory waveform, the inference unit 36 ​​may further perform processing for converting the data into a heart rate, a respiratory rate, or the like. The inference unit 36 ​​supplies the generated inference result for vital information to the output control unit 37.

[0068] The output control unit 37 performs processing to output the inference result of the vital information supplied from the inference unit 36 ​​as estimated vital information. In this case, the output control unit 37 may display or output the estimated vital information as sound using an output device such as a display or a speaker, or may store or transmit the vital information in the storage device 2 or another device.

[0069] The components of the data expansion unit 14, the conversion unit 15, the inference unit 16, and the learning unit 17 can be realized, for example, by the processor 11 executing a program. Similarly, the components of the conversion unit 35, the inference unit 36, and the output control unit 37 can be realized, for example, by the processor 31 executing a program. Alternatively, the necessary programs may be recorded on any non-volatile storage medium and installed as needed to realize each component. At least some of these components may not be realized by software programs, but may be realized by any combination of hardware, firmware, and software. At least some of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program consisting of the above components. At least some of the components may be configured using an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various types of hardware. The above also applies to other embodiments described later. Furthermore, each of these components may be realized by the cooperation of multiple computers using, for example, cloud computing technology.

[0070] (5) Processing Flow FIG. 7 is an example of a flowchart showing the processing procedure performed by the learning device 1.

[0071] First, the learning device 1 acquires clip videos to be used for learning from the learning data storage unit 21 (step S11). Then, the learning device 1 applies data extension to the clip videos acquired in step S11 (step S12). As a result, the learning device 1 generates extended clip videos using the clip videos acquired in step S11 as original clip videos. In this case, the learning device 1 applies the same data extension to the original clip videos that form the correct pair, thereby generating extended clip videos that form the correct pair.

[0072] The learning device 1 then converts each piece of video data (including the original clip video and the extended clip video) into a spatiotemporal map (step S13). Next, the learning device 1 acquires an inference result related to vital information from each spatiotemporal map using an inference model corresponding to the camera that generated the clip video that is the basis of each spatiotemporal map (step S14). This allows for the acquisition of an inference result of the inference model corresponding to each of the spatiotemporal maps acquired in step S13.

[0073] The learning device 1 then performs contrastive learning of the inference models so as to bring inference results based on clips of the same subject captured simultaneously closer together and to move inference results based on clips of different subjects or different imaging periods further apart (step S15). In this case, the learning device 1 determines parameters for each inference model so as to minimize the loss function shown in equation (1), and updates the model information for each inference model stored in the model information storage unit 22 with the determined parameters.

[0074] (6) Modifications Next, modifications that can be applied to the above-described embodiment will be described. The following modifications may be implemented in any combination.

[0075] (Variation 1) When either the first inference model or the second inference model is a trained inference model, only the other inference model may be trained by contrastive learning based on this embodiment. Hereinafter, for ease of explanation, the first inference model will be considered to be a trained inference model. Figure 8 shows an overview of the learning of the inference model in Variation 1.

[0076] In this case, the first inference model is trained in advance using a training data set including multiple records each pairing a clip video generated by the first camera with a teacher label (correct answer label) indicating the correct answer that the first inference model should output when the clip video is input, and the trained parameters are stored in the model information storage unit 22. The teacher label is, for example, data acquired from a sensor (e.g., a pulse wave sensor or a respiratory band) that senses the subject's vital information when the clip video is captured.

[0077] The learning device 1 then executes the flowchart shown in Figure 7. In this case, in step S15, the parameters of the first inference model are fixed, and the parameters of the second inference model are changed so as to minimize the loss function shown in equation (1) etc. Note that the learning device 1 may also perform further learning on the first inference model without fixing the parameters of the first inference model. In other words, the learning device 1 may change the parameters of both the first and second inference models so as to minimize the loss function shown in equation (1) etc.

[0078] (Variation 2) The inference model may be a model that uses three-dimensional feature information (three-dimensional feature map) as input data, instead of a model that uses a spatiotemporal map, which is two-dimensional feature information, as input data.

[0079] In this case, the inference model is a model that is trained to output an inference result of vital information based on the clip video when three-dimensional feature information based on the clip video is input. For example, in step S13 of Fig. 7, the learning device 1 generates three-dimensional feature information from the video clip, having three axes corresponding to two-dimensional spatial directions (i.e., the same two-dimensional coordinate axes as the image) and the time direction, respectively, and in step S14, the learning device 1 inputs the three-dimensional feature information to the inference model to obtain an inference result of vital information from the inference model.

[0080] Note that the input data for the inference model is not limited to two-dimensional or three-dimensional feature information, but may be input data of any format represented by tensors.

[0081] (Modification 3) The learning data storage unit 21 may store measurement data generated by a sensor other than a camera in addition to clipped video generated by a camera.

[0082] Examples of such sensors include a microphone that collects sounds related to the subject's heartbeat and breathing, and a temperature sensor that measures the subject's breathing-related temperature. In this case, the learning data storage unit 21 contains measurement data measured by the sensor at the same time as each clip video. Then, in step S13 of FIG. 7, the learning device 1 converts each clip video and the measurement sensor generated at the same time as each clip video into a spatiotemporal map. This aspect allows the learning device 1 to learn an inference model that uses information other than video.

[0083] (Modification 4) The learning data storage unit 21 may store video clips generated from three or more cameras.

[0084] In this case, for example, if the number of cameras is "Nc", the learning device 1 calculates Ｎｃ C ２ A set of camera pairs is formed, and for each pair, contrastive learning of an inference model corresponding to the camera of each pair is performed. In this case, the learning device 1 acquires clipped video from the cameras of the target pair in step S11 of the flowchart in FIG. 7, acquires an inference result using the inference model corresponding to the cameras of the target pair in step S14, and learns the parameters of the inference model corresponding to the cameras of the target pair based on the inference result in step S15. Note that in step S15, the learning device 1 may define an overall loss function corresponding to the Nc cameras and determine parameters of the Nc inference models corresponding to the Nc cameras so as to minimize the loss function. The loss function used in this case is, for example, Ｎｃ C ２ Corresponding to a pair of cameras Ｎｃ C ２ The loss function may be a sum of the loss functions of equation (1) for each of the above.

[0085] (Variation 5) The learning device 1 may learn a single inference model that functions as both the first inference model and the second inference model. In this case, the first inference model and the second inference model are common inference models regardless of the camera.

[0086] In this case, the model information storage unit 22 stores model information for one inference model, and in step S14 of Fig. 7, the learning device 1 acquires inference results from each spatiotemporal map using one inference model constructed with reference to the model information storage unit 22. Then, when performing contrastive learning in step S15 of Fig. 7, the learning device 1 updates the parameters of the one inference model so as to bring the inference results based on the positive example clip video closer and move the inference results based on the negative example clip video further away.

[0087] According to this modification, the learning device 1 can learn an inference model that is not dependent on an individual camera.

[0088] (Variation 6) The inference model is not limited to a model that outputs inference results regarding vital information of a subject who is the subject of a clip video input to the inference model, but may also be a model that outputs inference results regarding any condition information of the subject. In this case, examples of the condition information include information regarding any internal condition of the subject, such as stress, and information regarding the subject's external condition (e.g., the degree of swelling).

[0089] (Variant 7) The estimation device 3 may determine a course of action to present to the subject based on the inference results of vital information (or other status information based on variant 6) output by an inference model based on, for example, clip footage of the subject.

[0090] In this case, the estimation device 3 determines a countermeasure to be presented to the subject based on a model generated by machine learning the correspondence between the estimated vital information and the countermeasure, and the estimated vital information of the subject. The trained parameters of the above-mentioned model are stored in advance in, for example, the storage device 2 or the memory 32. Note that the method for determining the countermeasure is not limited to the above-mentioned method. In this way, the inference model obtained by learning can be used to support decision-making by users (medical professionals (doctors, nurses, public health nurses, laboratory technicians, therapists, trainers, etc.), public health nurses, healthcare counselors, etc.).

[0091] 9 shows a schematic configuration of a learning device 1X according to a second embodiment. The learning device 1X mainly includes a feature information acquisition unit 15X and a learning unit 17X. The learning device 1X may be composed of multiple devices.

[0092] The feature information acquisition means 15X acquires first feature information, which is feature information in time and space generated from a first video of the subject captured by a first camera, and acquires second feature information, which is feature information in time and space generated from a second video of the subject captured by a second camera. The feature information acquisition means 15X may itself perform the process of generating the first feature information from the first video and the process of generating the second feature information from the second video. In this case, the feature information acquisition means 15X may be, for example, the conversion unit 15 in the first embodiment. In another example, the feature information acquisition means 15X may receive the first feature information and the second feature information from a device other than the learning device 1X that performs the above-mentioned process of generating the feature information.

[0093] The learning means (training means) 17X learns an inference model that infers the relationship between the feature information and the condition information of the subject based on the first feature information and the second feature information. In this case, the learning means 17X performs learning by treating the first feature information and the second feature information based on images of the same subject captured during the same period as a positive pair. The learning means 17X can be, for example, the learning unit 17 in the first embodiment.

[0094] 10 is an example of a flowchart showing a processing procedure executed by the learning device 1X in the second embodiment. The feature information acquisition means 15X acquires first feature information, which is feature information in time and space generated from a first video of the subject captured by a first camera, and acquires second feature information, which is feature information in time and space generated from a second video of the subject captured by a second camera (step S21). Then, the learning means 17X learns an inference model that infers a relationship between the feature information and the subject's condition information based on the first feature information and the second feature information. In this case, the learning means 17X learns the first feature information and the second feature information based on videos of the same subject captured during the same period as a positive pair (step S22).

[0095] According to the second embodiment, the learning device 1X can preferably perform learning of an inference model using feature information based on images of the same subject taken during the same period.

[0096] In the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor or the like. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can be supplied to a computer via wired communication paths such as electric wires and optical fibers, or via wireless communication paths.

[0097] In addition, some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0098] [Supplementary Note 1] A learning device comprising: a feature information acquisition means for acquiring first feature information, which is feature information in time and space generated from a first video of a subject captured by a first camera, and acquiring second feature information, which is the feature information generated from a second video of the subject captured by a second camera; and a learning means for learning an inference model that infers a relationship between the feature information and state information of the subject based on the first feature information and the second feature information, wherein the learning means performs the learning by treating the first feature information and the second feature information, which are based on videos of the same subject captured during the same period, as a positive pair. [Supplementary Note 2] The learning device according to Supplementary Note 1, comprising: a first inference model that infers a relationship between the first feature information and the state information, and first inference means that generates a first inference result regarding the state information based on the first feature information; a second inference model that infers a relationship between the second feature information and the state information, and second inference means that generates a second inference result regarding the state information based on the second feature information; wherein the learning means learns at least one of the first inference model or the second inference model based on the first inference result and the second inference result. [Supplementary Note 3] The learning device according to Supplementary Note 2, wherein the learning means performs contrastive learning of at least one of the first inference model or the second inference model based on the first inference result and the second inference result. [Supplementary Note 4] The learning device according to Supplementary Note 2, wherein the learning means updates parameters of at least one of the first inference model or the second inference model so as to bring the first inference result and the second inference result, respectively, generated based on the positive pair of the first feature information and the second feature information, closer to each other. [Supplementary Note 5] The learning device according to Supplementary Note 3 or 4, wherein the learning means updates parameters of at least one of the first inference model or the second inference model so as to move away from the first inference result and the second inference result, which are generated based on the first feature information and the second feature information, respectively, based on images in which at least one of the subject and the imaging period is different. [Supplementary Note 6] The learning device according to Supplementary Note 2, wherein the first inference model is a trained model, and the learning means trains the second inference model based on the first inference result and the second inference result.[Supplementary Note 7] The learning device according to Supplementary Note 2, wherein the learning means learns one inference model that functions as the first inference model and the second inference model. [Supplementary Note 8] The learning device according to Supplementary Note 1, further comprising: data extension means that generates first and second extended images by applying common data extension to the first image and the second image corresponding to the first feature information and the second feature information that form the positive pair; [Supplementary Note 9] The learning device according to Supplementary Note 2, wherein the feature information acquisition means acquires third feature information, which is the feature information generated from a third video of the subject captured by a third camera, and the learning means has third inference means for generating a third inference result regarding the status information based on the third feature information and a third inference model that infers a relationship between the third feature information and the status information, and the learning means learns at least one of the first inference model, the second inference model, and the third inference model based on the first inference result, the second inference result, and the third inference result. [Supplementary Note 10] The learning device according to Supplementary Note 1, wherein the status information is at least one of information regarding the vital signs of the subject, information regarding the internal state of the subject, and information regarding the external state of the subject. [Supplementary Note 11] A learning method in which a computer acquires first feature information, which is feature information in time and space generated from a first image of a subject captured by a first camera, acquires second feature information, which is the feature information generated from a second image of the subject captured by a second camera, and learns an inference model that infers a relationship between the feature information and status information of the subject based on the first feature information and the second feature information, and performs the learning by treating the first feature information and the second feature information based on images of the same subject captured during the same period as a positive pair.[Supplementary Note 12] A storage medium storing a program that causes a computer to execute a process of acquiring first feature information, which is feature information in time and space generated from a first video of a subject captured by a first camera, acquiring second feature information, which is the feature information generated from a second video of the subject captured by a second camera, learning an inference model that infers a relationship between the feature information and status information of the subject based on the first feature information and the second feature information, and performing the learning process with the first feature information and the second feature information based on videos of the same subject captured during the same period as a positive pair.

[0099] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent documents and other documents are incorporated herein by reference.

[0100] REFERENCE SIGNS LIST 1, 1X Learning device 2 Storage device 3 Estimation device 5 Camera 11, 31 Processor 12, 32 Memory 13, 33 Interface 21 Learning data storage unit 22 Model information storage unit 100 Vital information estimation system

Claims

1. a feature information acquiring means for acquiring first feature information, which is feature information in time and space generated from a first image of a subject captured by a first camera, and for acquiring second feature information, which is the feature information generated from a second image of the subject captured by a second camera; a learning means for learning an inference model that infers a relationship between the feature information and state information of the subject based on the first feature information and the second feature information; and the learning means performs the learning by treating the first feature information and the second feature information based on images of the same subject taken during the same period as a positive pair. Learning device.

2. a first inference model for inferring a relationship between the first feature information and the state information, and a first inference means for generating a first inference result regarding the state information based on the first feature information; a second inference model for inferring a relationship between the second feature information and the state information; and a second inference means for generating a second inference result regarding the state information based on the second feature information, The learning device according to claim 1 , wherein the learning means learns at least one of the first inference model or the second inference model based on the first inference result and the second inference result.

3. The learning device according to claim 2 , wherein the learning means performs contrastive learning of at least one of the first inference model or the second inference model based on the first inference result and the second inference result.

4. The learning device described in claim 2, wherein the learning means updates parameters of at least one of the first inference model or the second inference model so as to bring closer the first inference result and the second inference result, respectively generated based on the positive pair of the first feature information and the second feature information.

5. The learning device described in claim 3 or 4, wherein the learning means updates parameters of at least one of the first inference model or the second inference model so as to move away from the first inference result and the second inference result, which are respectively generated based on the first feature information and the second feature information based on images in which at least one of the subject or the shooting period is different.

6. the first inference model is a trained model, The learning device according to claim 2 , wherein the learning means learns the second inference model based on the first inference result and the second inference result.

7. The learning device according to claim 2 , wherein the learning means performs learning of one inference model that functions as the first inference model and the second inference model.

8. further comprising a data extension means for generating a first extended image and a second extended image by applying common data extension to the first image and the second image corresponding to the first feature information and the second feature information that form the positive pair; The learning device according to claim 1 , wherein the learning means performs the learning by treating the first feature information based on the first extended video and the second feature information based on the second extended video as a positive pair.

9. the feature information acquisition means acquires third feature information, which is the feature information generated from a third video image of the subject captured by a third camera; a third inference means for generating a third inference result regarding the state information based on the third feature information and a third inference model for inferring a relationship between the third feature information and the state information; The learning device described in claim 2, wherein the learning means learns at least one of the first inference model, the second inference model, or the third inference model based on the first inference result, the second inference result, and the third inference result.

10. The learning device according to claim 1 , wherein the condition information is at least one of information relating to the subject's vital signs, information relating to the subject's internal condition, and information relating to the subject's external appearance.

11. The computer acquiring first feature information, which is feature information in time and space generated from a first image of the subject captured by a first camera; and acquiring second feature information, which is the feature information generated from a second image of the subject captured by a second camera; learning an inference model that infers a relationship between the feature information and state information of the subject based on the first feature information and the second feature information; the learning is performed with the first feature information and the second feature information based on images of the same subject photographed during the same period as a positive pair; How to learn.

12. acquiring first feature information, which is feature information in time and space generated from a first image of the subject captured by a first camera; and acquiring second feature information, which is the feature information generated from a second image of the subject captured by a second camera; learning an inference model that infers a relationship between the feature information and state information of the subject based on the first feature information and the second feature information; A program that causes a computer to execute a process of performing the learning by treating the first feature information and the second feature information based on images of the same subject taken during the same period as a positive pair.