Information processing device, information processing method, and information processing program
The information processing system uses a reduced complexity neural network with incremental learning to achieve real-time, accurate estimation of subject conditions from moving images, addressing the limitations of existing techniques in processing load and estimation speed.
Patent Information
- Application Number
- JP2024502032
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-10-16
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2043-10-16
AI Technical Summary
Existing techniques for estimating various information fail to provide real-time appropriate estimations due to high information processing loads and inability to perform robust state estimation from moving images.
An information processing system utilizing a convolutional neural network with a new learning method that reduces the size and complexity of the neural network while maintaining accuracy, enabling real-time estimation of subject conditions from moving images by gradually incorporating additional information during learning.
The system achieves highly accurate, real-time estimation of subject conditions with reduced computational load by optimizing the neural network through incremental learning, ensuring robustness against additional information.
Smart Images

Figure 0007582728000001 
Figure 0007582728000002 
Figure 0007582728000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]
[0002] Conventionally, techniques for making estimations from various pieces of information are known. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2017-029318 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, with the above-mentioned technology, it is not possible to perform appropriate estimation in real time when making estimations from various pieces of information.
[0005] The present application has been made in consideration of the above, and has an object to perform appropriate estimation in real time when making estimations from various pieces of information. [Means for solving the problem]
[0006] The information processing device of the present application is characterized by having an acquisition unit that acquires first learning information including at least first information whose importance in the estimation satisfies a predetermined condition among information for allowing an estimator that performs an estimation based on input information to learn features, and second learning information including the first information as well as predetermined second information, and a learning unit that uses the first learning information to train the estimator, and after training using the first learning information, uses the second learning information to train the estimator that has been trained using the first learning information. Effect of the Invention
[0007] According to one aspect of the embodiment, when making an estimation from various pieces of information, it is possible to advantageously make an estimation appropriately in real time. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of information processing according to the embodiment. [Diagram 3] FIG. 3 is an explanatory diagram for explaining the overall flow of the model. [Figure 4] FIG. 4 is an explanatory diagram for explaining the learning process according to the embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of the configuration of an information display device according to an embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a sensor information storage unit according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a model information storage unit according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of the architecture of the first model. [Figure 10] FIG. 10 is a diagram illustrating an example of the architecture of the second model. [Figure 11] FIG. 11 is a diagram illustrating an example of the architecture of the third model. [Figure 12] FIG. 12 is a flowchart (1) showing an example of a procedure of information processing according to the embodiment. [Figure 13] FIG. 13 is a flowchart (2) showing an example of the procedure of information processing according to the embodiment. [Figure 14] FIG. 14 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, the information processing device, the information processing method, and the information processing program according to the present application will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to the embodiments. In addition, the same parts in the following embodiments are given the same reference numerals, and duplicated explanations are omitted.
[0010] Photoplethysmography (PPG) is a known technology that uses light to measure blood flow. For example, animals with various circulatory systems have blood flowing through their bodies, and the amount of this blood flow, i.e., blood flow, changes depending on the pulsation of the heart. The speed (i.e., pulse rate) and strength (i.e., blood pressure) of such heart pulsation change depending on the physical and mental conditions of the animal. For this reason, it can be said that the physical and mental conditions of an animal can be estimated from changes in blood flow.
[0011] Here, it is known that the blood of general warm-blooded animals (especially mammals) selectively absorbs light of a specific wavelength (especially green light) because it contains hemoglobin. When blood flow increases due to an increase in blood pressure or heart rate, the amount of light absorbed of the specific wavelength also increases. A measuring device that measures blood flow using PPG irradiates light onto the skin, etc., and measures the reflected light using a photodiode or the like. The measuring device then calculates the absorbance of the light of the specific wavelength from the intensity of the light of the specific wavelength in the irradiated light and the intensity of the light of the specific wavelength in the reflected light, and can estimate the blood flow based on the calculated absorbance.
[0012] In addition, by applying PPG technology, it is also possible to measure pulses that indicate fluctuations in blood flow from moving images, that is, PPG pulses. The technology to measure PPG pulses from moving images is also called VPPG (this technology is also sometimes called Remote PPG or rPPG). For example, because blood flow exists in the skin, changes in this blood flow cause changes in the absorbance of light of specific wavelengths, and as a result, changes in blood flow cause changes in skin tone. VPPG is a technology that measures PPG pulses based on changes in skin tone. Since the face has a high concentration of blood vessels, it is known that facial skin is particularly suitable for measurement.
[0013] It is known that there are various methods for VPPG. For example, when the subject is not in a moving state such as sitting, a method is known in which the face is detected using a predetermined algorithm (e.g., the Viola-Jones method) for detecting the face from a moving image, and the PPG pulse is measured using a channel of the optical signal of the whole face or a part of the face. In addition, a method is known in which the heart rate signal is separated from the raw signal using Blind Source Separation (BSS), and the PPG pulse is measured. In addition, a method is known in which the heart rate signal is extracted from the raw signal using Power Spectral Density (PSD) analysis, and the PPG pulse is measured. In addition, a method is known in which the PPG pulse is measured by adjusting the brightness of the background using an NLMS (Normalized Least Mean Squared) adaptive filter and suppressing the influence of changes in brightness.
[0014] In addition, methods for suppressing the effects of changes in brightness are not limited to methods for suppressing the effects of changes in motion. For example, a method using an algorithm that enables robust estimation using a pixel-based remote PPG sensor is known. In addition, a method is known that takes into account the translational movement of each image pixel between frames using an optical flow algorithm using the Farneback method. Note that the optical flow algorithm using the Farneback method is a high-density optical flow algorithm, and may impose a larger load on information processing than the optical flow algorithm using the Lucas-Kanade method.
[0015] In addition, the shift in color space can effectively suppress the effects of changes in brightness and changes in movement. In addition, the calculation efficiency is good, so the delay in BVP (volume pulse wave) estimation can also be suppressed. In addition, RGB (Red Green Blue) requires filtering processing. In addition, since PCA (Principal Component Analysis) and ICA (Independent Component Analysis) are required, a shift in color space may be necessary. Note that both PCA and ICA are methods of scaling dimensions, and the delay in BVP estimation may be large.
[0016] The above-mentioned VPPG method is known to use artificial convolutional neural networks (CNNs) combined with long short-term memory layers and attention masks that measure VPPG from video images. However, these neural networks tend to increase the number of nodes and layers to improve accuracy, which can be a heavy load for real-time use.
[0017] In the following embodiment, a convolutional neural network is used to measure VPPG. In addition, the use of a convolutional neural network reduces the size of the network, making it lightweight and enabling real-time use.
[0018] In addition, conventionally, there are known techniques for making inferences from various information. For example, there is known a technique for grasping a subject's condition by reading the subject's biological information from a moving image. For example, there is known a technique for generating heart rate data from a face image to grasp the subject's stress state.
[0019] However, the above-mentioned technology cannot estimate appropriately in real time when estimating from various information. For example, the above-mentioned technology cannot estimate the state of a subject appropriately in real time from a moving image because the information processing load is large and robust state estimation is not possible.
[0020] (Embodiment) [1. Configuration of information processing system] First, an information processing system 1 according to an embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of the information processing system 1 according to the embodiment. As shown in FIG. 1, the information processing system 1 includes an information display device 10 and an information processing device 100. The information display device 10 and the information processing device 100 are connected to each other via a predetermined communication network (network N) so as to be able to communicate with each other by wire or wirelessly. Note that the information processing system 1 shown in FIG. 1 may include a plurality of information display devices 10 and a plurality of information processing devices 100.
[0021] The information display device 10 is an information processing device used by a subject. The information display device 10 may be any device that can realize the processing in the embodiment. The information display device 10 may be a device such as a smartphone, a tablet terminal, a notebook PC, a desktop PC, a mobile phone, or a PDA. The example shown in FIG. 2 shows a case where the information display device 10 is a notebook PC.
[0022] The information display device 10 is, for example, a smart device such as a smartphone or tablet, and is a mobile terminal device capable of communicating with any server device via a wireless communication network such as 4G to 5G (Generation) or LTE (Long Term Evolution). The information display device 10 has a screen such as a liquid crystal display with a touch panel function, and may receive various operations on displayed data such as content, such as tapping, sliding, and scrolling, performed by a subject using a finger or a stylus. In FIG. 2, the information display device 10 is used by a subject U. For example, the information display device 10 1 〜10 N are used by the subjects U1 to UN, respectively. X is used by the subject UX. The subjects U1 to UN are subjects collected to create training data for the neural network, and the subject UX is a subject whose state is estimated by applying the trained neural network. The information display device 10 may also have an imaging device such as a camera to capture the subject U.
[0023] The information processing device 100 is an information processing device intended to appropriately estimate the state of a subject from a moving image in real time, and may be any device as long as it can realize the processing in the embodiment. The information processing device 100 is realized, for example, by a server device or a cloud system that provides a service based on the state of the subject. For example, the information processing device 100 creates a skin image including at least a part of the skin of the subject from the moving image, and estimates the state of the subject in real time by using a convolutional neural network.
[0024] Although FIG. 1 shows a case where the information display device 10 and the information processing device 100 are separate devices, the information display device 10 and the information processing device 100 may be integrated.
[0025] Before describing an example of information processing executed by the information processing device 100, an overview will be given of how the information processing device 100 can realize VPPG using a neural network in real time.
[0026] The information processing device 100 estimates the subject's condition from an image by roughly three stages of information processing. Specifically, it performs a process of extracting the range in which the subject is photographed from the image, a process of estimating the subject's PPG pulse from the extracted range, and a process of estimating the subject's condition from the PPG pulse. In each of these processes, a separate neural network is adopted.
[0027] In general, a convolutional neural network used in a process for estimating a subject's PPG pulse from an extracted range has a large number of nodes and layers in order to estimate a PPG pulse from an image, and tends to become large in scale. As a result, the amount of calculations required for processing using a neural network increases, and real-time performance may be impaired.
[0028] Here, in order to improve the accuracy of information processing by a neural network, it may be possible to achieve this not only by increasing the scale of the NN, such as the number of nodes or layers, but also by improving the learning method. The applicant has discovered a method for improving the accuracy of a NN by using the learning method of the NN, without increasing the scale of the NN.
[0029] If the accuracy of the NN can be improved, it will be possible to ensure accuracy equal to or greater than that of a larger-scale NN, even if the NN is of a scale that can be executed in real time. In other words, when a certain level of accuracy is required in VPPG, if learning is performed using the NN learning method discovered by the applicant, it is possible to train a NN that is smaller in scale than conventional NNs and can be executed in real time.
[0030] Therefore, the information processing device 100 performs NN learning using a new NN learning method, thereby preventing an increase in the size of the NN and providing an NN capable of executing VPPG in real time.
[0031] In addition, by adopting a new NN learning method, real-time performance can be ensured not only for the NN that estimates the subject's PPG pulse from the extracted range, but also for the NN that extracts the range in which the subject is photographed from the image, and the NN that estimates the subject's condition from the PPG pulse.
[0032] [2. An example of information processing] FIG. 2 is a diagram showing an example of information processing of the information processing system 1 according to the embodiment. The subjects U1 to UN are in a sitting position and are fitted with a biosensor (e.g., a PPG sensor or a breathing sensor). For example, a PPG sensor may be fitted to the earlobe and a breathing sensor may be fitted under the chest. The PPG sensor and the breathing sensor are examples of biosensors and are not limited to these examples. For example, the subjects U1 to UN are initially encouraged to maintain a relaxed state for learning the neural network, and are then encouraged to solve mental arithmetic problems that cause a stressful state. The subject UX is in a sitting position but is not fitted with a biosensor such as a PPG sensor.
[0033] In the following embodiment, a case will be described in which the information processing device 100 extracts a face image as an example of a skin image including at least a part of the skin of the subject U, but the extracted skin image may not be limited to a face image. For example, in the following embodiment, the information processing device 100 may extract an arm image or a leg image of the subject U. Also, in the following embodiment, a case will be described in which the information processing device 100 estimates a stress state as an example of the state of the subject U, but the estimated state may not be limited to a stress state. For example, in the following embodiment, the information processing device 100 may estimate the state related to the emotions of the subject U (e.g., joy, anger, sadness, happiness, etc.).
[0034] (Learning process) The learning process of the information processing device 100 will be described below. The information processing device 100 acquires shooting information (corresponding to a moving image) of subjects U1 to UN (step S101). For example, the information processing device 100 acquires shooting information of subjects U1 to UN in real time. Specifically, the shooting information is images constituting a moving image.
[0035] The information processing device 100 extracts face images from the acquired photographic information as skin images including at least a part of the skin of the subjects U1 to UN (step S102). For example, the information processing device 100 extracts face images by applying a single-shot multi-box detector (SSD). A model used to extract face images from photographic information is hereinafter referred to as a "first model" as appropriate. Note that in the following embodiments, any method may be used as long as it is a method for extracting face images from photographic information, and is not limited to a method using the first model. The single-shot multi-box detector is an example of the first model.
[0036] The information processing device 100 applies the extracted face image to a Convolutional Attention Neural Network (CAN) to generate information (e.g., heart rate data, respiration data, etc.) for estimating the stress state of the subjects U1 to UN (step S103). The model used to generate information for estimating the stress state from the face image is hereinafter referred to as a "second model" as appropriate. The Convolutional Attention Neural Network is an example of the second model (corresponding to an estimator).
[0037] The information processing device 100 estimates the stress state of the subjects U1 to UN from the generated information (step S104). For example, the information processing device 100 estimates the stress state by applying a residual convolutional neural network (for example, a parallel 1D residual convolutional neural network, etc.). The model used to estimate the stress state is hereinafter referred to as a "third model" as appropriate. Note that in the following embodiment, any method may be used as long as it is a method for estimating the stress state, and is not limited to a method using the third model. The residual convolutional neural network is an example of the third model.
[0038] In the following embodiment, the first model, the second model, and the third model are collectively referred to as "models" as appropriate. The information processing device 100 learns the model using the information of the subjects U1 to UN as correct answer data (step S105). For example, the information processing device 100 learns the second model using the information of the subjects U1 to UN as correct answer data. At this time, the information processing device 100 may learn at least one of the first model and the third model using the information of the subjects U1 to UN as correct answer data. In addition, the information processing device 100 learns the model using the information output from the model and information from an actual biosensor such as a PPG sensor so that the information output from the model approaches the information from the biosensor. For example, the information processing device 100 trains a model (first model, second model, third model, first model / second model, first model / third model, second model / third model, or first model / second model / third model) so that information output from the third model (information indicating a stress state) approaches the information of the biosensor.
[0039] Here, the overall flow of the model will be described with reference to Fig. 3. Fig. 3 is an explanatory diagram for explaining the overall flow of the model. In the first step, a first model is used to extract an area suitable as an estimation source of subject state information from a moving image, such as a human face (for example, an area including human skin, such as a human face or hand, that is, an area estimated to change color according to blood flow), and generate an extracted moving image.
[0040] In the second step, the second model is used to generate information that is the basis for estimating the subject's state from the extracted video image, such as biosuggestive information that suggests blood flow and pulse. The second model has a blood flow detectable area extraction unit that extracts an area in which blood flow is detectable from the extracted video image, and an optical flow processing unit that processes the optical flow characteristics of the extracted video image. The blood flow detectable area extraction unit generates information indicating which area of each pixel in the extracted image should be emphasized, that is, attention, and the optical flow processing unit generates biosuggestive information from the optical flow image, taking into account the importance of each area of the optical flow image according to the attention. In the third step, the third model is used to generate subject state information from the biosuggestive information. The subject state information is, for example, information on the emotions of the subject U.
[0041] (Learning Environment) The learning environment will be described below. Note that the learning environment according to the embodiment may not be limited to the following example. As an example of the learning environment according to the embodiment, for example, a biosensor may be connected to a Biosignal Plux Hub. For example, the sampling rate of the biosensor may be set to a predetermined sampling rate (for example, 256 Hz or 700 Hz). For example, a general webcam may be used to capture a video image with a resolution of a predetermined pixel (for example, 680×420 pixels) and a frame rate of a predetermined FPS (for example, 30 FPS). For example, a camera may be installed at a position at a predetermined distance (for example, 0.5 m) from the face. For example, a black cloth may be used so that the background is black in order to increase the contrast. For example, after each session of a session encouraging a relaxed state and a session encouraging solving mental arithmetic problems, a subjective evaluation may be used to evaluate pleasure, discomfort, stress, etc. (for example, the degree of pleasure, discomfort, stress, etc. may be evaluated) using a Self-Assessment Manikin (SAM). For example, the image may be captured by multiple cameras to increase the number of samples. Also, for example, the subject may be photographed wearing a mask to be applicable even when part of the face is obscured.
[0042] Also, for example, a dataset may be created with different time frames, different shooting angles, different shooting distances, and different shooting illuminance (lighting settings). Also, for example, a dataset may be created with subjects with different attributes (for example, age, sex, nationality, skin color, etc.). Also, for example, a dataset may be created with a subject wearing a mask in some learning environments, and a subject not wearing a mask in some learning environments. Also, for example, a dataset may be created with a background in some learning environments, and a background other than black (for example, white, etc.) in some learning environments. Also, for example, a dataset may be created with a subject in a normal state (for example, a stress-free state, etc.) in some learning environments, and a subject in a stressed state in some learning environments. Also, for example, a dataset without a face may be created by extracting background videos of areas other than the face in each video image. Also, for example, a dataset that can be applied to a fake face may be created by shooting a fake face video of a face in a magazine or on a screen. Also, for example, a dataset including various information such as a plurality of different angles, illuminance, age, distance, and skin color may be created.
[0043] The information processing device 100 may use the data set created in this way to acquire information including information on a person who is a subject as first information, and information including information other than the person who is a subject as second information. The information processing device 100 may use the data set created in this way to acquire information including a predetermined type of information of the person who is a subject as first information, and information including information of a type other than the predetermined type of information of the person who is a subject as second information. The information processing device 100 may use the data set created in this way to acquire information including information on the skin of the person who is a subject as first information, and information including information other than the skin of the person who is a subject as second information.
[0044] (Data Augmentation and Data Preprocessing) Hereinafter, data augmentation and data preprocessing will be described. Note that data augmentation and data preprocessing according to the embodiment may not be limited to the following examples. In order to obtain highly accurate results in the estimation process, at least one of data augmentation and data preprocessing may be performed before learning the model. In addition, data augmentation or data preprocessing may be performed by the information processing device 100 or via an external information processing device.
[0045] As an example of data augmentation according to the embodiment, for example, a predetermined conversion process may be performed on the shooting information. For example, conversion such as rotation (e.g., random rotation, etc.), addition (e.g., pixel addition, etc.), inversion (e.g., left-right inversion, up-down inversion, etc.), and shearing may be performed. These conversions may also be performed randomly. Data augmentation may be performed to artificially increase the number of samples. Data augmentation is also effective in generalizing a model so that it can be applied even when it is not included in the original data set.
[0046] As an example of data pre-processing according to the embodiment, for example, a predetermined conversion process may be performed on the shooting information. For example, the face size may be converted to a predetermined pixel size (for example, 36×36 pixels, etc.). In addition, for example, optical flow features may be extracted and normalized for model learning. In addition, as an example of data pre-processing according to the embodiment, for example, a predetermined conversion process may be performed on information from a biosensor such as a PPG sensor. For example, noise removal may be performed by filtering.
[0047] (Estimation process) The estimation process of the information processing device 100 will be described below. Note that the description of the same as the learning process will be omitted as appropriate. The information processing device 100 acquires shooting information of the subject UX (step S201). For example, the information processing device 100 acquires shooting information of the subject UX in real time.
[0048] The information processing device 100 extracts a face image as a skin image including at least a part of the skin of the subject UX from the acquired photographing information (step S202). At this time, the information processing device 100 may use a first model such as a single-shot multi-box detector. Then, the information processing device 100 applies the extracted face image to a second model, a convolutional attention neural network, which is a learned second model, to generate information (biologically suggestive information) for estimating the stress state of the subject UX (step S203). In this way, the information processing device 100 generates information (biologically suggestive information) for estimating the stress state of the subject UX by applying the information to a learned model learned by the learning process. Then, the information processing device 100 estimates the stress state of the subject UX from the generated information (step S204). At this time, the information processing device 100 may use a third model such as a residual convolutional neural network.
[0049] Here, the learning process according to the embodiment will be further described with reference to FIG. 4. FIG. 4 is an explanatory diagram for explaining the learning process according to the embodiment. Specifically, by gradually learning additional data that gradually adds information from essential data, it becomes possible to generate a model with high final accuracy even if the size is small. For example, the information processing device 100 generates only parts where the skin is exposed, such as the face (that is, only parts where color changes according to blood flow can be expected), as learning data, and generates image groups in which additional information, such as body parts and background parts, is gradually increased. Then, data groups such as the first data group, the second data group, and the third data group are generated in order of decreasing amount of additional information, and learning is performed in order of decreasing amount of additional information. After learning with the first data group, when the error becomes equal to or less than a predetermined threshold, learning is performed with the second data group. By doing this, the model does not suddenly perform processing based on the characteristics of the clothing and the characteristics of the background, but performs basic learning from the characteristics of the parts that are basically desired to be the basis of the estimation process, and gradually increases the amount of additional information, thereby making it possible to perform learning such as estimating exceptional conditions. As a result, model learning is performed by optimizing accuracy based on fundamental information, and gradually optimizing accuracy further by considering additional information, so that accuracy is not optimized based on exceptions, but is optimized based on fundamentals while considering exceptions, which makes it possible to improve accuracy. As a result, accuracy can be guaranteed even for models with small model sizes, and real-time performance can be guaranteed.For example, in the first data group, learning is performed using a black background to eliminate the features of the background, in the second data group, learning is performed using a background with black and other colors added to the features of the background, in the third data group, learning is performed to generate samples without faces using the features of the background parts up to the second data group and sections of the frame where no face is shown (for example, in the third data group, if there are parts in the frame where a face is shown and parts where a face is not shown, the frame parts where a face is not shown may be trained to be "no face present"), and in the fourth data group, learning is performed using the features up to the third data group and fake face frames obtained from magazines, newspapers, external screens, etc. In this way, a data group containing only the principles and multiple data groups with additional information gradually increased are prepared, and learning is performed for each data group in order of the amount of additional information. In this way, instead of learning from exceptions, by learning the principles and gradually increasing the additional information, it is possible to improve robustness against errors due to additional information.
[0050] 3. Configuration of Information Display Device Next, the configuration of the information display device 10 according to the embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the configuration of the information display device 10 according to the embodiment. As shown in Fig. 5, the information display device 10 has a communication unit 11, an input unit 12, an output unit 13, and a control unit 14.
[0051] (Communications Department 11) The communication unit 11 is realized by, for example, a network interface card (NIC) etc. The communication unit 11 is connected to a predetermined network N by wire or wirelessly, and transmits and receives information to and from the information processing device 100 etc. via the predetermined network N.
[0052] (Input section 12) The input unit 12 accepts various operations from the subject. In the example shown in Fig. 2, various operations are accepted from the subject U (subjects U1 to UN and UX). For example, the input unit 12 may accept various operations from the subject via a display surface using a touch panel function. The input unit 12 may also accept various operations from buttons provided on the information display device 10 or a keyboard or mouse connected to the information display device 10.
[0053] (Output section 13) The output unit 13 is a display screen of a tablet terminal or the like realized by, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and is a display device for displaying various information. For example, the output unit 13 displays information transmitted from the information processing device 100.
[0054] (Control unit 14) The control unit 14 is, for example, a controller, and is realized by executing various programs stored in a storage device inside the information display device 10 using a RAM (Random Access Memory) as a working area by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). For example, the various programs include application programs installed in the information display device 10. For example, the various programs include an application program for photographing a subject in real time. The control unit 14 is also realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).
[0055] As shown in FIG. 5, the control unit 14 has an image capturing unit 141, and realizes or executes the information processing operations described below.
[0056] (Camera Division 141) The photographing unit 141 photographs in real time. For example, the photographing unit 141 photographs a subject in real time. In addition, the photographing unit 141 transmits photographed photographing information to an external information processing device in real time. For example, the photographing unit 141 transmits photographed photographing information to another information processing device such as the information processing device 100 in real time.
[0057] 4. Configuration of Information Processing Device Next, the configuration of the information processing device 100 according to the embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Fig. 6, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Note that the information processing device 100 may have an input unit (e.g., a keyboard, a mouse, etc.) that accepts various operations from an administrator of the information processing device 100, and a display unit (e.g., a liquid crystal display, etc.) that displays various information.
[0058] (Communication unit 110) The communication unit 110 is realized by, for example, a NIC etc. The communication unit 110 is connected to a network N by wire or wirelessly, and transmits and receives information to and from the information display device 10 etc. via the network N.
[0059] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 6, the storage unit 120 includes a sensor information storage unit 121 and a model information storage unit 122.
[0060] The sensor information storage unit 121 stores information of a biosensor such as a PPG sensor. For example, the sensor information storage unit 121 stores information of a biosensor attached to a subject for model learning. Here, FIG. 7 shows an example of the sensor information storage unit 121 according to the embodiment. As shown in FIG. 7, the sensor information storage unit 121 has items such as "sensor information ID," "subject ID," "learning environment," and "sensor information."
[0061] The "sensor information ID" indicates identification information for identifying the sensor information. The "subject ID" indicates identification information for identifying the subject. The "learning environment" indicates the learning environment. In the example shown in FIG. 7, an example was shown in which conceptual information such as "learning environment #1" and "learning environment #2" was stored in the "learning environment", but in reality, a numerical value indicating a sampling rate, a numerical value indicating a resolution, a numerical value indicating a frame rate, a numerical value indicating a distance from the subject, a numerical value indicating a shooting angle, a numerical value indicating a shooting illuminance, information indicating the subject's attributes, information indicating whether the subject is wearing a mask, information indicating the color of the background, information indicating the presence or absence of stress, and the like are stored. The "sensor information" indicates information of a biosensor. In the example shown in FIG. 7, an example was shown in which conceptual information such as "sensor information #1" and "sensor information #2" was stored in the "sensor information", but in reality, raw data such as heart rate data measured by a PPG sensor and breathing data measured by a breathing sensor are stored. In addition, data after data expansion or data preprocessing may be stored.
[0062] The model information storage unit 122 stores information about a model. For example, the model information storage unit 122 stores information about a model that has been trained to approximate information from a biosensor. Here, FIG. 8 shows an example of the model information storage unit 122 according to the embodiment. As shown in FIG. 8, the model information storage unit 122 has items such as "model ID," "model type," "model information," and "training data."
[0063] "Model ID" indicates identification information for identifying a model. "Model type" indicates a type for identifying whether it is a first model, a second model, or a third model. "Model information" indicates information about a model. In the example shown in FIG. 8, an example is shown in which conceptual information such as "model information #1" and "model information #2" is stored in "model information", but in reality, information indicating the elements and weights of a model is stored. "Learning data" indicates learning data used for learning a model. In the example shown in FIG. 8, an example is shown in which conceptual information such as "learning data #1" and "learning data #2" is stored in "learning data", but in reality, shooting information used for learning a model is stored. In addition, shooting information after data expansion or data preprocessing may be stored.
[0064] (Control unit 130) The control unit 130 is a controller, and is realized, for example, by a CPU, an MPU, or the like, executing various programs stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 130 is also realized, for example, by an integrated circuit such as an ASIC or an FPGA.
[0065] 6, the control unit 130 has an acquisition unit 131, an extraction unit 132, a generation unit 133, an estimation unit 134, and a learning unit 135, and realizes or executes the information processing action described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 6, and may be other configurations as long as they perform the information processing described below.
[0066] (Acquisition part 131) The acquisition unit 131 acquires various information. The acquisition unit 131 acquires various information from an external information processing device. For example, the acquisition unit 131 acquires various information from another information processing device such as the information display device 10.
[0067] The acquisition unit 131 acquires various pieces of information from the storage unit 120. For example, the acquisition unit 131 acquires various pieces of information from the sensor information storage unit 121 or the model information storage unit 122. In addition, the acquisition unit 131 stores the acquired various pieces of information in the storage unit 120. For example, the acquisition unit 131 stores the various pieces of information in the sensor information storage unit 121 or the model information storage unit 122.
[0068] The acquisition unit 131 acquires imaging information obtained by imaging a subject in real time. For example, the acquisition unit 131 acquires imaging information transmitted in real time from an external information processing device. For example, the acquisition unit 131 acquires imaging information transmitted in real time from another information processing device such as the information display device 10.
[0069] (Extraction part 132) The extraction unit 132 extracts a skin image (e.g., a face image, etc.) including at least a part of the skin of the subject from the photographing information acquired by the acquisition unit 131. For example, the extraction unit 132 extracts the skin image using a first model (e.g., a single-shot multi-box detector, etc.). At this time, the extraction unit 132 may extract the skin image using the first model, which is an existing model, or may extract the skin image using a first model generated by the learning unit 135 described later, or may extract the skin image using a first model after learning in which the existing first model is trained by the learning unit 135 described later.
[0070] Here, the first model according to the embodiment will be described in detail with reference to FIG. 9. Note that the description will be given taking a single-shot multi-box detector as an example. FIG. 9 is a diagram showing an example of the architecture of the first model. The first model may be an architecture based on a slicing method in which the imaging information is divided into regions of a predetermined size and detection is performed for each divided region, or an architecture based on a grid cell method in which the imaging information is divided in advance into a predetermined number of regions and detection is performed for each divided region.
[0071] In FIG. 9, IN1 is shooting information, which is input information input to A1, which is the architecture of the first model. OU1 is a skin image, which is output information output from A1. The first model may include multiple CNNs in the architecture. For example, the first model may include at least each of a full-layer convolutional neural network based on MobileNet-v2 and a residual convolutional neural network.
[0072] A learning method of the first model will be described. As an example of the learning method of the first model, a loss function such as Mean Squared Error may be used. Also, an optimization method such as ADAM (Lr=0.001) may be used. Also, for example, padding may be set to "enabled" or the like. Also, for example, the number of epochs may be set to "32" or the like. Also, for example, the batch size may be set to "32" or the like. The first model may be learned by correcting the connection coefficient between nodes by backpropagation or the like so as to extract areas where the skin is exposed, such as a person's face or hand, that is, areas having optical characteristics that allow blood flow to be estimated, when images constituting each frame of a video are input.
[0073] In the first model, for example, MobileNet-v2 may be trained using the ImageNet dataset. For example, the weights obtained from MobileNet-v2 may be fixed and trained. For example, the residual convolutional neural network may be trained using half of a predetermined dataset (for example, a public dataset such as WIDER that includes a face bounding box). For example, the weights may be released from the fix and the learning rate may be reduced to about 1 / 100,000. For example, further training may be performed using a predetermined dataset (for example, a public dataset such as WFLW that includes a face bounding box and 98 landmarks) to improve robustness when the subject wears a mask. For example, training may be performed using stride 2 without using a pooling layer.
[0074] (Generation unit 133) The generating unit 133 generates a second model that is a model with high accuracy even if it is small in size by gradually learning additional data that gradually adds information. Specifically, the generating unit 133 prepares a data group including only the principles and a plurality of data groups in which the additional information is gradually increased, and performs learning for each data group in order of the amount of additional information to generate the second model. For example, the generating unit 133 prepares (1) a data group for learning only with a face, (2) a data group for learning only with a face and a body, (3) a data group for learning with a face, a body, and a background (black and white), and (4) a data group for learning with a face, a body, and a background (color), and performs learning for each data group in order of the amount of additional information (1), (2), (3), and (4) to generate the second model. In addition, for example, the generating unit 133 similarly prepares a data group for learning with a fake face and body, and performs learning for each data group to generate the second model. For example, the generating unit 133 prepares a data group for learning with the subject's face and the subject's body, a data group for learning with a fake face and the subject's body, and a data group for learning with a fake face and a fake body, and generates a second model by performing learning for each data group. In addition, when the error becomes equal to or less than a predetermined threshold (when the loss convergence is reached again), the generating unit 133 performs learning with a data group with the next least additional information, and by repeating this, generates a second model by performing learning to estimate an exceptional condition. In this way, the generating unit 133 performs basic learning from the features of the part that is fundamentally desired to be the basis of the estimation process, and when the error becomes equal to or less than a predetermined threshold, performs learning with a data group with the next least additional information, and by gradually increasing the amount of additional information, generates a second model by performing learning to estimate an exceptional condition.
[0075] The generating unit 133 generates bio-suggested information of the subject (e.g., heart rate data, respiratory data, etc.) from the skin image extracted by the extracting unit 132. For example, the generating unit 133 generates the bio-suggested information of the subject using a second model (e.g., a convolutional attention neural network, etc.). In this case, the generating unit 133 may generate the bio-suggested information of the subject using, for example, the second model which is an existing model, may generate the bio-suggested information of the subject using a second model generated by the learning unit 135 described later, or may generate the bio-suggested information of the subject using a second model after learning in which the existing second model is learned by the learning unit 135 described later.
[0076] Here, the second model according to the embodiment will be described in detail with reference to Fig. 10. Note that the description will be given taking a convolutional attention neural network as an example. Fig. 10 is a diagram showing an example of the architecture of the second model.
[0077] In FIG. 10, IN2 is a skin image, which is input information input to A2, which is the architecture of the second model. OU2 is bio-indicative information of the subject, which is output information output from A2. OU2 may be detrended for signal balancing. IN21 is information generated by extracting optical flow features of IN2, and IN22 is information generated by normalizing IN2 for use in an attention mask. That is, A2 performs convolution by using IN22 as an attention mask and applying it to IN21. A2 includes a first component that extracts features based on a video image, and a second component that generates information for estimation based on optical flow features based on a video image.
[0078] A learning method of the second model will be described. As an example of the learning method of the second model, for example, a loss function such as mean square error may be used. For example, an optimization method such as Adaptive-Delta (Lr=1.0 to 0.1 to 0.01 to 0.001) may be used. For example, padding may be set to "enabled" or the like. For example, stride may be set to "1" or the like. For example, the number of epochs may be set to "370" or the like. For example, batch size may be set to "128" or the like. For example, batch size may be set to a multiple of frame depth (parameter set at the time of designing A2). For example, each batch may be set as an independent signal. The second model may be learned by correcting the connection coefficient between nodes by backpropagation or the like so as to extract bio-suggestive information based on the optical characteristics of an exposed part of the skin when a skin image including at least the exposed part of the skin is input.
[0079] In the second model, for example, learning may be performed on a predetermined data set (for example, a public data set such as PURE), and then learning may be performed on a data set from which unnecessary biological data has been removed (for example, a private data set). For example, when learning is performed using heart rate data, learning may be performed on a data set from which respiration data has been removed.
[0080] In the second model, for example, learning may be performed with a randomly set weight. For example, learning may be performed with a high learning rate of "1.0". For example, learning may be performed initially with only face video images. In this way, learning may be performed initially with only basic information. For example, a part of the subject's data may be used for learning. For example, about 20% of the data may be used for evaluation, and the remaining about 80% of the data may be used for learning. For example, after a predetermined number of repetitions, learning may be performed with a lowered learning rate. For example, learning may be performed with a tenth of the learning rate. For example, learning may be performed initially with only face video images, but then learning may be performed including background images. In this way, learning may be performed while gradually increasing information from basic information. For example, after loss convergence is reached (after the decrease in evaluation loss stops), learning may be performed until loss convergence is reached again. For example, learning may be performed with a fake face image. For example, after loss convergence is reached, learning may be performed with a lowered learning rate. For example, learning may be performed while fine-tuning the model by repeating the process of lowering the learning rate. Additionally, training may be performed by applying dedicated fine-tuning, for example, to adapt the model to a particular environment.
[0081] In addition, the second model may have a blood flow detectable area extraction unit in Figure 3 that extracts an area in which blood flow is detectable from the extracted video image, and an optical flow processing unit that processes the optical flow characteristics of the extracted video image.
[0082] (Estimation part 134) The estimation unit 134 estimates the state of the subject (e.g., a stress state, etc.) from the information generated by the generation unit 133. For example, the estimation unit 134 estimates the state of the subject using a third model (e.g., a residual convolutional neural network, etc.). In this case, the estimation unit 134 may estimate the state of the subject using, for example, the third model which is an existing model, or may estimate the state of the subject using a third model generated by the learning unit 135 described later, or may estimate the state of the subject using a third model after learning in which the existing third model is learned by the learning unit 135 described later.
[0083] Here, the third model according to the embodiment will be described in detail with reference to Fig. 11. Note that the explanation will be given by taking a residual convolutional neural network as an example. Fig. 11 is a diagram showing an example of the architecture of the third model.
[0084] 11, IN3 is information generated by the generating unit 133, and is input information input to A3, which is the architecture of the third model. OU3 is information indicating the state of the subject, and is output information output from A3. IN3 is, for example, information with a sampling rate of 128 Hz, a signal length of 10 seconds, and a slice of 2 seconds.
[0085] A learning method of the third model will be described. As an example of the learning method of the third model, for example, a loss function such as categorical cross entropy or binary cross entropy may be used. For example, an optimization method such as ADAM may be used. For example, padding may be set to "enabled" or the like. For example, the number of epochs may be set to "100" or the like. For example, the batch size may be set to "64" or the like. The third model may be learned by correcting the connection coefficient between nodes by backpropagation or the like so as to extract information indicating the subject's state estimated based on the optical characteristics of an exposed part of the skin when the biosuggestive information based on the optical characteristics of the part is input.
[0086] In the third model, for example, after training is performed on a given dataset (e.g., a public dataset such as WESAD), further training may be performed on a different dataset (e.g., a private dataset).
[0087] (Learning Section 135) The learning unit 135 learns the model. For example, the learning unit 135 learns at least one of a first model, a second model, and a third model. For example, the learning unit 135 trains the model such that when the learning data, which is information of a subject, is input to the model as correct answer data, information output from the model approaches actual biosensor information.
[0088] The learning unit 135 learns the first model using, for example, a predetermined data set (for example, WIDER or WFLW). The first model may be an existing model, a model generated by the learning unit 135, or a learned model obtained by training an existing model using the predetermined data set.
[0089] The learning unit 135 learns the second model using, for example, a predetermined data set (for example, PURE, etc.). The second model may be an existing model, may be a model generated by the learning unit 135, or may be a learned model obtained by learning an existing model using the predetermined data set.
[0090] The learning unit 135 learns the third model using, for example, a predetermined data set (for example, WESAD, etc.). The third model may be an existing model, a model generated by the learning unit 135, or a learned model obtained by training an existing model using the predetermined data set.
[0091] The learning unit 135 performs model learning using, for example, information (corresponding to the first learning information) including at least first information whose importance in estimation satisfies a predetermined condition, among information for making the model learn features. Then, the learning unit 135 performs further model learning using, for example, information (corresponding to the second learning information) including predetermined second information in addition to the first information.
[0092] The learning unit 135 performs learning using the first learning information, for example, until a difference between the first learning information and output information (corresponding to the first output information) output by the model when the first learning information is input to the model satisfies a predetermined condition. Also, the learning unit 135 performs learning of the model that has been trained using the first learning information, for example, until a difference between the second learning information and output information (corresponding to the second output information) output by the model when the second learning information is input to the model satisfies a predetermined condition.
[0093] The learning unit 135 performs learning using the first learning information until, for example, a loss of the first output information for the first learning information converges to a predetermined threshold or less. Also, the learning unit 135 performs learning using the second learning information until, for example, a loss of the second output information for the second learning information converges to a predetermined threshold or less.
[0094] The learning unit 135 causes the model to learn the features of the second learning information at a learning rate that is lower than the learning rate using the first learning information, for example.
[0095] [5. Information processing flow] Next, the procedure of information processing by the information processing system 1 according to the embodiment will be described with reference to Figures 12 and 13. Figures 12 and 13 are flowcharts showing the procedure of information processing by the information processing system 1 according to the embodiment.
[0096] As shown in FIG. 12, the information processing device 100 acquires imaging information obtained by imaging a target person in real time (step S301).
[0097] The information processing device 100 uses the first model to extract a skin image including at least a part of the skin of the subject (step S302).
[0098] The information processing device 100 generates information for estimating the state of the subject by using the second model (Step S303).
[0099] The information processing device 100 estimates the state of the subject by using the third model (step S304).
[0100] As shown in FIG. 13, the information processing device 100 acquires first learning information including at least first information and second learning information including predetermined second information in addition to the first information (Step S401).
[0101] The information processing device 100 uses the first learning information to learn a second model, and after learning using the first learning information, further learns the second model using the second learning information (step S402).
[0102] 6. Modifications The information processing system 1 according to the embodiment described above may be implemented in various different forms other than the above embodiment, so other embodiments of the information processing system 1 will be described below.
[0103] In the above embodiment, the information processing device 100 acquires shooting information of a subject photographed in real time by the information display device 10, but the present invention is not limited to this example. For example, the information processing device 100 may acquire pre-recorded recording information from an external information processing device. For example, the information processing device 100 may estimate the state of a subject appearing in a pre-recorded video. This makes it possible to visualize the state of a performer in a movie or drama at the time of shooting by applying the information processing device 100 to a pre-recorded video, for example.
[0104] 7. Effects As described above, the information processing device 100 according to the embodiment includes the acquisition unit 131 and the learning unit 135. The acquisition unit 131 acquires first learning information including at least first information whose importance in estimation satisfies a predetermined condition among information for causing an estimator that performs estimation based on input information to learn features, and second learning information including predetermined second information in addition to the first information. The learning unit 135 uses the first learning information to train the estimator, and after training using the first learning information, uses the second learning information to train the estimator that has been trained using the first learning information.
[0105] This allows the information processing device 100 according to the embodiment to appropriately perform estimation in real time when making estimation from various pieces of information.
[0106] In addition, the learning unit 135 performs learning using the first learning information until a difference between the first learning information and first output information output by the estimator when the first learning information is input to the estimator satisfies a predetermined condition, and performs learning of the estimator that has been trained using the first learning information until a difference between the second learning information and second output information output by the estimator when the second learning information is input to the estimator satisfies a predetermined condition.
[0107] As a result, the information processing device 100 according to the embodiment can perform learning by gradually increasing information starting from basic information, thereby enabling highly accurate estimation with a small model.
[0108] In addition, the learning unit 135 performs learning using the first learning information until the loss of the first output information for the first learning information converges to a predetermined threshold or less, and performs learning using the second learning information until the loss of the second output information for the second learning information converges to a predetermined threshold or less.
[0109] Thereby, the information processing device 100 according to the embodiment can enable more accurate learning by setting conditions related to learning when gradually increasing information.
[0110] Furthermore, the learning unit 135 causes the estimator to learn the features of the second learning information at a learning rate lower than that of the learning using the first learning information.
[0111] As a result, the information processing device 100 according to the embodiment can achieve highly accurate learning while reducing the load of the learning process.
[0112] In addition, the acquisition unit 131 acquires, as the first learning information, information including information about a person whose state is to be estimated using an estimator as the first information, and acquires, as the second learning information, information including information other than the person as the second information.
[0113] This enables the information processing device 100 according to the embodiment to be applied to estimating a state of a person.
[0114] In addition, the acquisition unit 131 acquires, as the first learning information, information including a predetermined type of information among the information of a person whose state is to be estimated using the estimator as the first information, and acquires, as the second learning information, information including a type of information of the person other than the predetermined type as the second information.
[0115] As a result, when the information processing apparatus 100 according to the embodiment is applied to estimating a state of a person, it is possible to learn a model using any information of a person as basic information.
[0116] In addition, the acquisition unit 131 acquires, as the first learning information, information including skin information of a person whose state is to be estimated using the estimator as the first information, and acquires, as the second learning information, information including information of the person other than skin as the second information.
[0117] As a result, the information processing apparatus 100 according to the embodiment can enable model learning using skin information of a person as basic information when applied to estimating a state of a person.
[0118] Moreover, the acquiring unit 131 acquires images constituting the moving image as the first learning information and the second learning information. Moreover, the learning unit 135, as an estimator, learns a neural network that estimates a state of a subject captured in the moving image based on the images constituting the moving image.
[0119] This enables the information processing device 100 according to the embodiment to estimate the state of the subject in real time from a moving image.
[0120] In addition, the learning unit 135 learns a neural network having a first component part which is a neural network that extracts features based on a moving image input as a neural network, and a second component part which generates information for estimation based on optical flow features based on the moving image and the features extracted by the first component part.
[0121] This enables the information processing device 100 according to the embodiment to perform learning based on the characteristics of the optical flow, thereby enabling efficient learning.
[0122] Furthermore, the learning unit 135 performs learning of a neural network that estimates the state of a subject captured in a video based on images that have been subjected to data expansion or data preprocessing for images that constitute the video.
[0123] As a result, the information processing device 100 according to the embodiment can perform more accurate learning by performing data augmentation or data preprocessing.
[0124] [8. Hardware Configuration] Moreover, the information display device 10 and the information processing device 100 according to the above-described embodiment are realized, for example, by a computer 900 having a configuration as shown in Fig. 14. Fig. 14 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information display device 10 and the information processing device 100. The computer 900 has a CPU 910, a RAM 920, a ROM 930, a HDD 940, a communication interface (I / F) 950, an input / output interface (I / F) 960, and a media interface (I / F) 970.
[0125] The CPU 910 operates and controls each unit based on a program stored in the ROM 930 or the HDD 940. The ROM 930 stores a boot program executed by the CPU 910 when the computer 900 is started up, programs that depend on the hardware of the computer 900, and the like.
[0126] The HDD 940 stores programs executed by the CPU 910, data used by such programs, etc. The communication interface 950 acquires data from other devices via a predetermined communication network and sends it to the CPU 910, and transmits data generated by the CPU 910 to other devices via the predetermined communication network.
[0127] The CPU 910 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via an input / output interface 960. The CPU 910 acquires data from the input devices via the input / output interface 960. The CPU 910 also outputs generated data to the output devices via the input / output interface 960.
[0128] The media interface 970 reads a program or data stored in the recording medium 980 and provides it to the CPU 910 via the RAM 920. The CPU 910 loads the program from the recording medium 980 onto the RAM 920 via the media interface 970, and executes the loaded program. The recording medium 980 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.
[0129] For example, when the computer 900 functions as the information display device 10 and the information processing device 100 according to the embodiment, the CPU 910 of the computer 900 executes programs loaded onto the RAM 920 to realize the functions of the control units 14 and 130. The CPU 910 of the computer 900 reads and executes these programs from the recording medium 980, but as another example, the CPU 910 may obtain these programs from other devices via a predetermined communication network.
[0130] [9. Others] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.
[0131] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.
[0132] Furthermore, the above-described embodiments can be appropriately combined as long as the processing contents are not contradictory.
[0133] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the Disclosure of the Invention section.
[0134] Furthermore, the above-mentioned "section, module, unit" can be read as "means" or "circuit", etc. For example, an acquisition section can be read as an acquisition means or an acquisition circuit. [Explanation of symbols]
[0135] 1. Information Processing Systems 10 Information display device 11 Communications Department 12 Input section 13 Output section 14 Control section 100 Information processing device 110 Communications Department 120 Storage section 121 Sensor information storage unit 122 Model information storage unit 130 Control section 131 Acquisition Department 132 Extraction part 133 Generation part 134 Estimation Department 135 Learning Department 141 Photography Department N Network
Claims
1. an acquisition unit that acquires first learning information including at least first information that is information on an exposed skin area and that is essential data on which the estimation is based, and that is of the highest importance in the estimation based on whether or not a color change according to blood flow can be expected, among information included in the input learning information and that allows an estimator that performs estimation based on the information to learn features, and second learning information including, in addition to the first information, predetermined second information that is information that does not include an exposed skin area, unlike the essential data on which the estimation is based; a learning unit that learns the estimator using the first learning information and output information output by the estimator when the first learning information is input, and after learning using the first learning information, learns the estimator that has been trained using the first learning information using the second learning information and output information output by the estimator that has been trained using the first learning information when the second learning information is input; 13. An information processing device comprising:
2. The acquisition unit is When comparing a data group including only the essential data with a plurality of data groups in which additional information is gradually increased, the second learning information is obtained, which is a data group selected as a data group having more additional information than the first learning information.
2. The information processing apparatus according to claim 1,
3. The learning unit is performing learning using the first learning information until a difference between the first learning information and first output information output by the estimator when the first learning information is input to the estimator satisfies a predetermined condition; The estimator that has been trained using the first learning information is trained until a difference between the second learning information and second output information output by the estimator when the second learning information is input to the estimator satisfies a predetermined condition.
2. The information processing apparatus according to claim 1,
4. The learning unit is performing learning using the first learning information until a loss of the first output information with respect to the first learning information converges to a predetermined threshold or less; Learning is performed using the second learning information until a loss of the second output information with respect to the second learning information converges to a predetermined threshold or less.
4. The information processing apparatus according to claim 3.
5. The learning unit is The feature of the second learning information is learned by the estimator at a learning rate lower than that of the learning using the first learning information.
5. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises:
6. The acquisition unit is As the first learning information, information including information of a person whose state is to be estimated using the estimator is acquired as first information, and as the second learning information, information including information other than the person is acquired as second information.
5. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises:
7. The acquisition unit is As the first learning information, information including a predetermined type of information among information of a person whose state is to be estimated using the estimator is acquired as first information, and as the second learning information, information including a type of information of the person other than the predetermined type is acquired as second information.
5. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises:
8. The acquisition unit is acquiring images constituting a moving image as the first learning information and the second learning information; The learning unit is As the estimator, a neural network is trained to estimate the state of a subject captured in the video based on images constituting the video.
5. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises:
9. The learning unit is As the neural network a first component which is a neural network for extracting features based on an input video; a second component for generating information for estimation based on optical flow features based on the video and the features extracted by the first component; We train a neural network with 9. The information processing apparatus according to claim 8,
10. The learning unit is A neural network is trained to estimate the state of a subject captured in the video based on images obtained after data expansion or data preprocessing of the images constituting the video.
9. The information processing apparatus according to claim 8,
11. 1. A computer-implemented information processing method, comprising: an acquisition step of acquiring first learning information including at least first information which is information on an exposed skin area and has the highest importance in the estimation based on whether a color change according to blood flow can be expected, among information included in the input learning information and which allows an estimator that performs an estimation based on the information to learn features, and second learning information including second information which is predetermined second information in addition to the first information and which is information that does not include an exposed skin area, unlike the essential data that is the basis of the estimation; a learning step of learning the estimator using the first learning information and output information output by the estimator when the first learning information is input, and after learning using the first learning information, learning the estimator that has been trained using the first learning information using the second learning information and output information output by the estimator that has been trained using the first learning information when the second learning information is input; 13. An information processing method comprising:
12. an acquisition step of acquiring first learning information including at least first information which is information on an exposed skin area and has the highest importance in the estimation based on whether a color change according to blood flow can be expected, among information included in the input learning information and which allows an estimator that performs an estimation based on the information to learn features, and second learning information including second information which is predetermined second information in addition to the first information and which is information that does not include an exposed skin area, unlike the essential data that is the basis of the estimation; a learning procedure of learning the estimator using the first learning information and output information output by the estimator when the first learning information is input, and after learning using the first learning information, learning the estimator that has been trained using the first learning information using the second learning information and output information output by the estimator that has been trained using the first learning information when the second learning information is input; An information processing program characterized by causing a computer to execute the above.
Citation Information
Patent Citations
Action Recognition in Video Using 3D Spatiotemporal Convolutional Neural Networks
JP2020519995A
Information processing apparatus and learning method
JP2023131287A
Information processing device and information processing method
WO2020027073A1
Method for processing image for stress-monitoring and program of the same
JP2017029318A
