State estimation system, state estimation method, and state estimation program

Abstract

According to the present invention, a feature expression model creation unit (180) creates a feature amount calculation model by self-distillation using an electrocardiogram feature expression model and a cardiac activity feature expression model. A state estimation model creation unit (130) calculates a feature amount for learning using the feature amount calculation model, and creates a state estimation model by learning the feature amount for learning. A state estimation unit (140) extracts cardiac activity information as target activity information from target sensor data obtained by sensing a target animal, calculates a feature amount of the target activity information as a feature amount for inference through the feature amount calculation model by using the target activity information as input, and estimates the state of the target animal through the state estimation model by using the feature amount for inference as input.

Description

State estimation system, state estimation method, and state estimation program

[0001] The present disclosure relates to techniques for estimating the state of a human or non-human animal.

[0002] One method for realizing state estimation using cardiac activity information is to use a machine learning approach. However, training a model requires a large amount of cardiac activity information. Furthermore, collecting cardiac activity information anew using sensors is time-consuming.

[0003] By utilizing cardiac activity information obtained from electrocardiograms, it is possible to reduce the effort required to collect data using sensors. Furthermore, electrocardiograms are already being measured in large quantities in hospitals and other facilities, making them easy to collect.

[0004] Patent Literature 1 discloses a technology for constructing a model capable of estimating heart rate variability from heart rates measured by a wearable device. This technology constructs a model capable of inferring heart rate variability indexes from heart rates measured by a wearable device using heart rate variability indexes obtained from an electrocardiogram. The technology of Patent Literature 1 only uses heart rate variability indexes obtained from an electrocardiogram.

[0005] Japanese Patent Application Laid-Open No. 2022-98608

[0006] The present disclosure aims to provide a technology for estimating the state of an animal using cardiac activity information obtained from an electrocardiogram or the like, whereby a model can be created by utilizing as much information contained in the electrocardiogram as possible.

[0007] The state estimation system of the present disclosure comprises an electrocardiogram learning unit that learns electrocardiogram data for feature learning and creates an electrocardiogram feature representation model; a cardiac activity learning unit that learns learning information based on the electrocardiogram data for feature learning and creates a cardiac activity feature representation model; a self-distillation unit that creates a feature calculation model by self-distillation using the electrocardiogram feature representation model and the cardiac activity feature representation model; a state estimation model creation unit that takes input information based on the electrocardiogram data for state learning as input, uses the feature calculation model to calculate learning features corresponding to the features of the cardiac activity information extracted from the electrocardiogram data for state learning, and learns the learning features to create a state estimation model; and a state estimation unit that extracts cardiac activity information as target activity information from target sensor data obtained by sensing a target animal, uses the target activity information as input, calculates the features of the target activity information as inference features using the feature calculation model, and estimates the state of the target animal using the state estimation model with the inference features as input.

[0008] According to the present disclosure, in a technique for estimating the state of an animal using cardiac activity information obtained from an electrocardiogram or the like, it is possible to create a model by utilizing as much information contained in the electrocardiogram as possible.

[0009] 1 is a configuration diagram of a state estimation system 100 according to the first embodiment. 2 is a configuration diagram of a feature representation model creation unit 180 according to the first embodiment. 3 is a configuration diagram of a state estimation model creation unit 130 according to the first embodiment. 4 is a configuration diagram of a state estimation unit 140 according to the first embodiment. 5 is a flowchart of a state estimation method according to the first embodiment. 6 is a flowchart of step S110 according to the first embodiment. 7 is a diagram showing types of cardiac activity information according to the first embodiment. 8 is a flowchart of step S120 according to the first embodiment. 9 is a flowchart of step S130 according to the first embodiment. 10 is a flowchart of step S140 according to the first embodiment. 11 is a schematic diagram of a state estimation method according to the first embodiment. 12 is a schematic diagram of a state estimation method according to the first embodiment. 13 is a schematic diagram of a state estimation system 100 according to the second embodiment. 14 is a configuration diagram of a domain transformation model creation unit 150 according to the second embodiment. 15 is a configuration diagram of a feature representation model creation unit 180 according to the second embodiment. 16 is a configuration diagram of a state estimation model creation unit 130 according to the second embodiment. 17 is a flowchart of a state estimation method according to the second embodiment. 18 is a flowchart of a feature representation model creation unit 130 according to the second embodiment. 19 is a flowchart of a state estimation method according to the second embodiment. 19 is a flowchart of step S210 according to the second embodiment. 19 is a flowchart of step S220 according to the second embodiment. Flowchart of step S230 in embodiment 2. Flowchart of step S240 in embodiment 2. Configuration diagram of state estimation system 100 in embodiment 3. Configuration diagram of feature representation model creation unit 180 in embodiment 3. Flowchart of a state estimation method in embodiment 3. Flowchart of step S330 in embodiment 3. Configuration diagram of feature representation model creation unit 180 in embodiment 4. Configuration diagram of state estimation model creation unit 130 in embodiment 4. Flowchart of a state estimation method in embodiment 4. Flowchart of step S410 in embodiment 4. Flowchart of step S420 in embodiment 4. Flowchart of step S430 in embodiment 4. Hardware configuration diagram of state estimation system 100 in the embodiments.

[0010] In the embodiments and drawings, the same or corresponding elements are denoted by the same reference numerals. The description of elements denoted by the same reference numerals as those already described will be omitted or simplified as appropriate. Arrows in the drawings primarily indicate the flow of data or the flow of processing.

[0011] Embodiment 1 An embodiment for estimating the state of a human or non-human animal will be described with reference to Figs.

[0012] ***Description of Configuration*** The configuration of the state estimation system 100 will be described with reference to Fig. 1. The state estimation system 100 is a computer including hardware such as a processor 101, a memory 102, an auxiliary storage device 103, and an input / output interface 104. These pieces of hardware are connected to one another via signal lines.

[0013] The processor 101 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 101 is a CPU, a DSP, or a GPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

[0014] The memory 102 is a volatile or non-volatile storage device. The memory 102 is also called a primary storage device or a main memory. For example, the memory 102 is a RAM. Data stored in the memory 102 is saved in the secondary storage device 103 as needed. RAM is an abbreviation for Random Access Memory.

[0015] The auxiliary storage device 103 is a non-volatile storage device. For example, the auxiliary storage device 103 is a ROM, a HDD, a flash memory, or a combination of these. Data stored in the auxiliary storage device 103 is loaded into the memory 102 as needed. ROM is an abbreviation for Read Only Memory. HDD is an abbreviation for Hard Disk Drive.

[0016] The input / output interface 104 is a port to which an input device, an output device, and a communication device are connected. For example, the input / output interface 104 is a USB terminal, the input devices are a keyboard and a mouse, and the output device is a display. Input and output to and from the state estimation system 100 are performed using the input / output interface 104. USB is an abbreviation for Universal Serial Bus.

[0017] The state estimation system 100 includes elements such as a feature expression model creation unit 180, a state estimation model creation unit 130, and a state estimation unit 140. These elements are realized by software.

[0018] The auxiliary storage device 103 stores a state estimation program for causing the computer to function as the feature representation model creation unit 180, the state estimation model creation unit 130, and the state estimation unit 140. The state estimation program is loaded into the memory 102 and executed by the processor 101. The auxiliary storage device 103 also stores an OS. At least a portion of the OS is loaded into the memory 102 and executed by the processor 101. The processor 101 executes the state estimation program while running the OS. OS is an abbreviation for Operating System.

[0019] Data (input data, output data, etc.) of the state estimation program is stored in the storage unit 190. The auxiliary storage device 103 functions as the storage unit 190. However, a storage device such as the memory 102, a register in the processor 101, or a cache memory in the processor 101 may function as the storage unit 190 instead of or together with the auxiliary storage device 103.

[0020] The state estimation program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or a flash memory.

[0021] The configuration of the feature representation model creation unit 180 will be described with reference to Fig. 2. The feature representation model creation unit 180 includes an information acquisition unit 110 and a feature representation learning unit 120. The information acquisition unit 110 includes an electrocardiogram acquisition unit 111 and a cardiac activity information extraction unit 112. The feature representation learning unit 120 includes an electrocardiogram learning unit 121, a cardiac activity learning unit 122, and a self-distillation unit 123. The electrocardiogram data 201, the electrocardiogram feature representation model 203, the cardiac activity feature representation model 204, and the feature quantity calculation model 205 are stored in the storage unit 190.

[0022] The configuration of the state estimation model creation unit 130 will be described with reference to Fig. 3. The state estimation model creation unit 130 includes a cardiac activity extraction unit 131, a feature calculation unit 132, and a feature learning unit 133. Electrocardiogram data 211, cardiac activity information 212, learning features 213, and a state estimation model 214 are stored in a storage unit.

[0023] The configuration of the state estimation unit 140 will be described with reference to Fig. 4. The state estimation unit 140 includes a cardiac activity extraction unit 141, a feature calculation unit 142, and a feature learning unit 143. The sensor data 221, cardiac activity information 222, inference features 223, and state information 224 are stored in the storage unit 190.

[0024] ***Description of Operation*** The operation procedure of the state estimation system 100 corresponds to a state estimation method. Also, the operation procedure of the state estimation system 100 corresponds to a processing procedure by a state estimation program.

[0025] The state estimation method will be described with reference to Fig. 5. In step S110, the information acquisition unit 110 acquires electrocardiogram data 201 and cardiac activity information 202.

[0026] Step S110 will be described in detail with reference to Fig. 6. In step S111, the electrocardiogram acquisition unit 111 acquires the electrocardiogram data 201.

[0027] The electrocardiogram data 201 is electrocardiogram data used in feature representation learning (electrocardiogram data for feature learning). The electrocardiogram data is data showing the electrocardiogram of an animal. The electrocardiogram data is obtained by measurement using an electrocardiograph. The "animal" means a human or a non-human animal.

[0028] The electrocardiogram data is provided as, for example, a one-dimensional signal, and may be data from multiple leads (for example, a 12-lead electrocardiogram) or single lead.

[0029] For example, a user inputs electrocardiogram data 201 to the condition estimation system 100 , and the electrocardiogram acquisition unit 111 receives the input electrocardiogram data 201 .

[0030] The electrocardiogram acquisition unit 111 may extract data (partial electrocardiogram signals) within an arbitrary time range from the electrocardiogram data 201. For example, the arbitrary time range is determined randomly. When data within the arbitrary time range is extracted from the electrocardiogram data 201, the extracted data is used as the electrocardiogram data 201 in subsequent processing.

[0031] In step S112 , the cardiac activity information extracting unit 112 extracts cardiac activity information 202 from the electrocardiogram data 201 .

[0032] The cardiac activity information 202 is cardiac activity information extracted from the electrocardiogram data 201 .

[0033] 7 shows an example of cardiac activity information. The cardiac activity information includes information about the heart rate, electrocardiogram peaks, etc. The information about the electrocardiogram peaks includes the R wave, RR interval, Q wave, T wave, QT interval, P wave, QRS interval, etc.

[0034] The cardiac activity information may be extracted by any method, and may be extracted using an existing method for each type of cardiac activity information, or may be extracted using a neural network or the like.

[0035] The cardiac activity information 202 is extracted from the partial electrocardiogram extracted as the electrocardiogram data 201. In other words, the cardiac activity information 202 is not a list of raw values ​​of cardiac activity information, but is data that has been shaped to match the dimension (sampling period) of the electrocardiogram data 201 (partial electrocardiogram signal). The dimension of the cardiac activity information is changed by smoothing or padding, etc. Padding is a process of filling in the same values. By unifying the dimensions, it is possible to make the configurations of the electrocardiogram feature representation model 203 and the cardiac activity feature representation model 204 identical.

[0036] 5, the description will continue from step S120. In step S120, the feature representation learning unit 120 creates a feature quantity calculation model 205 using the electrocardiogram data 201 and cardiac activity information 202.

[0037] Step S120 will be described in detail with reference to Fig. 8. In step S121, the electrocardiogram learning unit 121 learns the electrocardiogram data 201 and creates the electrocardiogram feature expression model 203.

[0038] The electrocardiogram feature expression model 203 is a feature expression model created by learning the electrocardiogram data 201 .

[0039] A feature representation model is a trained model for calculating input features. The feature representation model is created through feature representation training.

[0040] Specifically, the electrocardiogram learning unit 121 learns the feature representation of the electrocardiogram indicated by the electrocardiogram data 201 and creates the electrocardiogram feature representation model 203. For example, a neural network or the like is used for the learning.

[0041] In step S122 , the cardiac activity learning unit 122 learns the learning information based on the electrocardiogram data 201 and creates the cardiac activity feature representation model 204 .

[0042] The cardiac activity feature representation model 204 is a feature representation model created by learning cardiac activity information.

[0043] Specifically, the cardiac activity learning unit 122 learns cardiac activity information 202 extracted from electrocardiogram data 201 as learning information, and creates a cardiac activity feature representation model 204 .

[0044] Specifically, the cardiac activity learning unit 122 learns the feature representation of the cardiac activity information 202 to create the cardiac activity feature representation model 204. For example, a neural network or the like is used for the learning.

[0045] In step S123, the self-distillation unit 123 creates a feature quantity calculation model 205 by self-distillation using the electrocardiogram feature expression model 203 and the cardiac activity feature expression model 204.

[0046] The feature quantity calculation model 205 is a feature expression model.

[0047] Specifically, the self-distillation unit 123 performs self-distillation using the electrocardiogram feature representation model 203 as a teacher model and the cardiac activity feature representation model 204 as a student model to update the cardiac activity feature representation model 204. The updated cardiac activity feature representation model 204 becomes the feature calculation model 205.

[0048] Specifically, the self-distillation unit 123 updates the cardiac activity feature representation model 204 and the feature calculation model 205 so that the output of the cardiac activity feature representation model 204 and the output of the electrocardiogram feature representation model 203 approach each other. The amount of information of the electrocardiogram feature is greater than the amount of information of the cardiac activity feature. This framework allows knowledge distillation from the electrocardiogram to cardiac activity information.

[0049] 5, the description will continue from step S 130. In step S 130, the state estimation model creation unit 130 creates the state estimation model 214 using the feature quantity calculation model 205.

[0050] Step S130 will be described in detail with reference to Fig. 9. In step S131, the cardiac activity extraction unit 131 acquires the electrocardiogram data 211.

[0051] The electrocardiogram data 211 is electrocardiogram data (electrocardiogram data for state learning) used in learning for the state estimation model 214 .

[0052] For example, a user inputs electrocardiogram data 211 into the condition estimation system 100 , and the cardiac activity extraction unit 131 receives the input electrocardiogram data 211 .

[0053] The electrocardiogram data 211 may be the same as the electrocardiogram data 201 or may be different from the electrocardiogram data 201 .

[0054] Then, the cardiac activity extraction unit 131 extracts cardiac activity information 212 from the electrocardiogram data 211 .

[0055] The cardiac activity information 212 is cardiac activity information extracted from the electrocardiogram data 211 .

[0056] In step S132 , the feature calculation unit 132 receives input information based on the electrocardiogram data 211 and calculates the learning feature 213 using the feature calculation model 205 .

[0057] The learning feature amount 213 corresponds to the feature amount of the cardiac activity information 212 extracted from the electrocardiogram data 211 .

[0058] Specifically, the feature calculation unit 132 receives cardiac activity information 212 extracted from electrocardiogram data 211 as input information, and calculates learning features 213 using the feature calculation model 205 .

[0059] In step S133 , the feature learning unit 133 learns the learning feature 213 to create the state estimation model 214 .

[0060] The state estimation model 214 is a trained model for estimating the state of an animal based on the feature quantities of the animal's cardiac activity information.

[0061] For example, the feature learning unit 133 performs learning as follows: A correct label indicating the state of the animal corresponding to the electrocardiogram data 211 is input to the state estimation system 100. Then, the feature learning unit 133 receives the correct label and learns the relationship between the learning features 213 and the correct label. If there is no correct label, the feature learning unit 133 performs learning using a general anomaly detection approach. For example, a group of electrocardiogram data of an animal in a normal state is used as learning data, and the feature learning unit 133 determines an "abnormal state" based on the deviation from the feature space obtained from the learning data. The method for determining the deviation is not limited. For example, a method of setting a threshold value may be used, or a method of setting a threshold value that determines a normal state in 99% of the learning data may be used.

[0062] Returning to Fig. 5, step S140 will be described. In step S140, the state estimation unit 140 uses the feature calculation model 205 and the state estimation model 214 to estimate the state of the target animal.

[0063] Step S140 will be described in detail with reference to Fig. 10. In step S141, the cardiac activity extractor 141 acquires the sensor data 221.

[0064] The sensor data 221 is sensor data (target sensor data) obtained by sensing an animal (target animal) whose state is to be estimated.

[0065] The sensor data is data from which cardiac activity information can be extracted and is obtained from sensors other than electrocardiographs. For example, the sensor data is obtained from a wearable device, a non-contact biosensor, etc. Examples of the non-contact biosensor include an infrared camera, a camera, a radar, a radio wave sensor, and LiDAR. LiDAR is an abbreviation for Light Detection and Ranging.

[0066] For example, a user inputs sensor data 221 into the state estimation system 100 , and the cardiac activity extraction unit 141 receives the input sensor data 221 .

[0067] Then, the cardiac activity extraction unit 141 extracts cardiac activity information 222 from the sensor data 221 .

[0068] The cardiac activity information 222 is cardiac activity information (target activity information) extracted from the sensor data 221 .

[0069] In step S142 , the feature calculation unit 142 receives the cardiac activity information 222 as an input and calculates the inference feature 223 using the feature calculation model 205 .

[0070] The inference feature 223 is a feature of the cardiac activity information 222 .

[0071] In step S143, the feature learning unit 143 receives the inference feature 223 as an input and uses the state estimation model 214 to estimate the state of the target animal.

[0072] Then, the feature learning unit 143 outputs the state information 224 .

[0073] The state information 224 indicates the estimated state of the target animal.

[0074] For example, the feature learning unit 143 displays the state information 224 on a display.

[0075] Summary of First Embodiment The first embodiment relates to a state estimation system that acquires cardiac activity information from a wearable device or the like, extracts features from the cardiac activity information, and estimates the state of the individual. In the first embodiment, a feature representation model is constructed by self-distillation using a previously obtained electrocardiogram signal, capable of extracting features with higher expressiveness than those obtained from cardiac activity information alone. This improves the accuracy of state estimation. An overview of the first embodiment is shown in FIGS. 11 and 12 . As shown in (1) of FIG. 11 , feature representation learning is performed by self-distillation, with the electrocardiogram signal as the teacher and the cardiac activity information extracted from the electrocardiogram signal as the student. In self-distillation, learning is performed so that the features obtained from the teacher model and the features obtained from the student model are similar. This makes it possible to learn feature representations of cardiac activity information using an unlabeled electrocardiogram dataset, which is relatively easy to prepare. Furthermore, it is possible to learn features with higher expressiveness than learning using cardiac activity information alone. As shown in (2) of Fig. 11, the state estimation model is trained so that the state estimated by inputting the feature quantities of cardiac activity information extracted from the electrocardiogram signal matches the correct label. As shown in Fig. 12, during inference, the state is estimated by inputting cardiac activity information obtained from the sensor using the trained Student model and the trained state estimation model.

[0076] ***Effects of First Embodiment*** An electrocardiogram is a higher-level concept of cardiac activity information, and contains much information other than cardiac activity information. For example, an electrocardiogram contains minute changes and fluctuations in the waveform of an electrocardiogram signal. However, within the framework of conventional technology, much of the information contained in an electrocardiogram is omitted during learning.

[0077] On the other hand, in the first embodiment, "learning for state estimation using cardiac activity information as input" is performed without discarding electrocardiogram information. However, when estimating a person's state, such as disease detection, physical condition detection, or emotion estimation, only cardiac activity information (heart rate, pulse rate, RRI, etc.) is used. Using cardiac activity information as input offers the advantage of a smaller amount of data and a lighter measurement load than using electrocardiogram information itself.

[0078] The first embodiment relates to a state estimation method for estimating a person's state from cardiac activity information obtained from a wearable sensor or the like. The first embodiment enables feature extraction with higher expressiveness than features obtained using only cardiac activity information by self-distillation from a feature representation model obtained from an electrocardiogram signal to a feature representation model obtained from cardiac activity information. This framework makes it possible to achieve state estimation with higher accuracy than conventional methods.

[0079] Second Embodiment A mode of converting cardiac activity information extracted from electrocardiogram data into cardiac activity information extracted from sensor data will be described below, focusing on differences from the first embodiment, with reference to Figs.

[0080] ***Description of Configuration*** The configuration of the state estimation system 100 will be described with reference to Fig. 13. The state estimation system 100 further includes an element called a domain transformation model creation unit 150. The domain transformation model creation unit 150 is realized by software. The state estimation program further causes a computer to function as the domain transformation model creation unit 150.

[0081] The configuration of the domain conversion model creation unit 150 will be described with reference to Fig. 14. The domain conversion model creation unit 150 includes a cardiac activity extraction unit 151, a cardiac activity extraction unit 152, and a domain conversion learning unit 153. The electrocardiogram data 231, cardiac activity information 232, sensor data 233, cardiac activity information 234, and domain conversion model 235 are stored in the storage unit 190.

[0082] The configuration of the feature representation model creation unit 180 will be described with reference to Fig. 15. The information acquisition unit 110 further includes a domain conversion unit 113. The conversion activity information 206 is stored in the storage unit 190. The configuration of the feature representation learning unit 120 is the same as that in the first embodiment.

[0083] The configuration of the state estimation model creation unit 130 will be described with reference to Fig. 16. The state estimation model creation unit 130 further includes a domain conversion unit 134. The conversion activity information 215 is stored in the storage unit 190.

[0084] ***Explanation of Operation*** The state estimation method will be described with reference to Fig. 17. In step S210, the domain conversion model creation unit 150 creates a domain conversion model 235 using the electrocardiogram data 231 and the sensor data 233.

[0085] Step S210 will be described in detail with reference to Fig. 18. In step S211, the cardiac activity extraction unit 151 acquires the electrocardiogram data 231.

[0086] The electrocardiogram data 231 is electrocardiogram data that is used to create the domain conversion model 235 .

[0087] For example, a user inputs electrocardiogram data 231 into the condition estimation system 100 , and the cardiac activity extraction unit 151 receives the input electrocardiogram data 231 .

[0088] Then, the cardiac activity extraction unit 151 extracts cardiac activity information 232 from the electrocardiogram data 231 .

[0089] The cardiac activity information 232 is cardiac activity information extracted from the electrocardiogram data 231 .

[0090] In step S212, the cardiac activity extraction unit 152 acquires the sensor data 233.

[0091] The sensor data 233 is sensor data used to create the domain conversion model 235. The sensor data 233 is obtained by sensing the same animal at the same time as the electrocardiogram of the animal is measured to obtain the electrocardiogram data 231.

[0092] For example, a user inputs sensor data 233 into the state estimation system 100 , and the cardiac activity extraction unit 152 receives the input sensor data 233 .

[0093] Then, the cardiac activity extraction unit 152 extracts cardiac activity information 234 from the sensor data 233 .

[0094] The cardiac activity information 234 is cardiac activity information extracted from the sensor data 233 .

[0095] In step S213, the domain conversion learning unit 153 learns the cardiac activity information 232 and the cardiac activity information 234 and creates a domain conversion model 235.

[0096] The domain conversion model 235 is a trained model for converting cardiac activity information extracted from electrocardiogram data into cardiac activity information equivalent to cardiac activity information extracted from sensor data.

[0097] Any learning method may be used. For example, the domain transformation learning unit 153 performs learning using a neural network. For example, it may be realized using an autoencoder, a VAE, a transformer, etc. VAE stands for variational autoencoder. When there are multiple sensors, the domain transformation learning unit 153 performs learning for each sensor.

[0098] The domain conversion learning unit 153 has the following features. Cardiac activity information is not obtained at regular intervals. For example, the RR interval is output each time an R peak is detected in an electrocardiogram, so it is not possible to directly associate pieces of cardiac activity information with each other. Therefore, the domain conversion learning unit 153 resamples pieces of cardiac activity information onto the same time axis. In other words, the domain conversion learning unit 153 includes a process of converting pieces of cardiac activity information into vectors of the same dimension. This process is performed before inputting them into the model.

[0099] 17, the description will continue from step S220. In step S220, the information acquiring unit 110 acquires the electrocardiogram data 201 and the conversion activity information 206.

[0100] Step S220 will be described in detail with reference to Fig. 19. In step S221, the electrocardiogram acquisition unit 111 acquires the electrocardiogram data 201. Step S221 is the same as step S111 in the first embodiment.

[0101] In step S222, the cardiac activity information extracting unit 112 extracts cardiac activity information 202 from the electrocardiogram data 201. Step S222 is the same as step S112 in the first embodiment.

[0102] In step S223, the domain conversion unit 113 converts the cardiac activity information 202 using the domain conversion model 235. As a result, converted activity information 206 (learning information) is created.

[0103] The transformed activity information 206 is the cardiac activity information 202 that has been transformed using the domain transformation model 235 .

[0104] 17, the description will continue from step S230. In step S230, the feature representation learning unit 120 creates a feature quantity calculation model 205 using the electrocardiogram data 201 and the conversion activity information 206.

[0105] Step S230 will be described in detail with reference to Fig. 20. In step S231, the electrocardiogram learning unit 121 learns the electrocardiogram data 201 and creates the electrocardiogram feature expression model 203. Step S231 is the same as step S121 in the first embodiment.

[0106] In step S232, the cardiac activity learning unit 122 learns the conversion activity information 206 to create the cardiac activity feature expression model 204. Step S232 corresponds to the process of replacing the cardiac activity information 202 with the conversion activity information 206 in step S122 of the first embodiment.

[0107] In step S233, the self-distillation unit 123 creates the feature calculation model 205 by self-distillation using the electrocardiogram feature expression model 203 and the cardiac activity feature expression model 204. Step S233 is the same as step S123 in the first embodiment.

[0108] 17, the description will continue from step S240. In step S240, the state estimation model creation unit 130 creates the state estimation model 214 using the domain transformation model 235 and the feature amount calculation model 205.

[0109] Step S240 will be described in detail with reference to Fig. 21. In step S241, the cardiac activity extractor 131 extracts cardiac activity information 212 from the electrocardiogram data 211. Step S241 is the same as step S131 in the first embodiment.

[0110] In step S242, the domain conversion unit 134 converts the cardiac activity information 212 using the domain conversion model 235. As a result, converted activity information 215 (input information) is created.

[0111] The transformed activity information 215 is cardiac activity information 212 that has been transformed using the domain transformation model 235 .

[0112] In step S243, the feature calculation unit 132 receives the conversion activity information 215 as an input and calculates the learning feature 213 using the feature calculation model 205.

[0113] The learning feature 213 is a feature of the conversion activity information 215 .

[0114] In step S244, the feature learning unit 133 learns the learning feature 213 to create the state estimation model 214. Step S244 is the same as step S133 in the first embodiment.

[0115] 17, step S140 will be described. Step S140 is the same as that described in the first embodiment.

[0116] ***Effects of the Second Embodiment*** In the second embodiment, when there is sensor data linked to (measured simultaneously with) electrocardiogram data, a domain conversion model is constructed using a pair of electrocardiogram data and sensor data. The domain conversion model is used for feature representation learning through self-distillation of the electrocardiogram data and for learning of a state estimation model, thereby reducing the domain gap. The domain gap is the difference in characteristics between cardiac activity information obtained from electrocardiogram data and cardiac activity information obtained from sensor data.

[0117] The second embodiment relates to a state estimation method for estimating a person's state from cardiac activity information obtained from a wearable sensor or the like. In the second embodiment, a domain transformation is performed from cardiac activity information obtained from an electrocardiogram to cardiac activity information obtained from a sensor. Then, self-distillation is performed from a feature representation model obtained from the electrocardiogram signal to a feature representation model obtained from the domain-transformed cardiac activity information. This eliminates the domain gap and enables feature extraction with higher expressive power than features using only cardiac activity information. This framework makes it possible to achieve state estimation with higher accuracy than conventional methods.

[0118] Third Embodiment A third embodiment of creating a feature calculation model 205 by two-stage self-distillation will be described with reference to Figs. 22 to 25, focusing mainly on the differences from the first embodiment.

[0119] ***Description of Configuration*** The configuration of the state estimation system 100 will be described with reference to Fig. 22. The state estimation system 100 further includes an element called a domain transformation model creation unit 150. The domain transformation model creation unit 150 is realized by software. The state estimation program further causes a computer to function as the domain transformation model creation unit 150.

[0120] The configuration of the domain transformation model creation unit 150 is the same as that in the second embodiment.

[0121] The configuration of the feature expression model creation unit 180 will be described with reference to Fig. 23. The information acquisition unit 110 further includes a domain conversion unit 113. The feature expression learning unit 120 further includes a conversion activity learning unit 124. The conversion activity information 206 and the conversion activity feature expression model 207 are stored in the storage unit 190.

[0122] ***Description of Operation*** The state estimation method will be described with reference to Fig. 24. In step S310, the domain conversion model creation unit 150 creates the domain conversion model 235 using the electrocardiogram data 231 and the sensor data 233. Step S310 is the same as step S210 in the second embodiment.

[0123] In step S320, the information acquiring unit 110 acquires the electrocardiogram data 201, the cardiac activity information 202, and the conversion activity information 206. Step S320 is the same as step S220 in the second embodiment.

[0124] In step S330, the feature representation learning unit 120 creates a feature quantity calculation model 205 using the electrocardiogram data 201, the cardiac activity information 202, and the conversion activity information 206.

[0125] Step S330 will be described in detail with reference to Fig. 25. In step S331, the electrocardiogram learning unit 121 learns the electrocardiogram data 201 and creates the electrocardiogram feature expression model 203. Step S331 is the same as step S121 in the first embodiment.

[0126] In step S332, the cardiac activity learning unit 122 learns the cardiac activity information 202 to create the cardiac activity feature expression model 204. Step S332 is the same as step S122 in the first embodiment.

[0127] In step S333, the self-distillation unit 123 updates the cardiac activity feature representation model 204 by self-distillation using the electrocardiogram feature representation model 203 and the cardiac activity feature representation model 204. The method for updating the cardiac activity feature representation model 204 is the same as the method in step S123 in the first embodiment.

[0128] In step S334, the conversion activity learning unit 124 learns the conversion activity information 206 and creates the conversion activity feature expression model 207.

[0129] The conversion activity feature representation model 207 is a cardiac activity feature representation model created by learning the conversion activity information 206 .

[0130] In step S335, the self-distillation unit 123 uses the updated cardiac activity learning unit 122 and the conversion activity feature expression model 207 to create the feature calculation model 205 by self-distillation.

[0131] Specifically, the self-distillation unit 123 performs self-distillation using the updated cardiac activity feature representation model 204 as a teacher model and the transformation activity feature representation model 207 as a student model to update the transformation activity feature representation model 207. The updated transformation activity feature representation model 207 becomes the feature calculation model 205.

[0132] Specifically, the self-distillation unit 123 updates the transformed activity feature representation model 207 so that the output of the updated cardiac activity feature representation model 204 approaches the output of the transformed activity feature representation model 207. This framework performs knowledge distillation from cardiac activity information obtained from an electrocardiogram to cardiac activity information obtained from an electrocardiogram (after domain conversion).

[0133] 24, the description will continue from step S130. Steps S130 and S140 are the same as those described in the first embodiment.

[0134] ***Effects of the Third Embodiment*** In the third embodiment, when there is sensor data linked to (measured simultaneously with) electrocardiogram data, a domain conversion model is constructed using a pair of electrocardiogram data and sensor data. The domain gap is eliminated by performing a two-stage self-distillation: self-distillation of cardiac activity information obtained from the electrocardiogram and self-distillation of cardiac activity information after domain conversion.

[0135] In the third embodiment, after executing the flow of the first embodiment, self-distillation is performed from cardiac activity information obtained from an electrocardiogram to cardiac activity information (after domain conversion) obtained from an electrocardiogram. In the third embodiment, the cardiac activity feature representation model constructed within the framework of the first embodiment can be used as is. For example, when sensor data does not exist, the cardiac activity feature representation model is constructed using only electrocardiogram data. Then, when it becomes possible to collect sensor data, a domain-converted cardiac activity feature representation model is constructed from the cardiac activity feature representation model using the electrocardiogram data and the sensor data.

[0136] Fourth Embodiment A fourth embodiment in which knowledge distillation is performed instead of self-distillation will be described below, mainly with reference to Figs. 26 to 31, with respect to differences from the first embodiment.

[0137] ***Description of Configuration*** The configuration of the state estimation system 100 is the same as that in embodiment 1. However, the configurations of the feature expression model creation unit 180 and the state estimation model creation unit 130 are different from those in embodiment 1.

[0138] The configuration of the feature representation model creation unit 180 will be described with reference to Fig. 26. The configuration of the information acquisition unit 110 is the same as that in embodiment 1. The feature representation learning unit 120 includes a knowledge distillation unit 125 instead of the cardiac activity learning unit 122. The sensor data 208 is stored in the storage unit 190.

[0139] The configuration of the state estimation model creation unit 130 will be described with reference to Fig. 27. The sensor data 216 is stored in the storage unit 190.

[0140] ***Description of Operation*** The state estimation method will be described with reference to Fig. 28. In step S410, the information acquisition unit 110 acquires the electrocardiogram data 201 and the cardiac activity information 202.

[0141] Step S410 will be described in detail with reference to Fig. 29. In step S411, the electrocardiogram acquisition unit 111 acquires the electrocardiogram data 201. Step S411 is the same as step S111 in the first embodiment.

[0142] In step S412 , the cardiac activity information extraction section 112 acquires the sensor data 208 .

[0143] The sensor data 208 is sensor data used to create the cardiac activity feature representation model 204. The sensor data 208 is obtained by sensing the same animal at the same time as the electrocardiogram of the animal is measured to obtain the electrocardiogram data 201.

[0144] For example, a user inputs sensor data 208 into the state estimation system 100 , and the cardiac activity information extraction unit 112 receives the input sensor data 208 .

[0145] Then, the cardiac activity information extractor 112 extracts cardiac activity information 202 from the sensor data 208 .

[0146] Cardiac activity information 202 is cardiac activity information extracted from sensor data 208 .

[0147] 28, the description will continue from step S420. In step S420, the feature representation learning unit 120 creates a feature quantity calculation model 205 using the electrocardiogram data 201 and the cardiac activity information 202.

[0148] Step S420 will be described in detail with reference to Fig. 30. In step S421, the electrocardiogram learning unit 121 learns the electrocardiogram data 201 and creates the electrocardiogram feature expression model 203. Step S421 is the same as step S121 in the first embodiment.

[0149] In step S422, the cardiac activity learning unit 122 learns the cardiac activity information 202 and creates the cardiac activity feature expression model 204. Step S422 is the same as step S122 in the first embodiment.

[0150] In step S423, the knowledge distillation unit 125 creates the feature quantity calculation model 205 by knowledge distillation using the electrocardiogram feature expression model 203 and the cardiac activity feature expression model 204.

[0151] Specifically, the knowledge distillation unit 125 performs knowledge distillation using the electrocardiogram feature representation model 203 as a teacher model and the cardiac activity feature representation model 204 as a student model to update the cardiac activity feature representation model 204. The updated cardiac activity feature representation model 204 becomes the feature quantity calculation model 205.

[0152] 28, the description will continue from step S430. In step S430, the state estimation model creation unit 130 creates the state estimation model 214 using the feature quantity calculation model 205.

[0153] Details of step S430 will be described with reference to Fig. 31. In step S431, the cardiac activity extraction unit 131 acquires the sensor data 216.

[0154] The sensor data 216 is used to create the state estimation model 214 .

[0155] For example, a user inputs sensor data 216 into the state estimation system 100 , and the cardiac activity extraction unit 131 receives the input sensor data 216 .

[0156] Then, the cardiac activity extraction unit 131 extracts cardiac activity information 212 from the sensor data 216 .

[0157] Cardiac activity information 212 is cardiac activity information extracted from sensor data 216 .

[0158] In step S432 , the feature calculation unit 132 receives the cardiac activity information 212 as an input and calculates the learning feature 213 using the feature calculation model 205 .

[0159] The learning feature 213 is a feature of the cardiac activity information 212 .

[0160] In step S433, the feature learning unit 133 learns the learning feature 213 to create the state estimation model 214. Step S433 is the same as step S133 in the first embodiment.

[0161] Returning to Fig. 28, step S140 will be described. Step S140 is the same as that described in the first embodiment.

[0162] ***Effects of Embodiment 4*** In embodiment 4, when sensor data linked to electrocardiogram data (measured simultaneously) is available, the framework of embodiment 1 is realized by knowledge distillation from electrocardiogram data to sensor data.

[0163] The fourth embodiment relates to a state estimation method for estimating a person's state from cardiac activity information obtained from a wearable sensor or the like. The fourth embodiment enables feature extraction with higher expressiveness than features using only cardiac activity information by self-distillation from a feature representation model obtained from an electrocardiogram signal to a feature representation model obtained from cardiac activity information obtained from the electrocardiogram signal. This framework makes it possible to achieve state estimation with higher accuracy than conventional methods.

[0164] *** Supplementary Information about the Embodiment *** The hardware configuration of the state estimation system 100 will be described with reference to Fig. 32 . The state estimation system 100 includes a processing circuit 109. The processing circuit 109 is hardware that realizes the feature representation model creation unit 180, the state estimation model creation unit 130, the state estimation unit 140, and the domain transformation model creation unit 150. The processing circuit 109 may be dedicated hardware, or may be a processor 101 that executes a program stored in the memory 102.

[0165] When the processing circuit 109 is dedicated hardware, the processing circuit 109 may be, for example, a single circuit, a multiple circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field Programmable Gate Array.

[0166] The state estimation system 100 may include multiple processing circuits replacing the processing circuit 109 .

[0167] In the processing circuit 109, some functions may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware.

[0168] Thus, the functions of the state estimation system 100 can be realized by hardware, software, firmware, or a combination thereof.

[0169] Each embodiment is an example of a preferred embodiment and is not intended to limit the technical scope of the present disclosure. Each embodiment may be implemented in part or in combination with other embodiments. Procedures described using flowcharts, etc. may be modified as appropriate.

[0170] The state estimation system 100 may be realized by one device (computer) or by multiple devices. The "part" of each element of the state estimation system 100 may be read as a "process," a "step," a "circuit," or a "circuitry."

[0171] 100 State estimation system, 101 Processor, 102 Memory, 103 Auxiliary storage device, 104 Input / output interface, 109 Processing circuit, 110 Information acquisition unit, 111 Electrocardiogram acquisition unit, 112 Cardiac activity information extraction unit, 113 Domain conversion unit, 120 Feature representation learning unit, 121 Electrocardiogram learning unit, 122 Cardiac activity learning unit, 123 Self-distillation unit, 124 Conversion activity learning unit, 125 Knowledge distillation unit, 130 State estimation model creation unit, 131 Cardiac activity extraction unit, 132 Feature calculation unit, 133 Feature learning unit, 134 Domain conversion unit, 140 State estimation unit, 141 Cardiac activity extraction unit, 142 Feature calculation unit, 143 Feature learning unit, 150 Domain conversion model creation unit, 151 Cardiac activity extraction unit, 152 Cardiac activity extraction unit, 153 Domain conversion learning unit, 180 feature representation model creation unit, 190 memory unit, 201 electrocardiogram data, 202 cardiac activity information, 203 electrocardiogram feature representation model, 204 cardiac activity feature representation model, 205 feature calculation model, 206 converted activity information, 207 converted activity feature representation model, 208 sensor data, 211 electrocardiogram data, 212 cardiac activity information, 213 learning feature, 214 state estimation model, 215 converted activity information, 216 sensor data, 221 sensor data, 222 cardiac activity information, 223 inference feature, 224 state information, 231 electrocardiogram data, 232 cardiac activity information, 233 sensor data, 234 cardiac activity information, 235 domain conversion model.

Claims

1. A state estimation system comprising: an electrocardiogram learning unit that learns electrocardiogram data for feature learning and creates an electrocardiogram feature representation model; a cardiac activity learning unit that learns learning information based on the electrocardiogram data for feature learning and creates a cardiac activity feature representation model; a self-distillation unit that creates a feature calculation model by self-distillation using the electrocardiogram feature representation model and the cardiac activity feature representation model; a state estimation model creation unit that takes input information based on electrocardiogram data for state learning as input, and uses the feature calculation model to calculate learning features corresponding to the features of the cardiac activity information extracted from the electrocardiogram data for state learning, and learns the learning features to create a state estimation model; and a state estimation unit that extracts cardiac activity information as target activity information from target sensor data obtained by sensing a target animal, uses the target activity information as input, calculates the features of the target activity information as inference features using the feature calculation model, and estimates the state of the target animal using the state estimation model with the inference features as input.

2. The state estimation system of claim 1, wherein the self-distillation unit performs self-distillation using the electrocardiogram feature representation model as a teacher model and the cardiac activity feature representation model as a student model to create an updated cardiac activity feature representation model as the feature calculation model.

3. A state estimation system as described in claim 1 or claim 2, wherein the cardiac activity learning unit learns cardiac activity information extracted from the feature learning electrocardiogram data as the learning information to create the cardiac activity feature representation model, and the state estimation model creation unit uses the cardiac activity information extracted from the state learning electrocardiogram data as the input information to calculate the learning features using the feature calculation model.

4. The state estimation system according to claim 1 or claim 2, further comprising: a domain conversion unit that uses a domain conversion model to convert cardiac activity information extracted from the feature learning electrocardiogram data into cardiac activity information equivalent to cardiac activity information extracted from sensor data, to convert cardiac activity information extracted from the feature learning electrocardiogram data, thereby creating the learning information; and the state estimation model creation unit uses the domain conversion model to convert cardiac activity information extracted from the state learning electrocardiogram data, thereby creating the input information.

5. A state estimation system as described in claim 4, comprising a domain conversion model creation unit that creates the domain conversion model by learning cardiac activity information extracted from electrocardiogram data showing the electrocardiogram of an animal and cardiac activity information extracted from sensor data obtained by sensing the animal simultaneously with measuring the electrocardiogram.

6. The state estimation system according to claim 1, comprising: a domain conversion unit that uses a domain conversion model that converts cardiac activity information extracted from the electrocardiogram data for feature learning into cardiac activity information equivalent to cardiac activity information extracted from sensor data to convert the cardiac activity information extracted from the feature learning electrocardiogram data to create converted activity information; and a converted activity learning unit that learns the converted activity information to create a converted activity feature model, wherein the self-distillation unit updates the cardiac activity feature representation model by performing self-distillation using the electrocardiogram feature representation model as a teacher model and the cardiac activity feature representation model as a student model, and creates the updated converted activity feature model as the feature calculation model by performing self-distillation using the updated cardiac activity feature representation model as a teacher model and the converted activity feature model as a student model.

7. A state estimation system as described in claim 6, comprising a domain conversion model creation unit that creates the domain conversion model by learning cardiac activity information extracted from electrocardiogram data showing the electrocardiogram of an animal and cardiac activity information extracted from sensor data obtained by sensing the animal simultaneously with measuring the electrocardiogram.

8. A state estimation system as described in claim 6 or claim 7, wherein the cardiac activity learning unit learns cardiac activity information extracted from the feature learning electrocardiogram data as the learning information to create the cardiac activity feature representation model, and the state estimation model creation unit calculates the learning features using the feature calculation model with the cardiac activity information extracted from the state learning electrocardiogram data as the input information.

9. A state estimation method comprising: learning electrocardiogram data for feature learning to create an electrocardiogram feature representation model; learning learning information based on the electrocardiogram data for feature learning to create a cardiac activity feature representation model; creating a feature calculation model by self-distillation using the electrocardiogram feature representation model and the cardiac activity feature representation model; using input information based on electrocardiogram data for state learning as input, calculating learning features corresponding to the features of cardiac activity information extracted from the electrocardiogram data for state learning using the feature calculation model; learning the learning features to create a state estimation model; extracting cardiac activity information as target activity information from target sensor data obtained by sensing a target animal, using the target activity information as input, calculating the features of the target activity information as inference features using the feature calculation model, and estimating the state of the target animal using the inference features as input using the state estimation model.

10. A state estimation program for causing a computer to execute the following steps: an electrocardiogram learning process for learning electrocardiogram data for feature learning and creating an electrocardiogram feature representation model; a cardiac activity learning process for learning learning information based on the electrocardiogram data for feature learning and creating a cardiac activity feature representation model; a self-distillation process for creating a feature calculation model by self-distillation using the electrocardiogram feature representation model and the cardiac activity feature representation model; a state estimation model creation process for taking input information based on electrocardiogram data for state learning as input, using the feature calculation model to calculate learning features corresponding to the features of the cardiac activity information extracted from the electrocardiogram data for state learning, and learning the learning features to create a state estimation model; and a state estimation process for extracting cardiac activity information as target activity information from target sensor data obtained by sensing a target animal, using the target activity information as input, calculating features of the target activity information as inference features using the feature calculation model, and then using the inference features as input to infer the state of the target animal using the state estimation model.

11. An electrocardiogram learning unit that learns electrocardiogram data for feature learning that indicates the electrocardiogram of an animal and creates an electrocardiogram feature representation model; a cardiac activity learning unit that learns cardiac activity information extracted from feature learning sensor data obtained by sensing the animal simultaneously with measuring the electrocardiogram and creates a cardiac activity feature representation model; a knowledge distillation unit that performs knowledge distillation using the electrocardiogram feature representation model and the cardiac activity feature representation model to create an updated cardiac activity feature representation model as a feature calculation model; a state estimation model creation unit that receives cardiac activity information extracted from state learning sensor data as input, calculates feature quantities of the cardiac activity information extracted from the state learning sensor data using the feature calculation model, and learns the learning features to create a state estimation model; a state estimation unit that extracts cardiac activity information as target activity information from target sensor data obtained by sensing a target animal, calculates features of the target activity information as features for inference using the feature calculation model with the target activity information as input, and estimates the state of the target animal using the state estimation model with the features for inference as input.

12. A state estimation method comprising: creating an electrocardiogram feature representation model by learning electrocardiogram data for feature learning that indicates an animal's electrocardiogram; creating a cardiac activity feature representation model by learning cardiac activity information extracted from feature learning sensor data obtained by sensing the animal simultaneously with measuring the electrocardiogram; creating an updated cardiac activity feature representation model as a feature calculation model by performing knowledge distillation using the electrocardiogram feature representation model and the cardiac activity feature representation model; using cardiac activity information extracted from state learning sensor data as input, calculating feature quantities of the cardiac activity information extracted from the state learning sensor data as learning features using the feature calculation model; learning the learning features to create a state estimation model; extracting cardiac activity information as target activity information from target sensor data obtained by sensing a target animal, using the target activity information as input, calculating feature quantities of the target activity information as inference features using the feature calculation model; and estimating the state of the target animal using the state estimation model as input.

13. An electrocardiogram learning process that learns electrocardiogram data for feature learning that indicates the electrocardiogram of an animal and creates an electrocardiogram feature representation model; a cardiac activity learning process that learns cardiac activity information extracted from feature learning sensor data obtained by sensing the animal simultaneously with measuring the electrocardiogram and creates a cardiac activity feature representation model; a knowledge distillation process that performs knowledge distillation using the electrocardiogram feature representation model and the cardiac activity feature representation model to create an updated cardiac activity feature representation model as a feature calculation model; a state estimation model creation process that uses cardiac activity information extracted from state learning sensor data as input, calculates feature quantities of the cardiac activity information extracted from the state learning sensor data as learning features using the feature calculation model, and learns the learning features to create a state estimation model; a state estimation program for causing a computer to execute the steps of: extracting cardiac activity information as target activity information from target sensor data obtained by sensing a target animal; calculating features of the target activity information as features for inference using the feature calculation model with the target activity information as input; and estimating the state of the target animal using the feature for inference as input.

Images