Biological information inference device, biological information inference method, and program

Abstract

One aspect of the present disclosure is a biological information inference device comprising: an image acquisition unit that acquires a captured image obtained by imaging a subject; a preprocessing unit that performs prescribed preprocessing on the captured image; an extraction unit that extracts color information of the subject from the captured image subjected to the preprocessing performed by the preprocessing unit; an inference unit that infers, on the basis of the color information and subject information indicating information on the subject, a biological information value that is a value indicating biological information; and an output unit that outputs the biological information value. The preprocessing in the preprocessing unit includes color correction processing for correcting the color of the subject included in the captured image so as to be close to the actual color of the subject.

Description

Biological information estimation device, biological information estimation method, and program

[0001] The present invention relates to a biological information estimation device, a biological information estimation method, and a program.

[0002] Conventionally, techniques for estimating health status using images of the human body have been known. For example, Patent Document 1 describes a health status estimation system that estimates the health status of a subject by analyzing images of nails. The health status estimation system acquires images including the subject's nails, extracts features from the images, and estimates the subject's health status through machine learning based on the extracted features. Conventionally, techniques for performing tests such as urinalysis at home have been known. For example, Patent Document 2 describes a urine test device that captures images of a urine test reagent using a digital camera. The urine test device includes a through-hole for inserting the lens barrel of the digital camera into an opening of a container with the urine test reagent disposed on the inner bottom surface. This allows the digital camera to be fixed to the urine test device to capture an image of the urine test reagent after the criterion display and color reaction, and then the captured image can be displayed on a display monitor, allowing the color of the criterion display and the color of the urine test reagent to be compared and examined. Patent Document 3 describes a method in which multiple types of color change information of a color change portion that changes to multiple types of color tones upon contact with urine and multiple types of color change reference information that indicate the correspondence with the measured values ​​of biological information are both captured by an imaging device, and the color change information captured by the imaging device is compared with the color change reference information to identify the color change reference information that corresponds to the color change information.

[0003] JP 2021-125130 A JP 2009-229212 A JP 2015-187562 A

[0004] The technology described in Patent Document 1 uses images of the human body to estimate health status, but the camera device used to capture the images and the surrounding environment in which the images are captured vary depending on the subject. Because the color information of the image differs depending on the camera device and surrounding environment, it is difficult to estimate health status with high accuracy. The test described in Patent Document 2 uses images captured by an imaging device, but depending on the model and performance of the digital camera, accurate color images may not be obtained, resulting in inaccurate testing. The test described in Patent Document 3 requires simultaneous capture of color change information and color change reference information, which requires preparation of color change reference information for each capture. Furthermore, the color change information and color change reference information may change depending on the surrounding environment, and, as with Patent Document 1, accurate color images may not be obtained depending on the model and performance of the digital camera.

[0005] The present disclosure has been made in consideration of these circumstances, and aims to provide a biometric information estimation device, a biometric information estimation method, and a program that can acquire color information from a captured image and estimate biometric information with high accuracy.

[0006] The present disclosure has been made to solve the above-mentioned problems, and one aspect of the present disclosure is a bioinformation estimation device comprising: an image acquisition unit that acquires an image of a subject; a preprocessing unit that performs predetermined preprocessing on the image; an extraction unit that extracts color information of the subject from the image that has been preprocessed by the preprocessing unit; an estimation unit that estimates a bioinformation value that is a value indicating bioinformation based on the color information and subject information that indicates information about the subject; and an output unit that outputs the bioinformation value.

[0007] Another aspect of the present disclosure is a bioinformation estimation method including the steps of: acquiring an image of a subject; performing predetermined preprocessing on the image; extracting color information of the subject from the preprocessed image; estimating a bioinformation value, which is a value indicating bioinformation, based on the color information and subject information indicating information about the subject; and outputting the bioinformation value.

[0008] Another aspect of the present disclosure is a bioinformation estimation program that causes a computer to execute the steps of acquiring an image of a subject, performing predetermined preprocessing on the image, extracting color information of the subject from the preprocessed image, estimating a bioinformation value that is a value indicating bioinformation based on the color information and subject information that indicates information about the subject, and outputting the bioinformation value.

[0009] According to one aspect of the present invention, color information can be acquired from a captured image and biological information can be estimated with high accuracy.

[0010] FIG. 1 is a block diagram showing an example of a configuration of a biometric information estimation system according to a first embodiment. FIG. 2 is a block diagram showing an example of a configuration of a biometric information estimation system according to a second embodiment. FIG. 3 is a sequence diagram showing an example of a processing procedure of the biometric information estimation system according to the second embodiment. FIG. 4 is a diagram showing an example of a color information acquisition unit according to the second embodiment. FIG. 5 is a block diagram showing an example of a biometric information estimation system according to the second embodiment. FIG. 6 is a block diagram showing an example of a biometric information estimation system according to the second embodiment. FIG. 7 is a block diagram showing an example of a configuration of an inspection system according to a third embodiment. FIG. 8 is a sequence diagram showing an example of a processing procedure of the inspection system according to the third embodiment. FIG. 9 is a diagram showing an example of a display screen of a terminal device according to the third embodiment. FIG. 10 is a diagram showing an example of an image color correction unit according to the third embodiment. FIG. 11 is a block diagram showing an example of an inspection system according to the third embodiment.

[0011] A biological information estimation device, a biological information estimation method, and a program according to the present invention will be described below with reference to the accompanying drawings.

[0012] 1 is a block diagram showing an example of the configuration of a biological information estimation system 1 according to a first embodiment. The biological information estimation system 1 is a system in which a captured image is transmitted to a biological information estimation device 100 that is disposed remotely from a subject, and the biological information estimation device 100 acquires color information from the captured image and estimates biological information with high accuracy.

[0013] The specimen in the first embodiment is a body part of a subject, a reagent reacted with a user's bodily fluid, etc. The subject is a user of the terminal device 200, a patient, etc. Biological information is various numerical values ​​related to the subject's living body. Biological information is values ​​directly measured from the living body, such as body temperature, blood pressure, oxygen saturation, and pulse rate, or test values ​​obtained by reacting bodily fluid with a reagent, etc.

[0014] The biometric information estimation system 1 includes, for example, a biometric information estimation device 100 and a terminal device 200 equipped with a camera device 210. The biometric information estimation device 100 and the terminal device 200 are connected via, for example, a communication network. The biometric information estimation device 100 and the terminal device 200 each have a communication interface such as a network interface card (NIC) or a wireless communication module for connecting to a network such as the Internet. The communication network may include, for example, a general-purpose network such as the Internet, and a private network such as local 5G or Wi-Fi (registered trademark).

[0015] The terminal device 200 is an information processing device such as a smartphone or personal computer used by a user. The terminal device 200 includes a camera device 210, an input unit 220, and a display unit 230. The camera device 210 operates based on operations received by the input unit 220 and generates captured images of a subject. The input unit 220 is a device that receives user operations, such as a touch sensor built into the display unit 230. The display unit 230 is a device that displays various information, such as an organic EL display. The terminal device 200 executes application software to receive user operations, transmit captured images generated by the camera device 210 to the bioinformation estimation device 100, and display bioinformation (test results) transmitted from the bioinformation estimation device 100. Note that, although the terminal device 200 and the bioinformation estimation device 100 are separate entities in this embodiment, this is not limiting, and the functions of the bioinformation estimation device 100 may be built into the terminal device 200.

[0016] The camera device 210 may be a camera built into a smartphone, a single-lens reflex camera, a web camera, etc. The terminal device 200 may transmit other information, such as metadata indicating camera-related information such as the model of the camera of the terminal device 200, along with the captured image.

[0017] The captured image may be a still image or a moving image. The captured image may be a plurality of images generated by interval photography. The captured image may be an image of a part of a patient. The part of a patient may be, for example, a nail, the back of a hand, the face, lips, conjunctiva, eyes, eyelids, or tongue. The captured image may be an image of multiple parts of a patient. The captured image may be a still image of a reagent or a moving image of a reagent. The captured image may be a plurality of images generated by interval photography every elapsed seconds to observe the reaction of a reagent, for example. The captured image may be a plurality of images captured with different subject distances, with or without a flash, at different angles, etc. The captured image may be an image of a patient together with a reference object such as a color chart or white board for color correction. The captured image may include both an image including the patient and an image of the reference object. The captured image may include depth information measured using a depth sensor or the like.

[0018] The biometric information estimation device 100 is an information processing device that communicates with other devices and performs various processes. The biometric information estimation device 100 includes, for example, an image acquisition unit 110, a preprocessing unit 120, an extraction unit 130, an estimation unit 140, an output unit 150, and a storage unit 160. The image acquisition unit 110, the preprocessing unit 120, the extraction unit 130, the estimation unit 140, and the output unit 150 are functional units that are realized by a processor such as a CPU (Central Processing Unit) executing a biometric information estimation program stored in a program memory.

[0019] The image acquisition unit 110 acquires an image of a subject.

[0020] The preprocessing unit 120 performs predetermined preprocessing on the captured image. The preprocessing unit 120 may perform the preprocessing by referring to preprocessing setting information stored in the storage unit 160. The preprocessing setting information is information that identifies, for example, a processing procedure for the preprocessing, processing rules such as calculation formulas, programs, and machine learning models.

[0021] The pre-processing unit 120 may perform color correction processing to correct the color of the subject included in the captured image so that it approaches the color of the actual subject. The pre-processing unit 120 corrects the image so that it matches the "correct color" of the subject in the captured image. The correct color refers to a color that is correct in terms of colorimetry (defined color values ​​measured with a colorimeter). The pre-processing unit 120 may perform image processing useful for estimating biometric information values, such as illumination unevenness correction (shading correction), shadow correction, sharpness correction, and contrast correction. The pre-processing unit 120 may also perform processing such as enlarging, reducing, adjusting the shape, and extracting objects from the captured image, as long as it is useful for estimating biometric information values.

[0022] The extraction unit 130 extracts color information of the subject from the captured image preprocessed by the preprocessing unit 120. The extraction unit 130 may extract the color information by referring to extraction setting information stored in the storage unit 160. The extraction setting information is information used by the extraction unit 130 to perform the color information extraction process, and is, for example, information indicating the body part to be extracted or information indicating a learned spectral reflectance estimation model. This allows the extraction unit 130 to extract color information corresponding to each piece of biological information, even when estimating multiple pieces of biological information, for example.

[0023] The color information is information such as colorimetric values ​​indicating the patient's color for estimating biological information. The color information may be RGB values ​​with white balance correction or RGB values ​​reproduced in a specific color space, such as sRGB values. The color information may be spectral reflectance, XYZ values ​​in the XYZ color system, L*a*b* values ​​in the L*a*b* color system, erythema index, melanin index, absorbance, spectral distribution at a specific wavelength, a representative value for a specific region of a subject, or time-series information indicating changes over time in a video or interval-captured image with values ​​for each pixel. The color information may also include two or more of these types of information. The color information may be a single value, such as an average value for the extracted region, or image data (two-dimensional color information distribution) of a small region including the extracted region.

[0024] The estimation unit 140 estimates a biometric information value, which is a value indicating biometric information, based on color information and subject information. The subject information is information indicating information on the subject whose image has been captured. Specifically, the subject information is information indicating a part of the patient and information indicating a reagent. The estimation unit 140 may estimate the biometric information using a method such as multiple regression analysis, polynomial approximation, or machine learning. The estimation unit 140 may estimate the biometric information by referring to estimation setting information stored in the storage unit 160. The estimation setting information is, for example, information specifying processing rules such as estimation processing procedures and calculation formulas, programs, and machine learning models.

[0025] The output unit 150 outputs the biometric information value. The output unit 150 may transmit display data for displaying the biometric information value on the terminal device 200. The output unit 150 may output the biometric information to, for example, a device that stores the biometric information, or may output the biometric information to an inspection device that performs various inspections using the biometric information. Note that the output unit 150 may output multiple pieces of biometric information simultaneously.

[0026] The storage unit 160 stores preprocessing setting information, extraction setting information, and estimation setting information. The preprocessing setting information is, for example, information for correcting color information without using a color chart. For example, the preprocessing setting information is the camera spectral sensitivity and camera gradation characteristics for each camera model, a trained spectral reflectance estimation model for each subject or a general-purpose trained spectral reflectance estimation model, the illumination spectral distribution, and parameters necessary for estimating the illumination color from a captured image. The preprocessing setting information is, for example, information indicating the colorimetric values ​​of each patch on a color chart when color correction is performed by the preprocessing unit 120 using a color chart.

[0027] The memory unit 160 may store, as pre-processing setting information, white unevenness information such as a white paper image captured in advance to be used for color unevenness (shading) correction, black unevenness information such as a black paper image captured in advance to be used for shadow correction, and parameters for optimizing sharpness and contrast enhancement processing for each target when there are multiple targets for biometric information estimation.

[0028] Estimated setting information is stored in the storage unit 160. The estimated setting information is conversion information for estimating biological information when the specimen is a reagent. This conversion information is information showing a correspondence table between color information of the captured image for each reagent and the actual color of the reagent. Furthermore, the estimated setting information is conversion information for estimating biological information when the specimen is a body part, a blood component, or the like. This conversion information is, for example, information showing a correspondence table between color information of the captured image for each nail or blood component and the actual color of the reagent.

[0029] The biological information estimation system 1 may include a biological information identification unit that identifies the biological information estimated by the estimation unit 140. The biological information identification unit may be the input unit 220 that acquires information that identifies the biological information in response to a user's operation. The biological information identification unit may be the image acquisition unit 110 that acquires information that identifies the biological information input in response to a user's operation. The preprocessing unit 120 performs color correction on the subject corresponding to the biological information identified by the biological information identification unit. The subject corresponding to the biological information is a subject suitable for the identified biological information. The subject suitable for the biological information is, for example, a part of a patient's body for measuring blood pressure or a urine test strip for measuring urine test values.

[0030] The biological information estimation system 1 may include a subject identification unit that identifies a subject. The subject identification unit may be the input unit 220 that acquires information that identifies the subject in response to a user's operation. The subject identification unit may be the image acquisition unit 110 that acquires information that identifies the subject input in response to a user's operation. The preprocessing unit 120 performs color correction on the captured image in accordance with the subject identified by the subject identification unit. The extraction unit 130 estimates a biological information value based on the subject information corresponding to the subject identified by the subject identification unit.

[0031] The preprocessing unit 120 may switch conversion information indicating the correspondence between the color information of the captured image and the colorimetric values ​​of the object based on the type of object identified by the object identification unit. The conversion information is stored in the storage unit 160 as preprocessing setting information. The conversion information is information for converting the color information of the captured image and includes information indicating the correspondence between the pixel values ​​of the captured image and the colorimetric values ​​of the actual object. The correspondence between the pixel values ​​of the captured image and the colorimetric values ​​of the actual object depends on, for example, the color characteristics of the object. Therefore, conversion information is prepared for each object according to the color characteristics of the object. The color characteristics of the object are the tendency of spectral reflectance. The color characteristics of the object may be predicted using a learned spectral reflectance estimation model obtained by learning the tendency of the spectral reflectance of the object.

[0032] The preprocessing unit 120 may estimate the spectral reflectance of the captured image by inputting it into a machine learning model (trained spectral reflectance estimation model) trained using color information from the captured image specific to the subject and the spectral reflectance of the actual object, and may perform color correction on the captured image based on the colorimetric values ​​of the subject calculated from the estimated spectral reflectance. The trained spectral reflectance estimation model is a machine learning model trained using color information from the captured image specific to the subject and the color (spectral reflectance, correct value) of the actual object. Using the trained spectral reflectance estimation model, it is possible to estimate the spectral reflectance with high accuracy. Alternatively, the trained spectral reflectance estimation model may be a general-purpose trained spectral reflectance estimation model trained using a dataset that collects general captured images and color information (correct value) of the object without specifying the object. Using the estimated spectral reflectance, the color (colorimetric value) of the object can be obtained by applying the spectral distribution of the observation illumination and a color matching function. Here, the machine learning model for estimating the spectral reflectance may be a neural network, a method based on statistical analysis such as principal component analysis, or a machine learning algorithm such as multiple regression analysis.

[0033] The preprocessing unit 120 may estimate related information about the image capture from the captured image and perform color correction based on the estimated related information. For example, the preprocessing unit 120 may estimate the illumination color (or light source color or spectral distribution) around the subject as related information and perform color correction based on the estimated illumination color. The preprocessing unit 120 can estimate the illumination color (or light source color or spectral distribution) around the subject by referring to parameters necessary for estimating the illumination color from the captured image, for example, by estimating the time of day and whether the image was captured outdoors or indoors based on time information and latitude and longitude information added to the captured image. The preprocessing unit 120 may, for example, refer to conversion information prepared according to the estimated illumination color and correct the color information of the captured image to approximate the actual color of the subject. The preprocessing unit 120 may perform color correction based on camera model information included in the metadata of the captured image as related information. The camera model information includes camera color characteristic information such as the camera's spectral sensitivity and gradation characteristics according to the camera model. The preprocessing unit 120, for example, refers to conversion information prepared according to camera model information extracted from the metadata and corrects the color information of the captured image to approximate the actual color of the subject. Specifically, the preprocessing unit 120 estimates the spectral reflectance from the camera model information using the camera spectral sensitivity for each camera model stored in the storage unit 160, and can switch the conversion information based on the estimated spectral reflectance.

[0034] The preprocessing unit 120 may estimate a spectral information image indicating the spectral reflectance of the subject from an RGB image, which is a captured image of the subject, using a machine learning model. The preprocessing unit 120 inputs the captured image, the spectral sensitivity of the camera device 210 that captured the captured image, and the spectral distribution of the light source in the environment where the captured image was captured into the machine learning model, and estimates the spectral information image. This allows the preprocessing unit 120 to perform preprocessing on the captured image taking into account the spectral sensitivity of the camera device 210 and the light source in the environment where the captured image was captured.

[0035] The pre-processing unit 120 may perform color correction based on registration information received based on a user's operation. The registration information may be, for example, an illumination spectral distribution indicating an illumination color, etc., and the pre-processing unit 120 may perform color correction based on the illumination color. For example, the pre-processing unit 120 refers to conversion information that converts color information of the captured image corresponding to the illumination color into the color of the actual object, and corrects the color information of the captured image to approximate the color of the actual object.

[0036] The registered information is terminal information, and the preprocessing unit 120 may perform color correction based on camera model information corresponding to the terminal information. The preprocessing unit 120 refers to conversion information that converts the color information of the captured image corresponding to the registered camera model information into the color of the actual subject, and corrects the color information of the captured image so that it approaches the color of the actual subject.

[0037] The subject information may indicate an operation that induces a temporal change in the subject, and the estimation unit 140 may estimate the biometric information value based on a change in color information corresponding to the temporal change in the subject. The operation that induces a temporal change in the subject induces a color change or color enhancement in a part of the subject. The terminal device 200 prompts the patient to perform an operation that induces a temporal change before or during capture of the captured image, and transmits the subject information to the biometric information estimation device 100.

[0038] The subject information may be information indicating an action such as compression. For example, the action information may be information indicating actions such as before and after drug administration, before and after drug application, before and after application of electrical stimulation, before and after heating, before and after heating, before and after cooling, changes in indoor temperature and humidity, changes in oxygen concentration, before and after irradiation with light of a specific wavelength, and before and after administration of an allergen. The terminal device 200 may display a message to capture an image of the patient's nail, display a message to compress the nail while the image of the nail is captured, start capturing video images before and after compression, release the compression, and control the camera device 210 to continue capturing video images after the compression is released. In this way, the biological information estimation device 100 acquires the action of compressing a portion of the skin as subject information, and the terminal device 200 can transmit the video image of the compression of the portion of the skin and the subject information to the biological information estimation device 100. The subject information may also be information measuring the time it takes for the nail color to return to its pre-image color.

[0039] The estimation unit 140 estimates the biometric information based on the subject information and the color information. Specifically, the estimation unit 140 transmits subject information indicating the actions of compressing and releasing the nail to the terminal device 200, and the camera device 210 transmits captured images capturing the process of compressing and releasing the nail to the biometric information estimation device 100. The extraction unit 130 acquires color information from the captured images, and the estimation unit 140 estimates the biometric information from the subject information and the color information.

[0040] The subject information is biometric information of an individual user, and the estimation unit 140 may estimate the biometric information value based on the color information and the biometric information. The biometric information of an individual user includes the user's gender, age, medical history, biometric information values ​​from a previous test, etc.

[0041] The subject information is bio-related information or environmental information of an individual user, and the estimation unit 140 may estimate the bio-information value based on the color information and the bio-related information.

[0042] For example, in the case of a test using a reagent such as a test strip, the biological information is a correspondence table of test values ​​for each test item and the colors of the reaction reagent corresponding to the test item. Examples of test items include glucose test values ​​and pH values. The colors of the reaction reagent are, for example, colorimetric values ​​XYZ, L*a*b*, etc. The correspondence table includes, for example, colorimetric values ​​such as "50, 4, -1" for a glucose test value of "-" and "30, 2, -5" for a test value of "1+". For example, the correspondence table includes colorimetric values ​​such as "20, 0, -2" for a pH test value of "6.5," "40, 0, 2," for a pH test value of "7.0," and "60, 2, -5" for a pH test value of "7.5."

[0043] The biological information is, for example, a correspondence table between biological information and the color of a body part corresponding to the biological information. The color of the reaction reagent is, for example, colorimetric values ​​XYZ, L*a*b*, etc. For example, when the biological information is "hemoglobin" and the biological part is a nail, the correspondence table includes colorimetric values ​​"20,0,-2" for a test value of "6.0," colorimetric values ​​"40,0,2" for a test value of "8.0," and colorimetric values ​​"60,2,-5" for a test value of "10.0." For example, when the biological information is "SpO2" and the biological part is a finger, the correspondence table includes colorimetric values ​​"20,0,-2" for a test value of "100," colorimetric values ​​"40,0,2" for a test value of "98," and colorimetric values ​​"60,2,-5" for a test value of "96."

[0044] The estimation unit 140 can acquire colorimetric values ​​as color information from the extraction unit 130 and acquire test values ​​corresponding to the colorimetric values. For example, if the color information acquired from the extraction unit 130 is a colorimetric value of "50, 5, -1," the estimation unit 140 can classify and acquire the test value "-" that corresponds to the colorimetric value "50, 4, -1," which is closest in Euclidean distance from the correspondence table. The estimation unit 140 may also convert the test values ​​into continuous values ​​by performing regression processing. The classification or regression processing of the test values ​​by the estimation unit 140 may be performed using a machine learning method such as logistic regression analysis, multiple regression analysis, a support vector machine, or a neural network.

[0045] The image acquisition unit 110 acquires an image of the subject and a color chart image of a color chart, and the preprocessing unit 120 may refer to conversion information that converts color information included in the image of the subject using the color chart image and correct the color information included in the image based on the conversion information. The color chart image has areas corresponding to multiple patches. The preprocessing unit 120 compares the color information included in the color chart image with the color information of the captured image and identifies areas in the color chart to match, thereby estimating the actual color of the subject.

[0046] The bioinformation estimation device 100 may include a health state estimation unit that estimates a health state based on the bioinformation values ​​estimated by the estimation unit 140. The health state estimation unit may be implemented, for example, as one function of the estimation unit 140. The health state estimation unit estimates the user's health state based on the bioinformation values ​​estimated by the estimation unit 140. The health state estimation unit may estimate a disease name as the health state. The health state estimation unit, for example, refers to table information stored in the storage unit 160 that associates bioinformation values ​​with health states.

[0047] As described above, the biological information estimation system 1 according to the first embodiment performs predetermined preprocessing on a captured image, extracts color information of a subject from the preprocessed captured image, and estimates a biological information value based on the color information and the subject information. The biological information estimation system 1 extracts preprocessed color information and estimates biological information simply by capturing an image. This allows the biological information estimation system 1 to estimate biological information with high accuracy. Furthermore, the biological information estimation system 1 can reduce the effort required for a user to undergo medical treatment or testing.

[0048] Second Embodiment FIG. 2 is a block diagram showing an example of the configuration of a biometric information estimation system 1A according to a second embodiment. The biometric information estimation system 1A described in the second embodiment may be the same as the biometric information estimation system according to the above-described embodiment. The biometric information estimation system 1A is a system in which a captured image is transmitted to a biometric information estimation device 100A located remotely from a living body, such as a patient, and the biometric information estimation device 100A acquires color information from the captured image and estimates biometric information with high accuracy. In the second embodiment, the subject whose biometric information is to be estimated is a patient undergoing a health checkup, but is not limited to this and may be a human body or another living body.

[0049] The biometric information estimation system 1A includes, for example, a biometric information estimation device 100A and a terminal device 200A equipped with a camera device 210A. The biometric information estimation device 100A and the terminal device 200A are connected via, for example, a communication network. The biometric information estimation device 100A and the terminal device 200A each have a communication interface such as a network interface card (NIC) or a wireless communication module for connecting to a network such as the Internet. The communication network may include, for example, a general-purpose network such as the Internet, and a private network such as local 5G or Wi-Fi (registered trademark).

[0050] The terminal device 200A is an information processing device such as a smartphone or personal computer used by the patient. The terminal device 200A includes a camera device 210A. The camera device 210A operates based on an operation received by the terminal device 200A and generates a captured image of the patient. The terminal device 200A transmits the captured image to the biological information estimation device 100. Note that in the present embodiment, the terminal device 200A and the biological information estimation device 100A are separate entities, but this is not limiting, and the functions of the biological information estimation device 100A may be built into the terminal device 200A.

[0051] The camera device 210A may be a camera built into a smartphone, a single-lens reflex camera, a webcam, or the like. The terminal device 200A may transmit information indicating the type of camera along with the captured image. The captured image may be a still image or a moving image. The captured image may be a plurality of images generated by interval photography. The captured image may be an image of a portion of a patient. The portion of the patient may be, for example, a nail, the back of the hand, the face, the lips, the conjunctiva, the eyes, the eyelids, or the tongue. The captured image may be an image of multiple parts of the patient. The captured image may be a plurality of images captured with different subject distances, with or without a flash, at different angles, etc. The captured image may be an image of the patient together with a reference object such as a color chart or white board for color correction. The captured image may include both an image including the patient and an image of the reference object. The captured image may include depth information measured using a depth sensor, etc.

[0052] The biometric information estimation device 100A is an information processing device that communicates with other devices and performs various processes. The biometric information estimation device 100A includes, for example, an image acquisition unit 110A, a color information acquisition unit 120A, and a biometric information estimation unit 130A. The image acquisition unit 110A, the color information acquisition unit 120A, and the biometric information estimation unit 130A are functional units realized by a processor such as a CPU (Central Processing Unit) executing a program stored in a program memory.

[0053] The image acquisition unit 110A acquires an image of a patient.

[0054] The color information acquisition unit 120A corrects color information included in the captured image and acquires the corrected color information. The color information indicates the patient's color for estimating biological information. The color information may be RGB values ​​with white balance correction or RGB values ​​reproduced in a specific color space, such as sRGB values. The color information may be spectral reflectance, XYZ values ​​in the XYZ color system, L*a*b* values ​​in the L*a*b* color system, erythema index, melanin index, absorbance, spectral distribution at a specific wavelength, a representative value for a specific region of the subject, or time-series information indicating changes over time in a video or interval-captured image with values ​​for each pixel. The color information may also include two or more of these types of information. The color information may be a single value, such as an average value for the extracted region, or image data (two-dimensional color information distribution) for a small region including the extracted region.

[0055] The biological information estimation unit 130A estimates the patient's biological information based on color information. The biological information estimation unit 130A estimates the biological information using a method such as multiple regression analysis, polynomial approximation, or machine learning. The biological information estimation device 100A outputs the biological information. The biological information estimation device 100A may output the biological information to, for example, a terminal device 200A, a device that stores the biological information, or an inspection device that performs various inspections using the biological information. Note that the biological information estimation unit 130A may simultaneously estimate multiple pieces of biological information based on color information.

[0056] FIG. 3 is a sequence diagram illustrating an example of a processing procedure of the biometric information estimation system 1A according to the second embodiment. First, the terminal device 200A captures an image of the patient using the camera device 210A (step ST10) and transmits the captured image to the biometric information estimation device 100A. The image acquisition unit 110A acquires the captured image and outputs it to the color information acquisition unit 120A. The color information acquisition unit 120A corrects color information included in the captured image and acquires the corrected color information (step ST12). The color information acquisition unit 120A outputs the acquired color information to the biometric information estimation unit 130A. The biometric information estimation unit 130A estimates the patient's biometric information based on the color information (step ST14). The biometric information estimation device 100A transmits the biometric information estimated by the biometric information estimation unit 130A to the terminal device 200A. The terminal device 200A presents the biometric information to the patient by displaying it using the biometric information (step ST16).

[0057] 4 is a diagram showing an example of color information acquisition unit 120A in the second embodiment. Color information acquisition unit 120A may include at least one of correction unit 121A, conversion unit 122A, and estimation units 123A, 124A, and 125A. Correction unit 121A, conversion unit 122A, and estimation units 123A, 124A, and 125A are functional units realized by a processor such as a CPU executing a program stored in a program memory.

[0058] The correction unit 121A estimates the illumination color of the space in which the patient is imaged based on the captured image, and corrects the white balance of the RGB information included in the captured image. The correction unit 121A outputs the captured image including the color information with the white balance corrected to the biological information estimation unit 130A.

[0059] The image acquisition unit 110A acquires a color chart image obtained by capturing an image of a color chart and a captured image of a patient. The conversion unit 122A creates conversion information for converting color information included in the captured image of the patient using the color chart image and converts the color information included in the captured image based on the conversion information. The color chart is, for example, a plate-shaped object on which color samples are arranged. The conversion unit 122A creates a color conversion matrix (conversion information) based on the relationship between the colors indicated by the color chart and the colors of the color chart included in the captured image. The conversion unit 122A creates the color conversion matrix using the color chart by, for example, performing multiple regression analysis. The conversion unit 122A can convert the captured image of the patient based on the conversion information to the colors indicated by the color chart. The conversion unit 122A outputs the captured image with the converted color information to the biological information estimation unit 130A.

[0060] The estimation unit 123A estimates color information based on output information from a machine learning engine that estimates parameters that affect the color of a captured image when a patient is imaged. The parameters that affect the color of the captured image are, for example, spectral reflectance and absorbance. The machine learning engine is a parameter estimation engine that is trained using the captured image and parameters (spectral reflectance and absorbance) as training data. The parameter estimation engine estimates parameters in response to input of the captured image. The estimation unit 123A estimates color information of the captured image using the parameters estimated by the parameter estimation engine. The estimation unit 123A may generate other parameters, such as an erythema index and a melanin index, using the absorbance estimated by the parameter estimation engine. The estimation unit 123A estimates color information of the captured image using other parameters generated using the parameters estimated by the parameter estimation engine.

[0061] The estimation unit 124A estimates correct color information based on the captured image acquired by the image acquisition unit 110A and color information about the camera that acquired the captured image, such as spectral sensitivity. The terminal device 200A transmits camera information related to color information, such as the type and specifications (sensitivity, etc.) of the camera device 210A, along with the captured image, to the biological information estimation device 100. The estimation unit 124A estimates the color information of the patient based on the camera information included in the captured image.

[0062] The estimation unit 125A may estimate color information based on body part information acquired by the image acquisition unit 110A or output information of a machine learning engine that recognizes a patient's body part. The body part information is information indicating a body part specified by the user, such as a nail, back of the hand, face, or conjunctiva. The machine learning engine that recognizes a patient's body part is a body part estimation engine that has trained using the captured image and body part information for identifying the body part as training data. The body part estimation engine estimates the body part in response to the input of the captured image. The estimation unit 125A estimates color information of the captured image in response to the body part indicated by the body part information or the body part estimated by the body part estimation engine.

[0063] As described above, according to the embodiment of the biometric information estimation system 1A, an image of a patient is acquired, color information contained in the image is corrected, and the patient's biometric information is estimated based on the corrected color information. Therefore, even if the image varies depending on the camera device 210A that captured the image and the environment around the patient, color information can be acquired from the image and biometric information can be estimated with high accuracy by correcting the color information.

[0064] FIG. 5 is a block diagram showing an example of a biological information estimation system 1B according to an embodiment. The biological information estimation system 1B includes a biological information detection unit 220A that acquires biological information detected from a patient. The biological information detection unit 220A may be a device that measures the patient's biological information, such as a smart watch, or may be a device that receives operation by the patient or the like to input biological information. The biological information detection unit 220A detects, for example, biological information different from the biological information estimated by the biological information estimation unit 130B. The biological information detected by the biological information detection unit 220A is biological information that assists the biological information estimation unit 130B in estimating the biological information. The biological information detected by the biological information detection unit 220A may be body temperature, oxygen saturation, pulse, blood pressure, electrocardiogram, etc., oxygen saturation measured by a pulse oximeter, or manually inputted age, gender, blood type, height, weight, disease, etc. Furthermore, the biological information detection unit 220A may be communicatively connected to a server device and may acquire blood test results (hemoglobin), urine test results, various test results (blood pressure, melanin, electrocardiogram, etc.), medical history, etc. from the server device. The biological information detection unit 220A may estimate biological information from captured images acquired by the camera device 210A or captured images acquired by other means through internal processing. The biological information detection unit 220A may estimate, for example, burns, trauma, melanin, etc. from the captured images.

[0065] The biological information estimation unit 130B estimates biological information based on the biological detection information and color information acquired from the biological information detection unit 220A. Specifically, the biological information detection unit 220A transmits a captured image of a nail to the biological information estimation device 100B. The biological information detection unit 220A transmits oxygen saturation acquired by a pulse oximeter or a smartwatch to the biological information estimation device 100B as biological detection information. The color information acquisition unit 120A performs color correction on the captured image of the nail acquired by the image acquisition unit 110A to sRGB values ​​and averages the RGB values ​​of the nail. The biological information estimation unit 130B estimates the hemoglobin value using polynomial approximation from the color information corrected by the color information acquisition unit 120A and the oxygen saturation transmitted from the biological information detection unit 220A.

[0066] As described above, according to the embodiment of the biometric information estimation device 100B, in addition to the captured image of the patient, biometric detection information obtained by the biometric information detection unit 220A is also acquired, and the patient's biometric information can be estimated based on the corrected color information and the biometric detection information.

[0067] 6 is a block diagram showing an example of a biological information estimation system 1C according to the second embodiment. The biological information estimation system 1C includes a motion information acquisition unit 230A that acquires motion information of a patient when an image is captured. In the biological information estimation system 1C, a color information acquisition unit 120C acquires color information of the patient based on the captured image and the motion information, and a biological information estimation unit 130C estimates the biological information based on the color information.

[0068] The patient's motion information when the captured image is captured is information indicating the patient's motion that induces a color change or color enhancement. The motion information acquisition unit 230A may prompt the patient to perform an operation to induce a change over time before or during capture of the captured image. The motion information acquisition unit 230A may, for example, display a message prompting the patient to perform an operation on a screen on which the captured image is captured by the camera device 210A, and acquire motion information indicating the motion corresponding to the captured image acquired after the message is displayed.

[0069] The operation information may be information indicating an operation such as compression. For example, the operation information may be information indicating an operation from before administration of a drug to after administration, from before application of a drug to after application, from before application of an electrical stimulus to after application, from before heating to after heating, from before cooling to after cooling, a change in indoor temperature and humidity, a change in oxygen concentration, from before irradiation with light of a specific wavelength to after irradiation, from before administration of an allergen to after administration, etc. Furthermore, the operation information may be an amount of operation during shooting when capturing a video.

[0070] Specifically, the motion information acquisition unit 230A may control the camera device 210A to display a message to capture an image of the patient's nail, display a message to compress the nail while the image of the nail is captured, start capturing moving images before and after compression, release the compression, and continue capturing moving images after the compression is released. This allows the motion information acquisition unit 230A to acquire motion information indicating the action of compressing a portion of the skin, and the terminal device 200A can transmit the motion information and the moving image when the portion of the skin is compressed to the biological information estimation device 100C. The motion information acquisition unit 230A may also measure the time it takes for the color of the nail to return to its state before the image was captured and acquire this as motion information.

[0071] The biometric information estimation unit 130C estimates the biometric information based on the motion information and color information acquired from the motion information acquisition unit 230A. Specifically, the motion information acquisition unit 230A transmits motion information indicating the motion of compressing and releasing the nail to the terminal device 200A, and the camera device 210A transmits captured images capturing the process from compressing the nail to releasing it to the biometric information estimation device 100C. The color information acquisition unit 120C acquires color information from the captured images, and the biometric information estimation unit 130C estimates the biometric information from the motion information and color information.

[0072] As described above, according to the biometric information estimation device 100C in the second embodiment, in addition to the captured image of the patient, movement information can be acquired, and the patient's biometric information can be estimated based on the corrected color information and movement information.

[0073] Third Embodiment FIG. 7 is a block diagram showing an example of the configuration of a testing system 1D according to a third embodiment. The testing system 1D described in the third embodiment may be the same as the biological information estimation system described in the above-described embodiments. The testing system 1D is a system that transmits a captured image of a reagent and reagent-specific information to a testing device 100D located remotely from a reagent user, and the testing device 100D acquires color information from the captured image to estimate a test value with high accuracy. The reagent in the embodiment may be, for example, a urine test strip impregnated with a patient's urine, but is not limited thereto and may be any known reagent whose test value is read by its color.

[0074] The inspection system 1D includes, for example, an inspection device 100D and a terminal device 200B. The inspection device 100D and the terminal device 200D are connected, for example, via a communication network. The inspection device 100D and the terminal device 200D have a communication interface, such as a network interface card (NIC) or a wireless communication module, for connecting to a network such as the Internet. The communication network may include, for example, a general-purpose network such as the Internet, and a private network such as local 5G or Wi-Fi (registered trademark).

[0075] The terminal device 200D is an information processing device such as a smartphone or personal computer used by a reagent user. The terminal device 200D includes, for example, a camera device 202D, a processing unit 204D, an operation unit 206D, and a display unit 208D. The camera device 202D operates based on operations received by the terminal device 200D and generates captured images of the reagent. The processing unit 204 is a processor that executes application software to perform various processes and controls. The operation unit 206D is an operation interface such as a touch panel or buttons. The display unit 208D is a liquid crystal display or the like. Note that, in this embodiment, the terminal device 200D and the testing device 100D are separate entities, but this is not limited thereto, and the functions of the testing device 100D may be built into the terminal device 200D.

[0076] The processing unit 204D may read a three-dimensional code or the like written on the reagent using the camera device 202D and determine reagent identification information such as the reagent manufacturer through image recognition processing. The processing unit 204D may also generate reagent identification information in response to receiving an operation to input information about the reagent via the operation unit 206D. If the processing unit 204D is configured to execute dedicated application software corresponding to the reagent, the processing unit 204D may execute the application software to acquire an image and transmit the image and reagent identification information to the testing device 100D. The processing unit 204D may also recognize AR markers, registration marks, etc. to generate reagent identification information. The processing unit 204D may also predict the degree of deterioration of the paper based on the image and check whether the reagent is expired.

[0077] The camera device 202D may be a camera built into a smartphone, a single-lens reflex camera, a webcam, or the like. The terminal device 200D may transmit information indicating the type of camera along with the captured image. The captured image may be a still image of the reagent or a moving image of the reagent. The captured image may be, for example, a plurality of images generated by taking interval photographs every elapsed seconds to observe the reaction of the reagent. The captured image may be a plurality of images captured by changing the subject distance, the presence or absence of a flash, the angle, etc. The captured image may be an image captured with the capturing environment (lighting, angle of view) fixed depending on the case. The captured image may be an image of the reagent as well as an image of a reference object such as a color chart or white board for color correction. The captured image may include both an image including the reagent and an image of the reference object. The captured image may include depth information measured using a depth sensor, etc.

[0078] The inspection device 100D is an information processing device that communicates with other devices and performs various processes. The inspection device 100D includes, for example, an acquisition unit 110D, a color information acquisition unit 120D, a test value estimation unit 130D, a health condition estimation unit 140D, a display information output unit 150D, and a storage unit 160D. The acquisition unit 110D, the color information acquisition unit 120D, the test value estimation unit 130D, the health condition estimation unit 140D, and the display information output unit 150D are functional units realized by a processor, such as a central processing unit (CPU), executing a program stored in a program memory. The storage unit 160D is configured by any combination of storage media, such as a hard disk drive (HDD) and a solid state drive (SSD).

[0079] The acquisition unit 110D functions as an image acquisition unit that acquires a captured image of a reagent, and as a reagent identification information acquisition unit that acquires reagent identification information for identifying a reagent.

[0080] The color information acquisition unit 120D corrects the color information contained in the captured image and acquires the corrected color information. The color information indicates the color of the reagent used to estimate the test value. The color information may be RGB values ​​with white balance correction, RGB values ​​reproduced in a specific color space such as sRGB values, etc. The color information may be spectral reflectance, XYZ values ​​in the XYZ color system, L*a*b* values ​​in the L*a*b* color system, erythema index, melanin index, absorbance, spectral distribution at a specific wavelength, a representative value for a specific region of the subject, or time-series information indicating changes over time in a video or interval-captured image with values ​​for each pixel. It may also include two or more of these pieces of information. The color information may be a single value, such as an average value for the extracted area, or image data (two-dimensional color information distribution) of a small region including the extracted area.

[0081] The color information acquisition unit 120D includes, for example, an image preprocessing unit 122D, an image color correction unit 124D, and a reagent color extraction unit 126D. The image preprocessing unit 122D performs preprocessing on the captured image. The preprocessing is a step prior to color correction and may include shading (illumination unevenness) correction, specular reflection correction, white balance estimation, etc. The preprocessing may include processing to smoothly extract the color of the reagent (keystone correction, processing to identify the reagent portion of the image and cut it out into a rectangle). The image color correction unit 124D corrects the color information contained in the captured image that has been preprocessed, thereby acquiring color-corrected image information that accurately represents the reagent color. Color correction will be described with reference to FIG. 10. The reagent color extraction unit 126D extracts the color of the reagent based on the corrected color information. In the case of a reagent with multiple test areas as shown in FIG. 7, color information is extracted for each area. When the reaction time differs for each reagent and the recommended testing time differs, the reagent color extraction unit 126D extracts color information from the images captured at each recommended testing time among multiple images captured at intervals.In the case of a video, color information is extracted from the frame at the recommended time.The reagent color extraction unit 126D may detect the pattern (sedimentation) of the reagent instead of the color.The reagent color extraction unit 126D may identify the reagent based on the extracted color.

[0082] The test value estimation unit 130D estimates the test value of the reagent based on the reagent identification information and color information. If the reagent has multiple test areas, the test value estimation unit 130D estimates the test value for each test area. The test value estimation unit 130D determines the test value based on the distance between the color information of each test area extracted by the reagent color extraction unit 126D and the representative color for each test item. The test value estimation unit 130D may estimate the test value using techniques such as multiple regression analysis, polynomial approximation, or machine learning (CNN, SVM, etc.).

[0083] The test value estimation unit 130D may input the captured image and reagent identification information acquired by the acquisition unit 110D into a machine learning engine that has learned the captured image of the reagent, the reagent identification information, and the test value, and output the test value based on the output from the machine learning engine. Parameters for realizing the machine learning engine may be stored in the storage unit 160D as a trained engine for each reagent. During testing, the test value estimation unit 130D can read, for example, a trained model corresponding to the reagent identification information from the storage unit 160D, input color information into the trained model, and estimate the test value based on the output of the trained model.

[0084] The health condition estimation unit 140D estimates the health condition of the reagent user based on the test values ​​estimated by the test value estimation unit 130D. The health condition estimation unit 140D may estimate a disease name as the health condition. The display information output unit 150D outputs display information for displaying the test values ​​estimated by the test value estimation unit 130D and the health condition estimated by the health condition estimation unit 140D to the terminal device 200D. The storage unit 160D stores information processed by the acquisition unit 110D, the color information acquisition unit 120D, the test value estimation unit 130D, the health condition estimation unit 140D, and the display information output unit 150D.

[0085] The reagent identification information may be any information that identifies a reagent, such as information that identifies the type of reagent. In this case, test value estimation unit 130D estimates the test value of the reagent based on the type and color information of the reagent.

[0086] FIG. 8 is a sequence diagram showing an example of the processing procedure of the testing system 1D according to the third embodiment. First, the terminal device 200D captures an image of a reagent user using the camera device 202D, transmits the captured image to the testing device 100D, and transmits reagent identification information to the testing device 100D. The acquisition unit 110D acquires the captured image, and the image preprocessing unit 122D performs preprocessing on the captured image and outputs the preprocessed image to the image color correction unit 124D. The image color correction unit 124D acquires information necessary for correction from the storage unit 160D and corrects the color information of the preprocessed image. The image color correction unit 124D outputs the color-corrected image to the reagent color extraction unit 126. The reagent color extraction unit 126D extracts the color of the reagent from the color-corrected image and outputs the color information to the test value estimation unit 130D. Health condition estimation unit 140D estimates the health condition based on the test values, and display information output unit 150D generates display information including the test values ​​and the health condition and transmits it to terminal device 200D. Note that if test value estimation unit 130D fails to detect the color of the reagent or if health condition estimation unit 140D fails to test the health condition, test value estimation unit 130D may cause display information output unit 150D to output an error result to terminal device 200D. Display information output unit 150D may generate display information prompting the user to retake the captured image together with the error result and transmit it to terminal device 200.

[0087] FIG. 9 illustrates an example of a display screen of the terminal device 200D according to the third embodiment. The terminal device 200D first launches application software to display the initial screen (user page) shown in FIG. 9A. Upon detecting that the "Start Test" button has been selected via the operation unit 206D, the terminal device 200D displays the test setting screen shown in FIG. 9B. The test setting screen shown in FIG. 9B prompts the user to select a test strip (reagent) from a drop-down list and includes test items corresponding to the reagent. Upon receiving an operation to select a camera display button after a reagent has been selected, the terminal device 200D displays the image capture adjustment screen shown in FIG. 9C. Furthermore, the terminal device 200D generates reagent identification information corresponding to the selected reagent. The image capture adjustment screen shown in FIG. 9C displays, for example, a message such as "Please make sure the test strip fits within the frame" and a guide frame superimposed on top of the camera image. When the test start button is selected while the imaging adjustment screen is displayed, the terminal device 200D acquires the camera image within the frame as a captured image and transmits the captured image and reagent identification information to the testing device 100D. The terminal device 200D receives display information from the testing device 100D and displays the test result screen of FIG. 9(D) or the test failure screen of FIG. 9(E) based on the display information. The test result screen of FIG. 9(D) includes a reagent image as the captured result, test results (numerical values, health status), a reference image, a re-photograph button, and a save result button. This allows the reagent user to know the test values ​​and health status. When the re-photograph button is selected, the terminal device 200D displays the imaging adjustment screen of FIG. 9(C). When the save result button is selected, the terminal device 200D displays the initial screen of FIG. 9(A). When the save result button is selected, the terminal device 200D may transmit test results, etc. to other application software or a server device to provide services linked to the test results. 9(E) includes error information, a cancel button, and a retake button, which allows the terminal device 200D to prompt the user to take a retake of the reagent.

[0088] 10 is a diagram showing an example of an image color correction unit 124D according to the third embodiment. The image color correction unit 124D of the color information acquisition unit 120D may include at least one of a correction unit 124a, a conversion unit 124b, an estimation unit 124c, and an estimation unit 124d. The correction unit 124a, the conversion unit 124b, the estimation unit 124c, and the estimation unit 124d are functional units realized by a processor such as a CPU executing a program stored in a program memory.

[0089] The correction unit 124a estimates the illumination color of the space in which the reagent was captured based on the captured image, and corrects the white balance of the RGB information contained in the captured image. It is desirable for the captured image to have a secondary background, such as white, in the background, and a message prompting the camera device 202D to capture the image in an environment including a white background may be displayed. The correction unit 124a outputs the captured image, including color information with corrected white balance, to the reagent color extraction unit 126. The correction unit 124a may change the correction process depending on the type of reagent.

[0090] The acquisition unit 110D acquires a color chart image of a color chart and a captured image of a reagent. The conversion unit 124b uses the color chart image to create conversion information for converting color information included in the captured image of the reagent, and converts the color information included in the captured image based on the conversion information. The color chart is, for example, a plate-shaped object on which color samples are arranged. Note that, instead of a color chart, a captured image of the reagent before impregnation may be acquired and used as a color chart or reference. The conversion unit 124b creates a color conversion matrix (conversion information) based on the relationship between the colors indicated by the color chart and the colors of the color chart included in the captured image. The conversion unit 124b creates a color conversion matrix using the color chart by, for example, performing multiple regression analysis. The conversion unit 124b can convert the captured image of the reagent based on the conversion information to convert it into the colors indicated by the color chart. The conversion unit 124b outputs the captured image of the converted color information to the reagent color extraction unit 126D. Note that the conversion unit 124b may create conversion information for each type of reagent.

[0091] The estimation unit 124c estimates color information based on output information from a machine learning engine that estimates parameters that affect the color of a captured image when an image of a reagent is captured. The parameters that affect the color of the captured image are, for example, spectral reflectance and absorbance. The machine learning engine is a parameter estimation engine that is trained using the captured image and parameters (spectral reflectance and absorbance) as training data. The parameter estimation engine estimates parameters in response to input of the captured image. The estimation unit 124c estimates color information of the captured image using the parameters estimated by the parameter estimation engine. The estimation unit 124c may generate other parameters using the absorbance estimated by the parameter estimation engine. The estimation unit 124c estimates color information of the captured image using other parameters generated using the parameters estimated by the parameter estimation engine.

[0092] The estimation unit 124d estimates correct color information based on the captured image acquired by the acquisition unit 110D and color information about the camera that captured the captured image, such as spectral sensitivity. The terminal device 200 transmits camera information related to color information, such as the type and specifications (sensitivity, etc.) of the camera device 202D, along with the captured image to the inspection device 100D. The estimation unit 124d estimates the color information of the reagent user based on the camera information included in the captured image.

[0093] As described above, according to the testing system 1D of the embodiment, an image of a reagent user is acquired, color information contained in the image is corrected, and biological information of the reagent user is estimated based on the corrected color information. Therefore, even if the image varies depending on the camera device 202D that captured the image and the environment around the reagent user, color information can be acquired from the image and biological information can be estimated with high accuracy by correcting the color information.

[0094] 11 is a block diagram showing an example of a testing system 1E according to the third embodiment. The testing system 1E includes a biological information detection unit 210D that acquires biological detection information detected from a reagent user. The biological information detection unit 210D may be a device that measures the biological information of the reagent user, such as a smart watch, or may be a device that receives an operation from the reagent user to input the biological information. The biological information detected by the biological information detection unit 210D is biological information that assists the health condition estimation unit 140E in estimating a health condition.

[0095] The biological information detected by the biological information detection unit 210D may be personal information such as gender, height, weight, age, medical history, etc., values ​​detected by biological sensors such as body temperature, oxygen saturation, pulse, blood pressure, electrocardiogram, etc., or environmental information detected by environmental sensors such as temperature, humidity, time, etc. Of the information detected by the terminal device 200D, environmental information that may cause a change in the color of a reagent may be transmitted to the testing device 100E to assist the test value estimation unit 130E in estimating the test value.

[0096] The biological information detection unit 210D may be communicatively connected to a server device and may acquire blood test results (hemoglobin), urine test results, various test results (melanin, etc.), medical history, etc. from the server device. The biological information detection unit 210D may estimate biological information from captured images acquired by the camera device 202D or captured images acquired by other means through internal processing. The biological information detection unit 210D may estimate, for example, burns, external injuries, melanin, etc. from the captured images.

[0097] Test value estimation unit 130E estimates test values ​​based on environmental information indicating the environment in which the reagent was used, color information, and reagent identification information. Health condition estimation unit 140E estimates the health condition based on the biological information and test values ​​obtained from biological information detection unit 210D.

[0098] As described above, testing device 100E according to the embodiment can acquire not only an image of a reagent but also biological detection information acquired by biological information detection unit 210D, and estimate a health condition based on the corrected color information and the biological information. Furthermore, testing device 100E can estimate a test value with high accuracy based on environmental information indicating the environment in which the reagent was used, color information, and reagent identification information.

[0099] The terminal device and biometric information estimation device in the above-described embodiments may be implemented by a computer. In this case, a program for implementing this function may be recorded on a computer-readable recording medium, and the program may be loaded and executed by a computer system. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client. Furthermore, the program may be designed to implement a portion of the above-described functions, or may be capable of implementing the above-described functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0100] Although the term "unit" is used to explain the configuration of the bioinformation estimation device in the embodiments, the term "unit" may be replaced with other terms that represent the technical configuration, structure, function, action, etc. of the bioinformation estimation device, such as circuit, unit, module, component, etc. Furthermore, the terms "unit," "circuit," "unit," "module," "component," etc. may refer to modules or components that can be stored and / or executed by general-purpose hardware of a computer system (e.g., computer-readable medium, processor device, etc.), and / or hardware configured to execute software objects or software routines.

[0101] Although each embodiment and variant have been described, these are merely examples and are not intended to be limiting. For example, one aspect of the present invention may be realized by combining any of the embodiments or variants, or a part of each embodiment or a part of each variant, with one or more other embodiments or one or more other variants.

[0102] (Additional Notes) The embodiments can also be expressed as follows. (1-1) A biometric information estimation device comprising: an image acquisition unit that acquires a captured image of a living organism; a color information acquisition unit that corrects color information included in the captured image and acquires the corrected color information; and a biometric information estimation unit that estimates biometric information of the living organism based on the color information. (1-2) The biometric information estimation device according to (1-1), in which the color information acquisition unit estimates an illumination color of a space in which the living organism is captured based on the captured image and corrects white balance of RGB information included in the captured image. (1-3) The biometric information estimation device according to (1-2), in which the image acquisition unit acquires a color chart image of a color chart and a captured image of the living organism, and the color information acquisition unit creates conversion information for converting color information included in the captured image of the living organism using the color chart image, and converts the color information included in the captured image based on the conversion information. (1-4) The biological information estimation device according to (1-1), wherein the color information acquisition unit estimates color information based on output information of a machine learning engine that estimates parameters that affect the color of a captured image when a living organism is imaged. (1-5) The biological information estimation device according to (1-1), wherein the color information acquisition unit estimates color information of the living organism based on camera information included in the captured image. (1-6) The biological information estimation device according to (1-1), wherein the color information acquisition unit estimates color information based on body part information acquired by the image acquisition unit or output information of a machine learning engine that recognizes a body part of the living organism. (1-7) The biological information estimation device according to (1-1), further comprising a biological information detection unit that acquires living organism detection information detected from the living organism, and the biological information estimation unit estimates the living organism based on the living organism detection information and the color information acquired by the biological information detection unit. (1-8) The biological information estimation device described in (1-1) includes a motion information acquisition unit that acquires motion information of the living body when the captured image is captured, and the color information acquisition unit acquires color information of the living body based on the captured image and the motion information.(1-9) The biological information estimation device according to (1-1), wherein the image acquisition unit acquires a captured image of a patient as the living body, the color information acquisition unit corrects color information of the patient, and the biological information estimation unit estimates the biological information of the patient. (1-10) A biological information estimation method including the steps of: the biological information estimation device acquiring a captured image of a living body, the biological information estimation device correcting color information included in the captured image and acquiring the corrected color information, and the biological information estimation device estimating the biological information of the living body based on the color information. (1-11) A program causing a computer of an information processing device to execute the steps of: acquiring a captured image of a living body, correcting color information included in the captured image and acquiring the corrected color information, and estimating the biological information of the living body based on the color information. (2-1) An inspection device comprising: an image acquisition unit that acquires a captured image of a reagent; a reagent identification information acquisition unit that acquires reagent identification information for identifying the reagent; a color information acquisition unit that corrects color information included in the captured image and acquires the corrected color information; and an inspection value estimation unit that estimates an inspection value of the reagent based on the reagent identification information and the color information. (2-2) The inspection device described in (2-1), wherein the reagent identification information is information for identifying a type of the reagent, and the inspection value estimation unit estimates the inspection value of the reagent based on the type of the reagent and the color information. (2-3) The inspection device described in (2-1)1, wherein the color information acquisition unit estimates an illumination color of a space in which the reagent was imaged based on the captured image, and corrects the white balance of RGB information included in the captured image. (2-4) The inspection device described in (2-1), wherein the image acquisition unit acquires a color chart image obtained by capturing an image of a color chart and a captured image obtained by capturing an image of a reagent, and the color information acquisition unit creates conversion information for converting color information included in the captured image obtained by capturing an image of the reagent using the color chart image, and converts the color information included in the captured image based on the conversion information.(2-5) The testing device according to (2-1), wherein the color information acquisition unit estimates color information based on output information of a machine learning engine that estimates parameters that affect the color of an image captured when a reagent is imaged. (2-6) The testing device according to (2-1), wherein the color information acquisition unit estimates color information of the reagent based on camera information included in the captured image. (2-7) The testing device according to (2-1), wherein the test value estimation unit inputs the captured image acquired by the image acquisition unit and the reagent identifying information acquired by the reagent identifying information acquisition unit to a machine learning engine that has learned the captured image of the reagent, reagent identifying information, and test value, and outputs a test value based on an output from the machine learning engine. (2-8) The testing device according to (2-1), further comprising a health condition estimation unit that estimates a health condition of a user of the reagent based on the test value estimated by the test value estimation unit. (2-9) The testing device described in (2-8) includes a biological information detection unit that acquires biological information detected from a user of the reagent, wherein the health condition estimation unit estimates the health condition based on the biological information acquired by the biological information detection unit, the test value estimated by the test value estimation unit, and the reagent identification information. (2-10) The testing device described in (2-1), wherein the test value estimation unit estimates the test value based on environmental information that indicates an environment in which the reagent was used, the color information, and the reagent identification information. (2-11) A testing method including the steps of: the testing device acquiring a captured image of a reagent; the testing device acquiring reagent identification information for identifying the reagent; the testing device correcting color information included in the captured image and acquiring the corrected color information; and the testing device estimating the test value of the reagent based on the reagent identification information and the color information. (2-12) A program causing a computer of an information processing device to execute the steps of: acquiring an image of a reagent; acquiring reagent identification information for identifying the reagent; correcting color information included in the image and acquiring the corrected color information; and estimating a test value of the reagent based on the reagent identification information and the color information.

[0103] 1, 1A, 1B, 1C Biological information estimation system 1D, 1E Inspection system 100, 100A, 100B, 100C Biological information estimation device 100D, 100E Inspection device 110, 110A Image acquisition unit 110D Acquisition unit 120 Preprocessing unit 120A, 120C, 120D Color information acquisition unit 121A, 124a Correction unit 122A, 124b Conversion unit 122D Image preprocessing unit 123A, 124A, 124c, 124d, 125A Estimation unit 124D Image color correction unit 126D Reagent color extraction unit 130 Extraction unit 130A, 130B, 130C Biological information estimation unit 130D, 130E Test value estimation unit 140 Estimation unit 140D, 140E Health condition estimation unit 150 Output unit 150D Display information output unit 160, 160D Storage unit 200, 200A, 200B, 200D Terminal device 202D Camera device 204, 204D Processing unit 206D Operation unit 208D Display unit 210, 210A Camera device 210D, 220A Biometric information detection unit 220 Input unit 230 Display unit 230A Operation information acquisition unit

Claims

1. A bioinformation estimation device comprising: an image acquisition unit that acquires an image of a subject; a pre-processing unit that performs a predetermined pre-processing on the captured image; an extraction unit that extracts color information of the subject from the captured image pre-processed by the pre-processing unit; an estimation unit that estimates a bioinformation value, which is a value indicating bioinformation, based on the color information and subject information indicating information about the subject; and an output unit that outputs the bioinformation value.

2. The biological information estimation device according to claim 1, wherein the pre-processing in the pre-processing unit includes a color correction process for correcting the color of the subject included in the captured image to approximate the color of the actual subject.

3. A biometric information estimation device as described in claim 2, further comprising a biometric information identification unit that identifies the biometric information estimated by the estimation unit, and wherein color correction in the pre-processing unit performs color correction of the subject corresponding to the biometric information identified by the biometric information identification unit.

4. The bioinformation estimation device of claim 2, further comprising a subject identification unit that identifies the subject, wherein the preprocessing unit corresponds to the subject identified by the subject identification unit, and the estimation unit estimates the bioinformation value based on the subject information corresponding to the subject identified by the subject identification unit by performing color correction on the captured image.

5. The bioinformation estimation device of claim 4, wherein the pre-processing unit switches conversion information indicating the correspondence between the color information of the captured image and the colorimetric values ​​of the subject based on the type of the subject identified by the subject identification unit.

6. The bioinformation estimation device of claim 4, wherein the preprocessing unit estimates a spectral reflectance by inputting the captured image into a machine learning model trained on color information of the captured image specific to the subject and the spectral reflectance of an actual test subject, and performs color correction of the captured image based on the colorimetric value of the subject calculated from the estimated spectral reflectance.

7. The biometric information estimation device according to claim 2, wherein the preprocessing unit extracts related information about imaging from the captured image and performs color correction based on the extracted related information.

8. The biological information estimation device according to claim 7, wherein the preprocessing unit estimates an illumination color around the subject as the related information, and performs color correction based on the estimated illumination color.

9. The biometric information estimation device according to claim 7, wherein the preprocessing unit performs color correction based on camera model information included in metadata of the captured image as the related information.

10. The biometric information estimation device according to claim 2, wherein the preprocessing unit performs color correction based on registration information accepted based on a user's operation.

11. The biometric information estimation device according to claim 10, wherein the registration information is an illumination color, and the preprocessing unit performs color correction based on the illumination color.

12. The biometric information estimation device according to claim 10, wherein the registration information is terminal information, and the preprocessing unit performs color correction based on camera model information corresponding to the terminal information.

13. The bioinformation estimation device of claim 2, wherein the subject information indicates an operation that induces a temporal change in the subject, and the estimation unit estimates the bioinformation value based on a change in color information in response to the temporal change in the subject.

14. The bioinformation estimation device of claim 1, wherein the subject information is a user's personal bioinformation, and the estimation unit estimates the bioinformation value based on the color information and the bioinformation.

15. The bioinformation estimation device of claim 1, wherein the subject information is a user's personal bio-related information or environmental information, and the estimation unit estimates the bioinformation value based on the color information and the bio-related information.

16. The biological information estimation device of claim 1, wherein the image acquisition unit acquires an image of a subject and a color chart image of a color chart, and the pre-processing unit creates conversion information for converting color information contained in the image of the subject using the color chart image, and corrects the color information contained in the image based on the conversion information.

17. The bioinformation estimation device according to claim 1, further comprising a health condition estimation unit that estimates a health condition based on the bioinformation value estimated by the estimation unit.

18. A method for estimating biometric information, comprising the steps of: a computer acquiring an image of a subject; performing a predetermined pre-processing on the captured image; extracting color information of the subject from the pre-processed captured image; estimating a biometric information value, which is a value indicating biometric information, based on the color information and subject information indicating information about the subject; and outputting the biometric information value.

19. A bioinformation estimation program that causes a computer to execute the steps of: acquiring an image of a subject; performing a predetermined preprocessing on the captured image; extracting color information of the subject from the preprocessed captured image; estimating a bioinformation value that is a value indicating bioinformation based on the color information and subject information that indicates information about the subject; and outputting the bioinformation value.

Images