Terminal and program

Abstract

To provide a terminal which hardly causes misreading and has high character reading accuracy even in a health measuring machine having no back light.SOLUTION: This health measuring instrument 200 is provided with a photographing means for photographing the health measuring instrument 200, a model information recording means for recording model information of the health measuring instrument 200, a character area specifying means for specifying a plurality of character areas 311 to 333 from a photographed display part 211 based on the model information, a character selecting means for selecting a character having the highest degree of coincidence by template matching for each character area, a character deciding means for abandoning a character string when there is a character whose degree of coincidence by template matching is less than a prescribed value in the character area of at least one digit and deciding the character of the display part when all the degrees of coincidence of the character areas of at least one digit are equal to or more than a prescribed value, and a character deciding means for performing photographing of the display part a plurality of times and continuously deciding the same character twice or more.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a terminal device and a program for recording information displayed by a health measurement device. [Background technology]

[0002] The following has been proposed as a terminal device that records information displayed by a health measurement device by photographing the health measurement device. For example, Patent Document 1 (JP 2018-109801 A) proposes a terminal device that can more accurately acquire measurement results displayed on the display unit of a fitness device.

[0003] The terminal device described in Patent Document 1 executes a process of recognizing character strings indicating measurement values ​​displayed on the display of a health device from an image of the health device included in a captured image, and determines whether the accuracy representing the recognition accuracy of each of multiple recognized characters included in a recognized character string, which is the recognition result of the character string indicating the displayed measurement value, is less than a first threshold value. If the accuracy is less than the first threshold value, the terminal device further executes a process of recognizing character strings indicating the displayed measurement value from each of N (N is an integer equal to or greater than 1) captured images further generated by the imaging means. The terminal device determines a recognized character string consisting of each recognized character identified based on N+1 recognition results of the character strings indicating the displayed measurement value as the measurement value of the health device, and outputs the determination result.

[0004] Patent Document 2 (Japanese Patent Laid-Open Publication No. 2001-224557) proposes a data input device used in an electronic health monitor terminal device or the like equipped with various measuring instruments for performing health checks.

[0005] The data input device described in Patent Document 2 comprises an image acquisition means for acquiring an image of the data displayed on the data display unit of the measuring instrument, a number reading means for reading the numbers in the acquired image, and a display means for displaying the read numbers; the image acquisition means simultaneously acquires images of parts of the measuring instrument other than the data display unit, and the image recognition means reads information about the measuring instrument other than numbers from the acquired images; and the read information about the measuring instrument is used when the number reading means reads numbers.

[0006] Patent Document 3 (WO 2016 / 009740) proposes a measurement data processing system that recognizes measurement data from the measurement result display units of various measuring instruments and outputs the measurement results.

[0007] The measurement data processing system described in Patent Document 3 is a measurement data processing system that can stably recognize measurement values ​​without being affected by the environment, and includes a measuring instrument having a display with a light source that displays measurement results, multiple measuring instrument identification media installed on the display of the measuring instrument, each with a different light transmittance from the light source, an image acquisition means that captures images of the measurement results displayed on the display and the measuring instrument identification media, identifies the measuring instrument identification media from the acquired images, recognizes the measurement results according to image processing parameters searched for using the identified measuring instrument identification information, and outputs measurement data or error data.

[0008] Patent Document 4 (JP 2014-137713 A) proposes a data identification method that makes it possible to correct the image quality of numeric image information captured under different conditions.

[0009] The data identification method described in Patent Document 4 uses the photo mail function of a mobile phone to send the numerical information of measurement results on the LCD screen of a measuring instrument located in a place physically distant from the computer system, or product numbers on an LCD television image, or product numbers displayed in a catalog or flyer, to the computer system, where the image numerical information is identified and converted into digital numbers. This method employs an identification marker to facilitate the position extraction of the measurement value or product number, treats it as one octal digit using three points of the identification marker, selects the subject to be treated as one octal digit, an identification template, and a number font, and uses a correction font that uses the background for each target number digit.

[0010] Patent Document 5 (JP 2017-102797 A) proposes a medical data reading device that can improve versatility for reading objects.

[0011] The medical data reading device described in Patent Document 5 includes an imaging unit, a memory unit that stores device identification information that can identify a medical device and medical data identification information that can identify medical data displayed on the medical device, a medical device identification unit that identifies the medical device imaged by the imaging unit based on the image captured by the imaging unit and the device identification information, a medical data identification unit that identifies the medical data displayed on the imaged medical device based on the image captured, the medical data identification information, and the identification result of the medical device identification unit, and a recording unit that records the medical data identified by the medical data identification unit.

[0012] Patent Document 6 (JP 2014-53693 A) proposes an electronic device that accurately reads the measurement value displayed on the display unit of a health measurement device from an image captured of the display unit.

[0013] The electronic device described in Patent Document 6 has a camera that captures image data of a subject, including a display unit provided on a health measuring device that displays the measurement values ​​measured by the health measuring device, and a controller that identifies the area in which the image of the display unit included in the image data is displayed based on the brightness distribution of the image data captured by the camera, performs photometric processing on the identified area, and adjusts the exposure amount when acquiring an image of the subject based on the results of the photometric processing. [Prior art documents] [Patent documents]

[0014] [Patent Document 1] Japanese Patent Application Publication No. 2018-109801 [Patent Document 2] Japanese Patent Application Laid-Open No. 2001-224557 [Patent Document 3] International Publication No. 2016 / 009740 [Patent Document 4] Japanese Patent Application Laid-Open No. 2014-137713 [Patent Document 5] Japanese Patent Application Publication No. 2017-102797 [Patent Document 6] Japanese Patent Application Laid-Open No. 2014-53693 Summary of the Invention [Problem to be solved by the invention]

[0015] With the recent spread of smartphones and other devices, various data measured on health conditions are now being recorded and managed as biometric data. In particular, it is recommended to measure blood pressure and other parameters daily and record their fluctuations. In some cases, measurements are taken at multiple times each day, recorded, and stored to confirm diurnal fluctuations. From this perspective, there is a need to link health measurement results with a smartphone or personal computer to understand the user's health condition. Devices that measure various health conditions are being introduced not only in medical settings but also in homes, but many of these health measuring devices do not have communication functions, so users have to input the measurement results displayed on the display (such as an LCD display) into a smartphone or other device themselves.

[0016] Additionally, to eliminate the need for users to manually enter information into their smartphones, attempts have been made to use character recognition technology to identify information displayed on the display of health monitoring devices. However, compared to recognizing characters printed on paper, reading information displayed on the display of a health monitoring device is more difficult, resulting in the problem of incorrectly identifying values. In particular, the display units of home health measuring devices often have transparent surface panels to protect the LCD screen, which causes reflections and shadows that reduce the accuracy of character recognition. Furthermore, some home health monitoring devices prioritize portability, light weight, low cost, or low power consumption, and many of them use LCD displays without backlights. When users take a photo of the display of such health monitoring devices with their own smartphones, there are noticeable glare and shadows, which can lead to incorrect readings.

[0017] The terminal device of Patent Document 1 determines the character string by replacing some characters in the character string with recognized characters that are recognized more frequently when the character string cannot be correctly recognized and the accuracy is less than a threshold value. However, when reflections or shadows occur and some lines of a character are missing but the accuracy is high (for example, misidentifying 6 as 5), or when some lines of a character are missing but the accuracy is high (for example, misidentifying 1 as 7), the entire character string is judged to have been recognized correctly, resulting in the problem of recognizing an incorrect number.

[0018] The terminal device of Patent Document 2 reads an image of a marker or the like near a display unit, and performs display recognition based on the obtained detection auxiliary information (type, etc.). However, even in this case, there is a problem that if a reflection or shadow occurs on the display, an incorrect value may be recognized.

[0019] The measurement data processing system of Patent Document 3 improves character recognition accuracy by attaching a plurality of measurement instrument identification media to the display screen and utilizing the amount of light emitted by the display screen. However, in this case, the display must have a backlight, so there is almost no risk of misinterpretation due to reflections or shadows on the display.

[0020] The data identification method in Patent Document 4 improves the ability to identify measuring instruments with different fonts by photographing both the numeric information and the measurement value identification marker. Patent Document 4 also proposes using a correction font cut out from the background image for LCD screens without backlights, but this can lead to the problem of incorrect numerical values ​​being recognized when reflections or shadows occur on the display.

[0021] The medical data reading device of Patent Document 5 identifies the medical device from the shape of the image of the medical device, and identifies the contents of the medical data read from the display screen. However, even in this case, there is a problem that if a reflection or shadow occurs on the display screen, an incorrect value may be recognized.

[0022] The information providing system of Patent Document 6 adjusts the exposure amount based on the luminance distribution of image data to improve reading accuracy. However, even in this case, there is a problem that if reflections or shadows occur in the image data, incorrect values ​​may be recognized.

[0023] Therefore, an object of the present invention is to provide a terminal device and a program that are less likely to cause misreadings and have high character reading accuracy even when used with a health measuring device that does not have a backlight. Another object of the present invention is to provide a terminal device and a program that are less likely to cause misreadings and have high character reading accuracy even when there are various types of health measuring devices. [Means for solving the problem]

[0024] (1) According to one aspect, a terminal device is a terminal device capable of reading characters displayed on a display unit of a health measuring device that does not have a backlight, and includes: an imaging means for imaging the health measuring device; a model information recording means for recording model information of the health measuring device; a character area identification means for identifying multiple character areas from the imaged display unit based on the model information; a character selection means for selecting the character with the highest degree of match by template matching for each character area; a provisional determination means for discarding a character string if a character area of ​​at least one digit has a degree of match by template matching that is less than a predetermined value, and provisionally determining a character for the display unit if all the degrees of match of the character areas of at least one digit are equal to or greater than a predetermined value; and a character determination means for photographing the display unit multiple times and determining the provisionally determined character if all of the provisionally determined character strings are the same character two or more times in a row. The "for at least one digit" used in the provisional determination means, for example, that if the identified characters are a three-digit number, and there is a character whose template matching score is less than a predetermined value for "at least the tens digit and the ones digit," the character string is discarded and no provisional determination is made. On the other hand, if the matching score for all character regions "for the tens digit and the ones digit" is equal to or greater than a predetermined value, the character to be displayed is provisionally determined. Furthermore, if the identified characters include multiple character strings (e.g., systolic blood pressure and diastolic blood pressure), and any of the characters has a matching score less than a predetermined value (except when only the hundreds digit is below the threshold), the misidentified character string is discarded, and a provisional determination is made if the matching score for all characters in the character string is equal to or greater than a predetermined value. In this way, the character region is composed of multiple character strings, each representing a numerical value within a predetermined range, and if the matching score for the character of at least one digit in all of the multiple character strings is equal to or greater than a predetermined value, the character to be displayed is provisionally determined.

[0025] In recent years, various types of measuring devices have been released for consumer use, but many of them are used standalone and do not have communication functions. Furthermore, even if the screen of such devices is photographed with a mobile phone (smartphone), there is a problem in that the accuracy of character recognition is lower than that of printed characters when attempting to automatically identify the values ​​displayed on the screen. In particular, if the display does not have a backlight (light-emitting element), indoor lighting or the terminal device (smartphone, etc.) being photographed may be reflected on the surface panel, creating a shadow around the display's outline, which can lead to character misrecognition and significantly reduce recognition accuracy. In particular, the characters displayed on the display of health measuring devices are often displayed in large numbers to emphasize user visibility, and 7-segment characters are often placed near the display's outline. Consumer health measuring devices are often equipped with a surface panel to protect the LCD display, and shadows are likely to appear on the outline due to the difference in level between the surface panel and the LCD panel, so the 7-segment characters are often placed in a position where they are likely to overlap with the shadow of the outline. Normally, when humans look at an object with their eyes, the brain automatically cancels out visual information such as shadows, so there are few cases of misreading even when characters overlap with shadows, etc. However, when character recognition is performed mechanically, even if the captured image contains reflections or shadows, there is a problem in that if the degree of match with the template image (matching rate) is high, the misreading may not be noticed and the wrong numerical value may be identified.

[0026] According to one aspect of the terminal device, the individual character regions (segments) of each health measurement device are identified based on the model information, and if there is a character for which the degree of match in template matching for at least one digit of each measurement value is less than a predetermined value, the measurement value is rejected because it is considered that the conditions for character identification are not met due to a problem such as reflection in the captured image. Even if the degree of match for all characters is equal to or greater than the predetermined value and a provisional determination is made, it is further determined whether all of the provisionally determined characters have been identified as the same character multiple times in a row, and the characters are finally determined. Since the user takes pictures while holding the terminal device (such as a smartphone) in their hand, the positional relationship between the character part in the continuously captured images and the reflections and shadows will be shifted due to the movement or shaking of the hand. Therefore, even if a reflection or shadow overlaps with part of a character and accidentally leads to a provisional determination, this will not be left as a misreading, and only if all of the provisionally determined characters match two or more times in a row will the identified character be determined as the final measurement result. This makes it possible to create a terminal device with high character reading accuracy even for health measurement devices that do not have a backlight.

[0027] Furthermore, according to a terminal device according to one aspect, the character area is identified based on the model information of the health measurement device used by the user, so there is no risk of misreading the measurement results, such as mistakenly recognizing systolic blood pressure (maximum blood pressure) as diastolic blood pressure (minimum blood pressure).

[0028] (2) The terminal device according to the second aspect of the present invention is a terminal device according to one aspect of the present invention, and may further include an enlargement display means for enlarging and displaying the photographed health measurement device on the display screen of the terminal device.

[0029] Since the captured image is enlarged and displayed on the display screen of the terminal device, the user must take the image while keeping the health measurement device and the terminal device apart. When the display unit of a health measurement device is close to the terminal device (camera) held by the user, the display unit reflects a wider range of the indoor space, making it easier for the indoor lighting and other factors to be reflected. Therefore, the reflected lighting and other factors can obscure some of the characters, resulting in incorrect readings. Furthermore, images taken with smartphones and other devices often have their brightness automatically corrected, and if the indoor lighting is reflected, the entire display becomes dark, making it difficult to distinguish the characters. Furthermore, if the display unit of the health measurement device is close to the terminal device held by the user, the shadow of the terminal device is likely to be reflected on the display unit, and if the text and the outline of the shadow overlap, the user may recognize an incorrect number. In the terminal device of the second invention, the health measurement device photographed is displayed enlarged on the display screen (display) of the terminal device held by the user, so the user takes the photograph while keeping the health measurement device and the terminal device at a distance. This narrows the range of reflection in the indoor space and blurs the outline of the shadow of the terminal device, resulting in a terminal device with high character reading accuracy.

[0030] (3) The terminal device according to the third invention may be the terminal device according to the first aspect or the second invention, and may further include a threshold determination means for determining a threshold for extracting the character portion by determining the luminance distribution of the character region identified by the character region identification means and decomposing the character region into a background portion, the character portion, and a shadow portion caused by an outline portion, and a binarization processing means for performing binarization processing of the character region based on the threshold determination means.

[0031] The display unit of health measurement devices often has a surface panel on top of the liquid crystal display, and shadows often appear around the periphery of the liquid crystal screen of the display unit due to the difference in level between the liquid crystal screen and the surface panel. On the other hand, the seven-segment characters on the display are often designed to be displayed as large as possible on the display, which means that parts of the seven-segment characters tend to overlap with the shadows that accompany the outlines near the edges of the display, resulting in poor character recognition accuracy. In the terminal device according to the third aspect of the present invention, a brightness histogram of the character region is created, and the binarization threshold is determined by previously separating the components into the background, character, and shadow areas of the LCD screen. This minimizes the influence of the shadow areas and allows only the components of the character area to be extracted, resulting in a terminal device with high character reading accuracy.

[0032] (4) The terminal device according to the fourth invention may be the terminal device according to the third invention, further comprising an expansion process for expanding thin portions of characters when the thickness of the characters that have been binarized is not constant.

[0033] When binarization processing is performed as in the third invention, some of the straight line parts of the seven-segment characters may become thinner, which may result in a low degree of match with the template image when template matching is performed. Therefore, by performing an expansion process to expand thin portions before template matching, it is possible to prevent a decrease in the degree of matching that accompanies the binarization process.

[0034] (5) A terminal device according to a fifth aspect of the present invention is a terminal device according to the third or fourth aspect of the present invention, wherein the threshold determination means divides the character area into a plurality of areas in the vertical direction and determines a plurality of thresholds in the vertical direction based on the luminance distribution of each area, and the binarization processing means divides the character area into a plurality of areas in the vertical direction and performs binarization processing using the thresholds of each area.

[0035] When scanning characters in the 7-segment method horizontally, the character part always appears within the scanning range. Therefore, if one character area is divided into multiple areas vertically and each divided area is scanned horizontally, the luminance distribution of each divided area will always include the luminance of the character part and the background part. Therefore, a threshold for extracting the character part can be found based on the luminance distribution of each divided area. The terminal device according to the fifth aspect of the present invention can effectively remove the effects of glare and shadows that occur partially in the character area by scanning each of the divided areas horizontally and determining a threshold value for extracting the character portion.

[0036] (6) A terminal device according to a sixth aspect of the present invention is a terminal device according to any one of the first to fifth aspects of the present invention, further comprising a grayscale processing means for converting a photographed display unit into grayscale, wherein the template images for template matching include, for each character, an image in which the shadow of the outline is not reflected in the character area and an image in which the shadow of the outline is reflected in the character area, and the character selection means may output a character with a higher degree of match and the degree of match.

[0037] Among the shadows displayed on the display unit of a health measurement device, many of the shadows that lead to erroneous character recognition are caused by shadows on the outline of the surface panel. Therefore, the terminal device according to the sixth aspect of the present invention prepares template images of characters that include shadows at the outlines in addition to template images of normal characters, and performs template matching processing between these and the captured image. This allows template matching processing to be performed including shadows that can lead to erroneous character recognition, making it possible to provide a terminal device with high character reading accuracy regardless of whether shadows are present around the LCD screen.

[0038] (7) A terminal device according to a seventh aspect of the present invention is a terminal device according to any one of the first to sixth aspects of the present invention, and may be provided with a guidance display means for displaying user guidance determined based on model information on a display screen of the terminal device.

[0039] This automatically displays user guidance suited to the health measurement device, allowing the user to easily read data by following the user guidance.

[0040] (8) The terminal device according to the eighth invention is a terminal device according to any one of the aspects to the seventh invention, and may further include a device type code written on the health measurement device, and a model information determination means for determining model information of the health measurement device by photographing the device type code written on the health measurement device.

[0041] This saves the user the trouble of inputting model information of the health measurement device into the terminal device. Furthermore, even when a user inputs the measurement results of a plurality of health measurement devices into a terminal device, there is no need to switch between the model information of the health measurement devices. Therefore, even when various types of health measuring devices are mixed, the terminal device can have high character reading accuracy.

[0042] (9) The terminal device according to the ninth aspect of the present invention is a terminal device according to any one of the aspects to the eighth aspect of the present invention, and may further include a contour determination means for determining the contour of the photographed display unit based on the casing of the photographed health measurement device and the model information recorded in the model information recording means.

[0043] When a terminal device detects the position (and shape) of a display unit from a captured image, it usually detects a rectangle or other shape from the captured image and identifies the outline of the display unit. For example, if the border between the case and the display unit (the outline of the display unit) is clear, such as when the case is white and the display unit is gray, there is no problem. However, if the colors of the case and the display unit are similar, it becomes difficult to determine the position (and shape) of the outline of the display unit. The terminal device of the ninth invention identifies the position of the outline of the display unit from the appearance and shape of the housing of the health measurement device, so that the character area can be reliably identified even when the boundary between the housing and the display unit is unclear. This allows the terminal device to have high character reading accuracy even for health measuring devices with a wide variety of color variations.

[0044] (10) A terminal device according to a tenth aspect of the present invention is a terminal device according to any one of the aspects to the ninth aspect of the present invention, and may further include an icon area identification means for identifying an icon area from a photographed display unit based on model information, and an icon identification means for identifying the presence or absence of an icon display in the icon area by template matching.

[0045] Some health measurement devices notify the measurement results not only with text information but also with icons. According to the terminal device of the tenth aspect of the present invention, it is possible to distinguish whether an icon is displayed or not.

[0046] (11) The terminal device according to an eleventh aspect of the present invention is a terminal device according to any one of the first to tenth aspects of the present invention, wherein the icon display on the display unit further includes error confirmation information for the character string displayed by the health measuring device, and may further include a character string confirmation means for confirming whether there is an error in the character string determined by the character string determination means based on the identified error confirmation information.

[0047] Even in terminal devices that have various means for preventing misreading, there is a possibility that some kind of malfunction may cause misreading, resulting in errors in reading characters. The terminal device of the eleventh invention has error confirmation information (including error correction information) in addition to the character string indicating the measurement result. Therefore, even if a character string is misread due to some kind of malfunction, the terminal device can recognize that the character string is a misreading. Therefore, by performing template matching again (or performing error correction), the terminal device can be made to have extremely high character reading accuracy.

[0048] (12) A program according to another aspect is a program that causes a smartphone to function as a terminal device according to any one of the first to eleventh aspects of the present invention.

[0049] This eliminates the need for a dedicated terminal device and allows users to use their own smartphones. [Brief explanation of the drawings]

[0050] [Figure 1] 1 is a schematic perspective view of a system including a terminal device according to an embodiment. [Figure 2] FIG. 2(a) is a schematic perspective view of the sphygmomanometer, and FIG. 2(b) is an exploded view of the sphygmomanometer. [Figure 3] FIG. 2 is a schematic plan view of a display unit of a health measurement device for explaining a screen template of an embodiment. [Figure 4] FIG. 10 is a schematic explanatory diagram illustrating an example of misreading of the display unit of the health measurement device. [Figure 5] FIG. 2 is a schematic diagram for explaining a luminance distribution according to an embodiment. [Figure 6] FIG. 2 is a block diagram illustrating an example of the configuration of a terminal device according to the embodiment. [Figure 7] 10 is a flowchart illustrating a processing procedure of a terminal device according to an embodiment. [Figure 8] FIG. 2 is a schematic plan view illustrating an operation screen of the terminal device according to the embodiment. [Figure 9] FIG. 2 is a schematic explanatory diagram for explaining the operation of the terminal device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0051] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, the same components are designated by the same reference numerals. Furthermore, when the reference numerals are the same, the names and functions of the components are also the same. Therefore, detailed descriptions thereof will not be repeated.

[0052] (system) FIG. 1 is a schematic perspective view illustrating a system including a terminal device 100 and a health measurement device 200. As shown in FIG. As shown in FIG. 1, the health measurement device 200 of this embodiment is a blood pressure monitor 210, a weight scale 220, an alcohol checker 230, a thermometer 240, etc., but is not limited to these and may be, for example, a body fat monitor, a body composition monitor, a women's thermometer, an activity monitor, an oxygen saturation monitor, etc. The terminal device 100 of the embodiment is a mobile phone (smartphone) owned by a user, and is a device that can read information displayed on the display unit 211 of the health measurement device 200 by installing an app (software). In recent years, various health measurement devices 200 for home and commercial use have been released, and while some of these devices have a communication function and can transmit measurement results to the terminal device 100, many health measurement devices 200 do not have a communication function. Even in devices with a communication function, some users may find it difficult to pair their smartphone with the health measurement device 200. Furthermore, some types of health measurement devices 200 can only communicate with specific applications or communication terminals, making it impossible to collectively record and manage measurement results from multiple types of health measurement devices 200 using a single application. In such cases, the user had to input the measurement results displayed by the health measurement device 200 into the terminal device 100 every time a measurement was taken. In particular, some measurement results taken by the health measurement device 200 must be taken every day (or in some cases multiple times a day), and the user's health condition can only be understood by examining the trends, which places a significant burden on the user. Furthermore, there is a risk that input errors may occur during repeated input, which may affect the understanding of the user's health condition. According to the terminal device 100 of this embodiment, even if the health measurement device 200 does not have a communication function, or even if it has a communication function but the measurement data cannot be shared, the measurement results displayed on the display unit 211 of the health measurement device 200 can be accurately read.

[0053] In the following, in this embodiment, a blood pressure monitor 210 will be used as an example for explanation.

[0054] (Blood Pressure Monitor 210) 2A and 2B are schematic perspective views for explaining the sphygmomanometer 210, and FIG. 2A is a schematic perspective view for explaining the appearance of the sphygmomanometer 210. FIG. A blood pressure monitor 210 typically inflates the upper arm with a cuff 219, then gradually depressurizes the cuff while detecting arterial pulsation to calculate systolic and diastolic blood pressure. The blood pressure monitor 210 illustrated in Fig. 2 can also display the pulse rate and the presence or absence of irregular pulse waves. In this embodiment, the blood pressure monitor 210 is of a type in which the cuff 219 is wrapped around the upper arm, but it may also be of a fully automatic upper arm type, a wrist type, or the like. The blood pressure monitor 210 of this embodiment is turned on when the power switch 216 is pressed, and automatically starts measuring blood pressure after a few seconds. When blood pressure measurement is complete, the display unit 211 displays the systolic blood pressure (three digits), diastolic blood pressure (three digits), pulse rate (three digits), and, if necessary, an irregular pulse wave icon, a moving object detection icon, and the measurement date and time. Each number is displayed using seven-segment characters represented by a combination of seven straight lines. If the blood pressure monitor 210 is left for a while, it will automatically turn off after a few minutes, but it can also be turned off by pressing and holding the power switch 216. Also, pressing the memory switch 217 displays the results of previous measurements along with the measurement date and time.

[0055] FIG. 2(b) is a schematic exploded view for explaining the structure of the display unit 211 of the sphygmomanometer 210. The display unit 211 of this embodiment digitally displays measurement results and the like on a liquid crystal display 215 that does not have a backlight. Display unit 211 includes a surface panel 213 provided on the lid of the housing of sphygmomanometer 210 and a liquid crystal display 215 inside the housing, and is arranged so that liquid crystal display 215 can be seen through surface panel 213. Liquid crystal display 215 is fixed on an electronic circuit board, which is fixed inside housing 212. On the other hand, surface panel 213 is formed integrally with the lid of housing 212, is made of transparent resin, and has the function of protecting liquid crystal display 215. Outline portion 214 forms the outline between housing 212 and display unit 211. The front panel 213 of the display unit 211 and the liquid crystal display 215 are disposed close to each other but are slightly spaced apart. Therefore, depending on the angle of incidence of the indoor lighting, a shadow 430 caused by the outline 214 may be projected onto the liquid crystal display 215.

[0056] Here, the mechanism by which misreading occurs during character recognition in health monitoring device 200 that does not have a backlight will be described in detail. 2, most display units 211 of health measuring devices 200 are configured so that character portion 420 of liquid crystal display 215 is viewed through front panel 213 of housing 212. Therefore, the front panel 213 is prone to reflecting indoor lighting and the like, and the liquid crystal display 215 is prone to casting a shadow of outline portion 214. FIG. 4 is a schematic diagram illustrating an example of a misreading of display unit 211 of health measurement device 200. As illustrated in FIG. 4, when shadow portion 430 of outline portion 214 overlaps character portion 420 on liquid crystal display 215, the captured image appears as character region 313. Therefore, when a character that should be identified as a "1" may have a higher degree of match with template image 350 of "7," resulting in misrecognition of the character. Also, as illustrated in FIG. 4, when glare portion 440 from indoor lighting or the like overlaps character portion 420 on liquid crystal display 215, a portion of the character in the captured image becomes bright, as in character region 322 in FIG. 4, and when a character that should be identified as an "8" may have a higher degree of match with template image 350 of "9," resulting in misrecognition of the character. The LCD display 215 without a backlight is excellent in terms of power saving, light weight, cost, etc., but it is prone to shadows caused by the outline 214 or reflections caused by the surface panel 213, which causes the problem that misreadings are likely to occur, unlike character recognition (OCR) for printed paper.

[0057] (Terminal device 100) The terminal device 100 exemplified in this embodiment is a smartphone or the like owned by a user, and by installing an application, it can automatically read and record the measurement results of various health measurement devices 200. Note that in this embodiment, the terminal device 100 is a smartphone with a predetermined application installed, but it may also be a dedicated terminal used in a medical setting.

[0058] 6 is a block diagram showing an example configuration of a terminal device 100 (smartphone) according to an embodiment. The terminal device 100 includes a processor 110 capable of processing various programs, a camera 120 capable of capturing images of a health measurement device 200, a display 130 capable of displaying captured images or videos, an input unit 140 provided on the display 130 and capable of electrostatically receiving input from a user, a memory 150 capable of recording applications, input information, capture information, model information, and identified text information, a speaker 160 capable of transmitting sound to the user, a microphone 170 capable of receiving voice or sound from the user, and a wireless communication unit 180 capable of transmitting information in the memory 150 to or receiving information from other devices. Note that the wireless communication unit 180 may not be included in a dedicated terminal used in a medical setting. When health measuring device 200 is photographed with camera 120, terminal device 100 of this embodiment identifies the position and shape of display unit 211, identifies characters in character areas 311-333 and / or icons in icon areas 341-343 based on screen template 300 corresponding to health measuring device 200, and records the measurement results of health measuring device 200. Note that, if necessary, terminal device 100 may further have a function to accept measurement results from input unit 140 via user operation, and may further have a function to receive numerical values ​​read aloud by the user via microphone 170 and perform voice recognition to record the measurement results.

[0059] (Operation screen) FIG. 8 is a schematic plan view for explaining the operation screen of the terminal device according to the embodiment. When the user runs the application, a type selection screen (not shown) is displayed for selecting the type of health measuring device 200 to be read. When the user selects a predetermined blood pressure monitor 210 on the type selection screen, the model information (vendor information, product model number information) is recorded in the model information recording means, and during character recognition (described later), the screen template 300 corresponding to the selected blood pressure monitor 210 of the predetermined model number is read and started. The type selection screen may be displayed each time the application is launched, or may be displayed when the application is launched for the first time, and may not be displayed if a type has already been selected, and the type selection screen may be displayed from the settings menu. In addition, in this embodiment, the user selects or inputs the model information, but the model number information 218 written on the housing 212 may be identified by character recognition, or the device identification code Q (such as a two-dimensional QR code (registered trademark) containing product model number information) may be identified.

[0060] Next, when the user presses (clicks) the capture button 501 from the main menu 500 (Figure 8(a)), the display unit reading screen 510 is displayed, and the camera 120 of the terminal device 100 is activated to capture a video (or a series of photos). On the display unit reading screen 510, the image captured by the user with the camera 120 is displayed on the image display unit 511 (FIG. 8(b)). Here, the image captured by the camera 120 is enlarged and displayed on the image display unit 511, and if the distance between the terminal device 100 and the sphygmomanometer 210 is short, only a part of the image on the display unit is displayed. Therefore, the user takes an image with the terminal device 100 and the sphygmomanometer 210 at a distance from each other. As a result, even if a shadow of the terminal device 100 appears on the display unit 211 of the sphygmomanometer 210, the outline of the shadow becomes unclear, thereby reducing the influence of the shadow of the terminal device 100 on misreading. Furthermore, since the distance between the terminal device 100 and the sphygmomanometer 210 is increased, reflections on the display unit 211 are reduced. That is, when the distance between the camera 120 and the display unit 211 is decreased, the range of the indoor environment reflected on the display unit 211 is increased, so that indoor illumination light and the like are more likely to be reflected on the display unit, and misreading occurs due to overlapping with the characters on the liquid crystal display 215 (see FIG. 4). On the other hand, since the distance between the camera 120 and the display unit 211 is increased, the range of the indoor environment reflected on the display unit 211 is decreased, so that the possibility of illumination light and the like being reflected is reduced, and misreading is prevented. The magnification of the captured image on the image display unit 511 is preferably 1.2 to 3 times, and more preferably 1.4 to 2.5 times. In the terminal device 100 of this embodiment, the initial value is set to 2.0 times, and the magnification can be changed as needed by user operation. In addition, the distance between the camera 120 and the display unit 211 is preferably 10 cm to 20 cm, and more preferably 13 cm to 17 cm. This makes it possible to effectively prevent misreading while maintaining resolution.

[0061] In the example of the display unit reading screen 510 (FIG. 8(b)) in FIG. 8, user guidance 512 is displayed below the video display unit 511. User guidance 512 may be changed as appropriate depending on the type of health measurement device 200 being photographed, or general precautions regardless of the type may be displayed, or instructions on how to operate the software may be displayed. Furthermore, if a reflection or a shadow is detected, a message urging the user to change the angle may be displayed. Examples of guidance to be displayed when the frame lines in the video display unit 511 do not match the display unit 211 include "Take a photo of the LCD screen from the front and make sure it fits within the frame lines," "Please use in a bright environment" may be displayed when the brightness of the captured image is insufficient, "Please avoid this by changing the position or orientation or enlarging the image" may be displayed when a light or the like is reflected on the display unit 211, and "Please use in a situation where there is no shadow on the edge, such as by changing the position or orientation" may be displayed when there is a shadow portion 430 of the outline portion 214.

[0062] Once the image capture is complete, the screen transitions to the measurement value display screen 520 (Fig. 8(c)), where each identified value is displayed as a measurement result. If the user discovers a misreading here, they can correct each value. Furthermore, the user can add additional information such as notes as needed. When the user checks the measurement value on the measurement value display screen 520 and presses the check button, the read value is confirmed as the measurement value and recorded in the memory 150, and the screen returns to the main menu 500. If the user wishes to input the measurement results directly into the terminal device 100, the user can press the Add Measurement Value button 502 to input the measurement values ​​directly from the measurement value display screen 520 without automatically identifying the characters by photographing. Furthermore, if a health measurement device 200 with a communication function is present, the measurement results can be obtained by using the short-range communication function (BLUETOOTH (registered trademark)) of the wireless communication unit 180 of the terminal device 100 or the like.

[0063] The determined measurement values ​​are recorded in memory 150, and the measurement values ​​stored in memory 150 can be displayed as a time series graph or the like, as shown in FIG. 8(d). In addition, the measurement results (e.g., systolic blood pressure, diastolic blood pressure, weight, number of steps, etc.) of multiple health measurement devices 200 may be displayed together on a graph or the like, and the measurement results may be transmitted to a management server or the like via the wireless LAN communication function of the wireless communication unit 180.

[0064] (Screen template 300) When the user selects a predetermined model number of blood pressure monitor 210 on the type selection screen, a screen template 300 corresponding to the predetermined health measurement device 200 is read from memory 150 and activated. Fig. 3 shows an example of the screen template 300 corresponding to the blood pressure monitor 210. As shown in Fig. 3, the screen template 300 predefines areas for character portions to be identified from the display unit 211 of the sphygmomanometer 210. The screen template 300 in Fig. 3 includes character areas 311, 312, and 313 for systolic blood pressure, character areas 321, 322, and 323 for diastolic blood pressure, character areas 331, 332, and 333 for pulse rate, an icon area 342 for motion detection, and an icon area 343 for irregular pulse wave. As will be described in other embodiments, the screen template 300 may also include error check information 345 for checking for erroneous character recognition (or for correcting errors).

[0065] Measurement results such as systolic blood pressure, diastolic blood pressure, and heart rate are displayed in seven-segment characters on the display unit 211 of the sphygmomanometer 210. When the user photographs the display unit 211 with the camera 120, the terminal device 100 first detects the outline 214 of the display unit 211 based on the rectangular shape. Here, if the display unit 211 is photographed obliquely, the detected outline 214 may be distorted into a trapezoid. In this case, the terminal device 100 corrects the trapezoidally distorted display unit 211 into a rectangle. Alternatively, if no keystone correction is performed, the user may be instructed via the user guidance 512 to align the display unit 211 with the frame line within the video display unit 511. Furthermore, if the resolution of the photographed video differs from a predetermined value, image correction may be performed as appropriate to achieve the required resolution. In this way, when an image with a rectangular outline 214 is acquired, a predetermined screen template 300 is applied, and the three digits of the systolic blood pressure are identified as numbers from character areas 311, 312, and 313, the three digits of the diastolic blood pressure are identified as numbers from character areas 321, 322, and 323, and the three digits of the pulse rate are identified as numbers from character areas 331, 332, and 333.

[0066] In this embodiment, an example has been given of a case where contour 214 of display unit 211 is automatically detected from a photographed image of sphygmomanometer 210. However, if the colors of housing 212 and display unit 211 are similar, it may be difficult to detect contour 214. In such a case, the shape of housing 212 may be image-recognized based on model information (product model number information), and the position and shape of contour 214 may be estimated from the photographed shape of housing 212. By providing this function, the terminal device 100 can be made compatible with the health measurement device 200 in a wide variety of colors.

[0067] (template matching) The numbers in each of the character areas 311 to 333 are identified by comparing them with template images 350 stored in the memory 150 of the terminal device 100. The template images 350 of this embodiment use seven-segment characters (numbers 0 to 9) captured in advance on the display unit 211 using a camera or the like, and two types of template images 350 are provided: one for blood pressure values ​​with large characters and one for heart rate values ​​with small characters. Then, processor 110 compares the numbers in each character area 311-333 and the icons in icon areas 341-343 with template image 350, calculates the degree of match (correlation coefficient) between the two images, and selects the number with the highest degree of match from the numbers 0-9 (or identifies the presence or absence of an icon).

[0068] (Tentative decision) Here, a predetermined threshold is set for the degree of match in template matching. According to the processing method of this embodiment, for the most significant digit of each measurement value (the hundreds digit of the systolic blood pressure, the hundreds digit of the diastolic blood pressure, and the hundreds digit of the heart rate), if the degree of match is below the threshold, it is determined that there is no numerical value in that digit. For digits other than the most significant digit (the tens digit and ones digit of the systolic blood pressure, the tens digit and ones digit of the diastolic blood pressure, and the tens digit and ones digit of the heart rate), if the degree of match is below the threshold for any one digit, that measurement value is discarded, and character recognition is performed on the image obtained in the next capture. That is, for example, the measured value of systolic blood pressure may be two digits (e.g., 98 mHg) or three digits (e.g., 150 mHg). Therefore, if the highest digit (the hundreds digit) is below the threshold, it is identified as "none" and the measured value can be determined to be two digits. On the other hand, since the tens and ones digits of the systolic blood pressure cannot be "none," if they are below the threshold, it is determined that a misreading has occurred, the systolic blood pressure (character string) is discarded, and character strings are identified from another captured image. The same applies to diastolic blood pressure and heart rate. In this way, if there is a character in character areas 312, 313, 322, 323, 332, 333 (character areas of the tens digit and ones digit) of at least one digit of each measurement value whose degree of match by template matching is less than a predetermined value, the character string is discarded, but on the other hand, if the degree of match for all of the character areas 312, 313, 322, 323, 332, 333 (character areas of the tens digit and ones digit) is equal to or greater than a predetermined value, the characters in character areas 311-333 and the icons in icon areas 342, 343 obtained from the captured image are stored as provisional data in memory 150. This means that captured images with even a partial shadow or reflection are excluded from character identification, preventing misreading. Furthermore, the icon areas 341 to 343 are compared with the template image 350 to determine whether an icon is displayed if the value is equal to or greater than a threshold value, and whether an icon is not displayed if the value is less than the threshold value.

[0069] In addition to the processing method of this embodiment described above, the following processing method may be performed on the character areas 311, 321, and 331 of the most significant digit of each measurement value. Alternative processing method 1: The presence or absence of characters is checked in the character areas 311, 321, 331 of the highest digit, and if no characters are detected, i.e., if there is almost no variation in the brightness distribution of each character area 311, 321, 331, the character for that digit may be identified as "none." Alternative processing method 2: For the character areas 311, 321, and 331 of the highest digit, template matching may be performed including the template image 350 without characters, and if the degree of match with the template image 350 without characters is the highest, the character for that digit may be identified as "none." In another processing method, if there is a character in all character areas 311 to 333 whose degree of match by template matching is less than a predetermined value, the character string selected by template matching is discarded, and if the degree of match for all character areas 311 to 333 is greater than or equal to a predetermined value, the character (and icon) to be displayed may be provisionally determined.

[0070] In the case of the blood pressure monitor 210 illustrated in this embodiment, the range in which the systolic blood pressure is normally measured is 50 mmHg or more and 250 mmHg or less, the range in which the diastolic blood pressure is normally measured is 40 mmHg or more and 180 mmHg or less, and the range in which the pulse rate is normally measured is 40 beats / minute or more and 160 beats / minute or less. Therefore, the terminal device 100 of this embodiment defines the range of values ​​that each health measurement device 200 normally measures based on the model information (product model number information), and if the identified value exceeds the defined value (the above-mentioned range of values ​​that normally measures), it discards the character string selected by template matching and identifies the characters from the image obtained in the next capture.

[0071] (Decision) Next, the camera 120 of the terminal device 100 of this embodiment continuously captures images and performs template matching processing based on the second image (or the second frame image of the video). As a result, the characters in the character areas 311 to 333 and the icons in the icon areas 342 and 343 obtained from the second consecutive image are stored in the memory 150 as second provisionally determined data. Subsequently, template matching is performed based on the third image (or the third frame image of the video). As a result, the characters in the character areas 311 to 333 and the icon areas 342 and 343 obtained from the third consecutive image are stored in the memory 150 as third provisionally determined data. Subsequently, template matching is performed based on the fourth image (or the fourth frame image of the video). As a result, the characters in the character areas 311 to 333 and the icon areas 342 and 343 obtained from the fourth consecutive image are stored in the memory 150 as the fourth provisionally determined data.

[0072] 9 is a schematic diagram illustrating the operation of the terminal device according to the embodiment. In the example of FIG. 9, the systolic blood pressure does not match after the fourth imaging, so the previous count is discarded and a new value is acquired after the fifth imaging. The diastolic blood pressure does not match after the seventh imaging, so a new value is acquired after the eighth imaging. By the eleventh imaging, all values ​​have been the same as the previous values ​​for three or more consecutive imaging sessions, so the final determination is made. In this way, if all numbers and icons (systolic blood pressure, diastolic blood pressure, heart rate and icon) match three or more times in a row (four provisional determinations), the identification characters are finally determined as the measurement results. The user holds the terminal device 100 in his / her hand and takes a photograph. Therefore, even if a provisional decision is made by chance due to a reflection or shadow overlapping part of a character, the identified character is decided as the final decision as the measurement result only if all of the provisionally decided characters match four or more times in a row, so misreading can be prevented with high accuracy, and the terminal device can have high character reading accuracy. In this embodiment, an example is shown in which the final decision is made when characters match in four consecutively captured images, but the final decision may be made when characters match in two or more consecutively captured images (three images, five images, etc.). Making the final decision based on many images can reduce misreadings, but it may take a long time from capturing images to making the final decision. In addition, in this embodiment, "continuous" refers to images taken at intervals of less than 0.5 seconds, and if there is an interval of 0.5 seconds or more between the first and second images, the two taken images may be compared to improve the accuracy of reading characters.

[0073] Furthermore, when character-identifying the measurement results of health measurement device 200, it is preferable to perform template matching for each of character regions 311-333, excluding values ​​that are not expected as measurement results (for example, systolic blood pressure of 300 mmHg or more). For example, in the case of character areas 311 to 313 indicating systolic blood pressure, the template image 350 for character area 311 (hundreds digit) is selected from "1," "2," and "none," the template image 350 for character area 312 (tens digit) is selected from "0" to "9," and the template image 350 for character area 313 (units digit) is selected from "0" to "9." Similarly, it is preferable that the character area 321 for diastolic blood pressure is selected from "1" and "none," the character area 322 is selected from "0" to "9," and the character area 323 is selected from "0" to "9," and the character area 331 for pulse rate is selected from "1" and "none," the character area 332 is selected from "0" to "9," and the character area 333 is selected from "0" to "9." As a result, unnecessary template images 350 are excluded in advance for each of the character regions 311 to 333, which narrows the range of selection and improves the character recognition rate, while shortening the time required for template matching processing and reducing the load on the terminal device 100.

[0074] (binarization processing) When performing the template matching process, the image captured by the camera 120 may be subjected to binarization (black and white dot conversion). When performing binarization processing, first, the luminance distribution of the image in each of the character regions 311 to 333 is calculated. Figure 5 is a schematic diagram for explaining the luminance distribution, where Figure 5(a) is an example of the character region 313 in Figure 4, and Figure 5(b) shows the luminance distribution of the character region 313 as a histogram (the vertical axis is the number of pixels, and the horizontal axis is the luminance value). As shown in Figure 5(a), the character area 313 is divided into a character part 420 with the lowest brightness (black), a shadow part 430 with a medium brightness (gray), and a background part 410 with the highest brightness (the white part of the liquid crystal display 215). When this is decomposed into a brightness distribution, as shown in Figure 5(b), it is decomposed into a character component α, a shadow component β, and a background component γ. Therefore, a threshold value Th is set so that the character component α with the lowest brightness can be separated from this brightness distribution. By using the threshold value Th thus obtained to perform binarization processing on the character region 313, a binarized image in which only the character portion 420 is extracted from the character region 313 can be obtained. Therefore, by performing template matching using a binarized image that eliminates the influence of the shadow component β, character recognition can be performed while preventing misreading due to shadows. When performing such binarization processing, it is preferable to use a binarized image (a black and white binary image) as the template image 350.

[0075] Furthermore, when binarizing the character regions 311 to 333, the character regions 311 to 333 may be further divided into three regions in the vertical direction. Specifically, as shown in Figure 5(c), the 7-segment character part is divided into three regions, A, B, and C, and a binarization threshold Th is set for each region. A ,Th B ,Th C Set. Each threshold Th A ,Th B ,Th C The threshold value Th may be set for each of the regions A, B, and C in the same manner as the threshold value Th described above, but the following method may also be used. That is, the areas near the center of each of the regions A, B, and C that do not overlap with the horizontal segments are scanned horizontally, and the threshold value Th is applied to each of the scanning lines a, b, and c. A ,Th B ,Th C Alternatively, the luminance distribution may be calculated for each of the scanning lines a, b, and c, and a threshold value Th may be determined based on the luminance values ​​on the bright side and the luminance values ​​on the dark side. A ,Th B ,Th C This allows character recognition to be performed while preventing misreading due to shadows. In FIG. 5(c), the character regions 311 to 333 are divided into three regions in the vertical direction, and three threshold values ​​Th A ,Th B ,Th C However, the present invention is not limited to this example, and the character areas 311 to 333 may be divided into three or more parts in the vertical direction, or may be divided in the horizontal direction and subjected to binarization processing.

[0076] (grayscale processing) When performing template matching processing on an image captured by camera 120, grayscale processing may be performed on the captured image instead of the binarization processing described above. Here, the template image 350 is one that has been photographed in advance using a camera or the like to capture the display unit 211. In addition to the template image 350 that has only 7-segment characters (numbers 0 to 9) and does not include the shadow portion 430, a template image 350' that includes the shadow portion 430 is also prepared. Shadow portion 430 is a shadow created by the steps of outline portion 214 being reflected on liquid crystal display 215, and therefore appears around outline portion 214 depending on the angle of incidence of the illumination light. Therefore, for example, if shadow portion 430 appears in eight locations, namely the top, upper left, left, lower left, bottom, lower right, right, and upper right, template images 350' of seven-segment characters may be prepared for each of these locations. Furthermore, depending on the model (screen template 300), it may be predicted that shadow portion 430 of outline 214 will appear only at the top of character area 311, only at the top of character area 312, only at the top or right of character area 313, only at the right of character area 323, only at the bottom of character area 331, only at the bottom of character area 332, or only at the bottom or right of character area 333. Therefore, template matching may be performed by applying template image 350' in which shadow portion 430 is reflected only to areas where shadow portion 430 is expected to appear. Furthermore, a template image 350' for when reflections such as lighting are present may be prepared as necessary. This allows character recognition to be performed while preventing misreading.

[0077] (Character decision process) FIG. 7 illustrates an example of a character determination process performed by the terminal device 100 in this embodiment. <Screen template acquisition process: S2> When the capture button 501 is pressed, the screen template 300 corresponding to the model information (device model number) is obtained from the memory 150. <Shooting process: S3> Then, the display unit reading screen 510 is displayed, and the user is prompted to take an appropriate photograph of the health measurement device 200 by means of user guidance 512 or the like. <Contour detection process: S4> Next, the captured image is received from the camera 120, and the contour 214 of the display unit 211 is detected. The camera 120 continues to capture images (or continue to capture video) continuously until the contour 214 of the display unit 211 is detected. In the contour detection process, the number of pixels captured may be temporarily reduced to reduce the load on the terminal device 100. <Keystone correction process> If the detected outline 214 is a trapezoid, it is considered that the image was captured from an oblique angle, and the shape of the display unit 211 is corrected to become a rectangle. At this time, the image may be processed to have an optimal number of pixels for template matching. The trapezoid correction process may also be omitted as necessary. <Character area identification process: S5> The screen template 300 is applied to the obtained photographed image of the display unit 211, and images of the character areas 311-313, 321-323, 331-333 and icon areas 342, 343 are obtained for each character string (systolic blood pressure, diastolic blood pressure, pulse rate) and icon. <Template matching process: S6> The images in each of the character regions 311 to 333 and icon regions 342 and 343 are compared with each template image 350, and the numerical value (or icon) with the highest degree of coincidence (matching rate) is selected. <String destruction process: S7> If the degree of match of the character area 311, 321, 331 of the highest digit among the character areas 311 to 333 is less than a threshold, it is determined that there is no numerical value in that digit, and if it is equal to or greater than the threshold, the numerical value with the highest degree of match is selected. Furthermore, if the character areas 312, 313, 322, 323, 332, 333 of digits other than the highest digit contain numerical values ​​with a degree of match less than a predetermined threshold, the character string is discarded. Regarding the icon areas 342, 343, if the degree of match is less than the threshold, it is determined that there is no icon in that digit, and if it is equal to or greater than the threshold, it is determined that there is an icon in that digit. <Tentative decision step: S8> In the character areas 312, 313, 322, 323, 332, and 333 of digits other than the most significant digit, if the degree of coincidence of all the numerical values ​​is equal to or greater than the threshold, the selected character is tentatively determined. <Continuous shooting process: S10, S11> Next, the photographing process through the tentative determination process are repeated three times to determine whether all tentatively determined characters are the same numerical value (and whether or not an icon is present) based on a total of four photographed images. If any of the images contains a different tentatively determined character, the first image is acquired again. In the examples of Figures 7 and 9, the most recent tentatively determined value is compared with the previously tentatively determined value (S10), and this comparison is repeated three times in a row to determine whether the numerical values ​​and icons of a total of four photographed images are the same (S11). <Final decision process: S12>. If the provisionally determined character has the same number (and the same icon) in all four consecutively captured images, the character is finally determined.

[0078] (Other embodiments) In addition to the above embodiment, error check information 345 for detecting (or correcting) an error may be added near the icon area 343 of the display unit 211. This error check information 345 is preferably placed at a position away from the character areas 311 to 333. In particular, because the shadow area 430 caused by the outline 214 depends on the angle of incidence of the illumination light, no shadow area 430 occurs diagonally from the area where the shadow area 430 occurs (in the example of FIG. 4, the shadow area 430 is on the upper left side of the display unit 211, so there is no shadow area 430 on the lower right side). Therefore, by placing the error check information 345 diagonally from the character areas 311 to 333 on the display unit 211, it is possible to preferably detect the occurrence of a misreading or correct a misread character.

[0079] When template matching is performed on character areas 311 to 333, error check information 345 is also subjected to template matching, and the information is identified. Error check information 345 includes information for detecting the presence or absence of an error (or information for correcting the error) when the numerical value of the measurement result of health measuring device 200 is misread. Therefore, when identifying characters in character areas 311 to 333, if a character is misrecognized, the misrecognized character string can be discarded and a new image of display unit 211 can be taken (or the identified characters can be corrected). This error checking information 345 may be a 1-bit information display (e.g., an icon on / off), a 4-bit information display in a checkerboard pattern as shown in Fig. 3, or a multi-bit information display such as a QR code (registered trademark). As an example of the error checking information 345, error correction data known as a Reed-Solomon code can be used, which allows identified misread characters to be corrected.

[0080] In the present invention, terminal device 100 corresponds to the "terminal device", health measurement device 200 corresponds to the "health measurement device", display unit 211 corresponds to the "display unit", character areas 311 to 333 correspond to the "character areas", icon areas 341 to 343 correspond to the "icon areas", device identification code Q corresponds to the "device type code", housing 212 corresponds to the "housing", outline portion 214 corresponds to the "outline portion", character portion 420 corresponds to the "character portion", and shadow portion 430 corresponds to the "shadow portion".

[0081] Although a preferred embodiment of the present invention has been described above, the present invention is not limited thereto. It will be understood that various other embodiments can be made without departing from the spirit and scope of the present invention. Furthermore, although the actions and effects of the configuration of the present invention are described in this embodiment, these actions and effects are merely examples and do not limit the present invention. [Explanation of symbols]

[0082] 100 Terminal Device 200 Health measuring device 210 Sphygmomanometer 211 Display section 212 Case 213 Surface Panel 214 Contour 215 LCD display 218 Model Information 300 Screen Templates 311~333 Character area 341~343 Icon area 345 Error confirmation information 410 Background part 420 character part 430 Shadow area 440 Reflected area 500 Main Menu 510 Display reading screen 511 Video display unit 512 User Guidance 520 Measurement value display screen Q Device Identification Code

Claims

1. A terminal device capable of reading characters displayed on a display unit of a health measurement device that does not have a backlight, an imaging means for imaging the health measurement device; a model information recording means for recording model information of the health measurement device; a character area specifying means for specifying a plurality of character areas from the photographed display unit based on the model information; a character selection means for selecting, for each character region, a character with the highest degree of match by template matching; a provisional determination means for discarding a character string when there is a character in the character region of at least one digit whose degree of match by the template matching is less than a predetermined value, and for provisionally determining a character for the display unit when all the degrees of match of the character region of at least one digit are equal to or greater than the predetermined value; A terminal device comprising: a character determination means for taking multiple photographs of the display unit and determining the provisionally determined characters when all of the provisionally determined character strings are the same character two or more times in a row.

2. 2. The terminal device according to claim 1, further comprising an enlargement display unit for enlarging and displaying the photographed health measurement device on a display screen of the terminal device.

3. a threshold value determination means for determining a luminance distribution of the character region specified by the character region specification means, and for determining a threshold value for extracting the character region by decomposing the character region into a background portion, the character region, and a shadow portion due to an outline; 2. The terminal device according to claim 1, further comprising: a binarization processing unit that performs binarization processing of the character region based on the threshold value determining unit.

4. The terminal device according to claim 3 , further comprising an expansion process for expanding thin portions of the characters when the thickness of the characters subjected to the binarization process is not constant.

5. the threshold value determining means divides the character region into a plurality of regions in a vertical direction, and determines a plurality of threshold values ​​in a vertical direction based on the luminance distribution of each region; 4. The terminal device according to claim 3, wherein said binarization processing means divides said character region into a plurality of regions in the vertical direction and performs binarization processing using a threshold value for each region.

6. Further, the image capturing device includes a grayscale processing unit for converting the captured image of the display unit into a grayscale image, the template images for the template matching include, for each character, an image in which the shadow of the outline portion is not reflected in the character region and an image in which the shadow of the outline portion is reflected in the character region; 2. The terminal device according to claim 1, wherein said character selection means outputs characters with a higher degree of match and the degree of match.

7. 2. The terminal device according to claim 1, further comprising a guidance display means for displaying user guidance determined based on the model information on a display screen of the terminal device.

8. The health measurement device has a device type code written on it, The terminal device according to claim 1 , further comprising: a model information determination unit that determines model information of the health measurement device by photographing the device type code printed on the health measurement device.

9. 2. The terminal device according to claim 1, further comprising: a contour determining means for determining the contour of the photographed display unit based on the photographed housing of the health measurement device and the model information recorded in the model information recording means.

10. an icon area specifying means for specifying an icon area from the photographed display unit based on the model information; 2. The terminal device according to claim 1, further comprising: an icon identifying means for identifying whether or not an icon is displayed in said icon area by template matching.

11. The icon display on the display unit further includes error confirmation information for the character string displayed by the health measurement device, 11. The terminal device according to claim 10, further comprising: a character string checking unit that checks whether the character string determined by the character string determining unit has any errors based on the identified error check information.

12. A program that causes a smartphone to function as the terminal device according to any one of claims 1 to 11.

Images