Information display device, information display method, and program

Abstract

A user terminal (20) comprises: an actual image acquisition unit (31) that acquires an actual image showing an electronic timepiece; a device type determination unit (32) that, on the basis of the actual image, identifies information pertaining to the electronic timepiece; and an information display unit (35) that displays the information pertaining to the electronic timepiece, wherein the device type determination unit (32) determines, on the basis of the actual image, the presence / absence of the effect of external light which occurred when the actual image was acquired and which interferes with the identification of the information pertaining to the electronic timepiece.

Description

Technical Field

[0001] The present invention relates to an information display device, an information display method, and a program.

Background Art

[0002] Patent Document 1 discloses a technique for recognizing the model of an electronic clock using a user terminal such as a smartphone.

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] As disclosed in Patent Document 1, in a technique of photographing a device such as an electronic clock and recognizing predetermined information from the photographed image, there are cases where the device displayed in the image cannot be determined due to the influence of external light. When the determination result does not come out normally and the cause is not clear, the user simply misinterprets the device as being outside the determination target. As a result, the user cannot obtain the information that should originally be obtained. In addition, due to the influence of external light, there are also cases where the device is determined to be a different device from the photographed device.

[0005] The present invention has been made in view of the above problems, and its object is to increase the accuracy of accurately determining information related to the device displayed in the image.

Means for Solving the Problems

[0006] (1) An information display device comprising: an image acquisition unit that acquires an image of a device showing predetermined information by at least mechanical or electronic means; a device information identification unit that identifies information about the device based on the image; and an information display unit that displays information about the device, wherein the device information identification unit determines, based on the image, whether or not there is an influence of ambient light that occurs when acquiring the image and hinders the identification of information about the device.

[0007] (2) An information display device having a notification unit that performs a predetermined notification according to the determination result of the device information identification unit in (1).

[0008] (3) In (1) or (2), the effect of ambient light that hinders the identification of information about the device is the reflection of light on the surface of the device or halation due to backlighting, an information display device.

[0009] (4) In any of (1) to (3), the device has a transparent cover that covers a display unit that shows the predetermined information, and the effect of ambient light that hinders the identification of information related to the device is the reflection of ambient light on the surface of the transparent cover, an information display device.

[0010] (5) In any of (1) to (4), the device information identification unit determines the type of the device using a trained model that has been trained on training data including a real image of the device that has been captured in advance.

[0011] (6)(5) The trained model has learned training data including real images in which the device is affected by ambient light that hinders the identification of information about the device, and the device information identification unit uses the trained model to determine whether or not there is an effect of ambient light that hinders the identification of information about the device, an information display device.

[0012] (7)(5) The trained model has learned training data including real images of the equipment in a deteriorated state that have been taken in advance, and the equipment information identification unit determines whether or not the equipment is in a deteriorated state using the trained model, an information display device.

[0013] (8)(1) The information display unit is an information display device that displays guidance information regarding services corresponding to the equipment identified by the equipment information identification unit.

[0014] (9)(1) to (6) The information display device has a surface relief enhancement image generation unit that generates a surface relief enhancement image based on the actual image that represents a color difference corresponding to the difference between the height of a predetermined part of the device in the normal direction of the predetermined part of the device and the height of a part to be compared, and the device information identification unit identifies information about the device based on the surface relief enhancement image.

[0015] In (10)(1), the device is a clock with a display unit showing time information fixed to a building or structure, and the device information identification unit is an information display device that determines the installation location and the shooting direction using a learning model learned from learning data which includes a real image of the clock that has been photographed in advance and is labeled according to the installation location where the clock is installed and the shooting direction in which the clock was photographed.

[0016] (11) An information display method comprising the steps of: acquiring a real image in which a device showing predetermined information is displayed by at least mechanical or electronic means; identifying information relating to the device based on the real image; and displaying information relating to the device, wherein in the step of identifying information relating to the device, it is determined, based on the real image, whether or not there is an influence of ambient light that occurs when acquiring the real image and hinders the identification of information relating to the device.

[0017] (12) A program that causes a computer to execute steps of obtaining a real image in which a device indicating predetermined information is displayed by at least one of mechanical or electronic means, identifying information about the device based on the real image, and displaying the information about the device. In the step of identifying the information about the device, the program determines whether there is an influence of external light that occurs when obtaining the real image and inhibits the identification of the information about the device based on the real image.

Effect of the Invention

[0018] According to aspects (1) to (12) of the present invention described above, it is possible to increase the accuracy of accurately determining information about a device displayed in an image.

Brief Description of the Drawings

[0019] [Figure 1] It is a diagram showing the overall configuration of an information display system according to the first embodiment. [Figure 2] It is a diagram showing an example of a teacher image that a learned model has learned. [Figure 3] It is a diagram showing an example of a correspondence explanation table. [Figure 4] It is a functional block diagram showing an example of functions implemented in a user terminal according to the first embodiment. [Figure 5] It is a flowchart showing the operation of an information display system according to the first embodiment. [Figure 6A] It is a diagram showing an example of an image displayed on a display device of a user terminal according to the first embodiment. [Figure 6B] It is a diagram showing an example of an image displayed on a display device of a user terminal according to the first embodiment. [Figure 6C] It is a diagram showing an example of an image displayed on a display device of a user terminal according to the first embodiment. [Figure 6D] It is a diagram showing an example of an image displayed on a display device of a user terminal according to the first embodiment. [Figure 7] It is a diagram showing an example of an embossed and highlighted image. [Figure 8] It is a diagram schematically showing a part of the concavo-convex emphasized image. [Figure 9] It is a diagram showing an example of an image displayed on the display device of the user terminal of the second embodiment. [Figure 10] FIG. 10 is a diagram showing the overall configuration of the information display system according to the third embodiment. [Figure 11] It is a functional block diagram showing an example of functions implemented in the user terminal of the third embodiment. [Figure 12] It is a diagram showing an example of an image displayed on the display device of the user terminal of the third embodiment. [Figure 13] FIG. 17 is a diagram showing the overall configuration of the information display system according to the fourth embodiment. [Figure 14] It is a functional block diagram showing an example of functions implemented in the user terminal of the fourth embodiment. [Figure 15] It is a diagram showing an example of a concavo-convex emphasized image in which a part of the exterior case is displayed. [Figure 16] It is a diagram showing an example of a concavo-convex emphasized image in which a part of the leather band is displayed. [Figure 17] FIG. 29 is a diagram showing the overall configuration of the information display system according to the fifth embodiment. [Figure 18] It is a functional block diagram showing an example of functions implemented in the user terminal of the fifth embodiment.

Embodiments for Carrying Out the Invention

[0020] Hereinafter, each embodiment of the present invention will be described in detail based on the drawings.

[0021] [Information Display System S] FIG. 1 is a diagram showing the overall configuration of the information display system according to the first embodiment. FIG. 2 is a diagram showing an example of a teacher image that the learned model is learning. FIG. 3 is a diagram showing an example of a correspondence explanation table.

[0022] The information display system S is a system that, on the user terminal 20, determines the model of an electronic clock based on a real image obtained by photographing the electronic clock, and displays information regarding its handling according to the operating status of the clock's hands.

[0023] In the first embodiment, the electronic watch to be photographed is described using an analog wristwatch as an example, which includes buttons and a crown as operating elements and performs various functions through the operation of these elements by the user. Functions that the electronic watch to be photographed may include, for example, a function to adjust the time by receiving GPS signals including time information, and a function to generate power using a built-in solar panel. Furthermore, the electronic watch to be photographed may be a wristwatch having hands that move on the dial and a transparent cover (also called a crystal) that covers the hands.

[0024] As shown in Figure 1, the information display system S includes a server 10 and a user terminal 20 as an information display device, and these can be connected to a network N such as the Internet. In Figure 1, one server 10 and one user terminal 20 are shown, but their numbers are arbitrary.

[0025] [Information display system S: Server 10] Server 10 is a server computer. Server 10 includes a control unit 11, a storage unit 12, and a communication unit 13.

[0026] The control unit 11 includes at least one processor. The control unit 11 executes processing according to programs and data stored in the storage unit 12. The storage unit 12 is composed of volatile memory such as RAM, or non-volatile memory such as ROM, EEPROM, or flash memory. In the first embodiment, the storage unit 12 stores at least a learned model 12a and a correspondence description table 12b. The communication unit 13 is a communication interface for wired or wireless communication and performs data communication via the network N.

[0027] The trained model 12a is generated by machine learning using training data that includes real images of electronic clocks that have been captured in advance, and which are labeled for each model. The trained model 12a can be trained using any known machine learning algorithm; for example, it is preferable that the features contained in the training images are automatically learned in each layer of a multi-layered neural network.

[0028] In the first embodiment, as shown in Figure 2, the training image displays models 1, 2, etc., and includes images labeled for each model. The training image also includes reflections. Note that models 1, 2, etc. are names conveniently assigned in the first embodiment to distinguish between types of equipment; in practice, equipment types should be distinguished by model numbers. A model number indicates the official name of the product and is represented, for example, by a symbol combining letters and numbers.

[0029] The training images for each model were taken with randomly changed shooting angles, indicator times, etc. Figure 2 shows examples of multiple images displaying Model 1, where the shooting angle and indicator time differ from each other, and multiple images displaying Model 2, where the shooting angle and indicator time differ from each other.

[0030] A reflected image is an image of an electronic clock in which an object near the electronic clock (for example, the user's face) is reflected on the transparent cover that covers the hands that move on the dial, due to the reflection of ambient light on the surface of the cover. In the first embodiment, regardless of the model, a common label is assigned to the images of electronic clocks exhibiting reflections and the system is trained to recognize them. Figure 2 shows examples of reflected images of electronic clocks exhibiting reflections, representing multiple different models.

[0031] As shown in Figure 3, the correspondence explanation table 12b is a table that associates the model of the electronic clock and the operating state of the hands with the corresponding section (corresponding page) in the instruction manual. In Figure 3, for example, in the instruction manual for model 1, page 10 contains information on how to handle the clock when the hands are stopped, and page 15 contains information on how to handle the clock when the time indicated by the hands is different from the time measured by the timing unit 24 of the user terminal 20.

[0032] [Information display system S: User terminal 20] The user terminal 20 is a computer operated by the user. The user terminal 20 is, for example, a mobile phone (including a smartphone), a personal information terminal (including a tablet computer and a wearable device), or a personal computer. In the first embodiment, an example in which the user terminal 20 is a smartphone will be described.

[0033] As shown in Figure 1, the user terminal 20 includes a control unit 21, a storage unit 22, a communication unit 23, a timing unit 24, a camera 25, an input device 26, and a display device 27. The physical configurations of the control unit 21, storage unit 22, and communication unit 23 are similar to those of the control unit 11, storage unit 12, and communication unit 13, respectively, and a detailed explanation is omitted.

[0034] The timing unit 24 measures the current time. The current time measured by the timing unit 24 may be set using time synchronization technology such as NITZ (Network Identity and Time Zone). The camera 25 is a device for photographing the subject. The input device 26 may be a touch panel or the like. The display device 27 is a device having a screen on which information is displayed, and may be, for example, a liquid crystal display or an organic EL display.

[0035] Figure 4 is a functional block diagram showing an example of functions implemented in the user terminal of the first embodiment. Note that the user terminal 20 may also have functions other than those shown in Figure 4. As shown in Figure 4, the user terminal 20 includes a real image acquisition unit 31, a model determination unit (device information identification unit) 32, a surface texture enhancement image generation unit 33, an operation determination unit 34, an information display unit 35, and a notification unit 36. These units are preferably mainly implemented as a control unit 21.

[0036] The real image acquisition unit 31 acquires a real image in which an electronic clock (a device that displays predetermined information) showing time information is displayed. Here, a real image is an image that is captured by the camera 25 and displayed on the display device 27 of the user terminal 20, and is an image that has not been specially processed. In the first embodiment, the real image acquisition unit 31 may acquire at least two still images as real images used by the operation determination unit 34 to determine the operation state of the pointer, namely a first real image and a second real image taken after a predetermined time has elapsed since the first image was taken. Note that the real image acquired by the real image acquisition unit 31 may be a moving image composed of a plurality of consecutive still images.

[0037] The model determination unit 32 determines the model of the electronic clock using the trained model 12a based on the real image acquired by the real image acquisition unit 31. In the first embodiment, the model determination unit 32 may use deep learning technology to determine which of the pre-trained training images the given input image (the real image acquired by the real image acquisition unit 31 in the first embodiment) is closest to.

[0038] The trained model 12a may be sent from the server 10 in response to a request from the user terminal 20 and temporarily stored in the memory unit 22 of the user terminal 20. The model determination unit 32 may then use the trained model 12a stored in the memory unit 22 to determine the model of the electronic clock.

[0039] The surface contour enhancement image generation unit 33 generates a surface contour enhancement image based on the actual image acquired by the actual image acquisition unit 31. The surface contour enhancement image is an image generated by processing the actual image and represents the color difference corresponding to the difference between the height of the dial and the height of the hands in the normal direction of the dial of the electronic clock. Details of the surface contour enhancement image will be described later with reference to Figures 7 and 8.

[0040] The operation determination unit 34 determines the operating state of the hands of the electronic clock being photographed based on the surface texture enhancement image generated by the surface texture enhancement image generation unit 33. Specifically, the operation determination unit 34 determines whether the hands of the electronic clock are operating normally. More specifically, the operation determination unit 34 determines whether the hands of the electronic clock are stopped, and whether the time indicated by the hands of the electronic clock is off from the current time measured by the timekeeping unit 24. The hands whose operating state is determined by the operation determination unit 34 should preferably be at least the second hand. However, the operation determination unit 34 may also determine the operating state of the minute hand and hour hand, not just the second hand.

[0041] The information display unit 35 refers to the correspondence explanation table 12b described above and displays information regarding the handling of the electronic clock, corresponding to the model of the electronic clock being photographed and the operating status of the electronic clock's hands, on the display device 27 of the user terminal 20.

[0042] If the model determination unit 32 determines that the electronic clock displayed in the image acquired by the actual image acquisition unit 31 is a reflection, that is, if it determines that there is an influence of ambient light that occurs when acquiring the actual image and hinders the identification of information about the electronic clock, the notification unit 36 ​​prompts the user to retake the image with the camera 25. The notification unit 36 ​​may prompt the user to retake the image by displaying text prompting retake on the display device 27 of the user terminal 20, or by prompting retake by voice or the like. In addition, if the subject to be photographed displayed in the image acquired by the actual image acquisition unit 31 is a model not stored in the correspondence explanation table 12b, the notification unit 36 ​​may notify the user that there is an error.

[0043] [Operation of Information Display System S] Next, the operation of the information display system according to the first embodiment will be described with reference to Figures 5 and 6A to 6D. Figure 5 is a flowchart showing the operation of the information display system according to the first embodiment. Figures 6A to 6D are diagrams showing examples of images displayed on the display device of the user terminal of the first embodiment. Specifically, Figure 6A is an example of a screen when the camera is started, Figure 6B is an example of a screen showing the model determination result, Figure 6C is an example of a screen showing the operation determination result, and Figure 6D is an example of a screen showing the corresponding section of the instruction manual.

[0044] First, the user operates the input device 26 of the user terminal 20, which starts the application installed on the user terminal 20 (step S11). After the application starts, the user terminal 20 requests the trained model 12a from the server 10 (step S12). The server 10 receives the request from the user terminal 20 (step S21) and sends the trained model 12a to the user terminal 20 (step S22).

[0045] The user terminal 20 receives the trained model 12a and stores the received trained model 12a in the storage unit 22 (step S13). The user terminal 20 also activates the camera 25 (step S14). At this time, it is preferable that the screen of the user terminal 20 displays an image with digital information for alignment added, as shown in Figure 6A.

[0046] Figure 6A shows an example where a "circular" shape, corresponding to the outer shape of the dial of the electronic watch being photographed, is displayed. Figure 6A also shows an example where text indicating the positions of the crown and buttons is displayed along with the "circular" shape. This display on the screen allows the user to photograph the electronic watch with the 12 o'clock position facing upwards (upwards in Figure 6A). Furthermore, as shown in Figure 6A, it would be desirable for the user terminal 20 screen to display a message to the user regarding the photography method.

[0047] When the camera 25 of the user terminal 20 is activated and the subject to be photographed is captured, the actual image acquisition unit 31 acquires an actual image (step S15). The actual image acquisition unit 31 should acquire at least two actual images (two frames) taken at predetermined time intervals.

[0048] Subsequently, the model determination unit 32 uses the learned model 12a stored in the memory unit 22 to determine the model of the electronic clock displayed in the actual image, and the determination result is obtained (step S16). Figure 6B shows an example of the screen when the determination result is that the model number is "ABC-12".

[0049] If the determination result is that the image is a reflection (Yes in step S17), the notification unit 36 ​​prompts the user to take another picture with the camera 25 (step S18). In other words, it prompts the user to reacquire a real image with the real image acquisition unit 31. The model determination unit 32 may determine that an image is a reflection if any one of the multiple real images acquired by the real image acquisition unit 31 is a reflection.

[0050] If the result indicates that the image is not a reflection (No. in step S17), the surface texture enhancement image generation unit 33 generates a surface texture enhancement image based on the actual image (step S19). It is preferable that at least two surface texture enhancement images be generated based on actual images taken at predetermined time intervals. The predetermined time interval should be 1 second or more and within 59 seconds. This is because if the two actual images are taken within 1 second of each other, the indicator will be judged as stopped even though the indicator is moving normally. Also, if the second actual image is taken exactly 60 seconds after the first actual image is taken, the indicator will be judged as stopped even though the second hand is in the same position and moving normally.

[0051] Furthermore, the operation determination unit 34 acquires time information related to the current time measured by the timing unit 24 (S110). Then, the operation determination unit 34 determines the operating state of the electronic clock's hands based on the time information related to the current time and the relief-enhanced image, and obtains the determination result (step S111). Figures 6A to 6D show how the current time measured by the timing unit 24 is displayed on the screen of the user terminal 20. Specifically, it is shown that the current time is 14:15:45.

[0052] Specifically, the operation determination unit 34 compares the time indicated by the pointer displayed in at least one texture-enhanced image with the current time measured by the timing unit 24. If they do not match, it determines that there is a "time discrepancy". The operation determination unit 34 also compares at least two texture-enhanced images. If the positions of the pointers displayed in those images match, the operation determination unit 34 determines that the pointer has stopped. Figure 6C shows an example of the screen when the operation determination unit 34 determines that the pointer has stopped.

[0053] Meanwhile, the operation determination unit 34 compares the time indicated by the pointer displayed in at least two relief-enhanced images with the current time measured by the timing unit 24, and if they match, it determines that the hands are moving normally.

[0054] The user terminal 20 transmits information about the model and operating status to the server 10 (step S112). The server 10 receives the model and information about the model (step S23), and, referring to the correspondence explanation table 12b stored in the storage unit 12, reads out information about how to handle the equipment according to the model and operating status (hereinafter referred to as correspondence explanation information) (step S24). Then, the server 10 transmits the read correspondence explanation information to the user terminal 20 (step S25).

[0055] The user terminal 20 receives corresponding explanatory information (step S113). Then, the information display unit 35 displays the corresponding explanatory information on the display device 27 of the user terminal 20 (step S114). Figure 6D shows a portion of the instruction manual for the electronic clock model "ABC-12" when the hands are stopped. Specifically, it shows a page from the instruction manual for model "ABC-12" that shows an example of the operations the user needs to perform when adjusting the time by receiving radio waves. Note that the switch from the screen shown in Figure 6C to the screen shown in Figure 6D may be performed automatically or based on the user's operation of the input device 26. For example, after the screen shown in Figure 6C is displayed, a link to access the instruction manual may be displayed on the display device 27, and after the user accesses the link by operating the input device 26, the screen may switch to the screen shown in Figure 6D.

[0056] It should be noted that there may be multiple possible causes for the hands of an electronic clock to stop, or for the time indicated by the hands to be inaccurate. For example, possible causes of stopping include a drop in power supply voltage and a malfunction in the microcomputer contained in the electronic clock. Possible causes of time inaccuracy include wear and tear on the mechanism, magnetic influence, and dust ingress. When there are multiple possible causes, the information display unit 35 should prioritize displaying explanatory information on the cause that is most likely to be the cause of the stoppage or time inaccuracy. If the problem is not resolved, the information display unit 35 should then display explanatory information on the next most likely cause. It should be noted that information regarding the possible causes of the stoppage or time inaccuracy should be stored in the user terminal 20 in advance based on past examples.

[0057] The correspondence explanation information may include text data and be displayed as text on the display device 27 as shown in Figure 6D. The information display unit 35 partially displays the text data corresponding to the operating status of the guide. Here, "partial display" may refer to the corresponding page in the instruction manual, or only the corresponding part of the corresponding page. However, the correspondence explanation information is not limited to text data; for example, it may also refer to videos or animations that dynamically explain how to use the equipment.

[0058] If the operation determination unit 34 determines that the pointer has stopped, then if the electronic clock has a built-in solar panel, it is sufficient to display the section of the instruction manual that explains how to generate power. If the electronic clock does not have a built-in solar panel, it is sufficient to display the section of the instruction manual that explains how to replace the battery.

[0059] Furthermore, if the operation determination unit 34 determines that there is a "time discrepancy," if it is a radio-controlled clock, it is sufficient to display the section of the instruction manual that explains how to forcibly receive signals including time information. If it is not a radio-controlled clock, it is sufficient to display the section of the instruction manual that explains how to adjust the pointer position appropriate for that model.

[0060] If the operation determination unit 34 determines that the hands are moving normally, the information display unit 35 may simply display that fact on the display device 27 of the user terminal 20, and then terminate the application's operation. Alternatively, if the operation determination unit 34 determines that the hands are moving normally, the information display unit 35 may, for example, display the front page of the instruction manual for that model on the screen of the user terminal 20. The user can then access the page of the instruction manual they wish to read by, for example, scrolling the screen of the user terminal 20.

[0061] [Image emphasizing unevenness] Here, with reference to Figures 7 and 8, the surface texture enhancement image generated by the surface texture enhancement image generation unit 33 will be explained. Figure 7 is a diagram showing an example of a surface texture enhancement image. Figure 8 is a diagram schematically showing a part of a surface texture enhancement image. Each grid-like section shown in Figure 8 represents one pixel. In Figure 8, lighter colored sections represent pixels with low brightness, and darker colored sections represent pixels with high brightness.

[0062] Here, the real image acquisition unit 31 acquires a real image corresponding to the color of the actual object being photographed. For example, in the case of an electronic clock that is the object being photographed, if the color of the dial and the color of the hands are similar in color, the colors of the dial and hands displayed in the real image will also be similar in color. Therefore, if the color of the dial and the color of the hands are close to each other, it may become difficult for the operation determination unit 34 to recognize the hands, making it difficult to determine the operation of the hands. It would be good to have a difference between the color of the dial and the color of the hands, but in that case the degree of design freedom would decrease. Therefore, in the first embodiment, a configuration was adopted that improves the accuracy of the operation determination unit 34's determination of the operation state of the hands without impairing the degree of design freedom.

[0063] Specifically, as described above, the user terminal 20 employs a configuration in which the surface contour enhancement image generation unit 33 generates a surface contour enhancement image based on the actual image acquired by the actual image acquisition unit 31.

[0064] The relief-enhanced image is an image from which information about the color of each pixel of the real image has been removed, and in which each pixel represents the color corresponding to the height of each part in the normal direction of the dial. In the first embodiment, the relief-enhanced image is an image that represents the color difference corresponding to the difference between the height of the dial and the height of the hands in the normal direction of the dial. The relief-enhanced image may be an image that is generated by utilizing the fact that the degree of reflection of ambient light differs depending on the difference in height between the dial and the hands.

[0065] Furthermore, the relief-enhancing image should be one that emphasizes the pointer by highlighting the shadows around it. Specifically, the area around the pointer, which is higher than the dial in the normal direction, receives less ambient light and appears dark due to the shadows. By representing these shadowed areas with pixels of low brightness, the pointer is displayed more prominently. The relief-enhancing image should ideally be, for example, a known normal vector map.

[0066] The motion determination unit 34 may, for example, recognize as a guide area in the relief-enhanced image a portion where pixels of the same color are lined up linearly for a predetermined length or longer. Alternatively, the motion determination unit 34 may recognize as a guide area a portion surrounded by pixels with low brightness. In Figure 8, the portion recognized as a guide area by the motion determination unit 34 is shown with a dashed line.

[0067] In this way, by eliminating the actual color of the electronic clock and adopting a configuration that determines the operation state of the hands based on an image that enhances the unevenness according to the height of each part of the electronic clock, the accuracy of determining the operation state of the hands is improved. Furthermore, since the color of the dial, the color of the hands, etc. can be freely determined, the degree of design freedom is not compromised.

[0068] [summary] In the information display system S described above, users can easily access information related to the operation of the electronic watch (information related to the device). Specifically, users can access the corresponding section of the instruction manual for their electronic watch simply by taking a picture of their electronic watch with the camera 25 mounted on the user terminal 20. Therefore, even if the paper instruction manual is lost or the model of the electronic watch owned by the user is unknown, they can still access the corresponding section of the instruction manual for that electronic watch. Furthermore, for electronic watches with complex functions, the instruction manual tends to be lengthy, making it difficult for users to find the relevant section. However, by adopting the information display system S of the first embodiment, it becomes possible to easily access the section of the instruction manual that needs to be read depending on the operating state of the electronic watch's hands. In addition, the first embodiment employs a configuration that determines the operating state of the hands based on an image with reflections removed, thus enabling accurate determination of the operating state.

[0069] In the first embodiment, an example was described in which the model of an electronic clock is determined using a trained model 12a in order to improve the accuracy of model determination. However, this is just one example, and the model of an electronic clock may also be determined by image recognition without using the trained model 12a. Furthermore, in the first embodiment, an example was described in which the trained model 12a is trained using reflected images as training images. However, the invention is not limited to this, and a trained model that has not been trained on reflected images may also be used.

[0070] Furthermore, while the first embodiment described an example of determining the pointer's operating state based on an image with enhanced surface texture, the invention is not limited to this, and the pointer's operating state may also be determined based on a real image that has not undergone surface texture enhancement processing.

[0071] In the first embodiment, an example was described in which functions such as model detection are implemented on the user terminal 20. However, the invention is not limited to this, and some of the functions shown in Figure 4 may be executed on the server 10 side.

[0072] In the first embodiment, an example of determining the operating state of the second hand as a guideline was described, but the system is not limited to this. For example, if the target of the camera is an electronic clock that does not have a second hand, it is preferable to determine the operating state of the minute hand. For example, the presence or absence of a second hand can be determined based on the actual image acquired by the actual image acquisition unit 31, and if it is determined that there is no second hand, a process can be performed to determine the operating state of the minute hand. In this case, for example, the operation determination unit 34 can determine the operating state of the minute hand based on the first image and a second actual image taken one minute or more after the first image was taken. It is preferable that the information display unit 35 displays on the display device 27 of the user terminal 20 that the shooting time (shooting interval) will be long. Alternatively, before the camera is activated (step S14 in Figure 5), information regarding whether or not the electronic clock to be photographed has a second hand may be input in advance by user operation. Then, if information that the electronic clock to be photographed does not have a second hand is input by user operation, a process can be performed to determine the operating state of the minute hand.

[0073] Furthermore, if the electronic clock being photographed has the ability to connect to the user terminal 20 via short-range wireless communication using Bluetooth® or the like, the information display unit 35 may display corresponding explanatory information on the display device 27 of the user terminal 20, as well as information regarding whether or not to synchronize with the time of the user terminal 20. If the user selects to synchronize the time, the electronic clock's time may be corrected based on the current time of the user terminal 20. Alternatively, the time may be corrected automatically without confirming whether or not to synchronize the time. After the time correction is complete, the information display unit 35 may display on the display device 27 of the user terminal 20 that the time has been corrected.

[0074] In the case of wristwatches, even if the watch body has the same shape, different band types (color scheme, pattern, shape, material, etc.) may result in different model numbers. Therefore, it is preferable that the training images pre-trained in the trained model 12a are labeled with the model (model number) assigned based on the combination of the watch body and band type. Furthermore, it is preferable that the image acquired by the real image acquisition unit 31 displays at least a part of the band in addition to the watch body. The model determination unit 32 then uses the trained model 12a to determine the model (device information) of the electronic watch based on the real image including the watch body and band.

[0075] Furthermore, although the first embodiment used an electronic clock as an example of a device to be photographed, it is not limited to this, and any device equipped with a pointer that moves on a dial is preferable. For example, the device to be photographed may be a water meter or a pressure meter equipped with a pointer that moves on a dial. For example, if the device to be photographed is a water meter, the user terminal 20 may determine the model of the water meter using the model determination unit 32, determine the operating state of the pointer using the operation determination unit 34, and then display information regarding the handling of the water meter (e.g., how to read the meter) on the screen based on these determination results.

[0076] The information display device of the first embodiment described above preferably includes: an image acquisition unit that acquires an image of a device equipped with a pointer that moves on a dial; a model determination unit that determines the type of the device based on the image; an operation determination unit that determines the operating state of the pointer based on the image; and an information display unit that displays information regarding the handling of the device according to the type of device and the operating state.

[0077] Furthermore, the device type determination unit may determine the type of device using a trained model that has been trained on training data including actual images of the device that have been captured in advance.

[0078] Furthermore, the trained model has learned training data including real images in which the device is displayed in which external light is reflected on the surface of the transparent cover that covers the indicator, and the model determination unit may use the trained model to determine whether or not external light is reflected on the surface of the transparent cover.

[0079] Furthermore, the information display device may also have a notification unit that prompts the user to reacquire the actual image using the actual image acquisition unit if the model determination unit determines that external light reflection is occurring on the surface of the transparent cover.

[0080] Furthermore, the information display device may include text data for information regarding the handling of the equipment, and the information display unit may partially display the parts of the text data corresponding to the operating status of the guideline.

[0081] Furthermore, the information display device may further include a timing unit that measures the current time, and the operation determination unit may determine the operating state by comparing the time indicated by the indicator of the device displayed in the actual image with the current time measured by the timing unit.

[0082] Furthermore, the actual image includes a first actual image and a second actual image taken after a predetermined time has elapsed since the first image was taken, and the operation determination unit may determine the operation state by comparing the first actual image and the second actual image.

[0083] Furthermore, the information display device has a relief enhancement image generation unit that generates a relief enhancement image based on the actual image, which represents the color difference corresponding to the difference between the height of the dial and the height of the pointer in the normal direction of the dial, and the operation determination unit determines the operation state of the pointer based on the relief enhancement image.

[0084] In the information display method of the first embodiment, it is preferable to include the steps of: acquiring an actual image of a device having a pointer that moves on a dial; determining the type of the device based on the actual image; determining the operating state of the pointer based on the actual image; and displaying information regarding the handling of the device according to the type of device and the operating state.

[0085] In the program of the first embodiment, it is preferable to have the computer execute the following steps: acquire an actual image showing a device equipped with a pointer that moves on a dial; determine the type of the device based on the actual image; determine the operating state of the pointer based on the actual image; and display information regarding the handling of the device according to the type of device and the operating state.

[0086] [Second Embodiment] Next, a second embodiment will be described with reference to Figure 9. Figure 9 is a diagram showing an example of an image displayed on the display device of the user terminal in the second embodiment.

[0087] In the first embodiment described above, an example was given in which the instruction manual is displayed on the display device 27 according to the model and operating status of the electronic clock. However, the information about the device displayed on the display device 27 is not limited to this. Furthermore, it is not limited to displaying information about the device according to the model and operating status of the electronic clock; it may also display information about the device according only to the model of the electronic clock.

[0088] In the second embodiment, an example is described in which guidance information regarding services corresponding to the model of the electronic clock is displayed. The user terminal 220 in the second embodiment does not necessarily have the operation determination unit 34 described in the first embodiment. In the second embodiment, the same reference numerals are used for components that have the same functions as those described in the first embodiment, and their descriptions are omitted.

[0089] Figure 9 shows an example where the display device 27 of the user terminal 220 displays "Model A," indicating the model of the photographed electronic watch, and a predetermined authentication key, which is guidance information. The authentication key is information used to receive services corresponding to the model of the electronic watch. For example, it would be good if the user of the electronic watch could enter the authentication key on a specific website to obtain an image issued exclusively to users who own that electronic watch. The issued image could be, for example, an illustration that can be sent instead of text when users converse with each other via chat. Furthermore, the displayed guidance information is not limited to the authentication key, and may also be the URL (Uniform Resource Locator) of a website where the image issued exclusively to users who own that electronic watch can be obtained. The owner of the electronic watch may then be able to obtain the image issued exclusively to the owner of that electronic watch by accessing the website.

[0090] In the second embodiment, as in the first embodiment, if the model determination unit 32 determines that the electronic clock displayed in the image acquired by the actual image acquisition unit 31 is a reflection image, the notification unit 36 ​​may prompt the user to retake the image with the camera 25. The notification unit 36 ​​may prompt the user to retake the image by displaying text prompting retake on the display device 27 of the user terminal 20, or by prompting retake through voice or other means. As a result of retaking the image, an image that is not a reflection image can be acquired. Therefore, in the second embodiment, the accuracy of providing a service according to the model of the electronic clock can be increased.

[0091] [Third Embodiment] Next, a third embodiment will be described with reference to Figures 10 to 12. Figure 10 is a diagram showing the overall configuration of the information display system according to the third embodiment. Figure 11 is a functional block diagram showing an example of the functions implemented in the user terminal of the third embodiment. Figure 12 is a diagram showing an example of an image displayed on the display device of the user terminal of the third embodiment.

[0092] In the third embodiment, an example is described in which the device information identification unit 332 determines the deterioration state of the electronic clock, and the information display unit 335 displays information regarding the deterioration state as information about the device. In the third embodiment, the same reference numerals are used for components that have the same functions as those described in the first embodiment, and their descriptions are omitted.

[0093] In the third embodiment, the server 310 is a server computer. The server 310 includes a control unit 311, a storage unit 312, and a communication unit 313.

[0094] The control unit 311 includes at least one processor. The control unit 311 performs processing according to programs and data stored in the storage unit 312. The storage unit 312 consists of volatile memory such as RAM, or non-volatile memory such as ROM, EEPROM, or flash memory. In the third embodiment, the storage unit 312 stores at least a learned model 312a and a corresponding information table 312b. The communication unit 313 is a communication interface for wired or wireless communication and performs data communication via the network N.

[0095] In the third embodiment, the trained model 312a is generated by machine learning from training data that includes real images of electronic clocks that have been previously captured, and which are labeled according to the model and the deterioration state of that model. The trained model 312a can be trained using any known machine learning algorithm; for example, features included in the training images may be automatically learned in each layer of a multilayer neural network.

[0096] In the third embodiment, the training images display models 1, 2, etc., photographed with randomly changed shooting angles, the time indicated by the pointer, etc., and also include images labeled according to the model and the deterioration state of that model. Each training image is a photograph of an electronic clock with varying degrees of deterioration and locations of deterioration. Specifically, the training images may include, for example, an image of model 1 without deterioration, an image of model 1 with deterioration, an image of model 2 without deterioration, an image of model 2 with deterioration, etc.

[0097] Furthermore, the training images include reflection images. A reflection image is an image of an electronic clock in which an object near the electronic clock (for example, the user's face) is reflected in the transparent cover, due to the reflection of ambient light on the surface of the transparent cover that covers the hands that move on the dial. In the third embodiment, it is preferable to assign a common label to images in which reflections occur, regardless of the model or state of deterioration, and to train the system accordingly.

[0098] Furthermore, the electronic clock displayed in the teacher image may not only be one that has deteriorated over time due to use, but may also be one that has deteriorated through accelerated degradation tests such as temperature and humidity tests or light resistance tests.

[0099] The user terminal 320, which is an information display device in the third embodiment, includes a real image acquisition unit 331, a device information identification unit 332, an information display unit 335, and a notification unit 336, as shown in Figure 11. These units are preferably mainly implemented as the control unit 21 of the user terminal 320.

[0100] The real image acquisition unit 331 acquires a real image in which an electronic clock (a device that displays predetermined information) showing time information is displayed. Here, a real image is an image that is captured by the camera 25 and displayed on the display device 27 of the user terminal 320, and is an image that has not been specially processed. In the third embodiment, the real image acquisition unit 331 may acquire a still image.

[0101] The device information identification unit 332 determines the model and deterioration status of the electronic watch displayed in the actual image based on the actual image acquired by the actual image acquisition unit 331. In the third embodiment, the device information identification unit 332 may use deep learning technology to determine which of the pre-trained training images the given input image (the actual image acquired by the actual image acquisition unit 331 in the third embodiment) is closest to. The deterioration status may be determined, for example, based on the color information of the actual image in which the electronic watch is displayed. Specifically, if the brightness of the color in the actual image has decreased due to discoloration or peeling paint on the leather band of the electronic watch displayed in the actual image, it may be determined that the electronic watch is in a deteriorated state.

[0102] The trained model 312a may be transmitted from the server 10 in response to a request from the user terminal 320 and temporarily stored in the memory unit 22 of the user terminal 320. The device information identification unit 332 may then use the trained model 12a stored in the memory unit 22 to determine the model and deterioration status of the electronic clock.

[0103] The information display unit 335 displays the degradation information determined by the equipment information identification unit 332 on the display device. The notification unit 336 provides a predetermined notification according to the determination result of the equipment information identification unit 332. Specifically, if the equipment information identification unit 332 determines that the image is a reflection, the notification unit 336 may prompt the user to retake the image with the camera 25. By adopting this configuration for determining whether or not the image is a reflection, it is possible to prompt the user to retake the image and increase the accuracy of accurately determining the degradation state. The notification unit 336 may prompt the user to retake the image by displaying text prompting retake on the display device 27 of the user terminal 20, or by prompting retake through voice or other means.

[0104] The correspondence information table 312b is a table that associates the model and deterioration status of an electronic watch with the information displayed when it is determined that the watch is in a deteriorated state. Figure 12 shows an example of the screen when the electronic watch is model A and the band is determined to be in a deteriorated state. Also in Figure 12, as an example of the information displayed when the band is determined to be in a deteriorated state, the information display unit 335 shows an example of access information to stores that provide band replacement services. In the example in Figure 12, the user can access information about stores that provide band replacement services for their electronic watch by clicking "here" where the access information is embedded.

[0105] By displaying information as shown in Figure 12, users of electronic watches can be encouraged to replace or change their watch bands. The information displayed when a band is determined to be in a deteriorated state may also include information about the average cost of replacement or exchange.

[0106] [Fourth Embodiment] Next, a fourth embodiment will be described with reference to Figures 13 to 16. Figure 13 is a diagram showing the overall configuration of the information display system according to the fourth embodiment. Figure 14 is a functional block diagram showing an example of a function implemented in the user terminal of the fourth embodiment. Figure 15 is a diagram showing an example of an image with enhanced bumps and ridges showing a part of the outer case. Figure 16 is a diagram showing an example of an image with enhanced bumps and ridges showing a part of the leather band.

[0107] In the fourth embodiment, an example is described in which the deterioration state of an electronic clock is determined based on an image with enhanced surface texture, and information regarding the deterioration state is displayed. In the fourth embodiment, the same reference numerals are used for components that have the same functions as those described in the first embodiment, and their descriptions are omitted.

[0108] The fourth embodiment is particularly effective when the deterioration of the electronic watch manifests as surface irregularities such as scratches on the outer case or tears in the leather band.

[0109] In the fourth embodiment, the server 410 is a server computer. The server 410 includes a control unit 411, a storage unit 412, and a communication unit 413.

[0110] The control unit 411 includes at least one processor. The control unit 411 executes processing according to programs and data stored in the storage unit 412. The storage unit 412 is composed of volatile memory such as RAM, or non-volatile memory such as ROM, EEPROM, or flash memory. In the fourth embodiment, the storage unit 412 stores at least a first learned model 412a, a plurality of second learned models 412b, and a corresponding information table 412c. The communication unit 413 is a communication interface for wired or wireless communication and performs data communication via the network N.

[0111] The first trained model 412a is generated by machine learning using training data that includes real images of electronic clocks that have been previously captured and labeled for each model. The first trained model 412a can be trained using any known machine learning algorithm; for example, it is preferable that the features contained in the training images are automatically learned in each layer of a multilayer neural network.

[0112] Each of the multiple second pre-trained models 412b is generated by machine learning from training data that includes a pre-photographed image of an electronic clock in a deteriorated state, along with an enhanced image of the clock's surface, each labeled for a predetermined part of the electronic clock. The predetermined parts may be, for example, the surface of the electronic clock's outer case when viewed from the front, the surface of the electronic clock's outer case when viewed from the left side, the leather band attached to the top of the electronic clock, the leather band attached to the bottom of the electronic clock, etc.

[0113] The user terminal 420, which is an information display device in the fourth embodiment, includes, as shown in Figure 14, a real image acquisition unit 431, a device information identification unit 432, a surface texture enhancement image generation unit 433, a part selection unit 41, a model selection unit 42, an information display unit 435, and a notification unit 436. These units are preferably mainly implemented as the control unit 21 of the user terminal 320.

[0114] The real image acquisition unit 431 acquires a real image in which an electronic clock (a device that displays predetermined information) showing time information is displayed. Here, a real image is an image that is captured by the camera 25 and displayed on the display device 27 of the user terminal 420, and is an image that has not been specially processed. In the fourth embodiment, the real image acquisition unit 431 may acquire a still image.

[0115] The device information identification unit 432 determines the model of the electronic clock displayed in the real image based on the real image acquired by the real image acquisition unit 431. In the fourth embodiment, the device information identification unit 432 may use deep learning technology to determine which of the pre-trained training images the given input image (in this case, the real image acquired by the real image acquisition unit 431) is closest to.

[0116] The first trained model 412a may be transmitted from the server 410 in response to a request from the user terminal 420 and temporarily stored in the memory unit 22 of the user terminal 420. The device information identification unit 432 may then use the first trained model 412a stored in the memory unit 22 to determine the model of the electronic clock.

[0117] The part selection unit 41 selects the part to be used for determining its deterioration status. The part selection by the part selection unit 41 is preferably performed in response to input operations by the user on the user terminal 420. For example, if the user wants to check the deterioration status of a leather band, the part selection unit 41 may select the leather band based on the user's input.

[0118] Furthermore, after the part selection unit 41 has selected the part to be judged for deterioration, the user should align the camera 25 with the part to be judged for deterioration. As a result, the image acquisition unit 431 should acquire an image in which the part to be judged for deterioration is displayed.

[0119] The model selection unit 42 selects a second trained model 412b from among several second trained models 412b that corresponds to the part of the electronic clock selected by the part selection unit 41.

[0120] The surface texture enhancement image generation unit 433 generates a surface texture enhancement image based on the actual image acquired by the actual image acquisition unit 431, which displays the area to be determined as being in a state of deterioration. The surface texture enhancement image is an image generated by processing the actual image and represents the color difference corresponding to the surface texture of the object being photographed.

[0121] The device information identification unit 432 determines the deterioration state of the electronic clock parts being photographed based on the surface texture enhancement image generated by the surface texture enhancement image generation unit 433, using the second trained model 412b selected by the model selection unit 42. In other words, the device information identification unit 432 uses deep learning techniques to determine which of the pre-trained training images the given input image (in this case, the surface texture enhancement image generated by the surface texture enhancement image generation unit 433) is closest to.

[0122] The second trained model 412b may be transmitted from the server 410 in response to a request from the user terminal 420 and temporarily stored in the memory unit 22 of the user terminal 420. The device information identification unit 432 may then use the second trained model 412b stored in the memory unit 22 to determine the deterioration state of the electronic clock.

[0123] The information display unit 435 displays the degradation information determined by the equipment information identification unit 432 on the display device. The notification unit 436 provides a predetermined notification according to the determination result of the equipment information identification unit 432. Specifically, if the equipment information identification unit 432 determines that the image is a reflection, the notification unit 436 may prompt the user to retake the image with the camera 25. By adopting this configuration for determining whether or not the image is a reflection, it is possible to prompt the user to retake the image and increase the accuracy of accurately determining the degradation state. The notification unit 436 may prompt the user to retake the image by displaying text prompting retake on the display device 27 of the user terminal 20, or by prompting retake through voice or other means.

[0124] The correspondence information table 412c is a table that associates parts of the electronic watch with information that is displayed when it is determined that the part is in a deteriorated state. In the fourth embodiment, as in the third embodiment described with reference to Figure 12, it is preferable that access information to stores that provide band replacement services be displayed on the display device 27.

[0125] Figure 15 shows an example of an enhanced image of the surface texture of a deteriorated exterior case. In this example, the enhanced image represents the color difference corresponding to the difference in height in the normal direction of the exterior case between areas where vertical and horizontal scratches have formed on the surface and areas where the surface is flat.

[0126] The device information identification unit 432 eliminates the actual color of the outer casing and employs a configuration that determines the state of deterioration based on an image that enhances the unevenness of the outer casing according to the height of the surface of the outer casing. This makes it possible to identify scratches that are difficult to see with the naked eye. In other words, it becomes possible to accurately determine the state of deterioration of the outer casing.

[0127] Figure 16 shows an example of an enhanced image of the surface texture of a deteriorated leather band. In this example, the enhanced image represents the color difference corresponding to the difference in height in the normal direction of the leather band between the areas where a tear has occurred on the surface and the areas where no tear has occurred.

[0128] The device information identification unit 432 eliminates the actual color of the leather band and employs a configuration that determines the state of deterioration based on an enhanced image of surface irregularities represented according to the height of the leather band's surface, making it possible to identify deterioration that is difficult to identify by visual inspection. In other words, it becomes possible to accurately determine the state of deterioration of the leather band. In particular, with leather bands, it is difficult to visually confirm whether it is just a streak or a tear.

[0129] In the fourth embodiment, the deterioration state of the outer case and leather band was used as an example, but the invention is not limited to this. For example, the deterioration state of the surface of a metal band may also be determined.

[0130] In the first to fourth embodiments described above, an example was shown where the object to be photographed is an analog wristwatch including a pointer. However, the invention is not limited to this, and any device that displays predetermined information, such as time information, by at least mechanical or electronic means is preferable. For example, it may be a digital wristwatch equipped with a display unit that digitally displays predetermined information, such as time information.

[0131] [Fifth Embodiment] Next, a fifth embodiment will be described with reference to Figures 17 and 18. Figure 17 is a diagram showing the overall configuration of the information display system according to the fifth embodiment. Figure 18 is a functional block diagram showing an example of the functions implemented in the user terminal of the fifth embodiment. In the fifth embodiment, the same reference numerals are used for components that have the same functions as those described in the first embodiment, and their descriptions are omitted.

[0132] In the first to fourth embodiments described above, an example was given in which a real image of a wristwatch is acquired and various information is displayed, but the invention is not limited to this. In the fifth embodiment, an example will be given in which a real image of a display clock is acquired and predetermined information is displayed. A display clock is a device also called a clock tower, clock tower, monument clock, etc. A display clock is preferably a clock that is powered and driven by an external power source. A display clock has a display unit that shows time information, and this display unit is fixed and placed at a high place on an exterior wall or pole of a building or monument, etc., located in front of a station, inside a station, in a park, a commercial facility, etc.

[0133] In the fifth embodiment, the server 510 is a server computer. The server 510 includes a control unit 511, a storage unit 512, and a communication unit 513.

[0134] The control unit 511 includes at least one processor. The control unit 511 performs processing according to programs and data stored in the storage unit 512. The storage unit 512 consists of volatile memory such as RAM, or non-volatile memory such as ROM, EEPROM, or flash memory. In the fifth embodiment, the storage unit 512 stores at least a learned model 512a and a corresponding information table 512b. The communication unit 513 is a communication interface for wired or wireless communication and performs data communication via the network N.

[0135] In the fifth embodiment, the trained model 512a is generated by machine learning from training data that includes real images of the equipment clock that have been previously photographed, and which are labeled for each installation location and shooting direction of the equipment clock. The trained model 512a can be trained using any known machine learning algorithm, for example, by having features contained in the training images automatically learned in each layer of a multilayer neural network. The installation location is the place where the equipment clock is installed, for example, in front of the north exit of XX Station, YY Park, etc.

[0136] In the fifth embodiment, the training image displays the equipment clocks 1, 2, etc., and includes images labeled according to the direction in which each equipment clock is photographed. The training image may also include the background of each equipment clock. Specifically, the training image may include, for example, an image obtained when equipment clock 1 is photographed facing northwest, an image obtained when equipment clock 1 is photographed facing southeast, an image obtained when equipment clock 2 is photographed facing north, an image obtained when equipment clock 2 is photographed facing southwest, and so on.

[0137] Furthermore, the training images should include images with halation. Halation refers to the phenomenon where the screen becomes blurry or cloudy due to excessive light from backlighting. Halation is more likely to occur when photographing subjects outdoors, especially on sunny days. Halation images are images in which halation occurs. In the fifth embodiment, it is preferable to assign a common label to images with halation, regardless of the installation location of the equipment clock and the shooting direction, and to train the model accordingly.

[0138] As shown in Figure 18, the user terminal 520 of the fifth embodiment has a real image acquisition unit 531, a device information identification unit 532, an information display unit 535, and a notification unit 536. The real image acquisition unit 531 acquires a real image on which a device clock (a device that displays predetermined information) indicating time information is displayed. Here, a real image is an image that is captured by the camera 25 and displayed on the display device 27 of the user terminal 20, and is an image that has not been subjected to any special processing.

[0139] The device information identification unit 532 determines the installation location and shooting direction of the equipment clock using a trained model based on the actual image acquired by the actual image acquisition unit 531. In the fifth embodiment, the device information identification unit 532 may use deep learning techniques to determine which of the pre-trained training images the given input image (the actual image acquired by the actual image acquisition unit 531 in the fifth embodiment) is closest to.

[0140] The trained model 512a may be transmitted from the server 10 in response to a request from the user terminal 520 and temporarily stored in the memory unit 22 of the user terminal 520. The equipment information identification unit 532 may then use the trained model 512a stored in the memory unit 22 to determine the installation location and shooting direction of the equipment clock.

[0141] The corresponding information table 512b is a table in which the equipment clock and an image showing the past appearance of the location where the equipment clock is installed are associated for each shooting direction. For example, if the user terminal 20 photographs the equipment clock facing southwest, the information display unit 535 should display an image showing the equipment clock and the surrounding landscape 50 years ago as seen from a southwest-facing perspective on the display device 27. This allows the user to see the landscape from 50 years ago from the same viewpoint as the current shooting position. The information displayed is not limited to images showing the past appearance; it can be any information related to the photographed equipment clock. Furthermore, the information displayed is not limited to still images; it may also be animations such as videos.

[0142] If the notification unit 536 determines that the equipment clock displayed in the image acquired by the actual image acquisition unit 531 by the equipment information identification unit 532 is affected by ambient light that hinders the identification of information related to the equipment clock, that is, if it is determined to be a halated image, it prompts the user to retake the image with the camera. As a result of retaking the image, the user can acquire an image without halation. Therefore, the accuracy of determining the installation location and shooting direction of the equipment clock can be increased. The notification unit 536 may prompt the user to retake the image by displaying text prompting retaking on the display device 27 of the user terminal 20, or by prompting retaking with voice or the like.

[0143] In the fifth embodiment, the equipment clock may have distinctive design features in the dial and surrounding structure on which the time information display unit is provided. Furthermore, the dial may be fixed in a predetermined direction. As a result, the appearance of the dial and surrounding structure may change depending on the direction from which the equipment clock is viewed. In other words, the equipment clock may have different design features depending on the viewing direction. In the fifth embodiment, by utilizing the design features of the equipment clock, the installation location and shooting direction of the equipment clock can be determined with high accuracy, and information related to the equipment clock can be displayed.

[0144] In the fifth embodiment, an example was described in which the device information identification unit 532 determines whether or not an image is a halation image using a trained model 512a, but the invention is not limited to this. For example, the device information identification unit 532 may determine whether or not an image is a halation image based on the brightness of the pixels in the actual image. Specifically, the device information identification unit 532 may determine whether or not an image is a halation image based on the average value of the RGB values ​​of all pixels in the actual image. The device information identification unit 532 may determine that an image is a halation image if the average value of the RGB values ​​of all pixels in the actual image is greater than or equal to a predetermined value.

Claims

1. A real image acquisition unit that acquires a real image of a device displaying predetermined information by at least mechanical or electronic means, Based on the aforementioned actual image, an equipment information identification unit identifies information about the equipment, An information display unit that displays information related to the aforementioned equipment, It has, The device information identification unit determines, based on the actual image, whether or not there is an influence of ambient light that occurs when acquiring the actual image and hinders the identification of information about the device, and determines the type of the device using a trained model that has been trained on training data including a real image in which the device is displayed, which has been captured in advance. The aforementioned trained model has learned training data including real images in which the device is displayed, in which ambient light is present that interferes with the identification of information about the device that has been captured in advance. The device information identification unit uses the trained model to determine whether or not there is an influence of ambient light that hinders the identification of information about the device. Information display device.

2. A real image acquisition unit that acquires a real image of a device displaying predetermined information by at least mechanical or electronic means, Based on the aforementioned actual image, an equipment information identification unit identifies information about the equipment, An information display unit that displays information related to the aforementioned equipment, It has, The device information identification unit determines, based on the actual image, whether or not there is an influence of ambient light that occurs when acquiring the actual image and hinders the identification of information about the device, and determines the type of the device using a trained model that has been trained on training data including a real image in which the device is displayed, which has been captured in advance. The aforementioned trained model has been trained on training data that includes real images of the device in a degraded state, which have been captured in advance. The device information identification unit uses the learned model to determine whether or not the device is in a deteriorated state. Information display device.

3. The device has a notification unit that performs a predetermined notification according to the determination result of the device information identification unit. The information display device according to claim 1 or 2.

4. The influence of ambient light that hinders the identification of information regarding the aforementioned device is the reflection of light on the surface of the device or halation due to backlighting. The information display device according to claim 1 or 2.

5. The device has a transparent cover that covers the display unit that shows the predetermined information, The influence of ambient light that hinders the identification of information regarding the aforementioned device is the reflection of ambient light on the surface of the transparent cover. The information display device according to claim 1 or 2.

6. A real image acquisition unit that acquires a real image of a device displaying predetermined information by at least mechanical or electronic means, Based on the aforementioned actual image, an equipment information identification unit identifies information about the equipment, An information display unit that displays information related to the aforementioned equipment, It has, The device information identification unit determines, based on the actual image, whether or not there is an influence of ambient light that occurs when acquiring the actual image and hinders the identification of information about the device. The information display unit displays guidance information used to obtain an image corresponding to the type of equipment identified by the equipment information identification unit. Information display device.

7. A real image acquisition unit that acquires a real image of a device displaying predetermined information by at least mechanical or electronic means, Based on the aforementioned actual image, an equipment information identification unit identifies information about the equipment, An information display unit that displays information related to the aforementioned equipment, It has, The device information identification unit determines, based on the actual image, whether or not there is an influence of ambient light that occurs when acquiring the actual image and hinders the identification of information about the device. The device has a surface texture enhancement image generation unit that generates a surface texture enhancement image based on the actual image, which represents the color difference corresponding to the difference between the height of the predetermined part and the height of the part to be compared in the normal direction of the predetermined part of the device, The aforementioned device information identification unit identifies information about the device based on the surface enhancement image. Information display device.

8. A real image acquisition unit that acquires a real image of a device displaying predetermined information by at least mechanical or electronic means, Based on the aforementioned actual image, an equipment information identification unit identifies information about the equipment, An information display unit that displays information related to the aforementioned equipment, It has, The device information identification unit determines, based on the actual image, whether or not there is an influence of ambient light that occurs when acquiring the actual image and hinders the identification of information about the device. The aforementioned device is a clock with a display unit showing time information that is fixed to a building or structure. The equipment information identification unit determines the installation location and the shooting direction using a learning model learned from learning data that includes pre-captured real images showing the equipment clock, and is labeled according to the installation location where the equipment clock is installed and the shooting direction from which the equipment clock was photographed. Information display device.

9. The steps include: acquiring a real image in which a device displaying predetermined information is shown by at least mechanical or electronic means; The steps include identifying information about the device based on the actual image, The steps include: displaying information about the aforementioned device; It has, In the step of identifying information about the device, based on the actual image, it is determined whether or not there is an influence of ambient light that occurs when acquiring the actual image and hinders the identification of information about the device, and the type of the device is determined using a trained model that has been trained on training data including a real image of the device that has been previously captured. The aforementioned trained model has learned training data including real images in which the device is displayed, in which ambient light is present that interferes with the identification of information about the device that has been captured in advance. In the step of identifying information about the equipment, the trained model is used to determine whether or not there is an influence of ambient light that hinders the identification of information about the equipment. Information display method.

10. The steps include: acquiring a real image in which a device displaying predetermined information is shown by at least mechanical or electronic means; The steps include identifying information about the device based on the actual image, The steps include: displaying information about the aforementioned device; A program that causes a computer to execute, In the step of identifying information about the device, based on the actual image, it is determined whether or not there is an influence of ambient light that occurs when acquiring the actual image and hinders the identification of information about the device, and the type of the device is determined using a trained model that has been trained on training data including a real image of the device that has been previously captured. The aforementioned trained model has learned training data including real images in which the device is displayed, in which ambient light is present that interferes with the identification of information about the device that has been captured in advance. In the step of identifying information about the equipment, the trained model is used to determine whether or not there is an influence of ambient light that hinders the identification of information about the equipment. program.

Images