Information processing device, information processing system, recording medium and information processing method

Abstract

An information processing device, an information processing system, a recording medium, and an information processing method. The information processing device comprises: a photographing unit; a recognition unit that recognizes that an object is displayed in a photographic image being photographed by the photographing unit; and a control unit that, when the recognition unit recognizes the object, causes a first image in which an image based on object information related to the object is added to the photographic image to be displayed on a display unit, and, when the recognition unit does not recognize the object, causes a second image different from the first image and including an image based on the object information to be displayed on the display unit.

Description

Technical Field

[0001] The present invention relates to an information processing device, an information processing system, a recording medium and an information processing method. Background Art

[0002] When an object such as a device is photographed and the object is recognized from the photographed image, the photographed image is sometimes displayed, and an image based on information related to the object is added to the photographed image and displayed. For example, this display is sometimes performed using AR (Augmented Reality) technology.

[0003] Patent document 1 describes an information processing device that determines a photographic part based on a photographic image taken by an image forming device, and when a designated part where an event to be notified occurs is included in the photographic part, sends first display information corresponding to the designated part to a mobile terminal, and when the designated part is not included in the photographic part, sends guidance information to the designated part to the mobile terminal.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2012-156872

[0005] In the technology in which, when an object is identified from a photographic image, an image based on information related to the object is attached to the photographic image and displayed, if the object is not identified from the photographic image, the image based on the information related to the object is not displayed and the user cannot confirm the information related to the object. Summary of the invention

[0006] An object of the present invention is to enable a user to confirm information related to an object even when the object is not recognized from a photographic image, when an image based on information related to the object is displayed when the object is not recognized from the photographic image.

[0007] The invention involved in Scheme 1 is an information processing device, which comprises: a photographic unit; a recognition unit that recognizes an object displayed in a photographic image being photographed by the photographic unit; and a control unit that, when the recognition unit recognizes the object, causes a first image in the photographic image to be attached with an image based on object information related to the object and displayed on the display unit; and, when the recognition unit does not recognize the object, causes a second image that is different from the first image and includes an image based on the object information to be displayed on the display unit.

[0008] The invention involved in Scheme 2 is the information processing device described in Scheme 1, wherein when the recognition unit recognizes the object, the control unit causes the first image to be displayed on the display unit, and after the recognition unit recognizes the object, when the recognition unit no longer recognizes the object, causes the second image to be displayed on the display unit.

[0009] The invention involved in Scheme 3 is an information processing device as described in Scheme 1 or Scheme 2, wherein the second image is an image based on the object information attached to an image that is different from the photographic image being photographed by the photographic unit and represents the object.

[0010] The invention according to claim 4 is the information processing device according to claim 1 or 2, further comprising a storage unit for storing the first image, wherein the second image is the first image stored in the storage unit.

[0011] The invention involved in Scheme 5 is an information processing device as described in any one of Schemes 1 to 4, wherein the control unit prevents the second image from being displayed on the display unit from the moment the first image is displayed on the display unit until a predetermined time has passed.

[0012] The invention involved in Scheme 6 is the information processing device described in Scheme 2, wherein after the recognition unit recognizes the object, when the state in which the recognition unit does not recognize the object continues for more than a predetermined time, the control unit causes the second image to be displayed on the display unit.

[0013] The invention involved in Scheme 7 is an information processing device as described in any one of Schemes 1 to 3, wherein when the information processing device is stationary in a state where the recognition unit does not recognize the object, the control unit causes the second image to be displayed on the display unit.

[0014] The invention according to claim 8 is the information processing device according to claim 7, wherein when the information processing device is moved, the control unit causes the photographic image to be displayed on the display unit.

[0015] The invention according to claim 9 is the information processing device according to any one of claims 1 to 8, wherein the control unit causes the display unit to display the second image together with the photographic image.

[0016] The invention involved in Scheme 10 is an information processing device according to any one of Schemes 1 to 8, wherein the control unit further causes an image to be displayed on the display unit, the image being used to switch between display of the photographic image and display of the second image.

[0017] The invention involved in Scheme 11 is an information processing device described in any one of Schemes 1 to 8, wherein when the second image is displayed on the display unit, the control unit also causes information to be displayed on the display unit, and the information is used to urge the user to switch the second image to the first image.

[0018] The invention involved in Scheme 12 is an information processing system, which includes an information processing device and an external device, the information processing device having a photographing unit and a control unit for displaying an image on a display unit, the external device having: a recognition unit for recognizing that an object is displayed in a photographic image being photographed by the photographing unit; and a generation unit for generating a first image by appending an image based on object information related to the object to the photographic image, and generating a second image that is different from the first image and includes an image based on the object information, wherein when the recognition unit recognizes the object, the control unit displays the first image on the display unit, and when the recognition unit does not recognize the object, the control unit displays the second image on the display unit.

[0019] The invention involved in Scheme 13 is a recording medium, which records a program that causes a computer to function as the following units: a recognition unit that recognizes an object displayed in a photographic image being photographed by a photographic unit; and a control unit that, when the recognition unit recognizes the object, causes a first image in which an image based on object information related to the object is attached to the photographic image to be displayed on a display unit, and when the recognition unit does not recognize the object, causes a second image that is different from the first image and includes an image based on the object information to be displayed on the display unit.

[0020] The invention described in Scheme 14 is an information processing method, which includes the following steps: an identification step, identifying an object displayed in a photographic image being photographed by a photographic unit; and a control step, when the object is identified in the identification step, causing a first image in the photographic image in which an image based on object information related to the object is attached to be displayed on the display unit, and when the object is not identified in the identification step, causing a second image different from the first image and containing an image based on the object information to be displayed on the display unit.

[0021] Effects of the Invention

[0022] According to the inventions involved in schemes 1, 12, 13 and 14, when an image based on information related to an object is displayed when the object is recognized from a photographic image, the user can confirm the information related to the object even when the object is not recognized from the photographic image.

[0023] According to the invention involved in claim 2, when an image based on information related to an object is displayed when the object is recognized from a photographic image, the user can continue to confirm the information based on the object even if the object is not recognized from the photographic image.

[0024] According to the invention pertaining to claim 3, even when the object is not recognized from the photographic image, the user can confirm information on the object from an image different from the photographic image.

[0025] According to the invention pertaining to claim 4, even when the object is not recognized from the photographed image, the user can confirm information related to the object based on the first image stored in the storage means.

[0026] According to the inventions according to the fifth and sixth aspects, when the object has not yet been recognized from the photographed image, the second image is not displayed.

[0027] According to the invention as set forth in claim 7, it is possible to display an image suitable for a situation in which the information processing apparatus is stationary.

[0028] According to the invention as set forth in claim 8, it is possible to display an image suitable for the situation in which the information processing apparatus is moving.

[0029] According to the invention pertaining to claim 9, the user can check the captured image while checking the information on the object.

[0030] According to the invention according to claim 10, it is possible to display an image desired by the user.

[0031] According to the invention pertaining to claim 11, it is possible to urge the user to photograph the object in a manner that allows the user to recognize the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Embodiments of the present invention will be described in detail with reference to the following drawings.

[0033] Figure 1 is a block diagram showing the structure of an information processing system according to the present embodiment;

[0034] Figure 2 is a block diagram showing the structure of a terminal device according to the present embodiment;

[0035] Figure 3 is a diagram schematically showing a terminal device and an image forming device;

[0036] Figure 4 is a diagram showing the structure of the first image data;

[0037] Figure 5is a diagram showing an example of display of a photographic image;

[0038] Figure 6 is a diagram showing a display example of the first image;

[0039] Figure 7 is a diagram showing an example of display of a photographic image;

[0040] Figure 8 is a diagram showing the structure of the second image data;

[0041] Fig. 9 is a diagram showing a display example of the second image;

[0042] Fig.10 is a flowchart showing processing based on a terminal device;

[0043] Fig.11 is a flowchart showing the processing involved in Modification Example 1;

[0044] Fig.12 is a diagram showing display conditions of the first image or the second image;

[0045] Fig.13 is a diagram showing a display example of a second image and a photographic image;

[0046] Fig.14 is a diagram showing an example of display of a photographic image;

[0047] Fig.15 is a diagram showing a display example of the second image;

[0048] Fig.16 is a diagram showing an example of display of a photographic image;

[0049] Fig.17 is a diagram showing a display example of the first recommended information;

[0050] Fig.18 is a diagram showing a display example of the second recommended information;

[0051] Fig.19 is a flowchart showing the processing involved in Modification Example 6;

[0052] Fig. 20 is a block diagram showing the structure of an information processing system according to Modification 7;

[0053] Fig.21 This is a block diagram showing the structure of a server involved in Modification Example 7.

[0054] Explanation of symbols

[0055] 10-terminal device, 12-object, 14-information management database, 22-photographing unit, 24-determining unit, 26-recognition unit, 28-acquisition unit. DETAILED DESCRIPTION

[0056] Below, reference Figure 1 , an information processing system involved in an embodiment of the present invention is described. Figure 1 An example of the configuration of the information processing system according to the present embodiment is shown in FIG.

[0057] The information processing system involved in this embodiment includes a terminal device 10 as an example of an information processing device. The terminal device 10 is, for example, a personal computer (hereinafter referred to as "computer"), a tablet computer, a smart phone, or a mobile phone. As an example, the terminal device 10 is a portable terminal device such as a tablet computer, a smart phone, or a mobile phone.

[0058] The object 12 is an object that is the object of information acquisition. As described later, the object 12 is photographed by the terminal device 10. The object 12 can be any object that can be photographed by the terminal device 10. For example, the object 12 is a tangible object that physically occupies a part of the space. Specifically, the object 12 is a device, furniture, a car, or a building. Of course, objects other than these can also be the object 12. Devices are, for example, objects called electronic devices, information devices, video devices, audio devices, or household appliances. For example, devices that process information or devices with communication functions are equivalent to an example of the object 12. In addition, the object 12 can also be a device that can communicate with the terminal device 10 and send and receive information. Furniture is, for example, a chair, a table, a sofa, etc. Buildings are, for example, a house, a store, etc. In addition, the object 12 can be an image displayed on a display device, characters or graphics recorded on paper, etc., a 3D image drawn in space, etc.

[0059] exist Figure 1 In the example shown, the terminal device 10 and the object 12 are connected to a communication path N and can communicate with other devices. Of course, the terminal device 10 and the object 12 may not be connected to the communication path N. In addition, a plurality of objects 12 may be connected to the communication path N. The communication path N is, for example, a network such as a LAN (Local Area Network) or the Internet. The communication path N can be constructed by wired communication or by wireless communication such as Wi-Fi (registered trademark). Furthermore, the terminal device 10 and the object 12 can communicate directly with other devices through short-range wireless communication, etc., instead of through the communication path N. As short-range wireless communication, for example, Bluetooth (registered trademark), RFID, NFC, etc. are used.

[0060] The information management database 14 is a device for managing object information of each object 12. The terminal device 10 can obtain the object information from the information management database 14 via the communication path N or by short-range wireless communication. The information management database 14 is set in the object 12, so that the terminal device 10 can obtain the object information from the object 12. In addition, the information management database 14 may not be used. In this case, the terminal device 10 can obtain the object information from an external device such as a server other than the information management database 14. In addition, all or part of the object information may be stored in the terminal device 10.

[0061] The object information is information related to the object 12. The object information may be information that changes according to the state of the object 12, or may be information that does not change according to the state of the object 12.

[0062] The state of the object 12 may be, for example, the state of the appearance of the object 12, the state of the interior of the object 12 (i.e., the state that cannot be seen from the outside), or the state of both the appearance and the interior. Information that changes according to the state of the object 12 is information that changes according to the state of the appearance or the interior. For example, the state of the appearance of a device as the object 12 is the opening and closing of a cover provided on the device. The object information changes according to the opening and closing of the cover. The internal state of the device is whether an error has occurred in the device. The object information changes according to the presence or absence of an error. The change in the state of the object 12 may be caused by the operation of the user who photographs the object 12. For example, the above-mentioned cover may be opened and closed by the user.

[0063] Information that does not change depending on the state of the object 12 is, for example, inherent information associated with the object 12. For example, identification information for identifying the object 12, installation location information indicating the location where the object 12 is installed, etc. are equivalent to an example of inherent information. Identification information is, for example, the name, ID, IP address, MAC address, etc. of the object 12. Installation location information is information indicating the floor where the object 12 is installed, information indicating the location where the object 12 is installed on the floor, information indicating the address where the object 12 is installed, coordinate information obtained by GPS (Global Positioning System), etc.

[0064] The object information includes, for example, the identification information, the location information, the type information indicating the type of the object 12, the state information indicating the state of the object 12, and the structure information indicating the structure of the object 12. Some of the location information, type information, state information, and structure information may be included in the object information. In addition, information indicating the content of services provided by the object 12 and information indicating the price of the object 12 may be included in the object information.

[0065] Below, reference Figure 2 , the structure of the terminal device 10 is described in detail. Figure 2 An example of the structure of the terminal device 10 is shown in FIG.

[0066] The communication unit 16 is a communication interface, and has a function of transmitting information to other devices and a function of receiving information from other devices. The communication unit 16 may have a wireless communication function or a wired communication function.

[0067] The UI unit 18 is a user interface unit, including a display unit and an operation unit. The display unit is a display device such as a liquid crystal display. The operation unit is an input device such as a keyboard and a mouse. The UI unit 18 may be a touch panel that has both a display unit and an operation unit. In addition, the UI unit 18 may include a voice output unit such as a speaker.

[0068] The storage unit 20 is one or more storage areas for storing information. Each storage area is constituted by, for example, one or more storage devices (for example, physical drives such as hard disk drives and memories) provided in the terminal device 10.

[0069] The photographing unit 22 is a photographing device such as a camera, and is configured to generate photographic image data by photographing. The photographic image data may be still image data or moving image data.

[0070] The determination unit 24 is configured to determine the object 12 as the object to be acquired for the object information. The determination unit 24 can determine the object 12 identified by the recognition unit 26 described later as the object 12 as the object to be acquired for the object information, and can determine the object 12 specified by the user as the object 12 as the object to be acquired for the object information. For example, a list of identification information of the object 12 is displayed on the display unit of the UI unit 18, and the user can specify the object 12 as the object to be acquired for the object information from the list. In addition, when the user inputs the identification information of the object 12 using the UI unit 18, the determination unit 24 can determine the object 12 having the identification information as the object 12 as the object to be acquired for the object information. The determination unit 24 outputs the identification information of the object 12 as the object to be acquired for the object information to the acquisition unit 28 described later. For example, when the object 12 is identified by the recognition unit 26, the object 12 acquires the identification information, and the determination unit 24 outputs the identification information to the acquisition unit 28. When the user specifies the object 12, the specifying unit 24 outputs the identification information of the object 12 specified by the user to the acquiring unit 28. When the user inputs the identification information, the specifying unit 24 outputs the identification information input by the user to the acquiring unit 28.

[0071] The recognition unit 26 is configured to recognize the object 12, which is displayed in the photographic image data generated by the photographing by the photographing unit 22. The recognition unit 26 recognizes the object 12 using the photographic image data currently photographed by the photographing unit 22. The photographic image data used for recognizing the object 12 is, for example, dynamic image data. The recognition unit 26 obtains the recognition information of the object 12 by recognizing the object 12.

[0072] The recognition unit 26 recognizes the object 12 by using, for example, a known image recognition technology. As the image recognition technology, for example, a markerless recognition technology or a marker recognition technology is used.

[0073] The markerless recognition technology is a technology for recognizing the object 12 by matching image data. For example, image data representing the entirety, a part, or a characteristic portion of the object 12 is associated with the identification information of the object 12 and stored in the terminal device 10, an external device such as a server, etc. The recognition unit 26 matches the photographic image data currently being photographed by the photographing unit 22 with the image data stored in the terminal device 10, an external device, etc., thereby recognizing the object 12 displayed in the photographic image data and acquiring the identification information associated with the image data. The recognition unit 26 can perform matching using features extracted from the photographic image data.

[0074] The mark type recognition technology is a technology for recognizing the object 12 using a mark pre-set on the object 12. The mark is a component that can read the identification information of the object 12. The mark may be, for example, a component that indicates that the identification information is encoded (e.g., a barcode, a two-dimensional code, etc.), or a component that indicates unencoded identification information. If the mark is photographed by the photographing unit 22, the recognition unit 26 obtains the identification information from the image data representing the mark to recognize the object 12. For example, when the two-dimensional code is photographed by the photographing unit 22, the recognition unit 26 obtains the identification information by decoding the two-dimensional code to recognize the object 12.

[0075] The acquisition unit 28 is configured to acquire object information of the object 12 determined by the determination unit 24. To explain in more detail, the acquisition unit 28 acquires object information including identification information output from the determination unit 24 to the acquisition unit 28. The acquisition unit 28 may acquire the object information from the information management database 14, may acquire the object information from the object 12, or may acquire the object information from an external device such as a server. In the case where all or part of the object information is stored in the storage unit 20, the acquisition unit 28 may acquire all or part of the object information from the storage unit 20. The acquisition unit 28 may recognize changes in the appearance of the object 12 based on the photographic image data, and acquire state information indicating the changes from the photographic image data. For example, in the case where the object 12 does not have a communication function, the acquisition unit 28 may acquire the state information from the photographic image data.

[0076] The control unit 30 is configured to control the operation of each unit of the terminal device 10. For example, the control unit 30 causes the display unit of the UI unit 18 to display various information.

[0077] For example, the control unit 30 causes the display unit of the UI unit 18 to display a photographic image based on the photographic image data currently being photographed by the photographing unit 22. When the photographing unit 22 photographs photographic image data as dynamic image data, the control unit 30 generates dynamic image data by photographing, and then causes the display unit of the UI unit 18 to display a dynamic image based on the dynamic image data.

[0078] The control unit 30 may display various information on the display unit of the UI unit 18 using augmented reality (AR) technology.

[0079] For example, when the recognition unit 26 recognizes the object 12, the control unit 30 displays the first image based on the first image data on the display unit of the UI unit 18; when the recognition unit 26 does not recognize the object 12, the control unit 30 displays the second image based on the second image data on the display unit of the UI unit 18.

[0080] The first image data is image data that is added to the photographed image data currently being photographed by the photographing unit 22 based on the object information acquired by the acquisition unit 28 .

[0081] The second image data is image data different from the first image data and includes image data based on the object information. The second image data is, for example, image data based on the object information added to image data representing the object 12 that is different from the photographic image data currently being photographed by the photographing unit 22.

[0082] Hereinafter, image data based on the object information is referred to as “additional image data.” Furthermore, image data that is different from the image data currently being photographed by the photographing unit 22 and that represents the object 12 is referred to as “substitute image data.”

[0083] The substitute image data may be, for example, image data schematically representing the object 12, image data reflecting the inherent structure of the object 12, or general image data not reflecting the inherent structure of the object 12. Furthermore, the substitute image data may be two-dimensional image data or three-dimensional image data. The substitute image data may be generated in advance for each object 12, and stored in the storage unit 20, the information management database 14, or an external device in association with the identification information of the object 12. Furthermore, the substitute image data may be generated in advance for each type of the object 12, and stored in the storage unit 20, the information management database 14, or an external device in association with the type information of the object 12. Furthermore, when the photographic image data is stored in association with the identification information of the object 12, the substitute image data of the object 12 may be the photographic image data stored in the storage unit 20. Furthermore, when the first image data is stored in the storage unit 20, the second image data may be the first image data stored in the storage unit 20.

[0084] When the recognition unit 26 no longer recognizes the object 12 after recognizing the object 12 , the control unit 30 may display a second image based on second image data regarding the object 12 on the display unit of the UI unit 18 .

[0085] The additional image data is, for example, image data indicating all or part of information included in the object information, image data generated by processing the object information, etc. For example, the control unit 30 generates the additional image data based on the object information.

[0086] The additional image data may be image data indicating identification information included in the object information, image data indicating status information included in the object information, or image data indicating structural information included in the object information. Specifically, image data indicating a message of an error occurring in the object 12 as a device, image data indicating a method for eliminating the error, image data indicating the name of the object 12, image data indicating the price of the object 12, etc. are examples of the additional image data. Of course, the additional image data may be image data indicating information other than this and included in the object information.

[0087] The addition of an image is, for example, the superposition of images. The first image based on the first image data may be, for example, an image in which an additional image based on the additional image data is superimposed on a photographic image based on the photographic image data currently being photographed by the photographing unit 22. When the recognition unit 26 recognizes the object 12, the control unit 30 causes the photographic image based on the photographic image data currently being photographed by the photographing unit 22 to be displayed on the display unit of the UI unit 18, and the additional image is superimposed on the photographic image and displayed on the display unit of the UI unit 18.

[0088] The second image based on the second image data may be, for example, an image in which an additional image is superimposed on an alternative image based on the alternative image data. When the recognition unit 26 does not recognize the object 12, the control unit 30 causes the display unit of the UI unit 18 to display an alternative image based on the alternative image data, and displays the additional image superimposed on the alternative image on the display unit of the UI unit 18.

[0089] Of course, image superposition is only one example of image addition. For example, the additional image may not be superimposed on the photographic image, but displayed side by side with the photographic image. Similarly, the additional image may not be superimposed on the substitute image, but displayed side by side with the substitute image.

[0090] Hereinafter, the processing by the terminal device 10 will be described in more detail. Figure 3 Schematically shows a terminal device 10 and an image forming device 12A as an example of the object 12. The image forming device 12A is an example of a device and is a device having at least one image forming function among a printing function, a scanning function, a copying function, and a facsimile function.

[0091] Figure 4 2 shows an example of the structure of the first image data. The first image data 32 includes, for example, photographed image data 34 and additional image data 36 generated based on object information. The photographed image data 34 is image data currently being photographed by the photographing unit 22. The object information is information acquired by the acquisition unit 28.

[0092] When the user photographs the image forming device 12A using the photographing unit 22 of the terminal device 10, photographed image data 34 representing the image forming device 12A and the surrounding scene (e.g., background, etc.) of the image forming device 12A is generated. An object image 38 representing the image forming device 12A is displayed in the photographed image data 34. In addition, the image representing the surrounding scene is omitted from illustration.

[0093] exist Figure 4 In the example shown, the additional image data 36 is image data representing a character string indicating the state of the image forming device 12A and an arrow indicating the direction pointing to the image forming device 12A. For example, a paper jam error occurs in the image forming device 12A, and the object information includes state information indicating the occurrence of the error. The arrow may point to the location where the error occurred. Furthermore, an animation or a character string for conveying to the user a method of recovering from the error may be included in the additional image data 36. The control unit 30 generates the additional image data 36 based on the object information. The additional image data 36 may be included in the object information. In this case, the control unit 30 does not generate the additional image data 36, ​​but uses the additional image data 36 included in the object information, thereby causing the first image based on the first image data to be displayed on the display unit of the UI unit 18.

[0094] Figure 5 , which is a display example of a photographic image. The control unit 30 causes the display unit of the UI unit 18 to display a photographic image based on the photographic image data 34. The photographic image is displayed on the screen 40 of the display unit. The photographic image is, for example, a dynamic image that the photographing unit 22 is currently photographing. In the field of view of the photographing unit 22, that is, in the range photographed by the photographing unit 22, the image forming device 12A is included, and an object image 38 representing the image forming device 12A is displayed in the photographic image. In this case, since the object information is not acquired by the acquisition unit 28, the additional image based on the additional image data 36 is not displayed.

[0095] When the acquisition unit 28 acquires the object information of the image forming device 12A and the recognition unit 26 recognizes the image forming device 12A, the control unit 30 causes the display unit of the UI unit 18 to display the first image based on the first image data 32 .

[0096] For example, the recognition unit 26 recognizes the image forming device 12A based on the photographic image data currently being photographed by the photographing unit 22, and obtains the identification information of the image forming device 12A. The determination unit 24 determines the image forming device 12A identified by the recognition unit 26 as the object 12 to be obtained as the object information, and outputs the identification information of the image forming device 12A to the acquisition unit 28. The acquisition unit 28 acquires the object information corresponding to the identification information from the information management database 14, the image forming device 12A, an external device or the storage unit 20. The control unit 30 generates additional image data based on the object information obtained by the acquisition unit 28. Then, the control unit 30 displays a first image based on the first image data, which is obtained by adding the additional image data to the photographic image data, on the display unit of the UI unit 18. In addition, the user inputs the identification information of the image forming device 12A, whereby the image forming device 12A can be determined as the object 12 to be obtained as the object information.

[0097] Figure 6 4 shows an example of display of the first image. A photographic image including an object image 38 and an additional image 42 based on additional image data 36 are displayed on a screen 40. The additional image 42 is displayed by being attached to the photographic image. For example, the additional image 42 is displayed overlapping the photographic image. The additional image 42 may be displayed around the object image 38 without overlapping the object image 38, or may be displayed overlapping the object image 38. For example, the first image including the photographic image and the additional image is displayed by AR technology.

[0098] For example, when image forming device 12A is included in the field of view of imaging unit 22 to such an extent that recognition unit 26 can recognize image forming device 12A and image forming device 12A is shown in captured image data 34, recognition unit 26 recognizes image forming device 12A and displays the first image.

[0099] If the image forming device 12A is not included in the viewing angle of the photographing unit 22 to such an extent that the recognition unit 26 can recognize the image forming device 12A, the recognition unit 26 does not recognize the image forming device 12A. For example, only a portion of the image forming device 12A is included in the viewing angle of the photographing unit 22, such as Figure 7 As shown, the captured image displayed on the screen 40 includes only a partial image 44. When the recognition unit 26 does not recognize the image forming device 12A based on only the partial image 44, the control unit 30 causes the display unit of the UI unit 18 to display a second image based on the second image data.

[0100] For example Figure 6 As shown, after the recognition unit 26 recognizes the image forming device 12A, Figure 7As shown, when the recognition unit 26 no longer recognizes the image forming apparatus 12A, the control unit 30 causes the display unit of the UI unit 18 to display the second image instead of the first image.

[0101] The control unit 30 obtains the substitute image data corresponding to the identification information of the image forming device 12A from the storage unit 20, the information management database 14, or an external device. As another example, the control unit 30 may obtain the substitute image data corresponding to the type information of the image forming device 12A. Then, the control unit 30 generates the second image data by adding the additional image data to the substitute image data, and displays the second image based on the second image data on the display unit of the UI unit 18.

[0102] Figure 8 An example of the structure of the second image data is shown in FIG. The second image data 46 includes, for example, substitute image data 48 and additional image data 36. Substitute image data 48 may be, for example, 2D image data or 3D image data schematically representing image forming device 12A, image data generated by photographing image forming device 12A in the past, general image data, or data of a simple graphic (e.g., a rectangle, a circle, an ellipse, etc.). For example, in FIG. Figure 4 When the data of the object image 38 shown is stored in the storage unit 20 , the substitute image data 48 may be the data of the object image 38 .

[0103] Fig. 9 4 shows a display example of the second image. A substitute image 50 based on substitute image data 48 and an additional image 42 based on additional image data 36 are displayed on screen 40. The additional image 42 is displayed by being added to the substitute image 50. For example, the additional image 42 may be displayed around the substitute image 50 without being overlapped with the substitute image 50, or may be displayed by being overlapped with the substitute image 50.

[0104] Furthermore, when the state (for example, error) indicated by the object information is resolved while the first image or the second image is being displayed, the control unit 30 may make the additional image 42 non-displayed.

[0105] Below, reference Fig.10 , the process of processing based on the terminal device 10 is explained. Fig.10 A flowchart showing the processing by the terminal device 10 is shown in FIG.

[0106] First, the acquisition unit 28 acquires the object information of the image forming device 12A (S01). For example, when the user uses the UI unit 18 to specify the image forming device 12A as the object 12 for acquiring the object information, the determination unit 24 determines the image forming device 12A as the object 12 for acquiring the object information. In addition, the determination unit 24 may determine the image forming device 12A identified by the recognition unit 26 as the object 12 for acquiring the object information. The acquisition unit 28 acquires the object information of the image forming device 12A thus determined.

[0107] For example, the information management database 14 is provided as MIB (Management Information Base) in the image forming device 12A, and the acquisition unit 28 acquires the object information from the information management database 14 provided in the image forming device 12A. Specifically, the communication unit 16 communicates with the image forming device 12A via the communication path N. The address information (such as the IP address, etc.) of the image forming device 12A used in the communication may be input by the user, may be acquired from the image forming device 12A through short-range wireless communication, or may be pre-stored in the storage unit 20. The acquisition unit 28 acquires the object information from the information management database 14 through the communication established by the communication unit 16. For example, SNMP (Simple Network Management Protocol) or the like is used in the communication. The object information of the image forming device 12A includes, for example, identification information, type information, status information, and structure information. When a paper jam error occurs in the image forming device 12A, information indicating the paper jam error is included in the status information. When a toner out error occurs in the image forming apparatus 12A, the status information includes information indicating the toner out error. This information is sent from the image forming apparatus 12A and other devices to the MIB and managed by the MIB. The configuration information includes information indicating the number of selectable trays, for example.

[0108] The acquisition unit 28 may acquire the object information from an external device such as a server, may acquire the object information from captured image data, or may acquire the object information from the storage unit 20 .

[0109] Next, the user takes a picture using the photographing unit 22 of the terminal device 10 (S02). Photographic image data is generated by the photographing, and a photographic image based on the photographic image data is displayed on the display unit of the UI unit 18. For example, Figure 5 The photographic images shown in Figure 7 The photographic images shown in , etc.

[0110] Before the acquisition unit 28 acquires the object information, the user may take a picture using the photography unit 22. In this case, when the recognition unit 26 recognizes the image forming device 12A based on the photographed image data, the identification unit 24 may identify the recognized image forming device 12A as the object 12 to be acquired for the object information.

[0111] When the recognition unit 26 recognizes the image forming device 12A from the photographic image data currently being photographed by the photographing unit 22 (S03, Yes), the control unit 30 generates additional image data based on the object information acquired by the acquisition unit 28 (S04). The control unit 30 adds the additional image data to the photographic image data currently being photographed by the photographing unit 22, and displays the first image based on the first image data on the display unit of the UI unit 18 (S05). For example, the display unit 18 displays the first image based on the first image data. Figure 6 For example, an image of an arrow pointing to the location where the paper jam occurred may be displayed, or an animation showing a state where the cover is closed may be displayed.

[0112] When the recognition unit 26 does not recognize the image forming device 12A from the photographic image data currently being photographed by the photographing unit 22 (S03, No), the control unit 30 generates additional image data based on the object information acquired by the acquisition unit 28 (S06). The control unit 30 adds the additional image data to the substitute image data corresponding to the image forming device 12A, and displays the second image based on the second image data on the display unit of the UI unit 18 (S07). For example, the display Fig. 9 Thus, even when the image forming apparatus 12A is not recognized, the user can confirm the object information related to the image forming apparatus 12A.

[0113] When the recognition unit 26 recognizes the image forming apparatus 12A while the second image is being displayed, the control unit 30 may cause the display unit of the UI unit 18 to display the first image instead of the second image.

[0114] Furthermore, after the recognition unit 26 recognizes the image forming apparatus 12A and displays the first image, if the recognition unit 26 no longer recognizes the image forming apparatus 12A, the control unit 30 may display the second image instead of the first image on the display unit of the UI unit 18. Thus, even if the image forming apparatus 12A is not recognized, the user can continue to confirm the object information related to the image forming apparatus 12A.

[0115] The control unit 30 may display the first image or the second image, or may not display the first image and the second image, and may generate information included in the object information as voice from the voice output unit of the UI unit 18. For example, information indicating that an error has occurred and information indicating a method for recovering from the error may be generated as voice from the voice output unit.

[0116] Furthermore, the control unit 30 may perform either voice output or second image display according to the state of the image forming device 12A. For example, when the error generated in the image forming device 12A is simple, the control unit 30 may cause the information included in the object information to be generated as voice from the voice output unit of the UI unit 18. When the error generated in the image forming device 12A is complex, the control unit 30 may cause the second image to be displayed on the display unit of the UI unit 18. When a plurality of jobs are to be executed, the control unit 30 may perform either voice output or second image display for each job according to the job content.

[0117] When the recognition unit 26 does not recognize the image forming apparatus 12A, the control unit 30 may cause an error message or the like corresponding to the state of the image forming apparatus 12A, which is stored in advance in the storage unit 20, to be displayed on the display unit of the UI unit 18. When the error message or the like is displayed, if the recognition unit 26 recognizes the image forming apparatus 12A, the control unit 30 causes the first image to be displayed on the display unit of the UI unit 18 instead of the error message or the like. In addition, the error message or the like may be generated as voice.

[0118] Furthermore, when the image forming device 12A is displayed in the photographic image captured by the photographing unit 22, the control unit 30 causes the display unit of the UI unit 18 to display the first image, and when the image forming device 12A is no longer displayed in the photographic image after the image forming device 12A is displayed in the photographic image, the control unit 30 causes the display unit of the UI unit 18 to display the second image. For example, when the image forming device 12A is displayed in the photographic image to the extent that the recognition unit 26 can recognize the image forming device 12A, the control unit 30 causes the display unit to display the first image, and when the image forming device 12A is not displayed in the photographic image to the extent that the recognition unit 26 can recognize the image forming device 12A, the control unit 30 causes the display unit of the UI unit 18 to display the second image. Thus, even when the image forming device 12A is not displayed in the photographic image, the user can continue to confirm the object information related to the image forming device 12A.

[0119] (Variant 1)

[0120] Below, reference Fig.11 , the processing involved in variant example 1 is explained. Fig.11 A flowchart showing the processing involved in Modification Example 1 is shown in FIG.

[0121] Fig.11 The processing of steps S10 to S14 in Fig.10 Since the processing of steps S01 to S05 is the same as that of step S01, the description of the processing of steps S10 to S14 is omitted, and the processing after step S15 is described.

[0122] The control unit 30 stores the first image data in the storage unit 20 (S15). Furthermore, the control unit 30 stores the object information acquired by the acquisition unit 28 in the storage unit 20 in correspondence with the first image data. The control unit 30 may store the photographic image data and the additional image data in the storage unit 20 separately, or may store the additional image data in the storage unit 20 by adding it to the first image data in the photographic image data. The control unit 30 stores the latest first image data in the storage unit 20, and may delete data other than the latest first image data from the storage unit 20. As another example, the control unit 30 may update the first image data stored in the storage unit 20 to the newly generated first image data at each predetermined period. As yet another example, when there is a change in the object information, the control unit 30 may change the first image data stored in the storage unit 20 to the newly generated first image data.

[0123] When the recognition unit 26 does not recognize the image forming device 12A from the photographic image data currently being photographed by the photographing unit 22 ( S12 , No), the process proceeds to step S16 .

[0124] When object information identical to the object information acquired in step S10 is stored in the storage unit 20 (S16, yes), the control unit 30 reads out the first image data stored in the storage unit 20 in correspondence with the object information, and displays the first image based on the first image data on the display unit of the UI unit 18 as the second image of the image forming device 12A (S17).

[0125] When the first image based on the first image data stored in the storage unit 20 is displayed, in the case where the recognition unit 26 recognizes the image forming device 12A, the control unit 30 causes the first image based on the first image data including the photographic image data currently being photographed by the photographic unit 22 to be displayed on the display unit of the UI unit 18 instead of the first image based on the first image data stored in the storage unit 20.

[0126] Furthermore, the control unit 30 may store the latest first image data in the storage unit 20. In this case, after the image forming apparatus 12A is recognized by the recognition unit 26 and the first image is displayed in step S14, if the image forming apparatus 12A is not recognized by the recognition unit 26, the control unit 30 may display the first image based on the latest first image data stored in the storage unit 20 on the display unit of the UI unit 18.

[0127] If the object information identical to the object information acquired in step S10 is not stored in the storage unit 20, the control unit 30 generates additional image data based on the object information acquired in step S10 (S18). The control unit 30 adds the additional image data to the alternative image data and displays the second image based on the second image data on the display unit of the UI unit 18 (S19).

[0128] For example, if the name, type, and structure indicated by the object information acquired in step S10 and the object information indicating the same name, type, and structure are not stored in the storage unit 20, the image forming device 12A is a different object from the object 12 photographed in the past. Therefore, the control unit 30 prevents the first image based on the first image data stored in the storage unit 20 from being displayed on the display unit of the UI unit 18. Furthermore, even if the name, type, and structure indicated by the object information acquired in step S10 and the object information indicating the same name, type, and structure are stored in the storage unit 20, if the state indicated by the object information acquired in step S10 and the object information indicating the same state are not stored in the storage unit 20, the current state of the image forming device 12A is different from the state when the image forming device 12A was photographed in the past. Therefore, the control unit 30 prevents the first image based on the first image data stored in the storage unit 20 from being displayed on the display unit of the UI unit 18.

[0129] (Variant 2)

[0130] The following describes processing according to Modification 2. In Modification 2, control unit 30 prevents the second image from being displayed on the display unit of UI unit 18 until a predetermined time has passed from the time when the first image is displayed on the display unit of UI unit 18.

[0131] For example, after the recognition unit 26 recognizes the image forming apparatus 12A, if the recognition unit 26 no longer recognizes the image forming apparatus 12A for a predetermined time or longer, the control unit 30 causes the second image to be displayed on the display unit of the UI unit 18. This time may be changed by the user or may be set for each type of the object 12 or each object information.

[0132] In other words, when the state in which the recognition unit 26 does not recognize the image forming device 12A does not continue for a predetermined time period or longer, the control unit 30 prevents the display unit of the UI unit 18 from displaying the second image.

[0133] For example, when the image forming device 12A photographed by the photographing unit 22 is out of the field of view of the photographing unit 22 and the recognition unit 26 no longer recognizes the image forming device 12A, the second image is not displayed.

[0134] (Variant 3)

[0135] The following describes the processing involved in Modification 3. In Modification 3, when the terminal device 10 is stationary and the recognition unit 26 does not recognize the image forming device 12A, the control unit 30 causes the display unit of the UI unit 18 to display the second image. More specifically, after the recognition unit 26 recognizes the image forming device 12A, when the terminal device 10 is stationary and the recognition unit 26 no longer recognizes the image forming device 12A, the control unit 30 causes the display unit of the UI unit 18 to display the second image.

[0136] For example, a detection unit such as a gyro sensor or an acceleration sensor is provided in the terminal device 10, and the control unit 30 detects the movement of the terminal device 10 based on the detection result of the detection unit. In addition, the control unit 30 can detect the movement of the terminal device 10 based on the photographic image data. For example, when the amount of change in the photographic image data becomes greater than a threshold, the control unit 30 determines that the terminal device 10 is moving. When the movement of the terminal device 10 is not detected, the control unit 30 determines that the terminal device 10 is stationary.

[0137] refer to Fig.12 , variant example 3 is described in more detail. Fig.12 The condition for displaying the first image or the second image is shown in . This condition is a combination of the length of time that the recognition unit 26 does not recognize the object 12 and the length of time that the terminal device 10 is stationary.

[0138] Fig.12 The X hours in φ is a threshold value related to the length of time that has passed since the recognition unit 26 stopped recognizing the object 12 . Fig.12 The Y hours in the above is a threshold value related to the length of time that the terminal device 10 is stationary. As an example, the relationship between the length of X hours and the length of Y hours is X hours>Y hours. For example, X hours is 5 seconds and Y hours is 1 second. These values ​​are just examples, and other values ​​can be used or changed by the user.

[0139] For example, after the recognition unit 26 recognizes the image forming device 12A, in a state where the recognition unit 26 no longer recognizes the image forming device 12A, and when the terminal device 10 moves, after X hours have passed since the recognition unit 26 no longer recognizes the image forming device 12A, the control unit 30 causes the display unit of the UI unit 18 to display the second image. Fig.12 The display control is performed when the condition (3) is met. As a situation satisfying the condition (3), for example, it is assumed that the terminal device 10 is stored in the user's clothing pocket.

[0140] After the recognition unit 26 recognizes the image forming device 12A, when the recognition unit 26 no longer recognizes the image forming device 12A and the terminal device 10 is stationary, the control unit 30 displays the second image on the display unit of the UI unit 18 after Y hours have passed since the terminal device 10 was stationary. Fig.12 The display control is performed when the condition (1) or the condition (2) is met. As a situation where the condition (1) or the condition (2) is met, for example, a situation where the terminal device 10 is placed on the floor or the like is assumed.

[0141] Under conditions other than conditions (1) to (3), the control unit 30 causes the first image to be displayed on the display unit of the UI unit 18. For example, when Fig.12 The first image is displayed when the condition (4) is met. As a situation satisfying the condition (4), for example, a situation in which the image forming device 12A is photographed by the photographing unit 22 but the image forming device 12A is not recognized yet can be assumed.

[0142] In addition, the relationship between the length of X hours and the length of Y hours can be changed by the user.

[0143] Furthermore, after the recognition unit 26 recognizes the image forming device 12A, the terminal device 10 is stationary in a state where the recognition unit 26 no longer recognizes the image forming device 12A, and thus after the second image is displayed on the display unit of the UI unit 18, when the terminal device 10 moves, the control unit 30 causes the photographic image based on the photographic image data currently being photographed by the photographing unit 22 to be displayed on the display unit of the UI unit 18. For example, when the above-mentioned condition (1) or condition (2) is satisfied, when the terminal device 10 moves in a state where the second image is displayed on the display unit of the UI unit 18, the photographic image is displayed on the display unit of the UI unit 18.

[0144] When the terminal device 10 is stationary and the second image is displayed on the display unit of the UI unit 18, if the terminal device 10 moves for a predetermined time or longer, the control unit 30 may display the photographic image instead of the second image on the display unit of the UI unit 18. This time may be changed by the user or may be set for each type of the object 12 or each object information.

[0145] In other words, when the terminal device 10 is stationary and the second image is displayed on the display unit of the UI unit 18, even if the terminal device 10 moves, if the terminal device 10 has not moved for more than a predetermined time, the control unit 30 displays the second image on the display unit of the UI unit 18 instead of switching the displayed image from the second image to the photographic image.

[0146] (Variant 4)

[0147] Below, reference Fig.13 and Fig.14 , the processing involved in variant example 4 is explained. Fig.13 and Fig.14 In Modification 4, the control unit 30 causes the display unit of the UI unit 18 to display both the second image and the photographed image.

[0148] Fig.13 The control unit 30 causes the second image to be displayed on the display unit of the UI unit 18. Fig. 9 Similarly to the display example shown, the substitute image 50 and the additional image 42 are displayed on the screen 40. The control unit 30 also causes the photographic image 52 currently being photographed by the photographic unit 22 to be displayed on the screen 40 on which the second image is displayed. The photographic image 52 is displayed at a size that does not interfere with the display of the substitute image 50 and the additional image 42, at a position that does not interfere with the display. For example, at a position that does not overlap with the image of the object in the substitute image 50 and the additional image 42, the photographic image 52 is displayed at a size that does not overlap. Specifically, the photographic image 52 is displayed near the lower corner of the screen 40. The size of the photographic image 52 is, for example, smaller than the size of the substitute image 50. In addition, the display position of the photographic image 52 can be changed by the user.

[0149] Furthermore, when the user designates the photographic image 52 (for example, when the photographic image 52 is clicked or single-clicked), as shown in FIG. Fig.14 As shown, the control unit 30 enlarges the photographic image 52 and displays it on the display unit of the UI unit 18. The photographic image 52 may be displayed superimposed on the second image or may be displayed instead of the second image. The size of the photographic image 52 may be changed by the user.

[0150] When the user designates the enlarged photographic image 52 or issues an instruction to reduce the display, Fig.13As shown, the photographic image 52 is displayed in reduced size. For example, a reduction button image is displayed at a corner of the enlarged photographic image 52, and when the user clicks the reduction button image, the photographic image 52 is displayed in reduced size. Then, the second image is displayed.

[0151] Furthermore, the zoomed display and the reduced display of the photographed image 52 can be switched according to the user's voice instruction.

[0152] The control unit 30 may display the second image and the photographic image 52 side by side on the display unit of the UI unit 18. For example, the screen 40 is divided into a display area for the second image and a display area for the photographic image 52, the second image is displayed in the display area for the second image, and the photographic image 52 is displayed in the display area for the photographic image 52. The size of the display area for the second image and the size of the display area for the photographic image 52 may be the same or different. Furthermore, the size or position of each display area may be changed by the user.

[0153] Furthermore, when the terminal device 10 is stationary, Fig.13 As shown, when the second image and the reduced photographic image 52 are displayed, when the terminal device 10 moves, as shown in FIG. Fig.14 As shown, the control unit 30 may enlarge the photographed image 52 and display it on the display unit of the UI unit 18. The control unit 30 may enlarge the photographed image 52 when the terminal device 10 continues to move for more than a predetermined time.

[0154] (Variant 5)

[0155] Below, reference Fig.15 and Fig.16 , the processing involved in variant example 5 is explained. Fig.15 and Fig.16 In the fifth modification, the control unit 30 causes the display unit of the UI unit 18 to display an image for switching between the display of the photographic image currently being photographed by the photographing unit 22 and the display of the second image. Hereinafter, the image for this switching is referred to as a "switching button image".

[0156] Fig.15 The control unit 30 displays the second image on the display unit of the UI unit 18. Fig. 9Similarly to the display example shown, the replacement image 50 and the additional image 42 are displayed on the screen 40. The control unit 30 also causes the switching button image 54 to be displayed on the screen 40 on which the second image is displayed. For example, the switching button image 54 is displayed at a position that does not interfere with the display of the replacement image 50 and the additional image 42, and at a size that does not interfere with the display. For example, the switching button image 54 is displayed at a position that does not overlap the replacement image 50 and the additional image 42, and at a size that does not overlap. Specifically, the switching button image 54 is displayed near the bottom corner of the screen 40. Furthermore, the display position or size of the switching button image 54 can be changed by the user.

[0157] The switch button image 54 when the second image is displayed indicates that the displayed image is the second image. For example, the switch displayed on the switch button image 54 is displayed at a position indicating the display of the second image. In addition, a character string (for example, a character string "OFF") indicating that the displayed image is not a photographic image but the second image is displayed together with the switch button image 54.

[0158] When the user instructs to switch the display of the second image to the display of the photographed image by operating the switching button image 54 while the second image is being displayed, Fig.16 As shown in FIG. 1 , the control unit 30 causes the image currently being photographed by the photographing unit 22 to be displayed on the display unit of the UI unit 18 instead of the second image. Figure 7 In the same display example shown, Fig.16 An image 44 representing a part of the image forming apparatus 12A is displayed on the screen 40 shown in FIG. 1 . In this state, the recognition unit 26 does not recognize the image forming apparatus 12A, and no additional image is displayed. In addition, a switching button image 54 is displayed.

[0159] When a photographic image is displayed, the switch button image 54 indicates that the displayed image is a photographic image. For example, the switch displayed on the switch button image 54 is displayed at a position indicating the display of the photographic image. In addition, a character string indicating that the displayed image is a photographic image (for example, a character string "ON") is displayed together with the switch button image 54.

[0160] When the user instructs to switch the display of the photographic image to the display of the second image by operating the switching button image 54 while the photographic image is being displayed, Fig.15 As shown, the control unit 30 causes the display unit of the UI unit 18 to display the second image instead of the photographed image.

[0161] like Fig.15 As shown in FIG. 1 , when the second image is displayed, the control unit 30 causes the switching button image 54 to be displayed on the display unit of the UI unit 18. Fig.15As shown in FIG. 1 , after the second image is temporarily displayed, when the displayed image is switched from the second image to the photographic image, as shown in FIG. Fig.16 As shown, when a photographic image is displayed, the control unit 30 also causes the switching button image 54 to be displayed on the display unit of the UI unit 18. Of course, the switching button image 54 may be displayed when the second image is displayed, but not displayed when the photographic image is displayed. Furthermore, the switching button image 54 may be displayed when the first image is displayed, or may be displayed all the time. Furthermore, when the displayed image is switched from the second image to the photographic image, and then, when the image forming device 12A is recognized by the recognition unit 26 and the first image is displayed, the switching button image 54 may not be displayed.

[0162] Furthermore, the display sizes of the second image and the photographic image can be changed by operating the switching button image 54. Fig.13 The switching button image 54 is displayed on the screen 40 shown in FIG. Fig.13 As shown, when the photographic image 52 is displayed as being smaller than the second image, if the user operates the switching button image 54, the control unit 30 makes the photographic image 52 larger than the second image and displays it on the display unit of the UI unit 18. When the photographic image 52 is displayed as being larger than the second image, if the user operates the switching button image 54, the control unit 30 makes the photographic image 52 smaller than the second image and displays it on the display unit of the UI unit 18. In addition, the concept of a large image includes a large area of ​​the displayed image or a large outer edge of the displayed image.

[0163] (Variant 6)

[0164] The following describes the processing according to Modification 6. In Modification 6, the control unit 30 displays recommendation information for urging the user to switch between the first image and the second image on the display unit of the UI unit 18. In addition, a voice expressing the recommendation information may be emitted.

[0165] The control unit 30 may cause the display unit of the UI unit 18 to display first recommendation information for urging the user to switch the displayed image from the second image to the first image while the second image is being displayed.

[0166] For example, when the predetermined first condition is satisfied during display of the second image, the control unit 30 causes the first recommendation information to be displayed on the display unit of the UI unit 18. Specifically, the state of the image forming apparatus 12A is such that information cannot be conveyed to the user unless the additional image is attached to the photographic image that has actually been photographed and displayed, and this is an example of satisfying the first condition. The control unit 30 determines whether the state of the image forming apparatus 12A corresponds to such a state, for example, based on the state information included in the object information.

[0167] For example, since the method of recovering an error that occurs in the image forming device 12A is complicated, it is easier for the user to understand the recovery method when information indicating the recovery method is attached to the photographic image that has actually been taken and displayed, compared with the case where the second image is displayed. In this case, the first recommended information is displayed. In addition, in the case where the substitute image in the second image is an image that schematically represents the image forming device 12A, or a general image that does not reflect the inherent structure of the image forming device 12A, the substitute image sometimes does not accurately or in detail represent the structure of the actual image forming device 12A. If the second image is displayed in this case, it is difficult for the user to understand the recovery method, so the first recommended information is displayed. In addition, in the case where object information is output by voice together with or instead of the display of the second image, it is sometimes difficult for the user to understand the recovery method by voice. In this case, the first recommended information is displayed.

[0168] In addition, a case where information should be transmitted in consideration of the environment around the image forming device 12A is an example of satisfying the first condition. The control unit 30 recognizes the surrounding environment based on the photographic image being photographed by the photographing unit 22, for example, and displays the first recommended information on the display unit of the UI unit 18 based on the recognition result. For example, if there is an obstacle around the image forming device 12A, if the obstacle should also be displayed, the control unit 30 displays the first recommended information on the display unit of the UI unit 18. In this case, the control unit 30 recognizes the obstacle based on the photographic image, and displays the first image including the additional image on the display unit of the UI unit 18 while avoiding the display position of the obstacle.

[0169] Fig.17 4 shows a display example of the first recommended information. The alternative image 50 and the additional image 42 included in the second image are displayed on the screen 40. In addition, an image 56 based on the first recommended information is displayed on the screen 40. For example, the image 56 showing a message "If you take a picture of the device, you can confirm the detailed information." is displayed. This message is a message for urging the user to take a picture so that the image forming device 12A can be recognized by the recognition unit 26.

[0170] For example, when the first condition is not satisfied at the time when the second image is displayed, the control unit 30 does not display the image 56, but displays the second image on the screen 40. When the state represented by the new object information acquired by the acquisition unit 28 after the second image is displayed satisfies the first condition, the control unit 30 displays the image 56 based on the first recommendation information on the screen 40.

[0171] The image 56 based on the first recommended information may be displayed, for example, as a pop-up, or may be displayed at a position and in a size that does not interfere with the display of the alternative image 50 and the additional image 42. Of course, the image 56 may be displayed overlapping the alternative image 50 or the additional image 42. In addition, a voice expressing the first recommended information may be generated.

[0172] Furthermore, the control unit 30 may cause the display unit of the UI unit 18 to display second recommendation information for urging the user to switch the displayed image from the first image to the second image while the first image is being displayed.

[0173] For example, when the predetermined second condition is satisfied during the display of the first image, the control unit 30 causes the second recommendation information to be displayed on the display unit of the UI unit 18. Specifically, a situation where the user needs to release the terminal device 10 due to a phenomenon that the user should work with both hands is an example of satisfying the second condition. For example, the control unit 30 identifies the state of the image forming device 12A based on the state information included in the object information, and causes the second recommendation information to be displayed on the display unit of the UI unit 18 when the state of the image forming device 12A is a state where the user should work with both hands.

[0174] Fig.18 An example of displaying the second recommended information is shown in FIG. Figure 6 Similarly to the display example shown in , the first image is displayed on the screen 40. In addition, an image 58 based on the second recommendation information is displayed on the screen 40. For example, the image 58 showing a message "It is preferable to place the terminal device." is displayed. This message is a message for urging the user to place the terminal device 10 on the floor or table and release both hands.

[0175] For example, when the second condition is not satisfied at the time when the first image is displayed, the control unit 30 does not display the image 58, but displays the first image on the screen 40. When the state represented by the new object information acquired by the acquisition unit 28 after the first image is displayed satisfies the second condition, the control unit 30 displays the image 58 based on the second recommendation information on the screen 40.

[0176] The image 58 based on the recommendation information may be displayed as a pop-up, or may be displayed at a position that does not interfere with the display of the object image 38 and at a size that does not interfere with the display, or may be displayed at a position that does not interfere with the display of the object image 38 and the additional image 42 and at a size that does not interfere with the display. Of course, the image 58 may be displayed superimposed on the object image 38 or the additional image 42. In addition, a voice expressing the second recommendation information may be emitted.

[0177] Furthermore, the first recommended information may be displayed in the case described below. Fig.19A flowchart showing this process is shown in FIG.

[0178] Fig.19 The processing of steps S20 and S21 in Fig.10 Since the processing of steps S01 and S02 is the same as that of steps S01 and S02, the description of the processing of steps S20 and S21 is omitted, and the processing after step S22 is described.

[0179] If the first image needs to be displayed and the recognition unit 26 does not recognize the image forming apparatus 12A (S22, YES), the control unit 30 displays the first recommended information on the display unit of the UI unit 18 (S23). For example, the first recommended information is displayed when the first condition described above is satisfied.

[0180] If the condition of step S22 is not satisfied (S22, No), the process ends. In this case, the photographed image may be displayed or the second image may be displayed. If the image forming device 12A is recognized by the recognition unit 26, the first image is displayed.

[0181] (Variant 7)

[0182] The following is an explanation of the information processing system involved in variant example 7. In variant example 7, a part of the processing performed in the terminal device 10 is performed by an external device such as a server. For example, the processing of each of the determination unit 24, the identification unit 26 and the acquisition unit 28 is performed by an external device. In this case, the determination unit 24, the identification unit 26 and the acquisition unit 28 are not provided in the terminal device 10, and these processing are not performed in the terminal device 10. Of course, a part of the determination unit 24, the identification unit 26 and the acquisition unit 28 can be provided in the terminal device 10, and the part of the processing is performed in the terminal device 10. In this case, the structure that is not provided in the terminal device 10 is provided in the external device, and the processing of the structure is performed in the external device. In addition, even if the determination unit 24, the identification unit 26 and the acquisition unit 28 are all provided in the terminal device 10, these processing can be performed by the external device.

[0183] Below, reference Fig. 20 The information processing system according to Modification 7 will be described in detail.

[0184] Fig. 20 An example of the structure of the information processing system involved in the modification example 7 is shown.

[0185] The information processing system according to the modification example 7 includes a terminal device 10 and a server 60 as an example of an external device. The server 60 can communicate with the terminal device 10, the object 12, and the information management database 14 via a communication path N. A plurality of terminal devices 10 may be included in the information processing system according to the modification example 7, and a plurality of servers 60 may be included in the information processing system according to the modification example 7.

[0186] The server 60 generates the first image data and the second image data, and transmits the first image data and the second image data to the terminal device 10. The photographic image data used to generate the first image data is transmitted from the terminal device 10 to the server 60. When a plurality of terminal devices 10 are included in the information processing system, the server 60 may receive the photographic image data from each terminal device 10. In this case, the server 60 may transmit the first image data generated based on the photographic image data transmitted from a certain terminal device 10 to another terminal device 10.

[0187] Below, reference Fig.21 The structure of the server 60 will be described in detail. Fig.21 An example of the configuration of the server 60 is shown in FIG.

[0188] The communication unit 62 is a communication interface, and has a function of transmitting information to other devices and a function of receiving information from other devices. The communication unit 62 may have a wireless communication function or a wired communication function.

[0189] The storage unit 64 is one or more storage areas for storing information. Each storage area is constituted by, for example, one or more storage devices (for example, physical drives such as hard disk drives and memories) provided in the server 60 .

[0190] The determining unit 66 is configured to determine the object 12 to be acquired as the object information, similarly to the determining unit 24 provided in the terminal device 10. The determining unit 66 may determine the object 12 identified by the identification unit 68 described later as the object 12 to be acquired as the object information, or may determine the object 12 specified by the user using the terminal device 10 as the object 12 to be acquired as the object information.

[0191] The recognition unit 68 receives the photographic image data transmitted from the terminal device 10 to the server 60, and is configured to recognize the object 12 displayed in the photographic image data, similarly to the recognition unit 26 provided in the terminal device 10. The recognition unit 68 recognizes the object 12 using the photographic image data currently being photographed by the photographing unit 22.

[0192] The acquisition unit 70 is configured to acquire the object information of the object 12 specified by the specifying unit 66, similarly to the acquisition unit 28 provided in the terminal device 10. For example, the acquisition unit 70 may acquire the object information from the information management database 14 or from the object 12. Furthermore, when the object information is stored in the storage unit 64, the acquisition unit 70 may acquire the object information from the storage unit 64.

[0193] The image generating unit 72 is configured to generate first image data and second image data. The image generating unit 72 generates the first image data by, for example, adding additional image data based on the object information acquired by the acquiring unit 70 to the photographic image data transmitted from the terminal device 10 to the server 60. Furthermore, the image generating unit 72 generates the second image data by adding the additional image data to the substitute image data. The substitute image data is stored in the storage unit 64, for example.

[0194] The control unit 74 is configured to control the operation of each unit of the server 60. For example, the control unit 74 controls the communication unit 62 so as to transmit the first image data or the second image data generated by the image generating unit 72 to the terminal device 10. Furthermore, the control unit 74 may store the first image data or the second image data in the storage unit 64 in association with the object information, or may store the photographic image data in the storage unit 64.

[0195] When the first image data and the second image data are generated by the server 60, any one of the processes in the above-mentioned embodiment and Modifications 1 to 6 is performed. In this case, information is transmitted and received between the terminal device 10 and the server 60, and the first image or the second image is displayed on the display unit of the terminal device 10.

[0196] Furthermore, first image data is generated based on photographic image data sent from a terminal device 10 used by a certain user to a server 60, and the first image data is sent as second image data to a terminal device 10 used by another user, so that a first image based on the first image data can be displayed on the terminal device 10 used by another user.

[0197] For example, suppose that a user needs to perform a three-step operation in order to eliminate an error that occurred in the image forming device 12A. In this case, in the terminal device 10 used by user A, the first image is displayed in all steps, and the operation of all steps is performed by user A. The first image data displayed in each step is stored in the storage unit 64 of the server 60. Then, in the case where another user B eliminates the same error, the first image data used by user A and stored in the storage unit 64 is sent to the terminal device 10 used by user B, and the first image is displayed on the terminal device 10 used by user B. User B performs the operation of each step while referring to the first image. For example, in the terminal device 10 used by user B, even if the image forming device 12A in which the error occurred is not recognized, the first image based on the first image data used by other user A is displayed on the terminal device 10 used by user B.

[0198] In addition, the processing of the embodiment and each modified example has been described by taking the image forming device 12A as an example of the object 12, but the embodiment and each modified example can be applied to a device other than the image forming device 12A or an object other than the device. For example, when a commodity sold in a store is photographed and recognized, a first image related to the commodity may be displayed, and when the commodity is not recognized, a second image related to the commodity may not be displayed.

[0199] The functions of the various parts of the above-mentioned terminal device 10 and server 60 are implemented as an example through the collaboration of hardware and software. Specifically, the terminal device 10 and the server 60 have one or more processors such as a CPU not shown in the figure. The functions of the various parts of the terminal device 10 and the server 60 are implemented by reading out the program stored in the storage device not shown in the figure through the one or more processors. The above-mentioned program is stored in the storage device via a recording medium such as a CD or DVD, or via a communication path such as a network. As another example, the functions of the various parts of the terminal device 10 and the server 60 can be implemented by hardware resources such as a processor, an electronic circuit or an ASIC (Application Specific Integrated Circuit). Devices such as memory can be used in this implementation. As another example, the functions of the various parts of the terminal device 10 and the server 60 can be implemented by a DSP (Digital Signal Processor) or an FPGA (Field Programmable Gate Array).

[0200] The above-mentioned embodiments of the present invention are provided for the purpose of illustration and description. In addition, the embodiments of the present invention do not fully and exhaustively include the present invention, and do not limit the present invention to the disclosed methods. Obviously, various modifications and changes are self-evident to those skilled in the art to which the present invention belongs. The present embodiment is selected and described in order to most easily understand the principles of the present invention and its application. Thus, other technicians in the art can understand the present invention through various variations optimized for specific uses assumed to be various embodiments. The scope of the present invention is defined by the above claims and their equivalents.

Claims

1. An information processing device, comprising: Photography Unit; a recognition unit that recognizes that an object is displayed in the photographic image being photographed by the photographing unit; and A control unit determines whether the object in the photographic image is recognized, and when the recognition unit recognizes the object in the photographic image, causes a first image in the photographic image to be displayed on a display unit, in which an image based on object information related to the object is attached. When the recognition unit does not recognize the object in the photographic image, causes a second image obtained by attaching the image based on the object information to an image different from the photographic image and representing the object to be displayed on the display unit.

2. The information processing device according to claim 1, characterized in that The control unit displays the first image on the display unit when the recognition unit recognizes the object, and displays the second image on the display unit when the recognition unit no longer recognizes the object after the recognition unit recognizes the object.

3. The information processing device according to claim 1 or 2, characterized in that: The information processing device further comprises a storage unit for storing the first image. The second image is the first image stored in the storage unit.

4. The information processing device according to claim 1 or 2, characterized in that: The control unit prevents the second image from being displayed on the display unit until a predetermined time has passed from the time when the first image is displayed on the display unit.

5. The information processing device according to claim 2, characterized in that: After the recognition unit recognizes the object, if the recognition unit does not recognize the object for a period of time longer than a predetermined period, the control unit causes the display unit to display the second image.

6. The information processing device according to claim 1 or 2, characterized in that: When the information processing device is stationary in a state where the recognition unit does not recognize the object, the control unit causes the display unit to display the second image.

7. The information processing device according to claim 6, characterized in that: When the information processing device moves, the control unit causes the display unit to display the photographic image.

8. The information processing device according to claim 1 or 2, characterized in that: The control unit causes the display unit to display the second image together with the photographic image.

9. The information processing device according to claim 1 or 2, characterized in that: The control unit further causes the display unit to display an image for switching between display of the photographic image and display of the second image.

10. The information processing device according to claim 1 or 2, characterized in that: When the second image is displayed on the display unit, the control unit further causes the display unit to display information for urging a user to switch the second image to the first image.

11. An information processing system comprising an information processing device and an external device, The information processing device comprises: Photography Unit; and a control unit to display the image on the display unit, The external device has: a recognition unit that recognizes that an object is displayed in the photographic image being photographed by the photographing unit; and a generating unit that generates a first image by adding an image based on object information related to the object to the photographic image, and generates a second image by adding the image based on the object information to an image that is different from the photographic image and represents the object, The control unit determines whether the object in the photographic image is recognized. When the recognition unit recognizes the object in the photographic image, the control unit displays the first image on the display unit. When the recognition unit does not recognize the object in the photographic image, the control unit displays the second image on the display unit.

12. A recording medium having recorded thereon a program for causing a computer to function as: a recognition unit that recognizes that an object is displayed in a photographic image being photographed by the photographing unit; and A control unit determines whether the object in the photographic image is recognized, and when the recognition unit recognizes the object in the photographic image, causes a first image in the photographic image to be displayed on a display unit, in which an image based on object information related to the object is attached. When the recognition unit does not recognize the object in the photographic image, causes a second image obtained by attaching the image based on the object information to an image different from the photographic image and representing the object to be displayed on the display unit.

13. An information processing method comprising the following steps: a recognition step of recognizing that an object is displayed in a photographic image being photographed by a photographing unit; and A control step for determining whether the object in the photographic image is recognized. When the object in the photographic image is recognized in the recognition step, a first image in which an image based on object information related to the object is attached to the photographic image is displayed on a display unit. When the object in the photographic image is not recognized in the recognition step, a second image obtained by attaching the image based on the object information to an image different from the photographic image and representing the object is displayed on the display unit.

Images