Information processing device, information processing method, and recording medium

Abstract

The present invention discloses an information processing device, an information processing method and a recording medium, wherein the information processing device includes a processor, wherein, when a second device different from a first device is photographed after a first device is photographed, the processor controls the display of a collaborative function that can be executed by combining a first function of the first device and a second function of the second device, the first function being different from the second function.

Description

[0001] This application is a divisional application of the Chinese invention patent application with application number 201710307933.9, application date May 4, 2017, and invention name “Information processing device and information processing method”. Technical Field

[0002] The present invention relates to an information processing device, an information processing method and a recording medium. Background Art

[0003] Japanese Patent No. 5737906 discloses a technology for displaying an image showing an operation guide on an operation panel that is detachable from a device body and can communicate wirelessly with the device body.

[0004] Japanese Patent Laying-Open No. 2014-10769 discloses a relay device that relays data between a client device and a service providing device that provides a service.

[0005] Japanese Patent Laying-Open No. 2014-48672 discloses a portable terminal that generates an augmented reality image in which additional information including status information of an image forming device is superimposed on an image captured by a capturing unit.

[0006] Japanese Patent Laying-Open No. 2014-238786 discloses a technology for displaying the usage status of each cloud server by a user on a display of a device operated by the user when logging in, in a communication system that communicates with a plurality of cloud servers.

[0007] However, regarding the utilization environment of an apparatus such as an image forming apparatus, a utilization environment in which multiple users utilize one apparatus is generally assumed. In contrast, in the following, a utilization environment in which multiple users utilize multiple apparatuses is assumed. In addition, a user interface such as a touch panel is removed from the apparatus, or a scene in which the apparatus is temporarily utilized at a user's destination is assumed. In such a utilization environment, the user may not be able to simply understand which function can be executed by the apparatus. Summary of the invention

[0008] The present invention aims to provide information about functions that can be performed by a device.

[0009] According to a first aspect of the present invention, there is provided an information processing device comprising a processor, wherein, in a case where a second device different from a first device is photographed after a first device is photographed, the processor controls the display of a collaborative function that can be executed by combining a first function of the first device and a second function of the second device, the first function being different from the second function.

[0010] According to a second aspect of the present invention, there is provided an information processing device according to the first aspect, wherein the processor controls display of the collaborative function utilizing the first function and the second function in response to capturing the second device after the image of the first device disappears from a display.

[0011] According to a third aspect of the present invention, there is provided an information processing device according to the first aspect, wherein the first device includes a plurality of parts having different functions, and the processor performs the following processing: when a first part of the first device is photographed when photographing the first device, a function of the first part is determined as the first function.

[0012] According to a fourth aspect of the present invention, an information processing device according to the third aspect is provided, wherein the second device includes a plurality of parts having different functions, and the processor performs the following processing: when a second part of the second device is photographed when photographing the second device, the function of the second part is determined as the second function.

[0013] According to a fifth aspect of the present invention, an information processing device according to the fourth aspect is provided, wherein, based on the display positions of a partial image within a first image related to the first function and a partial image within a second image related to the second function, the display of a collaborative function that can be executed by combining the first function related to the partial image within the first image and the second function related to the partial image within the second image is controlled.

[0014] According to a sixth aspect of the present invention, there is provided an information processing device according to the first aspect, wherein the first device and the second device include a plurality of parts having different functions, and each partial image related to the plurality of parts is associated with a different function.

[0015] According to a seventh aspect of the present invention, there is provided an information processing device according to the first aspect, wherein, when the first device and the second device are photographed, the device used as the starting point of the collaborative function is changed according to the positional relationship with the photographing device performing the photographing.

[0016] According to an eighth aspect of the present invention, there is provided an information processing device according to the first aspect, wherein, when the first device and the second device are photographed, a first image related to the first device and a second image related to the second device are displayed in a display unit, and the processor controls the display of the collaborative function by changing the starting point of the device utilized in the collaborative function and the next collaborative target according to the display positions of the first image and the second image in the display unit.

[0017] According to a ninth aspect of the present invention, there is provided an information processing device according to the first aspect, wherein, when the first device and the second device are photographed, a first image related to the first device and a second image related to the second device are displayed in a display unit, and the processor switches the display of the collaborative function according to the display positions of the first image and the second image in the display unit.

[0018] According to a tenth aspect of the present invention, there is provided the information processing apparatus according to any one of the first to fifth aspects, wherein the cooperative function is a function that cannot be utilized before the cooperation.

[0019] According to the eleventh aspect of the present invention, a recording medium is provided, which records an information processing program, which enables a computer to function as a control unit. When a second device different from the first device is photographed after a first device is photographed, the control unit controls the display of a collaborative function that can be executed by combining a first function of the first device and a second function of the second device, the first function being different from the second function.

[0020] According to the twelfth aspect of the present invention, there is provided a recording medium recording an information processing program, wherein the information processing program enables a computer to function as a control unit, wherein the control unit, when a second device different from a first device is captured after a first device is captured, controls the display of a collaborative function that can be executed by combining a first function of the first device and a second function of the second device, the first function being different from the second function.

[0021] According to the thirteenth aspect of the present invention, there is provided an information processing device comprising a processor, wherein the processor performs the following processing: according to the display positions of a partial image within a first image related to a first device and a partial image within a second image related to a second device, switching the display of a collaborative function that can be executed by combining a function related to the partial image within the first image and a function related to the partial image within the second image.

[0022] According to a fourteenth aspect of the present invention, there is provided an information processing device according to the thirteenth aspect, wherein the first image is an image obtained by photographing the first device; and the second image is an image obtained by photographing the second device.

[0023] According to the fifteenth aspect of the present invention, there is provided an information processing device according to the fourteenth aspect, wherein the processor performs the following processing: identifies the first device based on the first image, identifies the second device based on the second image, and when the first device and the second device are identified through the identification, switches the display of the collaborative function according to the display position.

[0024] According to the sixteenth aspect of the present invention, there is provided an information processing device comprising a processor, wherein the processor controls the display of a collaborative function that can be executed by combining a function related to a partial image within a first image related to a first function and a function related to a partial image within a second image related to a second function, based on the display positions of a partial image within a first image related to a first function and a partial image within a second image related to a second function.

[0025] According to a seventeenth aspect of the present invention, there is provided the information processing apparatus according to any one of the thirteenth to sixteenth aspects, wherein each partial image is associated with a different function.

[0026] According to an eighteenth aspect of the present invention, there is provided the information processing apparatus according to any one of the thirteenth to sixteenth aspects, wherein the cooperative function is a function that cannot be used before the cooperation.

[0027] According to the nineteenth aspect of the present invention, there is provided a recording medium having a program recorded thereon, the program causing a computer to function as a control unit, the control unit switching the display of a collaborative function executable by combining a function related to a partial image within a first image related to a first device and a function related to a partial image within a second image related to a second device, according to the display positions of a partial image within a first image related to a first device and a partial image within a second image related to a second device.

[0028] According to the twentieth aspect of the present invention, there is provided a recording medium recording a program, which enables a computer to function as a control unit, and the control unit controls the display of a collaborative function that can be executed by combining a function related to a partial image within a first image related to a first function and a function related to a partial image within a second image related to a second function, based on the display positions of a partial image within a first image related to a first function and a partial image within a second image related to a second function.

[0029] According to the twenty-first aspect of the present invention, an information processing method is provided, wherein, based on the display positions of a partial image within a first image related to a first device and a partial image within a second image related to a second device, the display of a collaborative function that can be executed by combining a function related to the partial image within the first image and a function related to the partial image within the second image is switched.

[0030] According to the twenty-second aspect of the present invention, an information processing method is provided, wherein, based on the display positions of a partial image within a first image related to a first function and a partial image within a second image related to a second function, the display of a collaborative function that can be executed by combining a function related to the partial image within the first image and a function related to the partial image within the second image is controlled.

[0031] According to the first to twenty-second aspects, there is provided information of functions executable by a device. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a block diagram showing an image forming system according to an embodiment of the present invention;

[0033] Figure 2 is a block diagram showing an image forming apparatus according to the present embodiment;

[0034] Figure 3 is a block diagram showing a server according to the present embodiment;

[0035] Figure 4 is a block diagram showing a terminal device according to the present embodiment;

[0036] Figure 5 is a schematic diagram showing the appearance of an image forming apparatus;

[0037] Figure 6 is a diagram showing an example of a device function management table;

[0038] Figure 7 is a diagram showing an example of a collaborative function management table;

[0039] Figure 8 is a diagram showing a device that is utilized alone;

[0040] Fig. 9 is a diagram showing an example of a function display screen;

[0041] Fig.10 is a diagram showing an example of a function display screen;

[0042] Fig.11 is a diagram representing an apparatus for coordinating objectives;

[0043] Fig.12 is a diagram showing an example of a function display screen;

[0044] Fig.13 is a diagram showing an example of a display screen of the device;

[0045] Fig.14 is a diagram showing an example of a display screen of the device;

[0046] Fig.15 is a diagram showing an example of a display screen of a device according to Modification Example 1;

[0047] Fig.16 is a diagram showing an example of a function display screen of Modification Example 1;

[0048] Fig.17 is a diagram showing an example of a display screen of a device according to Modification Example 1;

[0049] Fig.18 is a diagram showing an example of a function display screen of Modification Example 1;

[0050] Fig.19 is a diagram showing an example of a display screen of a device according to Modification Example 1;

[0051] Fig. 20 is a diagram showing an example of a display screen of a device according to Modification Example 2;

[0052] Fig.21 is a diagram showing an example of a display screen of a device according to Modification Example 2;

[0053] Fig. 22 is a diagram showing an example of a display screen of a device according to Modification Example 3;

[0054] Fig.23 is a diagram showing an example of a display screen of a device according to Modification Example 4;

[0055] Fig.24 is a diagram showing an example of a display screen of a device according to Modification Example 4;

[0056] Fig.25 is a diagram showing an example of a device function management table according to Modification 5;

[0057] Fig.26 is a diagram showing an example of a cooperative function management table according to Modification 5;

[0058] Fig. 27 is a diagram showing an example of a display screen of a device according to Modification Example 5;

[0059] Fig.28 is a diagram showing an example of a display screen of a device according to Modification Example 5;

[0060] Fig.29 is a diagram showing an example of a display screen of a device according to Modification Example 6;

[0061] Fig.30 is a diagram showing an example of a display screen of a device according to Modification Example 6;

[0062] Fig.31 is a diagram showing an example of a display screen of a device according to Modification Example 7;

[0063] Fig.32 is a diagram showing an example of a display screen of a device according to Modification Example 7; and

[0064] Fig.33 This is a sequence diagram showing connection processing in the image forming system. DETAILED DESCRIPTION

[0065] Reference Figure 1 , an image forming system as an information processing system according to an embodiment of the present invention is described. Figure 1 An example of the image forming system according to the present embodiment is shown.

[0066] The image forming system of this embodiment includes a plurality of devices (e.g., devices 10 and 12), a server 14 as an example of an external device, and a terminal device 16 as an example of an information processing device. The devices 10 and 12, the server 14, and the terminal device 16 have a function of communicating with each other via a communication path N such as a network. Of course, the devices 10 and 12, the server 14, and the terminal device 16 may communicate with other devices via different communication paths. Figure 1 In the example shown, the image forming system includes two devices (devices 10 and 12 ), but the image forming system may include three or more devices. In addition, the image forming system may include a plurality of servers 14 or a plurality of terminal devices 16 .

[0067] Devices 10 and 12 are devices with specific functions, for example, image forming devices with image forming functions, personal computers (PCs), projectors, display devices such as liquid crystal displays or projectors, telephones, clocks, surveillance cameras, and the like. In addition, devices 10 and 12 have the function of sending and receiving data between them and other devices. In this embodiment, device 10 is an image forming device as an example. The image forming device (device 10) is, for example, a device having at least one of a scanning function, a printing function, a copying function, and a fax function.

[0068] The server 14 is a device that manages the functions of each device. For example, the server 14 manages the coordinated functions using the functions of each device and the functions of multiple devices. In addition, the server 14 has the function of sending and receiving data between it and other devices.

[0069] The server 14 may also manage functions available to each user. Functions available to users include, for example, functions provided to users free of charge, functions provided to users for a fee and purchased by users, etc. The server 14 may also manage available function information (for example, function purchase history information) indicating functions available to users for each user. Of course, since there are functions available for free, additional update functions, functions specially managed by administrators, etc., the server 14 may not uniformly manage functions based on whether they are purchased or not. The purchase process of functions is performed, for example, by the server 14. Of course, the purchase process may also be performed by other devices.

[0070] The terminal device 16 is a device such as a personal computer (PC), a tablet PC, a smartphone, or a mobile phone, and has a function of transmitting and receiving data between itself and other devices. The terminal device 16 functions as a user interface unit (UI unit) when the device is used, for example.

[0071] In the image forming system of this embodiment, for example, when one device is recognized, the display of the functions of the device is controlled, and when multiple devices are recognized, the display of the coordinated functions using the functions of the multiple devices is controlled. For example, when one device is photographed, the display of the functions of the device is controlled, and when multiple devices are photographed, the display of the coordinated functions using the functions of the multiple devices is controlled.

[0072] Hereinafter, each device included in the image forming system of this embodiment will be described in detail.

[0073] Reference Figure 2 The structure of the device 10 as an image forming device will be described in detail. Hereinafter, the device 10 will be referred to as the image forming device 10. Figure 2 , the structure of the image forming apparatus 10 is shown.

[0074] The communication unit 18 is a communication interface having a function of transmitting data to other devices and a function of receiving data from other devices. The communication unit 18 may be a communication interface having a wireless communication function or a communication interface having a wired communication function.

[0075] The image forming unit 20 has an image forming function. Specifically, the image forming unit 20 has at least one of a scanning function, a printing function, a copying function, and a fax function. By executing the scanning function, the original is read to generate scan data (image data). By executing the printing function, the image is printed on a recording medium such as paper. By executing the copying function, the original is read and printed on a recording medium. By executing the fax function, image data is sent or received by fax. In addition, a collaborative function that enables multiple functions to work together can also be executed. For example, a scanning and transmitting function that combines a scanning function and a transmitting function (transmitting function) can also be executed. By executing the scanning and transmitting function, the original is read to generate scan data (image data), and the scan data is sent to a receiving address (for example, an external device such as a terminal device 16). Of course, this collaborative function is just an example, and other collaborative functions can also be executed.

[0076] The storage unit 22 is a storage device such as a hard disk or a memory (such as an SSD). The storage unit 22 stores, for example, information indicating a command for image formation (such as job information), image data to be printed, scan data generated by executing a scanning function, device address information indicating an address of another device, server address information indicating an address of the server 14, various control data, various programs, etc. Of course, these may be stored in different storage devices or in one storage device.

[0077] The UI unit 24 is a user interface unit, including a display unit and an operating unit. The display unit is, for example, a display device such as a liquid crystal display. The operating unit is, for example, an input device such as a touch panel or a keyboard. Of course, it may also be a user interface that serves as both a display unit and an operating unit (for example, a device including a touch-type display, or an electronically displayed keyboard on a display, etc.). In addition, the image forming device 10 may not have the UI unit 24, or may not have a display unit but may have a hardware user interface unit (hardware UI unit) as hardware. The hardware UI unit is, for example, a hard key specialized for digital input (for example, a numeric keypad), a hard key specialized for directional indication (for example, a directional indication key), etc.

[0078] The control unit 26 controls the operation of each unit of the image forming apparatus 10 .

[0079] Below, refer to Figure 3 The structure of the server 14 will be described in detail. Figure 3 , the structure of the server 14 is shown.

[0080] The communication unit 28 is a communication interface having a function of transmitting data to other devices and a function of receiving data from other devices. The communication unit 28 may be a communication interface having a wireless communication function or a communication interface having a wired communication function.

[0081] The storage unit 30 is a storage device such as a hard disk or a memory (such as an SSD, etc.). The storage unit 30 stores, for example, device function information 32, collaborative function information 34, various data, various programs, device address information indicating the address of each device, server address information indicating the address of the server 14, etc. Of course, they can be stored in different storage devices or in one storage device. The device function information 32 and collaborative function information 34 stored in the storage unit 30 are provided to the terminal device 16 periodically or at a specified time, thereby also updating the information stored in the terminal device 16. The device function information 32 and collaborative function information 34 are described below.

[0082] The device function information 32 is information indicating the functions of each device (e.g., the image forming device 10 or the device 12) included in the image forming system, and is information indicating the association between device identification information for identifying a device and function identification information for identifying the functions of the device. The device identification information is, for example, information indicating the device ID, device name, type of device, model, location information, an appearance image indicating the appearance of the device, and the like. The function identification information is, for example, a function ID or a function name, and the like. For example, in the case where the image forming device 10 has a scanning function, a printing function, a copying function, and a scanning and transmitting function, in the device identification information of the image forming device 10, function identification information indicating the scanning function, function identification information indicating the printing function, function identification information indicating the copying function, and function identification information indicating the scanning and transmitting function are associated. By referring to the device function information 32, the functions of each device are specified (identified).

[0083] The collaborative function information 34 is information indicating a collaborative function that is executed by making a plurality of functions collaborate. One or more collaborative functions are executed by making a plurality of functions collaborate. The collaborative function can be executed by making a plurality of functions of one device (e.g., the image forming device 10 or the device 12) collaborate, or by making a plurality of functions of a plurality of devices (e.g., the image forming device 10 and the device 12) collaborate. In addition, the terminal device (the terminal device 16 in this embodiment) that issues the operation instruction is also included in the device for identifying the target, and the function of the terminal device can be used as part of the collaborative function.

[0084] The cooperative function information 34 is information indicating, for example, a combination of function identification information for identifying each function used in a cooperative function and an association between cooperative function identification information for identifying the cooperative function. The cooperative function identification information is, for example, a cooperative function ID or a cooperative function name.

[0085] When a plurality of devices are coordinated, the coordinated function information 34 is information indicating a coordinated function utilizing a plurality of functions possessed by the plurality of devices, and is information indicating a combination of device identification information for identifying each device utilized by the coordinated function and association of the coordinated function identification information.

[0086] A collaborative function may be a function that is executed by making a plurality of different functions collaborate, or a function that is executed by making the same function collaborate. A collaborative function may also be a function that cannot be used before collaboration. The function that cannot be used before collaboration may be a function that can be used by using the same function among the functions of the collaborative target device, or a function that can be used by combining different functions. For example, a copy function as a collaborative function is realized by making a device (printer) with a printing function and a device (scanner) with a scanning function collaborate. That is, a copy function is realized by making a printing function and a scanning function collaborate. In this case, the copy function as a collaborative function and a combination of a printing function and a scanning function are associated. In the collaborative function information 34, for example, a combination of collaborative function identification information for identifying a copy function as a collaborative function and device identification information for identifying a device with a printing function and device identification information for identifying a device with a scanning function is associated.

[0087] In addition, available function information may also be stored in the storage unit 30. Available function information is information indicating functions available to each user, for example, information indicating the association between user identification information for identifying a user and function identification information (including collaborative function identification information) indicating functions available to the user. The functions available to the user may be, for example, functions provided free of charge to the user, or functions purchased by the user, as described above, or functions possessed by a single device, or collaborative functions. User identification information is, for example, user account information such as a user ID or name. By referring to the available function information, the functions available to each user are identified. The available function information is updated, for example, each time a function is provided to a user (for example, each time a function is provided to a user free of charge or for a fee).

[0088] The control unit 36 ​​controls the operation of each unit of the server 14. The control unit 36 ​​also includes a specifying unit 38.

[0089] The specifying unit 38 receives the device identification information for identifying the device, and specifies the function identification information of the function associated with the device identification information in the device function information 32 stored in the storage unit 30. Thus, the function of the device is specified (identified). For example, the device identification information is sent from the terminal device 16 to the server 14, and the function identification information of the function associated with the device identification information is specified by the specifying unit 38. Information about the function (such as the function identification information or the description information of the function, etc.) is sent from the server 14 to the terminal device 16 and displayed on the terminal device 16. Thus, information about the function of the device specified by the device identification information is displayed on the terminal device 16.

[0090] In addition, the specific unit 38 receives device identification information for identifying each device of the collaborative target, and specifies the collaborative function identification information of the collaborative function associated with the combination of each device identification information in the collaborative function information 34 stored in the storage unit 30. Thus, the collaborative function performed by coordinating the functions possessed by each device of the collaborative target is specified (identified). For example, a plurality of device identification information is sent from the terminal device 16 to the server 14, and the collaborative function identification information of the collaborative function associated with the plurality of device identification information is specified by the specific unit 38. Information about the collaborative function (such as the collaborative function identification information or the description information of the collaborative function, etc.) is, for example, sent from the server 14 to the terminal device 16 and displayed on the terminal device 16. Thus, information about the collaborative function performed by the plurality of devices specified by the plurality of device identification information is displayed on the terminal device 16.

[0091] For example, when one device is identified (for example, when one device is photographed), the specific unit 38 receives device identification information for identifying the device, and specifies function identification information of the function associated with the device identification information in the device function information 32. Thus, when one device is identified (for example, when one device is photographed), the function of the device is specified (identified). When multiple devices are identified (for example, when multiple devices are photographed), the specific unit 38 receives device identification information for identifying each device included in the multiple devices, and specifies cooperative function identification information of the cooperative function associated with the combination of each device identification information in the cooperative function information 34. Thus, when multiple devices are identified (for example, when multiple devices are photographed), the cooperative function that utilizes the functions of the multiple devices is specified (identified). In this way, the specific unit 38 switches the specific processing of the function according to the number of devices identified (for example, according to the number of devices photographed). In the terminal device 16, information about the function of a single device or information about the cooperative function that utilizes the functions of multiple devices is displayed according to the number of devices identified (for example, photographed). Of course, even when a plurality of devices are captured, the specifying unit 38 may receive the device identification information of each of the plurality of devices and specify (identify) the function of each device by referring to the device function information 32 .

[0092] Furthermore, when functions available to the user are managed, the specifying unit 38 may receive user identification information for identifying the user, and specify function identification information of each function associated with the user identification information in the available function information stored in the storage unit 30. Thus, the function group available to the user is specified (identified). For example, the user identification information is sent from the terminal device 16 to the server 14, and the function identification information of each function associated with the user identification information is specified by the specifying unit 38. Information about each function available to the user (for example, information indicating the name of each function) is sent from the server 14 to the terminal device 16 and displayed on the terminal device 16. Thus, information about each function available to the user specified by the user identification information is displayed on the terminal device 16. For example, the specifying unit 38 receives device identification information and user identification information, specifies function identification information of a function associated with the device identification information in the device function information 32, and specifies function identification information of a function associated with the user identification information in the available function information. Thus, the function possessed by the device specified by the device identification information, that is, the function available to the user specified by the user identification information, is specified.

[0093] The control unit 36 ​​may also execute a purchase process of a function and manage the purchase history. For example, if a user purchases a chargeable function, the control unit 36 ​​may apply a charge process to the user.

[0094] In addition, the control unit 36 ​​may also perform functions related to image processing, such as a character recognition function, a translation function, an image processing function, and an image forming function. Of course, the control unit 36 ​​may also perform functions related to processing other than image processing. By performing the character recognition function, characters in an image are recognized and character data representing the characters are generated. By performing the translation function, characters in an image are translated into characters represented in a specific language, and character data representing the translated characters are generated. By performing the image processing function, an image is processed. For example, the control unit 36 ​​receives scan data generated by performing a scanning function from the image forming device 10, and performs functions related to image processing, such as a character recognition function, a translation function, and an image processing function, with the scan data as a target. The control unit 36 ​​may also receive image data from the terminal device 16, and perform various functions with the image data as a target. The character data and image data generated by the control unit 36 ​​are sent to the terminal device 16, for example, from the server 14. The server 14 is used as an external device, and the collaborative function may also be a function that utilizes functions possessed by multiple devices including the server 14.

[0095] Below, refer to Figure 4 The structure of the terminal device 16 will be described in detail. Figure 4 , the structure of the terminal device 16 is shown.

[0096] The communication unit 40 is a communication interface having a function of transmitting data to other devices and a function of receiving data from other devices. The communication unit 40 may be a communication interface having a wireless communication function or a communication interface having a wired communication function.

[0097] The camera 42 as a photographing unit generates image data (e.g., still image data or moving image data) by photographing a photographing target. In addition, the communication unit 40 may receive image data photographed not only by the camera of the terminal device 16 but also by an external camera connected to a communication path such as a network, and display the image data on the UI unit 46 so that the user can operate the image data.

[0098] The storage unit 44 is a storage device such as a hard disk or a memory (for example, an SSD, etc.), which stores various programs, various data, address information of the server 14, address information of each device (for example, address information of the image forming device 10 or the device 12), information about the identified device, information about the device of the identified collaborative target, information about the functions of the identified device, information about the collaborative function, etc.

[0099] The UI unit 46 is a user interface unit, including a display unit and an operation unit. The display unit is, for example, a display device such as a liquid crystal display. The operation unit is, for example, an input device such as a touch panel, a keyboard, a mouse, etc. Of course, it can also be a user interface that serves as both a display unit and an operation unit (for example, a device including a touch display, or a device that electronically displays a keyboard on a display, etc.).

[0100] The control unit 48 controls the operation of each unit of the terminal device 16. The control unit 48 functions as, for example, a display control unit (control unit), and displays various information on the display unit of the UI unit 46.

[0101] The display section of the UI section 46 displays, for example, an image captured by the camera 42, an image related to a device identified as a target device (e.g., a device to be used alone or a device to be used in cooperation), and the like. The image related to the device may be an image (still image or moving image) representing the device captured by the camera 42, or an image (e.g., an icon) schematically representing the device. The data of the schematically represented image may be stored in the server 14 and provided to the terminal device 16 from the server 14, may be pre-stored in the terminal device 16, or may be stored in another device and provided to the terminal device 16 from the other device.

[0102] The device function information 32 may be stored in the storage unit 44 of the terminal device 16. In this case, the device function information 32 may not be stored in the storage unit 30 of the server 14. Similarly, the collaborative function information 34 may be stored in the storage unit 44 of the terminal device 16. In this case, the collaborative function information 34 may not be stored in the storage unit 30 of the server 14. The control unit 48 of the terminal device 16 may also have the above-mentioned specific unit 38, identify the device based on the device identification information, identify the function of the device, and may also specify a collaborative function that utilizes multiple functions. In this case, the server 14 may not have the specific unit 38.

[0103] When available function information is created, the available function information may be stored in the storage unit 44 of the terminal device 16. In this case, the available function information may not be stored in the storage unit 30 of the server 14. The control unit 48 of the terminal device 16 may also manage the history of user purchase of functions. In this case, the control unit 36 ​​of the server 14 may not have the management function. The control unit 48 of the terminal device 16 may also specify the function available to the user based on the user identification information.

[0104] Furthermore, as another example, the device function information 32 or the cooperative function information 34 may be stored in the image forming device 10 or the device 12, and the image forming device 10 or the device 12 may include the specific unit 38. That is, the processing performed by the specific unit 38 of the server 14 may be performed in the server 14, in the terminal device 16, or in the image forming device 10 or the device 12.

[0105] In this embodiment, as an example, by applying AR (Augmented Reality) technology (augmented reality technology), device identification information is obtained and the device using the target is identified. For example, by applying AR technology, the device identification information of a device used alone is obtained and the device is identified. In addition, the device identification information of the device of the collaborative target is obtained and the device of the collaborative target is identified. As AR technology, well-known AR technology is used. For example, a marking type AR technology using a mark such as a two-dimensional barcode, a marking-free AR technology using image recognition technology, and a location information AR technology using location information are used. Of course, it is also possible to obtain device identification information and identify the device using the target without applying AR technology. For example, if it is a device connected to the network, the device can be identified based on the IP address, or the device ID can be read out to identify the device. Furthermore, if it is a device or terminal device with various wireless communication functions such as infrared communication, visible light communication, Wi-Fi (Wireless Fidelity) (registered trademark), and Bluetooth (Bluetooth (registered trademark)), it is also possible to obtain the device ID of the collaborative device by using these wireless communication functions to identify the device and perform the collaborative function.

[0106] Below, refer to Figure 5 The processing for acquiring the device identification information will be described in detail. As an example, a case where the device identification information of the image forming apparatus 10 is acquired will be described. Figure 5The appearance of the image forming device 10 is schematically shown. Here, the process for obtaining device identification information by applying the marker-type AR technology is described. A marker 50 such as a two-dimensional barcode is provided on the frame of the image forming device 10. The marker 50 is information that encodes the device identification information of the image forming device 10. The user activates the camera 42 of the terminal device 16 and uses the camera 42 to photograph the marker 50 provided on the image forming device 10 of the utilization target. Thus, image data representing the marker 50 is generated. The image data is transmitted from the terminal device 16 to the server 14, for example. In the server 14, the control unit 36 ​​extracts the device identification information by applying a decoding process to the marker image represented by the image data. Thus, the image forming device 10 of the utilization target (the image forming device 10 having the photographed marker 50) is identified. The specifying unit 38 of the server 14 specifies the function identification information of the function associated with the extracted device identification information in the device function information 32. Thus, the function possessed by the image forming device 10 of the utilization target is specified (identified).

[0107] Furthermore, the control unit 48 of the terminal device 16 may extract the device identification information by applying a decoding process to the image data representing the mark 50. In this case, the extracted device identification information is transmitted from the terminal device 16 to the server 14. The specifying unit 38 of the server 14 specifies the function identification information of the function associated with the device identification information transmitted from the terminal device 16 in the device function information 32. In the case where the storage unit 44 of the terminal device 16 stores the device function information 32, the control unit 48 of the terminal device 16 may also specify the function identification information of the function associated with the extracted device identification information in the device function information 32.

[0108] Function identification information of the functions of the image forming device 10 may be encoded in the mark 50. In this case, by applying a decoding process to the image data representing the mark 50, the device identification information of the image forming device 10 is extracted, and at the same time, the function identification information of the functions of the image forming device 10 is also extracted. In this way, the image forming device 10 is specified (identified), and at the same time, the functions of the image forming device 10 are specified (identified). The decoding process may be performed by the server 14 or by the terminal device 16.

[0109] When executing a cooperative function using functions possessed by a plurality of devices, the cooperative function is specified (identified) by acquiring device identification information of each device by imaging a mark of each cooperative target device.

[0110] In the case of applying the markerless AR technology to obtain device identification information, for example, the user uses the camera 42 of the terminal device 16 to shoot all or part of the appearance of the target device (such as the image forming device 10). Of course, it is beneficial to obtain information for a specific device such as the name (such as the product name) or model of the target device from the appearance shooting. By shooting, appearance image data representing all or part of the appearance of the target device is generated. The appearance image data is sent from the terminal device 16 to the server 14, for example. In the server 14, the control unit 36 ​​identifies the target device based on the appearance image data. For example, in the storage unit 30 of the server 14, for each device, appearance image association information representing the association between the appearance image data and the device identification information of the device is stored, and the appearance image data represents all or part of the appearance of the device. The control unit 36, for example, compares the appearance image data sent from the terminal device 16 with each appearance image data contained in the appearance image association information, and specifies the device identification information of the target device based on the comparison result. For example, the control unit 36 ​​extracts the features of the appearance of the device being used as the target from the appearance image data sent from the terminal device 16, specifies the appearance image data that represents the same or similar features as the features of its appearance in the appearance image data group contained in the appearance image associated information, and specifies the device identification information associated with the appearance image data. Thus, the device being used as the target (the device photographed by the camera 42) is identified. As another example, when the name (e.g., product name) or model of the photographed device is generated to generate appearance image data that displays the name or model, the device being used as the target can also be identified based on the name or model represented by the appearance image data. The specific unit 38 of the server 14 specifies the function identification information of each function associated with the specific device identification information in the device function information 32. Thus, the functions possessed by the device being used as the target (e.g., the image forming device 10) are specified.

[0111] In addition, the control unit 48 of the terminal device 16 may compare the appearance image data representing all or part of the appearance of the device (e.g., the image forming device 10) of the utilization target with each appearance image data contained in the appearance image associated information, and specify the device identification information of the device of the utilization target based on the comparison result. The appearance image associated information may also be stored in the storage unit 44 of the terminal device 16. In this case, the control unit 48 of the terminal device 16 specifies the device identification information of the device of the utilization target by referring to the appearance image associated information stored in the storage unit 44 of the terminal device 16. Alternatively, the control unit 48 of the terminal device 16 may obtain the appearance image associated information from the server 14, and specify the device identification information of the device of the utilization target by referring to the appearance image associated information.

[0112] When executing a cooperative function utilizing functions of a plurality of devices, the cooperative function is specified (identified) by acquiring device identification information of each device by photographing all or part of the appearance of each cooperative target device.

[0113] In the case of obtaining device identification information by applying the location information AR technology, for example, by using the GPS (Global Positioning System) function, location information indicating the location where a device (such as the image forming device 10) is installed is obtained. For example, each device has a GPS function, and device location information indicating the location of the device itself is obtained. The terminal device 16 outputs information indicating a request to obtain device location information to the device of the target, and receives the device location information of the device from the device as a response to the acquisition request. The device location information is sent from the terminal device 16 to the server 14, for example. In the server 14, the control unit 36 ​​identifies the device of the target based on the device location information. For example, in the storage unit 30 of the server 14, location association information indicating the association between device location information and device identification information of the device is stored for each device, and the device location information indicates the location where the device is installed. The control unit 36 ​​specifies the device identification information associated with the device location information sent from the terminal device 16 in the location association information. Thus, the device of the target is specified (identified). The specifying unit 38 of the server 14 specifies the function identification information of each function associated with the specific device identification information in the device function information 32. Thereby, the function of the target device (for example, the image forming device 10 ) is specified (identified).

[0114] In addition, the control unit 48 of the terminal device 16 may specify the device identification information associated with the location information of the device of the utilization target in the location-related information. The location-related information may also be stored in the storage unit 44 of the terminal device 16. In this case, the control unit 48 of the terminal device 16 specifies the device identification information of the device of the utilization target by referring to the location-related information stored in the storage unit 44 of the terminal device 16. As another example, the control unit 48 of the terminal device 16 may obtain the location-related information from the server 14, and specify the device identification information of the device of the utilization target by referring to the location-related information.

[0115] When executing a cooperative function using a plurality of devices, the device location information of each cooperative target device is acquired, and the device identification information of each device is specified based on the device location information. Thus, the cooperative function is specified (identified).

[0116] Hereinafter, the image forming system according to the present embodiment will be described in detail.

[0117] Reference Figure 6 , the device function information 32 is described in detail. Figure 6 , an example of a device function management table as device function information 32 is shown. In the device function management table, as an example, a device ID, information indicating a device name (e.g., a type of device), information indicating a function possessed by the device (function identification information), and an image ID are associated with each other. The device ID and the device name are equivalent to an example of device identification information. The image ID is an example of image identification information for identifying an image representing a device (e.g., an image representing the appearance of the device, an image schematically representing the device (e.g., an icon), etc.). In addition, the device function management table may not contain an image ID. For example, a device with a device ID of "B" is a multifunction machine (an image forming device having multiple image forming functions) having functions such as a printing function or a scanning function. An image ID for identifying an image representing the device is associated with the device. Data representing an image of a device is stored, for example, in the storage unit 30 of the server 14.

[0118] For example, by applying AR technology, a device ID for identifying a device that is a target of utilization is obtained. The specific unit 38 of the server 14 specifies the device name, function, and image ID associated with the device ID by referring to the device function management table. Thus, the device that is a target of utilization is identified. For example, information representing the device name and data representing the image of the device are sent from the server 14 to the terminal device 16, and they are displayed on the UI unit 46 of the terminal device 16. The image representing the device is displayed as an image related to the device. Of course, the image itself captured by the camera 42 can also be displayed in the UI unit 46 of the terminal device 16. In addition, in the UI unit 46 of the terminal device 16, when the user specifies an image related to the device (such as an image captured by the camera 42 or an image schematically representing the device), information about the functions of the device (such as function identification information or function description information, etc.) can be sent from the server 14 to the terminal device 16 and displayed in the UI unit 46 of the terminal device 16.

[0119] Below, refer to Figure 7 The cooperative function information 34 will be described in detail. Figure 7An example of a collaborative function management table as collaborative function information 34 is shown. In the collaborative function management table, as an example, a combination of device IDs, information indicating the device name of the collaborative target (e.g., the type of each device), and information indicating the collaborative function (collaborative function identification information) are associated with each other. For example, a device with a device ID of "A" is a PC (personal computer), and a device with a device ID of "B" is a multifunction machine. By making the PC (A) and the multifunction machine (B) collaborate, as collaborative functions, for example, "scan and transmit function" and "print function" are realized. The "scan and transmit function" is a function of transmitting image data generated by scanning by the multifunction machine (B) to the PC (A). The "print function" is a function of sending data (e.g., image data or document data) stored in the PC (A) to the multifunction machine (B) and printing it using the multifunction machine (B).

[0120] Below, refer to Figure 8 , the processing when using a single device is described. Figure 8 , an example of a device used alone is shown. As an example, the image forming device 10 is a device used alone. The image forming device 10 is, for example, a multifunction peripheral. The image forming device 10 is a device existing in the real space. Figure 8 The terminal device 16 shown is a device existing in the real space, for example, a portable terminal device such as a smart phone or a mobile phone.

[0121] For example, a mark 50 such as a two-dimensional barcode is provided on the frame of the image forming device 10. When using the mark-type AR technology or the markless AR technology, the user shoots the image forming device 10 of the target through the camera 42 of the terminal device 16 (such as a smartphone). As a result, image data representing the mark 50 or appearance image data representing the appearance of the image forming device 10 is generated. A device display screen 52 is displayed on the display unit of the UI unit 46 of the terminal device 16, and a device image 54 related to the image forming device 10 is displayed on the device display screen 52. The device image 54 is, for example, an image generated by shooting with the camera 42 (it can be an image with the size at the time of shooting, or an enlarged or reduced image).

[0122] The image data generated by the image captured by the camera 42 is transmitted from the terminal device 16 to the server 14. In the server 14, the control unit 36 ​​extracts the device identification information of the image forming device 10 by applying a decoding process to the image data, thereby identifying the image forming device 10. As another example, appearance image data representing the appearance of the image forming device 10 may be generated, and the appearance image data may be transmitted from the terminal device 16 to the server 14. In this case, in the server 14, the control unit 36 ​​specifies the device identification information of the image forming device 10 by referring to the appearance image-related information. Thus, the image forming device 10 is identified.

[0123] In addition, the specifying unit 38 of the server 14 refers to the device function information 32 (for example, Figure 6 The device function management table shown in FIG. 10 specifies (identifies) the functions of the image forming device 10. Figure 6 The image forming apparatus 10 is, for example, a "multifunctional peripheral (B)". The specifying unit 38 specifies Figure 6 The functions associated with the multifunction machine (B) in the device function management table shown in the figure are identified. Thus, the functions possessed by the multifunction machine (B) are identified. Information on the identified functions is sent from the server 14 to the terminal device 16. Of course, the processing for identifying the device and the function can also be performed by the terminal device 16.

[0124] In addition, on the device display screen 52, instead of the image generated by photographing by the camera 42, a pre-prepared image related to the identified image forming device 10 (not the image itself obtained by photographing, but a schematic image (such as an icon)) or an image generated by photographing by an external camera may be displayed as the device image 54.

[0125] For example, when using image data obtained by a photographing device, since the appearance of the current device itself (for example, the appearance reflecting scratches, notes, stickers affixed to the device, etc.) is reflected in the image, it has the effect of allowing the user to visually understand the difference from other similar devices.

[0126] When using a schematic image, the data of the schematic image is transmitted from the server 14 to the terminal device 16. For example, when recognizing the image forming device 10, the specifying unit 38 of the server 14 refers to the Figure 6 The device function management table (device function information 32) shown in FIG. 1 specifies a schematic image related to the image forming device 10. The data of the schematic image is transmitted from the server 14 to the terminal device 16, and the schematic image is displayed as a device image 54 on the device display screen 52. The data of the schematic image may be stored in the terminal device 16 in advance. In this case, when the image forming device 10 is recognized, the device image 54 stored in the terminal device 16 is displayed on the device display screen 52. The data of the schematic image may be stored in a device other than the server 14 or the terminal device 16.

[0127] In addition, when the device is recognized, information indicating the name of the device is transmitted from the server 14 to the terminal device 16, and the name of the device may be displayed on the device display screen 52 of the terminal device 16. Figure 8 In the illustrated example, the image forming apparatus 10 is a multifunction peripheral, and its name “Multifunction peripheral (B)” is displayed.

[0128] When the functions of the image forming apparatus 10 are specified, for example, Fig. 9 As shown, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display a function display screen 56, and displays information about the function on the function display screen 56. As the information about the function, for example, a button image for instructing the execution of the function is displayed. The multifunction machine (B) as the image forming device 10 has, for example, a printing function, a scanning function, a copying function, and a fax function, and therefore, a button image for instructing the execution of these functions is displayed on the function display screen 56. For example, when a user uses the terminal device 16 to specify a button image representing a printing function and instructs the execution of the printing function, execution instruction information indicating an execution instruction of the printing function is sent from the terminal device 16 to the image forming device 10. The execution instruction information includes control data for executing the printing function, image data applied to the printing function, and the like. When the image forming device 10 receives the execution instruction information, it executes printing in accordance with the execution instruction information.

[0129] exist Fig.10 Another example of a function display screen is shown in FIG. Figure 8 As shown, the function display screen 58 is a screen displayed on the UI unit 46 of the terminal device 16 when a single device is used. As described above, the device (e.g., the image forming device 10) of the utilization target is specified, and the functions of the device of the utilization target are specified. In addition, the user who uses the device of the utilization target can also specify the function identification information indicating the function associated with the user identification information, that is, the function available to the user. Furthermore, since the functions of the device of the utilization target are specified, the functions that the device of the utilization target does not have in the function group that becomes the provision target can also be specified. This information can also be displayed on the function display screen 58.

[0130] exist Fig.10On the function display screen 58 shown, as an example of function identification information, a button image 60 indicating function A, a button image 62 indicating function B, and a button image 64 indicating function C are displayed. Function A is a function possessed by the target device (e.g., the identified image forming device 10), that is, a function available to the user. Function B is a function possessed by the target device, that is, a function not available to the user. By accepting the provision of function B, the user can use function B. If function B is a paid function, the user can use function B by purchasing function B, and if function B is a free function, the user can use function B by accepting the provision of function B for free. Function C is a function that the target device does not have, that is, a function that the target device does not correspond to. The control unit 48 of the terminal device 16 may also change the display mode of the button image according to whether the function represented by the button image is a function possessed by the target device. In addition, the control unit 48 may also change the display mode of the button image according to whether the function represented by the button image is a function available to the user. For example, the control unit 48 may also change the color or shape of the button image. exist Fig.10 In the example shown, button images 60, 62, and 64 are displayed in different colors. As an example, a button image that indicates a function possessed by the device of the target, i.e., a function available to the user (e.g., button image 60 showing function A) is displayed in blue. A button image that indicates a function possessed by the device of the target, i.e., a function not available to the user (e.g., button image 62 showing function B) is displayed in yellow. A button image that indicates a function that the device of the target does not have (e.g., button image 64 showing function C) is displayed in gray. As another example, the control unit 48 can change the shape of the button images 60, 62, and 64, or change the font of the function display name. Of course, the display method can also be changed by another method. In this way, the availability of each function can be conveyed to the user with good visibility.

[0131] For example, when a user specifies a button image 60 representing function A using the terminal device 16 and gives an execution instruction for function A, execution instruction information representing the execution instruction for function A is sent from the terminal device 16 to the target device. The execution instruction information includes control data for executing function A or data such as image data applied to the processing performed by function A. When the target device receives the execution instruction information, function A is executed according to the execution instruction information. For example, when the target device is the image forming device 10 and function A is a scan transmission function, the image forming unit 20 of the image forming device 10 executes the scan function, thereby generating scan data (image data). The scan data is sent from the image forming device 10 to the set receiving address (e.g., the terminal device 16).

[0132] In addition, when the user specifies the button image 62 representing the function B using the terminal device 16 and instructs the provision of the function B, the provision process is executed. When the provision process is performed by the server 14, the terminal device 16 accesses the server 14. As a result, as information for allowing the user to use the function B, a screen (such as a website) for accepting the provision of the function B is displayed on the UI unit 46 of the terminal device 16. By performing a provision program on this screen, the user can use the function B. For example, a program of a web browser is stored in the terminal device 16, and access from the terminal device 16 to the server 14 is achieved by using the web browser. When the user accesses the server 14 using the web browser, a function provision screen (such as a website) is displayed on the UI unit 46 of the terminal device 16, and the function is provided through the website. Of course, the provision process may also be performed by a server other than the server 14. As another example, as information for allowing the user to use the function B, a use permission entrustment screen (such as a website) for entrusting the use of the function B to an administrator or the like may be displayed on the UI unit 46 of the terminal device 16. The user can delegate permission for use of the function B to an administrator or the like via the permission for use delegation screen, and can use the function B if the permission is obtained.

[0133] Below, refer to Fig.11 This section explains how to use the collaborative function. Fig.11 , an example of a device of a collaborative target is shown. As an example, an image forming device 10 and a projector as a device 12 (hereinafter referred to as projector 12) are used as devices of a collaborative target. The image forming device 10, the projector 12, and the terminal device 16 are devices existing in the real space.

[0134] For example, a mark 50 such as a two-dimensional barcode is provided on the frame of the image forming device 10, and a mark 66 such as a two-dimensional barcode is provided on the frame of the projector 12. The mark 66 is information that encodes the device identification information of the projector 12. When using the marker-type AR technology or the marker-free AR technology, the user uses the camera 42 of the terminal device 16 (such as a smartphone) to shoot the image forming device 10 and the projector 12 of the collaborative target. Fig.11In the example shown, the image forming device 10 and the projector 12 are captured together in a state where the capturing area of ​​the camera 42 includes both the image forming device 10 and the projector 12. As a result, image data representing each mark 50, 66 is generated. A device display screen 68 is displayed on the display unit of the UI unit 46 of the terminal device 16, and a device image 70 related to the image forming device 10 and a device image 72 related to the projector 12 are displayed on the device display screen 68. The device images 70, 72 are, for example, images generated by capturing by the camera 42 (they may be images of the size at the time of capturing, or may be enlarged or reduced images).

[0135] The image data generated by the image capturing by the camera 42 is transmitted from the terminal device 16 to the server 14. In the server 14, the control unit 36 ​​extracts the device identification information of the image forming device 10 and the device identification information of the projector 12 by applying a decoding process to the image data, thereby identifying the image forming device 10 and the projector 12. As another example, appearance image data representing the appearance of each of the image forming device 10 and the projector 12 may be generated and transmitted from the terminal device 16 to the server 14. In this case, in the server 14, the control unit 36 ​​specifies the device identification information of the image forming device 10 and the device identification information of the projector 12 by referring to the appearance image related information. Thus, the image forming device 10 and the projector 12 are identified.

[0136] In addition, the specific unit 38 of the server 14 refers to the cooperative function information 34 (for example Figure 7 The cooperative function management table shown in FIG. 1 is used to specify (identify) cooperative functions that utilize the functions of the image forming apparatus 10 and the functions of the projector 12. Figure 7 This will be described in detail. The image forming apparatus 10 is exemplified as a "multifunctional peripheral (B)", and the projector 12 is exemplified as a "projector (C)". The specifying unit 38 specifies Figure 7 The cooperative function management table shown in FIG. 1 shows the cooperative function associated with the combination of the multifunction device (B) and the projector (C). Thus, the cooperative function to be executed by making the multifunction device (B) and the projector (C) cooperate with each other is specified. Information about the specific cooperative function is sent from the server 14 to the terminal device 16. Of course, the processing for identifying the device or the cooperative function can also be performed by the terminal device 16.

[0137] In addition, on the device display screen 68, instead of the image generated by the camera 42, a pre-prepared image (e.g., a schematic image (e.g., an icon)) related to the recognized image forming device 10 or an image generated by the external camera may be displayed as the device image 70. Similarly, a pre-prepared image related to the recognized projector 12 or an image generated by the external camera may be displayed as the device image 72. As described above, the data of the schematic image may be transmitted from the server 14 to the terminal device 16, may be stored in advance in the terminal device 16, or may be stored in a device other than these.

[0138] In addition, when a device is identified, information indicating the name of the device may be transmitted from the server 14 to the terminal device 16, and the name of the device may be displayed on the device display screen 68 in the terminal device 16. Fig.11 In the illustrated example, the name of the image forming apparatus 10 “multifunctional peripheral (B)” and the name of the projector 12 “projector (C)” are shown.

[0139] Even when a plurality of devices are captured, the identifying unit 38 of the server 14 can identify the function of each device by referring to the device function information 32. Fig.11 In the example shown, the specifying unit 38 may specify the functions of the image forming apparatus 10 and the functions of the projector 12. Information on the specified functions may be transmitted from the server 14 to the terminal device 16.

[0140] When specific collaborative functions, e.g. Fig.12 As shown, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display a function display screen 74, and causes the function display screen 74 to display information about the collaborative function. As the information about the collaborative function, for example, a button image for indicating the execution of the collaborative function is displayed. By making the multifunction machine (B) cooperate with the projector (C), for example, a collaborative function can be executed in which the projector (C) projects an image generated by scanning the multifunction machine (B), or the multifunction machine (B) prints an image projected by the projector (C). The button image for indicating the execution of these collaborative functions is displayed on the function display screen 74. For example, when the user uses the terminal device 16 to specify a button image and instruct the execution of the collaborative function, the execution instruction information indicating the execution instruction of the collaborative function is sent from the terminal device 16 to the image forming device 10 and the projector 12. When the image forming device 10 and the projector 12 receive the execution instruction information, they execute the collaborative function specified by the user.

[0141] The device of the cooperative target can also be specified by the user's operation. For example, by shooting the image forming device 10 and the projector 12 by the camera 12, as shown in FIG. Fig.11 As shown, the device image 70 related to the image forming device 10 and the device image 72 related to the projector 12 are displayed on the UI part 46 of the terminal device 16. In addition, the image related to the device can be an image obtained by shooting with the camera 42, or it can be a pre-prepared image related to the identified device (for example, a schematic image (for example, an icon)). By the user specifying the device images 70 and 72 on the device display screen 68, the image forming device 10 and the projector 12 are specified as devices of the collaborative target. For example, when the user specifies the device image 70, the marker-type AR technology or the marker-free AR technology is applied to the device image 70, thereby specifying (identifying) the image forming device 10. Similarly, when the user specifies the device image 72, the marker-type AR technology or the marker-free AR technology is applied to the device image 72, thereby specifying (identifying) the projector 12. As a result, the collaborative function performed by the image forming device 10 and the projector 12 is specified, and information about the collaborative function is displayed on the UI part 46 of the terminal device 16.

[0142] As another example, by touching the device image 70 on the device display screen 68 and operating with a finger until the device image 72 (for example, drawing with a finger, etc.), the device images 70 and 72 are designated, and the image forming device 10 and the projector 12 can also be designated as the devices of the collaborative target. The order of touching the device images 70 and 72 or the direction of drawing can also be opposite to the above example. Of course, a screen contact device other than a finger such as a pen for drawing the device display screen 68 can also be used. The device of the collaborative target can be designated by a drawing operation such as drawing a circle, or by specifying a device image related to the device to be coordinated within a preset time. When canceling the collaboration, the user can specify the device of the cancellation target on the device display screen 68, or press the collaboration cancellation button. In the case where a device that is not the collaborative target is projected on the image, the user can also specify the device on the device display screen 68 to remove the device from the collaborative target device group. The device of the cancellation target can also be designated by a preset operation such as marking a cross mark.

[0143] Each device of the collaborative target may also be photographed separately. For example, by performing the photographing of the camera 42 in multiple times, each device of the collaborative target is identified. In the case where the photographing of the camera 42 is divided into multiple times, the device identification information of the device identified in each photographing is stored in the storage unit of the server 14 or the terminal device 16. For example, the image forming device 10 is photographed in a state where the photographing area of ​​the camera 42 includes the image forming device 10, and then the projector 12 is photographed in a state where the photographing area of ​​the camera 42 includes the projector 12. Thus, image data about the image forming device 10 and image data about the projector 12 are generated, and by applying the marker-type AR technology or the marker-free AR technology to each image data, the image forming device 10 and the projector 12 are specified (identified), and the collaborative function that utilizes the functions of the image forming device 10 and the projector 12 is specified (identified). For example, it is not limited to configuring each device of the collaborative target at a position close to each other and the photographing area of ​​the camera 42 must include both the image forming device 10 and the projector 12. Sometimes, it is possible to cope with the situation by changing the angle of the shooting area of ​​the camera 42 or by enlarging or reducing the shooting area. However, when such operations cannot be used to cope with the situation, a method of providing a device for identifying collaborative targets by taking pictures multiple times can also be used as a solution.

[0144] As another example, the device of the collaborative target may be pre-set as the basic collaborative device. For example, the image forming device 10 is pre-set as the basic collaborative device. The device identification information of the basic collaborative device may also be pre-stored in the storage unit of the server 14 or the terminal device 16. The user may also specify the basic collaborative device using the terminal device 16. When the basic collaborative device is set, the user uses the camera 42 of the terminal device 16 to photograph the device of the collaborative target other than the basic collaborative device. Thus, the device of the collaborative target is specified (identified), and the collaborative function that utilizes the basic collaborative device and the functions possessed by the photographed device is specified (identified).

[0145] In addition, in the case where the device of the shooting target cannot be identified even if AR technology is applied, the device image representing the device of the shooting target may not be displayed on the device display screen. In this way, the visibility of the identified device can be improved. For example, in the case where the identified device and the unidentified device are mixed and are respectively photographed by the camera 42, the device image representing the unidentified device is set to non-display. In this way, the device image representing the identified device is displayed and will not be covered by the device image representing the unidentified device, thereby improving the visibility of the identified device. In addition, the device image representing the identified device may also be highlighted. For example, the device image representing the identified device may be displayed in a specific color, the edge of the device image may be emphasized, the device image may be enlarged and displayed, the device image may be displayed in three dimensions, or the device image may be displayed in a flashing manner. In this way, the visibility of the identified device can be improved.

[0146] Hereinafter, switching control between the display of the function of utilizing the device independently (hereinafter referred to as “independent utilization function”) and the display of the cooperative function will be described in detail.

[0147] For example, when only one device is identified within a preset identification time, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the functions of the one device (e.g., the image forming device 10) as information about the function of using the single function. The length of the identification period can also be changed by the user. In addition, the device can be identified by applying AR technology, and the device can be identified by applying technology other than AR technology. The identification process of the device can be performed by the server 14 or by the terminal device 16. The starting point (starting point) of the identification time can be, for example, the time point when a device is identified, or the time point specified by the user (e.g., the time point when the identification process starts).

[0148] For example, when no other device is recognized within the recognition time from the time when one device is recognized, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the function of the one device as information about the single use function. In this case, the one device is treated as a device recognized within the recognition period. The information about the device may be information sent from the server 14 to the terminal device 16, or may be information pre-stored in the terminal device 16.

[0149] As another example, when a time point specified by the user is used as the starting time point of the identification time, and only one device is identified within the identification time from the starting time point, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the functions of the one device as information about the individual use of the function.

[0150] As another example, when a display instruction of the independent use function is given after a device is recognized, the control unit 48 of the terminal device 16 may cause the UI unit 46 of the terminal device 16 to display information about the function of the device as the information about the independent use function. The control unit 48 of the terminal device 16 may cause the UI unit 46 of the terminal device 16 to display information about the function of the device at all times or when a device is photographed (or when a device is recognized). Figure 8 The device display screen 52 shown in the figure displays a button image for instructing the display of the individual utilization function. When the user presses the button image, the control unit 48 causes the UI unit 46 to display information on the function of the one device.

[0151] At the time point when the recognition time has passed, the control unit 48 of the terminal device 16 may also cause the UI unit 46 of the terminal device 16 to display a confirmation screen. The confirmation screen is, for example, a screen for the user to instruct the extension of the recognition time. When the user instructs the extension of the recognition time via the confirmation screen, if no other device is captured within the extended time, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the functions of the recognized device.

[0152] The display control of information about the single-use function is described in more detail. For example, the device is identified by marker-type AR technology or marker-free AR technology. For example, when only one device is photographed within a preset shooting time, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the functions possessed by the one device as information about the single-use function. The starting point (starting point) of the shooting time can be the time point when a device is photographed, or it can be the time point indicated by the user (for example, the shooting start time point). The length of the shooting time can also be changed by the user. In addition, when the one device is photographed, the one device is identified by marker-type AR technology or marker-free AR technology. The recognition process can be performed by the server 14 or by the terminal device 16.

[0153] For example, when no other device is captured within the capturing time from the time when one device is captured, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the function of the one device as information about the single-use function. In this case, the one device is treated as a device captured within the capturing time.

[0154] As another example, when a time point specified by the user is used as the starting time point of the shooting time, and only one device is photographed during the shooting time from the starting time point, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the function of the one device as information about the single use function.

[0155] As another example, when a display indication of a separate function is given after a device is photographed, the control unit 48 of the terminal device 16 may cause the UI unit 46 of the terminal device 16 to display information about the function of the device as information about the separate function.

[0156] For example, when only one device is photographed within the shooting time (for example, when no other device is photographed within the shooting time from the time point when the first device is photographed, or when only one device is photographed within the shooting time from the start time point specified by the user), the control unit 48 of the terminal device 16 sends the image data generated by the shooting to the server 14. The shooting time can be measured by the control unit 48 or by a timer. The specific unit 38 of the server 14 specifies (identifies) the device based on the image data and specifies the function of the device. Information about the function is sent from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16. Of course, the terminal device 16 may not manage the time, but the server 14 may manage the time and send information about the function of the identified device to the terminal device 16.

[0157] At the time point when the shooting time has passed, the control unit 48 of the terminal device 16 may also cause the UI unit 46 of the terminal device 16 to display a confirmation screen. The confirmation screen is, for example, a screen for the user to instruct the extension of the shooting time. When the user instructs the extension of the shooting time via the confirmation screen, if no other device is photographed within the extended time, the control unit 48 of the terminal device 16 sends the image data obtained by the shooting to the server 14, and accordingly causes the UI unit 46 of the terminal device 16 to display the function of the one device. The length of the extended time may also be changed by the user.

[0158] As another example, after photographing a device and providing a display indication of a separate function, the control unit 48 of the terminal device 16 sends the image data generated by the photographing to the server 14, and accordingly, can also receive information about the functions of the photographed device from the server 14.

[0159] As another example, the control unit 48 of the terminal device 16 may send the image data to the server 14 each time the device is photographed to generate image data, and accordingly receive information about the function of the photographed device from the server 14. In this case, when only one device is photographed during the photographing time, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display the information about the one device as the information about the single use function.

[0160] On the other hand, when a plurality of devices are recognized within the recognition time, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about a collaborative function using the functions of the plurality of devices recognized within the recognition time. When the recognition time is extended, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about a collaborative function using the functions of the plurality of devices recognized within the recognition time including the extended time. The plurality of devices recognized within the recognition time are designated as collaborative target devices. The information about the collaborative function may be information sent from the server 14 to the terminal device 16, or may be information pre-stored in the terminal device 16.

[0161] For example, when another device is recognized within the recognition time from the time when the first device is recognized, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about a cooperative function using functions of multiple devices including the first device and the other device. In this case, the first device is also treated as a device recognized within the recognition time.

[0162] In addition, when another device is recognized within the recognition time from the time point when the first device is recognized, the control unit 48 of the terminal device 16 may also set a new recognition time with the time point when the other device is recognized as the starting time point. In the same way, when another device is further recognized within the new recognition time, a new recognition time is further set. In this case, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the cooperative function using the functions of the multiple devices (including the first device) recognized within the full recognition time.

[0163] As another example, when a time point specified by the user is used as the starting time point of the recognition time, and multiple devices are photographed within the recognition time from the starting time point, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about collaborative functions utilizing the functions of the multiple devices.

[0164] As another example, when a display instruction of a cooperative function is given after multiple devices are recognized, the control unit 48 of the terminal device 16 may also cause the UI unit 46 of the terminal device 16 to display information about a cooperative function using the functions of the multiple devices. The control unit 48 of the terminal device 16, for example, always or when multiple devices are photographed (or when multiple devices are recognized), causes the device display screen 68 to display a button image for indicating the display of a cooperative function. When the user presses the button image, the control unit 48 causes the UI unit 46 to display information about a cooperative function using the functions of the multiple devices.

[0165] As another example, when another device is identified after the first device is identified and before an instruction to execute the function of the first device is given, the control unit 48 of the terminal device 16 may cause the UI unit 46 of the terminal device 16 to display information about a collaborative function using the functions of the multiple devices. Fig. 9 or Fig.10 As shown, when the UI unit 46 of the terminal device 16 displays the function display screens 56 and 58, when other devices are photographed before the user instructs the execution of a function, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about collaborative functions utilizing the initially identified device and the functions possessed by the other devices.

[0166] The display control of information about the collaborative function is described in more detail. For example, multiple devices are identified by marker-type AR technology or marker-free AR technology. For example, when multiple devices are photographed within a preset shooting time, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about collaborative functions that utilize the functions of the multiple devices photographed within the shooting time. When the shooting time is extended, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about collaborative functions that utilize the functions of the multiple devices photographed within the shooting time including the extended time. The multiple devices photographed within the shooting time are designated as collaborative target devices. In addition, if multiple devices are photographed, the multiple devices are identified by marker-type AR technology or marker-free AR technology. The recognition process can be performed by the server 14 or by the terminal device 16.

[0167] For example, when another device is photographed within the photographing time from the time when the first device is photographed, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about a collaborative function using functions of multiple devices including the first device and the other device. In this case, the first device is also treated as a device photographed within the photographing time.

[0168] In addition, when another device is captured within the capturing time from the time point when the first device is captured, the control unit 48 of the terminal device 16 may set a new capturing time with the time point when the other device is captured as the starting time point. In the same manner, when another device is further captured within the new capturing time, a new capturing time is further set. In this case, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the collaborative function using the functions of the multiple devices (including the first device) captured within the full capturing time.

[0169] As another example, when a time point specified by the user is used as the starting time point of the shooting time, and multiple devices are photographed within the shooting time from the starting time point, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the collaborative function utilizing the functions of the multiple devices.

[0170] As another example, when a plurality of devices are photographed and a display instruction for a cooperative function is given, the control unit 48 of the terminal device 16 may cause the UI unit 46 of the terminal device 16 to display information about a cooperative function utilizing functions of the plurality of devices.

[0171] As another example, when another device is photographed after the first device is photographed and before the execution instruction of the function of the first device is given, the control unit 48 of the terminal device 16 may cause the UI unit 46 of the terminal device 16 to display information about the cooperative function using the functions of the multiple devices. Fig. 9 or Fig.10 As shown, when the UI unit 46 of the terminal device 16 displays the function display screens 56 and 58, in a case where another device is photographed before the user instructs the execution of a function, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about a collaborative function utilizing the device initially photographed and the function possessed by the other device.

[0172] For example, when multiple devices are photographed within the shooting time (for example, when other devices are photographed within the shooting time from the time point when the first device is photographed, or when multiple devices are photographed within the shooting time from the start time point specified by the user), the control unit 48 of the terminal device 16 sends the multiple image data generated by the shooting to the server 14. The specifying unit 38 of the server 14 specifies (identifies) each device based on each image data, and specifies the collaborative function of the functions of the identified multiple devices. Information about the collaborative function is sent from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16. Of course, the functions that each device has individually can also be specified, and the information about the functions that each device has can be sent from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16.

[0173] As another example, after photographing multiple devices and providing a display indication of a collaborative function, the control unit 48 of the terminal device 16 sends multiple image data generated by the photographing to the server 14, and accordingly, can also receive information about the collaborative function that utilizes the functions of the multiple devices photographed from the server 14.

[0174] As another example, when other devices are photographed after the first device is photographed and before an instruction to execute a function of the first device is given, the control unit 48 of the terminal device 16 sends multiple image data generated by the photographing to the server 14, and accordingly, information about collaborative functions utilizing the functions of the multiple devices photographed can also be received from the server 14.

[0175] As another example, the control unit 48 of the terminal device 16 sends the image data to the server 14 every time it captures a device and generates image data, and accordingly, it can also receive information about the collaborative function using the functions of the multiple devices captured from the server 14. In this case, when the control unit 48 of the terminal device 16 captures multiple devices within the capturing time, it causes the UI unit 46 of the terminal device 16 to display the information about the collaborative function using the functions of the multiple devices.

[0176] Next, another display example of information on functions will be described.

[0177] exist Fig.13 , an example of display of a single-use function is shown. The control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display a device display screen 52, and causes the device display screen 52 to display a device image 54 representing the device (e.g., the image forming device 10) photographed, and information about the functions of the device. Fig.13In the example shown, information about the functions of the image forming apparatus 10 (e.g., the names of the functions) is displayed in a balloon-shaped display area 76. The shape of the display area 76 is merely an example, and a display area 76 having another shape may be displayed. When the user specifies information about the function displayed in the display area 76 and gives an instruction to execute the function, the specified function is executed.

[0178] exist Fig.14 , a display example of a collaborative function is shown. The control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display a device display screen 68, and displays device images 70 and 72 representing multiple devices (such as the image forming device 10 and the projector 12) that have been photographed on the device display screen 68. In addition, the control unit 48 of the terminal device 16 causes the device display screen 68 to display information about a collaborative function that utilizes the functions of the multiple devices. Fig.14 In the example shown, information about the cooperative function (e.g., the name or description of the cooperative function) is displayed in the balloon-shaped display area 78. The display area 78 is merely an example, and a display area 78 having another shape may be displayed. When the user specifies the information about the cooperative function displayed in the display area 78 and gives an instruction to execute the cooperative function, the specified cooperative function is executed.

[0179] In addition, the control unit 48 of the terminal device 16 may display information on the individual utilization functions of each device of the cooperation target on the device display screen 68. Fig.14 In the example shown, information about the functions of the image forming device 10 is displayed in the balloon-shaped display area 80, and information about the functions of the projector 12 is displayed in the balloon-shaped display area 82. The control unit 48 of the terminal device 16 may display information about the cooperative function and information about the independent use function separately on the device display screen 68. For example, the display area 78 may be displayed in a color different from that of the display areas 80 and 82, and the shape of the display area 78 may be different from that of the display areas 80 and 82. As another example, the display positions of the display areas 78, 80, and 82 may be different according to the independent use function and the cooperative function. For example, the display area 80 is displayed near the corresponding device image 70, the display area 82 is displayed near the corresponding device image 72, and the display area 78 is displayed at a position farther away from the device images 70 and 72 than the display areas 80 and 82. In this way, by displaying the display area 78 separately from the display areas 80 and 82, the user can easily distinguish between the independent use function and the cooperative function.

[0180] The control unit 48 of the terminal device 16 may also control the display of information about the function to be used alone or the display of information about the cooperative function according to the user's instructions. For example, when the user specifies a device image (e.g., device image 70), the control unit 48 of the terminal device 16 may also cause the device display screen 68 to display information about the function possessed by the device (e.g., image forming device 10) associated with the specified device image. When the user specifies a device image and gives a display instruction to use a function to be used alone, the control unit 48 of the terminal device 16 may cause the device display screen 68 to display information about the function possessed by the device associated with the specified device image. When the user specifies multiple device images (e.g., device images 70, 72), the control unit 48 of the terminal device 16 may also cause the device display screen 68 to display information about the cooperative function that utilizes the functions possessed by multiple devices (e.g., image forming device 10 and projector 12) associated with the specified multiple device images. For example, after the user specifies the first device image, if the user specifies another device image within a preset time, the control unit 48 of the terminal device 16 causes the device display screen 68 to display information about the cooperative function using the functions of the multiple devices associated with the multiple device images. As another example, if the user specifies multiple device images and gives a display instruction for the cooperative function, the control unit 48 of the terminal device 16 may cause the device display screen 68 to display information about the cooperative function using the functions of the multiple devices associated with the specified multiple device images.

[0181] As described above, according to this embodiment, when one device is identified (e.g., photographed), information about the functions of the one device is displayed, and when multiple devices are identified (e.g., photographed), information about cooperative functions using the functions of the multiple devices is displayed. Thus, information about functions that can be executed using the identified (e.g., photographed) devices is provided to the user, which is more convenient for the user.

[0182] In addition, by executing the cooperative function, functions that cannot be executed by each device alone can be executed, which is more convenient.

[0183] In addition, since individual functions and collaborative functions can be utilized by simply applying AR technology to the recognition device, each function can be utilized with simple operations compared to a situation where the user manually performs settings for executing functions, thereby saving the user's time and effort.

[0184] In addition, according to the present embodiment, for example, in an environment where multiple users use multiple devices, information about functions used alone or information about cooperative functions is appropriately displayed on the terminal device 16. For example, even when a user interface such as a touch panel is removed from the device, the terminal device 16 can be used as a user interface. As another scenario, for example, when a device is temporarily used at a user's destination, the terminal device 16 can also implement a user interface suitable for the user, that is, a user interface that displays the functions of the device specified by the user or the cooperative functions using the device.

[0185] Furthermore, in the above-described embodiment, the cooperative function using two devices has been described. However, it goes without saying that the cooperative function may also be a function using three or more devices.

[0186] In addition, even when a plurality of devices are recognized (eg, photographed), a device specified by the user can be excluded from the devices of the cooperation target.

[0187] (Variant 1)

[0188] The following describes variant example 1. In variant example 1, multiple devices are identified and information about the collaborative function is displayed. As in the above-mentioned embodiment, the device can be identified by using AR technology or by using technology other than AR technology. The device identification process can be performed by the server 14 or by the terminal device 16.

[0189] In Modification 1, the control unit 48 of the terminal device 16 changes the cooperative function according to the display position of the device image on the device display screen 68, and causes the UI unit 46 of the terminal device 16 to display information about the cooperative function. For example, the order of use of each device used in the cooperative function is changed according to the display position of the device image, and the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the cooperative function in accordance with the order of use. In other words, the order of movement of data moved between each device used in the cooperative function is changed according to the display position of the device image, and the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the cooperative function in accordance with the order of movement. Modification 1 is described in detail below.

[0190] exist Fig.15, a device display screen 68 of variant example 1 is shown. On the device display screen 68, a starting point display area 86 for displaying a device image related to the device that becomes the starting point, and a collaborative target display area 88 for displaying a device image related to the device that becomes the next collaborative target are pre-set, for example, by a boundary line 84. As an example, the device display screen 68 is divided into left and right, with the starting point display area 86 set on the left and the collaborative target display area 88 set on the right. Of course, the boundary line 84 or the character column representing the display area (such as "left" or "right") may not be displayed. Fig.15 In the example shown, the device image 70 is displayed in the starting point display area 86, and the device image 72 is displayed in the cooperation target display area 88. Therefore, the image forming device 10 (multifunction machine (B)) associated with the device image 70 is designated as the starting point device, and the projector 12 (projector (C)) associated with the device image 72 is designated as the cooperation target device. In this case, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the cooperation function that uses the image forming device 10 as the starting point device and uses the projector 12 as the cooperation target device. For example, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the cooperation function that first uses the image forming device 10 as the starting point device and then uses the projector 12 as the cooperation target device. In other words, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the cooperation function that moves data from the image forming device 10 as the starting point device to the projector 12 as the cooperation target device. Furthermore, the device images 70 and 72 may be images generated by photographing with a camera, or may be schematic images (eg, icons).

[0191] exist Fig.16 , an example of displaying information about the cooperative function is shown. A function display screen 90 is displayed on the UI unit 46 of the terminal device 16. Information about the cooperative function using the image forming device 10 as the starting device and the projector 12 as the cooperative target device is displayed on the function display screen 90. Fig.16 In the example shown, the cooperative function is a function of moving image data obtained by scanning from the image forming device 10 (multifunctional peripheral (B)) to the projector 12 (projector (C)) and projecting the image by the projector 12. Specifically, when the cooperative function is executed, the image forming device 10 is used as the starting device, the image data generated by scanning by the image forming device 10 is sent to the projector 12 as the cooperative target device, and the projector 12 projects the image.

[0192] exist Fig.17, another example of the device display screen 68 of the modification example 1 is shown. On the device display screen 68, the starting point display area 92 and the cooperative target display area 94 are set by the boundary line 84. Fig.15 Unlike the example shown, on the device display screen 68, a starting point display area 92 is set on the right side and a cooperation target display area 94 is set on the left side. In this case, the projector 12 (projector (C)) associated with the device image 72 is designated as the starting point device, and the image forming device 10 (multifunction device (B)) associated with the device image 70 is designated as the cooperation target device. In this case, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the cooperation function that uses the projector 12 as the starting point device and uses the image forming device 10 as the cooperation target device. For example, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the cooperation function that first uses the projector 12 as the starting point device and then uses the image forming device 10 as the cooperation target device. In other words, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the cooperation function that moves data from the projector 12 as the starting point device to the image forming device 10 as the cooperation target device.

[0193] exist Fig.18 , an example of displaying information about the cooperative function is shown. A function display screen 96 is displayed on the UI unit 46 of the terminal device 16, and information about the cooperative function using the projector 12 as the starting device and the image forming device 10 as the cooperative target device is displayed on the function display screen 96. Fig.18 In the example shown, the collaborative function is a function of printing (printing) the projected image. Specifically, when the collaborative function is executed, the projector 12 is used as the starting device, the image data projected by the projector 12 is sent to the image forming device 10 as the collaborative target device, and the image forming device 10 prints the image data.

[0194] In variant example 1, the terminal device 16 sends the data of the device image displayed in the starting point display area 86 (92) as the image data of the starting point device to the server 14, and sends the data of the device image displayed in the collaborative target display area 88 (94) as the image data of the collaborative target device to the server 14. The specific unit 38 of the server 14 treats the device associated with the image data of the starting point device as the starting point device, and treats the device associated with the image data of the collaborative target device as the collaborative target device. The server 14 stores data of a collaborative function management table of variant example 1, in which the device identification information of the starting point device, the device identification information of the collaborative target function, and the collaborative function identification information are associated with each other. The specific unit 38 of the server 14 specifies in the collaborative function management table the collaborative function identification information associated with the device identification information of the device specified as the starting point device and the device identification information of the device specified as the collaborative target device. For example, the device identification information of the image forming device 10 as the starting point device, the device identification information of the projector 12 as the collaborative target device, and the collaborative function ( Fig.16 In addition, the device identification information of the projector 12 as the starting device, the device identification information of the image forming device 10 as the collaborative target device, and the collaborative function ( Fig.18 Therefore, by referring to the cooperative function management table of Modification 1, the cooperative function that utilizes the functions of the origin device and the cooperative target device is identified.

[0195] In variant 1, the display position of the device image can also be changed. In this case, the device display screen showing the device image functions as an editing screen for changing the display position of the device image. In addition, the screen can be turned to the editing screen in the shooting state, or the editing screen for indicating the collaborative function can be displayed in the state where the device image is held, such as when the device image is shot in advance or when the device image is received. Of course, other methods can also be used to transition the screen to the editing screen. For example, Fig.19As shown, the device image 70 is displayed in the starting point display area 86, and the device image 72 is displayed in the cooperation target display area 88. For example, when the user moves the device image 70 from the starting point display area 86 to the cooperation target display area 88 in the direction indicated by the arrow 98, the control unit 48 of the terminal device 16 displays the device image 70 in the cooperation target display area 88 according to the movement instruction, and does not display the device image 70 in the starting point display area 86. As a result, the image forming device 10 (multifunction machine (B)) related to the device image 70 is specified (recognized) as the cooperation target device. In addition, when the user moves the device image 72 from the cooperation target display area 88 to the starting point display area 86 in the direction indicated by the arrow 100, the control unit 48 of the terminal device 16 displays the device image 72 in the starting point display area 86 according to the movement instruction, and does not display the device image 72 in the cooperation target display area 88. As a result, the projector 12 (projector (C)) related to the device image 72 is specified (recognized) as the starting point device. After the display position is changed and the user has given a display instruction for the collaborative function, information about the collaborative function corresponding to the changed display position can also be displayed on the UI unit 46 of the terminal device 16. Even if the user has not given a display instruction for the collaborative function, information about the collaborative function corresponding to the changed display position can be displayed on the UI unit 46 of the terminal device 16 along with the change of the display position.

[0196] In the above-mentioned modification 1, the device display screen 68 is divided in the left-right direction, but this is only an example, and it may be divided in the upper-lower direction, and the starting point display area and the cooperative target display area may be displayed side by side in the upper-lower direction. As another example, the starting point display area may be displayed near the center of the screen, and the cooperative target display area may be displayed in a manner surrounding the starting point display area. Conversely, the cooperative target display area may be displayed near the center of the screen, and the starting point display area may be displayed in a manner surrounding the cooperative target display area.

[0197] In addition, in the above-mentioned variation 1, the collaborative function using two devices is described, but it goes without saying that the collaborative function can also be a function using more than three devices. For example, the control unit 48 of the terminal device 16 causes the device display screen 68 to display multiple collaborative target display areas. A utilization order is pre-set for each collaborative target display area, and the utilization order (collaborative order) of each device is determined according to the utilization order. For example, the device associated with the device image displayed in the first collaborative target display area is used as the first collaborative target device, and the device associated with the device image displayed in the second collaborative target display area is used as the second collaborative target device. The collaborative function in this case is a function of first utilizing the starting device, then utilizing the first collaborative target device, and then utilizing the second collaborative target device. In other words, the collaborative function is a function of moving data in the order of the starting device, the first collaborative target device, and the second collaborative target device. The utilization order set in each collaborative target display area can also be changed by the user.

[0198] As described above, according to Modification 1, since the cooperative function is changed according to the display position of the device image, the operability of changing the cooperative function can be improved. In addition, the cooperative function can be changed with a simple operation.

[0199] (Variant 2)

[0200] Modification 2 is described below. In modification 2, a plurality of devices are recognized similarly to modification 1. In addition, information on a function to be used independently or information on a cooperative function is displayed depending on the display position of the device image.

[0201] For example, when a plurality of device images are arranged in a specific direction (e.g., left-right direction) on the device display screen of the UI unit 46 of the terminal device 16, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about a cooperative function using functions of the plurality of devices associated with the plurality of device images. Fig. 20 As shown, the device display screen 68 is divided into left and right by the boundary line 102, and when the device image 70 is displayed in the left area 104 and the device image 72 is displayed in the right area 106, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information on the cooperative function using the image forming device 10 related to the device image 70 and the projector 12 related to the device image 72. As in the first modification, one of the areas 104 and 106 may be used as the starting display area, and the other area may be used as the cooperative target display area.

[0202] When a plurality of device images are arranged in a direction intersecting the above-mentioned specific direction (e.g., up and down direction) on the device display screen, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the functions (independent use functions) possessed by each device associated with each device image. Fig.21 As shown, when the device display screen 68 is divided into upper and lower parts by the boundary line 108, and the device image 70 is displayed in the upper area 110 and the device image 72 is displayed in the lower area 112, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the functions (independent use functions) of the image forming device 10 related to the device image 70, and information about the functions (independent use functions) of the projector 12 related to the device image 72.

[0203] Furthermore, the boundary lines 102 and 108 and the character strings on the left and right and on the top and bottom may or may not be displayed.

[0204] For example, when the user gives a display instruction of the display area for the cooperative function using the UI unit 46 of the terminal device 16, as shown in FIG. Fig. 20 As shown in FIG. 1 , the control unit 48 of the terminal device 16 divides the device display screen 68 into left and right sections and sets the sections 104 and 106. On the other hand, when the user gives a display instruction for the display area for the separate use function, as shown in FIG. Fig.21 As shown, the control unit 48 of the terminal device 16 divides the device display screen 68 into upper and lower parts and sets the areas 110 and 112. Of course, even if the user does not give a display instruction for the display area, when a plurality of device images are arranged in the left-right direction, information on cooperative functions can be displayed, and when a plurality of device images are arranged in the upper-lower direction, information on independently used functions can be displayed.

[0205] When multiple device images are arranged in the left-right direction, information about the individual functions can be displayed, and when multiple device images are arranged in the up-down direction, information about the cooperative functions can be displayed. Of course, when multiple device images are arranged in directions other than these, either information about the individual functions or information about the cooperative functions can be displayed according to their directions.

[0206] As described above, according to Modification 2, since information on the independently utilized functions or information on the coordinated functions is displayed according to the display positions of the plurality of device images, these information can be switched and displayed with a simple operation.

[0207] (Variant 3)

[0208] The following describes a variant example 3. In the variant example 3, when the device image associated with the captured device does not satisfy the pre-set area reference, the device cannot be identified as a device of the utilization target. Fig. 22 As shown, the image forming device 10 as a multifunction machine (B) is photographed by the camera 42. Trees are arranged as obstacles in front of the image forming device 10 (between the image forming device 10 and the terminal device 16), and the device image 70 related to the image forming device 10 (for example, the image generated by the photography performed by the camera 42) overlaps with the image 114 of the trees. That is, the device image 70 is missing due to the image of the obstacle. In this case, the image forming device 10 cannot be identified as a device for utilization. For example, the appearance image data obtained by photographing the device is sent from the terminal device 16 to the server 14, and the specific unit 38 of the server 14 specifies (identifies) the device based on the appearance image data. In this case, when the appearance image data contains a missing portion, the specific unit 38 of the server 14 cannot specify (identify) the device based on the appearance image data. As a result, the device (for example, the image forming device 10) related to the appearance image data containing the missing portion cannot be identified as a device for utilization.

[0209] Furthermore, obstacles are objects that cannot be recognized as devices by the specific unit 38 of the server 14. Since trees and the like cannot be registered as devices, even if appearance image data representing trees is sent to the server 14, the trees cannot be recognized as devices but are recognized as obstacles.

[0210] The specific unit 38 of the server 14 may also specify (identify) the device based on the appearance image data according to the size of the area other than the missing portion in the appearance image data. For example, when the size of the area other than the missing portion in the appearance image data is greater than a preset threshold, the specific unit 38 of the server 14 may also specify (identify) the device based on the area other than the missing portion in the appearance image data. As another example, when the area other than the missing portion is larger than the missing portion, the specific unit 38 of the server 14 may also specify (identify) the device based on the area other than the missing portion in the appearance image data. As another example, when the markerless AR technology is applied to the appearance image data and the device can be specified, even if the appearance image data contains a missing portion, the specific unit 38 of the server 14 may specify (identify) the device based on the area other than the missing portion in the appearance image data.

[0211] When the image forming device 10 (multifunction machine (B)) is not recognized as the device to be used, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the functions of other devices that have been photographed. For example, the projector 12 as the projector (C) is photographed, and the device image 72 related to the projector 12 is displayed, and the camera (D) is photographed, and the device image 116 related to the camera (D) is displayed. In this case, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the cooperative function of the functions of the projector (C) and the camera (D), or information about the functions (independent use functions) respectively possessed by the projector (C) and the camera (D) according to the processing of the above-mentioned embodiment.

[0212] In addition, a collaborative function can be constructed by utilizing functions related to image parts other than the missing part in the appearance image data. Fig. 22 In the example shown, when the main body of the multifunction machine (B) (image forming device 10) is shown in the device image 70, the control unit 48 of the terminal device 16 may also display information about a cooperative function using the function of the main body or information about a function (independently used function) of the main body in the UI unit 46 of the terminal device 16 according to the processing of the above-mentioned embodiment. For example, in each part of the device, an image of the appearance part representing the part and the function of the part are associated with each other, and the device function information 32 representing the association is stored in the storage unit 30 of the server 14. By referring to the device function information 32, the function related to the image part other than the missing part is specified (identified). In addition, the cooperative function information 34 representing the cooperative function using the functions of multiple parts is stored in the storage unit 30 of the server 14. Of course, the cooperative function information 34 may also include information representing the cooperative function using the function of the part of the device and the function of the entire device. By referring to the cooperative function information 34, the cooperative function using the function related to the image part other than the missing part is specified (identified).

[0213] In the case where the device image is missing, the specific unit 38 of the server 14 can also specify (identify) the device related to the device image by using multiple recognition technologies such as marker-type AR technology or location information AR technology, thereby specifying (identifying) the functions of the device and utilizing the collaborative functions of the functions of the device.

[0214] As described above, according to Modification 3, the device associated with the missing device image is removed from the target device. This prevents the device associated with the missing device image from being recognized as a device that the user does not expect to be the target device, and as a result, prevents information about the function of the device or information about a cooperative function using the function of the device from being displayed.

[0215] (Variant 4)

[0216] Modification 4 is described below. In modification 4, a plurality of devices are identified similarly to modification 1. In addition, the device used as the starting point of the cooperative function is changed according to the positional relationship between the imaging device that captures the device and each device.

[0217] For example, the closer the device (the device photographed) is to the shooting device (for example, the terminal device 16 with the camera 42), the more preferentially it is used as the starting point device of the collaborative function. In this case, the specific unit 38 of the server 14 obtains information indicating the distance between the terminal device 16 and each device, and gives priority to each device based on the distance. The location information of the terminal device 16 or each device is obtained, for example, by utilizing the GPS (Global Positioning System) function. Of course, the location information can also be obtained by other technologies. For example, the terminal device 16 or each device has a GPS function and obtains its own location information. The terminal device 16 obtains location information from each device and sends the location information of the terminal device 16 and each device to the server 14. The specific unit 38 of the server 14 calculates the distance between the terminal device 16 and each device based on the location information of the terminal device 16 and each device. For example, the device with the shortest distance to the terminal device 16 is used as the first (highest priority) device, and the device with the second shortest distance to the terminal device 16 is used as the second device. In this case, the specific unit 38 of the server 14 specifies (identifies) the collaborative functions to be used in sequence from the functions of the devices with higher priority, and the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to preferentially display the collaborative functions to be used in sequence from the functions of the devices with higher priority.

[0218] For example, Fig.23As shown, the image forming device 10 as the multifunction machine (B), the projector 12 as the projector (C), the camera (D), and the display (E) are photographed by the camera 42. In this case, the device image 70 related to the multifunction machine (B), the device image 72 related to the projector (C), the device image 116 related to the camera (D), and the device image 118 related to the display (E) are displayed on the UI unit 46 of the terminal device 16. For example, when the projector (C), the camera (D), the display (E), and the multifunction machine (B) are determined to be close to the terminal device 16 in this order, the cooperative function used in the order of the projector (C), the camera (D), the display (E), and the multifunction machine (B) is specified (recognized), and the cooperative function is preferentially displayed on the UI unit 46 of the terminal device 16.

[0219] As another example, the device related to the device image displayed at a position closer to the reference position 120 set on the device display screen 68 may be preferentially used as the starting device of the cooperative function. In this case, the terminal device 16 displays the distance between the reference position 120 and each device image (for example, Fig.23 The specific unit 38 of the server 14 sends information about the length of each arrow shown in the figure to the server 14. The specific unit 38 of the server 14 gives priority to each device associated with each device image based on the distance between the reference position 120 and each device image. For example, the device associated with the device image with the shortest distance from the reference position 120 is used as the first (highest priority) device, and the device associated with the device image with the second shortest distance from the reference position 120 is used as the second device. In this case, the specific unit 38 of the server 14 specifies (identifies) collaborative functions to be used in order from the functions of the devices with higher priority, and the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display the collaborative functions. In addition, the control unit 48 of the terminal device 16 may display a mark (for example, a circular mark) indicating the reference position 120 or an arrow indicating the distance between images on the device display screen 68, or may not display it.

[0220] The control unit 48 of the terminal device 16 may associate information indicating the priority (e.g., numbers corresponding to the order of positions close to the terminal device 16) with each device image and display it on the device display screen 68. For example, the information indicating the priority (e.g., numbers) may be displayed near each device image. Fig.23 In the example shown, a number indicating the first (highest priority) is displayed near the device image 72 related to the projector (C). The same applies to the other device images.

[0221] As described above, according to modification 4, the collaborative functions that are used more preferentially are displayed by the devices that are closer to the camera (e.g., terminal device 16). Thus, when the user desires to use the devices in order starting from the devices that are located closer to the camera (e.g., terminal device 16), information about the collaborative functions that are suitable for the user's expectations is displayed, which is convenient.

[0222] The positional relationship of each device may also be determined based on the defect relationship between device images (appearance images) related to each device. Fig.24 As shown, when the device image 70 related to the multifunction machine (B) is displayed superimposed on the device image 118 related to the display (E) (for example, the appearance image obtained by the camera 42), it is determined that the multifunction machine (B) is installed at a position closer to the terminal device 16 than the display (E). This determination can be made by the terminal device 16 or by the server 14. In this case, the specifying unit 38 of the server 14 specifies (recognizes) the cooperative function of each device in the order of the multifunction machine (B) and the display (E), and the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to preferentially display information about the cooperative function.

[0223] In addition, since the device image 70 is superimposed on the device image 118, the device image 118 is missing. In this case, the specifying unit 38 of the server 14 specifies (identifies) the display (E) related to the device image 118 by using a plurality of recognition technologies such as the marker-type AR technology or the position information AR technology, thereby specifying (identifying) the function of the display (E) or the cooperative function using the function of the display (E).

[0224] (Variant 5)

[0225] Modification 5 is described below. In modification 5, functions are assigned to the respective parts of the device, and information about the function assigned to the recognized (e.g., photographed) part and information about the coordinated function using the function assigned to the recognized (e.g., photographed) part are displayed. Modification 5 is described below in detail.

[0226] exist Fig.25, an example of a device function management table of variant example 5 is shown. The data of the device function management table of variant example 5 is stored in the server 14 as device function information 32. In the device function management table of variant example 5, as an example, the device ID, information indicating the device name (e.g., the type of device), information indicating the part name of the device (e.g., the type of part), the part ID as part identification information for identifying the part, information indicating the function assigned to the part (the function possessed by the part), and the part image ID for identifying the device part image related to the part are associated with each other. The device part image is an image representing the appearance of a part of the device obtained by shooting with a camera. Of course, a partial image schematically representing a part of the device can also be associated with the part. For example, different functions are assigned to each part of the device.

[0227] To illustrate with a specific example, the display portion of PC (A) is equipped with a screen display function, and information indicating the screen display function is associated with a partial image ID of a device partial image associated with the display portion. The screen display function is a function for displaying information on PC (A). The main body portion of PC (A) is equipped with a data storage function, and information indicating the data storage function is associated with a partial image ID of a device partial image associated with the main body portion. The data storage function is a function for storing data on PC (A).

[0228] In addition, the main body of the multifunction machine (B) is equipped with a printing function, and information indicating the printing function is associated with a partial image ID of a device partial image related to the main body. A scanning function is assigned to a reading portion of the multifunction machine (B) (for example, a portion corresponding to an original cover, an original glass, or an automatic original paper feeder of the multifunction machine (B)), and information indicating the scanning function is associated with a partial image ID of a device partial image related to the reading portion. A stapling function is assigned to a post-processing device of the multifunction machine (B), and information indicating the stapling function is associated with a partial image ID of a device partial image related to the post-processing device. The stapling function is a function of stapling outputted sheets of paper using staples.

[0229] The function assigned to a portion of the device is specified (identified), for example, by using a markerless AR technology. For example, when a portion of the device is photographed by a camera (e.g., camera 42 of terminal device 16), appearance image data representing the portion is sent from terminal device 16 to server 14. The specific unit 38 of server 14 specifies (identifies) the function associated with the appearance image data in the device function management table. Thus, the function assigned to the photographed portion is specified (identified). For example, when the main body of a multifunction machine (B) is photographed by camera 42, appearance image data representing the main body of the multifunction machine (B) is sent from terminal device 16 to server 14. The specific unit 38 of server 14 specifies the printing function associated with the appearance image data in the device function management table. Thus, the function assigned to the main body of the multifunction machine (B) is specified as the printing function.

[0230] Of course, the functions assigned to the parts of the device can also be specified (identified) by using marker-type AR technology. For example, each part of the device is provided with a mark such as a two-dimensional barcode that encodes the partial identification information (e.g., partial ID) used to identify the part. When the mark set on the part is photographed by a camera and the marker-type AR technology is applied, the partial identification information (e.g., partial ID) of the part is obtained. The application of marker-type AR technology can be performed on the terminal device 16 or on the server 14. When the partial identification information is obtained in this way, the specific unit 38 of the server 14 specifies (identifies) the function associated with the partial identification information (e.g., partial ID) in the device function management table.

[0231] exist Fig.26 , an example of a collaborative function management table of modification 5 is shown. Data of the collaborative function management table of modification 5 is stored in the server 14 as collaborative function information 34. The collaborative function management table of modification 5 is information indicating a collaborative function that utilizes functions possessed by a plurality of parts. In the collaborative function management table, as an example, information indicating a combination of parts of an apparatus, information indicating a combination of part IDs, and information indicating a collaborative function that utilizes functions possessed by a plurality of parts included in the combination are associated with each other. Of course, in the collaborative function management table, information indicating a combination of a part of an apparatus and the entirety of the apparatus, and information indicating a collaborative function that utilizes functions possessed by a part of the apparatus and functions possessed by the entirety of the apparatus may be associated.

[0232] To explain by taking a specific example, a printing function as a cooperative function is assigned to the combination of the display unit of PC (A) and the main body of the multifunction machine (B), and information indicating the printing function as a cooperative function is associated with information indicating the combination of the partial ID of the display unit of PC (A) and the partial ID of the main body of the multifunction machine (B). The printing function as a cooperative function is, for example, a function of sending data stored in PC (A) to the multifunction machine (B) and printing the data by the multifunction machine (B).

[0233] In addition, a printing function as a cooperative function is allocated to the combination of the main body of the multifunction machine (B) and the main body of the projector (C), and information indicating the printing function as a cooperative function is associated with information indicating the combination of the partial ID of the main body of the multifunction machine (B) and the partial ID of the main body of the projector (C). The printing function as a cooperative function is, for example, a function of sending data projected by the projector (C) to the multifunction machine (B) and printing the data by the multifunction machine (B).

[0234] In addition, a scanning and projection function as a cooperative function is allocated to the combination of the reading unit of the multifunction machine (B) and the main body of the projector (C), and information indicating the scanning and projection function as a cooperative function is associated with information indicating the combination of the partial ID of the reading unit of the multifunction machine (B) and the partial ID of the main body of the projector (C). The scanning and projection function as a cooperative function is, for example, a function of sending data generated by scanning by the multifunction machine (B) to the projector (C) and projecting the data by the projector (C).

[0235] Furthermore, the cooperative function may be a function utilizing functions of multiple parts included in the same device, or a function utilizing functions of parts of multiple different devices, or a function utilizing functions of three or more parts.

[0236] For example, when multiple parts of a device (e.g., multiple parts of multiple different devices, or multiple parts of the same device) are specified (identified) using the marker-type AR technology or the markerless AR technology, the specific unit 38 of the server 14 specifies (identifies) the collaborative function associated with the combination of the identified multiple parts in the collaborative function management table. Thus, the collaborative function using the functions possessed by the identified (e.g., photographed) multiple parts is specified (identified). For example, when the main body of the multifunction machine (B) and the main body of the projector (C) are photographed by the camera 42 of the terminal device 16 and the main body of the multifunction machine (B) and the main body of the projector (C) are identified, the specific unit 38 of the server 14 specifies the printing function or the like as a collaborative function associated with the combination of the main body of the multifunction machine (B) and the main body of the projector (C) in the collaborative function management table.

[0237] In the fifth modification, similar to the above-mentioned embodiment, information about the function to be used alone or information about the cooperative function may be displayed on the UI unit 46 of the terminal device 16. For example, when the user gives a display instruction to use the function alone, or when only one part in the device is recognized, the control unit 48 of the terminal device 16 may cause the UI unit 46 of the terminal device 16 to display information about the function of the recognized part. On the other hand, when multiple parts of multiple devices are recognized, or multiple parts in the same device are recognized, or a display instruction to use the cooperative function is given after multiple parts are recognized, the control unit 48 of the terminal device 16 may cause the UI unit 46 of the terminal device 16 to display information about the cooperative function using the functions of the recognized multiple parts.

[0238] The following is a specific example to explain the modification 5. For example, the main body of the multifunction machine (B) as the image forming device 10 is photographed by the camera 42 of the terminal device 16, and the main body of the multifunction machine (B) is identified (recognized) by the marker-type AR technology or the marker-free AR technology. In this case, for example, Fig. 27 As shown, a device display screen 52 is displayed on the UI unit 46 of the terminal device 16, and a device portion image 121 related to the main body of the multifunction device (B) is displayed on the device display screen 52. When the reading unit or the post-processing device of the multifunction device (B) is not recognized, the device portion image 122 related to the reading unit and the device portion image 124 related to the post-processing device are not displayed on the device display screen 52.

[0239] For example, when the user gives a display instruction to use a function independently, or when only one part is photographed instead of multiple parts, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the function of the main unit of the multifunction device (B) as information about the function to be used independently. The identification process of the function of the main unit of the multifunction device (B) can be performed by the terminal device 16 or by the server 14. Fig. 27 In the example shown, information about the functions of the main body of the multifunction device (B) (e.g., the names of each function) is displayed in a display area 126 provided in the device display screen 52. When the information about the function displayed in the display area 126 is designated by the user and an instruction to execute the function is given, the designated function is executed. The display area 126 is arranged near the device part image 121 related to the main body of the multifunction device (B). Of course, the control unit 48 of the terminal device 16 may cause the UI unit 46 of the terminal device 16 to display a function display screen different from the device display screen 52, and display information about the functions of the main body of the multifunction device (B) in the function display screen.

[0240] In addition, when the UI unit 46 of the terminal device 16 displays a device image representing the entire device, information about the function corresponding to the part specified by the user in the device image can also be displayed on the UI unit 46 of the terminal device 16 as information about the function to be used separately. For example, different functions are assigned to each position in the device image, and information about each function corresponding to each position is stored in the server 14 as device function information 32. Information representing the position specified by the user in the device image is sent from the terminal device 16 to the server 14, and the specifying unit 38 of the server 14 specifies (identifies) the function corresponding to the specified position. Information about the function is displayed on the UI unit 46 of the terminal device 16 as a function to be used separately. As another example, the device image representing the entire device can also be composed of a plurality of device part images representing a plurality of parts constituting the device. As described above, since the function is assigned to the device part image, the function corresponding to the device part image specified by the user is specified, and information about the function is displayed on the UI unit 46 of the terminal device 16.

[0241] When the entire device is captured, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about one or more functions of the device. When a portion of the device is captured, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the functions assigned to the portion. For example, when the entire multifunction device (B) is captured, Fig. 9 As shown in FIG. 1 , information about the functions of the multifunction device (B) is displayed. When the main body of the multifunction device (B) is photographed, as shown in FIG. Fig. 27 As shown, information about the functions of the main body can also be displayed.

[0242] In addition, the camera 42 of the terminal device 16 captures the main body of the multifunction machine (B) as the image forming device 10 and the main body of the projector (C) as the projector 12, and the main body of the multifunction machine (B) and the main body of the projector (C) are specified (identified) by the marker-type AR technology or the markerless AR technology. In this case, for example, Fig.28 As shown, a device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and a device portion image 121 related to the main body of the multifunction peripheral (B) and a device portion image 128 related to the main body of the projector (C) are displayed on the device display screen 68.

[0243] For example, when a user gives an instruction to display a collaborative function, or when multiple parts are captured within the capture time, or when multiple parts are captured before an instruction to execute a function assigned to one part is given, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about a collaborative function using the function of the main body of the multifunction device (B) and the function of the main body of the projector (C). The recognition process of the collaborative function may be performed by the terminal device 16 or by the server 14. Fig.28 In the example shown, information about the cooperative function (e.g., the name of the cooperative function) is displayed in the display area 130 provided in the device display screen 68. When the user specifies the information about the function displayed in the display area 130 and gives an instruction to execute the cooperative function, the specified cooperative function is executed. Fig.28 In the example shown, information about the functions of the main body of the multifunction machine (B) is displayed in the display area 132, and information about the functions of the main body of the projector (C) is displayed in the display area 134. The display area 132 is arranged near the device part image 121 related to the main body of the multifunction machine (B), and the display area 134 is arranged near the device part image 128 related to the main body of the projector (C). Of course, the control unit 48 of the terminal device 16 can also make the UI unit 46 of the terminal device 16 display a function display screen different from the device display screen 68, and display information about the cooperative function or information about the independent use function in the function display screen. In addition, the display areas 132 and 134 may not be displayed, and the display and non-display may be switched according to the user's instructions.

[0244] In addition, the control unit 48 of the terminal device 16 may also display the name of the function of the device related to the device part image (such as printer, scanner, etc.) on the device display screen 68. In this way, the user is provided with information that makes it easy to understand which function is assigned to each part. Of course, the name of the function may not be displayed.

[0245] As described above, according to variant example 5, when a part of the device is identified, information about the functions of the part is displayed, and when multiple parts are identified, information about collaborative functions using the functions of the multiple parts is displayed. For example, by identifying (e.g. photographing) the part of the target to be used in the device, information about the functions of the part is displayed. Therefore, compared with the case where information about all functions of the device is displayed, it is less likely to display information that is unnecessary for the user, so it is more convenient. Similarly, by identifying (e.g. photographing) multiple parts, information about collaborative functions using the functions of the multiple parts is displayed. Therefore, it is less likely to display information that is unnecessary for the user.

[0246] (Variant 6)

[0247] Next, modification example 6 is described. Modification example 6 is a combination of modification examples 1 and 5. For example, the main body of the multifunction device (B) and the main body of the projector (C) are photographed. In this case, for example, Fig.29 As shown, a device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and a device portion image 121 related to the main body of the multifunction peripheral (B) and a device portion image 128 related to the main body of the projector (C) are displayed on the device display screen 68. Similar to modification example 1, a starting point display area 86 and a collaborative target display area 88 are set on the device display screen 68. In this case, a portion of the device related to the device portion image displayed in the starting point display area 86 is designated as the starting point, and a portion of the device related to the device portion image displayed in the collaborative target display area 88 is designated as the collaborative target. Fig.29 In the example shown, since the device part image 121 is displayed in the starting point display area 86, the main body of the multifunction machine (B) is specified as the starting point, and since the device part image 128 is displayed in the cooperation target display area 88, the main body of the projector (C) is specified as the cooperation target. In this case, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the cooperation function that uses the main body of the multifunction machine (B) as the starting point and uses the main body of the projector (C) as the cooperation target. For example, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the cooperation function that first uses the main body of the multifunction machine (B) as the starting point and then uses the main body of the projector (C) as the cooperation target. In other words, the control unit 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information about the cooperation function that moves data from the main body of the multifunction machine (B) as the starting point to the projector (C) as the cooperation target.

[0248] In addition, the display position of the device portion image may be changed as in Modification 1. For example, when the user moves the device portion image 121 to the cooperative target display area 88, the main body of the multifunction machine (B) associated with the device portion image 121 is treated as part of the cooperative target device. Similarly, when the user moves the device portion image 128 to the starting point display area 86, the main body of the projector (C) associated with the device portion image 128 is treated as part of the starting point device.

[0249] As in the first modification, the terminal device 16 transmits the data of the device portion image displayed in the starting point display area 86 as the image data of the starting point device portion to the server 14, and transmits the data of the device portion image displayed in the cooperation target display area 88 as the image data of the cooperation target device portion to the server 14. The specifying unit 38 of the server 14 sets the device portion related to the image data of the starting point device portion as the starting point, and sets the device portion related to the image data of the cooperation target device portion as the cooperation target. The server 14 stores data of the cooperation function management table of the sixth modification, in which the identification information of the starting point device portion, the identification information of the cooperation target device portion, and the cooperation function identification information are associated. The specifying unit 38 of the server 14 specifies the cooperation function identification information in which the identification information of the starting point device portion and the identification information of the cooperation target device portion are associated in the cooperation function management table. When the cooperation function identification information is specified in this way, the information about the specified cooperation function is displayed on the UI unit 46 of the terminal device 16.

[0250] In addition, the device partial image representing a part may be separated from the device image representing the entire device, and the display position of the device partial image may be changed. Fig.30 As shown, the device image 70 representing the entire multifunction device (B) is displayed in the starting point display area 86. In this case, as shown by the arrow 136, when the user moves the device portion image 122 included in the device image 70 from the starting point display area 86 to the cooperation target display area 88, the function of the portion related to the device portion image 122 (for example, the scanning function of the reading unit) is set as the function of the cooperation target. Similarly, when the user moves the device portion image included in the device image displayed in the cooperation target display area 88 to the starting point display area 86, the function of the portion related to the device portion image is set as the function of the starting point.

[0251] As described above, according to Modification 6, since the cooperative function is changed according to the display position of the device partial image, the operability of changing the cooperative function can be improved. In addition, the cooperative function can be changed by a simple operation.

[0252] (Variant 7)

[0253] Below, refer to Fig.31 and Fig.32 Modification 7 is described below. Fig.31 and Fig.32, a device display screen 68 of Modification 7 is shown. In Modification 7, as in Modification 1 and the like, a starting point display area 86 and a cooperation target display area 88 are set on the device display screen 68, and a plurality of device images (e.g., device images 72, 116, 118) are displayed in the same cooperation target display area 88. In addition, a device image 70 is displayed in the starting point display area 86. In this case, the order of use of each device related to each device image displayed in the cooperation target display area 88 (the order of cooperation) may be specified by the user, or may be automatically set based on the relationship between the display positions of each device image.

[0254] When the order of cooperation is set according to the user's instructions, the user specifies, for example, in order the device images displayed in the cooperation target display area 88. For example, when the user specifies the device images in the order of the device image 72 associated with the projector (C), the device image 118 associated with the display (E), and the device image 116 associated with the camera (D), the multifunction machine (B) associated with the device image 70 displayed in the starting point display area 86 is used first, and then the devices of the cooperation target are cooperated (used) in the specified order.

[0255] When the coordination order is automatically set, the coordination order may be assigned to each device in the order of proximity to the device image 70 displayed in the starting point display area 86. Fig.31 In the example shown, the device image 72, the device image 118, and the device image 116 are displayed in the cooperation target display area 88 in the order of being closest to the device image 70. In this case, the multifunction device (B) is used first, and then the devices are cooperated (used) in the order of the projector (C), the display (E), and the camera (D).

[0256] As another example, the device associated with the device image displayed at the position closest to the device image 70 displayed in the starting point display area 86 may be set as the first coordination target device, and the coordination order of the second and subsequent coordination target devices may be determined based on the positional relationship between the device images displayed in the same coordination target display area 88. Fig.32As shown, the device associated with the device image 72 displayed at the position closest to the device image 70 is set as the first collaborative target device. Next, in the collaborative target display area 88, the camera (D) associated with the device image 116 displayed at the position closest to the device image 72 is set as the second collaborative target device. Next, in the collaborative target display area 88, with device images other than the device images associated with the device that has been set as the collaborative target (for example, the device image 72) as the target, the display (E) associated with the device image 118 displayed at the position closest to the device image 116 is set as the third collaborative target device. The same applies to the subsequent collaborative order.

[0257] As described above, when multiple device images are displayed in the same collaborative target display area 88, the collaborative order is automatically set based on the relationship between the display positions of the device images, which saves the user's time and effort and is more convenient. In addition, when multiple device partial images are displayed in the same collaborative target display area 88, the collaborative order of each device portion is determined by the same processing as above. Of course, the device partial image can also be displayed in the starting point display area 86.

[0258] In the above-mentioned embodiments and modifications, the cooperative function may be a function utilizing a combination of the entire devices, a function utilizing a combination of parts of the devices, or a function utilizing a combination of the entire device and parts of the device.

[0259] In addition, the functions of each variant can also be switched according to the user's instructions. For example, when the user instructs to execute the function of variant 1, the function of variant 1 is executed, thereby setting the starting point display area and the cooperative target display area on the device display screen. The same is true for other variants. When the user instructs to execute the function of a variant, the function of the variant is executed.

[0260] In the following, in the above-mentioned embodiments and variations, the processing when the function of the device is executed is described. As an example, the processing when the collaborative function is executed is described. In this case, a connection request is sent from the terminal device 16 to the collaborative target device, and the terminal device 16 is connected to the collaborative target device. Fig.33 This connection processing will be described. Fig.33 is a timing diagram showing this process.

[0261] First, the user instructs the start of an application (program) for executing the cooperative function using the terminal device 16. The control unit 48 of the terminal device 16 starts the application according to the instruction (S01). The application may be pre-stored in the storage unit 44 of the terminal device 16 or downloaded from the server 14 or the like.

[0262] Next, by applying the marker-type AR technology, the marker-free AR technology or the location information AR technology, the device of the collaborative target is identified (S02). Of course, the device of the collaborative target can also be identified by technology other than AR technology. In the case of applying the marker-type AR technology or the marker-free AR technology, the user photographs the device of the collaborative target through the camera 42 of the terminal device 16. As an example, in the case of using the image forming device 10 (multifunction machine (B)) and the projector 12 (projector (C)) as collaborative devices, the user photographs the image forming device 10 and the projector 12 through the camera 42. The device identification information of each of the image forming device 10 and the projector 12 is obtained, and the image forming device 10 and the projector 12 are identified as the device of the collaborative target. In the case of applying the location information AR technology, the location information of each of the image forming device 10 and the projector 12 is obtained, and the device identification information of each of the image forming device 10 and the projector 12 is specified based on the location information, thereby identifying the image forming device 10 and the projector 12.

[0263] As described above, when multiple devices are identified, or an instruction to display a collaborative function is given, or when other devices are identified before an instruction to execute a function that is independently utilized by one device is given, a collaborative function that utilizes the functions of multiple devices is identified. Information about the identified collaborative function is displayed on the UI unit 46 of the terminal device 16 (S03). The collaborative function identification process can be performed by the server 14 or by the terminal device 16.

[0264] Next, in the terminal device 16, when the user specifies the collaborative function of the target to be executed, the terminal device 16 sends information indicating a connection request to the device of the collaborative target that executes the collaborative function (e.g., the image forming device 10 and the projector 12) (S04). For example, in the case where the server 14 stores address information indicating the address of the device of the collaborative target, the terminal device 16 obtains the address information of the device of the collaborative target from the server 14. In the case where the device identification information includes the address information, the terminal device 16 may also obtain the respective address information from the device identification information of the device of the collaborative target. As another example, the address information of the device of the collaborative target may also be stored in the terminal device 16. Of course, the terminal device 16 may also obtain the address information of the device of the collaborative target by other methods. The terminal device 16 uses the address information of the device of the collaborative target (e.g., the image forming device 10 and the projector 12) to send information indicating a connection request to the device of the collaborative target (e.g., the image forming device 10 and the projector 12).

[0265] The image forming device 10 and the projector 12 that receive the information indicating the connection request permit or disallow the connection with the terminal device 16 (S05). For example, in the case where the image forming device 10 and the projector 12 correspond to devices whose connection is not permitted, or in the case where the number of devices requesting connection exceeds the upper limit, the connection is not permitted. In addition, in the case where the connection from the terminal device 16 is permitted, the change operation may be prohibited in a manner that does not change the inherent setting information of the image forming device 10 and the projector 12 from the terminal device 16. For example, changes in the color parameters of the image forming device 10, the setting time when transitioning to power saving, etc. may also be prohibited. Thus, the security of the device with respect to the collaborative target is improved. As another example, in the case of making the device collaborative, the change of the setting information may be restricted compared to the case where the device is not collaborative and used alone. For example, compared to the case where the device is used alone, changes in fewer setting items may be permitted. In addition, the reading of other users' personal information such as the operation history may also be prohibited. Thus, the security of the user's personal information can be improved.

[0266] Result information indicating whether the connection is permitted or not is transmitted from the image forming device 10 and the projector 12 to the terminal device 16 (S06). If the connection to the image forming device 10 and the projector 12 is permitted, communication is established between the terminal device 16 and the image forming device 10 and the projector 12.

[0267] Next, the user instructs execution of the cooperative function using the terminal device 16 (S07). In response to the instruction, execution instruction information indicating the execution instruction of the cooperative function is sent from the terminal device 16 to the image forming device 10 and the projector 12 (S08). The execution instruction information sent to the image forming device 10 includes information indicating the processing to be executed by the image forming device 10 (e.g., job information), and the execution instruction information sent to the projector 12 includes information indicating the processing to be executed by the projector 12 (e.g., job information).

[0268] The image forming device 10 and the projector 12 that have received the execution instruction information execute the function according to the execution instruction information (S09). For example, when the cooperative function includes a process of sending and receiving data between the image forming device 10 and the projector 12, such as a function of transmitting scan data from the image forming device 10 (multifunctional peripheral (B)) to the projector 12 (PC (A)) and projecting the data by the projector 12, communication is established between the image forming device 10 and the projector 12. In this case, for example, the execution instruction information sent to the image forming device 10 includes the address information of the projector 12, and the execution instruction information sent to the projector 12 includes the address information of the image forming device 10, and the communication is established between the image forming device 10 and the projector 12 using these address information.

[0269] When the execution of the cooperative function ends, information indicating the end of the execution of the cooperative function is sent from the image forming device 10 and the projector 12 to the terminal device 16 (S10). Information indicating the end of the execution of the cooperative function is displayed on the UI unit 46 of the terminal device 16 (S11). In addition, when the information indicating the end of the execution is not displayed even after a preset time has passed since the time when the execution instruction was given, the control unit 48 of the terminal device 16 can cause the UI unit 46 to display information indicating an error, and send the execution instruction information or the information indicating the connection request to the image forming device 10 and the projector 12 again.

[0270] Next, the user confirms whether to cancel the collaborative state between the image forming device 10 and the projector 12 (S12), and executes processing corresponding to whether to cancel or not (S13). When canceling the collaborative state, the user gives a cancellation instruction using the terminal device 16. As a result, the communication between the terminal device 16 and the image forming device 10 and the projector 12 is canceled. Similarly, the communication between the image forming device 10 and the projector 12 is also canceled. In the case where the collaborative state is not canceled, the execution instruction can continue to be given.

[0271] In addition, additional devices may be added as cooperative targets. For example, device identification information of a third device may be obtained to identify a cooperative function performed by coordinating three devices including the image forming device 10 and the projector 12. In addition, information indicating that the image forming device 10 and the projector 12 have been identified as cooperative functions is stored in the server 14 or the terminal device 16.

[0272] The device identification information of the device that represents the collaborative target and the collaborative function identification information that represents the executed collaborative function may also be stored in the terminal device 16 or the server 14. For example, the user account information (user identification information) of the user who uses the terminal device 16 may be obtained, and historical information representing the association of the user account information, the device identification information of the device that represents the collaborative target, and the collaborative function identification information that represents the executed collaborative function may be created and stored in the terminal device 16 or the server 14. The historical information may be created by the terminal device 16 or the server 14. By referring to the historical information, it is possible to identify which collaborative function is executed using which device group.

[0273] In addition, the device of the collaborative target (such as the image forming device 10 and the projector 12) can also store the user account information of the user who requested the connection and the terminal identification information of the terminal device 16 indicating the connection request source as historical information. By referring to the historical information, the user who used the device is identified. For example, when a user used a specific device when the device was damaged, or when charging for consumables, etc., the historical information can be fully utilized to identify the user. The historical information can be stored in the server 14 or the terminal device 16, and can also be stored in other devices.

[0274] The user account information is stored in advance in the storage unit 44 of the terminal device 16, for example. The control unit 48 of the terminal device 16 functions as an example of a user identification unit, reads the user account information of the user from the storage unit 44, and identifies the user using the terminal device 16. In the case where the storage unit 44 stores the user account information of multiple users, the user specifies his own user account information using the terminal device 16. Thus, the user account information of the user is read in, and the user is identified. As another example, the control unit 48 of the terminal device 16 can also identify the user by reading the user account information of the user logged in to the terminal device 16. As another example, in the case where only one user account information is stored in the same terminal device 16, the control unit 48 of the terminal device 16 can also identify the user by reading the user account information. In addition, in the case where the user account is not set and the user account information is not created, the initial setting is performed, thereby creating the user account information.

[0275] It is also possible to manage the usage history of the collaborative function for each user, and display information indicating the collaborative function used by the user in the past represented by the read user account information on the UI unit 46 of the terminal device 16. The information indicating the usage history may be stored in the terminal device 16 or in the server 14. In addition, it is also possible to display information indicating the collaborative function used at a frequency higher than a preset usage frequency. By providing such a shortcut function, it is possible to save time and effort of the user's operation on the collaborative function.

[0276] When the individual use function is executed, information indicating an execution instruction of the individual use function is transmitted from the terminal device 16 to the device for executing the individual use function. The device executes the individual use function in accordance with the execution instruction.

[0277] The above-mentioned image forming device 10, server 14 and terminal device 16 are respectively implemented by the cooperation of hardware and software as an example. Specifically, the image forming device 10, server 14 and terminal device 16 are respectively equipped with one or more processors such as a CPU not shown in the figure. The one or more processors read and execute the program stored in the storage device not shown in the figure to realize the functions of the image forming device 10, server 14 and terminal device 16. The above-mentioned program is stored in the storage device via a recording medium such as a CD or DVD, or via a communication path such as a network. As another example, each part of the image forming device 10, server 14 and terminal device 16 can also be implemented by hardware resources such as a processor or a circuit. Devices such as a memory can also be used in this implementation. As another example, each part of the image forming device 10, server 14 and terminal device 16 can also be implemented by a DSP (Digital Signal Processor) or an FPGA (Field Programmable Gate Array).

Claims

1. An information processing device comprising a processor, wherein: When a second device different from the first device is photographed after the first device is photographed, the processor controls display of a cooperative function executable by combining a first function of the first device and a second function of the second device, the first function is different from the second function, The processor controls display of the cooperative function using the first function and the second function in response to capturing the second device after the image of the first device disappears from the display.

2. The information processing device according to claim 1, wherein: The first device comprises a plurality of parts having different functions, The processor performs the following processing: When a first part of the first device is photographed when photographing the first device, a function of the first part is determined as the first function.

3. The information processing device according to claim 2, wherein: The second device comprises a plurality of parts having different functions, The processor performs the following processing: When a second part of the second device is photographed when photographing the second device, a function of the second part is determined as the second function.

4. The information processing device according to claim 3, wherein: Based on the display positions of a partial image within a first image related to the first function and a partial image within a second image related to the second function, the display of a collaborative function that can be executed by combining the first function related to the partial image within the first image and the second function related to the partial image within the second image is controlled.

5. The information processing device according to claim 1, wherein: The first device and the second device include a plurality of parts having different functions, Each of the partial images related to the plurality of portions is associated with a different function.

6. The information processing device according to claim 1, wherein: When the first device and the second device are captured, the device used as the starting point of the cooperative function is changed according to the positional relationship with the imaging device that captured the image.

7. The information processing device according to claim 1, wherein: In the case where the first device and the second device are photographed, a first image related to the first device and a second image related to the second device are displayed on a display unit, The processor controls display of the cooperative function by changing a start point of a device utilized in the cooperative function and a next cooperative target according to display positions of the first image and the second image in the display unit.

8. The information processing device according to claim 1, wherein: In the case where the first device and the second device are photographed, a first image related to the first device and a second image related to the second device are displayed on a display unit, The processor switches display of the cooperative function according to display positions of the first image and the second image in the display unit.

9. The information processing device according to any one of claims 1 to 4, wherein: The cooperative function is a function that cannot be used before the cooperation.

10. A recording medium having recorded thereon an information processing program for causing a computer to function as a control unit. When a second device different from the first device is photographed after the first device is photographed, the control unit controls display of a cooperative function executable by combining a first function of the first device and a second function of the second device, the first function is different from the second function, In response to the second device being captured after the image of the first device disappears from the display, display of the cooperative function using the first function and the second function is controlled.

11. An information processing method, wherein: When a second device different from the first device is photographed after the first device is photographed, controlling display of a cooperative function executable by combining a first function of the first device and a second function of the second device, the first function is different from the second function, In response to the second device being captured after the image of the first device disappears from the display, display of the cooperative function using the first function and the second function is controlled.

12. An information processing device comprising a processor, wherein: The processor performs the following processing: switching display of a cooperative function executable by combining a function related to the partial image in the first image and a function related to the partial image in the second image according to display positions of a partial image in a first image related to the first device and a partial image in a second image related to the second device; The processor controls the display of the collaborative function utilizing the function associated with the partial image within the first image and the function associated with the partial image within the second image in response to capturing the second image associated with the second device after the first image associated with the first device disappears from the display.

13. The information processing device according to claim 12, wherein: The first image is an image obtained by photographing the first device; The second image is an image obtained by photographing the second device.

14. The information processing device according to claim 13, wherein: The processor performs the following processing: identifying the first device according to the first image, identifying the second device according to the second image, When the first device and the second device are identified by the identification, the display of the cooperative function is switched according to the display position.

15. An information processing device comprising a processor, wherein: the processor controlling display of a cooperative function executable by combining a function related to the partial image in the first image and a function related to the partial image in the second image, based on display positions of a partial image in a first image related to a first function and a partial image in a second image related to a second function, The processor controls the display of the collaborative function utilizing the function associated with a partial image within the first image and the function associated with a partial image within the second image in response to capturing the second image associated with the second function after the first image associated with the first function disappears from the display.

16. The information processing device according to any one of claims 12 to 15, wherein: Each partial image is associated with a different function.

17. The information processing device according to any one of claims 12 to 15, wherein: The cooperative function is a function that cannot be used before the cooperation.

18. A recording medium having a program recorded thereon, the program causing a computer to function as a control unit, the control unit switches display of a cooperative function executable by combining a function related to the partial image in the first image and a function related to the partial image in the second image, according to display positions of a partial image in a first image related to the first device and a partial image in a second image related to the second device, The control unit controls the display of the collaborative function utilizing the function associated with the partial image within the first image and the function associated with the partial image within the second image in response to the second image associated with the second device being captured after the first image associated with the first device disappears from the display.

19. A recording medium having a program recorded thereon, the program causing a computer to function as a control unit, the control unit controls display of a cooperative function executable by combining a function related to the partial image in the first image and a function related to the partial image in the second image, based on display positions of a partial image in a first image related to a first function and a partial image in a second image related to a second function, The control unit controls the display of the collaborative function utilizing the function related to the partial image within the first image and the function related to the partial image within the second image in response to the second image related to the second function being captured after the first image related to the first function disappears from the display.

20. An information processing method, wherein: switching display of a cooperative function executable by combining a function related to the partial image in the first image and a function related to the partial image in the second image according to display positions of a partial image in a first image related to the first device and a partial image in a second image related to the second device, In response to capturing the second image associated with the second device after the first image associated with the first device disappears from the display, controlling the display of the collaborative function utilizing the function associated with the partial image within the first image and the function associated with the partial image within the second image.

21. An information processing method, wherein: controlling display of a cooperative function executable by combining a function related to the partial image in the first image and a function related to the partial image in the second image, according to display positions of a partial image in a first image related to a first function and a partial image in a second image related to a second function, In response to capturing the second image related to the second function after the first image related to the first function disappears from the display, the display of the collaborative function utilizing the function related to the partial image within the first image and the function related to the partial image within the second image is controlled.

Images