Information processing apparatus, information processing system, device management method, and program

Abstract

To configure settings for an electronic device using information used in another electronic device.SOLUTION: A management apparatus configured to communicate with a plurality of electronic devices and an imaging apparatus over a communication link includes: a state storage unit; a state management unit which includes an object state management unit, a device state management unit, and an output information determination unit; an image receiving unit which manages the electronic devices and a state of an object, decides content of an operation instruction to be transmitted to the electronic devices, updates, based on a received image, a registered object list stored in the state storage unit, updates, based on received device operation information, a registered device list and the registered object list, and transmits the received image to the state management unit; an operation information receiving unit which receives the device operation information from the electronic devices and transmits the received device operation information to the state management unit; an output information determination unit which determines content of an operation instruction to be transmitted to the electronic devices on the basis of the stored state management information, and transmits the determined operation instruction to an operation instruction unit; and the operation instruction unit which transmits the operation instruction received from the state management unit to the electronic devices.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, a device management method, and a program. [Background technology]

[0002] In cloud computing, there is a technology that realizes single sign-on by linking software. For example, Patent Document 1 discloses an invention that uses a common authentication infrastructure across multiple services, allowing a user authenticated in one service to omit authentication in other services.

[0003] In authentication using hardware such as electronic devices, there is a technology that associates the authentication process of each piece of hardware. For example, Patent Document 2 discloses an invention that links a system that manages entry and exit to an area where confidential information is handled with electronic devices in the area, thereby not allowing authentication on the electronic devices unless the user has been authenticated by the entry and exit management system.

[0004] It is difficult to determine the identity of a user when linking hardware devices. The invention disclosed in Patent Document 2 uses the authentication status of an access control system for authentication on an electronic device, but each user must perform authentication operations separately. For example, if it is possible to determine the identity of a user who has performed authentication on a first electronic device and a user who is near a second electronic device, the second electronic device can perform authentication using the authentication information entered by the user on the first electronic device. Summary of the Invention [Problem to be solved by the invention]

[0005] An embodiment of the present invention aims to configure a second electronic device using information used in a first electronic device. [Means for solving the problem]

[0006] In order to solve the above problem, an information processing device that is one embodiment of the present invention is an information processing device that can communicate with a first electronic device, a second electronic device, and a photographing device via a network, and includes: a first image receiving unit that receives a first image of a first user using the first electronic device from the photographing device; a first user identification unit that identifies the first user using the first electronic device based on usage information of the first electronic device and the first image; an input information receiving unit that receives from the first electronic device authentication information entered on the first electronic device by the first user identified by the first user identification unit; a second image receiving unit that receives a second image of a second user present at an operable position of the second electronic device from the photographing device; a second user identification unit that identifies the second user present at an operable position of the second electronic device from the second image; and an operation instruction unit that transmits authentication information or screen setting information in the authentication information to the second electronic device based on information on the first user identified by the first user identification unit and the second user identified by the second user identification unit and the authentication information. [Effects of the Invention]

[0007] According to one embodiment of the present invention, the second electronic device can be configured using information used in the first electronic device. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of a device management system according to an embodiment. [Figure 2] FIG. 2 illustrates an example of a hardware configuration of a computer according to an embodiment. [Figure 3] FIG. 1 is a diagram illustrating an example of a hardware configuration of an omnidirectional imaging device according to an embodiment. [Figure 4] FIG. 2 illustrates an example of a hardware configuration of an MFP according to an embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of an electronic whiteboard according to an embodiment. [Figure 6]FIG. 1 illustrates an example of a functional configuration of a device management system according to an embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of state management information according to an embodiment. [Figure 8] FIG. 2 is a diagram illustrating an example of a first basic flowchart according to an embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of a second basic flowchart according to an embodiment. [Figure 10] FIG. 1 illustrates an example of a device management method according to an embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of an object state update process according to an embodiment. [Figure 12] FIG. 10 is a diagram illustrating an example of a detected object list creation process according to an embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of a registered object list update process according to an embodiment. [Figure 14] FIG. 10 is a diagram illustrating an example of a registered object list update process according to an embodiment. [Figure 15] FIG. 10 is a diagram illustrating a first example of a group ID assignment process according to an embodiment. [Figure 16] FIG. 10 is a diagram illustrating a second example of a group ID assignment process according to an embodiment. [Figure 17] FIG. 10 illustrates an example of a device status update process according to an embodiment. [Figure 18] FIG. 10 is a diagram illustrating an example of an output information determination process according to an embodiment. [Figure 19] FIG. 10 is a diagram illustrating an example of an output condition determination process according to an embodiment. [Figure 20] FIG. 10 is a diagram illustrating an example of device output information according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, components having the same functions are designated by the same reference numerals, and duplicated explanations will be omitted.

[0010] [Embodiment] One embodiment of the present invention is a device management system that manages multiple electronic devices. The device management system in this embodiment realizes cooperation between electronic devices by instructing other electronic devices to perform operations using information input into one electronic device.

[0011] In the past, when linking hardware devices, the identity of a user was determined by whether the user possessed the same device (for example, an integrated circuit (IC) card or a mobile information terminal). As a result, even when linking hardware devices, authentication operations were required for each piece of hardware. With single sign-on through software linking, when a user authenticated in one service accesses another service, the operation is performed from the same device, making it easy to determine the identity of the user.

[0012] In the device management system of this embodiment, a management device tracks a person moving between electronic devices based on images captured by the management device of a space where multiple electronic devices are installed, allowing the management device to determine whether a user operating a certain electronic device is the same as a user near another electronic device.

[0013] If the identity of a user can be determined, an operation using information entered on one electronic device can be executed on another electronic device. For example, if an electronic device in the vicinity of the user performs authentication using authentication information entered by the user on another electronic device, single sign-on through hardware linkage can be realized.

[0014] <Overall configuration of equipment management system> First, the overall configuration of the device management system according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the overall configuration of the device management system according to this embodiment.

[0015] 1, the device management system 1 in this embodiment includes a management device 10, one or more monitoring devices 20, and multiple electronic devices 30. The management device 10, the monitoring devices 20, and the electronic devices 30 are each connected to a communication network N1.

[0016] The communication network N1 is configured so that the devices connected thereto can communicate with each other. The communication network N1 is constructed by a wired communication network such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

[0017] The communication network N1 may include not only wired communication, but also wireless communication networks such as wireless LAN or short-range wireless communication, or mobile communication networks such as WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), or 5G (5th Generation).

[0018] The monitoring device 20 and the electronic device 30 are installed in a managed space R1. The managed space R1 may be a single space such as a room in a building, or multiple spaces connected by accessible spaces such as doors or corridors.

[0019] An example of the managed space R1 is a conference room or work room in an office. Another example of the managed space R1 is a sales floor or back yard in a small store such as a convenience store. Another example of the managed space R1 is a hotel lobby or guest room. Another example of the managed space R1 is an airport, a bookstore, or a factory. The managed space R1 is not limited to these, and may be any space in which electronic devices that can be operated by multiple users are installed.

[0020] The management device 10 is an information processing device such as a PC (Personal Computer), a workstation, or a server that manages the electronic device 30. The management device 10 transmits operation instructions to the electronic device 30 based on images acquired by the monitoring device 20. An example of the management device 10 is a computer.

[0021] The monitoring device 20 is an electronic device that acquires images including the vicinity of the electronic devices 30 installed in the management target space R1. The monitoring device 20 may acquire video (i.e., a time series of images). The monitoring device 20 is installed in a position where it can capture images of all of the multiple electronic devices 30.

[0022] An example of the monitoring device 20 is a spherical imaging device. Another example of the monitoring device 20 is a plurality of network cameras. When the monitoring device 20 is a plurality of network cameras, the angle of view of each network camera is adjusted and positioned so that no blind spots occur within the managed space R1.

[0023] Hereinafter, when there are multiple monitoring devices 20, they will be distinguished from one another by using sub-numbers such as "monitoring device 20-1" and "monitoring device 20-2."

[0024] The electronic device 30 is any of various electronic devices used by a user. The electronic device 30 is configured so that it can be used only by authenticated users. The electronic device 30 is configured so that authentication can be performed using hardware such as an IC (Integrated Circuit) card.

[0025] An example of the electronic device 30 is an image forming device (such as a printer, a fax machine, an MFP (Multifunction Peripheral / Product / Printer: a digital multifunction machine), a scanner device, etc.) Another example of the electronic device 30 is an electronic whiteboard (an IWB (Interactive White Board: a whiteboard with an electronic whiteboard function that allows mutual communication)).

[0026] Hereinafter, when the plurality of electronic devices 30 are to be distinguished from one another, they will be referred to using sub-numbers such as "electronic device 30-1" and "electronic device 30-2."

[0027] The electronic device 30 is not limited to an image forming device or an electronic whiteboard as long as it is a device equipped with a communication function. That is, the electronic device 30 may be, for example, an output device such as a PJ (Projector), digital signage, a HUD (Head Up Display) device, industrial machinery, an imaging device, a sound collection device, a medical device, a network home appliance, an automobile (Connected Car), a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game console, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, a desktop PC, or the like.

[0028] <Hardware configuration of the equipment management system> Next, the hardware configuration of each device included in the device management system of this embodiment will be described with reference to FIGS.

[0029] <Computer hardware configuration> FIG. 2 is a diagram illustrating an example of a hardware configuration in which the management device 10 is realized by a computer.

[0030] As shown in FIG. 2, the computer in one embodiment includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, a HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0031] Of these, the CPU 501 controls the operation of the entire computer. The ROM 502 stores programs used to drive the CPU 501, such as an IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501.

[0032] The display 506 displays various types of information such as a cursor, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various types of external devices. In this case, the external devices are, for example, USB (Universal Serial Bus) memories or printers. The network I / F 509 is an interface for data communication using the communication network N1. The bus line 510 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 501 shown in FIG. 2.

[0033] The keyboard 511 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515, such as a flash memory.

[0034] <Hardware configuration of the spherical imaging device> 3 is a diagram showing an example of the hardware configuration when monitoring device 20 is realized by an omnidirectional imaging device. In the following, the omnidirectional imaging device is assumed to be an omnidirectional (all-directional) imaging device using two image sensors, but any number of image sensors may be used as long as it is two or more. In addition, the device does not necessarily have to be dedicated to omnidirectional imaging; a regular digital camera, smartphone, or the like may be equipped with an omnidirectional imaging unit as a retrofit, so that it has substantially the same functions as an omnidirectional imaging device.

[0035] As shown in FIG. 3 , the omnidirectional imaging device according to one embodiment includes an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an antenna 617 a, an acceleration / direction sensor 618, a gyro sensor 619, an acceleration sensor 620, and a concave terminal 621 for Micro USB.

[0036] Of these, the imaging unit 601 includes wide-angle lenses (so-called fisheye lenses) 602a and 602b, each with a field angle of 180° or more for forming a hemispherical image, and two imaging elements 603a and 603b provided corresponding to each wide-angle lens. The imaging elements 603a and 603b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image captured by the fisheye lenses 602a and 602b into image data in the form of an electrical signal and outputs the image data, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters required for the operation of the imaging elements are set.

[0037] The imaging elements 603a and 603b of the imaging unit 601 are each connected to an image processing unit 604 via a parallel I / F bus. On the other hand, the imaging elements 603a and 603b of the imaging unit 601 are connected to an imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, imaging control unit 605, and sound processing unit 609 are connected to a CPU 611 via a bus 610. Furthermore, a ROM 612, an SRAM 613, a DRAM 614, an operation unit 615, an external device connection I / F (Interface) 616, a communication unit 617, an acceleration / direction sensor 618, and the like are also connected to the bus 610.

[0038] The image processing unit 604 takes in the image data output from the image sensors 603a and 603b via a parallel I / F bus, performs predetermined processing on each piece of image data, and then synthesizes the image data to create equirectangular projection image data.

[0039] The imaging control unit 605 generally sets commands and the like in the registers of the imaging elements 603a and 603b using an I2C bus, with the imaging control unit 605 acting as a master device and the imaging elements 603a and 603b acting as slave devices. Necessary commands and the like are received from the CPU 611. The imaging control unit 605 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 603a and 603b and send it to the CPU 611.

[0040] Furthermore, the imaging control unit 605 instructs the image sensors 603a and 603b to output image data when the shutter button on the operation unit 615 is pressed. Some omnidirectional imaging devices have a preview display function or a video display function on a display (for example, a smartphone display). In this case, the image data is output continuously from the image sensors 603a and 603b at a predetermined frame rate (frames per minute).

[0041] As will be described later, the imaging control unit 605 also functions as a synchronization control unit that synchronizes the output timing of image data from the image sensors 603a and 603b in cooperation with the CPU 611. Note that, although the omnidirectional imaging device is not provided with a display in this embodiment, a display unit may be provided.

[0042] The microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in the sound data output from the microphone 608 via the I / F bus and performs predetermined processing on the sound data.

[0043] The CPU 611 controls the overall operation of the omnidirectional imaging device and executes necessary processes. The ROM 612 stores various programs for the CPU 611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611, data in the middle of processing, etc. In particular, the DRAM 614 stores image data in the middle of processing by the image processing unit 604 and data of processed equirectangular projection images.

[0044] The operation unit 615 is a general term for operation buttons such as a shutter button 615a, etc. The user operates the operation unit 615 to input various shooting modes, shooting conditions, and the like.

[0045] The external device connection I / F 616 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory, a PC (Personal Computer), etc. The data of the equirectangular projection image stored in the DRAM 614 is recorded on an external medium via the external device connection I / F 616, or transmitted to an external terminal (device) such as a smartphone via the external device connection I / F 616 as needed.

[0046] The communication unit 617 communicates with an external terminal (device) such as a smartphone via an antenna 617a provided in the omnidirectional imaging device using short-range wireless communication technology such as Wi-Fi, NFC (Near Field Communication), Bluetooth (registered trademark), etc. The communication unit 617 can also transmit data of the equirectangular projection image to an external terminal (device) such as a smartphone.

[0047] The acceleration and orientation sensor 618 calculates the orientation of the omnidirectional imaging device from the Earth's magnetism and outputs orientation information. This orientation information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of captured images. The related information also includes data such as the date and time the image was captured and the data size of the image data. The acceleration and orientation sensor 618 is a sensor that detects angle changes (roll angle, pitch angle, yaw angle) that occur as the omnidirectional imaging device 6 moves. The angle changes are an example of related information (metadata) according to Exif, and are used for image processing such as image correction of captured images.

[0048] Furthermore, the acceleration and orientation sensor 618 is a sensor that detects acceleration in three axial directions. The omnidirectional imaging device calculates the attitude (angle with respect to the direction of gravity) of its own device (the omnidirectional imaging device) based on the acceleration detected by the acceleration and orientation sensor 618. Providing the acceleration and orientation sensor 618 in the omnidirectional imaging device improves the accuracy of image correction.

[0049] <MFP hardware configuration> FIG. 4 is a diagram showing an example of a hardware configuration when the electronic device 30 is realized by an MFP.

[0050] As shown in FIG. 4, the MFP in one embodiment includes a controller 910, a short-range communication circuit 920, an engine control unit 930, an operation panel 940, and a network I / F (Interface) 950.

[0051] Of these, the controller 910 has a CPU (Central Processing Unit) 901, which is the main part of the computer, a system memory (MEM-P) 902, a north bridge (NB) 903, a south bridge (SB) 904, an ASIC (Application Specific Integrated Circuit) 906, a local memory (MEM-C) 907, which is a storage unit, an HDD (Hard Disk Drive) controller 908, and an HD (Hard Disk) 909, which is also a storage unit, and is configured such that the NB 903 and the ASIC 906 are connected by an AGP (Accelerated Graphics Port) bus 921.

[0052] Of these, the CPU 901 is a control unit that performs overall control of the MFP. The NB 903 is a bridge that connects the CPU 901 with the MEM-P 902, SB 904, and AGP bus 921, and includes a memory controller that controls reading and writing to the MEM-P 902, a PCI (Peripheral Component Interconnect) master, and an AGP target.

[0053] The MEM-P 902 comprises a ROM (Read Only Memory) 902a, which is memory for storing programs and data that realize the functions of the controller 910, and a RAM (Random Access Memory) 902b, which is used for expanding the programs and data, and as a drawing memory during memory printing. The programs stored in the RAM 902b may be provided by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD-ROM, CD-R, or DVD.

[0054] The SB 904 is a bridge for connecting the NB 903 with PCI devices and peripheral devices. The ASIC 906 is an integrated circuit (IC) for image processing applications that has hardware elements for image processing and serves as a bridge connecting the AGP bus 921, PCI bus 922, HDD 908, and MEM-C 907. The ASIC 906 includes a PCI target and AGP master, an arbiter (ARB) that forms the core of the ASIC 906, a memory controller that controls the MEM-C 907, multiple direct memory access controllers (DMACs) that perform image data rotation using hardware logic, and a PCI unit that transfers data between the scanner unit 931 and printer unit 932 via the PCI bus 922. A USB (Universal Serial Bus) interface or an IEEE 1394 (Institute of Electrical and Electronics Engineers) interface may also be connected to the ASIC 906.

[0055] The MEM-C907 is a local memory used as an image buffer for copying and a code buffer. The HD909 is a storage for storing image data, font data used during printing, and forms. The HD909 controls the reading and writing of data from and to the HD909 under the control of the CPU901. The AGP bus 921 is a bus interface for a graphics accelerator card proposed to speed up graphics processing, and direct high-throughput access to the MEM-P902 enables the graphics accelerator card to operate at high speed.

[0056] Further, the short-distance communication circuit 920 includes a short-distance communication circuit 920a. The short-distance communication circuit 920 is a communication circuit such as NFC or Bluetooth.

[0057] Furthermore, the engine control unit 930 is made up of a scanner unit 931 and a printer unit 932. The operation panel 940 is equipped with a panel display unit 940a, such as a touch panel, that displays current setting values ​​and selection screens and receives inputs from the operator, and an operation panel 940b that includes a numeric keypad that receives setting values ​​for image formation conditions such as density setting conditions and a start key that receives a copy start instruction. The controller 910 controls the entire MFP, and controls, for example, drawing, communication, input from the operation panel 940, etc. The scanner unit 931 or the printer unit 932 includes an image processing unit such as error diffusion and gamma conversion.

[0058] The MFP can sequentially switch among the document box function, copy function, printer function, and facsimile function using the application switching key on the operation panel 940. When the document box function is selected, the MFP enters document box mode, when the copy function is selected, the MFP enters copy mode, when the printer function is selected, the MFP enters printer mode, and when the facsimile mode is selected, the MFP enters facsimile mode.

[0059] The network I / F 950 is an interface for performing data communication using the communication network N1. The short-range communication circuit 920 and the network I / F 950 are electrically connected to the ASIC 906 via a PCI bus 922.

[0060] <Hardware configuration of the electronic whiteboard> FIG. 5 is a diagram showing an example of a hardware configuration in which the electronic device 30 is realized as an electronic whiteboard.

[0061] As shown in FIG. 5, the electronic whiteboard in one embodiment includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, a network I / F (Interface) 205, and an external device connection I / F 206.

[0062] Of these, the CPU 201 controls the overall operation of the electronic whiteboard. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201. The SSD 204 stores various data, such as programs for the electronic whiteboard. The network controller 205 controls communication with the communication network N1. The external device connection I / F 206 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory 230 and external devices (a microphone 240, a speaker 250, and a camera 260).

[0063] The electronic whiteboard also includes a capture device 211, a GPU 212, a display controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication circuit 219, an antenna 219a of the short-range communication circuit 219, a power switch 222, and selection switches 223.

[0064] Of these, the capture device 211 displays video information as still images or moving images on the display of an external PC (Personal Computer) 270. The GPU (Graphics Processing Unit) 212 is a semiconductor chip that specializes in graphics. The display controller 213 controls and manages screen display to output images output from the GPU 212 to a display 280 or the like.

[0065] The contact sensor 214 detects contact of the electronic pen 290, the user's hand H, or the like on the display 280. The sensor controller 215 controls the processing of the contact sensor 214. The contact sensor 214 inputs and detects coordinates using an infrared blocking method. This coordinate input and coordinate detection method involves two light receiving and emitting devices installed at both ends of the upper side of the display 280 emitting multiple infrared rays parallel to the display 280, and receiving light that is reflected by a reflecting member installed around the periphery of the display 280 and returns along the same optical path as the light emitted by the light receiving element. The contact sensor 214 outputs the IDs of the infrared rays radiated by the two light receiving and emitting devices that are blocked by an object to the sensor controller 215, and the sensor controller 215 identifies the coordinate position that is the contact position of the object.

[0066] The electronic pen controller 216 communicates with the electronic pen 290 to determine whether the pen tip or the pen tail is touching the display 280. The short-range communication circuit 219 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 222 is a switch for turning the power of the electronic whiteboard on and off. The selection switches 223 are a group of switches for adjusting, for example, the brightness and color of the display 280.

[0067] The electronic whiteboard further includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201 shown in FIG.

[0068] The contact sensor 214 is not limited to an infrared blocking type, and various detection means may be used, such as a capacitive touch panel that identifies the contact position by detecting a change in capacitance, a resistive film touch panel that identifies the contact position by a change in voltage between two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by a contacting object coming into contact with the display unit. Also, the electronic pen controller 216 may determine whether or not the part of the electronic pen 290 that the user grips or other parts of the electronic pen have been touched, in addition to the pen tip and pen butt.

[0069] <Functional configuration of equipment management system> Next, an example of the functional configuration of the device management system in this embodiment will be described with reference to Fig. 6. Fig. 6 is a block diagram illustrating the functional configuration of each device included in the device management system in this embodiment.

[0070] <Functional configuration of management device> 6, the management device 10 in this embodiment includes an image receiving unit 11, an operation information receiving unit 12, a status management unit 13, an operation instruction unit 14, and a status storage unit 100. The status management unit 13 in this embodiment includes an object status management unit 131, a device status management unit 132, and an output information determination unit 133.

[0071] The image receiving unit 11, the operation information receiving unit 12, the status management unit 13, and the operation instruction unit 14 are realized, for example, by processing executed by the CPU 501 and the HDD controller 505 of a program expanded from the HD 504 shown in FIG. 2 onto the RAM 503.

[0072] The state storage unit 100 is realized, for example, by using the HD 504 shown in Fig. 2. Reading or writing of data stored in the HD 504 is performed via an HDD controller 505, for example.

[0073] The state storage unit 100 stores state management information for managing the states of devices and objects present in the management target space R1. The state management information in this embodiment includes a registered object list, a registered device list, and a detected object list.

[0074] The registered object list is a list for managing information about objects that exist in the management target space R1. In this embodiment, the objects are people.

[0075] The registered device list is a list for managing information about devices that exist in the management target space R1.

[0076] The detected object list is a list for managing objects detected in the management target space R1. The detected object list is a temporary list used to update the registered object list.

[0077] Here, the state management information in this embodiment will be described with reference to Fig. 7. Fig. 7(A) is a conceptual diagram showing an example of a registered object list.

[0078] As shown in FIG. 7(A), the registered object list in this embodiment has the following data items: registered object ID, group ID, existence confirmation flag, attribute information, and device operation information.

[0079] The registered object ID is identification information that identifies an object included in the registered object list.

[0080] A group ID is identification information that identifies a group to which multiple registered objects belong. A group ID is assigned when some of the registered objects included in the registered object list are determined to belong to the same group. A default value (e.g., zero) is set as the group ID for a registered object that is not determined to belong to a group.

[0081] A group can be, for example, colleagues, parents and children, or friends. Members of a group may jointly hold a meeting, go shopping, or perform other tasks. Therefore, by recognizing groups, it is expected that the behavior of individuals can be grasped more accurately. For example, when multiple employees are holding a meeting in a conference room equipped with an MFP and an electronic whiteboard, it would be very convenient if one employee could authenticate using the MFP and other employees could omit authentication when using the electronic whiteboard.

[0082] The existence confirmation flag is a flag indicating whether or not the registered object exists in the management target space R1. For example, the existence confirmation flag is set to 1 if the registered object exists, and 0 if the registered object does not exist.

[0083] The attribute information is information associated with a registered object and is necessary for management. In this embodiment, the attribute information is the position of the registered object and the time when the position was confirmed (hereinafter also referred to as "image acquisition time"). The position of the registered object is expressed in three-dimensional coordinates (i.e., numerical values ​​on each axis of a Cartesian coordinate system, such as the X axis, Y axis, and Z axis).

[0084] The method for acquiring location information varies depending on the type of monitoring device 20. For example, if the monitoring device 20 is a network camera, a known technique can be used, such as acquiring the distance to an object with a stereo camera and mapping it together with the direction onto three-dimensional coordinates.

[0085] In addition to the position of the registered object, the attribute information may include various information indicating the characteristics of the registered object. For example, the color or shape of the registered object can be used. If the object is a person, it is preferable to use color because the shape changes depending on the posture. Since the number of different colors or the area of ​​each color also changes depending on the posture, it is preferable to use as a condition whether at least one color is continuous when determining whether the objects are the same.

[0086] Device operation information is information about device operations performed by a registered object. The device operation information may include input information entered by a user in the device operation. The device operation information may be, for example, the function performed by the user on the electronic device, the time of the operation, fee information as compensation for the operation, the language used, etc. The input information may be authentication information in an authentication operation or setting information in a device setting operation, etc.

[0087] The number of objects existing in the management target space R1 changes from moment to moment, so the number of registered objects included in the registered object list is variable.

[0088] Fig. 7(B) is a conceptual diagram showing an example of a registered device list. As shown in Fig. 7(B), the registered device list in this embodiment has the following data items: registered device ID, attribute information, device input information, and device output information.

[0089] The registered device ID is identification information that identifies a device included in the registered device list.

[0090] The attribute information is information associated with a registered device and is necessary for management. In this embodiment, the attribute information includes the location of the registered device, power on / off status, whether it is in sleep mode, whether authentication is required, whether options are installed, and status information such as under maintenance / scheduled maintenance time / in energy saving mode. The location of the registered device is expressed in three-dimensional coordinates, just like the registered object.

[0091] Device input information (hereinafter also referred to as "usage information") is predetermined information that is managed when a registered device is operated. Device input information may include input information entered by the user when operating the device. Device input information may include, for example, functions executed by the user on a registered device, the time of operation, and fee information to be charged as consideration for the operation. Device input information may also include screen setting information set by the user when operating the device. Screen setting information may include, for example, the language used, the font of characters used, the size of characters used, etc.

[0092] The device output information is information that is pre-registered for a registered device. The device output information is a list that associates output conditions that trigger the transmission of an operation instruction with operation instructions that are to be transmitted to the registered device when the output conditions are satisfied.

[0093] The number of devices installed in the management target space R1 may change, so the number of registered devices included in the registered device list is variable.

[0094] 7C is a conceptual diagram showing an example of a detected object list. As shown in FIG. 7C, the detected object list in this embodiment has detected object IDs, group IDs, and attribute information as data items.

[0095] The detected object ID is identification information that identifies an object included in the detected object list.

[0096] A group ID is identification information that identifies a group containing multiple detected objects. A group ID is assigned when some of the detected objects included in the detected object list are determined to belong to the same group. A default value (e.g., zero) is set as the group ID for detected objects that are not determined to belong to a group.

[0097] The attribute information is information associated with the detected object and is necessary for management. In this embodiment, the attribute information is the position of the detected object and the image capture time. The position of the detected object is expressed in three-dimensional coordinates, just like the registered objects.

[0098] The number of objects detected in the management target space R1 changes from moment to moment, so the number of detected objects included in the detected object list is variable.

[0099] Returning to Fig. 6, the image receiving unit 11 receives an image from the monitoring device 20. The image receiving unit 11 sends the received image to the state management unit 13 in response to a request from the state management unit 13.

[0100] The operation information receiving section 12 receives device operation information from the electronic device 30. The operation information receiving section 12 sends the received device operation information to the state management section 13 in response to a request from the state management section 13.

[0101] The state management unit 13 manages the states of the electronic device 30 and the object using an object state management unit 131 and an equipment state management unit 132. In addition, the state management unit 13 determines the content of an operation instruction to be transmitted to the electronic device 30 using an output information determination unit 133.

[0102] The object state management unit 131 updates the registered object list stored in the state storage unit 100 based on the image received from the image receiving unit 11 .

[0103] The device state management unit 132 updates the registered device list and registered object list stored in the state storage unit 100 based on the device operation information received from the operation information receiving unit 12 .

[0104] The output information determination unit 133 determines the content of the operation instruction to be transmitted to the electronic device 30 based on the state management information stored in the state storage unit 100. The output information determination unit 133 sends the determined operation instruction to the operation instruction unit 14.

[0105] The operation instruction unit 14 transmits the operation instruction received from the state management unit 13 to the electronic device 30.

[0106] <Functional configuration of the monitoring device> As shown in FIG. 6, the monitoring device 20 in this embodiment includes an image acquisition unit 21 and an image transmission unit 22.

[0107] The image acquisition unit 21 acquires an image including the vicinity of the electronic device 30 installed in the management target space R1. The image acquisition unit 21 is realized, for example, by processing that the CPU 611 and the imaging control unit 605 execute in accordance with a program loaded from the ROM 612 onto the SRAM 613 shown in FIG.

[0108] The image transmission unit 22 transmits the image acquired by the image acquisition unit 21 to the management device 10. The image transmission unit 22 is realized, for example, by a process in which a program loaded from the ROM 612 onto the SRAM 613 shown in FIG. 3 is executed by the CPU 611 and the external device connection I / F 616.

[0109] <Functional configuration of electronic devices> As shown in FIG. 6, an electronic device 30 according to this embodiment includes an operation information transmitting unit 31 and a device control unit 32.

[0110] The operation information transmitting unit 31 and the device control unit 32 are realized by, for example, processing that is executed by the CPU 901 and the HDD controller 908 in accordance with a program loaded from the HD 909 onto the RAM 902b shown in FIG.

[0111] The operation information transmitting unit 31 transmits to the management device 10 device operation information relating to an operation performed by a user.

[0112] The device control unit 32 receives an operation instruction from the management device 10. The device control unit 32 controls the operation of the electronic device 30 based on the received operation instruction. For example, the device control unit 32 authenticates the user using authentication information included in the operation instruction. Furthermore, for example, the device control unit 32 changes the settings of the electronic device 30 using setting information included in the operation instruction.

[0113] <Device management procedure> Next, a device management method executed by the device management system in this embodiment will be described with reference to FIGS.

[0114] <Basic Flowchart> The device management method of this embodiment includes many double-loop processes that process all combinations of data included in one list and data included in another list. Therefore, in this embodiment, a basic flowchart showing the framework of the double-loop process is introduced, and the following explanation focuses on the processes in the basic flowchart. Note that each process in the basic flowchart is performed for one combination of two data.

[0115] The basic flowchart in this embodiment includes two basic flowcharts. The first basic flowchart is used when two data items contained in different lists are processed in a double loop. The second basic flowchart is used when two data items contained in the same list are processed in a double loop.

[0116] FIG. 8 is a flowchart showing an example of a first basic flowchart in this embodiment.

[0117] In step S101, list A is read out. The number of data items in list A is assumed to be N. In step S102, list B is read out. The number of data items in list B is assumed to be M.

[0118] In step S103, a variable n is initialized to 1. In step S104, a variable m is initialized to 1. In step S105, processing A1 is executed.

[0119] In step S106, it is determined whether the variable m is equal to the number of data M. This means whether the combinations of the nth data in list A with all data in list B have been processed. If the variable m is different from the number of data M (NO), proceed to step S107. If the variable m is equal to the number of data M (YES), proceed to step S109.

[0120] In step S107, process A3 is executed. In step S108, variable m is incremented. Thereafter, the process returns to step S106.

[0121] In step S109, process A2 is executed. In step S110, it is determined whether variable n is equal to the number of data N. This means whether all combinations of all data in list A with all data in list B have been processed. If variable n is different from the number of data N (NO), proceed to step S111. If variable n is equal to the number of data N (YES), proceed to step S113.

[0122] In step S111, process A5 is executed. In step S112, the variable n is incremented. Then, the process returns to step S104.

[0123] In step S113, process A4 is executed. With the above, the processes for all combinations of data included in list A and list B are completed.

[0124] FIG. 9 is a flowchart showing an example of the second basic flowchart in this embodiment.

[0125] In step S201, list A is read. The number of data items in list A is set to N.

[0126] In step S202, a variable n is initialized to 1. In step S203, n+1 is substituted into a variable m. In step S204, process B1 is executed.

[0127] In step S205, it is determined whether variable m is equal to N. This means whether or not all combinations of the nth data in list A with all data in list A have been processed. If variable m is different from N (NO), proceed to step S206. If variable m is equal to N (YES), proceed to step S208.

[0128] In step S206, process B3 is executed. In step S207, variable m is incremented. Then, the process returns to step S204.

[0129] In step S208, process B2 is executed. In step S209, it is determined whether variable n is equal to N-1. This means that all combinations of all data in list A with all other data in list A have been processed. If variable n is different from N-1 (NO), proceed to step S210. If variable n is equal to N-1 (YES), proceed to step S212.

[0130] In step S210, process B5 is executed. In step S211, the variable n is incremented. Then, the process returns to step S203.

[0131] In step S212, process B4 is executed. With the above, processing for all combinations of data included in list A is completed.

[0132] ≪Device management method≫ FIG. 10 is a flowchart showing an example of a device management method executed by the device management system according to this embodiment.

[0133] The device management system repeatedly executes the flowchart shown in Fig. 10 at predetermined time intervals. The time interval may be set arbitrarily, for example, to 3 seconds.

[0134] In step S1, the image acquisition unit 21 included in the monitoring device 20 acquires a first image capturing an area near the electronic device 30-1. The image acquisition unit 21 also acquires a second image capturing an area near the electronic device 30-2. The monitoring device 20 is installed so that the areas near the electronic devices 30-1 and 30-2 installed in the management target space R1 are included in the angle of view. Therefore, the first image acquired by the image acquisition unit 21 captures an area near the electronic device 30-1. The second image acquired by the image acquisition unit 21 captures an area near the electronic device 30-2.

[0135] The vicinity of electronic device 30-1 or electronic device 30-2 refers to a range of positions where a user can operate electronic device 30-1 or electronic device 30-2 (hereinafter also referred to as "operable position"). The operable position is a position where a user can reach electronic device 30-1 or electronic device 30-2. The operable position is, for example, a position within one meter of electronic device 30-1 or electronic device 30-2. Therefore, the first image is an image of electronic device 30-1 and a user using electronic device 30-1. Furthermore, the second image is an image of electronic device 30-2 and a user using electronic device 30-2.

[0136] If the operable positions of the electronic devices 30-1 and 30-2 can be captured in a single image, the operable positions of the electronic devices 30-1 and 30-2 may be captured in a single image. In this case, the first image and the second image are the same image.

[0137] The first and second images acquired by the image acquisition unit 21 are not limited to still images, but may be moving images.

[0138] Next, the image acquisition unit 21 sends the acquired first image and second image to the image transmission unit 22. The image transmission unit 22 receives the first image and the second image from the image acquisition unit 21. Next, the image transmission unit 22 transmits the received first image and second image to the management device 10. In the management device 10, the image reception unit 11 receives the first image and the second image from the monitoring device 20.

[0139] In step S2, the state management unit 13 included in the management device 10 requests the first image and the second image from the image receiving unit 11. Next, the state management unit 13 inputs the first image and the second image received from the image receiving unit 11 to the object state management unit 131.

[0140] Next, the object state management unit 131 executes an object state update process, which will be described later, and updates the registered object list of the state management information stored in the state storage unit 100. If there are multiple monitoring devices 20, the object state management unit 131 executes the object state update process for each of the multiple images received from each of the multiple monitoring devices 20.

[0141] <Object state update process> The object state update process in this embodiment will now be described with reference to Fig. 11 to Fig. 16. Fig. 11 is a flowchart showing an example of the object state update process (step S2 in Fig. 10) executed by the object state management unit 131 in this embodiment.

[0142] In step S21, the object state management unit 131 executes a detected object list creation process, which will be described later, and creates a detected object list based on the image input from the state management unit 13.

[0143] FIG. 12 is a flowchart showing an example of the detected object list creation process (step S21 in FIG. 11) executed by the object state management unit 131 in this embodiment.

[0144] In step S21-1, the object state management unit 131 clears the detected object list stored in the state storage unit 100. In other words, all data included in the detected object list is deleted.

[0145] In step S21-2, the object state management unit 131 acquires an image input from the state management unit 13. The image may be one image received from one monitoring device 20, or multiple images received from multiple monitoring devices 20.

[0146] Next, the object state management unit 131 divides the image into predetermined blocks. Then, for each block, the object state management unit 131 measures the distance between the object captured in the image and the camera. The object state management unit 131 also calculates three-dimensional coordinates from the measured distances. A known method may be used to convert the distances into three-dimensional coordinates.

[0147] In step S21-3, the object state management unit 131 acquires the image acquisition time T from the acquired image. If the image acquisition time cannot be acquired from the image, the current time may be acquired as the image acquisition time T.

[0148] In step S21-4, the object state management unit 131 analyzes the acquired image and detects an object captured in the image. In this embodiment, it detects a person captured in the image.

[0149] A known method can be used to detect people. For example, pattern matching using a machine learning model can be used. Specifically, a known method such as Region Based Convolutional Neural Networks (R-CNN) can be used to extract people from an image.

[0150] Furthermore, the accuracy of detection can be improved by pattern matching the extracted image of the person with a pre-stored comparison image. Note that various other known person detection methods can also be used.

[0151] In step S21-5, the object state management unit 131 assigns a detected object ID that identifies the detected object.

[0152] In step S21-6, the object state management unit 131 registers the detected object ID and attribute information (the position of the detected object and the image acquisition time) in the detected object list. At this stage, the group ID is set to an initial value indicating that the object does not belong to a group.

[0153] Returning to Fig. 11, in step S22, the object state management unit 131 executes a registered object list update process, which will be described later, and updates the registered object list based on the detected object list.

[0154] 13 and 14 are flowcharts showing an example of the registered object list update process (step S22 in FIG. 11) executed by the object state management unit 131 in this embodiment.

[0155] The registered object list update process is performed within the framework of the first basic flowchart (see FIG. 8). Specifically, the detected object list is List A, the registered object list is List B, and process A1 (FIG. 13) and process A4 (FIG. 14) are executed for all combinations of detected objects and registered objects. Note that there are no processes equivalent to processes A2, A3, and A5.

[0156] 13 is a flowchart showing an example of process A1 of the registered object list update process. Process A1 is a process for determining the identity of a registered object and a detected object, and updating the position of the registered object.

[0157] Here, it is assumed that processing is executed for the combination of the nth detected object in the detected object list and the mth registered object in the registered object list.

[0158] In step S22-1, the object state management unit 131 clears the existence confirmation flag of the mth registered object. That is, the existence confirmation flag of the mth registered object is set to 0. This means that it is unknown whether the registered object exists in the management target space R1.

[0159] In step S22-2, the object state management unit 131 calculates the distance X between the three-dimensional coordinates of the mth registered object and the three-dimensional coordinates of the nth detected object.

[0160] In step S22-3, the object state management unit 131 determines whether the distance X is equal to or less than a predetermined threshold (for example, 1 meter). If the distance X exceeds the threshold (NO), the object state management unit 131 ends process A1. This means that the two are far apart and therefore it has been determined that they are not the same person. If the distance X is equal to or less than the threshold (YES), the object state management unit 131 proceeds to step S22-4.

[0161] In step S22-4, the object state management unit 131 adds the three-dimensional position and image acquisition time of the nth detected object to the attribute information of the mth registered object. This means that because the distance between the two is short, it has been determined that they are the same person, and that the position of the nth detected object is the position of the mth registered object at the image acquisition time T.

[0162] The attribute information in the registered object list increases each time the object state update process is executed, as long as the registered object is detected in the image. In other words, the attribute information in the registered object list represents the time series of the positions where the registered object was detected in the managed space. Therefore, the attribute information in the registered object list can be used to track the movement of the registered object.

[0163] In step S22-5, the object state management unit 131 sets the existence confirmation flag for the m-th registered object. That is, the existence confirmation flag for the m-th registered object is set to 1. This means that an object matching the registered object was detected in step S22-3, and therefore it was determined that the registered object existed in the management target space R1 even at the image acquisition time T.

[0164] 14 is a flowchart showing an example of process A4 of the registered object list update process. Process A4 is a process for deleting registered objects whose existence could not be confirmed in process A1 from the registered object list.

[0165] In step S22-6, the object state management unit 131 initializes a variable m to one.

[0166] In step S22-7, the object state management unit 131 determines whether the existence of the mth registered object has been confirmed. Specifically, it determines whether the existence confirmation flag is 1 or 0. If the existence confirmation flag is 1 (YES), the object state management unit 131 proceeds to step S22-9. If the existence confirmation flag is 0 (NO), the object state management unit 131 proceeds to step S22-8.

[0167] In step S22-8, the object state management unit 131 deletes the m-th registered object from the registered object list.

[0168] In step S22-9, the object state management unit 131 determines whether the variable m is equal to the number of data M. If the variable m is different from the number of data M (NO), the object state management unit 131 proceeds to step S22-10. If the variable m is equal to the number of data M (YES), the object state management unit 131 ends the process.

[0169] In step S22-10, the object state management unit 131 increments the variable m, and then returns the process to step S22-7.

[0170] Returning to Fig. 11, in step S23, the object state management unit 131 executes a group ID assignment process, which will be described later, and assigns a group ID to the registered object included in the registered object list.

[0171] The group ID assignment process is performed within the framework of the second basic flowchart (see FIG. 9). Specifically, the registered object list is List A, and process B1 (FIG. 15 or 16) is executed for all combinations of two registered objects. Note that there are no processes corresponding to processes B2-B5.

[0172] There are various methods for determining whether a certain object and another object belong to the same group. In this embodiment, group determination based on positional proximity and group behavior will be described. However, the group determination methods are not limited to these, and any technology that can determine the group of objects from an image may be used.

[0173] 15 is a flowchart showing a first example of the group ID assignment process (step S23 in FIG. 11) executed by the object state management unit 131 in this embodiment. The first example of the group ID assignment process is group determination based on positional proximity.

[0174] Group determination based on proximity is a process of registering as a group two objects that have been detected in close proximity for a predetermined number of consecutive times in the last few days. The predetermined number is, for example, five times. If the time interval for executing the device management method is three seconds, two objects that have been detected in close proximity for 15 consecutive seconds will be determined to be in the same group.

[0175] Here, it is assumed that processing is executed for the combination of the mth registered object and the nth registered object in the registered object list.

[0176] In step S23A-1, the object state management unit 131 initializes variables k and j to 1. Variable k is a counter that indicates the number of times it has been determined whether two objects are close to each other. Variable j is a counter that indicates the number of times it has been determined that two objects are close to each other.

[0177] In step S23A-2, the object state management unit 131 calculates the distance X between the three-dimensional coordinates of the mth registered object k times before and the three-dimensional coordinates of the nth registered object k times before.

[0178] In step S23A-3, the object state management unit 131 determines whether the distance X is less than a predetermined threshold (e.g., 1 meter). If the distance X is less than the threshold (YES), the object state management unit 131 proceeds to step S23A-4. If the distance X is equal to or greater than the threshold (NO), the object state management unit 131 proceeds to step S23A-5.

[0179] In step S23A-4, the object-state management unit 131 increments the variable j.

[0180] In step S23A-5, the object-state management unit 131 increments the variable k.

[0181] In step S23A-6, the object state management unit 131 determines whether the variable k is equal to the predetermined number of times K. If the variable k is different from the predetermined number of times K (NO), the object state management unit 131 returns the process to step S23A-2. If the variable k is equal to the predetermined number of times K (YES), the object state management unit 131 proceeds to step S23A-7.

[0182] In step S23A-7, the object state management unit 131 determines whether the variable j is equal to the predetermined number of times K. If the variable j is different from the predetermined number of times K (NO), the object state management unit 131 ends the processing. If the variable j is equal to the predetermined number of times K (YES), the object state management unit 131 proceeds to step S23A-8.

[0183] In step S23A-8, the object state management unit 131 determines whether a group ID has been assigned to either the mth registered object or the nth registered object. If a group ID has been assigned (YES), the object state management unit 131 proceeds to step S23A-9. If a group ID has not been assigned (NO), the object state management unit 131 proceeds to step S23A-10.

[0184] In step S23A-9, the object state management unit 131 determines one of the group IDs assigned to the m-th registered object and the n-th registered object as the group ID to be assigned. Next, the object state management unit 131 sets the determined group ID as the group ID of the m-th registered object and the n-th registered object in the registered object list.

[0185] The object state management unit 131 determines the group ID to be assigned as follows: If a group ID is assigned to only one of the registered objects, that group ID is used as the group ID to be assigned. In other words, a registered object to which no group ID is assigned is added as a member of an existing group.

[0186] If both registered objects have been assigned group IDs, first, the m+1th and subsequent registered objects are sorted by group ID to identify the registered object with the same group ID as the mth registered object. Next, the group IDs of the mth registered object and all identified registered objects are updated with the group ID of the nth registered object. As a result, all members of the group to which the mth registered object belongs will belong to the same group as the nth registered object. This makes it possible to identify groups that contain three or more registered objects.

[0187] In step S23A-10, the object state management unit 131 issues a new group ID that does not overlap with any other group IDs. Next, the object state management unit 131 sets the issued new group ID as the group ID of the m-th registered object and the n-th registered object in the registered object list.

[0188] 16 is a flowchart showing a second example of the group ID assignment process (step S23 in FIG. 11) executed by the object-state management unit 131 in this embodiment. The second example of the group ID assignment process is group determination based on group behavior.

[0189] Group determination based on group behavior is a process of performing group determination by image analysis after creating a detected object list. Therefore, when performing group determination based on group behavior, step S23 is executed between step S21 and step S22.

[0190] Here, it is assumed that processing is executed for the combination of the mth detected object and the nth detected object in the detected object list.

[0191] In step S23B-1, object state management unit 131 calculates the distance X between the three-dimensional coordinates of the nth detected object and the three-dimensional coordinates of the mth detected object.

[0192] In step S23B-2, the object state management unit 131 determines whether the distance X is less than a predetermined threshold (for example, 1 meter). If the distance X is equal to or greater than the threshold (NO), the object state management unit 131 ends the process. If the distance X is less than the threshold (YES), the object state management unit 131 proceeds to step S23B-3.

[0193] In step S23B-3, object state management unit 131 calculates the midpoint between the three-dimensional coordinates of the nth detected object and the mth detected object. Specifically, object state management unit 131 divides the sum of the X coordinate, Y coordinate, and Z coordinate of the two detected objects by 2.

[0194] In step S23B-4, the object-state management unit 131 extracts an image P having a radius of Y pixels centered on the midpoint. The radius Y is set according to the resolution of the image so that the range included in the image P is approximately 1 m in real space.

[0195] In step S23B-5, the object state management unit 131 reads out a pre-stored group action image. The group action image is a collection of images representing actions determined to belong to the same group. For example, the group action image may be an image of two people greeting or shaking hands.

[0196] In step S23B-6, the object state management unit 131 calculates the similarity between the video P and each of the group action images by pattern matching or the like.

[0197] In step S23B-7, the object state management unit 131 determines whether any of the calculated similarities exceeds a predetermined threshold. If any similarity exceeds the threshold (YES), the object state management unit 131 proceeds to step S23B-8. If no similarity exceeds the threshold (NO), the object state management unit 131 ends the process.

[0198] In step S23B-8, the object state management unit 131 issues a new group ID that does not overlap with any other group IDs. Next, the object state management unit 131 sets the issued new group ID as the group ID for the m-th detected object and the n-th detected object in the detected object list.

[0199] In step S22-4 of the registered object list update process executed thereafter (see FIG. 13), the object state management unit 131 sets the group ID of the registered object to the group ID of the detected object determined to be the same object. Note that if an existing group ID has been assigned to the registered object, steps S23A-8 to S23A-10 of the first example of the group ID assignment process (see FIG. 15) are executed.

[0200] Returning to Fig. 10, in step S3, the operation information transmitting unit 31 included in the electronic device 30-1 determines, in accordance with an operation performed by a user on the electronic device 30-1, whether or not the operation content is information that should be notified to the management device 10. This determination is made based on whether or not the operation content matches any of predetermined operation contents.

[0201] When it is determined that the operation content is one that should be notified, the operation information transmitting unit 31 transmits device operation information regarding the device operation to the management device 10. In the management device 10, the operation information receiving unit 12 receives the device operation information from the electronic device 30-1.

[0202] The operation contents and operation information in this embodiment are exemplified below.

[0203] <Example 1> Operation: Device login Operation information: Registered device ID, device authentication ID (if authentication is required), login time

[0204] <Example 2> Operation details: Processing that incurs charges (copy output, use of pay-per-use software, etc.) Operation information: registered device ID, device authentication ID (if authentication is required), fee, input data, language used

[0205] <Example 3> Operation: None Operation information: Status information such as maintenance and energy saving mode

[0206] The registered device ID is identification information included in the registered device list stored in the state storage unit 100 of the management device 10. A registered device ID is assigned to each electronic device 30 installed in the management target space R1. The device authentication ID is authentication information used for authentication for a user of the electronic device 30 to use the electronic device 30.

[0207] In step S4, the state management unit 13 included in the management device 10 requests operation information from the operation information receiving unit 12. Next, the state management unit 13 inputs the operation information received from the operation information receiving unit 12 to the device state management unit 132.

[0208] Next, the device status management unit 132 executes a device status update process, which will be described later, and updates the registered device list of the status management information stored in the status storage unit 100.

[0209] <Device status update process> The device status update process in this embodiment will now be described with reference to Fig. 17. Fig. 17 is a flowchart showing an example of the device status update process (step S4 in Fig. 10) executed by the device status management unit 132 in this embodiment.

[0210] In step S41, the device status management unit 132 identifies a registered device included in the registered device list by the registered device ID included in the device operation information. Next, the device status management unit 132 sets the received device operation information as the device input information of the identified registered device.

[0211] In step S42, the device state management unit 132 initializes a variable n to one.

[0212] In step S43, the device status management unit 132 calculates the distance X between the three-dimensional coordinates of the identified registered device and the latest three-dimensional coordinates of the n-th registered object. The three-dimensional coordinates of the registered device are set in advance using the same method as the three-dimensional coordinates of the registered object. Since the installation location of the registered device may be moved, the three-dimensional coordinates of the registered device must be updated periodically, but the update frequency may be low.

[0213] In step S44, the device status management unit 132 determines whether the distance X is less than a predetermined threshold (for example, 1 meter). If the distance X is equal to or greater than the threshold (NO), the device status management unit 132 proceeds to step S47. If the distance X is less than the threshold (YES), the device status management unit 132 proceeds to step S45.

[0214] In step S45, the device state management unit 132 adds the device input information of the identified registered device to the device operation information of the n-th registered object. The device operation information in the registered object list is configured to be able to store a predetermined number of pieces of device input information.

[0215] In step S46, the device status management unit 132 determines whether the variable n is equal to the number of data N. If the variable n is different from the number of data N (NO), the device status management unit 132 proceeds to step S47. If the variable n is equal to the number of data N (YES), the device status management unit 132 ends the process.

[0216] In step S47, the device status management unit 132 increments the variable n, and then returns the process to step S43.

[0217] 10. In step S5, the output information determination unit 133 included in the management device 10 executes an output information determination process, which will be described later, and determines an operation instruction to be transmitted to the electronic device 30 based on the state management information stored in the state storage unit 100.

[0218] <<Output information determination process>> The output information determination process in this embodiment will now be described with reference to Fig. 18 and Fig. 19. Fig. 18 is a flowchart showing an example of the output information determination process (step S5 in Fig. 10) executed by the output information determination unit 133 in this embodiment.

[0219] The output information determination process is performed within the framework of the first basic flowchart (see FIG. 8). Specifically, the registered device list is List A, the registered object list is List B, and process A1 (FIG. 18) is executed for all combinations of registered devices and registered objects. Note that there are no processes equivalent to processes A2-A5.

[0220] Here, it is assumed that processing is executed for the combination of the nth registered device in the registered device list and the mth registered object in the registered object list.

[0221] In step S51, the output information determination unit 133 calculates the distance X between the latest three-dimensional coordinates of the nth registered device and the latest three-dimensional coordinates of the mth registered object.

[0222] In step S52, the output information determination unit 133 determines whether the distance X is less than a predetermined threshold (for example, 1 meter). If the distance X is equal to or greater than the threshold (NO), the device status management unit 132 ends the process. If the distance X is less than the threshold (YES), the device status management unit 132 proceeds to step S53.

[0223] In step S53, the output information determination unit 133 acquires the output condition included in the device output information of the n-th registered device from the registered device list stored in the state storage unit 100.

[0224] In step S54, the output information determination unit 133 acquires the device operation information of the m-th registered object from the registered object list stored in the state storage unit 100.

[0225] In step S55, the output information determination unit 133 determines whether any of the device operation information acquired in step S54 satisfies the output condition acquired in step S53. If none of the device operation information satisfies the output condition (NO), the device status management unit 132 ends the process. If any of the device operation information satisfies the output condition (YES), the device status management unit 132 proceeds to step S56.

[0226] In step S56, the output information determination unit 133 acquires operation instructions corresponding to the output conditions acquired in step S53 from the device output information of the n-th registered device. Note that the operation instructions for the registered device may include adaptive processing according to the attribute information of the registered device. For example, the operation instructions may include a condition that the operation instructions are not sent when the registered device is in an inoperable state. An inoperable state may be, for example, a power-off state or a hibernation state.

[0227] FIG. 19 is a flowchart showing a modified example of the output condition determination process (step S55 in FIG. 18) executed by the output information determination unit 133 in this embodiment.

[0228] In the output information determination process shown in Fig. 18, it is determined in step S55 whether the output conditions are met based only on the device operation information of the m-th registered object. In the output information determination process shown in Fig. 19, it is determined whether the output conditions are met by referring to device operation information of devices other than the m-th registered device.

[0229] In step S55-1, the output information determination unit 133 determines whether any of the device operation information for the mth registered object satisfies the output condition of the nth registered device. If any of the device operation information satisfies the output condition (YES), the output information determination unit 133 proceeds to step S55-10. If none of the device operation information satisfies the output condition (NO), the output information determination unit 133 proceeds to step S55-2.

[0230] In step S55-2, the output information determination unit 133 determines whether or not device operation information for another registered object is necessary based on the output conditions of the nth registered device. If device operation information for another registered object is necessary (YES), the output information determination unit 133 proceeds to step S55-3. If device operation information for another registered object is not necessary (NO), the output information determination unit 133 proceeds to step S55-9.

[0231] In step S55-3, the output information determination unit 133 initializes a variable k to one.

[0232] In step S55-4, the output information determination unit 133 determines whether the variable k is equal to the variable m. If the variable k is equal to the variable m (YES), the output information determination unit 133 proceeds to step S55-5. If the variable k is not equal to the variable m (NO), the output information determination unit 133 proceeds to step S55-7.

[0233] In step S55-5, the output information determination unit 133 acquires the device operation information of the k-th registered object from the registered object list stored in the state storage unit 100.

[0234] In step S55-6, the output information determination unit 133 determines whether any of the device operation information for the kth registered object satisfies the output condition of the nth registered device. If any of the device operation information satisfies the output condition (YES), the output information determination unit 133 proceeds to step S55-10. If none of the device operation information satisfies the output condition (NO), the output information determination unit 133 proceeds to step S55-7.

[0235] In step S55-7, the output information determination unit 133 determines whether the variable k is equal to the number of data items M. If the variable k is different from the number of data items M (NO), the output information determination unit 133 proceeds to step S55-8. If the variable m is equal to the number of data items M (YES), the output information determination unit 133 proceeds to step S55-9.

[0236] In step S55-8, the output information determination unit 133 increments the variable k, and then returns the process to step S55-4.

[0237] In step S55-9, the output information determination section 133 determines that the device operation information acquired in step S54 does not satisfy the output condition acquired in step S53, and ends the process.

[0238] In step S55-10, the output information determination section 133 determines that the device operation information acquired in step S54 satisfies the output condition acquired in step S53, and ends the process.

[0239] 20 is a diagram showing an example of device output information in this embodiment. A first example of an output condition for device output information in this embodiment is an output condition in which device operation information for other registered objects is not required. A second example of an output condition for device output information in this embodiment is an output condition in which device operation information for other registered objects is required.

[0240] As shown in FIG. 20, a first example of an output condition in this embodiment is that the authentication information input by the mth registered object (hereinafter also referred to as "person α") on another electronic device (here, an MFP) is authentication information that can be authenticated on the nth electronic device (here, an electronic whiteboard).

[0241] As shown in FIG. 20, if it is determined that the first example of the output condition is satisfied, one or more of the following operational instructions are determined as the operational instructions to be transmitted to the registered device. The first operational instruction is to have the electronic whiteboard automatically authenticate person α. ​​In this case, the operational instruction to be transmitted includes the authentication information entered by person α on the MFP. The second operational instruction is to display "Mr. α, automatic authentication has been performed" on the electronic whiteboard for several seconds.

[0242] The third operational instruction is to display the message "Do you want to display the information just entered on device XX (the name of the MFP)?" along with a selection button for "Yes" or "No" when person α is entering information on the MFP (for example, scanning). If "Yes" is selected, the information entered on the scanner is displayed on the interactive whiteboard.

[0243] The fourth operational instruction is to identify the language set in the MFP from person α's device operation information and reflect it in the display on the electronic whiteboard. The fifth operational instruction is to not send an operational instruction if the attribute information of the electronic whiteboard is in a pause state such as "in energy saving mode."

[0244] 20, a second example of an output condition in this embodiment is that the authentication information input by the k-th registered object (hereinafter also referred to as "person β") in another electronic device (here, an MFP) is authentication information that can be authenticated in the n-th electronic device (here, an electronic whiteboard). Note that person β has the same group ID as person α.

[0245] As shown in FIG. 20, if it is determined that the second example of the output condition is satisfied, one or more of the following operational instructions are determined as the operational instructions to be transmitted to the registered device. The first operational instruction is to automatically authenticate person α on the electronic whiteboard. In this case, the operational instruction to be transmitted includes the authentication information entered by person β on the MFP. The second operational instruction is to display on the electronic whiteboard for several seconds, "Automatic authentication has been performed using person β's authentication information."

[0246] The third operational instruction is to display the message "Do you want to display the information entered by person β on device XX (the name of the MFP)?" along with a selection button for "Yes" or "No" when person β is entering information on the MFP (for example, scanning). If "Yes" is selected, the information entered by the scanner is displayed on the interactive whiteboard.

[0247] 10, in step S6, the output information determination unit 133 included in the management device 10 sends operation instruction information representing an operation instruction to the operation instruction unit 14. The operation instruction information includes the determined operation instruction and information indicating the destination electronic device 30-2.

[0248] The operation instruction unit 14 receives the operation instruction information from the output information determination unit 133. Next, the operation instruction unit 14 transmits the operation instruction included in the operation instruction information to the destination electronic device 30-2 included in the operation instruction information.

[0249] In the electronic device 30-2, the device control unit 32 receives an operation instruction from the management device 10. Next, the device control unit 32 executes an operation using input information included in the operation instruction in accordance with the received operation instruction. For example, the device control unit 32 authenticates the user using authentication information included in the operation instruction. Furthermore, for example, the device control unit 32 changes the settings of the electronic device 30-2 using setting information included in the operation instruction.

[0250] <Effects of the embodiment> The device management system 1 in this embodiment tracks the movement of a person based on images captured near the electronic device 30, and transmits an operation instruction using information input by a user to a first electronic device 30-1 to a second electronic device 30-2 that detects that the user is nearby. Therefore, the device management system 1 in this embodiment can cause the second electronic device to execute an operation using information input to the first electronic device.

[0251] In particular, by transmitting an authentication instruction using authentication information authenticated by the first electronic device 30-1 to the second electronic device 30-2, authentication can be automatically performed when the user authenticated by the first electronic device 30-1 moves near the second electronic device 30-2. In other words, the device management system in this embodiment can achieve single sign-on through hardware cooperation.

[0252] [Application example] In the above embodiment, the managed space is assumed to be a conference room or office, and the electronic device 30 is managed as an OA (Office Automation) device. However, the device management system 1 can be applied to a variety of usage scenarios, and is not limited to this.

[0253] For example, the device management system 1 can be configured to manage electronic devices installed in a hotel. A hotel is equipped with various electronic devices. For example, an automatic check-in machine may be installed in the lobby. Also, a set-top box capable of playing television broadcasts and on-demand videos may be installed in each guest room. By managing these devices with the device management system 1, it becomes possible to automatically set the language used by the hotel guest at the automatic check-in machine as the language used by the set-top box.

[0254] Furthermore, for example, the device management system 1 can be configured to manage electronic devices installed at airports. At airports, passengers must go through procedures such as ticketing, check-in, and baggage inspection before boarding an aircraft, and boarding passes are processed by dedicated electronic devices at each procedure. By managing these devices with the device management system 1, it becomes possible to automatically set the language set on the electronic device performing the previous procedure as the language set on the electronic device performing the next procedure.

[0255] Furthermore, for example, the device management system 1 can be configured to manage electronic devices such as search terminals installed in bookstores. By linking searches performed by customers on search terminals with information on bookshelves to which they subsequently move, this can be used to strengthen marketing.

[0256] Furthermore, for example, the device management system 1 can be configured to manage electronic devices such as PCs installed in a factory. By correlating and verifying the actions of workers with the operation details of devices installed at the work site, the appropriateness of the workers' work can be monitored.

[0257] Furthermore, for example, the device management system 1 can be configured to manage electronic devices installed in unmanned convenience stores. Convenience stores are equipped with electronic devices such as MFPs and payment terminals. By managing these devices with the device management system 1, it becomes possible, for example, to set the language set by a customer on the MFP as the language set on the payment terminal.

[0258] [supplement] In each of the above embodiments, the management device 10 is an example of an information processing device. The device management system 1 is an example of an information processing system. The monitoring device 20 is an example of an imaging device. The status management unit 13 is an example of a user identification unit. The operation information receiving unit 12 is an example of an input information receiving unit.

[0259] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0260] The devices described in the example are merely one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, management device 10 includes multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0261] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0262] 1. Equipment management system 10 Management device 11 Image receiving unit 12 Operation information receiving unit 13 State Management Unit 14 Operation instruction section 100 State memory unit 131 Object State Management Unit 132 Equipment Status Management Unit 133 Output information determination unit 20 Monitoring equipment 21 Image acquisition unit 22 Image transmission unit 30 Electronic equipment 31 Operation information transmission unit 32 Equipment control section [Prior art documents] [Patent documents]

[0263] [Patent Document 1] Patent No. 6064636 [Patent Document 2] Patent No. 5238409

Claims

1. An information processing apparatus capable of communicating with a first electronic device and a second electronic device via a network, a first image receiving unit that receives a first image of a first user present at an operable position of the first electronic device; a first user identifying unit that identifies the first user based on the first image; an input information receiving unit that receives input information input by the first user on the first electronic device from the first electronic device; a second image receiving unit that receives a second image of a second user present at an operable position of the second electronic device; a second user identifying unit that identifies the second user based on the second image; a transmitting unit that transmits information to the second electronic device based on the input information when the first user identified by the first user identifying unit matches the second user identified by the second user identifying unit; An information processing apparatus comprising:

2. The input information includes usage information of the first user using the first electronic device, The information transmitted by the transmitting unit includes the input information, The information processing apparatus according to claim 1.

3. The input information includes authentication information, The information processing apparatus according to claim 1.

4. The information transmitted by the transmitting unit includes the input information, The information processing apparatus according to claim 1.

5. The information transmitted by the transmitting unit includes an instruction for an operation in the second electronic device, The information processing apparatus according to claim 1.

6. The information processing apparatus further comprises a state management unit that records position information of the first user and the second user in time series using the first image and the second image. The information processing apparatus according to claim 1.

7. The state management unit identifies the first user who operated the first electronic device based on the time series of the position information. The information processing apparatus according to claim 6.

8. The state management unit records the position information using a plurality of the first images and the second images. The information processing apparatus according to claim 7.

9. The input information receiving unit further receives state information representing the state of the second electronic device, The transmitting unit does not transmit the information to the second electronic device whose state information represents a standby state. The information processing apparatus according to claim 1.

10. An information processing system in which a first electronic device, a second electronic device, and an information processing apparatus can communicate via a network, wherein the information processing apparatus, a first image receiving unit that receives a first image of a first user present at an operable position of the first electronic device, a first user specifying unit that specifies the first user based on the first image, an input information receiving unit that receives input information input by the first user on the first electronic device from the first electronic device, a second image receiving unit that receives a second image of a second user present at an operable position of the second electronic device, a second user specifying unit that specifies the second user based on the second image, and a transmission unit that transmits information to the second electronic device based on the input information when the first user specified by the first user specifying unit matches the second user specified by the second user specifying unit. An information processing system comprising the above.

11. A computer capable of communicating with a first electronic device and a second electronic device via a network, a first image receiving procedure for receiving a first image of a first user present at an operable position of the first electronic device, a first user specifying procedure for specifying the first user based on the first image, an input information receiving procedure for receiving input information input by the first user on the first electronic device from the first electronic device, a second image receiving procedure for receiving a second image of a second user present at an operable position of the second electronic device, a second user specifying procedure for specifying the second user based on the second image, and a transmission procedure for transmitting information to the second electronic device based on the input information when the first user specified by the first user specifying procedure matches the second user specified by the second user specifying procedure. A device management method for executing the above.

12. On a computer capable of communicating with a first electronic device and a second electronic device via a network, a first image receiving procedure for receiving a first image of a first user present at an operable position of the first electronic device, a first user specifying procedure for specifying the first user based on the first image, an input information receiving procedure for receiving input information input by the first user on the first electronic device from the first electronic device, A second image receiving procedure for receiving a second image that captures a second user present at an operable position of the second electronic device; A second user identifying procedure for identifying the second user based on the second image; A transmission procedure for transmitting information to the second electronic device based on the input information when the first user identified by the first user identifying procedure matches the second user identified by the second user identifying procedure; A program for causing the above to be executed.

Images