Systems, methods, and programs

Abstract

A user operating an electronic device transmits usage information based on that operation to a nearby payment device. [Solution] The information processing device is an information processing device that can communicate with an electronic device, a payment device and a shooting device via a network, and comprises: a first image receiving unit that receives a first image taken by the shooting device of the user of the electronic device and the electronic device; a first user identification unit that identifies the first user from the first image; a usage information receiving unit that receives usage information from the electronic device based on the operation of the first user identified by the first user identification unit; a second image receiving unit that receives a second image taken by the shooting device of the payment device and a second user who can use the payment device; a second user identification unit that identifies the second user from the second image; and a usage information transmitting unit that transmits usage information to the payment device based on the first user identified by the first user identification unit and the second user identified by the second user identification unit.

Description

Technical Field

[0001] This invention relates to an information processing apparatus, an information processing system, a fee management method, and a program.

Background Art

[0002] For example, in a retail store such as a convenience store or a supermarket, there is a technology for realizing a unmanned store by tracking a user using the video of a camera installed in the store. For example, Patent Document 1 discloses a unmanned store system that identifies a user by performing face authentication at the time of entering the store, at the time of settlement, and at the time of leaving the store, and calculates the price of a product by object recognition at the time of settlement.

[0003] On the other hand, in a retail store, there may be installed electronic devices such as a digital multifunction machine or a multimedia terminal. Usually, since operation of these electronic devices generates usage fees, means for settling the usage fees is provided. For example, Patent Document 2 discloses an image forming apparatus that can settle usage fees at a POS (Point Of Sales) terminal in a store.

[0004] However, in the prior art, it is difficult to settle including usage fees for which a user has operated an electronic device in a retail store. For example, in the technology disclosed in Patent Document 1, the price of a product is calculated by object recognition, but it is difficult to recognize complex operations of an electronic device from video or the like and calculate usage fees. Also, for example, in a store where unspecified users come to the store, it is difficult to determine the identity between a user who has operated an electronic device and a user during settlement.

Summary of the Invention

Problems to be Solved by the Invention

[0005] An embodiment of this invention aims to transmit usage information based on an operation to a settlement device near a user who has operated an electronic device.

Means for Solving the Problems

[0006] To solve the above problems, an information processing device, which is one embodiment of the present invention, is an information processing device that can communicate with an electronic device, a payment device, and a shooting device via a network, and comprises: a first image receiving unit that receives a first image taken by the shooting device of the user of the electronic device and the electronic device; a first user identification unit that identifies the first user from the first image; a usage information receiving unit that receives usage information from the electronic device based on the operation of the first user identified by the first user identification unit; a second image receiving unit that receives a second image taken by the shooting device of the payment device and a second user who can use the payment device; a second user identification unit that identifies the second user from the second image; and a usage information transmitting unit that transmits usage information to the payment device based on the first user identified by the first user identification unit and the second user identified by the second user identification unit. [Effects of the Invention]

[0007] According to one embodiment of this invention, a user who operates an electronic device can transmit usage information based on that operation to a nearby payment device. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the overall configuration of a billing management system in one embodiment. [Figure 2] This figure shows an example of a computer hardware configuration in one embodiment. [Figure 3] This figure shows an example of the hardware configuration of a 360-degree imaging device in one embodiment. [Figure 4] This figure shows an example of the hardware configuration of an MFP in one embodiment. [Figure 5] This figure shows an example of the functional configuration of a billing management system in one embodiment. [Figure 6] This figure shows an example of state management information in one embodiment. [Figure 7] This figure shows an example of a first basic flowchart in one embodiment. [Figure 8] This figure shows an example of a second basic flowchart in one embodiment. [Figure 9] This figure shows an example of a fee management method in one embodiment. [Figure 10] This figure shows an example of the overall configuration of a billing management system that includes multiple monitoring devices. [Figure 11] This figure shows an example of the functional configuration of a billing management system that includes multiple monitoring devices. [Figure 12] This figure shows an example of object state update processing in one embodiment. [Figure 13] This figure shows an example of the process for creating a list of detected objects in one embodiment. [Figure 14] This figure shows an example of the process for updating the registered object list in one embodiment. [Figure 15] This figure shows an example of the process for updating the registered object list in one embodiment. [Figure 16] This figure shows a first example of the group ID assignment process in one embodiment. [Figure 17] This figure shows a second example of the group ID assignment process in one embodiment. [Figure 18] This figure shows an example of equipment status update processing in one embodiment. [Figure 19] This figure shows an example of the output information determination process in one embodiment. [Figure 20] This figure shows an example of the output condition determination process in one embodiment. [Figure 21] This figure shows an example of device output information in one embodiment. [Modes for carrying out the invention]

[0009] The embodiments of this invention will be described in detail below with reference to the drawings. Note that components having the same function are numbered the same in the drawings, and redundant explanations are omitted.

[0010] [Embodiment] One embodiment of the present invention is a fee management system that performs settlement including usage fees based on operations of electronic devices installed in a store. In the fee management system of this embodiment, when a person who has operated an electronic device is near the settlement device, the settlement device presents usage fees based on the operations performed by the person on the electronic device.

[0011] For example, in the unmanned store system disclosed in Patent Document 1, the price is calculated by object recognition of an image of goods on the settlement counter. However, since complex operations are performed on the electronic devices installed in the store, it is difficult to calculate usage fees by video recognition. For example, when a copying operation is performed on a digital multifunction device, the usage fees vary depending on the mode such as black and white or color. Since the previous settings may be carried over in these operations, it is impossible to calculate usage fees without recognizing the entire operation of the user, which is not practical.

[0012] The electronic device itself knows the usage fees, and if the electronic device has a communication function, it is possible to transmit fee information representing the usage fees to the settlement device. However, the settlement device cannot determine the identity between the user who has operated on the electronic device and the user who is in the process of settlement. In the invention disclosed in Patent Document 1, the user is identified by face recognition from entry to exit, but prior registration of the user is required. In a store where unspecified users come, it is difficult to determine the identity between the user who has operated the electronic device and the user who is near the settlement device.

[0013] In the fee management system of this embodiment, the management device tracks a user moving within the space based on an image of the space where the electronic device and the settlement device are installed. Thereby, the management device can determine the identity between the user who has operated the electronic device and the user who is near the settlement device.

[0014] If the user's identity can be determined, the usage fee based on the operation of the electronic device can be displayed from a payment device located near that user. If usage fees incurred by the electronic device can be settled at a payment device, there is no need to provide a payment method for each electronic device, thus reducing capital investment. In addition, users do not need to settle usage fees each time they operate an electronic device, making it highly convenient.

[0015] <Overall structure of the billing management system> First, the overall configuration of the billing management system in this embodiment will be explained with reference to Figure 1. Figure 1 is a diagram showing an example of the overall configuration of the billing management system in this embodiment.

[0016] As shown in Figure 1, the billing system 1 in this embodiment includes a management device 10, one or more monitoring devices 20, one or more electronic devices 30, and a settlement device 40. The management device 10, monitoring devices 20, electronic devices 30, and settlement device 40 are each connected to a communication network N1.

[0017] The communication network N1 is configured so that each connected device can communicate with one another. The communication network N1 is constructed using wired communication networks such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

[0018] The communication network N1 may include not only wired communication but also wireless communication such as Wi-Fi or short-range wireless communication, or mobile communication networks such as WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), or 5G (5th Generation).

[0019] The monitoring device 20, electronic equipment 30, and settlement device 40 are installed in the managed space R1. The managed space R1 may be a single space such as a room in a building, or it may be multiple spaces connected by an accessible space such as a door or corridor.

[0020] An example of a managed space R1 is a sales area in a small store such as a convenience store. Another example of a managed space R1 is a hotel lobby or guest room. A managed space R1 is not limited to these, and can be any space where electronic devices that may incur usage fees through operation are installed.

[0021] The management device 10 is an information processing device such as a PC (Personal Computer), workstation, or server that manages the electronic equipment 30 and the payment device 40. Based on the images received from the monitoring device 20 and the fee information received from the electronic equipment 30, the management device 10 transmits operation instructions to the payment device 40. An example of the management device 10 is a computer.

[0022] The monitoring device 20 is an electronic device that acquires images including the vicinity of the electronic devices 30 and the payment device 40 installed in the managed space R1. The monitoring device 20 may acquire video (i.e., a time series of images). The monitoring device 20 is installed in a position where it can photograph all of the multiple electronic devices 30 and the payment device 40.

[0023] One example of the monitoring device 20 is a 360-degree camera. Another example of the monitoring device 20 is a set of multiple network cameras. When the monitoring device 20 consists of multiple network cameras, each network camera is positioned with its field of view adjusted so that there are no blind spots within the managed space R1.

[0024] Hereafter, when there are multiple monitoring devices 20, they will be distinguished using sub-numbers such as "monitoring device 20-1," "monitoring device 20-2," etc.

[0025] Electronic equipment 30 refers to various types of electronic equipment used by the user. An example of electronic equipment 30 is an image forming apparatus (printer, fax machine, MFP (Multifunction Peripheral / Product / Printer: digital multifunction device), scanner, etc.). Another example of electronic equipment 30 is a multimedia terminal, coffee machine, and product recognition device, etc.

[0026] In this embodiment, the product recognition device is installed on or near the product shelf. The product recognition device recognizes products that a user has taken from or returned to the product shelf, for example, using a weight sensor, infrared sensor, or image sensor. The product recognition device also outputs price information related to the products taken from the product shelf.

[0027] Hereafter, when distinguishing between multiple electronic devices 30, they will be described using sub-numbers such as "electronic device 30-1," "electronic device 30-2," etc.

[0028] The settlement device 40 is an information processing device that settles the price of goods purchased and / or charges for services used by customers of a store. The settlement device 40 displays the amount to be charged to the customer on a display or the like and processes the settlement using the money received from the customer. The settlement device 40 may be externally connected to a barcode scanner, payment terminal, cash receiver, etc., as needed. One example of the settlement device 40 is a computer. Another example of the settlement device 40 is a cash register.

[0029] Furthermore, the electronic device 30 is not limited to an image forming apparatus, as long as it is a device equipped with communication functions. In other words, the electronic device 30 may be, for example, a projector, an output device such as a digital signage display, a head-up display (HUD) device, industrial machinery, an imaging device, a sound collection device, medical equipment, networked home appliances, an automobile (connected car), a personal computer (notebook PC), a mobile phone, a smartphone, a tablet device, a game console, a personal digital assistant (PDA), a digital camera, a wearable PC, or a desktop PC.

[0030] <Hardware configuration of the billing management system> Next, the hardware configuration of each device included in the billing management system in this embodiment will be described with reference to Figures 2 to 4.

[0031] Computer Hardware Configuration Figure 2 shows an example of a hardware configuration when the management device 10 and the settlement device 40 are implemented using a computer.

[0032] As shown in Figure 2, a computer in one embodiment includes a CPU (Central Processing Unit) 501, ROM (Read Only Memory) 502, RAM (Random Access Memory) 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, DVD-RW (Digital Versatile Disk Rewritable) drive 514, and media I / F 516.

[0033] Of these components, the CPU 501 controls the operation of the entire computer. The ROM 502 stores programs used to drive the CPU 501, such as the IPL (Initial Program Loader). The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501.

[0034] The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection interface 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network interface 509 is an interface for data communication using the communication network N1. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 2.

[0035] The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, and various instructions. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting processing targets, and moving the cursor. The DVD-RW drive 514 controls the reading or writing of various data to the DVD-RW 513, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may also be DVD-R, etc. The media interface 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0036] ≪Hardware configuration of the 360-degree imaging system≫ Figure 3 shows an example of the hardware configuration when the monitoring device 20 is implemented as a 360-degree imaging device. In the following, the 360-degree imaging device is assumed to be a 360-degree (omnidirectional) imaging device using two image sensors, but any number of image sensors other than two is acceptable. Furthermore, it is not necessarily required to be a device dedicated to omnidirectional imaging; a standard digital camera or smartphone can be retrofitted with an omnidirectional imaging unit to achieve essentially the same functionality as a 360-degree imaging device.

[0037] As shown in Figure 3, the 360-degree imaging device in one embodiment consists of an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection interface 616, a communication unit 617, an antenna 617a, an acceleration / direction sensor 618, a gyroscope sensor 619, an acceleration sensor 620, and a recessed terminal 621 for Micro USB.

[0038] Of these, the imaging unit 601 includes wide-angle lenses (so-called fisheye lenses) 602a and 602b, each having a field of view of 180° or more for forming hemispherical images, and two image sensors 603a and 603b, each corresponding to the wide-angle lens. The image sensors 603a and 603b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the fisheye lenses 602a and 602b into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters necessary for the operation of the image sensors are set.

[0039] The image sensors 603a and 603b of the imaging unit 601 are each connected to the image processing unit 604 via a parallel I / F bus. On the other hand, the image sensors 603a and 603b of the imaging unit 601 are connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to the CPU 611 via bus 610. Furthermore, ROM 612, SRAM 613, DRAM 614, operation unit 615, external device connection I / F (Interface) 616, communication unit 617, and acceleration / direction sensor 618 are also connected to bus 610.

[0040] The image processing unit 604 receives image data output from image sensors 603a and 603b via a parallel I / F bus, performs predetermined processing on each image data, and then combines these image data to create equirectangular projection image data.

[0041] The imaging control unit 605 generally uses the I2C bus to set commands and other information in the registers of the image sensors 603a and 603b, with the imaging control unit 605 acting as the master device and the image sensors 603a and 603b as slave devices. It receives necessary commands and other information from the CPU 611. The imaging control unit 605 also uses the I2C bus to acquire status data and other information from the registers of the image sensors 603a and 603b and send it to the CPU 611.

[0042] Furthermore, the imaging control unit 605 instructs the image sensors 603a and 603b to output image data when the shutter button on the operation unit 615 is pressed. Some 360-degree imaging devices may also have functions to display a preview image on a display (for example, a smartphone display) or to display video. In this case, the image data output from the image sensors 603a and 603b is performed continuously at a predetermined frame rate (frames / minute).

[0043] Furthermore, as will be described later, the imaging control unit 605 also functions as a synchronization control means that works in cooperation with the CPU 611 to synchronize the output timing of image data from the image sensors 603a and 603b. In this embodiment, the 360-degree imaging device is not provided with a display, but a display unit may be provided.

[0044] Microphone 608 converts sound into sound (signal) data. Sound processing unit 609 receives the sound data output from microphone 608 through the I / F bus and performs predetermined processing on the sound data.

[0045] The CPU 611 controls the overall operation of the 360-degree imaging system and performs necessary processing. The ROM 612 stores various programs for the CPU 611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611 and data in progress. In particular, the DRAM 614 stores image data in progress from the image processing unit 604 and data of completed equirectangular projection images.

[0046] The control unit 615 is a collective term for the operation buttons, such as the shutter button 615a. The user inputs various shooting modes and shooting conditions by operating the control unit 615.

[0047] The external device connection interface 616 is an interface for connecting various external devices. These external devices include, for example, USB (Universal Serial Bus) memory and PCs (Personal Computers). The equirectangular projection image data stored in the DRAM 614 can be recorded to external media via the external device connection interface 616, or transmitted to external terminals (devices) such as smartphones via the external device connection interface 616 as needed.

[0048] The communication unit 617 communicates with external terminals (devices) such as smartphones via an antenna 617a provided on the 360-degree imaging device, using short-range wireless communication technologies such as Wi-Fi, NFC (Near Field Communication), and Bluetooth (registered trademark). This communication unit 617 can also transmit equirectangular projection image data to external terminals (devices) such as smartphones.

[0049] The acceleration / direction sensor 618 calculates the direction of the 360-degree camera from the Earth's magnetic field and outputs direction information. This direction information is an example of related information (metadata) in accordance with Exif and is used for image processing such as image correction of captured images. The related information also includes the date and time the image was captured and the data size of the image data. The acceleration / direction sensor 618 is also a sensor that detects changes in angle (Roll angle, Pitch angle, Yaw angle) associated with the movement of the 360-degree camera 6. Changes in angle are an example of related information (metadata) in accordance with Exif and are used for image processing such as image correction of captured images.

[0050] Furthermore, the acceleration / direction sensor 618 is a sensor that detects acceleration in three axes. The 360-degree imaging device calculates its orientation (angle relative to the direction of gravity) based on the acceleration detected by the acceleration / direction sensor 618. By providing the acceleration / direction sensor 618 to the 360-degree imaging device, the accuracy of image correction is improved.

[0051] ≪MFP Hardware Configuration≫ Figure 4 shows an example of a hardware configuration when the electronic device 30 is implemented using an MFP.

[0052] As shown in Figure 4, one embodiment of the MFP includes a controller 910, a short-range communication circuit 920, an engine control unit 930, an operation panel 940, and a network interface 950.

[0053] Of these, the controller 910 includes the main components of the computer: the CPU (Central Processing Unit) 901, system memory (MEM-P) 902, northbridge (NB) 903, southbridge (SB) 904, ASIC (Application Specific Integrated Circuit) 906, local memory (MEM-C) 907, HDD (Hard Disk Drive) controller 908, and HD (Hard Disk) 909, with the NB 903 and ASIC 906 connected by an AGP (Accelerated Graphics Port) bus 921.

[0054] Of these, the CPU901 is a control unit that performs overall control of the MFP. The NB903 is a bridge that connects the CPU901 to the MEM-P902, SB904, and AGP bus 921, and has a memory controller that controls reading and writing to the MEM-P902, as well as a PCI (Peripheral Component Interconnect) master and an AGP target.

[0055] MEM-P902 consists of a ROM (Read Only Memory) 902a, which is a memory for storing programs and data that realize the various functions of the controller 910, and a RAM (Random Access Memory) 902b, which is used for program and data deployment and drawing during memory printing. The programs stored in RAM 902b may be configured to be provided as installable or executable files recorded on a computer-readable recording medium such as a CD-ROM, CD-R, or DVD.

[0056] SB904 is a bridge for connecting NB903 to PCI devices and peripherals. ASIC906 is an integrated circuit (IC) for image processing applications that has hardware elements for image processing, and acts as a bridge connecting the AGP bus 921, PCI bus 922, HDD908, and MEM-C907, respectively. This ASIC906 consists of a PCI target and AGP master, an arbiter (ARB) that forms the core of the ASIC906, a memory controller that controls the MEM-C907, multiple DMACs (Direct Memory Access Controllers) that perform image data rotation using hardware logic, and a PCI unit that performs data transfer via PCI bus 922 between the scanner unit 931 and the printer unit 932. Note that the ASIC906 may also be connected to a USB (Universal Serial Bus) interface or an IEEE1394 (Institute of Electrical and Electronics Engineers 1394) interface.

[0057] MEM-C907 is local memory used as a copy image buffer and code buffer. HD909 is storage for storing image data, font data used during printing, and forms. HD909 controls data reading or writing to it according to the control of CPU901. The AGP bus 921 is a bus interface for graphics accelerator cards proposed to speed up graphics processing, and by directly accessing MEM-P902 with high throughput, the graphics accelerator card can be made faster.

[0058] Furthermore, the short-range communication circuit 920 is equipped with a short-range communication circuit 920a. The short-range communication circuit 920 is a communication circuit such as NFC or Bluetooth.

[0059] Furthermore, the engine control unit 930 is composed of a scanner unit 931 and a printer unit 932. The operation panel 940 includes a panel display unit 940a, such as a touch panel, which displays current settings and selection screens and accepts input from the operator, and an operation panel 940b, which consists of a numeric keypad that accepts setting values ​​for image formation conditions such as density settings and a start key that accepts a copy start command. The controller 910 controls the entire MFP, for example, controlling drawing, communication, and input from the operation panel 940. The scanner unit 931 or the printer unit 932 includes an image processing section, such as error diffusion and gamma conversion.

[0060] Furthermore, the MFP allows you to sequentially switch between document box function, copy function, printer function, and facsimile function using the application switching key on the control panel 940. When the document box function is selected, it enters document box mode; when the copy function is selected, it enters copy mode; when the printer function is selected, it enters printer mode; and when the facsimile mode is selected, it enters facsimile mode.

[0061] Furthermore, the network interface 950 is an interface for data communication using the communication network N1. The short-range communication circuit 920 and the network interface 950 are electrically connected to the ASIC 906 via the PCI bus 922.

[0062] <Functional Configuration of the Billing Management System> Next, an example of the functional configuration of the billing management system in this embodiment will be described with reference to Figure 5. Figure 5 is a block diagram illustrating the functional configuration of each device included in the billing management system in this embodiment.

[0063] ≪Functional Configuration of the Management Device≫ As shown in Figure 5, the management device 10 in this embodiment includes an image receiving unit 11, an operation information receiving unit 12, a state management unit 13, an operation instruction unit 14, and a state storage unit 100. The state management unit 13 in this embodiment includes an object state management unit 131, an equipment state management unit 132, and an output information determination unit 133.

[0064] The image receiving unit 11, operation information receiving unit 12, state management unit 13, and operation instruction unit 14 are realized, for example, by a process in which a program deployed from the HD 504 shown in Figure 2 onto the RAM 503 causes the CPU 501 and HDD controller 505 to execute.

[0065] The state storage unit 100 is implemented, for example, using the HD504 shown in Figure 2. Reading or writing data stored in the HD504 is performed, for example, via the HDD controller 505.

[0066] The state storage unit 100 stores state management information for managing the state of equipment and objects present in the managed space R1. In this embodiment, the state management information includes a registered object list, a registered equipment list, and a detected object list.

[0067] The registered object list is a list that manages information about objects present in the managed space R1. In this embodiment, the object is a person.

[0068] The registered equipment list is a list that manages information about equipment present in the managed space R1. In this embodiment, the equipment consists of electronic equipment 30 and payment equipment 40.

[0069] The detected object list is a list that manages objects detected in the managed space R1. The detected object list is a temporary list used to update the registered object list.

[0070] Here, the state management information in this embodiment will be explained with reference to Figure 6. Figure 6(A) is a conceptual diagram showing an example of a registered object list.

[0071] As shown in Figure 6(A), the registered object list in this embodiment includes, as data items, a registered object ID, a group ID, an existence confirmation flag, attribute information, and equipment operation information.

[0072] The registered object ID is identification information that identifies an object included in the registered object list.

[0073] A group ID is an identifier that identifies a group to which multiple registered objects belong. A group ID is assigned when it is determined that some of the registered objects in the registered object list belong to the same group. For registered objects that are not determined to belong to a group, the group ID is set to an initial value (e.g., zero).

[0074] A group can be, for example, colleagues, parents and children, or friends. Members of a group often work together to perform tasks such as shopping. Therefore, recognizing a group can be expected to provide a more accurate understanding of an individual's behavior. For example, when parents and children are shopping together, it would be highly convenient if the parent could pay for the MFP (Multifunction Printer) usage fees for the child's photocopying.

[0075] The existence confirmation flag indicates whether or not the registered object exists in the managed space R1. For example, the existence confirmation flag is set to 1 if it exists, and 0 if it does not.

[0076] Attribute information is information associated with a registered object and is necessary for its management. In this embodiment, it is the position of the registered object and the time when that position was confirmed (hereinafter also referred to as the "image acquisition time"). The position of the registered object is represented by three-dimensional coordinates (i.e., numerical values ​​on each axis in a Cartesian coordinate system such as the X, Y, and Z axes).

[0077] The method for acquiring location information varies depending on the type of monitoring device 20. For example, if the monitoring device 20 is a network camera, known techniques can be used, such as acquiring the distance to an object with a stereo camera and mapping it to three-dimensional coordinates along with its direction.

[0078] Furthermore, attribute information may include various pieces of information that describe the characteristics of the registered object, in addition to its location. For example, the color or shape of the registered object can be used. If the object is a person, the shape changes depending on their posture, so it is preferable to use color. Since the number of different colors or the area of ​​each color also changes depending on the posture, it is preferable to use the condition that at least one color is continuous when determining whether objects are the same.

[0079] Equipment operation information is information about equipment operations performed by a registered object. Equipment operation information may include usage information that describes how the electronic device was used. For example, if the electronic device is an MFP, the usage information may include information such as "5 color copies were made and 10 monochrome copies were made." Equipment operation information may also include fee information that describes the charges incurred during the equipment operation. Equipment operation information may include, for example, the functions performed by the user on the electronic device, the time of operation, usage information describing the content of the operation, fee information as compensation for the operation, and the language used.

[0080] If the equipment operation information includes charge information, a paid flag is added to each charge information entry. The paid flag is a boolean value indicating whether or not the charge information has been paid. The initial value of the paid flag is 0 (=false), indicating that it is unpaid.

[0081] The number of objects in the managed space R1 changes constantly. Therefore, the number of registered objects included in the registered object list is variable.

[0082] Figure 6(B) is a conceptual diagram showing an example of a registered device list. As shown in Figure 6(B), the registered device list in this embodiment has the following data items: registered device ID, attribute information, device input information, and device output information.

[0083] The registered device ID is identification information that identifies the device included in the registered device list.

[0084] Attribute information is information associated with the registered device and is necessary for its management. In this embodiment, this includes the location of the registered device, power on / off status, whether it is in sleep mode or not, whether authentication is required, whether options are installed, and status information such as maintenance in progress / scheduled maintenance time / energy saving mode. The location of the registered device is represented by three-dimensional coordinates, similar to the registered object.

[0085] Device input information is predetermined information that is managed when a registered device is operated. Device input information may include fee information representing the charges incurred during the operation of the device. Examples of device input information include the functions performed by the user on the registered device, the time of operation, fee information to be charged as compensation for the operation, and the language used.

[0086] Device output information is information pre-registered for registered devices. It is a list that associates the output conditions that trigger the transmission of an operation command with the operation command to be transmitted to the registered device when those output conditions are met.

[0087] The number of devices installed in the managed space R1 may change. Therefore, the number of registered devices included in the registered device list is variable.

[0088] Figure 6(C) is a conceptual diagram showing an example of a detection object list. As shown in Figure 6(C), the detection object list in this embodiment has a detection object ID, a group ID, and attribute information as data items.

[0089] The detected object ID is identification information that identifies an object included in the detected object list.

[0090] A group ID is an identifier that identifies a group containing multiple detected objects. A group ID is assigned when it is determined that some of the detected objects in the detected object list belong to the same group. For detected objects that are not determined to belong to a group, the group ID is set to an initial value (e.g., zero).

[0091] Attribute information is information associated with the detected object and is necessary for management. In this embodiment, it is the position of the detected object and the time of image acquisition. The position of the detected object is represented by three-dimensional coordinates, similar to registered objects.

[0092] The number of objects detected in the managed space R1 changes moment by moment. Therefore, the number of detected objects included in the detected object list is variable.

[0093] Let's return to Figure 5 for explanation. The image receiving unit 11 receives an image from the monitoring device 20. The image receiving unit 11 sends the received image to the status management unit 13 in response to a request from the status management unit 13.

[0094] The operation information receiving unit 12 receives device operation information from the electronic device 30. The operation information receiving unit 12 sends the received device operation information to the status management unit 13 in response to a request from the status management unit 13. In this embodiment, the device operation information includes charge information based on the device operation.

[0095] The status management unit 13 manages the status of the electronic equipment 30 and objects using the object status management unit 131 and the equipment status management unit 132. The status management unit 13 also determines the content of the operation instructions to be transmitted to the settlement device 40 using the output information determination unit 133.

[0096] The object state management unit 131 updates the registered object list stored in the state storage unit 100 based on the image received from the image receiving unit 11.

[0097] The device status management unit 132 updates the registered device list and registered object list stored in the status storage unit 100 based on the device operation information received from the operation information receiving unit 12.

[0098] The output information determination unit 133 determines the content of the operation instruction to be transmitted to the settlement device 40 based on the state management information stored in the state storage unit 100. The output information determination unit 133 sends the determined operation instruction to the operation instruction unit 14.

[0099] The operation instruction unit 14 transmits the operation instruction received from the status management unit 13 to the settlement device 40. In this embodiment, the operation instruction includes charge information contained in the equipment operation information.

[0100] ≪Functional Configuration of Monitoring Devices≫ As shown in Figure 5, the monitoring device 20 in this embodiment includes an image acquisition unit 21 and an image transmission unit 22.

[0101] The image acquisition unit 21 acquires an image that includes the vicinity of the electronic device 30 installed in the managed space R1. The image acquisition unit 21 is implemented, for example, by a process in which a program loaded from the ROM 612 shown in Figure 3 onto the SRAM 613 causes the CPU 611 and the imaging control unit 605 to execute.

[0102] The image transmission unit 22 transmits the image acquired by the image acquisition unit 21 to the management device 10. The image transmission unit 22 is implemented, for example, by a process in which a program loaded from the ROM 612 shown in Figure 3 onto the SRAM 613 is executed by the CPU 611 and the external device connection I / F 616.

[0103] ≪Functional Configuration of Electronic Devices≫ As shown in Figure 5, the electronic device 30 in this embodiment includes an operation information transmission unit 31.

[0104] The operation information transmission unit 31 is implemented, for example, by a process in which a program deployed from the HD909 shown in Figure 4 onto the RAM902b causes the CPU901 and the network I / F950 to execute.

[0105] The operation information transmission unit 31 transmits equipment operation information related to the operations performed by the user to the management device 10.

[0106] ≪Functional Configuration of the Payment System≫ As shown in Figure 5, the payment device 40 in this embodiment includes a payment settlement unit 41.

[0107] The billing unit 41 is implemented, for example, by a program deployed from the HD 504 shown in Figure 2 onto the RAM 503, which is then executed by the CPU 501, display 506, and network I / F 509.

[0108] The fare settlement unit 41 receives operation instructions from the management device 10. Based on the received operation instructions, the fare settlement unit 41 controls the operation of the fare settlement device 40. For example, the fare settlement unit 41 presents the fare information included in the operation instructions to the user. The fare settlement unit 41 also settles the presented fare according to the user's operation.

[0109] <Processing procedure for fee management methods> Next, the billing management method performed by the billing management system in this embodiment will be described with reference to Figures 7 to 21.

[0110] ≪Basic Flowchart≫ The fee management method in this embodiment involves many nested loop processes that handle all combinations of each data item in one list and each data item in another list. Therefore, in this embodiment, a basic flowchart illustrating the framework of the nested loop processes is introduced, and the following explanation will focus on the processes in the basic flowchart. Note that each process in the basic flowchart is performed on one combination of two data items.

[0111] The basic flowchart in this embodiment includes two basic flowcharts. The first basic flowchart is used when processing two data items in different lists using a nested loop. The second basic flowchart is used when processing two data items in the same list using a nested loop.

[0112] Figure 7 is a flowchart showing an example of the first basic flowchart in this embodiment.

[0113] In step S101, list A is read. The number of data points in list A is N. In step S102, list B is read. The number of data points in list B is M.

[0114] In step S103, the variable n is initialized to 1. In step S104, the variable m is initialized to 1. In step S105, process A1 is executed.

[0115] In step S106, it is determined whether the variable m is equal to the number of data points M. This means whether all combinations of the nth data point in list A with all the data points in list B have been processed. If the variable m is different from the number of data points M (NO), proceed to step S107. If the variable m is equal to the number of data points M (YES), proceed to step S109.

[0116] In step S107, process A3 is executed. In step S108, the variable m is incremented. Then, the process returns to step S106.

[0117] In step S109, process A2 is executed. In step S110, it is determined whether the variable n is equal to the number of data points N. This means whether all combinations of all data points in list A with all data points in list B have been processed. If the variable n is different from the number of data points N (NO), proceed to step S111. If the variable n is equal to the number of data points N (YES), proceed to step S113.

[0118] In step S111, process A5 is executed. In step S112, the variable n is incremented. Then, the process returns to step S104.

[0119] In step S113, process A4 is executed. This completes the processing for all combinations of data included in list A and list B.

[0120] Figure 8 is a flowchart showing an example of a second basic flowchart in this embodiment.

[0121] In step S201, list A is read. The number of data points in list A is N.

[0122] In step S202, the variable n is initialized to 1. In step S203, n+1 is assigned to the variable m. In step S204, process B1 is executed.

[0123] In step S205, it is determined whether the variable m is equal to N. This means whether the combinations of the nth data item in list A with all the data items in list A have been processed. If the variable m is different from N (NO), proceed to step S206. If the variable m is equal to N (YES), proceed to step S208.

[0124] In step S206, process B3 is executed. In step S207, the variable m is incremented. Then, the process returns to step S204.

[0125] In step S208, process B2 is executed. In step S209, it is determined whether the variable n is equal to N-1. This means that all the data in list A have been processed in combination with all the other data in list A. If the variable n is different from N-1 (NO), proceed to step S210. If the variable n is equal to N-1 (YES), proceed to step S212.

[0126] In step S210, process B5 is executed. In step S211, the variable n is incremented. Then, the process returns to step S203.

[0127] In step S212, process B4 is executed. This completes the processing for all combinations of data included in list A.

[0128] ≪Method of managing fees≫ Figure 9 is a flowchart showing an example of a billing management method performed by the billing management system in this embodiment.

[0129] The billing management system repeatedly executes the flowchart shown in Figure 9 at predetermined time intervals. The time interval can be set arbitrarily, but for example, it is 3 seconds.

[0130] In step S1, the image acquisition unit 21 of the monitoring device 20 acquires a first image of the vicinity of the electronic device 30. The image acquisition unit 21 also acquires a second image of the vicinity of the payment device 40. The monitoring device 20 is installed so that the vicinity of the electronic device 30 and the payment device 40 installed in the managed space R1 is within its field of view. Therefore, the first image acquired by the image acquisition unit 21 captures the vicinity of the electronic device 30. The second image acquired by the image acquisition unit 21 captures the vicinity of the payment device 40.

[0131] The vicinity of the electronic device 30 or the payment device 40 refers to the range of locations where a user can use the electronic device 30 and the payment device 40 (hereinafter also referred to as the "usable location"). The usable location differs depending on the electronic device 30 or the payment device 40. Therefore, it is advisable to pre-set the usable location for each of the electronic device 30 and the payment device 40. Accordingly, the first image is a photograph of the electronic device 30 and the user using the electronic device 30. The second image is a photograph of the payment device 40 and the user using the payment device 40.

[0132] The monitoring device 20 may consist of two or more cameras. In this case, a first image may be acquired by the first camera, capturing the available location of the electronic device 30, and a second image may be acquired by the second camera, capturing the available location of the payment device 40.

[0133] Figure 10 shows an example of the overall configuration of a fare management system when the monitoring device 20 consists of two or more cameras. As shown in Figure 10, the fare management system 1 in this case includes a monitoring device 20-1 that photographs the available locations of the electronic device 30 and a monitoring device 20-2 that photographs the available locations of the payment device 40.

[0134] Figure 11 shows an example of the functional configuration of a fee management system when the monitoring device 20 is composed of two or more cameras. As shown in Figure 11, monitoring device 20-1 and monitoring device 20-2 are equipped with an image acquisition unit 21 and an image transmission unit 22, respectively.

[0135] If the usable locations of the electronic device 30 and the payment device 40 can be captured by a single camera, the usable locations of the electronic device 30 and the payment device 40 may be captured by that camera. In this case, the first image and the second image will be the same image.

[0136] Next, the image acquisition unit 21 sends the acquired first image and second image to the image transmission unit 22. The image transmission unit 22 receives the first image and second image from the image acquisition unit 21. Next, the image transmission unit 22 transmits the received first image and second image to the management device 10. In the management device 10, the image receiving unit 11 receives the first image and second image from the monitoring device 20.

[0137] If the monitoring device 20 consists of two or more cameras, the image acquisition unit 21 of monitoring device 20-1 sends the acquired first image to the image transmission unit 22. Next, the image transmission unit 22 of monitoring device 20-1 transmits the received first image to the management device 10. Also, the image acquisition unit 21 of monitoring device 20-2 sends the acquired second image to the image transmission unit 22. Next, the image transmission unit 22 of monitoring device 20-2 transmits the received second image to the management device 10. In the management device 10, the image receiving unit 11 receives the first image and the second image from monitoring device 20-1 and monitoring device 20-2.

[0138] In step S2, the state management unit 13 of the management device 10 requests the first image and the second image from the image receiving unit 11. Next, the state management unit 13 inputs the first image and the second image received from the image receiving unit 11 to the object state management unit 131.

[0139] Next, the object state management unit 131 executes the object state update process described later and updates the registered object list of state management information stored in the state storage unit 100. If there are multiple monitoring devices 20, the object state management unit 131 executes the object state update process for each of the multiple images received from each of the multiple monitoring devices 20.

[0140] ≪Object State Update Process≫ Here, the object state update process in this embodiment will be explained with reference to Figures 12 to 17. Figure 12 is a flowchart showing an example of the object state update process (step S2 in Figure 9) executed by the object state management unit 131 in this embodiment.

[0141] In step S21, the object state management unit 131 executes the detected object list creation process described later and creates a detected object list based on the image input from the state management unit 13.

[0142] Figure 13 is a flowchart showing an example of the process for creating a detected object list (step S21 in Figure 12) performed by the object state management unit 131 in this embodiment.

[0143] In step S21-1, the object state management unit 131 clears the detected object list stored in the state storage unit 100. That is, it deletes all data included in the detected object list.

[0144] In step S21-2, the object state management unit 131 acquires an image input from the state management unit 13. The image may be a single image received from one monitoring device 20, or multiple images received from multiple monitoring devices 20.

[0145] Next, the object state management unit 131 divides the image into predetermined blocks. Subsequently, the object state management unit 131 measures the distance between the object captured in the image and the camera for each block. The object state management unit 131 also calculates three-dimensional coordinates from the measured distances. A known method can be used to convert the distance to three-dimensional coordinates.

[0146] In step S21-3, the object state management unit 131 obtains the image acquisition time T from the acquired image. If the image acquisition time cannot be obtained from the image, the current time may be used as the image acquisition time T.

[0147] In step S21-4, the object state management unit 131 analyzes the acquired image and detects the objects captured in the image. In this embodiment, it detects people captured in the image.

[0148] Human detection can be performed using known methods. For example, pattern matching using machine learning models can be used. Specifically, known region-based convolutional neural networks (R-CNNs) can be used to extract humans from images.

[0149] Furthermore, the accuracy of the detection can be improved by pattern matching the extracted image of the person with a pre-stored comparison image. In addition, various other known person detection methods can also be used.

[0150] In step S21-5, the object state management unit 131 assigns a detected object ID to identify the detected object.

[0151] In step S21-6, the object state management unit 131 registers the detected object ID and attribute information (location of the detected object and image acquisition time) in the detected object list. At this stage, the group ID is set to an initial value indicating that it does not belong to a group.

[0152] Let's return to Figure 12 for explanation. In step S22, the object state management unit 131 executes the registered object list update process described later and updates the registered object list based on the detected object list.

[0153] Figures 14 and 15 are flowcharts showing an example of the registered object list update process (step S22 in Figure 12) performed by the object state management unit 131 in this embodiment.

[0154] The registered object list update process is performed within the framework of the first basic flowchart (see Figure 7). Specifically, the detected object list is denoted as List A, and the registered object list as List B. Processes A1 (Figure 14) and A4 (Figure 15) are executed for all combinations of detected and registered objects. There are no processes corresponding to A2, A3, and A5.

[0155] Figure 14 is a flowchart showing an example of process A1 in the registered object list update process. Process A1 is a process for determining the identity between a registered object and a detected object and updating the position of the registered object.

[0156] Here, we will perform processing on the combination of the nth detected object in the detected object list and the mth registered object in the registered object list.

[0157] In step S22-1, the object state management unit 131 clears the existence confirmation flag for the m-th registered object. That is, it sets the existence confirmation flag for the m-th registered object to 0. This means that it is unknown whether or not the registered object exists in the managed space R1.

[0158] In step S22-2, the object state management unit 131 calculates the distance X between the three-dimensional coordinates of the m-th registered object and the three-dimensional coordinates of the n-th detected object.

[0159] In step S22-3, the object state management unit 131 determines whether the distance X is less than or equal to a predetermined threshold (for example, 1 meter). If the distance X exceeds the threshold (NO), the object state management unit 131 terminates process A1. This means that it has been determined that the two objects are not the same person because they are far apart. If the distance X is less than or equal to the threshold (YES), the object state management unit 131 proceeds to step S22-4.

[0160] In step S22-4, the object state management unit 131 adds the three-dimensional position and video acquisition time of the nth detected object to the attribute information of the mth registered object. This means that because the distance between the two is small, it is determined that they are the same person, and the position of the nth detected object is the position of the mth registered object at the image acquisition time T.

[0161] The attribute information in the registered object list increases each time the object state update process is performed, as long as the registered object is detected in the image. Conversely, the attribute information in the registered object list represents the time series of positions in which the registered object was detected within the managed space. Therefore, the movement of the registered object can be tracked using the attribute information in the registered object list.

[0162] In step S22-5, the object state management unit 131 sets the existence confirmation flag for the m-th registered object. That is, it sets the existence confirmation flag for the m-th registered object to 1. This means that since an object matching the registered object was detected in step S22-3, it was determined that the registered object was also present in the managed space R1 at the image acquisition time T.

[0163] Figure 15 is a flowchart showing an example of process A4 in the registered object list update process. Process A4 is the process of deleting registered objects from the registered object list whose existence could not be confirmed in process A1.

[0164] In step S22-6, the object state management unit 131 initializes the variable m to 1.

[0165] In step S22-7, the object state management unit 131 determines whether the existence of the m-th registered object has been confirmed. Specifically, it determines whether the existence confirmation flag is 1 or 0. If the existence confirmation flag is 1 (YES), the object state management unit 131 proceeds to step S22-9. If the existence confirmation flag is 0 (NO), the object state management unit 131 proceeds to step S22-8.

[0166] In step S22-8, the object state management unit 131 deletes the m-th registered object from the registered object list.

[0167] In step S22-9, the object state management unit 131 determines whether the variable m is equal to the number of data points M. If the variable m is different from the number of data points M (NO), the object state management unit 131 proceeds to step S22-10. If the variable m is equal to the number of data points M (YES), the object state management unit 131 terminates the process.

[0168] In step S22-10, the object state management unit 131 increments the variable m. After that, the object state management unit 131 returns to step S22-7.

[0169] Let's return to Figure 12 for explanation. In step S23, the object state management unit 131 executes the group ID assignment process described later and assigns a group ID to the registered objects included in the registered object list.

[0170] The group ID assignment process is performed within the framework of the second basic flowchart (see Figure 8). Specifically, the registered object list is designated as List A, and process B1 (Figure 16 or Figure 17) is executed for all combinations of two registered objects. Note that there are no processes corresponding to processes B2-B5.

[0171] Various methods can be considered for determining whether one object belongs to the same group as another object. In this embodiment, group determination by proximity and group determination by group behavior will be described. However, the group determination method is not limited to these, and any technology that enables group determination of objects from an image may be used.

[0172] Figure 16 is a flowchart showing a first example of the group ID assignment process (step S23 in Figure 12) performed by the object state management unit 131 in this embodiment. The first example of the group ID assignment process is group determination based on proximity.

[0173] Group determination based on proximity is a process that registers two objects detected in close proximity for a predetermined number of consecutive times as a group. The predetermined number of times is, for example, 5 times. If the time interval for executing the fee management method is 3 seconds, then two objects detected in close proximity for 15 seconds consecutively will be determined to be the same group.

[0174] Here, we will perform processing on the combination of the mth registered object and the nth registered object in the registered object list.

[0175] In step S23A-1, the object state management unit 131 initializes variables k and j to 1. Variable k is a counter that represents the number of times it has been determined whether or not two objects are in close proximity. Variable j is a counter that represents the number of times it has been determined that two objects are in close proximity.

[0176] In step S23A-2, the object state management unit 131 calculates the distance X between the three-dimensional coordinates of the m-th registered object k steps prior and the three-dimensional coordinates of the n-th registered object k steps prior.

[0177] In step S23A-3, the object state management unit 131 determines whether the distance X is less than a predetermined threshold (for example, 1 meter). If the distance X is less than the threshold (YES), the object state management unit 131 proceeds to step S23A-4. If the distance X is greater than or equal to the threshold (NO), the object state management unit 131 proceeds to step S23A-5.

[0178] In step S23A-4, the object state management unit 131 increments the variable j.

[0179] In step S23A-5, the object state management unit 131 increments the variable k.

[0180] In step S23A-6, the object state management unit 131 determines whether the variable k is equal to a predetermined number of times K. If the variable k is different from the predetermined number of times K (NO), the object state management unit 131 returns to step S23A-2. If the variable k is equal to the predetermined number of times K (YES), the object state management unit 131 proceeds to step S23A-7.

[0181] In step S23A-7, the object state management unit 131 determines whether the variable j is equal to a predetermined number of times K. If the variable j is different from the predetermined number of times K (NO), the object state management unit 131 terminates the process. If the variable j is equal to the predetermined number of times K (YES), the object state management unit 131 proceeds to step S23A-8.

[0182] In step S23A-8, the object state management unit 131 determines whether a group ID has been assigned to either the m-th registered object or the n-th registered object. If a group ID has been assigned (YES), the object state management unit 131 proceeds to step S23A-9. If a group ID has not been assigned (NO), the object state management unit 131 proceeds to step S23A-10.

[0183] In step S23A-9, the object state management unit 131 determines which of the group IDs assigned to the m-th registered object and the n-th registered object will be assigned as the group ID to be assigned. Next, the object state management unit 131 sets the determined group ID for the group IDs of the m-th registered object and the n-th registered object in the registered object list.

[0184] The object state management unit 131 determines the group ID to be assigned as follows: If a group ID has been assigned to only one registered object, that group ID will be used as the group ID to be assigned. In other words, registered objects that have not been assigned a group ID will be added as members of an existing group.

[0185] If both registered objects have a group ID, first, sort the m+1th and subsequent registered objects by group ID to identify the registered objects that share the same group ID as the mth registered object. Next, update the group IDs of the mth registered object and all identified registered objects with the group ID of the nth registered object. This ensures that all members of the group to which the mth registered object belongs also belong to the same group as the nth registered object. This allows us to identify groups that contain three or more registered objects.

[0186] In step S23A-10, the object status management unit 131 issues a new group ID that does not overlap with other group IDs. Next, the object status management unit 131 sets the newly issued group ID to the group IDs of the m-th and n-th registered objects in the registered object list.

[0187] Figure 17 is a flowchart showing a second example of the group ID assignment process (step S23 in Figure 12) executed by the object state management unit 131 in this embodiment. The second example of the group ID assignment process is group determination based on group actions.

[0188] Group determination by group action is a process that performs group determination by image analysis after creating a list of detected objects. Therefore, when performing group determination by group action, step S23 is executed between step S21 and step S22.

[0189] Here, we will perform processing on the combination of the mth detected object and the nth detected object in the list of detected objects.

[0190] In step S23B-1, the object state management unit 131 calculates the distance X between the three-dimensional coordinates of the nth detected object and the three-dimensional coordinates of the mth detected object.

[0191] In step S23B-2, the object state management unit 131 determines whether the distance X is less than a predetermined threshold (for example, 1 meter). If the distance X is greater than or equal to the threshold (NO), the object state management unit 131 terminates the process. If the distance X is less than the threshold (YES), the object state management unit 131 proceeds to step S23B-3.

[0192] In step S23B-3, the object state management unit 131 calculates the midpoint between the three-dimensional coordinates of the nth detected object and the three-dimensional coordinates of the mth detected object. Specifically, the object state management unit 131 divides the sum of the X, Y, and Z coordinates of the two detected objects by 2.

[0193] In step S23B-4, the object state management unit 131 extracts an image P with radius Y pixels centered on the midpoint. The radius Y is set according to the resolution of the image so that the area included in the image P is approximately 1m in real space.

[0194] In step S23B-5, the object state management unit 131 reads a pre-stored group action image. A group action image is a set of images representing actions that are determined to belong to the same group. For example, an image of one person putting an item into a shopping basket held by another person.

[0195] In step S23B-6, the object state management unit 131 calculates the similarity between the video P and the group action image using pattern matching or the like.

[0196] In step S23B-7, the object state management unit 131 determines whether any of the calculated similarities exceed a predetermined threshold. If there are similarities that exceed the threshold (YES), the object state management unit 131 proceeds to step S23B-8. If there are no similarities that exceed the threshold (NO), the object state management unit 131 terminates the process.

[0197] In step S23B-8, the object state management unit 131 issues a new group ID that does not overlap with any other group IDs. Next, the object state management unit 131 sets the newly issued group ID to the group IDs of the m-th detected object and the n-th detected object in the detected object list.

[0198] In step S22-4 (see Figure 14) of the subsequent registered object list update process, the object state management unit 131 sets the group ID of the detected object determined to be the same object as the registered object. If an existing group ID has been assigned to the registered object, steps S23A-8 to S23A-10 (see Figure 16) of the first example of the group ID assignment process are executed.

[0199] Let's return to Figure 9 for explanation. In step S3, the operation information transmission unit 31 of the electronic device 30-1 determines whether the operation performed by the user on the electronic device 30-1 is information that should be notified to the management device 10. This determination is made based on whether or not it matches one of the predetermined operation contents.

[0200] If the operation information transmission unit 31 determines that the operation content should be notified, it transmits equipment operation information related to the equipment operation to the management device 10. In the management device 10, the operation information receiving unit 12 receives the equipment operation information from the electronic device 30-1.

[0201] The operation content and operation information in this embodiment are illustrated below.

[0202] <Example 1> Operation details: Device login Operation information: Registered device ID, device authentication ID (if authentication is required), login time <Example 2> Operation details: Processes that incur charges (copying, outputting, use of pay-per-use software, etc.) Operation information: Registered device ID, device authentication ID (if authentication is required), fee, input data, language used <Example 3> Operation details: None Operation information: Status information such as maintenance or power saving mode. <Example 4> Operation details: The action of taking an item from the shelf. Operation information: Registered device ID, product identification information, product price

[0203] The registered device ID is identification information included in the registered device list stored in the state storage unit 100 of the management device 10. A registered device ID is assigned to each electronic device 30 installed in the managed space R1. The device authentication ID is authentication information used for authentication by the user of the electronic device 30 to use that electronic device 30.

[0204] In step S4, the status management unit 13 of the management device 10 requests operation information from the operation information receiving unit 12. Next, the status management unit 13 inputs the operation information received from the operation information receiving unit 12 to the equipment status management unit 132.

[0205] Next, the device status management unit 132 executes the device status update process described later and updates the registered device list of status management information stored in the status storage unit 100.

[0206] <<Equipment Status Update Process>> Here, the equipment status update process in this embodiment will be described with reference to Figure 18. Figure 18 is a flowchart showing an example of the equipment status update process (step S4 in Figure 9) executed by the equipment status management unit 132 in this embodiment.

[0207] In step S41, the device status management unit 132 identifies a registered device included in the registered device list based on the registered device ID contained in the device operation information. Next, the device status management unit 132 sets the received device operation information in the device input information of the identified registered device.

[0208] In step S42, the device status management unit 132 initializes the variable n to 1.

[0209] In step S43, the equipment status management unit 132 calculates the distance X between the three-dimensional coordinates of the identified registered equipment and the latest three-dimensional coordinates of the nth registered object. The three-dimensional coordinates of the registered equipment are set in advance using the same method as the three-dimensional coordinates of the registered objects. Since the installation location of the registered equipment may be moved, it is necessary to update the three-dimensional coordinates of the registered equipment periodically, but the update frequency may be low.

[0210] In step S44, the equipment status management unit 132 determines whether the distance X is less than a predetermined threshold (for example, 1 meter). If the distance X is greater than or equal to the threshold (NO), the equipment status management unit 132 proceeds to step S47. If the distance X is less than the threshold (YES), the equipment status management unit 132 proceeds to step S45.

[0211] In step S45, the equipment status management unit 132 adds the equipment input information of the identified registered equipment to the equipment operation information of the nth registered object. The equipment operation information in the registered object list is configured to store a predetermined number of equipment input information.

[0212] In step S46, the device status management unit 132 determines whether the variable n is equal to the number of data points N. If the variable n is different from the number of data points N (NO), the device status management unit 132 proceeds to step S47. If the variable n is equal to the number of data points N (YES), the device status management unit 132 terminates the process.

[0213] In step S47, the equipment status management unit 132 increments the variable n. Then, the equipment status management unit 132 returns to step S43.

[0214] Let's return to Figure 9 for explanation. In step S5, the output information determination unit 133 of the management device 10 executes the output information determination process described later and determines the operation instruction to be sent to the settlement device 40 based on the state management information stored in the state storage unit 100.

[0215] <<Output Information Determination Process>> Here, the output information determination process in this embodiment will be explained with reference to Figures 19 and 20. Figure 19 is a flowchart showing an example of the output information determination process (step S5 in Figure 9) executed by the output information determination unit 133 in this embodiment.

[0216] The output information determination process is carried out within the framework of the first basic flowchart (see Figure 7). Specifically, the registered equipment list is designated as List A, and the registered object list as List B. Process A1 (Figure 19) is executed for all combinations of registered equipment and registered objects. Note that there are no processes corresponding to A2-A5.

[0217] Here, processing will be performed on the combination of the nth registered device in the registered device list and the mth registered object in the registered object list.

[0218] In step S51, the output information determination unit 133 calculates the distance X between the latest three-dimensional coordinates of the nth registered device and the latest three-dimensional coordinates of the mth registered object.

[0219] In step S52, the output information determination unit 133 determines whether the distance X is less than a predetermined threshold (for example, 1 meter). If the distance X is greater than or equal to the threshold (NO), the equipment status management unit 132 terminates the process. If the distance X is less than the threshold (YES), the equipment status management unit 132 proceeds to step S53.

[0220] In step S53, the output information determination unit 133 obtains the output conditions included in the device output information of the nth registered device from the registered device list stored in the state storage unit 100.

[0221] In step S54, the output information determination unit 133 obtains the equipment operation information of the m-th registered object from the registered object list stored in the state storage unit 100.

[0222] In step S55, the output information determination unit 133 determines whether any of the equipment operation information acquired in step S54 satisfies the output conditions acquired in step S53. If none of the equipment operation information satisfies the output conditions (NO), the equipment status management unit 132 terminates the process. If any of the equipment operation information satisfies the output conditions (YES), the equipment status management unit 132 proceeds to step S56.

[0223] In step S56, the output information determination unit 133 obtains an operation instruction corresponding to the output condition acquired in step S53 from the device output information of the nth registered device. Note that the operation instruction for the registered device may include instructions that are processed adaptively according to the attribute information of the registered device. For example, it may include a condition that does not send an operation instruction when the registered device is in an inoperable state. An inoperable state is, for example, a power-off state or a hibernation state.

[0224] Figure 20 is a flowchart showing a modified example of the output condition determination process (step S55 in Figure 19) executed by the output information determination unit 133 in this embodiment.

[0225] In the output information determination process shown in Figure 19, step S55 determines whether the output conditions are met based solely on the equipment operation information of the m-th registered object. In the output information determination process shown in Figure 20, the output conditions are met by referring to the equipment operation information of devices other than the m-th registered device.

[0226] In step S55-1, the output information determination unit 133 determines whether any of the device operation information of the m-th registered object satisfies the output conditions of the n-th registered device. If any of the device operation information satisfies the output conditions (YES), the output information determination unit 133 proceeds to step S55-10. If none of the device operation information satisfies the output conditions (NO), the output information determination unit 133 proceeds to step S55-2.

[0227] In step S55-2, the output information determination unit 133 determines whether or not device operation information for other registered objects is required based on the output conditions of the nth registered device. If device operation information for other registered objects is required (YES), the output information determination unit 133 proceeds to step S55-3. If device operation information for other registered objects is not required (NO), the output information determination unit 133 proceeds to step S55-9.

[0228] In step S55-3, the output information determination unit 133 initializes the variable k to 1.

[0229] In step S55-4, the output information determination unit 133 determines whether variable k is equal to variable m. If variable k is equal to variable m (YES), the output information determination unit 133 proceeds to step S55-5. If variable k is not equal to variable m (NO), the output information determination unit 133 proceeds to step S55-7.

[0230] In step S55-5, the output information determination unit 133 obtains the equipment operation information of the k-th registered object from the registered object list stored in the state storage unit 100.

[0231] In step S55-6, the output information determination unit 133 determines whether any of the equipment operation information of the k-th registered object satisfies the output conditions of the n-th registered equipment. If any of the equipment operation information satisfies the output conditions (YES), the output information determination unit 133 proceeds to step S55-10. If none of the equipment operation information satisfies the output conditions (NO), the output information determination unit 133 proceeds to step S55-7.

[0232] In step S55-7, the output information determination unit 133 determines whether variable k is equal to the number of data points M. If variable k is different from the number of data points M (NO), the output information determination unit 133 proceeds to step 55-8. If variable m is equal to the number of data points M (YES), the output information determination unit 133 proceeds to step S55-9.

[0233] In step S55-8, the output information determination unit 133 increments the variable k. After that, the output information determination unit 133 returns to step S55-4.

[0234] In step S55-9, the output information determination unit 133 determines that the equipment operation information acquired in step S54 does not satisfy the output conditions acquired in step S53, and terminates the process.

[0235] In step S55-10, the output information determination unit 133 determines that the equipment operation information acquired in step S54 satisfies the output conditions acquired in step S53, and terminates the process.

[0236] Figure 21 shows an example of device output information in this embodiment. The first example of the output conditions for device output information in this embodiment is an output condition in which device operation information of other registered objects is not required. The second example of the output conditions for device output information in this embodiment is an output condition in which device operation information of other registered objects is required.

[0237] As shown in Figure 21, a first example of the output conditions in this embodiment is that there is device operation information that the m-th registered object (hereinafter also referred to as "person α") performed an operation on an electronic device (here, referred to as an MFP) that incurs charges, and the paid flag is 0 (=false), which indicates unpaid.

[0238] As shown in Figure 21, if it is determined that the first example of the output conditions is met, one or more of the following operation instructions will be determined to be sent to the registered device. The first operation instruction is to add the MFP usage fee to the billing amount presented to person α. ​​The second operation instruction is to instruct the management device 10 to update the paid flag in person α's device operation information to 1 (=true), which indicates that it has been paid.

[0239] As shown in Figure 21, a second example of the output conditions in this embodiment is that there is device operation information that the k-th registered object (hereinafter also referred to as "person β") performed an operation on an electronic device (here, MFP) that incurs charges, and the paid flag is 0 (=false), which indicates unpaid. Person β is assumed to have the same group ID as person α.

[0240] As shown in Figure 21, if it is determined that the second example of the output conditions is met, one or more of the following operation instructions will be determined to be sent to the registered device. The first operation instruction is to add the MFP usage fee for person β to the billing amount presented to person α. ​​The second operation instruction is to display an option indicating that "person β, not person α, will pay" the MFP usage fee for person β, along with a "yes" or "no" selection button. If "yes" is selected, the billing amount presented to person α is updated to the amount after subtracting the MFP usage fee for person β. The third operation instruction is to instruct the management device 10 to update the paid flag in person β's device operation information to 1 (=true), indicating that it has been paid, when person α has paid the billing amount including the MFP usage fee for person β.

[0241] Let's return to Figure 9 for explanation. In step S6, the output information determination unit 133 of the management device 10 sends operation instruction information representing an operation instruction to the operation instruction unit 14. The operation instruction information includes information indicating the determined operation instruction.

[0242] The operation instruction unit 14 receives operation instruction information from the output information determination unit 133. Next, the operation instruction unit 14 transmits the operation instruction included in the operation instruction information to the settlement device 40.

[0243] In the payment device 40, the payment unit 41 receives an operation instruction from the management device 10. Next, the payment unit 41 performs an operation using the payment information included in the received operation instruction. For example, the payment unit 41 displays the payment information included in the operation instruction on the display 506 in addition to the price of the goods purchased by the user.

[0244] Furthermore, for example, when the billing unit 41 receives billing information for all members of a group to which a user belongs, it displays the amount to be billed on the display 506 in a format that allows the user to select the amount to be billed. If the user wishes to bill individually, they can exclude charges based on the operation of electronic devices by other users when making a settlement.

[0245] <Effects of the Embodiment> In this embodiment, the fare management system 1 tracks a person's movement based on an image taken near the electronic device 30, and transmits an operation instruction using information representing the charges incurred as a result of the user operating the electronic device 30 to a payment device 40 that has detected the user's presence nearby. Therefore, according to the fare management system 1 in this embodiment, a user who has operated an electronic device can transmit charge information based on that operation to a nearby payment device.

[0246] In particular, the billing management system 1 in this embodiment identifies a group to which multiple users belong, and when it detects that any user belonging to the group is near the payment device 40, it transmits the billing information of all users belonging to that group to the payment device 40. This allows users to settle the usage fees of all members of the group together.

[0247] [Application Examples] In the above embodiment, the managed space was assumed to be a small store such as a convenience store, and the example mainly described involved using the electronic device 30 as an MFP to settle charges. However, the usage scenarios to which the charge management system 1 can be applied are not limited to this, and it can be applied to various usage scenarios.

[0248] For example, the billing system 1 can be configured to manage the usage fees for electronic devices installed in a hotel. Hotels have various electronic devices installed. For example, there may be a PC in the lobby for searching local information. Also, guest rooms may have a set-top box that can play television broadcasts and on-demand videos. By managing the billing information generated by these devices using the billing system 1, hotel guests can settle their bills all at once at checkout, rather than having to pay each time they use the device.

[0249] Furthermore, for example, the fare management system 1 can be configured to manage the usage fees for electronic devices installed at the airport. For example, airports may have paid lounges. These lounges may have MFPs (Multi-Function Printers) or similar devices. By managing the fare information generated by these devices using the fare management system 1, passengers can settle their bills all at once upon leaving the lounge, rather than having to pay each time.

[0250] [supplement] In each of the above embodiments, the management device 10 is an example of an information processing device. The fee management system 1 is an example of an information processing system. The monitoring device 20 is an example of a camera. The status management unit 13 is an example of a user identification unit. The operation information receiving unit 12 is an example of a usage information receiving unit. The operation instruction unit 14 is an example of a usage information transmission unit.

[0251] Each of the embodiments described above can be implemented by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each of the functions described above.

[0252] The apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the management device 10 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein.

[0253] Although embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications or changes are possible within the scope of the gist of the present invention as described in the claims. [Explanation of Symbols]

[0254] 1. Billing Management System 10 Management device 11 Image receiving unit 12 Operation Information Receiving Unit 13. Status Management Department 14 Operation instruction section 100 State memory unit 131 Object State Management Unit 132 Equipment Status Management Department 133 Output Information Determination Unit 20 Monitoring equipment 21 Image acquisition unit 22 Image transmission unit 30 Electronic equipment 31 Operation Information Transmission Unit 40 Payment device 41 Payment Department [Prior art documents] [Patent Documents]

[0255] [Patent Document 1] Japanese Patent Publication No. 2020-166638 [Patent Document 2] Japanese Patent Publication No. 2001-242754

Claims

1. An information processing device that can communicate with electronic devices, payment devices and imaging devices via a network, The aforementioned imaging device includes a first image receiving unit that receives a first image of the user of the electronic device and the electronic device itself, A first user identification unit that identifies a first user from the first image, A user information receiving unit that receives user information based on the operation of the first user identified by the first user identification unit from the electronic device, The aforementioned imaging device includes a second image receiving unit that receives a second image of the settlement device and a second user who is able to use the settlement device, A second user identification unit identifies a second user from the second image, A user information transmission unit that transmits the user information to the settlement device based on the first user identified by the first user identification unit and the second user identified by the second user identification unit, An information processing device equipped with the following features.

2. An information processing apparatus according to claim 1, The aforementioned usage information is the usage fee. Information processing device.

3. An information processing apparatus according to claim 1, The user information transmission unit transmits the user information to the payment device when the first user and the second user belong to the same group. Information processing device.

4. An information processing apparatus according to claim 1, The aforementioned imaging device consists of a first camera and a second camera. The first camera takes the first image, The second camera takes the second image. Information processing device.

5. An information processing system comprising the electronic device, the settlement device, the imaging device, and the information processing device according to any one of claims 1 to 4, which are capable of communicating via a network.

6. Electronic devices, payment devices, and cameras, and a computer capable of communicating via a network, A first image receiving procedure in which the imaging device receives a first image of the user of the electronic device and the electronic device itself, A first user identification procedure for identifying a first user from the first image, A user information receiving procedure for receiving user information based on the operations of the first user identified in the first user identification procedure from the electronic device, A second image receiving procedure in which the imaging device receives a second image of the payment device and a second user who can use the payment device, A procedure for identifying a second user from the second image, A user information transmission procedure for transmitting the user information to the settlement device based on the first user identified in the first user identification procedure and the second user identified in the second user identification procedure, A method for managing fees to implement this.

7. A computer capable of communicating with electronic devices, payment devices, and imaging devices via a network, A first image receiving procedure in which the imaging device receives a first image of the user of the electronic device and the electronic device itself, A first user identification procedure for identifying a first user from the first image, A user information receiving procedure for receiving user information based on the operations of the first user identified in the first user identification procedure from the electronic device, A second image receiving procedure in which the imaging device receives a second image of the payment device and a second user who can use the payment device, A procedure for identifying a second user from the second image, A user information transmission procedure for transmitting the user information to the settlement device based on the first user identified in the first user identification procedure and the second user identified in the second user identification procedure, A program to execute.

Images