Object type identification device, object type identification method, and object type identification program

Abstract

To provide an object type specifying device, an object type specifying method, and an object type specifying program capable of correctly specifying the type of an object picked up by a person.SOLUTION: The object type specifying device is provided with: a position detecting unit for detecting the position of an object; a storing unit for storing information indicative of a type of the object at the position of each object; a determining unit for determining, based on information from a sensor, whether the object has been taken out or the object has been placed; an object specifying unit for specifying, when the determining unit determines that the object has been taken out, the type of the taken-out object based on the output of the position detecting unit and the information stored in the storage unit; and an updating unit for updating, when the determining unit determines that the object has been placed, the information stored in the storing unit by using images captured by a camera for capturing the arrangement of each object from the front side.SELECTED DRAWING: Figure 18

Description

Technical Field

[0001] The present invention relates to an object type identification device, an object type identification method, and an object type identification program for identifying the type of an object to be targeted.

Background Art

[0002] Various methods for identifying the type of an object placed at a specific position are known. For example, Patent Document 1 describes a system that photographs customers and displayed products and automatically monitors for unauthorized removal of products based on the photographed image. In the system described in Patent Document 1, a camera is arranged facing the shelf on which the object is placed, and the photographed image is analyzed to perform action recognition of which product a person has picked up. Then, the system described in Patent Document 1 uses the information stored in the memory to identify the product picked up by the person.

[0003] Further, Patent Document 2 describes a device that monitors the state of products displayed on a shelf with a photographed image and determines whether product arrangement is necessary. The device described in Patent Document 2 also arranges a camera facing the shelf on which the object is placed and determines whether a store clerk is arranging products based on the photographed image.

[0004] Note that Patent Document 3 describes arranging a camera on a shelf to perform action recognition of a person.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0006] On the other hand, in the case of the system described in Patent Document 1, since motion recognition is performed based on image analysis, it is difficult to accurately recognize a person's movements, and it may not be possible to identify the type of product.

[0007] Furthermore, in the system described in Patent Document 1, if a person accidentally moves an item to a different location or if a new item is placed on the shelf, the system cannot correctly compare it with the information stored in memory. As a result, there is a problem in that the type of item picked up by the person cannot be correctly identified. The same problem exists when using the method described in Patent Document 3.

[0008] Furthermore, the device described in Patent Document 2 is designed to determine whether or not a store employee is organizing merchandise, and it is inherently difficult to identify the type of merchandise.

[0009] Therefore, the present invention aims to provide an object type identification device, an object type identification method, and an object type identification program that can correctly identify the type of object picked up by a person. [Means for solving the problem]

[0010] The object type identification device according to the present invention is characterized by comprising: position acquisition means for acquiring the position of an object; storage means for storing information indicating the type of object at the position of each object; determination means for determining whether an object has been removed or placed based on sensor information; object identification means for identifying the type of the removed object based on the output of the position acquisition means and the information stored in the storage means when the determination means determines that an object has been removed; and update means for updating the information stored in the storage means using images captured by a camera that captures the arrangement of each object from the front when the determination means determines that an object has been placed.

[0011] The object type identification method according to the present invention is characterized by determining whether an object has been removed or placed based on sensor information, acquiring the position of the object if it is determined that an object has been removed, identifying the type of object that has been removed based on the acquired position of the object and the information stored in a storage means that stores information indicating the type of object at the position of each object, and updating the information stored in the storage means using an image captured by a camera that captures the arrangement of each object from the front.

[0012] The recording medium according to the present invention records an object type identification program in which a computer performs a position acquisition process to acquire the position of an object, a determination process to determine whether an object has been removed or placed based on sensor information, an object identification process to identify the type of object that has been removed based on the acquisition result of the position acquisition and information stored in a storage means that stores information indicating the type of object at the position of each object, and an update process to update the information stored in the storage means using an image captured by a camera that captures the arrangement of each object from the front. [Effects of the Invention]

[0013] According to the present invention, the type of object picked up by a person can be correctly identified. [Brief explanation of the drawing]

[0014] [Figure 1] This is an explanatory diagram illustrating an example of a situation in which the object type identification method of the embodiment is used. [Figure 2] This is an explanatory diagram showing an example of a configuration that realizes the object type identification method of the embodiment. [Figure 3] This is an explanatory diagram illustrating the schematic of an object type identification device according to an embodiment. [Figure 4] This is a block diagram showing an example configuration of an object type identification device according to the first embodiment. [Figure 5]It is an explanatory diagram showing an example of setting a correspondence relationship based on a photographed image. [Figure 6] It is an explanatory diagram showing an example of a camera image after an object has been changed. [Figure 7] It is an explanatory diagram showing an example of a process for updating a correspondence relationship. [Figure 8] It is a flowchart showing an example of the operation of the object type identification device according to the first embodiment. [Figure 9] It is a flowchart showing an example of a process for identifying a taken-out product. [Figure 10] It is an explanatory diagram showing an example of information stored in the correspondence relationship storage unit. [Figure 11] It is an explanatory diagram showing an example of a list of taken-out products. [Figure 12] It is a flowchart showing an example of a process for updating information stored in the correspondence relationship storage unit. [Figure 13] It is a block diagram showing a modified example of the object type identification device according to the first embodiment. [Figure 14] It is a block diagram showing a configuration example of the object type identification device according to the second embodiment. [Figure 15] It is a flowchart showing an example of the operation of the object type identification device according to the second embodiment. [Figure 16] It is a block diagram showing a configuration example of the object type identification device according to the third embodiment. [Figure 17] It is a flowchart showing an example of the operation of the object type identification device according to the third embodiment. [Figure 18] It is a block diagram showing an overview of the object type identification device according to the embodiment. [Figure 19] It is a block diagram showing an example of the hardware configuration of the object type identification device according to the embodiment.

Mode for Carrying Out the Invention

[0015] First, with reference to Figure 1, the method for identifying the type of object in this embodiment will be outlined. Figure 1 is an explanatory diagram showing an example of a situation in which the object type identification method is used. In the example shown in Figure 1, multiple objects are placed on a shelf 11, and the movements of a person present in front of the shelf 11 are recognized using a camera 12 and an information processing terminal 13. While goods are given as an example of an object, the term "object" is not limited to goods intended for sale. An object means something that has a shape and can be seen or touched by a person, such as an item, product, or article. The information processing terminal 13 also maintains a correspondence table 14 that associates the type of object with the coordinates indicating the position of that object (hereinafter referred to as object coordinates). Below, the method for identifying the position of each object based on object coordinates will be described, but the method for identifying the position of an object is not limited to object coordinates.

[0016] Object coordinates may be represented, for example, as a position (x,y,z) in a three-dimensional space with a point in real space as the origin, or as a position (x,y) in a two-dimensional space on the front of shelf 11 with a point on the front of shelf 11 as the origin. In the example shown in Figure 1, the position of each object is represented by two-dimensional coordinates with the upper left of the front of shelf 11 as the origin. In the example shown in Figure 1, the type of object identified by object coordinates (1,0) is "C".

[0017] Here, if an object moves from the position indicated by its object coordinates, a discrepancy occurs between the correspondence between the object and its object coordinates set in the correspondence table 14 and the actual correspondence. When such a discrepancy occurs, the accuracy of object recognition decreases. Therefore, in the object type identification method of this embodiment, the object position is appropriately updated to match the actual position of the object.

[0018] Figure 2 is an explanatory diagram showing an example of a configuration for realizing an object type identification method. The object type identification system 20 illustrated in Figure 2 is an example of a system including the object type identification device of the present invention. The object type identification system 20 uses an information processing terminal 23 connected to a camera 22 to recognize the movements of a person present in front of a shelf 21, obtains the coordinates of an object on the shelf 21, and recognizes the target object using a correspondence table (for example, correspondence table 14). The object type identification system 20 also uses a camera 24 and an information processing terminal 25 to capture an image of the front (front side) of the shelf 21, obtains the object type and object coordinates through image recognition, and updates the correspondence table 14. The object type obtained through image recognition may include, for example, the object's name, product name, size, price, etc.

[0019] Figure 3 is an explanatory diagram illustrating the general structure of an object type identification device. Table T, illustrated in Figure 3, corresponds to the correspondence table 14 illustrated in Figure 1. In the example shown in Figure 3, the location of the product is represented by two-dimensional coordinates, and the cumulative record of the product's location over time is shown. The contents of Table T are updated at a predetermined update timing or periodically. The time in Table T indicates, for example, the update timing. In the example shown in Figure 3, the correspondence is recorded cumulatively, but a single master may be provided, and its contents may be overwritten and updated.

[0020] Furthermore, the upper part of Figure 3 illustrates the process of identifying an object. When an image sensor such as an RGB camera or depth camera detects that a person P puts their hand into a shelf S, the object type identification device detects the position where the hand was placed based on the sensor information (step S1). For example, if the detected position is the position identified by object coordinates (4,4) and the time is 16:45, the object type identification device refers to table T and identifies the object as product B (step S2).

[0021] Furthermore, the lower part of Figure 3 illustrates the process of recognizing the type of object. A camera C is installed to capture images of the arrangement of each object from the front, and each object placed on the shelf S is captured by this camera C. Camera C corresponds to camera 24 in Figure 2. In the present invention, table T is updated at the timing when an object is placed on the shelf, or at a predetermined timing (step S3). In this way, table T is updated appropriately as needed, resulting in improved image recognition accuracy, which in turn allows for the correct identification of the type of object picked up by a person.

[0022] The embodiments will be described below with reference to the drawings. Embodiment 1. Figure 4 is a block diagram showing an example configuration of an object type identification device in the first embodiment. The object type identification device 200 of this embodiment includes an action recognition unit 201, an object coordinate acquisition unit 202, an object recognition unit 203, a correspondence relationship storage unit 204, a correspondence relationship update unit 205, and an update method determination unit 206.

[0023] The motion recognition unit 201 recognizes the actions of the subject. In this embodiment, the subject is anything that causes some kind of change to the state of an object, such as a person or a robot. The method by which the motion recognition unit 201 recognizes the actions of the subject is arbitrary. In this embodiment, the motion recognition unit 201 determines whether an object has been removed or placed based on the information from the camera 22 (sensor). For this reason, the motion recognition unit 201 can be called a determination unit.

[0024] The motion recognition unit 201 may use an image sensor such as an RGB camera or a depth camera as the camera 22 to recognize actions such as "object removed," "object placed," or "no action" by the subject from changes in color or volume around a specific part of the body, such as a hand. A depth camera is a camera that can measure depth information from the camera to the object being imaged, along with RGB information acquired by normal imaging.

[0025] Furthermore, if, for example, a pressure sensor is pre-installed at the object's placement location (e.g., the surface on which the object is placed), the motion recognition unit 201 may determine whether the object has been removed or placed based on the information from the pressure sensor instead of the camera 22.

[0026] Furthermore, the sensors used to determine the motion are not limited to pressure sensors; for example, infrared sensors may also be used. In addition, the motion recognition unit 201 may recognize the motion of a person or robot using any other method capable of determining motion.

[0027] The object coordinate acquisition unit 202 acquires the position of the object that is the target of the action. In this embodiment, the object coordinate acquisition unit 202 acquires object coordinates as the position of the object. Therefore, the object coordinate acquisition unit 202 can be called an object position acquisition unit. As described above, object coordinates refer to the coordinates on which the object that a person, robot, or other entity is targeting is located.

[0028] The object coordinate acquisition unit 202 acquires object coordinates using, for example, an image captured by the camera 22. Specifically, the object coordinate acquisition unit 202 identifies the two-dimensional coordinates of parts of a person, such as the face or hands, on the image from image information obtainable from an RGB camera such as a surveillance camera. The object coordinate acquisition unit 202 may acquire object coordinates from the image by pre-associating the identified two-dimensional coordinates with the object coordinates in real space.

[0029] Furthermore, if a depth camera is used instead of an RGB camera, it is possible to obtain the real-space 3D coordinates of parts of a person, and the object coordinate acquisition unit 202 may use these as object coordinates.

[0030] In addition to using images, the object coordinate acquisition unit 202 may also acquire object coordinates using the pressure sensor described above. For example, if a sensor such as a pressure sensor is already installed on the surface on which the object is placed, the object coordinate acquisition unit 202 may acquire the coordinates where the pressure changed significantly when the object was "acquired" (removed) as the object coordinates.

[0031] However, the method by which the object coordinate acquisition unit 202 acquires object coordinates is not limited to the method described above.

[0032] The correspondence storage unit 204 stores information indicating the type of object at the location of each object. Specifically, the correspondence storage unit 204 stores the correspondence between the type of object to be recognized and the object coordinates, which are the coordinates in which the object is located. In addition to the correspondence between objects and object coordinates, the correspondence storage unit 204 may also store the update time associated with the update of that correspondence. In this embodiment, the correspondence storage unit 204 is assumed to hold the correspondence table 14 illustrated in Figure 1 (i.e., the correspondence between object coordinates and object types).

[0033] The correspondence between object coordinates and object types is predetermined. Figure 5 is an explanatory diagram showing an example of setting the correspondence based on captured images. For example, suppose the object type identification device 200 is connected to the camera 24 and information processing terminal 25 exemplified in Figure 2. In this case, when the camera 24 acquires a camera image 31 as exemplified in Figure 5, the information processing terminal 25 may determine which 2D coordinates in the camera image 31 correspond to the 2D coordinates of the shelf 32, determine the object coordinates for each object, and set them in the correspondence table. In addition, the correspondence storage unit 204 may store not only the correspondence between objects and object coordinates, but also the image feature quantities of each object extracted from the camera image 31 or other images in the object database 33. The object database 33 is stored, for example, in the correspondence storage unit 204.

[0034] The object recognition unit 203 recognizes the type of object that was the target of the operation. Specifically, when the motion recognition unit 201 determines that an object has been retrieved, the object recognition unit 203 identifies the type of object that was retrieved based on the object's position acquired by the object coordinate acquisition unit 202 and the information stored in the correspondence relationship storage unit 204. Therefore, the object recognition unit 203 can be called an object identification unit.

[0035] The object recognition unit 203 may refer to the object coordinates acquired by the object coordinate acquisition unit 202 and the correspondence table 14 held by the correspondence relationship storage unit 204, and recognize the type of object associated with the object coordinates as the recognition result. In addition to the recognized type of object, the object recognition unit 203 may also include the action of the subject recognized by the action recognition unit 201 as part of the recognition result.

[0036] For example, suppose the correspondence relationship storage unit 204 stores the correspondence table 14 illustrated in Figure 1. Then, suppose the action recognition unit 201 recognizes the subject's "acquisition" action, and the object coordinate acquisition unit 202 acquires the object coordinates (1,0). In this case, the object recognition unit 203 recognizes the object that was the target of the action as "C" based on the position indicated by the acquired object coordinates. At this time, the object recognition unit 203 may recognize object "C" as the recognition result, or the content of "acquiring object "C"" including the action as the recognition result, or further, the content of "acquiring object "C" from object coordinates (1,0)" including the object coordinates as the recognition result.

[0037] However, the method by which the object recognition unit 203 recognizes the object targeted for action is not limited to the method described above. In the example described above, the object recognition unit 203 obtains the type of object whose object coordinates match from the correspondence table 14, but it may also obtain the type of object that corresponds to the object when the object coordinates are within a certain range.

[0038] For example, p(o) is the coordinate of the object associated with object o in correspondence table 14, Distance(p1,p2) is the distance between p1 and p2, threshold distIf we take the threshold value and the set O represents the types of objects in correspondence table 14, then the identity of an object is expressed by equation 1, which is exemplified below.

[0039]

number

[0040] The correspondence update unit 205 updates the correspondence between the information indicating the type of object stored in the correspondence storage unit 204 and the position of that object. Specifically, the correspondence update unit 205 updates the type and position of the object when the motion recognition unit 201 determines that an object has been placed. In this embodiment, the correspondence update unit 205 updates the information stored in the correspondence storage unit 204 using images of the arrangement of each object captured from the front by the camera 24.

[0041] The correspondence update unit 205 recognizes object coordinates and object types from images captured by the camera 24 (for example, the camera image 31 illustrated in Figure 5), and identifies the object type and object coordinates on the image by matching the image with image features stored in the database (for example, the object database 33 illustrated in Figure 5). Then, the correspondence update unit 205 refers to the correspondence stored in the correspondence storage unit 204 and updates the correspondence between the identified object coordinates and the object type.

[0042] Figure 6 is an explanatory diagram showing an example of a camera image after an object has been changed. Figure 7 is an explanatory diagram showing an example of a process for updating correspondences. For example, when camera 24 takes the camera image exemplified in Figure 6, the correspondence update unit 205 identifies the type of object and the object's coordinates on the image from the captured camera image. In this example, the correspondence update unit 205 identifies that, from the initial state exemplified in Figure 5, the object at the position indicated by object coordinates (1,0) has changed from "C" to "X", and the object at the position indicated by object coordinates (1,1) has changed from "D" to "Y". Therefore, the correspondence update unit 205 updates the object corresponding to object coordinates (1,0) to "X" and the object corresponding to object coordinates (1,1) to "Y".

[0043] The update method determination unit 206 determines the update method for the correspondence stored in the correspondence storage unit 204. That is, the correspondence update unit 205 updates the information in the correspondence storage unit 204 based on the determination of the update method determination unit 206. Note that if the update method is predetermined, the object type identification device 200 does not need to be equipped with the update method determination unit 206.

[0044] In this embodiment, when the motion recognition unit 201 determines that an object has been placed, the correspondence relationship update unit 205 updates the type and position of that object. Therefore, the update method determination unit 206 detects that an object has been placed by the motion recognition unit 201 and determines the detected timing as the update timing T100. However, the update timing determined by the update method determination unit 206 is not limited to the timing when an object is placed. Other methods for determining the update timing are described below.

[0045] The update method determination unit 206 may, for example, determine the update timing T101 as a predetermined time interval. For example, if the time interval is predetermined to be 30 minutes, the update method determination unit 206 may determine the update timing T101 as 9:00, 9:30, 10:00, etc.

[0046] Furthermore, the update method determination unit 206 may determine the update timing T102 using the camera image. Specifically, if the difference between the image used during the previous update of the correspondence storage unit 204 and the newly captured image is large (i.e., the difference between the two images exceeds a predetermined threshold), the update method determination unit 206 may determine the time the image was captured as the update timing T102.

[0047] The update method determination unit 206, for example, acquires camera images taken at multiple times using a fixed camera and calculates the background difference of the target area in the image. At this time, the update method determination unit 206 may determine the time when a change of a certain level or more is detected as the update timing T102. Here, the target area may be, for example, the front of a shelf on which an object is placed, or the environment around the shelf.

[0048] Furthermore, the update method determination unit 206 may determine the update timing T103 using information obtained from a sensor installed on the surface on which the object is placed. For example, if a pressure sensor is installed on the surface on which the object is placed, the update method determination unit 206 may receive output values ​​from the pressure sensor in a time series and set the time when the change in the output value exceeds a predetermined threshold as the update timing T103.

[0049] Furthermore, the update method determination unit 206 may re-determine a new update timing based on a plurality of update timings being considered. For example, the update method determination unit 206 may generate an update queue when any of the above-described update timings T100, T101, T102, or T103 is determined, and if that queue exists, it may re-determine the update timing T104 by another method.

[0050] For example, we can prepare an IsToUpdated flag to indicate whether or not an update queue has been created. The update method determination unit 206 sets the flag IsToUpdated=true when an update queue is generated. Only when the flag IsToUpdated=true, the update method determination unit 206 detects other objects or people other than the object to be recognized within the target area of ​​the image captured by the camera. If no other objects or people are detected, the update method determination unit 206 may set the time of no detection as the update timing T104. The update method determination unit 206 should set the flag IsToUpdated=false at the time the update timing T104 is determined.

[0051] For example, if the update method is determined solely by the camera, there is a risk that the update may not be performed properly if something (such as a person) obstructs the target of the update timing. However, in this embodiment, the update method determination unit 206 determines the update method based on multiple pieces of information, thus preventing inappropriate updates from being performed.

[0052] The motion recognition unit 201, the object coordinate acquisition unit 202, the object recognition unit 203, the correspondence relationship update unit 205, and the update method determination unit 206 are implemented by a computer's CPU (Central Processing Unit) that operates according to a program (object type identification program). For example, as shown in Figure 19, the program may be stored in the storage unit of the object type identification device 200 (for example, the correspondence relationship storage unit 204), and the CPU may read the program and operate as the motion recognition unit 201, the object coordinate acquisition unit 202, the object recognition unit 203, the correspondence relationship update unit 205, and the update method determination unit 206 according to the program.

[0053] Furthermore, the motion recognition unit 201, the object coordinate acquisition unit 202, the object recognition unit 203, the correspondence relationship update unit 205, and the update method determination unit 206 may each be implemented with dedicated hardware. Also, the object type identification device may be configured by connecting two or more physically separate devices via wired or wireless connections. The correspondence relationship storage unit 204 may be implemented, for example, by a magnetic disk drive.

[0054] Next, the operation of the object type identification device of this embodiment will be described. Figure 8 is a flowchart showing an example of the operation of the object type identification device of this embodiment. It should be assumed that the correspondence relationship storage unit 204 stores information indicating the type of object at the location of each object.

[0055] First, the motion recognition unit 201 detects whether or not a person has made the action of putting their hand into the shelf (step S101). The motion recognition unit 201 detects the action of a person putting their hand into the shelf based on information acquired, for example, from an RGB camera or a depth camera.

[0056] If the motion of a person reaching into a shelf is detected (YES in step S102), the motion recognition unit 201 determines whether the motion is to take out an object or to place an object on it (step S103). The motion recognition unit 201 determines the nature of the person's motion based, for example, on images captured by the camera 22 or information obtained from the shelf's switch (pressure sensor).

[0057] When an action to remove an object is detected ("removal" in step S103), the object recognition unit 203 identifies the removed product (step S104). The process then proceeds to step S108.

[0058] On the other hand, if an action of placing an object is detected ("Placement" in step S103), the update method determination unit 206 determines the update timing (step S105). If no action of a person putting their hand into the shelf is detected (NO in step S102), and it is the scheduled update time (YES in step S106), the update method determination unit 206 determines the update timing (step S105).

[0059] The correspondence relationship update unit 205 updates the information stored in the correspondence relationship storage unit 204 based on the determined update timing (step S107). After that, the process proceeds to step S108.

[0060] If the update time has not arrived in step S106 (NO in step S106), or after step S104 or step S107, the operation recognition unit 201 determines whether or not the end time of the series of processes has arrived (step S108). If the end time has not arrived (NO in step S108), the processes from step S101 onwards are performed. On the other hand, if the end time has arrived (YES in step S108), the process ends.

[0061] Figure 9 is a flowchart showing an example of the process for identifying the retrieved product in step S104 of Figure 8. When the motion recognition unit 201 detects the motion of retrieving an object, the object coordinate acquisition unit 202 identifies the coordinates of the object that was the target of the motion (step S111). The object recognition unit 203 reads the corresponding coordinate information stored in the correspondence relationship storage unit 204 (step S112).

[0062] Figure 10 is an explanatory diagram showing an example of information stored in the correspondence storage unit 204. In the example shown in Figure 10, product A is placed at the position represented by object coordinates (1,0) and product B is placed at the position represented by object coordinates (1,1). For example, when the object coordinate acquisition unit 202 acquires object coordinates (1,1), the object recognition unit 203 reads product B at coordinates (1,1) from the correspondence storage unit 204. As a result, the object recognition unit 203 identifies the retrieved product.

[0063] Subsequently, the object recognition unit 203 may create a list of the retrieved items (step S113). Figure 11 is an explanatory diagram showing an example of a list of retrieved items. In the example shown in Figure 11, the list includes the acquired object coordinates, product name, and time. The object recognition unit 203 may also write the product names to the list in real time, and this list may be used for offline analysis later.

[0064] Figure 12 is a flowchart showing an example of the process of updating the information stored in the correspondence storage unit 204 in step S107 of Figure 8. Once the update timing is determined, the update method determination unit 206 recognizes a person in the target area of ​​the image captured by the camera 24 (step S121). If a person is recognized in the target area of ​​the image (YES in step S122), the process returns to step S121. On the other hand, if no person is recognized in the target area of ​​the image (NO in step S122), the object recognition unit 203 recognizes the placed product (step S123).

[0065] Then, the correspondence update unit 205 updates the information in the correspondence table 14 stored in the correspondence storage unit 204 (step S124). At this time, instead of updating the same table in the correspondence storage unit 204, the correspondence update unit 205 may manage the correspondence between products and locations by creating a new table or record.

[0066] As described above, in this embodiment, the motion recognition unit 201 determines whether an object has been removed or placed based on sensor information. If the motion recognition unit 201 determines that an object has been removed, the object recognition unit 203 identifies the type of object that was removed based on the output of the object coordinate acquisition unit 202, which acquires the position of the object, and the information stored in the correspondence relationship storage unit 204 (the relationship between the type of object and the position of that object). On the other hand, if the motion recognition unit 201 determines that an object has been placed, the correspondence relationship update unit 205 updates the information stored in the correspondence relationship storage unit 204 using the image captured by the camera 24. With such a configuration, the type of object picked up by a person can be correctly identified.

[0067] In other words, in this embodiment, two pieces of hardware (camera 22 (sensor) and camera 24) are linked together, and detection by one piece of hardware triggers the operation of the other piece of hardware. With such a configuration, it becomes possible to correctly identify the type of object that a person is holding.

[0068] Generally, the same product is placed in the same position on a display shelf. Therefore, even if an object is removed, the type of object generally does not change, so there is little need to update the contents of the correspondence table 14. On the other hand, when an object is placed in a certain position on a display shelf, the contents of that object are often unknown. Therefore, in this embodiment, the correspondence table 14 is updated when the placement of the object is detected.

[0069] Furthermore, in this embodiment, the update method determination unit 206 appropriately determines the timing for updating the correspondence table 14, and the correspondence relationship update unit 205 updates the correspondence table 14 to match the actual state. As a result, the recognition of the object to be operated on can be performed with high accuracy.

[0070] Next, a modified example of this embodiment will be described. Figure 13 is a block diagram showing a modified example of the object type identification device 200 of the first embodiment. The object type identification device 200a illustrated in Figure 13 includes an motion recognition unit 201, an object coordinate acquisition unit 202, an object recognition unit 203, a correspondence relationship storage unit 204, a correspondence relationship update unit 205, an update method determination unit 206, and a feature quantity conversion unit 207. In other words, the object type identification device 200 of this modified example is further equipped with a feature quantity conversion unit 207 compared to the object type identification device 200 of the first embodiment.

[0071] In the first embodiment, the information (database) stored in the correspondence storage unit 204 is shared by multiple devices (sensors such as cameras 22 and 24). The feature conversion unit 207 registers feature information such as the color and size of objects in the image captured by camera 24 in the correspondence storage unit 204 (for example, the object database 33), and further converts these feature information into a format usable by camera 22 (sensor). Specifically, the feature conversion unit 207 converts the image captured by camera 24 into a format usable by the object recognition unit 203 for identifying the type of object.

[0072] In this case, the object recognition unit 203 can recognize objects based on their size, using the size of the object captured by the camera 24 as the basis for recognition on the camera 22 (sensor) side. Furthermore, the object recognition unit 203 can also perform matching on the camera 22 (sensor) side based on the color information captured by the camera 24. In other words, by using such converted information, it is possible to improve the accuracy of object recognition. Moreover, when performing such a conversion, it is also possible to use the object's position as one of the features. Embodiment 2. Next, a second embodiment will be described. Figure 14 is a block diagram showing an example configuration of the object type identification device according to the second embodiment. The object type identification device 300 of this embodiment includes an motion recognition unit 301, an object coordinate acquisition unit 302, an object recognition unit 303, a correspondence relationship storage unit 304, a correspondence relationship update unit 305, and an update method determination unit 306.

[0073] The motion recognition unit 301, object coordinate acquisition unit 302, object recognition unit 303, and correspondence relationship storage unit 304 of this embodiment are the same as the motion recognition unit 201, object coordinate acquisition unit 202, object recognition unit 203, and correspondence relationship storage unit 204 of the first embodiment, respectively. The object type identification device 300 may also include a feature quantity conversion unit 207, which is a modified example of the first embodiment.

[0074] The update method determination unit 306 determines the update timing and update method based on at least the recognition result by the motion recognition unit 301 and the object coordinate information acquired by the object coordinate acquisition unit 302. Here, the update method refers to the method for determining the object coordinates to be updated and the method for determining the candidate types of objects to be updated. For example, the method used to determine the update timing is the method used by the update method determination unit 206 in the first embodiment to determine the update timing.

[0075] The update method determination unit 306 determines, based on the object coordinates acquired by the object coordinate acquisition unit 302, which objects within a predetermined range will be the target for updating, either by their position or type. Then, the correspondence relationship update unit 305 updates the determined target (specifically, the correspondence between the object's position and type) from the information stored in the correspondence relationship storage unit 304.

[0076] The update method determination unit 306 may determine the update method based, for example, on the object coordinates acquired by the object coordinate acquisition unit 302. Specifically, with respect to the object coordinates to be updated, the update method determination unit 306 may determine that the coordinates to be updated are those that are within a certain threshold distance from the object coordinates acquired by the object coordinate acquisition unit 302.

[0077] Furthermore, the update method determination unit 306 may determine the update method when it recognizes that the motion recognition unit 301 has moved an object. In this case, the update method determination unit 306 may determine that coordinates that are within a threshold distance from the source object coordinates and destination object coordinates acquired by the object coordinate acquisition unit 302 are the targets for update. In this case, the update method determination unit 306 may also decide to limit the update candidates to two types of objects: the type of object originally placed at that location, or the type of object from which the object was moved.

[0078] The correspondence relationship update unit 305 updates the information in the correspondence relationship storage unit 304 (specifically, the information in the correspondence table 14) based on the update timing and update method determined by the update method determination unit 306. In this embodiment, the correspondence relationship update unit 305 may update the correspondence table 14 held by the correspondence relationship storage unit 304 only for object coordinates determined by the update method determination unit 306, and only for candidate object types determined by the update method determination unit 306.

[0079] Similar to the first embodiment, the motion recognition unit 301, the object coordinate acquisition unit 302, the object recognition unit 303, the correspondence relationship update unit 305, and the update method determination unit 306 are all implemented by a computer CPU that operates according to a program (object type identification program).

[0080] Next, the operation of the object type identification device of this embodiment will be described. Figure 15 is a flowchart showing an example of the operation of the object type identification device of this embodiment. It should be assumed that the correspondence relationship storage unit 304 stores information indicating the type of object at the location of each object.

[0081] The operation of this embodiment is the same as that of the first embodiment illustrated in Figure 8. However, it differs from the first embodiment in that in step S205, the update method determination unit 306 determines the update timing and update method.

[0082] In step S205, the update method determination unit 306 determines, based on the object coordinates obtained by the object coordinate acquisition unit 302, which object's position or type is within a predetermined range and is the target for updating.

[0083] Then, in step S107, the correspondence relationship update unit 305 updates the information stored in the correspondence relationship storage unit 304 based on the determined update timing and update method. Specifically, at the determined update timing, the correspondence relationship update unit 305 updates the determined target (specifically, the correspondence between the position and type of an object) from the information stored in the correspondence relationship storage unit 304. The other processing is the same as in the first embodiment.

[0084] As described above, in this embodiment, the update method determination unit 306 determines which objects to update are those whose positions or types are within a predetermined range from the object's position acquired by the object coordinate acquisition unit 302. Then, the correspondence relationship update unit 305 updates the determined target from the information stored in the correspondence relationship storage unit 304. With this configuration, in addition to the effects of the first embodiment, the correspondence table 14 can be updated efficiently and accurately. Embodiment 3. Next, a third embodiment will be described. Figure 16 is a block diagram showing an example configuration of the object type identification device of the third embodiment. The object type identification device 400 of this embodiment includes an action recognition unit 401, an object coordinate acquisition unit 402, an object recognition unit 403, a correspondence relationship storage unit 404, a correspondence relationship update unit 405, an update method determination unit 406, and a display quantity storage unit 407. In other words, the object type identification device 400 of this embodiment further includes a display quantity storage unit 407 in addition to the object type identification device 200 of the first embodiment or the object type identification device 300 of the second embodiment.

[0085] The motion recognition unit 401, object coordinate acquisition unit 402, and correspondence relationship storage unit 404 of this embodiment are the same as the motion recognition unit 201, object coordinate acquisition unit 202, and correspondence relationship storage unit 204 of the first embodiment, respectively. The object type identification device 400 may include a feature quantity conversion unit 207, which is a modified example of the first embodiment.

[0086] The display quantity storage unit 407 stores display information, including the type and number of objects placed at a single object coordinate. Specifically, the display quantity storage unit 407 maintains display information that associates the type and number of objects with each object coordinate. The display quantity storage unit 407 may also store display information that includes the display order. Display information is pre-set in the display quantity storage unit 407. The display quantity storage unit 407 is implemented, for example, by a magnetic disk.

[0087] The object recognition unit 403 updates the display information based on the object recognition results. Specifically, the object recognition unit 403 updates the number of objects based on the action recognition results (specifically, the determination results indicating whether an object has been taken out or placed) by the action recognition unit 401.

[0088] For example, if the recognition result is "object "C" acquired from object coordinates (1,0)", the object recognition unit 403 subtracts 1 from the number of objects "C" at object coordinates (1,0) included in the display information stored in the display count storage unit 407.

[0089] The correspondence relationship update unit 405 and the update method determination unit 406 are the same as the correspondence relationship update unit 205 and the update method determination unit 206 in the first embodiment. Furthermore, in this embodiment, the update method determination unit 406 determines the update timing based on the display information stored in the display quantity storage unit 407.

[0090] The update method determination unit 406 may update the information stored in the correspondence relationship storage unit 404 when the motion recognition unit 401 determines that an object has been removed, and as a result of subtracting the number of objects at the position where the object was removed, the number of objects at that position becomes 0. For example, if the number of objects "C" at object coordinates (1,0) becomes 0 in the display information stored in the display count storage unit 407, the update method determination unit 406 decides to immediately update object coordinates (1,0) to "no object". Based on the decision of the update method determination unit 406, the correspondence relationship update unit 405 updates the type of object corresponding to object coordinates (1,0) in the correspondence table 14 stored in the correspondence relationship storage unit 404 to "no object".

[0091] Furthermore, the type of object placed at a single object coordinate is not limited to one type. In other words, the display quantity storage unit 407 may store the display order of objects at each object coordinate as display information. For example, suppose the display quantity storage unit 407 stores display information including the display order "C", "A", "A" at object coordinates (1,0) (object "C" is placed first, followed by two points of object "A"). Then, suppose the object recognition unit 403 recognizes that it has "acquired" object "C" from object coordinates (1,0).

[0092] In this case, the update method determination unit 406 detects from the display information stored in the display quantity storage unit 407 that object "A" has been placed at object coordinates (1,0). Since this detection indicates that the type of object has changed, the update method determination unit 406 decides to immediately update the type of object at object coordinates (1,0) to "A". Based on the decision of the update method determination unit 406, the correspondence relationship update unit 405 updates the type of object corresponding to object coordinates (1,0) in the correspondence table 14 stored in the correspondence relationship storage unit 404 to "A".

[0093] Similar to the first embodiment, the motion recognition unit 401, the object coordinate acquisition unit 402, the object recognition unit 403, the correspondence relationship update unit 405, and the update method determination unit 406 are all implemented by a computer CPU that operates according to a program (object type identification program).

[0094] Next, the operation of the object type identification device of this embodiment will be described. Figure 17 is a flowchart showing an example of the operation of the object type identification device of this embodiment. It should be assumed that the correspondence relationship storage unit 404 stores information indicating the type of object at the location of each object.

[0095] The operation of this embodiment is the same as that of the first embodiment illustrated in Figure 8. It differs from the first embodiment in that a process to update the display information (step S301) is added after the product retrieved in step S104 is identified, and the process to determine the update timing based on the update of the display information (step S302) is modified.

[0096] In step S301, the object recognition unit 403 updates the display information stored in the display quantity storage unit 407 according to the type of object identified. In step S302, the update method determination unit 406 determines the update timing based on the display information stored in the display quantity storage unit 407. Specifically, the update method determination unit 406 may decide to update the information stored in the correspondence relationship storage unit 404 when it is determined that an object has been removed, when the number of objects at the removal location becomes 0, or when the type of object at that location changes. Other processing is the same as in the first embodiment.

[0097] As described above, in this embodiment, the display quantity storage unit 407 stores display information that associates the position of an object with the type and number of that object. The object recognition unit 403 then updates the display information according to the type of object it identifies. Specifically, when the motion recognition unit 401 determines that an object has been removed, the object recognition unit 403 subtracts the number of objects at the removed position from the display information stored in the display quantity storage unit 407, and updates the information stored in the storage unit when that number becomes 0. With this configuration, in addition to the effects of the first embodiment, the correspondence table 14 can be updated efficiently and accurately.

[0098] In other words, in this embodiment, the update method determination unit 406 determines the update timing and update method based on the display information held by the display quantity storage unit 407, and the correspondence relationship update unit 405 updates the correspondence table 14 based on this. Therefore, it becomes possible to update the correspondence table 14 efficiently and accurately.

[0099] Next, we will explain the overview of the object type identification device. Figure 18 is a block diagram illustrating the overview of the object type identification device. The object type identification device 80 includes a position acquisition unit 81 (e.g., an object coordinate acquisition unit 202) that acquires the position of an object (e.g., object coordinates), a storage unit 82 (e.g., a correspondence relationship storage unit 204) that stores information indicating the type of object at the position of each object, a determination unit 83 (e.g., an motion recognition unit 201) that determines whether an object has been removed or placed based on information from a sensor (e.g., an image sensor such as an RGB camera or a depth camera), an object identification unit 84 (e.g., an object recognition unit 203) that identifies the type of the removed object based on the output of the position acquisition unit 81 and the information stored in the storage unit 82 when the determination unit 83 determines that an object has been removed, and an update unit 85 (e.g., an update method determination unit 206, a correspondence relationship update unit 205) that updates the information stored in the storage unit 82 using images captured by a camera (e.g., a camera 24) that captures the arrangement of each object from the front when the determination unit 83 determines that an object has been placed.

[0100] Such a configuration allows for the accurate identification of the type of object a person picks up.

[0101] Furthermore, the update unit 85 (for example, the update method determination unit 306, the correspondence relationship update unit 305) may determine which objects to update are those whose positions or types are located within a predetermined range from the position of the object acquired by the position acquisition unit 81, and update the determined objects from the information stored in the storage unit 82.

[0102] Furthermore, the object type identification device 80 may include a display quantity storage unit (for example, a display quantity storage unit 407) that stores display information that associates the location of an object with the type and number of that object. The object identification unit 84 (for example, an object recognition unit 403) may update the display information according to the type of object it identifies.

[0103] Specifically, if the determination unit 83 determines that an object has been removed, the object identification unit 84 may subtract the number of objects at the removed location from the display information stored in the display count storage unit, and update the information stored in the storage unit 82 when that number becomes 0.

[0104] Furthermore, the update unit 85 may update the information stored in the storage unit 82 if no object or person other than the target is detected in the image captured by the camera (for example, camera 24).

[0105] Furthermore, the update unit 85 may update the information stored in the storage unit 82 if the difference between the image used during the previous update of the storage unit and the newly captured image exceeds a predetermined threshold.

[0106] Furthermore, the object type identification device 80 may include a feature quantity conversion unit (for example, a feature quantity conversion unit 207) that converts an image captured by the camera into a format that can be used by the object identification unit 84 to identify the type of object.

[0107] Furthermore, the determination unit 83 may determine whether an object has been removed or placed based on information from a pressure sensor provided at the object's placement location.

[0108] Furthermore, some or all of the above embodiments may also be described as follows, but are not limited to the following.

[0109] (Note 1) An object type identification device comprising: a position acquisition unit that acquires the position of an object; a storage unit that stores information indicating the type of object at the position of each object; a determination unit that determines whether an object has been removed or placed based on sensor information; an object identification unit that identifies the type of the removed object based on the output of the position acquisition unit and the information stored in the storage unit when the determination unit determines that an object has been removed; and an update unit that updates the information stored in the storage unit using images captured by a camera that captures the arrangement of each object from the front when the determination unit determines that an object has been placed.

[0110] (Note 2) The update unit determines which objects within a predetermined range from the position of an object acquired by the position acquisition unit are to be updated, and updates the determined target from the information stored in the storage unit, as described in Note 1.

[0111] (Note 3) An object type identification device according to Note 1 or Note 2, comprising a display quantity storage unit that stores display information associating the position of an object with the type and number of the object, and an object identification unit that updates the display information according to the type of object identified.

[0112] (Note 4) The object identification unit is the object type identification device described in Note 3, which, when the determination unit determines that an object has been removed, subtracts the number of objects at the removed location from the display information stored in the display number storage unit, and updates the information stored in the storage unit when the number becomes 0.

[0113] (Note 5) The update unit is an object type identification device described in any one of Notes 1 to 4, which updates the information stored in the memory unit when no object or person other than the target is detected in the image captured by the camera.

[0114] (Note 6) The update unit is an object type identification device described in any one of Notes 1 to 5, which updates the information stored in the memory unit when the difference between the image used during the previous update of the memory unit and a newly taken image exceeds a predetermined threshold.

[0115] (Note 7) An object type identification device according to any one of Notes 1 to 6, comprising a feature conversion unit that converts an image captured by a camera into a format usable for the identification of the type of object by an object identification unit.

[0116] (Note 8) The determination unit is an object type identification device described in any one of Notes 1 to 7, which determines whether an object has been removed or placed based on information from a pressure sensor installed at the object's placement location.

[0117] (Note 9) An object type identification method characterized by determining whether an object has been removed or placed based on sensor information, obtaining the position of the object if it is determined that an object has been removed, identifying the type of object that has been removed based on the information stored in a storage unit that stores information indicating the type of object at the position of each object and the acquired position of the object, and updating the information stored in the storage unit using an image captured by a camera that captures the arrangement of each object from the front side if it is determined that an object has been placed.

[0118] (Note 10) The method for identifying the type of object described in Note 9, which determines which object's location or type is to be updated from the acquired object's location within a predetermined range, and updates the determined target from the information stored in the memory unit.

[0119] (Note 11) An object type identification program that causes a computer to perform a position acquisition process to acquire the position of an object, a determination process to determine whether an object has been removed or placed based on sensor information, an object identification process to identify the type of object that has been removed based on the acquisition result of the position acquisition and information stored in a storage unit that stores information indicating the type of object at the position of each object, and an update process to update the information stored in the storage unit using images captured by a camera that captures the arrangement of each object from the front side, if the determination process determines that an object has been placed.

[0120] (Note 12) An object type identification program as described in Note 11, which causes the computer to determine, in the update process, the location or type of an object that is within a predetermined range from the location of the object obtained in the location acquisition process to be the target for updating, and to update the determined target from the information stored in the memory unit.

[0121] This application claims priority based on Japanese Patent Application No. 2016-076511, filed on April 6, 2016, and incorporates all of its disclosures herein. [Industrial applicability]

[0122] The object type identification device described in the above embodiment is suitably applied, for example, to analyzing optimal shelf placement in retail stores such as convenience stores. By using the object type identification device for such analyses, it becomes possible to obtain useful marketing information.

[0123] The present invention has been described above using the embodiments described above as exemplary examples. However, the present invention is not limited to the embodiments described above. That is, the present invention can be applied in various forms that can be understood by those skilled in the art, within the scope of the present invention. [Explanation of symbols]

[0124] 11, 21, 32 shelves 12,22 Camera (Sensor) 13, 23, 25 Information Processing Terminals 14 Compatible Tables 24 cameras 31 Camera images 33 Object Database 201,301,401 Motion recognition section 202,302,402 Object coordinate acquisition unit 203,303,403 Object recognition unit 204,304,404 Correspondence Relationship Memory Unit 205,305,405 Correspondence Update Section 206,306,406 Update method determination section 207 Feature Conversion Unit 407 Display number storage unit

Claims

1. A position acquisition means that acquires the position in real space of an object that the person is acting on, based on the two-dimensional coordinates of a person's hand image identified from an image captured by a camera, and information on the correspondence between the two-dimensional coordinates and coordinates in real space. A storage means for storing information indicating the type of object at the object's location, A determination means that determines, based on an image captured by a camera, whether an object has been removed from its placement location or placed in a location different from the location stored in the storage means, When the determination means determines that an object has been removed, an object identification means identifies the type of the removed object based on the acquired position of the object and the information stored in the storage means. The system includes an update means that, in response to the determination means determining that the object has been placed in the different position, updates the position information of the placed object stored in the storage means to the different position. A device for identifying the type of object, characterized by the following features.

2. The update means determines which objects to update are those whose positions or types are located within a predetermined range from the position of the object acquired by the position acquisition means, and updates the determined objects from the information stored in the storage means. The object type identification device according to claim 1.

3. The update means updates the information stored in the storage means when no object or person other than the target is detected in the image captured by the camera. The object type identification device according to claim 1 or 2.

4. The update means updates the information stored in the storage means when the difference between the image used during the previous update of the storage means and the newly captured image exceeds a predetermined threshold. The object type identification device according to any one of claims 1 to 3.

5. The determination means determines whether an object has been removed or placed based on information from a pressure sensor installed at the object's placement location. The object type identification device according to any one of claims 1 to 4.

6. Computers Based on the image captured by the camera, it is determined whether the object has been removed from its placement location, or whether the object has been placed in a location different from the location stored in the storage means that stores information indicating the type of object at the object's location. If it is determined that the object has been removed, Based on the two-dimensional coordinates of a person's hand image identified from an image captured by a camera, and information on the correspondence between the two-dimensional coordinates and coordinates in real space, the real-space position of the object that the person was acting on is obtained. Based on the location of the acquired object and the information stored in the storage means, the type of the retrieved object is identified. When it is determined that an object has been placed in a different location, the storage means updates the location information of the placed object to the different location. A method for identifying the type of object, characterized by the following features.

7. On the computer, A position acquisition process that acquires the real-space position of the object that the person was acting on, based on the two-dimensional coordinates of a person's hand image identified from an image captured by a camera, and information on the correspondence between the two-dimensional coordinates and coordinates in real space. A determination process that determines, based on an image captured by a camera, whether an object has been removed from its placement location, or whether an object has been placed in a location different from the location stored in a storage means that stores information indicating the type of object at the object's location. When it is determined that an object has been retrieved, an object identification process is performed to identify the type of the retrieved object based on the location of the retrieved object and the information stored in the storage means, and If the determination process determines that the object has been placed in the different location, an update process is performed to update the location information of the placed object stored in the storage means to the different location. A program for identifying the type of object to execute an action.

Images