Object type specifying device, object type specifying method and object type specifying program
The object type identification device accurately identifies objects by using sensors and cameras to track positions and update information in real-time, addressing the challenge of dynamic object placement and movement.
Patent Information
- Application Number
- JP2024150476
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2016-04-06
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2037-04-03
AI Technical Summary
Existing systems struggle to accurately identify the type of object picked up by a person, especially when there are changes in object placement or movement, leading to incorrect identification.
An object type identification device that uses a combination of sensors and cameras to track object positions, update information in real-time, and determine the type of object based on sensor data and image analysis, ensuring accurate identification.
Enables precise identification of the type of object picked up by a person, even in dynamic environments, by continuously updating the object database with camera images and sensor data.
Smart Images

Figure 2024164265000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an object type identification device, an object type identification method, and an object type identification program for identifying the type of a target object. [Background technology]
[0002] There are various known methods for identifying the type of object placed at a specific location. For example, Patent Document 1 describes a system that photographs customers and displayed products and automatically monitors for unauthorized removal of products based on the photographed images. In the system described in Patent Document 1, a camera is placed facing the shelves on which the objects are placed, and the image photographed by the camera is analyzed to recognize the action of the person picking up which product. The system described in Patent Document 1 then uses information stored in a memory to identify the product that the person picked up.
[0003] Patent Document 2 describes a device that monitors the state of products displayed on shelves using captured images and determines whether or not the products need to be rearranged. The device described in Patent Document 2 also has a camera positioned facing the shelves on which objects are placed, and determines whether or not a store clerk is rearranging the products based on the captured images.
[0004] Patent Document 3 describes placing a camera on a shelf to recognize a person's movements. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] JP 2004-171240 A [Patent Document 2] Patent No. 5673888 [Patent Document 3] US Patent Application Publication No. 2014 / 0132728 Summary of the Invention [Problem to be solved by the invention]
[0006] On the other hand, in the case of the system described in Patent Document 1, since motion recognition is performed based on image analysis, it is difficult to accurately recognize a person's motion, and there are cases where the type of product cannot be identified.
[0007] Furthermore, in the case of the system described in Patent Document 1, if a person mistakenly moves a product to another location, or if a new product is placed on a shelf, the information stored in the memory cannot be correctly compared. Therefore, there is a problem that the type of product picked up by the person cannot be correctly identified. The same applies when the method described in Patent Document 3 is used.
[0008] Furthermore, the device described in Patent Document 2 is designed to determine whether or not a store clerk is arranging products, and it is difficult to identify the type of product in the first place.
[0009] Therefore, an object of the present invention is to provide an object type identification device, an object type identification method, and an object type identification program that can correctly identify the type of an object that a person has picked up. [Means for solving the problem]
[0010] The object type identification device according to the present invention is characterized in that it comprises a position acquisition means for acquiring the position of an object, a memory means for storing information indicating the type of object at the position of each object, a determination means for determining whether an object has been removed or placed based on information from the sensor, an object identification means for identifying the type of object removed based on the output of the position acquisition means and the information stored in the memory means when the determination means determines that an object has been removed, and an update means for updating the information stored in the memory means using an image captured by a camera that captures the arrangement of each object from the front side when the determination means determines that an object has been placed.
[0011] The object type identification method according to the present invention is characterized in that it determines whether an object has been removed or placed based on sensor information, obtains the position of the object if it is determined that an object has been removed, identifies the type of object removed based on the information stored in a memory means that stores information indicating the type of object at each object's position and the obtained object position, and if it is determined that an object has been placed down, updates the information stored in the memory means using an image captured by a camera that captures the arrangement of each object from the front side.
[0012] The recording medium according to the present invention records an object type identification program which causes a computer to execute a position acquisition process for acquiring the position of an object, a judgment process for determining whether an object has been removed or placed based on sensor information, an object identification process for identifying the type of object removed based on the position acquisition result and information stored in a storage means which stores information indicating the type of object at the position of each object if the judgment process determines that an object has been removed, and an update process for updating the information stored in the storage means using an image captured by a camera which captures the arrangement of each object from the front side if the judgment process determines that an object has been placed. Effect of the Invention
[0013] According to the present invention, the type of object picked up by a person can be correctly identified. [Brief description of the drawings]
[0014] [Figure 1] 1 is an explanatory diagram illustrating an example of a situation in which an object type identification method according to an embodiment is used; [Diagram 2] FIG. 2 is an explanatory diagram illustrating an example of a configuration for implementing an object type identification method according to an embodiment. [Diagram 3] 1 is an explanatory diagram showing an outline of an object type identification device according to an embodiment; [Figure 4] 1 is a block diagram showing an example of the configuration of an object type identification device according to a first embodiment. [Diagram 5]FIG. 11 is an explanatory diagram showing an example of setting a correspondence relationship based on a captured image. [Figure 6] FIG. 13 is an explanatory diagram showing an example of a camera image after an object has been changed. [Figure 7] FIG. 11 is an explanatory diagram illustrating an example of a process for updating a correspondence relationship; [Figure 8] 5 is a flowchart showing an example of the operation of the object type identification device of the first embodiment. [Figure 9] 13 is a flowchart illustrating an example of a process for identifying a taken-out product. [Figure 10] FIG. 4 is an explanatory diagram illustrating an example of information stored in a correspondence relationship storage unit. [Figure 11] FIG. 11 is an explanatory diagram showing an example of a list of products to be taken out. [Figure 12] 13 is a flowchart illustrating an example of a process for updating information stored in a correspondence relationship storage unit. [Figure 13] FIG. 11 is a block diagram showing a modified example of the object type identification device of the first embodiment. [Figure 14] FIG. 11 is a block diagram showing an example of the configuration of an object type identification device according to a second embodiment. [Figure 15] 10 is a flowchart showing an example of the operation of the object type identification device according to the second embodiment. [Figure 16] FIG. 13 is a block diagram showing an example of the configuration of an object type identification device according to a third embodiment. [Figure 17] 13 is a flowchart showing an example of the operation of the object type identification device according to the third embodiment. [Figure 18] 1 is a block diagram showing an overview of an object type identification device according to an embodiment; [Figure 19] 1 is a block diagram showing an example of a hardware configuration of an object type identification device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] First, the object type identification method of the embodiment will be outlined with reference to FIG. 1. FIG. 1 is an explanatory diagram showing an example of a situation in which the object type identification method is used. In the example shown in FIG. 1, a plurality of objects are arranged on a shelf 11, and the actions of a person in front of the shelf 11 are recognized using a camera 12 and an information processing terminal 13. Note that, although a commodity is given as an example of an object, the object is not limited to a commodity intended for sale. The object means an article, a commodity, or a thing that has a shape and can be seen or touched by a person. In addition, the information processing terminal 13 holds a correspondence table 14 that associates the type of object with coordinates (hereinafter, referred to as object coordinates) indicating the position of the object. In the following, a method of identifying the position of each object based on the object coordinates will be described, but the method of identifying the position of the object is not limited to the object coordinates.
[0016] The object coordinates may be expressed, for example, as a three-dimensional position (x, y, z) with a certain point in real space as the origin, or as a two-dimensional position (x, y) on the front surface of shelf 11 with a certain point on the front surface as the origin. In the example shown in Fig. 1, the position of each object is expressed in two-dimensional coordinates with the upper left corner of the front surface of shelf 11 as the origin. In the example shown in Fig. 1, the type of object identified by object coordinates (1, 0) is represented as "C".
[0017] Here, when an object moves from the position indicated by the object coordinates of the object, a discrepancy occurs between the correspondence relationship between the object and the object coordinates set in the correspondence table 14 and the actual correspondence relationship. When such a discrepancy occurs, the object recognition accuracy decreases. Therefore, in the object type identification method of this embodiment, the object position is appropriately updated to match the actual object position.
[0018] FIG. 2 is an explanatory diagram showing an example of a configuration for realizing an object type identification method. The object type identification system 20 illustrated in FIG. 2 is an example of a system including the object type identification device of the present invention. The object type identification system 20 recognizes the movement of a person in front of the shelf 21 using an information processing terminal 23 connected to a camera 22, acquires the object coordinates on the shelf 21, and recognizes the target object using a correspondence table (e.g., correspondence table 14). The object type identification system 20 also uses a camera 24 and an information processing terminal 25 to take an image of the front (front side) of the shelf 21, acquires the type of object and the object coordinates by image recognition, and updates the correspondence table 14. The type of object acquired by image recognition is, for example, the name, product name, size, price, etc. of the object.
[0019] FIG. 3 is an explanatory diagram showing an outline of an object type identification device. Table T shown in FIG. 3 corresponds to the correspondence table 14 shown in FIG. 1. In the example shown in FIG. 3, the position of the product is expressed in two-dimensional coordinates, and the position of the product for each time is cumulatively recorded. The contents of Table T are updated at a determined update timing or periodically. The time in Table T indicates, for example, the update timing. In the example shown in FIG. 3, the correspondence is cumulatively recorded, but one master may be provided, and the contents may be overwritten and updated.
[0020] 3 illustrates an example of an object identification operation. When an image sensor such as an RGB camera or a depth camera detects an operation of a person P putting his / her hand into a shelf S, the object type identification device detects the position where the hand was put based on the information from the sensor (step S1). For example, if the detected position is a position specified by object coordinates (4,4) and the time is 16:45, the object type identification device refers to table T and identifies the object as product B (step S2).
[0021] Moreover, the lower part of Fig. 3 illustrates an operation for recognizing the type of object. A camera C is installed to capture an image of the arrangement of each object from the front side, and each object placed on the shelf S is captured by this camera C. Camera C corresponds to camera 24 in Fig. 2. In the present invention, table T is updated when an object is placed on the shelf or at a predetermined timing (step S3). In this way, table T is appropriately updated as needed, and as a result, the accuracy of image recognition is improved, and it becomes possible to correctly identify the type of object that a person has picked up.
[0022] Each embodiment will be described below with reference to the drawings. Embodiment 1. FIG. 4 is a block diagram showing an example of the configuration of an object type identification device according to the first embodiment. The object type identification device 200 of this embodiment includes an action recognition unit 201 , an object coordinate acquisition unit 202 , an object recognition unit 203 , a correspondence relationship storage unit 204 , a correspondence relationship update unit 205 , and an update method determination unit 206 .
[0023] The action recognition unit 201 recognizes the action of a subject. In this embodiment, the subject is something that brings about some change to the state of an object, such as a person or a robot. The method by which the action recognition unit 201 recognizes the action of the subject is arbitrary. In this embodiment, the action recognition unit 201 judges whether an object has been taken out or placed based on information from the camera 22 (sensor). For this reason, the action recognition unit 201 can be called a judgment unit.
[0024] The action recognition unit 201 may use an image sensor such as an RGB camera or a depth camera as the camera 22, and recognize the action of the subject such as "taking out an object", "putting down an object", or "no action" from the change in color or volume around a specific part such as a hand. A depth camera is a camera that can measure depth information from the camera to the imaged object in addition to RGB information acquired by normal imaging.
[0025] Also, for example, if a pressure sensor is installed in advance at the location where the object is to be placed (e.g., the surface on which the object is placed), the action recognition unit 201 may determine whether the object has been removed or placed based on information from the pressure sensor instead of the camera 22.
[0026] The sensor used to determine the motion is not limited to a pressure sensor, and may be, for example, a sensor using infrared rays, etc. Furthermore, the motion recognition unit 201 may recognize the motion of a person, a robot, etc., using any other method capable of determining the motion.
[0027] The object coordinate acquisition unit 202 acquires the position of an object that is the target of an action. In this embodiment, the object coordinate acquisition unit 202 acquires object coordinates as the position of the object. Therefore, the object coordinate acquisition unit 202 can be said to be an object position acquisition unit. As described above, the object coordinates refer to the coordinates of the position of an object that is the target of an action of a person, a robot, or the like.
[0028] The object coordinate acquisition unit 202 acquires object coordinates using, for example, an image captured by the camera 22. Specifically, the object coordinate acquisition unit 202 specifies two-dimensional coordinates on an image of a person's body part, such as a face or a hand, from image information that can be acquired by, for example, an RGB camera such as a surveillance camera. The object coordinate acquisition unit 202 may acquire object coordinates from the image by previously associating the specified two-dimensional coordinates with object coordinates in real space.
[0029] Furthermore, when a depth camera is used instead of an RGB camera, it is possible to acquire real space three-dimensional coordinates of a person's body parts, and the object coordinate acquisition unit 202 may use these as object coordinates.
[0030] Further, other than the method using an image, the object coordinate acquisition unit 202 may acquire the object coordinates using the pressure sensor described above. For example, when a sensor such as a pressure sensor is installed in advance on the surface on which the object is placed, the object coordinate acquisition unit 202 may acquire, as the object coordinates, the coordinates at which the pressure changes significantly when an "acquire" (take out) action is performed on the object.
[0031] However, the method by which the object coordinate acquiring unit 202 acquires the object coordinates is not limited to the above-mentioned method.
[0032] The correspondence storage unit 204 stores information indicating the type of object at the position of each object. Specifically, the correspondence storage unit 204 stores the correspondence between the type of object to be recognized and the object coordinates, which are the coordinates at which the object is located. In addition to the correspondence between the object and the object coordinates, the correspondence storage unit 204 may also store the update time when the correspondence was updated in association with the correspondence. In this embodiment, it is assumed that the correspondence storage unit 204 holds the correspondence table 14 (i.e., the correspondence between the object coordinates and the object type) illustrated in FIG. 1.
[0033] The correspondence between object coordinates and object types is set in advance. FIG. 5 is an explanatory diagram showing an example of setting the correspondence based on a captured image. For example, assume that the object type identification device 200 is connected to the camera 24 and the information processing terminal 25 illustrated in FIG. 2. In this case, when the camera 24 acquires the camera image 31 illustrated in FIG. 5, the information processing terminal 25 may determine which two-dimensional coordinates of the camera image 31 correspond to the two-dimensional coordinates of the shelf 32, and may determine object coordinates for each object and set them in the correspondence table. In addition, the correspondence storage unit 204 may store not only the correspondence between the object and the object coordinates, but also the image feature amount of each object extracted from the camera image 31 and other images in the object database 33. The object database 33 is stored in the correspondence storage unit 204, for example.
[0034] The object recognition unit 203 recognizes the type of object that is the target of the action. Specifically, when the action recognition unit 201 determines that an object has been taken out, the object recognition unit 203 identifies the type of the taken out object based on the position of the object acquired by the object coordinate acquisition unit 202 and the information stored in the correspondence storage unit 204. Therefore, the object recognition unit 203 can be called an object identification unit.
[0035] The object recognition unit 203 may refer to the object coordinates acquired by the object coordinate acquisition unit 202 and the correspondence table 14 held in the correspondence storage unit 204, and may obtain the type of object associated with the object coordinates as a recognition result. Furthermore, the object recognition unit 203 may obtain the action of the subject recognized by the action recognition unit 201 as a recognition result, in addition to the type of recognized object.
[0036] For example, it is assumed that the correspondence storage unit 204 stores the correspondence table 14 illustrated in Fig. 1. It is assumed that the action recognition unit 201 recognizes the action of "acquire" by the subject, and the object coordinate acquisition unit 202 acquires the object coordinates (1,0). In this case, the object recognition unit 203 recognizes the object targeted by the action as "C" based on the position indicated by the acquired object coordinates. At this time, the object recognition unit 203 may recognize the object "C" as the recognition result, or may recognize the content "'acquire' object "C"" including the action as the recognition result, or may further recognize the content "'acquire' object "C" from object coordinates (1,0)" including the object coordinates as the recognition result.
[0037] However, the method by which the object recognition unit 203 recognizes an object that is the target of an action is not limited to the above-mentioned method. In the above-mentioned example, the object recognition unit 203 acquires the type of object whose object coordinates match from the correspondence table 14, but the type of the corresponding object may be acquired even when the object coordinates are within a certain range.
[0038] For example, p(o) is the object coordinate associated with object o in the correspondence table 14, Distance(p1, p2) is the distance between p1 and p2, and threshold distis a threshold value, and the types of objects in the correspondence table 14 are a set O. The identity of the objects is expressed by the following formula 1.
[0039]
number
[0040] The correspondence update unit 205 updates the correspondence between the information indicating the type of object stored in the correspondence storage unit 204 and the position of the object. Specifically, when the action recognition unit 201 determines that an object has been placed, the correspondence update unit 205 updates the type and position of the object. In this embodiment, the correspondence update unit 205 updates the information stored in the correspondence storage unit 204 using an image of the arrangement of each object captured from the front side by the camera 24.
[0041] The correspondence update unit 205 recognizes the object coordinates and the type of object from an image captured by the camera 24 (for example, camera image 31 shown in FIG. 5), and identifies the type of object and the object coordinates on the image by matching the image with image features stored in a database (for example, object database 33 shown in FIG. 5).Then, the correspondence update unit 205 refers to the correspondence stored in the correspondence storage unit 204, and updates the correspondence between the identified object coordinates and the object type.
[0042] FIG. 6 is an explanatory diagram showing an example of a camera image after an object has been changed. FIG. 7 is an explanatory diagram showing an example of a process for updating the correspondence relationship. For example, when the camera 24 captures the camera image shown in FIG. 6, the correspondence relationship update unit 205 identifies the type of object and the object coordinates on the image from the captured camera image. In this example, the correspondence relationship update unit 205 identifies that the object at the position indicated by the object coordinates (1,0) has been changed from "C" to "X" and the object at the position indicated by the object coordinates (1,1) has been changed from "D" to "Y" from the initial state shown in FIG. 5. Therefore, the correspondence relationship update unit 205 updates the object corresponding to the object coordinates (1,0) to "X" and updates the object corresponding to the object coordinates (1,1) to "Y".
[0043] Update method determination section 206 determines the method for updating the correspondence stored in correspondence storage section 204. That is, correspondence update section 205 updates the information in correspondence storage section 204 based on the determination by update method determination section 206. Note that if the update method is determined in advance, object type identification device 200 does not need to be equipped with update method determination section 206.
[0044] In this embodiment, when the action recognition unit 201 determines that an object has been placed, the correspondence update unit 205 updates the type and position of the object. Therefore, the update method determination unit 206 detects that the action recognition unit 201 has placed an object, and determines the detected timing as update timing T100. However, the update timing determined by the update method determination unit 206 is not limited to the timing when the object is placed. Other methods for determining the update timing will be described below.
[0045] The update method determination unit 206 may determine, for example, the update timing T101 to be a predetermined time interval. For example, if the time interval is predetermined to be 30 minutes, the update method determination unit 206 may determine the update timing T101 to be 9:00, 9:30, 10:00, and so on.
[0046] Furthermore, the update method determination unit 206 may determine the update timing T102 using a camera image. Specifically, when there is a large difference between an image used in the previous update of the correspondence storage unit 204 and a newly captured image (i.e., the difference between the two images exceeds a predetermined threshold), the update method determination unit 206 may determine the time when the image was captured as the update timing T102.
[0047] The update method determination unit 206, for example, obtains camera images taken by a fixed camera at multiple times and calculates background differences of target areas in the images. At this time, the update method determination unit 206 may determine the time at which a certain level of change is detected as the update timing T102. Here, the target area may be, for example, the front of a shelf on which an object is placed, or the environment around the shelf.
[0048] Furthermore, the update method determination unit 206 may determine the update timing T103 using information acquired from a sensor installed on the surface on which the object is placed. For example, if a pressure sensor is installed on the surface on which the object is placed, the update method determination unit 206 may receive output values from the pressure sensor in chronological order, and may determine the time when the change in the output value exceeds a pre-specified threshold as the update timing T103.
[0049] Furthermore, the update method determination unit 206 may re-determine a new update timing based on a plurality of update timings to be considered. For example, the update method determination unit 206 may generate an update queue when any one of the above-described update timings T100, T101, T102, and T103 is determined, and may re-determine the update timing T104 by another method when the queue exists.
[0050] For example, IsToUpdated is prepared as a flag indicating whether an update queue has occurred. When an update queue occurs, the update method determination unit 206 sets a flag IsToUpdated=true. Only when the flag IsToUpdated=true, the update method determination unit 206 detects objects or people other than the object to be recognized in the target area of the image captured by the camera. If no other object or person is detected, the update method determination unit 206 may determine the timing when no object or person is detected as the update timing T104. Note that the update method determination unit 206 may set the flag IsToUpdated=false at the timing when the update timing T104 is determined.
[0051] For example, if the update method is determined based only on the camera, there is a risk that the update will not be performed appropriately if there is something (such as a person) blocking the update timing target. However, in this embodiment, the update method determination unit 206 determines the update method based on multiple pieces of information, so that it is possible to prevent inappropriate updates from being performed.
[0052] Action recognition unit 201, object coordinate acquisition unit 202, object recognition unit 203, correspondence update unit 205, and update method determination unit 206 are realized by a CPU (Central Processing Unit) of a computer that operates according to a program (object type identification program). For example, as shown in Fig. 19, the program may be stored in a storage unit (e.g., correspondence storage unit 204) of object type identification device 200, and the CPU may read the program and operate as action recognition unit 201, object coordinate acquisition unit 202, object recognition unit 203, correspondence update unit 205, and update method determination unit 206 according to the program.
[0053] Moreover, the action recognition unit 201, the object coordinate acquisition unit 202, the object recognition unit 203, the correspondence update unit 205, and the update method determination unit 206 may each be realized by dedicated hardware. Moreover, the object type identification device may be configured by two or more physically separated devices connected by wire or wirelessly. The correspondence storage unit 204 is realized by, for example, a magnetic disk device.
[0054] Next, the operation of the object type identification device of this embodiment will be described. Fig. 8 is a flowchart showing an example of the operation of the object type identification device of this embodiment. It is assumed that the correspondence storage unit 204 stores information indicating the type of object at each object position.
[0055] First, the action recognition unit 201 detects whether or not a person has performed an action of putting a hand into a shelf (step S101). The action recognition unit 201 detects the action of the person putting a hand into a shelf based on information acquired from, for example, an RGB camera or a depth camera.
[0056] When the action of putting a hand into the shelf by the person is detected (YES in step S102), the action recognition unit 201 judges whether the action is an action of taking out an object or an action of placing an object (step S103). The action recognition unit 201 judges the action content of the person based on, for example, an image captured by the camera 22 or information acquired from a switch (pressure sensor) on the shelf.
[0057] When the action of taking out an object is detected ("take out" in step S103), the object recognition unit 203 identifies the taken-out product (step S104). After that, the process proceeds to step S108.
[0058] On the other hand, if a motion of placing an object is detected ("Place" in step S103), the update method determination unit 206 determines the update timing (step S105). If a motion of putting a hand on the shelf is not detected (NO in step S102) and the regular update time has arrived (YES in step S106), the update method determination unit 206 determines the update timing (step S105).
[0059] The correspondence update unit 205 updates the information stored in the correspondence storage unit 204 based on the determined update timing (step S107). After that, the process proceeds to step S108.
[0060] If it is not the update time in step S106 (NO in step S106), or after step S104 or step S107, the action recognition unit 201 determines whether it is the end time of a series of processes (step S108). If it is not the end time (NO in step S108), the processes from step S101 onwards are performed. On the other hand, if it is the end time (YES in step S108), the process ends.
[0061] Fig. 9 is a flow chart showing an example of a process for identifying a taken-out product in step S104 in Fig. 8. When the action recognition unit 201 detects an action of taking out an object, the object coordinate acquisition unit 202 identifies the object coordinates targeted by the action (step S111). The object recognition unit 203 reads information on the corresponding coordinates stored in the correspondence storage unit 204 (step S112).
[0062] Fig. 10 is an explanatory diagram showing an example of information stored in the correspondence storage unit 204. The example shown in Fig. 10 shows that product A is placed at a position represented by object coordinates (1,0), and product B is placed at a position represented by object coordinates (1,1). For example, when the object coordinate acquisition unit 202 acquires object coordinates (1,1), the object recognition unit 203 reads product B at coordinates (1,1) from the correspondence storage unit 204. As a result, the object recognition unit 203 identifies the taken out product.
[0063] Thereafter, the object recognition unit 203 may create a list of the taken-out products (step S113). FIG. 11 is an explanatory diagram showing an example of the list of taken-out products. In the example shown in FIG. 11, the list includes the acquired object coordinates, product name, and time. Note that the object recognition unit 203 may write out the product name in the list in real time, and may later use this list for offline analysis.
[0064] Fig. 12 is a flowchart showing an example of a process for updating the information stored in the correspondence storage unit 204 in step S107 of Fig. 8. When the update timing is determined, the update method determination unit 206 recognizes a person in the target area of the image captured by the camera 24 (step S121). If a person is recognized in the target area of the image (YES in step S122), the process returns to step S121. On the other hand, if a person is not recognized in the target area of the image (NO in step S122), the object recognition unit 203 recognizes the placed product (step S123).
[0065] Then, the correspondence update unit 205 updates the information of the correspondence table 14 stored in the correspondence storage unit 204 (step S124). At this time, the correspondence update unit 205 may create a new table or record to manage the correspondence between the product and the location, instead of updating the same table in the correspondence storage unit 204.
[0066] As described above, in this embodiment, the action recognition unit 201 judges whether an object has been taken out or placed down based on the sensor information. When the action recognition unit 201 judges that an object has been taken out, the object recognition unit 203 identifies the type of the taken out object based on the output of the object coordinate acquisition unit 202 that acquires the position of the object and the information stored in the correspondence relationship storage unit 204 (the relationship between the type of object and the position of the object). On the other hand, when the action recognition unit 201 judges that an object has been placed down, the correspondence relationship update unit 205 updates the information stored in the correspondence relationship storage unit 204 using an image captured by the camera 24. With such a configuration, the type of object picked up by a person can be correctly identified.
[0067] That is, in this embodiment, two pieces of hardware (camera 22 (sensor) and camera 24) are linked to each other, and detection by one piece of hardware triggers the operation of the other piece of hardware. With such a configuration, it becomes possible to correctly identify the type of object picked up by a person.
[0068] Generally, the same product is placed at the same position on the display shelf. Therefore, even if an object is removed, the type of the object generally does not change, and there is little need to update the contents of the correspondence table 14. On the other hand, when an object is placed at a certain position on the display shelf, the content of the object is often unknown, so in this embodiment, the placement of the object is detected and the correspondence table 14 is updated.
[0069] In this embodiment, the update method determination unit 206 appropriately determines the timing to update the correspondence table 14, and the correspondence update unit 205 updates the correspondence table 14 in accordance with the actual state. Therefore, it is possible to accurately recognize an object that is the target of an action.
[0070] Next, a modified example of this embodiment will be described. Fig. 13 is a block diagram showing a modified example of object type identification device 200 of the first embodiment. Object type identification device 200a illustrated in Fig. 13 includes an action recognition section 201, an object coordinate acquisition section 202, an object recognition section 203, a correspondence storage section 204, a correspondence update section 205, an update method determination section 206, and a feature conversion section 207. That is, object type identification device 200 of this modified example further includes feature conversion section 207 in addition to object type identification device 200 of the first embodiment.
[0071] In the first embodiment, the information (database) stored in the correspondence storage unit 204 is shared by a plurality of devices (sensors such as the camera 22 and the camera 24). Therefore, the feature conversion unit 207 registers feature amounts such as color information and size of an object in an image captured by the camera 24 in the correspondence storage unit 204 (for example, the object database 33), and further converts the feature amounts into a format usable by the camera 22 (sensor). Specifically, the feature conversion unit 207 converts the image captured by the camera 24 into a format usable for the object recognition unit 203 to identify the type of object.
[0072] In this case, the object recognition unit 203 can recognize an object based on size on the camera 22 (sensor) side, based on the size of the object photographed by the camera 24. The object recognition unit 203 can also perform matching on the camera 22 (sensor) side, based on color information photographed by the camera 24. That is, by using such converted information, it becomes possible to improve the recognition accuracy of the object. Furthermore, when performing such conversion, it becomes possible to use the position of the object as one of the feature quantities. Embodiment 2. Next, a second embodiment will be described. Fig. 14 is a block diagram showing an example of the configuration of an object type identification device of the second embodiment. An object type identification device 300 of this embodiment includes an action recognition unit 301, an object coordinate acquisition unit 302, an object recognition unit 303, a correspondence storage unit 304, a correspondence update unit 305, and an update method determination unit 306.
[0073] The action recognition unit 301, object coordinate acquisition unit 302, object recognition unit 303 and correspondence storage unit 304 of this embodiment are similar to the action recognition unit 201, object coordinate acquisition unit 202, object recognition unit 203 and correspondence storage unit 204 of the first embodiment, respectively. The object type identification device 300 may include a feature conversion unit 207 of the modified example of the first embodiment.
[0074] The update method determination unit 306 determines the update timing and the update method based on at least the recognition result by the action recognition unit 301 and the information on the object coordinates acquired by the object coordinate acquisition unit 302. Here, the update method means a method for determining the object coordinates to be updated and a method for determining candidates for the type of object to be updated. For example, the method for determining the update timing used is the method used by the update method determination unit 206 in the first embodiment.
[0075] The update method determination unit 306 determines that the position or type of an object existing within a predetermined range from the object coordinates acquired by the object coordinate acquisition unit 302 is to be updated. Then, the correspondence update unit 305 updates the determined target (specifically, the correspondence between the object position and type) from among the information stored in the correspondence storage unit 304.
[0076] The update method determination unit 306 may determine the update method based on, for example, the object coordinates acquired by the object coordinate acquisition unit 302. Specifically, the update method determination unit 306 may determine that, with regard to the object coordinates to be updated, the coordinates that are at a distance equal to or less than a certain threshold from the object coordinates acquired by the object coordinate acquisition unit 302 are to be updated.
[0077] Furthermore, the update method determination unit 306 may determine the update method when the action recognition unit 301 recognizes an action of moving an object. In this case, the update method determination unit 306 may determine, as the update target, coordinates that are at a distance equal to or less than a threshold value from the source object coordinates and the destination object coordinates acquired by the object coordinate acquisition unit 302. In this case, the update method determination unit 306 may determine that, with regard to the type of the destination object to be updated, update candidates are limited to two types: the type of the object originally placed at that location, and the type of the source object.
[0078] The correspondence updating unit 305 updates the information in the correspondence storage unit 304 (specifically, the information in the correspondence table 14) based on the update timing and update method determined by the update method determination unit 306. In this embodiment, the correspondence updating unit 305 may update the correspondence table 14 held in the correspondence storage unit 304 only for the object coordinates determined by the update method determination unit 306 and only for the candidate object types determined by the update method determination unit 306.
[0079] As in the first embodiment, the action recognition unit 301, the object coordinate acquisition unit 302, the object recognition unit 303, the correspondence update unit 305, and the update method determination unit 306 are realized by the CPU of a computer that operates in accordance with a program (object type identification program).
[0080] Next, the operation of the object type identification device of this embodiment will be described. Fig. 15 is a flowchart showing an example of the operation of the object type identification device of this embodiment. It is assumed that the correspondence storage unit 304 stores information indicating the type of object at each object position.
[0081] The operation of this embodiment is similar to that of the first embodiment illustrated in Fig. 8. However, it differs from the first embodiment in that in step S205, the update method determination unit 306 determines the update timing and the update method.
[0082] In step S205, the update method determination unit 306 determines that the position or type of an object existing within a predetermined range from the object coordinates acquired by the object coordinate acquisition unit 302 is to be updated.
[0083] Then, in step S107, the correspondence update unit 305 updates the information stored in the correspondence storage unit 304 based on the determined update timing and update method. Specifically, the correspondence update unit 305 updates the determined target (specifically, the correspondence between the position and type of object) among the information stored in the correspondence storage unit 304 at the determined update timing. The other processes are the same as those in the first embodiment.
[0084] As described above, in this embodiment, the update method determination unit 306 determines that the position or type of an object existing within a predetermined range from the position of the object acquired by the object coordinate acquisition unit 302 is to be updated. Then, the correspondence update unit 305 updates the determined object from among the information stored in the correspondence storage unit 304. With such a configuration, in addition to the effects of the first embodiment, the correspondence table 14 can be updated efficiently and accurately. Embodiment 3. Next, a third embodiment will be described. Fig. 16 is a block diagram showing an example of the configuration of an object type identification device of the third embodiment. An object type identification device 400 of this embodiment includes an action recognition unit 401, an object coordinate acquisition unit 402, an object recognition unit 403, a correspondence relationship storage unit 404, a correspondence relationship update unit 405, an update method determination unit 406, and a display number storage unit 407. That is, the object type identification device 400 of this embodiment further includes the display number storage unit 407 in addition to the object type identification device 200 of the first embodiment or the object type identification device 300 of the second embodiment.
[0085] The action recognition unit 401, the object coordinate acquisition unit 402, and the correspondence storage unit 404 of this embodiment are similar to the action recognition unit 201, the object coordinate acquisition unit 202, and the correspondence storage unit 204 of the first embodiment, respectively. The object type identification device 400 may include a feature conversion unit 207 of the modified example of the first embodiment.
[0086] The display number storage unit 407 stores display information including the type and number of objects arranged at one object coordinate. Specifically, the display number storage unit 407 holds display information in which the type of object and the number of objects are associated with each object coordinate. The display number storage unit 407 may hold display information including the display order. The display information is set in advance in the display number storage unit 407. The display number storage unit 407 is realized by, for example, a magnetic disk or the like.
[0087] The object recognition unit 403 updates the display information based on the object recognition result. Specifically, the object recognition unit 403 updates the number of objects based on the action recognition result by the action recognition unit 401 (specifically, the determination result indicating whether an object has been taken out or placed).
[0088] For example, if the recognition result is “obtained object “C” from object coordinates (1,0)”, the object recognition unit 403 subtracts 1 from the number of objects “C” at object coordinates (1,0) included in the display information stored in the display number memory unit 407.
[0089] The correspondence updating unit 405 and the update method determining unit 406 are similar to the correspondence updating unit 205 and the update method determining unit 206 of the first embodiment. Furthermore, in this embodiment, the update method determining unit 406 determines the update timing based on the display information stored in the display number storage unit 407.
[0090] The update method determination unit 406 may update the information stored in the correspondence relationship storage unit 404 when the action recognition unit 401 determines that an object has been removed, and the number of objects at the removed position is subtracted, resulting in the number of objects at that position becoming 0. For example, when the number of object "C" at object coordinate (1,0) in the display information stored in the display number storage unit 407 becomes 0, the update method determination unit 406 determines to immediately update the object coordinate (1,0) to "no object". The correspondence relationship update unit 405 updates the type of object corresponding to the object coordinate (1,0) in the correspondence table 14 stored in the correspondence relationship storage unit 404 to "no object" based on the determination by the update method determination unit 406.
[0091] The type of object arranged at one object coordinate is not limited to one type. That is, the display number storage unit 407 may store the display order of the objects at each object coordinate as display information. For example, suppose that the display number storage unit 407 stores display information including the display order of "C", "A", "A" (object "C" is placed at the beginning, followed by two objects "A") at object coordinates (1,0). Then, suppose that the object recognition unit 403 recognizes that "object "C" is "obtained" from object coordinates (1,0)".
[0092] In this case, the update method determination unit 406 detects from the display information stored in the display number storage unit 407 that an object "A" has been placed at the object coordinates (1,0). Since this detection indicates that the type of object has changed, the update method determination unit 406 decides to immediately update the type of object at the object coordinates (1,0) to "A". Based on the decision of the update method determination unit 406, the correspondence relationship update unit 405 updates the type of object corresponding to the object coordinates (1,0) in the correspondence table 14 stored in the correspondence relationship storage unit 404 to "A".
[0093] As in the first embodiment, the action recognition unit 401, the object coordinate acquisition unit 402, the object recognition unit 403, the correspondence update unit 405, and the update method determination unit 406 are realized by the CPU of a computer that operates in accordance with a program (object type identification program).
[0094] Next, the operation of the object type identification device of this embodiment will be described. Fig. 17 is a flowchart showing an example of the operation of the object type identification device of this embodiment. It is assumed that the correspondence storage unit 404 stores information indicating the type of object at each object position.
[0095] The operation of this embodiment is similar to that of the first embodiment illustrated in Fig. 8. It differs from the first embodiment in that a process for updating the display information (step S301) is added after the product taken out in step S104 is identified, and a process for determining the update timing based on the update of the display information (step S302) is changed.
[0096] In step S301, the object recognition unit 403 updates the display information stored in the display number storage unit 407 according to the type of the identified object. In step S302, the update method determination unit 406 determines the update timing based on the display information stored in the display number storage unit 407. Specifically, when it is determined that an object has been removed, the update method determination unit 406 may determine to update the information stored in the correspondence relationship storage unit 404 when the number of objects at the removed position becomes 0 or the type of object at that position has changed. The other processing is the same as in the first embodiment.
[0097] As described above, in this embodiment, the display number storage unit 407 stores display information that associates the position of an object with the type and number of the object. Then, the object recognition unit 403 updates the display information according to the identified type of object. Specifically, when the action recognition unit 401 determines that an object has been removed, the object recognition unit 403 subtracts the number of objects at the removed position from the display information stored in the display number storage unit 407, and updates the information stored in the storage unit when the number becomes 0. With such a configuration, in addition to the effects of the first embodiment, the correspondence table 14 can be updated efficiently and accurately.
[0098] That is, in this embodiment, the update method determination unit 406 determines the update timing and update method based on at least the display information stored in the display number storage unit 407, and the correspondence update unit 405 updates the correspondence table 14 based on the determined update timing and update method. This makes it possible to update the correspondence table 14 efficiently and accurately.
[0099] Next, an overview of the object type identification device will be described with reference to Fig. 18, which is a block diagram showing the overview of the object type identification device. The object type identification device 80 includes a position acquisition unit 81 (e.g., object coordinate acquisition unit 202) that acquires the position of an object (e.g., object coordinates), a memory unit 82 (e.g., correspondence memory unit 204) that stores information indicating the type of object at the position of each object, a determination unit 83 (e.g., action recognition unit 201) that determines whether an object has been taken out or placed based on information from a sensor (e.g., an image sensor such as an RGB camera or a depth camera), an object identification unit 84 (e.g., object recognition unit 203) that identifies the type of the object taken out based on the output of the position acquisition unit 81 and the information stored in the memory unit 82 when the determination unit 83 determines that an object has been taken out, and an update unit 85 (e.g., update method determination unit 206, correspondence update unit 205) that updates the information stored in the memory unit 82 using an image captured by a camera (e.g., camera 24) that captures the arrangement of each object from the front side when the determination unit 83 determines that an object has been placed.
[0100] Such a configuration makes it possible to correctly identify the type of object a person has picked up.
[0101] In addition, the update unit 85 (e.g., update method determination unit 306, correspondence update unit 305) may determine that the position or type of an object existing within a predetermined range from the position of the object acquired by the position acquisition unit 81 is to be updated, and update the determined target among the information stored in the memory unit 82.
[0102] Furthermore, the object type identification device 80 may include a display number storage unit (e.g., display number storage unit 407) that stores display information that associates the position of an object with the type and number of the object. Then, the object identification unit 84 (e.g., object recognition unit 403) may update the display information according to the identified type of object.
[0103] Specifically, when the determination unit 83 determines that an object has been removed, the object identification unit 84 may subtract the number of objects at the removed position from the display information stored in the display number memory unit, and update the information stored in the memory unit 82 when the number becomes zero.
[0104] Furthermore, the update unit 85 may update the information stored in the storage unit 82 when an object or person other than the target is not detected in an image captured by a camera (for example, the camera 24).
[0105] Furthermore, the update unit 85 may update the information stored in the storage unit 82 when the difference between the image used in the previous update of the storage unit and the newly captured image exceeds a predetermined threshold value.
[0106] Furthermore, the object type identification device 80 may include a feature conversion unit (for example, feature conversion unit 207) that converts the image captured by the camera into a format that can be used by the object identification unit 84 to identify the type of object.
[0107] Furthermore, the determination unit 83 may determine whether an object has been removed or placed, based on information from a pressure sensor provided at the place where the object is placed.
[0108] Furthermore, some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.
[0109] (Supplementary Note 1) An object type identification device comprising: a position acquisition unit that acquires the position of an object; a memory unit that stores information indicating the type of object at the position of each object; a determination unit that determines whether an object has been removed or placed based on sensor information; an object identification unit that, when the determination unit determines that an object has been removed, identifies the type of the object removed based on the output of the position acquisition unit and the information stored in the memory unit; and, when the determination unit determines that an object has been placed, an update unit that updates the information stored in the memory unit using an image captured by a camera that captures the arrangement of each object from the front side.
[0110] (Supplementary Note 2) An object type identification device as described in Supplementary Note 1, wherein the update unit determines that the position or type of an object existing within a predetermined range from the position of the object acquired by the position acquisition unit is to be updated, and updates the determined object among the information stored in the memory unit.
[0111] (Supplementary Note 3) An object type identification device as described in Supplementary Note 1 or Supplementary Note 2, further comprising a display number memory unit that stores display information that corresponds the position of an object to the type and number of the object, and an object identification unit that updates the display information according to the type of object identified.
[0112] (Appendix 4) An object type identification device as described in Appendix 3, wherein when the judgment unit determines that an object has been removed, the object identification unit subtracts the number of objects at the removed position from the display information stored in the display number memory unit, and updates the information stored in the memory unit when the number becomes zero.
[0113] (Appendix 5) An object type identification device described in any one of Appendices 1 to 4, wherein the update unit updates the information stored in the memory unit when an object or person other than the target is not detected in an image captured by the camera.
[0114] (Appendix 6) An object type identification device according to any one of Appendices 1 to 5, wherein the update unit updates the information stored in the memory unit when a difference between an image used in the previous update of the memory unit and a newly captured image exceeds a predetermined threshold.
[0115] (Appendix 7) An object type identification device according to any one of appendices 1 to 6, comprising a feature conversion unit that converts an image captured by the camera into a format that can be used by the object identification unit to identify the type of object.
[0116] (Appendix 8) An object type identification device according to any one of Appendices 1 to 7, wherein the determination unit determines whether an object has been removed or placed based on information from a pressure sensor provided at a location where the object is placed.
[0117] (Appendix 9) A method for identifying an object type, comprising: determining whether an object has been removed or placed based on sensor information; acquiring the position of the object when it is determined that an object has been removed; identifying the type of object removed based on the acquired position of the object and the information stored in a memory unit that stores information indicating the type of object at each object position; and updating the information stored in the memory unit using an image captured by a camera that captures the arrangement of each object from the front side when it is determined that an object has been placed.
[0118] (Appendix 10) An object type identification method as described in Appendix 9, which determines the position or type of an object existing within a predetermined range from the acquired object position as an object to be updated, and updates the determined object among the information stored in the memory unit.
[0119] (Supplementary Note 11) An object type identification program for causing a computer to execute a position acquisition process for acquiring the position of an object, a judgment process for judging whether an object has been removed or placed based on sensor information, an object identification process for identifying the type of object removed based on the result of the position acquisition and information stored in a memory unit that stores information indicating the type of object at the position of each object when it is determined in the judgment process that an object has been removed, and an update process for updating the information stored in the memory unit using an image captured by a camera that captures the arrangement of each object from the front side when it is determined in the judgment process that an object has been placed.
[0120] (Appendix 12) An object type identification program as described in Appendix 11, which causes a computer to determine, in an update process, the position or type of an object existing within a predetermined range from the position of the object acquired in the position acquisition process as the object to be updated, and updates the determined object among the information stored in the memory unit.
[0121] This application claims priority based on Japanese Patent Application No. 2016-076511, filed April 6, 2016, the disclosure of which is incorporated herein in its entirety. [Industrial Applicability]
[0122] The object type identification device described in the above embodiment is preferably applied to, for example, analysis of shelf layouts that generate good sales in retail stores such as convenience stores. By using the object type identification device for such analysis, it is possible to obtain useful marketing information.
[0123] The present invention has been described above by taking the above-mentioned embodiment as an exemplary example. However, the present invention is not limited to the above-mentioned embodiment. That is, the present invention can be applied in various aspects that can be understood by a person skilled in the art within the scope of the present invention. [Explanation of symbols]
[0124] 11,21,32 Shelves 12,22 Camera (sensor) 13,23,25 Information processing terminal 14 Corresponding Table 24 Camera 31 Camera Images 33 Object Database 201,301,401 Motion recognition section 202,302,402 Object coordinate acquisition unit 203,303,403 Object recognition unit 204, 304, 404 Correspondence memory section 205,305,405 Correspondence Update Section 206,306,406 Update method determination section 207 Feature Transformation Unit 407 Display number storage unit
Claims
1. A position acquisition means for acquiring a position of an object; a storage means for storing information indicating the type of each object at the position of the object; a determining means for determining whether an object has been placed at a position different from the position stored in the memory means, based on an image captured by a camera; and an update means for updating the position information of the placed object stored in the storage means to the different position in response to the determination means determining that the object has been placed at the different position.
1. An object type identification device comprising:
2. The update means determines a position or a type of an object that exists within a predetermined range from the position of the object acquired by the position acquisition means as a target for updating, and updates the determined target among the information stored in the storage means.
2. The object type identification device according to claim 1.
3. The update means updates the information stored in the storage means when an object or person other than the target is not detected in the image captured by the camera.
3. The object type identification device according to claim 1.
4. The update means updates the information stored in the storage means when a difference between an image used in a previous update of the storage means and a newly captured image exceeds a predetermined threshold value.
4. The object type identification device according to claim 1.
5. The determining means determines whether an object has been removed or placed on the basis of information from a pressure sensor provided at a location where the object is placed.
5. The object type identification device according to claim 1.
6. determining whether the object has been placed at a position different from a position stored in a storage means that stores information indicating the type of the object at the position of each object, based on an image captured by the camera; In response to the determination that the object has been placed at the different position, the position information of the placed object stored in the storage means is updated to the different position.
13. A method for identifying an object type, comprising:
7. On the computer, A position acquisition process for acquiring the position of an object; a determination process for determining whether an object has been placed at a position different from a position stored in a storage means that stores information indicating the type of the object at the position of each object, based on an image captured by the camera; and an updating process for updating the position information of the placed object stored in the storage means to the different position in response to the determination process that the object has been placed at the different position; An object type identification program for executing the above.
Citation Information
Patent Citations
Article management system
JP2010006557A
Personal behavior analysis device, personal behavior analysis system, personal behavior analysis method, and monitoring device
JP2015011649A
Monitoring method, monitoring device, and monitoring program
JP2015176227A
Article management system, information processing device, and control method and control program therefor
WO2014136559A1
Information processing device, product rack management system, and product rack management method
WO2015140851A1
