System, method and program

Abstract

To provide a self-checkout system capable of appropriately managing products to be purchased by customers even when the products are returned to locations different from locations from which the products were taken.SOLUTION: Change detection means 810 detects changes in a display state of products on the basis of photographed images in which the products are photographed. Rearrangement detection means 820 detects that the products are return to locations different from locations from which the product were taken, on the basis of a change in a display state of the products detected by the change detection means 810 and persons included in the photographed images or persons whose traffic lines are detected in a store.SELECTED DRAWING: Figure 36

Description

[Technical Field]

[0001] The present invention relates to a system, method, and program for automatically managing products purchased by customers. [Background technology]

[0002] Convenience stores, supermarkets, and other retail outlets have introduced self-checkout registers (self-registers) where customers themselves operate the register terminal (hereafter referred to as "register"). In a typical self-checkout, customers make payments by having the register terminal read the product's identification code (e.g., barcode). However, because it is time-consuming and tedious for customers to determine the location of the identification code, various methods have been proposed to automatically identify the products that customers are purchasing.

[0003] For example, U.S. Patent No. 6,277,949 describes a system for tracking the removal or placement of items in an inventory location having a material handling facility. The system described in U.S. Patent No. 6,277,949 captures an image of a user's hand, detects when an item is removed from the inventory location, and adds the item to a user item list in response to the detection.

[0004] Furthermore, Patent Document 2 describes a POS (Point Of Sales) system for making payments for products. The POS system described in Patent Document 2 detects the flow of customers using captured images, identifies the customer who is about to make payment for a product, and recognizes the product to be paid for from among the products displayed in positions corresponding to the customer's flow.

[0005] Non-Patent Document 1 describes a method for subtracting a background image using an adaptive Gaussian mixture model. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Special Publication No. 2016-532932 [Patent Document 2] International Publication No. 2015 / 140853 [Non-patent literature]

[0007] [Non-Patent Document 1] Zoran Zivkovic, "Improved Adaptive Gaussian Mixture Model for Background Subtraction", Proceedings of the 17th International Conference on Pattern Recognition (ICPR'04), USA, IEEE Computer Society, August 2004, Volume2-Volume02, p.28-31 Summary of the Invention [Problem to be solved by the invention]

[0008] On the other hand, implementing the system described in Patent Document 1 requires an imaging device (camera) that can identify the item in each customer's hand, which increases the implementation cost. Also, with the system described in Patent Document 1, if an item is returned to a location different from where it was taken, it is difficult to identify the item, which makes it difficult to properly manage the user item list.

[0009] Furthermore, the system described in Patent Document 2 considers products associated with flow-line data as candidates for payment processing, but is unable to determine whether the customer returns the product before payment. Therefore, it is difficult to detect when a customer returns a product to a location different from where it was taken. It is therefore desirable to be able to appropriately manage products purchased by customers even when the product is returned to a location different from where it was taken.

[0010] Therefore, an object of the present invention is to provide a self-checkout system, a purchased product management method, and a purchased product management program that can properly manage products purchased by customers even if the products are returned to a location different from where they were taken. [Means for solving the problem]

[0011] The system according to the present invention comprises: a change detection means for detecting a changed area of ​​a product shelf on which a product is placed, based on a captured image of the product and a background image; a tracking means for extracting an area where the amount of movement of the changed area is equal to or greater than a predetermined threshold, as a person area; an association generation means for extracting, based on the person area, people who intersect with the changed area from people captured at a time before the image capture time at which the changed area was detected, and generating association information that associates the person captured at the time closest to the image capture time with the changed area; a classification means for classifying, based on a first image of interest which is an image in which the changed area is extracted from the captured image, and a second image of interest which is an image in which the changed area is extracted from the background image, a change from the second image of interest to a state in the first image of interest; and an output means for outputting the classified change content and association information together with the first image of interest and the second image of interest. The classified changes are changes due to the product being removed, changes in the shelf environment, changes due to the product being placed on the shelf, or changes due to a change in the product's appearance. Changes due to a change in the product's appearance include changes in appearance due to a different product being placed on the shelf and changes in appearance due to a change in the product's posture. It is characterized by:

[0012] In a method according to the present invention, a computer detects a changed area of ​​a product shelf on which a product is placed, based on a captured image in which the product is captured and a background image; the computer extracts an area where the amount of movement of the changed area is equal to or greater than a predetermined threshold, as a person area; the computer extracts, based on the person area, people who intersected with the changed area from people captured at a time before the image capture time at which the changed area was detected, and generates relevance information that associates people captured at a time closest to the captured image with the changed area; the computer classifies the change from the second image of interest to the state in the first image of interest, based on a first image of interest which is an image in which the changed area is extracted from the captured image, and a second image of interest which is an image in which the changed area is extracted from the background image; and the computer outputs the classified change content and relevance information together with the first image of interest and the second image of interest. The classified changes are changes due to the product being removed, changes in the shelf environment, changes due to the product being placed on the shelf, or changes due to changes in the product's appearance. Changes due to changes in the product's appearance include changes in appearance due to different products being placed on the shelf and changes in appearance due to changes in the product's posture.It is characterized by:

[0013] The program according to the present invention causes a computer to execute a change detection process for detecting a changed area of ​​a product shelf on which a product is placed, based on a captured image of the product and a background image; a tracking process for extracting an area where the amount of movement of the changed area is equal to or greater than a predetermined threshold, as a person area; an association generation process for extracting, based on the person area, people who intersected with the changed area from people captured at a time before the image capture time at which the changed area was detected, and generating association information that associates people captured at a time closest to the image capture time with the changed area; a classification process for classifying a change from a second image of interest, which is an image in which a changed area is extracted from the captured image, to a state in a first image of interest, based on the second image of interest, which is an image in which a changed area is extracted from the background image; and an output process for outputting the classified change content and association information together with the first image of interest and the second image of interest. The classified changes are changes due to the product being removed, changes in the shelf environment, changes due to the product being placed on the shelf, or changes due to a change in the product's appearance. Changes due to a change in the product's appearance include changes in appearance due to a different product being placed on the shelf and changes in appearance due to a change in the product's posture. It is characterized by: [Effects of the Invention]

[0014] According to the present invention, products purchased by customers can be appropriately managed even if the products are returned to a location different from where they were taken. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a block diagram showing a configuration example of a first embodiment of a self-checkout system according to the present invention. [Figure 2] FIG. 1 is an explanatory diagram showing an example of a usage scene of the self-checkout system 1. [Figure 3] 1 is a block diagram showing an example of the configuration of an image processing device 100. FIG. [Figure 4] 4 is a block diagram showing an example of the configuration of a first change detection section and a first storage section. FIG. [Figure 5] 10A and 10B are explanatory diagrams showing an example of the operation of a foreground region detection unit; [Figure 6] 10 is an explanatory diagram showing an example of a classification result output by an area change classification unit; FIG. [Figure 7]FIG. 10 is an explanatory diagram showing an example of relevance information between a product and a person. [Figure 8] FIG. 10 is an explanatory diagram showing an example in which information on the relevance between a product and a person is integrated. [Figure 9] FIG. 2 is a block diagram showing an example of the configuration of a shopping list management device. [Figure 10] FIG. 10 is an explanatory diagram showing an example of shelf allocation information. [Figure 11] 4 is a flowchart showing an example of the operation of the image processing device 100 according to the first embodiment. [Figure 12] 4 is a flowchart showing an example of the operation of the self-checkout system according to the first embodiment. [Figure 13] FIG. 2 is an explanatory diagram showing an example of the configuration of an image processing device 200. [Figure 14] FIG. 10 is an explanatory diagram showing an example of information on the relevance between a product and a person; [Figure 15] FIG. 10 is an explanatory diagram showing an example in which information on the relevance between a product and a person is integrated. [Figure 16] 10 is a flowchart showing an example of the operation of the image processing device 200 according to the second embodiment. [Figure 17] FIG. 10 is an explanatory diagram showing a configuration example of a third embodiment of the self-checkout system according to the present invention. [Figure 18] FIG. 2 is an explanatory diagram showing an example of the configuration of an image processing device 300. [Figure 19] FIG. 10 is an explanatory diagram showing an example of flow line data. [Figure 20] FIG. 10 is an explanatory diagram showing an example of information on the relevance between a product and a person; [Figure 21] FIG. 10 is an explanatory diagram showing an example in which the association between a product and a person is integrated. [Figure 22] 10 is a flowchart showing an example of the operation of the image processing device 300 according to the third embodiment. [Figure 23] FIG. 10 is a block diagram showing a modified example of the self-checkout system of the third embodiment. [Figure 24] FIG. 2 is a block diagram showing an example of the configuration of an image processing device 300a. [Figure 25] FIG. 10 is an explanatory diagram showing another example of flow line data. [Figure 26] FIG. 2 is an explanatory diagram showing an example of the configuration of an image processing device 400. [Figure 27] 10 is a flowchart showing an example of the operation of the image processing device 400 according to the fourth embodiment. [Figure 28] FIG. 1 is an explanatory diagram showing an example of the configuration of an image processing device 500. [Figure 29] 10 is a flowchart showing an example of the operation of the image processing device 500 according to the fifth embodiment. [Figure 30] FIG. 10 is an explanatory diagram showing an example of an operation for detecting a foreground region. [Figure 31] FIG. 10 is an explanatory diagram showing an example of an operation for detecting a foreground region. [Figure 32] FIG. 10 is an explanatory diagram showing an example of an operation for detecting a foreground region. [Figure 33] FIG. 2 is an explanatory diagram showing an example of the operation of the self-checkout system in the first specific example. [Figure 34] FIG. 10 is an explanatory diagram showing an example of the operation of a self-checkout system in a second specific example. [Figure 35] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device that realizes the components of each device. [Figure 36] 1 is a block diagram showing an overview of a self-checkout system according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0017] [First embodiment] In the first embodiment, a self-checkout system 1 that manages a list of products that a customer intends to purchase (hereinafter referred to as a shopping list) based on captured images will be described. In particular, the self-checkout system 1 of this embodiment detects that a product has been returned to a location different from where it was taken, and appropriately manages the shopping list.

[0018] As will be described later, the self-checkout system 1 of this embodiment detects changes in the product shelves 3 based on images captured by the imaging device 2, and detects areas of people and objects included in the captured images. The self-checkout system 1 then associates the changes in the product shelves 3 with the extracted people, and integrates the associated information based on the people. In this way, the self-checkout system 1 is configured to associate changes in the product shelves 3 based on the extracted people. This allows the self-checkout system 1 to detect when a product has been returned to a location different from where it was taken, such as when a customer places a product they picked up on another product shelf 3.

[0019] 1 is a block diagram showing an example of the configuration of a first embodiment of a self-checkout system according to the present invention. The self-checkout system 1 of this embodiment includes a terminal 10 carried by a customer, an image processing device 100, an imaging device 2, a shopping list management device 20, an output device 30, a payment device 40, and a person identification device 50. The terminal 10, the image processing device 100, the imaging device 2, the shopping list management device 20, the output device 30, the payment device 40, and the person identification device 50 are connected to each other, for example, via a network N.

[0020] In the present embodiment, the image processing device 100, the imaging device 2, and the shopping list management device 20 are described as being separate components. However, each device may be built into the other device. For example, the imaging device 2 may have the functions of the image processing device 100 described below, and the image processing device 100 may have the functions of the imaging device 2. Furthermore, for example, the image processing device 100 and the shopping list management device 20 may be implemented by the same hardware, which receives images captured by the imaging device 2 and performs the processes described below.

[0021] 1 illustrates an example in which there is one imaging device 2. However, the number of imaging devices 2 included in the self-checkout system 1 is not limited to one, and may be two or more.

[0022] Fig. 2 is an explanatory diagram showing an example of a usage scene of the self-checkout system 1. Referring to Fig. 2, in the self-checkout system 1, the imaging device 2 captures an image of a product shelf 3 in a store. Then, the imaging device 2 transmits a video signal indicating the captured image to the image processing device 100.

[0023] The imaging device 2 is, for example, a surveillance camera installed in a store or the like. The imaging device 2 is installed, for example, in a predetermined position in a store or the like where it can capture images of the product shelves 3. A camera ID or the like for identifying the imaging device 2 is also assigned to the imaging device 2 in advance. The imaging device 2 acquires a captured image. At this time, the imaging device 2 associates the image capture time, which is the time when the captured image was acquired, with the acquired captured image, for example, by referring to a clock or the like that the imaging device 2 owns. In this way, the imaging device 2 acquires a captured image showing the state of the product shelves 3 or the like.

[0024] The video captured by the imaging device 2 may be a moving image or a series of still images. In this embodiment, the captured image acquired by the imaging device 2 is a color image (hereinafter referred to as an RGB (Red Green Blue) image). The captured image acquired by the imaging device 2 may be, for example, an image in a color space other than an RGB image.

[0025] As described above, the imaging device 2 transmits a video signal indicating the acquired captured image to the image processing device 100. The imaging device 2 may store the captured image in a storage device inside the imaging device 2 or in a storage device different from the image processing device 100.

[0026] The image processing device 100 is an information processing device that detects changes in the display state of products by analyzing captured images of the products. In this embodiment, the image processing device 100 detects that a product has been returned to a location different from the location where it was taken.

[0027] 3 is an explanatory diagram showing an example of the configuration of the image processing device 100. The image processing device 100 of this embodiment includes a first acquisition unit 110, a first change detection unit 120, a first storage unit 130, a first relevance generation unit 140, a relevance integration unit 150, and a display detection unit 160. Specifically, the image processing device 100 includes, for example, a calculation unit and a storage device (not shown). The image processing device 100 realizes each of the above processing units by, for example, causing the calculation unit to execute a program stored in the storage device.

[0028] Note that the image processing device 100 illustrated in Fig. 3 shows a configuration unique to the present disclosure. The image processing device 100 may have components not shown in Fig. 3. This also applies to the second and subsequent embodiments.

[0029] The first acquisition unit 110 acquires a video signal indicating a captured image obtained by the imaging device 2 capturing an image of the product shelf 3. The first acquisition unit 110 may receive a video signal transmitted from the imaging device 2. The first acquisition unit 110 may acquire a video signal converted based on a captured image stored inside the imaging device 2 or in a storage device different from the imaging device 2 and the image processing device 100.

[0030] As described above, the image processing device 100 may be configured to be built into the imaging device 2. In such a configuration, the first acquisition unit 110 may be configured to acquire the captured image itself.

[0031] The first acquisition unit 110 converts the acquired video signal into an RGB image constituting the video signal. Then, the first acquisition unit 110 supplies the converted RGB image to the first change detection unit 120. Note that the RGB image obtained by converting the video signal by the first acquisition unit 110 represents a captured image of the product shelf 3 captured by the imaging device 2, and is therefore also simply referred to as a captured image.

[0032] 4 is a block diagram showing an example of the configuration of first change detection section 120 and first storage section 130. First change detection section 120 and first storage section 130 will be described below with reference to FIG.

[0033] The first storage unit 130 is a storage device such as a hard disk or a memory. The first storage unit 130 stores background information 131, a shelf change model 132, foreground information 133, and person information 134. The first storage unit 130 may be realized by a storage device different from the image processing device 100, or may be built into the first change detection unit 120. The background information 131, the shelf change model 132, the foreground information 133, and the person information 134 may be stored in the same storage device or in separate storage devices.

[0034] The background information 131 is a reference image that the first change detection unit 120 uses to compare with the captured image. The background information 131 is also called a background image. The background information 131 is preferably, for example, the same type of image as the captured image. In this embodiment, as described above, the captured image is an RGB image. Therefore, it is preferable that the background information 131 is also an RGB image. Note that the background information 131 may be the captured image initially supplied from the first acquisition unit 110 to the first change detection unit 120, or may be an image provided in advance.

[0035] As will be described later, the background information 131 is updatable information. Details of the process for updating the background information 131 will be described later.

[0036] The shelf change model 132 is a model of a change in the product shelves 3 that has been learned in advance. The shelf change model 132 is, for example, stored in advance in the first storage unit 130. The shelf change model 132 may be obtained by learning using, for example, machine learning such as a commonly known convolutional neural network.

[0037] The shelf change model 132 represents, for example, a "change due to the absence of a product on the shelf 3" or a "change due to the presence of a new product on the shelf 3," which is learned using images of products on the shelf 3 and images of products not on the shelf 3. The shelf change model 132 also represents a "change due to a change in the appearance of products displayed on the shelf 3," which is learned using images of multiple products and multiple images of each product with a changed shape. The shelf change model 132 also represents a "change due to the presence of a person in front of the shelf 3," a "change due to the presence of a shopping cart in front of the shelf 3," etc., which are learned using images captured with no object in front of the shelf 3 and images captured with an object such as a person in front of the shelf 3. The shelf change model 132 may also represent, for example, a "change due to a change in lighting," which is learned using images captured under various environments.

[0038] The training data for the shelf change model 132 may be, for example, a six-channel image obtained by combining two RGB images before and after the change, or a two-channel image obtained by combining any one of the R, G, or B components of the two RGB images before and after the change. The training data may also be, for example, a four-channel image obtained by combining any two of the R, G, or B components of the two RGB images before and after the change, or a two-channel image obtained by converting the two RGB images before and after the change into grayscale images and then combining them. The training data may also be an image obtained by converting the RGB images before and after the change into another color space, such as the HSV (Hue Saturation Value) color space, and combining one or more channels in the color space after the conversion to the other color space.

[0039] Furthermore, the learning data of the shelf change model 132 may be generated from a color image such as an RGB image, or may be generated using both a color image and a distance image.

[0040] Foreground information 133 is information stored by foreground region detection unit 121. Foreground information 133 includes information indicating a foreground region (changing region) that is a region in the RGB image that differs from the background image, detected as a result of foreground region detection unit 121 comparing background information 131, which is a background image, with an RGB image, which is a captured image. Specifically, foreground information 133 is, for example, a binary image associated with the capture time of the captured image. Details of the process of storing foreground information 133 will be described later.

[0041] The person information 134 is information stored by the foreground region tracking unit 123. The person information 134 is generated, for example, by associating the person region extracted by the foreground region tracking unit 123 with the ID of the imaging device (camera ID), the person ID, the position on the captured image, the capture time of the captured image, and the like. As will be described later, the person region is, for example, an RGB image. In other words, the person information 134 may include information indicating, for example, the color, area, shape, and aspect ratio of the circumscribing rectangle of the stored person region. The process of storing the person information 134 will also be described in detail later.

[0042] The first change detection unit 120 detects a changed area relating to the product shelf 3.

[0043] For example, if a product displayed on a product shelf 3 included in a captured image is not included in an image (e.g., a background image) acquired before the captured image, the first change detection unit 120 detects the area of ​​the product. Also, for example, if a product displayed on a product shelf 3 included in a background image is not included in the captured image, the first change detection unit 120 detects the area of ​​the product. Also, for example, if the appearance of the product displayed on a product shelf 3 included in the captured image differs from that of the product included in the background image, the first change detection unit 120 detects the area of ​​the product. In this way, the first change detection unit 120 detects changes in the display state of products, such as a decrease in the number of products (disappearance), an increase in the number of products (new appearance), or a change in the appearance of the product, based on the captured image.

[0044] In addition, for example, if the captured image was captured when a person or object was present between the product shelf 3 and the imaging device 2, the first change detection unit 120 detects the area of ​​the person or object included in the captured image of the product shelf 3.

[0045] As described above, the first change detection unit 120 detects change areas related to the product shelf 3, such as change areas inside the product shelf 3 where the display state of the products has changed, and change areas in the captured image due to a person or object that exists between the product shelf 3 and the imaging device 2.

[0046] As illustrated in FIG. 4, the first change detection unit 120 includes a foreground region detection unit 121, a background information update unit 122, a foreground region tracking unit 123, a first extraction unit 124, a second extraction unit 125, and a region change classification unit 126.

[0047] The foreground region detection unit 121 receives the captured image supplied from the first acquisition unit 110. The foreground region detection unit 121 also acquires background information 131 corresponding to the captured image from the first storage unit 130. As described above, the background information 131 is an RGB image. The foreground region detection unit 121 compares the captured image, which are two RGB images, with the background information 131. The foreground region detection unit 121 then detects a region that has changed between the two compared RGB images as a changed region. Because the foreground region detection unit 121 compares the background information 131, which is a background image, with the RGB image, which is the captured image, it can be said that the foreground region detection unit 121 performs processing to detect a foreground region, which is a region different from the background image.

[0048] In this embodiment, the method by which foreground region detection unit 121 detects a changed region is not particularly limited. Foreground region detection unit 121 may detect a changed region using existing technology. For example, foreground region detection unit 121 may detect a changed region using a background subtraction method. Foreground region detection unit 121 may generate a binary image in which the pixel values ​​of the detected changed region are represented as 255 and the other pixel values ​​are represented as 0.

[0049] An example of the operation of the foreground region detection unit 121 will now be described in more detail with reference to Fig. 5. Fig. 5 is an explanatory diagram showing an example of the operation of the foreground region detection unit 121. Fig. 5(a) shows an example of a captured image, and Fig. 5(b) shows an example of background information 131 corresponding to the captured image stored in the first storage unit 130. Fig. 5(c) shows an example of a binary image that is a result of detection of a changed region.

[0050] 5(a) and 5(b), there are differences between the captured image and the background information 131 in three areas: product G1, product G2, and product G3. For example, in the example shown in FIGS. 5(a) and 5(b), product G1 is not included in the background information 131 but is included in the captured image. Product G3 is included in the background information 131 but not in the captured image. Furthermore, other products are displayed in the background information 131 at the position of product G2 included in the captured image. Therefore, the foreground region detection unit 121 detects the region of product G2 as a region where a change has occurred. In such a case, the foreground region detection unit 121 generates a binary image in which the areas corresponding to product G1, product G2, and product G3 are represented in white and the other areas are represented in black, as shown in FIG. 5(c), for example.

[0051] In the following description, the changed region refers to each white portion shown in Fig. 5(c). In other words, a changed region is, for example, a pixel with a pixel value of 255, and a set of pixels adjacent to that pixel with a pixel value of 255. In the example shown in Fig. 5(c), foreground region detection unit 121 detects three changed regions.

[0052] As described above, the foreground region detection unit 121 generates, for example, a binary image of the same size as the captured image. The foreground region detection unit 121 also associates the binary image, which is the detection result, with the capture time of the captured image used to generate the binary image. The foreground region detection unit 121 may associate, with the binary image, information indicating the captured image used to generate the binary image and information indicating background information 131. The foreground region detection unit 121 then supplies the detection result associated with the capture time of the captured image and the like to the background information update unit 122 and the foreground region tracking unit 123. The foreground region detection unit 121 also stores the detection result associated with the capture time of the captured image in the first storage unit 130 as foreground information 133.

[0053] The detection result by foreground region detection section 121 may include information indicating the detected changed region. For example, foreground region detection section 121 may associate information indicating the position and size of the detected changed region (region with a pixel value of 255) with information indicating the captured image used to detect the changed region and information indicating the background image, and output the result as the detection result. In this way, the detection result output by foreground region detection section 121 may be in any format.

[0054] Furthermore, the foreground region detection unit 121 may associate the imaging time of the captured image with the binary image that is the detection result, as well as with color information contained in the changed region extracted from the captured image. The foreground region detection unit 121 may associate the image of the changed region with the detection result, instead of the color information of the changed region. In this way, the foreground region detection unit 121 may associate information other than the imaging time with the detection result.

[0055] 5(c), the binary image generated by foreground region detection unit 121 may include multiple changed regions. In such a case, foreground region detection unit 121 may generate a binary image for each changed region. A method by which foreground region detection unit 121 generates a binary image for each changed region will be described later as a modified example of foreground region detection unit 121.

[0056] Foreground region tracking unit 123 tracks the changed region detected by foreground region detection unit 121 across multiple captured images. Depending on the tracking result, foreground region tracking unit 123 supplies a binary image to first extraction unit 124 and second extraction unit 125, or extracts a person region. In addition, foreground region tracking unit 123 supplies an update signal indicating an update of background information 131 to background information update unit 122.

[0057] Foreground region tracking unit 123 receives, for example, the detection result (binary image) supplied from foreground region detection unit 121. Furthermore, foreground region tracking unit 123 acquires foreground information 133, which is a binary image associated with the binary image that is the detection result and which is generated from a captured image captured before the captured image associated with the binary image, from first storage unit 130. Then, foreground region tracking unit 123 tracks the changed regions by performing a process of associating the changed regions represented by each binary image.

[0058] The foreground region tracking unit 123 can track the changed region using various methods. For example, the foreground region tracking unit 123 calculates the similarity based on at least one of the area, shape, and aspect ratio of the circumscribing rectangle of the changed region represented by the binary image supplied from the foreground region detection unit 121 and the foreground information 133 acquired from the first storage unit 130. The foreground region tracking unit 123 then tracks the changed region by matching the changed regions with the highest calculated similarity. Furthermore, in a configuration in which color information is associated with the detection result, the foreground region tracking unit 123 may perform tracking using the color information. The foreground region tracking unit 123 may also perform tracking based on an image of the changed region associated with the detection result.

[0059] The foreground region tracking unit 123 checks whether the tracking result is for a predetermined time or more, or whether the amount of movement of the changed region is equal to or greater than a predetermined threshold. Note that the predetermined time and the predetermined threshold used by the foreground region tracking unit 123 for the check are arbitrary values.

[0060] If the amount of movement of the changed region is less than a predetermined threshold and the tracking result is for a predetermined time or longer, foreground region tracking unit 123 supplies a binary image, which is the detection result supplied from foreground region detection unit 121, to first extraction unit 124 and second extraction unit 125. At this time, foreground region tracking unit 123, for example, adds information indicating the captured image used to generate the binary image and information indicating background information 131 to the binary image, and supplies the binary image to first extraction unit 124 and second extraction unit 125. For example, foreground region tracking unit 123 may supply the corresponding captured image and background information 131 together with the binary image to first extraction unit 124 and second extraction unit 125. Furthermore, if the binary image includes multiple changed regions and any of the changed regions has not been tracked for a predetermined time or longer, foreground region tracking unit 123 may supply the binary image to first extraction unit 124 and second extraction unit 125 together with information indicating the changed region that has been tracked for a predetermined time or longer.

[0061] If a binary image contains multiple changed regions, foreground region tracking unit 123 may generate multiple binary images so that each binary image contains one changed region. For example, binary images containing only changed regions that have been tracked for a predetermined time or more may be supplied to first extraction unit 124 and second extraction unit 125, and binary images containing changed regions that have not been tracked for a predetermined time or more may be discarded. Foreground region tracking unit 123 may receive a binary image for each changed region from foreground region detection unit 121 as the detection result.

[0062] Furthermore, if the amount of movement of the changed area is equal to or greater than a predetermined threshold, the foreground area tracking unit 123 determines that the object included in the changed area is a moving object. When the foreground area tracking unit 123 determines that the object included in the changed area is a moving object in this way, it refrains from supplying the changed area to the first extraction unit 124 and the second extraction unit 125. This allows the image processing device 100 to eliminate changes related to the product shelf 3 that are unrelated to the increase or decrease in the number of products, such as "changes due to the presence of a person in front of the product shelf 3." This makes it possible to monitor the display state of products more accurately.

[0063] The foreground region tracking unit 123 may associate the determination result that an object included in a changed region is a moving object with the changed region and supply it to the first extraction unit 124 and the second extraction unit 125. When the determination result is associated with a changed region, the region change classification unit 126 may classify the change related to the product shelf 3 in this changed region into a type related to a change other than the products displayed on the product shelf 3. For example, the region change classification unit 126 may classify the change related to the product shelf 3 in the changed region into a type related to a change other than the products, such as "a change due to the presence of a person in front of the product shelf 3" or "a change due to the presence of a shopping cart in front of the product shelf 3."

[0064] The foreground region tracking unit 123 extracts a changing region determined to be a moving object from the captured image as a person region. Specifically, the foreground region tracking unit 123 uses the captured image and a binary image of the same size as the captured image to extract, as a first image of interest, an image of a region on the captured image corresponding to a region in the binary image where the pixel value is 255. As described above, the captured image is an RGB image. Therefore, the extracted person region is also an RGB image.

[0065] For each changed area determined to be a moving object, foreground area tracking unit 123 may extract a person area of ​​the same shape as the changed area, or may extract as the person area an area surrounded by a frame of the same shape as a predetermined frame circumscribing the changed area. The shape of the frame circumscribing the changed area may be any shape, such as a rectangle or an ellipse. Furthermore, foreground area tracking unit 123 may extract as the person area an area surrounded by a frame that is a predetermined size larger than the frame circumscribing the changed area.

[0066] Next, foreground region tracking unit 123 associates the extracted person region with the ID (camera ID) of image capture device 2, a person ID assigned to each extracted person region, a position on the captured image, and the capture time of the captured image. Then, foreground region tracking unit 123 stores the associated information in the first storage unit as person information 134. Note that the position on the captured image may be expressed, for example, by the coordinate values ​​of the four corners of a circumscribing rectangle of the changing region determined to be a moving object, or may be expressed by the coordinate values ​​of at least one of the four corners and the width and height of the circumscribing rectangle.

[0067] Furthermore, the foreground region tracking unit 123 supplies an update signal indicating that the background information 131 should be updated to the background information update unit 122 .

[0068] For example, after tracking a changed region, when foreground region tracking unit 123 supplies a detection result indicating the changed region to first extraction unit 124 and second extraction unit 125, foreground region tracking unit 123 supplies an update signal with a value of 1 to background information update unit 122 together with information indicating the changed region. An update signal with a value of 1 indicates that the image of the portion corresponding to the changed region in background information 131 is to be updated. Furthermore, when foreground region tracking unit 123 does not supply a detection result to first extraction unit 124 and second extraction unit 125, foreground region tracking unit 123 may supply an update signal with a value of 0 to background information update unit 122 together with information indicating the changed region. An update signal with a value of 0 indicates that the image of the portion corresponding to the changed region in background information 131 is not to be updated. Note that detection results are not output to first extraction unit 124 and second extraction unit 125 when, for example, the tracking result is less than a predetermined time, or when the amount of movement of the changed region is equal to or greater than a predetermined threshold.

[0069] The foreground region tracking unit 123 may supply an update signal indicating an update of the background information 131 to the background information update unit 122 at timings other than those exemplified above. For example, the foreground region tracking unit 123 may output an update signal with a value of 1 to update the background of the store shelf 3 when it is determined that a product included in the changed region is likely to have been purchased or replenished based on product purchase information, stocking information, store clerk work information, etc. transmitted from an external device (not shown) of the image processing device 100. The foreground region tracking unit 123 may supply an update signal indicating an update of the background information 131 to the background information update unit 122 based on the tracking time, etc. included in the tracking result.

[0070] The background information update unit 122 updates the background information 131 based on the captured image supplied from the first acquisition unit 110, the detection result supplied from the foreground region detection unit 121, the RGB image that is the background information 131 stored in the first storage unit 130, and the update signal supplied from the foreground region tracking unit 123. The method by which the background information update unit 122 updates the background information 131 is not particularly limited. The background information update unit 122 may update the background information 131 using, for example, a method similar to that described in Non-Patent Document 1.

[0071] Note that background information update unit 122 does not have to update the image of a portion of the RGB image indicated by background information 131 that corresponds to a changed area indicated by the detection result supplied from foreground area detection unit 121. For example, when background information update unit 122 receives an update signal with the above-mentioned value of 0 from foreground area tracking unit 123, it does not update the background information of the area corresponding to the changed area.

[0072] As described above, when foreground region tracking unit 123 does not output a detection result to first extraction unit 124 and second extraction unit 125, it supplies an update signal with a value of 0 to background information update unit 122. In this way, when the tracking result satisfies the first predetermined condition, background information update unit 122 receives an update signal with a value of 0 and does not update the background information of the region corresponding to the changed region. In other words, when the tracking result satisfies the first predetermined condition, background information update unit 122 updates the region of background information 131 other than the region corresponding to the changed region. This makes it easier for foreground region detection unit 121 to detect, as a changed region, a region that was not updated in the captured image next acquired by first acquisition unit 110.

[0073] Furthermore, for example, when the value of the update signal supplied from the foreground region tracking unit 123 is 1, the background information update unit 122 updates the image of a portion of the RGB image represented by the background information 131 that corresponds to the changed region indicated by the detection result supplied from the foreground region detection unit 121. As described above, when the tracking result continues for a predetermined time or longer, the foreground region tracking unit 123 supplies the detection result indicating the tracked changed region to the first extraction unit 124 and the second extraction unit 125, and supplies an update signal with a value of 1 to the background information update unit 122. In other words, when the tracking result satisfies the second predetermined condition that the tracking result is the result of tracking for a predetermined time or longer, the background information update unit 122 receives the update signal with a value of 1 from the foreground region tracking unit 123 and updates the image of the portion of the background information 131 that corresponds to the changed region. In this way, the background information update unit 122 can bring the background information 131 stored in the first storage unit 130 closer to the captured image acquired by the first acquisition unit 110 at that time. Therefore, the image processing device 100 can prevent the foreground region detection section 121 from detecting, as a changed region, a region on the captured image that the first acquisition section 110 acquires next that corresponds to the changed region.

[0074] The first extraction unit 124 receives a binary image that is a detection result from the foreground region tracking unit 123. The first extraction unit 124 also acquires the captured image used to generate the binary image from the first acquisition unit 110. Note that the first extraction unit 124 may receive the captured image together with the binary image from the foreground region tracking unit 123.

[0075] The first extraction unit 124 extracts images of the changed region from the captured image. Specifically, the first extraction unit 124 uses the captured image and a binary image of the same size as the captured image to extract, as the first image of interest, an image of a region on the captured image corresponding to a region in the binary image where the pixel value is 255. For example, when the binary image is as shown in FIG. 5(c), the first extraction unit 124 extracts three first images of interest from the captured image. As described above, since the captured image is an RGB image, the extracted first images of interest are also RGB images.

[0076] For each changed area, the first extraction unit 124 may extract a first image of interest of an area having the same shape as the changed area, or may extract as the first image of interest an image of an area surrounded by a frame of the same shape as a predetermined frame circumscribing the changed area. The shape of the frame circumscribing the changed area may be any shape, such as a rectangle or an ellipse. The first extraction unit 124 may also extract as the first image of interest an image of an area surrounded by a frame that is a predetermined size larger than the frame circumscribing the changed area.

[0077] The first extraction unit 124 supplies the extracted first image of interest to the region change classification unit 126. The region of the first image of interest extracted by the first extraction unit 124 on the captured image is also referred to as the first region of interest. The first extraction unit 124 acquires position information of the first region of interest, associates the position information with the image capture time, and supplies them to the first correlation generation unit 140. The position information of the first region of interest may be, for example, the coordinate values ​​of the four corners of a circumscribing rectangle of the first region of interest, or may be expressed as the coordinate values ​​of at least one of the four corners and the width and height of the circumscribing rectangle. If the circumscribing rectangle is circular, the position information of the first region of interest may be, for example, the center coordinates of the circle and the radius of the circle. If the circumscribing rectangle is elliptical, the position information of the first region of interest may be, for example, the center coordinates of the ellipse and the major and minor axes of the ellipse.

[0078] Second extraction unit 125 receives a binary image that is the detection result from foreground region tracking unit 123. Second extraction unit 125 also acquires background information 131 used to generate the binary image from first storage unit 130. Note that second extraction unit 125 may receive background information 131 from foreground region tracking unit 123 together with the binary image.

[0079] The second extraction unit 125 extracts an image of a changed region from the background information 131. Specifically, the second extraction unit 125 uses the background information 131, which is a background image, and a binary image to extract, as a second image of interest, an image of a region on the background information 131 that corresponds to a region in the binary image where the pixel value is 255. The method for extracting the second image of interest is the same as the method for extracting the first image of interest. The second extraction unit 125 supplies the extracted second image of interest to the region change classification unit 126. Note that the region on the background information 131 of the second image of interest extracted by the second extraction unit 125 is also referred to as a second region of interest.

[0080] The area change classification unit 126 classifies the changes related to the product shelves 3 in the changed area, and supplies the classification results to the first relevance generation unit 140. The area change classification unit 126 classifies the change from the image state of the area corresponding to the detected changed area on the background image to the image state of the area corresponding to the changed area on the captured image, based on the first attention area and second attention area supplied from the first extraction unit 124 and the second extraction unit 125, and the shelf change model 132 stored in the first storage unit 130.

[0081] The state of the image may be, for example, whether or not a product is included in the image, whether or not a customer is included in the image, whether or not a shopping cart is included in the image, or whether or not a shopping cart is included in the image. Based on the shelf change model 132, the area change classification unit 126 classifies changes related to the product shelves 3 in the change area into types of changes, such as "a change due to products no longer being included in the product shelves 3," "a change due to new products being included in the product shelves 3," "a change due to a change in the appearance of products displayed on the product shelves 3," "a change due to the presence of a person in front of the product shelves 3," "a change due to the presence of a shopping cart in front of the product shelves 3," and "a change due to a change in lighting." Note that the types into which the area change classification unit 126 classifies changes in the state in the change area are merely examples and are not limited to these. Furthermore, for example, "a change due to a change in the appearance of products displayed on the product shelves 3" may be further classified into "a change in appearance due to the products being different" or "a change in the appearance of the products" or the like.

[0082] More specifically, the area change classification unit 126 receives a first image of interest from the first extraction unit 124. The area change classification unit 126 also receives a second image of interest from the second extraction unit 125. Then, based on the shelf change model 132 stored in the first storage unit 130, the area change classification unit 126 classifies the change from the state of the second image of interest to the state of the first image of interest corresponding to the second image of interest into, for example, the types described above. In other words, the area change classification unit 126 classifies the change from the state of the second image of interest to the state of the first image of interest based on the result of comparison with the shelf change model 132.

[0083] 6 is an explanatory diagram showing an example of a classification result output by the area change classification unit 126. The area change classification unit 126 outputs, for example, a classification result 90 shown in FIG.

[0084] As illustrated in Fig. 6, the classification result 90 includes, for example, a second image of interest 91, a first image of interest 92, and a type of change 93. Note that the classification result 90 illustrated in Fig. 6 is an example, and the classification result 90 may include information other than the information illustrated in Fig. 6. The classification result 90 may include, for example, information about the captured image (such as an identifier and a capture time) and information indicating the position of the first image of interest 92 in the captured image.

[0085] The area change classification unit 126 may classify the change relating to the product shelf 3 into one of the above types, for example, using a machine learning method (such as a convolutional neural network) that created the shelf change model 132.

[0086] An example of the configuration of first change detection section 120 has been described above.

[0087] The first relevance generation unit 140 receives the classification result of the changed area and the position information of the changed area from the first change detection unit 120. The first relevance generation unit 140 also acquires person information 134 from the first storage unit 130. The first relevance generation unit 140 then generates product-person relevance information indicating the relationship between the product (change in the display state of the product) corresponding to the changed area and the person, based on the image capture time of the changed area corresponding to the position information of the changed area and the image capture time of the person linked to the person information 134. The first relevance generation unit 140 then supplies the generated product-person relevance information to the relevance integration unit 150.

[0088] Specifically, the first association generation unit 140 extracts people who intersect with a changed area from people whose images were captured at a time before the image capture time at which the changed area was detected, and then associates the person whose image was captured at the time closest to the image capture time of the changed area among the extracted people with the changed area.

[0089] FIG. 7 is an explanatory diagram showing an example of the relationship information between products and people generated by the first relationship generation unit 140. FIG. 7 illustrates a camera ID indicating the ID of the imaging device, a person ID indicating the person captured by the imaging device, position information of a changed area on the product shelf, and a classification result of the change. In FIG. 7, the position information of the changed area is expressed by the coordinate value of one corner of a circumscribing rectangle of the changed area, and the width and height of the circumscribing rectangle. In addition, the type of change is shown as "a change due to a product no longer being included on the product shelf 3" as "a decrease in product" and "a change due to a new product being included on the product shelf 3" as "an increase in product."

[0090] Note that the first relevance generation unit 140 may, for example, associate the generated person-product relevance information with the person information 134 stored in the first storage unit 130 and supply the information to the relevance integrating unit 150. The first relevance generation unit 140 may also add information about the captured image (such as an identifier and a capture time) to the relevance information.

[0091] The relation integration unit 150 receives relation information between products and people from the first relation generation unit 140. If the received relation information includes relation information about the same person, the relation integration unit 150 integrates the information into one. Thereafter, the relation integration unit 150 supplies the integrated relation information to the display detection unit 160.

[0092] The relevance integration unit 150 calculates the similarity based on, for example, at least one of the color, area, shape, and aspect ratio of the circumscribing rectangle of the person area stored in the person information 134 in the first storage unit 130. Then, the relevance integration unit 150 determines that the person areas with the highest calculated similarity belong to the same person. As described above, the relevance integration unit 150 integrates the relevance information of the person areas determined to belong to the same person.

[0093] Fig. 8 is an explanatory diagram showing an example of integrating the relationship information between the products and the people shown in Fig. 7. In the example shown in Fig. 8, the relationship information between person ID=1 and person ID=4 shown in Fig. 7 is integrated into one. The relationship information between person ID=2 and person ID=3 shown in Fig. 7 is also integrated into one. In other words, Fig. 8 shows an example where the people shown in Fig. 7 with person ID=1 and person ID=4 are the same person, and the people shown in Fig. 7 with person ID=2 and person ID=3 are the same person.

[0094] In the example shown in Fig. 8, as an example of integration, when integrating the relationship information, two person IDs are compared and the person ID with the smaller value is used as the person ID of the integrated relationship information, but the person ID with the larger value may also be used. Also, for example, after integrating the relationship information, the person ID may be reassigned. Also, the person ID used when the person identification device 50 described later identifies the person may be used, or identification information that identifies the terminal 10 carried by the person may be used.

[0095] The display detection unit 160 receives the integrated relevance information from the relevance integration unit 150, and detects that a product has been returned to a location different from where it was taken, based on the relevance information integrated by the relevance integration unit 150. The display detection unit 160 detects that a product has been returned to a location different from where it was taken, for example, by comparing the location where the product was acquired with the location where the product was returned.

[0096] The operation of the display detection unit 160 will be described in detail below with reference to FIG. 8. The display detection unit 160 compares the position information and the type of change for each person ID. In the example shown in FIG. 7, a person with person ID = 1 retrieves an item from a location (10, 0) on a product shelf captured by an imaging device with camera ID = 1 and returns the item to a location (250, 300). The retrieved and returned items have circumscribed rectangles with a width and height of (30, 50), and are therefore determined to be the same item. From this, the display detection unit 160 detects that the person with person ID = 1 has returned the item to a different location on the same shelf as the one from which the item was retrieved. Note that the person with person ID = 2 returns the same item to the same location from which the item was retrieved. Therefore, the display detection unit 160 does not detect this behavior as a product being returned to a location different from where it was taken.

[0097] In this way, the display detection unit 160 detects that a product has been returned to a location different from where it was taken, for example, by detecting that the same person has returned a product to a different location. Note that the display detection unit 160 also detects that a product has been returned to a location different from where it was taken, for example, when the appearance of the product has changed even though it has been returned to the same location. Note that the display detection unit 160 can thus be called a rearrangement detection means, as it detects that a product has been returned to a location different from where it was taken and has been rearranged.

[0098] The above is an example of each configuration of the image processing device 100.

[0099] The terminal 10 is a device carried by a customer, and is realized by, for example, a mobile terminal or a tablet terminal. The terminal 10 stores information for identifying the customer, and is used by a person identification device 50 (described later) to associate the terminal 10 with the customer (person). The terminal 10 may display the information for identifying the customer as a label (such as a barcode), or may transmit the information via short-range wireless communication, for example.

[0100] Furthermore, the terminal 10 notifies the customer of various information in a manner that can be perceived by humans (such as by display, vibration, light, or sound) in response to a notification from the notification unit 23, which will be described later. The specific content of the notification by the notification unit 23 will be described later.

[0101] The person identification device 50 is a device that identifies people. In this embodiment, the person identification device 50 does not need to identify the characteristics of the person themselves (for example, gender, age, height, etc.), but only needs to be able to identify the person so that they can be distinguished from other people. For example, in the example shown in FIG. 8, it is sufficient to be able to identify the person with person ID=1, the person with ID=2, and the person with ID=5 as being different people. The person identification device 50 is installed, for example, at the entrance of a store to identify people.

[0102] In this embodiment, the person identification device 50 identifies a person based on a captured image. The person identification device 50 may identify a person using any method. For example, the person identification device 50 may acquire information (such as the color, area, shape, and aspect ratio of the circumscribing rectangle of the person area) from an image of a person entering a store, which the above-described relevance integration unit 150 uses to calculate the similarity of the person. The person identification device 50 may then identify the person using the acquired information.

[0103] Furthermore, the person identification device 50 may associate the identified person with the device (terminal 10) carried by that person. Specifically, the person identification device 50 may capture an image of the person when a sensor (not shown) installed at the entrance of the store detects the terminal 10, and associate the person identified in the captured image with the identification information of the terminal 10. In this case, the person identification device 50 is realized by a device including an imaging device and a sensor. Note that the imaging device and the sensor may be realized by separate hardware. Furthermore, the device carried by the person is not limited to hardware such as a mobile phone, but may be a medium such as an IC card, for example.

[0104] Specifically, a customer may launch an application program installed on the terminal 10 to display the customer's identification information, and when the customer has the identification information identified by the person identification device 50, the person identification device 50 may associate the person with the terminal 10.

[0105] 9 is a block diagram showing an example of the configuration of shopping list management device 20. Shopping list management device 20 manages shopping lists for each person. Shopping list management device 20 includes a shopping list generation unit 21, a shopping list update unit 22, a notification unit 23, and a shopping list storage unit 24.

[0106] The shopping list storage unit 24 stores a shopping list for each person. For example, the shopping list storage unit 24 may store the shopping list in association with the person ID described above. The shopping list storage unit 24 may also store the shopping list in association with an identifier assigned by the person identification device 50 described above when identifying the person. Furthermore, when a person is associated with a terminal 10, the shopping list storage unit 24 may store the person and the terminal 10 in association with the shopping list.

[0107] The shopping list generation unit 21 generates a shopping list and registers it in the shopping list storage unit 24. For example, when the person identification device 50 identifies a person, the shopping list generation unit 21 may generate a shopping list corresponding to that person. Alternatively, for example, when the person identification device 50 associates the person with the terminal 10, the shopping list generation unit 21 may generate a shopping list corresponding to that person. In this case, since the shopping list is associated with the terminal 10 carried by the person, the notification unit 23, which will be described later, can notify the terminal 10 of any changes that occur in the shopping list.

[0108] Furthermore, if the self-checkout system 1 does not include the person identification device 50 (i.e., if the terminal 10 is not associated with a shopping list), the shopping list generation unit 21 may generate a shopping list when the first association generation unit 140 generates association information between people and products. In this case, the shopping list generation unit 21 may also integrate the shopping list when the association integration unit 150 integrates the association information. In this way, the shopping list is managed in association with each person.

[0109] The shopping list update unit 22 updates the contents of the shopping list stored in the shopping list storage unit 24 based on changes in the display status of products. The shopping list update unit 22 may transmit the shopping list to the terminal 10 each time the contents of the shopping list are updated. In this embodiment, when the display status of products changes, location information of the changed area on the product shelf and a classification of the change are identified. Therefore, the shopping list update unit 22 identifies products for which a change in display status has been detected based on the planogram information of the product shelf on which the products are located.

[0110] The shelf allocation information is information that indicates the arrangement positions of products prepared in advance on the product shelves of each store. The shelf allocation information is, for example, information that associates a product shelf number (tier) and row number with the product name to be arranged at that position. Furthermore, the position where the image is captured by the imaging device 2 and the position of the product shelf are associated and managed in advance. Therefore, by associating the area of ​​the product shelf in the image captured by the imaging device 2 with the shelf allocation information, it becomes possible to identify the product.

[0111] FIG. 10 is an explanatory diagram showing an example of shelf allocation information. In the example shown in FIG. 10, image I1 is assumed to be an image captured by a camera with camera ID=1, which is an imaging device. Furthermore, shelf allocation information I2 shown in FIG. 10 represents products arranged in an area specified by the rows and columns of a product shelf. For example, it is assumed that the position information of product I3 included in image I1 matches the position information in the first row of the association information shown in FIG. 8 (i.e., position information (10,0,30,50)). In this case, the shopping list update unit 22 identifies product I3 as product a associated with row 1 of row 1 in shelf allocation information I2.

[0112] The relevance information includes the classification results of the changed area. In other words, it is possible to identify changes in the display state from the relevance information. Therefore, the shopping list update unit 22 identifies products whose display state has been detected to have changed due to a person picking up the product, based on the shelf allocation information of the product shelf on which the product is located. The shopping list update unit 22 then performs a registration process to register the identified product in the shopping list corresponding to the person. In the above example, a change in the display state due to a person picking up an product corresponds to "a change due to the product no longer being included on the product shelf 3," and corresponds to "a decrease in the number of products" in terms of the type of change.

[0113] The shopping list update unit 22 performs a process of registering an item in a shopping list corresponding to a person as a registration process for registering the item in the shopping list. Furthermore, as a registration process, the shopping list update unit 22 may cause the notification unit 23, which will be described later, to notify the terminal 10 of information that the item has been registered. At that time, the notification unit 23 may inquire of the customer via the terminal 10 whether the correct item has been added to the shopping list.

[0114] Furthermore, the shopping list update unit 22 performs a deletion process to delete the returned item from the shopping list corresponding to the person based on the detection result and the shelf allocation information by the display detection unit 160. In this embodiment, if the display detection unit 160 detects that an item has been returned to a location different from where it was taken, the shopping list update unit 22 performs a deletion process to delete the item (the item that was returned to a location different from where it was taken) from the shopping list corresponding to the person based on the detection result and the shelf allocation information.

[0115] The shopping list update unit 22 performs a process of deleting an item from a shopping list corresponding to a person as a deletion process for deleting the item from the shopping list. Furthermore, as a deletion process, the shopping list update unit 22 may cause the notification unit 23, which will be described later, to notify the terminal 10 of information that the item has been deleted. At that time, the notification unit 23 may inquire of the customer via the terminal 10 whether the correct item has been deleted from the shopping list.

[0116] Alternatively, the shopping list update unit 22 may not immediately delete the item from the shopping list, but may instead, as a deletion process, set a deletion flag for the item included in the shopping list, which identifies the item as being returned to a location different from where it was taken. The shopping list update unit 22 may then cause the notification unit 23, described below, to notify the terminal 10 of information that the deletion flag has been set. At this time, the notification unit 23 may similarly inquire of the customer via the terminal 10 whether the item for which the deletion flag has been set should be deleted. The shopping list update unit 22 may then receive, via the terminal 10 carried by the notified person, an instruction indicating whether to delete the item for which the deletion flag has been set, and, upon receiving the instruction to delete the item, delete the item from the shopping list.

[0117] The notification unit 23 notifies the terminal 10 of the shopping list information. As described above, the notification unit 23 may notify the terminal 10 of the registration and deletion of products and the setting of a deletion flag in response to the processing of the shopping list update unit 22.

[0118] Furthermore, if there is an item in the shopping list for which a deletion flag is set, the notification unit 23 may notify the payment device 40 (described later) that the payment process will be stopped. By sending such a notification, payment for an unauthorized item can be prevented from being made.

[0119] The output device 30 outputs the contents of the shopping list. The output device 30 may be installed, for example, near the payment device 40 described below, and output the contents of the shopping list during payment processing. Note that if the contents of the shopping list can be output to the terminal 10, the self-checkout system 1 does not need to be equipped with the output device 30. The output device 30 may be, for example, a display device such as a monitor, or a POS (point of sales) terminal. Furthermore, the output device 30 is not limited to these and may be, for example, a speaker or a mobile terminal.

[0120] The payment device 40 performs payment processing based on the contents of the shopping list. For example, if the shopping list is associated with a terminal 10, the payment device 40 may notify the terminal 10 associated with the shopping list of the total amount and perform payment processing. Note that methods for the payment device 40 to perform payments via an individual's terminal 10 are widely known, and detailed explanations thereof will be omitted here.

[0121] On the other hand, if a shopping list is not associated with the terminal 10, the payment device 40 may display the contents of the shopping list and the total amount on the output device 30 and accept payment processing such as deposit from the customer or card payment. Note that payment methods based on deposit or card payment are also widely known, so a detailed description thereof will be omitted here.

[0122] Furthermore, when the payment device 40 receives a notification from the notification unit 23 to stop the payment process (specifically, a notification that there are remaining items with deletion flags), the payment device 40 may stop the payment process based on the shopping list and display various alerts. For example, the payment device 40 may notify the terminal 10 that there are remaining deletion flags. Furthermore, the payment device 40 may prompt the customer to confirm the products with deletion flags by, for example, displaying the products with deletion flags on the output device 30 or by providing audio guidance.

[0123] The image processing device 100 (more specifically, the first acquisition unit 110, the first change detection unit 120, the first storage unit 130, the first relevance generation unit 140, the relevance integration unit 150, and the display detection unit 160) are realized by a CPU of a computer that operates according to a program. For example, the program may be stored in a storage unit (not shown) included in the image processing device 100, and the CPU may read the program and operate as the first acquisition unit 110, the first change detection unit 120, the first storage unit 130, the first relevance generation unit 140, the relevance integration unit 150, and the display detection unit 160 according to the program.

[0124] In addition, the first acquisition unit 110, first change detection unit 120, first memory unit 130, first relevance generation unit 140, relevance integration unit 150 and display detection unit 160 included in the image processing device 100 may each be realized by dedicated hardware.

[0125] The shopping list management device 20 (more specifically, the shopping list generation unit 21, the shopping list update unit 22, and the notification unit 23) are also realized by the CPU of a computer that operates according to a program. For example, the program may be stored in a memory unit (not shown) provided in the shopping list management device 20, and the CPU may read the program and operate as the shopping list generation unit 21, the shopping list update unit 22, and the notification unit 23 according to the program. The shopping list generation unit 21, the shopping list update unit 22, and the notification unit 23 included in the shopping list management device 20 may each be realized by dedicated hardware.

[0126] Next, the operation of the image processing device 100 of this embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the operation of the image processing device 100 of this embodiment.

[0127] The first acquisition unit 110 acquires a captured image, which is an RGB image, from a video signal capturing an image of the product shelf 3 (step S1001). The first acquisition unit 110 supplies the acquired captured image to the first change detection unit 120.

[0128] Foreground region detection unit 121 of first change detection unit 120 uses the captured image, which is an RGB image supplied from first acquisition unit 110, and background information 131, which is an RGB image stored in first storage unit 130, to detect a region that has changed between the two RGB images as a changed region (foreground region) (step S1002). Then, foreground region detection unit 121 supplies the changed region detection result to background information update unit 122 and foreground region tracking unit 123. For example, foreground region detection unit 121 generates a binary image in which pixels in the detected changed region are set to 255 and pixels in other regions are set to 0, and supplies the binary image to background information update unit 122 and foreground region tracking unit 123 as the changed region detection result.

[0129] Furthermore, the foreground region detection unit 121 stores the foreground information 133 in the first storage unit 130 (step S1003). As described above, the foreground information 133 is a detection result associated with the image capture time.

[0130] Foreground region tracking unit 123 tracks the changed region based on the detection result supplied from foreground region detection unit 121 and foreground information 133 (step S1004). Foreground region tracking unit 123 supplies a binary image indicating the changed region that has been tracked for a predetermined period of time or more to first extraction unit 124 and second extraction unit 125. Foreground region tracking unit 123 supplies an update signal indicating an update of background information 131 to background information update unit 122.

[0131] If the amount of movement of the changed area is equal to or greater than a predetermined threshold as a result of tracking, foreground area tracking unit 123 determines that the object included in the changed area is a moving object and extracts the determined changed area as a person area. After that, foreground area tracking unit 123 associates predetermined information with the person area and stores it in the first storage unit as person information 134.

[0132] The background information update unit 122 updates the background information 131 based on the captured image supplied from the first acquisition unit 110, the detection result of the changed area supplied from the foreground area detection unit 121, the background information 131, and the update signal supplied from the foreground area tracking unit 123 (step S1005). Note that step S1005 may be performed at any timing after step S1004.

[0133] Based on the captured image supplied from the first acquisition unit 110 and the detection result related to the captured image supplied from the foreground region tracking unit 123, the first extraction unit 124 extracts, as a first image of interest, an image of a region (first region of interest) in the captured image that corresponds to the changed region indicated by the detection result (step S1006). The first extraction unit 124 supplies the extracted first image of interest to the region change classification unit 126.

[0134] The second extraction unit 125 extracts a second image of interest from the background information 131 by performing the same operation as the first extraction unit 124, based on the detection result supplied from the foreground region tracking unit 123 and the background information 131 used to obtain the detection result, which is acquired from the first storage unit 130 (step S1007). The second extraction unit 125 supplies the extracted second image of interest to the region change classification unit 126. Note that steps S1006 and S1007 may be performed simultaneously or in reverse order.

[0135] The area change classification unit 126 classifies the change related to the product shelf 3 based on the first image of interest supplied from the first extraction unit 124, the second image of interest supplied from the second extraction unit 125, and the shelf change model 132 stored in the first storage unit 130 (step S1008). Specifically, the change related to the product shelf 3 is a change from the state in the second image of interest to the state in the first image of interest.

[0136] The first association generation unit 140 receives the classification result of the changed area and the position information of the changed area from the area change classification unit 126 of the first change detection unit 120. The first association generation unit 140 also acquires person information 134 from the first storage unit 130. The first association generation unit 140 then extracts people who intersect with the changed area from among people whose images were captured at a time before the image capture time at which the changed area was detected. Thereafter, the first association generation unit 140 associates the changed area with the person whose image was captured at the time closest to the image capture time of the extracted person (step S1009). In this way, the first association generation unit 140 generates association information.

[0137] The relevance integration unit 150 receives relevance information between products and people from the first relevance generation unit 140, and if there is relevance information about the same person, integrates it into one. The relevance integration unit 150 calculates the similarity based on, for example, at least one of the color, area, shape, and aspect ratio of the circumscribed rectangle of the person area stored in the person information 134 in the first storage unit 130. Then, the relevance integration unit 150 determines that the person areas with the highest calculated similarity belong to the same person. Thereafter, the relevance integration unit 150 integrates the relevance information including the people determined to be the same person into one (step S1010).

[0138] The display detection unit 160 receives the integrated relevance information from the relevance integration unit 150. Then, the display detection unit 160 detects that the product has been returned to a location different from where it was taken by, for example, comparing the location where the product was acquired with the location where the product was returned (step S1011). Furthermore, the display detection unit 160 detects that the product has been returned to a location different from where it was taken when, for example, the appearance of the product has changed.

[0139] The image processing device 100 determines whether the first acquisition unit 110 has received the next video signal (whether there is a next captured image) (step S1012). If there is a next captured image (YES in step S1012), the process proceeds to step S1001. On the other hand, if there is not a next captured image (NO in step S1012), the image processing device 100 ends its operation.

[0140] Next, the operation of the self-checkout system 1 of this embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the operation of the self-checkout system 1 of this embodiment. First, when a customer (person) enters a store, the person identification device 50 identifies the person and associates the person with the terminal 10 (step S1101). In addition, the shopping list generation unit 21 generates a shopping list corresponding to the person (step S1102). After the customer enters the store, the imaging device 2 acquires a captured image (step S1103), and the image processing device 100 detects a change in the display state of products based on the captured image (step S1104).

[0141] When image processing device 100 detects a change in the display state caused by a person picking up an item (YES in step S1105), shopping list update unit 22 identifies the item based on the product shelf allocation information and performs a registration process to register the identified item in the shopping list corresponding to the person (step S1106). On the other hand, when image processing device 100 does not detect a change in the display state caused by a person picking up an item (NO in step S1105), the process proceeds to step S1107.

[0142] On the other hand, if image processing device 100 detects that an item has been returned to a location different from where it was taken (YES in step S1107), shopping list update unit 22 performs a deletion process to delete the item that has been returned to a location different from where it was taken from the shopping list corresponding to the person based on the detection result and the shelf allocation information (step S1108). Note that even if image processing device 100 detects a change in the display state due to the person returning an item, shopping list update unit 22 identifies the item based on the shelf allocation information and deletes the identified item from the shopping list corresponding to the person. On the other hand, if it is not detected that the item has been returned to a location different from where it was taken (NO in step S1107), the process proceeds to step S1109.

[0143] The settlement device 40 performs settlement processing based on the contents of the shopping list (step S1109).

[0144] As described above, in this embodiment, the image processing device 100 (more specifically, the first change detection unit 120) detects a change in the display state of products based on a captured image in which the products are captured. Furthermore, the image processing device 100 (more specifically, the display detection unit 160) detects that a product has been returned to a location different from the location from which it was taken, based on the detected change in the display state of the products and a person included in the captured image. Then, the shopping list management device 20 (more specifically, the shopping list update unit 22) identifies a product whose display state has been detected to have changed due to a person picking up the product, based on the shelf allocation information of the product shelf on which the product is placed, and performs a registration process to register the identified product in a shopping list corresponding to the person. Furthermore, the shopping list management device 20 (more specifically, the shopping list update unit 22) performs a deletion process to delete a product that has been returned to a location different from the location from which it was taken, from the shopping list corresponding to the person, based on the detection result by the display detection unit 160 and the shelf allocation information. Therefore, even if a product is returned to a location different from the location from which it was taken, it is possible to appropriately manage products purchased by a customer.

[0145] In this embodiment, when the image processing device 100 detects that an item has been returned to a location different from where it was taken, the shopping list update unit 22 deletes the returned item from the shopping list corresponding to the person. However, depending on the product layout, the same item may be placed in multiple locations. This situation can be identified from the shelf allocation information. Therefore, the display detection unit 160 may detect items of the same type that are displayed in multiple locations based on the shelf allocation information, and may detect that an item has been returned to the same location even if it has been returned to a location different from where it was taken, if the location is a location where the same type of item is displayed.

[0146] As described above, the shelf change model 132 is a model that represents changes related to the product shelves 3. Therefore, the first change detection unit 120 may classify changes related to the product shelves 3 in an area detected as a changed area into types such as a product being removed from the product shelves 3 or a product being replenished.

[0147] Therefore, the image processing device 100 of this embodiment can not only detect that there has been a change in the products on the product shelf 3, but also identify the type of change. As a result, the image processing device 100 can more accurately determine the state of the product shelf 3, such as whether a product has been removed or whether the product shelf 3 has been replenished.

[0148] The image processing device 100 described in this embodiment can determine, for each person, whether the person took a product displayed on the shelf 3 or returned the product to the shelf 3, based on the classification results and person detection results. As a result, as described above, the image processing device 100 can detect that a product has been returned to a location different from the one from which it was taken. For example, if a product is returned to a location different from the one from which it was taken, such as returning a refrigerated product to a room-temperature shelf, sales opportunity losses and product waste may occur, significantly affecting store sales. Therefore, when such an event occurs, it is preferable to promptly carry out product management work to resolve the situation. By using the image processing device 100 of this embodiment, it is possible to reduce sales opportunity losses and product waste caused by products being returned to a location different from the one from which they were taken.

[0149] In the present embodiment, the case where the imaging device 2 captures an image of the product shelf 3 has been described. However, the object captured by the imaging device 2 is not limited to the product shelf 3. The imaging device 2 may, for example, capture an image of products stacked on a cart. That is, the captured image captured by the imaging device 2 may be an image of products stacked on a cart. The image processing device 100 may detect a change area by comparing the captured image of products stacked on a cart with a background image. As described above, the image processing device 100 is not limited to product shelves where the products are displayed so that all of their faces are visible, and can use captured images of products displayed in various display methods.

[0150] [Second embodiment] Next, a second embodiment of the present invention will be described. In the second embodiment, a self-checkout system 1 will be described in which multiple imaging devices 2 monitor different product shelves 3. As will be described later, an image processing device 200 in this embodiment detects that a product has been returned to a location different from where it was taken, based on the monitoring results of the different product shelves 3. In other words, the image processing device 200 described in this embodiment can detect that a product has been returned to a location different from where it was taken, even if the product shelf 3 from which the product was taken is different from the product shelf 3 to which the product was returned.

[0151] 1 described in the first embodiment, the image processing device 200 in this embodiment is communicably connected to the terminal 10, the imaging device 2, the shopping list management device 20, the output device 30, the payment device 40, and the person identification device 50. The terminal 10, the imaging device 2, the shopping list management device 20, the output device 30, the payment device 40, and the person identification device 50 in this embodiment are the same as those in the first embodiment.

[0152] FIG. 13 is an explanatory diagram showing an example configuration of an image processing device 200. The image processing device 200 of this embodiment includes a first acquisition unit 110, a second acquisition unit 210, a first change detection unit 120, a second change detection unit 220, a first storage unit 130, a second storage unit 230, a first relevance generation unit 140, a second relevance generation unit 240, a relevance integration unit 250, and a display detection unit 160. As described above, the image processing device 200 of this embodiment includes the relevance integration unit 250 instead of the relevance integration unit 150 included in the image processing device 100. Furthermore, the image processing device 200 includes the second acquisition unit 210, the second change detection unit 220, the second storage unit 230, and the second relevance generation unit 240 in addition to the configuration included in the image processing device 100.

[0153] In the above, elements having the same functions as elements included in the drawings explained in the first embodiment are denoted by the same reference numerals. The following describes the characteristic configuration of this embodiment.

[0154] The second acquisition unit 210 acquires an RGB image by performing the same operation as the first acquisition unit 110. Then, the second acquisition unit 210 supplies the RGB image to the second change detection unit 220. For example, the second acquisition unit 210 acquires a video signal from an imaging device 2 that monitors a product shelf 3 different from the imaging device 2 that is the transmission source of the video signal acquired by the first acquisition unit 110.

[0155] The second storage unit 230 has the same configuration as the first storage unit 130. Therefore, detailed description will be omitted. Note that the second storage unit 230 may be the same storage device as the first storage unit 130, or may be a different storage device.

[0156] The second change detection unit 220 has the same configuration as the first change detection unit 120. The second change detection unit 220 detects changed areas related to the product shelf 3 by performing the same operation as the first change detection unit 120. The second change detection unit 220 then classifies changes related to the product shelf 3 in the changed areas based on the detected changed areas and a shelf change model 132, which is a model of changes related to the product shelf 3 that has been learned in advance. The second change detection unit 220 then associates position information of the first area of ​​interest with the image capture time, and supplies this to the second relevance generation unit 240. The second change detection unit 220 also supplies the results of classifying the changes related to the product shelf 3 in the changed areas to the second relevance generation unit 240.

[0157] The second relationship generation unit 240 has the same configuration as the first relationship generation unit 140. The second relationship generation unit 240 generates relationship information between products and people by performing the same operation as the first relationship generation unit. Then, the second relationship generation unit 240 supplies the generated relationship information to the relationship integration unit 250.

[0158] FIG. 14 is an explanatory diagram showing an example of the relationship information between products and people generated by the first relationship generation unit 140 and the second relationship generation unit 240. Similar to FIG. 7, FIG. 14 illustrates a camera ID indicating the ID of an imaging device, a person ID indicating a person captured by the imaging device, position information of a changed area on the product shelf, and a classification result of the change. The example shown in FIG. 14 indicates that people with person ID=1 and person ID=2 were captured by the imaging device with camera ID=1, and people with person ID=3 and person ID=4 were captured by the imaging device with camera ID=2. Note that in the example shown in FIG. 14, for example, the information for camera ID=1 is information generated by the first relationship generation unit 140, and the information for camera ID=2 is information generated by the second relationship generation unit 240.

[0159] The relationship integration unit 250 receives relationship information between products and people from the first relationship generation unit 140 and the second relationship generation unit 240, respectively, and integrates the relationship information into one if there is relationship information about the same person. The relationship integration unit 250 then supplies the integrated relationship information to the display detection unit 160. The relationship integration unit 250 integrates multiple pieces of relationship information by, for example, operating in the same manner as the relationship integration unit 150.

[0160] Fig. 15 is an explanatory diagram showing an example in which the relationship integration unit 250 integrates the relationship information between the product and the person exemplified in Fig. 14. In the example shown in Fig. 15, the relationship information of person ID=1 and person ID=3 in Fig. 14 is integrated into one.

[0161] The display detection unit 160 receives the integrated relevance information from the relevance integration unit 250 and detects that the product has been returned to a location different from where it was taken by, for example, comparing the location where the product was acquired with the location where the product was returned.

[0162] The operation of the display detection unit 160 is the same as in the first embodiment. For example, the display detection unit 160 compares the position information and the type of change for each person ID. Specifically, in the example shown in FIG. 15, a person with person ID=1 acquires a product from a location (10,0,30,50) on a product shelf captured by an imaging device with camera ID=1, and returns the product to a location (100,250,50,70) on a product shelf captured by an imaging device with camera ID=2. As a result, the display detection unit 160 detects that the person with person ID=1 has returned the product to a shelf other than the shelf from which they acquired the product.

[0163] The above is an example of a characteristic configuration of the image processing device 200.

[0164] Next, the operation of the image processing device 200 of this embodiment will be described with reference to Fig. 16. Fig. 16 is a flowchart showing an example of the operation of the image processing device 200 of this embodiment.

[0165] 16, the first relationship creation unit 140 creates relationships (step S2001) by performing processes similar to the processes from step S1001 to step S1009 illustrated in Fig. 11 by the image processing device 200. Similarly, the second relationship creation unit 240 creates relationships (step S2002) by performing processes similar to the processes from step S1001 to step S1009. Note that the processes of step S2001 and step S2002 may be performed in parallel, or one may be performed first.

[0166] The relevance integration unit 250 receives the relevance information between products and people from the first relevance generation unit 140 and the second relevance generation unit 240. Then, when there is relevance information between products and people for the same person in the same imaging device 2 or among multiple imaging devices 2, the relevance integration unit 250 integrates the information into one (step S2003). Note that the relevance integration unit 250 determines whether the information is the same person by, for example, performing the same operation as the relevance integration unit 150.

[0167] The display detection unit 160 detects that the product has been returned to a location different from the location from which it was taken, based on the relevance information received from the relevance integration unit 150 (step S2004). The image processing device 200 determines whether the first acquisition unit 110 or the second acquisition unit 210 has received the next video signal (whether there is a next captured image) (step S2005). If there is a next captured image (YES in step S2005), the process proceeds to step S2001 and step S2002. On the other hand, if there is not a next captured image (NO in step S2005), the image processing device 200 ends its operation.

[0168] The processes in steps S2004 and S2005 are similar to the processes in steps S1011 and S1012 in FIG. 11 described in the first embodiment.

[0169] As described above, the image processing device 200 of this embodiment includes the first relevance generation unit 140 and the second relevance generation unit 240. The relevance integration unit 250 integrates the relevance information generated by the first relevance generation unit 140 and the relevance information generated by the second relevance generation unit. In other words, the relevance integration unit 250 integrates the relevance information between products and people for different product shelves 3 captured by multiple image capture devices 2. As a result, not only can the same effects as those of the first embodiment be obtained, but the image processing device 200 can also detect that a product has been returned to a different location from where it was taken, even if the shelf from which the customer obtained the product is different from the shelf to which the customer returned the product. This reduces sales opportunity losses and product waste due to improper product display across a wider area of ​​the store than in the first embodiment.

[0170] In the second embodiment, the image processing device 200 includes a first processing unit including a first acquisition unit 110, a first change detection unit 120, a first storage unit 130, and a first relevance generation unit 140, and a second processing unit including a second acquisition unit 210, a second change detection unit 220, a second storage unit 230, and a second relevance generation unit 240. That is, in the second embodiment, the image processing device 200 includes two various processing units. However, the number of various processing units included in the image processing device 200 is not limited to two. The image processing device 200 may include any number of various processing units, for example, three or more. In other words, the image processing device 200 may be configured to process captured images transmitted from imaging devices 2 monitoring three or more different product shelves 3.

[0171] [Third embodiment] Next, a third embodiment of the present invention will be described. In the third embodiment, an image processing device 300 will be described that has a configuration for generating customer flow line data in addition to the configuration of the image processing device 100 described in the first embodiment. As will be described later, the image processing device 300 generates relevance information using the generated flow line data. This makes it possible to detect with higher accuracy whether a product has been returned to a location different from where it was taken.

[0172] 17 is an explanatory diagram showing an example of the configuration of a third embodiment of a self-checkout system according to the present invention. A self-checkout system 4 of this embodiment includes an imaging device 2 that captures images of product shelves 3, as well as an imaging device 5 that captures images of aisles within a store similar to a general surveillance camera. Like the imaging device 2, the imaging device 5 is communicably connected to a terminal 10, a shopping list management device 20, an output device 30, a payment device 40, a person identification device 50, and an image processing device 300 via a network N. The terminal 10, imaging device 2, shopping list management device 20, output device 30, payment device 40, and person identification device 50 of this embodiment are also the same as those of the first embodiment.

[0173] The imaging device 5 is, for example, a surveillance camera installed in a store. The imaging device 5 is installed, for example, in a position where it can capture images of each aisle in the store. The configuration of the imaging device 5 may be the same as that of the imaging device 2.

[0174] The self-checkout system 4 may have one or more imaging devices 5. That is, the self-checkout system 4 may capture images of each aisle in the store using one imaging device 5, or may capture images of each aisle in the store using multiple imaging devices 5. In this embodiment, a case will be described in which the self-checkout system 4 has an imaging device 5 separate from the imaging device 2. However, the self-checkout system 4 may also acquire customer flow line data based on images captured by multiple imaging devices 2, for example. That is, the self-checkout system 4 may acquire customer flow line data based on images captured by multiple imaging devices 2 that capture images of the product shelves 3.

[0175] The image processing device 300 generates flow line data indicating the customer's movement route within the store from the RGB image captured by the imaging device 5. Then, the image processing device 300 generates information on the association between products and people based on the generated flow line data and the changed area of ​​the product shelf 3.

[0176] Fig. 18 is a block diagram showing an example of the configuration of an image processing device 300. As shown in Fig. 18, the image processing device 300 of this embodiment includes a first acquisition unit 110, a first change detection unit 120, a first storage unit 130, a first relationship generation unit 340, a third acquisition unit 310, a movement line data generation unit 320, a relationship integration unit 150, and a display detection unit 160. As such, the image processing device 300 of this embodiment includes the first relationship generation unit 340 instead of the first relationship generation unit 140 of the image processing device 100. Furthermore, the image processing device 300 includes a third acquisition unit 310 and a movement line data generation unit 320 in addition to the configuration of the image processing device 100.

[0177] In the above, elements having the same functions as elements included in the drawings explained in the first and second embodiments are denoted by the same reference numerals. The following describes the characteristic configuration of this embodiment.

[0178] The third acquisition unit 310 acquires RGB images from the imaging device 5 by performing the same operation as the first acquisition unit 110, and supplies the RGB images to the flow line data generation unit 320. The RGB images acquired by the third acquisition unit 310 are images of the aisles of a store, similar to those captured by a general surveillance camera.

[0179] The flow line data generator 320 generates data on the flow line of a person in a store using RGB images captured by at least one imaging device 5. The flow line data generated by the flow line data generator 320 includes, for example, a person ID that identifies the customer and a product shelf ID that identifies the product shelf 3 visited by the customer. The method by which the flow line data generator 320 generates the flow line data is not particularly limited. The flow line data generator 320 may generate the flow line data using, for example, the method described in Patent Document 2. That is, the flow line data generator 320 identifies customers in the captured images by, for example, performing facial recognition to detect people who provide identical data as the same person, or by extracting information indicating customer characteristics such as clothing to detect the same person. The flow line data generator 320 also determines that a customer has visited a product shelf 3 by, for example, determining whether the customer has stayed in front of the product shelf 3 for a certain period of time, or whether the distance between the customer and the product shelf 3 is equal to or less than a predetermined distance. Then, the flow line data generation unit 320 generates flow line data by associating a person ID for identifying a customer with the product shelf ID visited by the customer. Note that the flow line data generation unit 320 may generate flow line data using a method other than the above example.

[0180] FIG. 19 is an explanatory diagram showing an example of flow line data generated by the flow line data generation unit 320. The flow line data shown in FIG. 19 includes a person ID indicating a person, a product shelf ID indicating a product shelf visited by the person, and a camera ID indicating an imaging device capturing an image of the product shelf. The example shown in FIG. 19 shows that three people visited product shelf A, and that a person with person ID = 1 and a person with person ID = 2 visited product shelf A twice. Note that the example shown in FIG. 19 does not specify the times when the people visited the product shelves, but the flow line data may include the times when each product shelf was visited. The example shown in FIG. 19 shows that the people visited product shelf A in order from the top row to the oldest.

[0181] 19. Then, the flow line data generation unit 320 associates the generated flow line data with the time of the captured image of the flow line data, and supplies the associated data to the first association generation unit 340. In other words, the flow line data generation unit 320 includes in the flow line data the time at which each person in the flow line data visited each product shelf, and supplies the associated data to the first association generation unit 340.

[0182] The first relationship generation unit 340 receives the classification result of the changed area and the position information of the changed area from the first change detection unit 120. The first relationship generation unit 340 also acquires movement line data from the movement line data generation unit 320. The first relationship generation unit 340 then generates relationship information between the product and the person corresponding to the changed area based on the image capture time of the changed area linked to the position information of the changed area and the image capture time linked to the movement line data. The first relationship generation unit 340 then supplies the generated relationship information between the product and the person to the relationship integration unit 150.

[0183] Specifically, the first association generation unit 340 associates the change area with the person who visited the product shelf at the time closest to the image capture time of the change area from the traffic line data generated at a time before the image capture time when the change area was detected.

[0184] Fig. 20 is an explanatory diagram showing an example of the relevance information between a product and a person generated by the first relevance generation unit 340. Fig. 20 shows an example of a camera ID indicating the ID of an imaging device, a person ID indicating a person captured by the imaging device, a product shelf ID indicating a product shelf, position information of a changed area of ​​the product shelf, and a classification result of the change.

[0185] In this way, the first relationship generation unit 340 generates the relationship information by using the flow line data instead of the person information 134. Note that the first relationship generation unit 340 may also use the person information 134 when generating the relationship information.

[0186] The relationship integration unit 150 receives the relationships between products and people from the first relationship generation unit 340, and if there are relationships between the same person, integrates them into one. Then, the relationship integration unit 150 supplies the integrated relationship to the display detection unit 160.

[0187] Specifically, the relationship integration unit 150 integrates relationships between the same person based on the person ID. Fig. 21 is an explanatory diagram showing an example of integration of relationships between products and people shown in Fig. 20. In the example shown in Fig. 21, the relationship information of the person with person ID = 1 in Fig. 20 is integrated into one, and the relationship information of the person with person ID = 2 is also integrated into one.

[0188] The operation of the display detection unit 160 is the same as that described in the first and second embodiments, and therefore a description thereof will be omitted.

[0189] The above is an example of a characteristic configuration of the image processing device 300.

[0190] Next, the operation of the image processing device 300 of this embodiment will be described with reference to Fig. 22. Fig. 22 is a flowchart showing an example of the operation of the image processing device 300 of this embodiment.

[0191] Referring to FIG. 22, the image processing device 300 performs the same processes as steps S1001 to S1008 shown in FIG. 11, and the area change classification unit 126 classifies the change (step S3001).

[0192] The third acquisition unit 310 acquires a captured image (third captured image) that is an RGB image from the video signal transmitted from the imaging device 5 that captured an image of the aisle of the store (step S3002). The third acquisition unit 310 supplies the acquired captured image to the flow line data generation unit 320.

[0193] The flow line data generation unit 320 generates flow line data of people in the store using RGB images captured by at least one imaging device (step S3003). Then, the flow line data generation unit 320 associates the generated flow line data with the time of the captured image of the flow line data and supplies the data to the first association generation unit 340.

[0194] The processing of steps S3002 and S3003 may be executed in reverse order with respect to the processing of step S3001, or may be executed simultaneously.

[0195] The first association generation unit 340 receives the classification result of the changed area and the position information of the changed area from the first change detection unit 120. The first association generation unit 340 also acquires movement line data from the movement line data generation unit 320. The first association generation unit 340 then generates association information between the product and person corresponding to the changed area based on the image capture time of the changed area linked to the position information of the changed area and the image capture time linked to the movement line data (step S3004). Specifically, the first association generation unit 340 associates the changed area with the person who visited the shelf at the time closest to the image capture time of the changed area from the movement line data generated at a time before the image capture time at which the changed area was detected. The first association generation unit 340 then supplies the generated association information between the product and person to the association integration unit 150.

[0196] Thereafter, the image processing device 300 performs the processes from step S3005 to step S3007, which are the same as the processes from step S1010 to step S1012 shown in FIG.

[0197] As described above, the image processing device 300 of this embodiment includes the third acquisition unit 310 and the flow line data generation unit 320. With this configuration, the flow line data generation unit 320 generates flow line data based on the RGB image provided by the third acquisition unit 310. The image processing device 300 then generates relevance information between products and people based on change areas detected from the RGB image of the product shelf 3, the classification results of the change areas, and flow line data generated from the RGB image of the store aisle. The captured image of the store aisle captured by the imaging device 5 and used to generate the flow line data captures the entire body of the person. Therefore, it is easier to identify the person in the captured image than in the RGB image of the product shelf 3. In other words, generating relevance information using flow line data, as in the image processing device 300, is expected to generate more accurate relevance information than generating relevance information using the person information 134. Therefore, the image processing device 300 of this embodiment can integrate relevance information with higher accuracy than the image processing device 100 of the first embodiment. This improves the accuracy of detecting whether a product has been returned to a location different from where it was taken, and therefore, by using the image processing device 300, it is possible to reduce sales opportunity losses and product disposal losses due to improper product display more than the image processing device 100.

[0198] Next, a modified example of the third embodiment will be described. In the third embodiment, the imaging device 5 captures images of the aisles in the store to generate customer flow line data. However, the method of generating flow line data is not limited to the method using images. FIG. 23 is a block diagram showing a modified example of the self-checkout system of the third embodiment. The self-checkout system 4a illustrated in FIG. 23 includes a flow line detection device 6 that detects customer flow lines, instead of the imaging device 5 included in the self-checkout system 4 of the third embodiment. Furthermore, the self-checkout system 4a illustrated in FIG. 23 includes an image processing device 300a, instead of the image processing device 300 included in the self-checkout system 4 of the third embodiment.

[0199] The flow line detection device 6 is a device that detects the location of the terminal 10 carried by a customer. The flow line detection device 6 detects the location of the customer, for example, by detecting short-range wireless communication from the terminal 10. For example, when the person identification device 50 authenticates the terminal 10 at a predetermined location in the store (e.g., an entrance), the flow line detection device 6 tracks the authenticated person and detects the person's flow line. The shopping list generation unit 21 may generate a shopping list corresponding to the person carrying the authenticated terminal 10 at the time of this authentication.

[0200] Fig. 24 is a block diagram showing an example configuration of an image processing device 300a. As shown in Fig. 24, the image processing device 300a of this embodiment includes a first acquisition unit 110, a first change detection unit 120, a first storage unit 130, a third acquisition unit 310a, a movement line data generation unit 320a, a first relationship generation unit 340, a relationship integration unit 150, and a display detection unit 160. In this way, the image processing device 300a of this modification includes the third acquisition unit 310a and the movement line data generation unit 320a instead of the third acquisition unit 310 and the movement line data generation unit 320 of the image processing device 300.

[0201] The third acquisition unit 310a acquires customer location information from the flow line detection device 6. The customer location information includes, for example, the identification information of the terminal 10. The flow line data generation unit 320a generates data on the movement of people within a store. The flow line data generated by the flow line data generation unit 320a includes, for example, the identification information of the terminal 10 carried by the customer and a product shelf ID that is an identifier of the product shelf 3 visited by the customer. Note that the method by which the flow line data generation unit 320a generates the flow line data is not particularly limited. As in the third embodiment, the flow line data generation unit 320a determines that a customer has visited a product shelf 3 by, for example, whether the customer has stayed in front of the product shelf 3 for a certain period of time, or whether the distance between the customer and the product shelf 3 has become equal to or less than a predetermined distance. Then, the flow line data generation unit 320a generates the flow line data by associating the identification information of the terminal 10 carried by the customer with the product shelf ID visited by the customer.

[0202] FIG. 25 is an explanatory diagram showing another example of flow line data. For example, as shown in FIG. 25, the flow line data generation unit 320a generates flow line data in which the identification information of the terminal 10 is associated with a product shelf ID. Furthermore, the flow line data generation unit 320a associates the generated flow line data with the acquisition time of the flow line data and supplies the data to the first association generation unit 340. In other words, the flow line data generation unit 320a includes the time when each person in the flow line data visited each product shelf and supplies the data to the first association generation unit 340. The subsequent processing is the same as in the third embodiment. That is, in this modification, the display detection unit 160 detects that a product has been returned to a location different from the location from which it was taken, based on the change in the display state of the detected product and the person whose movement within the store is detected.

[0203] As described above, in this modification, the flow line detection device 6 generates flow line data based on the position information of the detected person, and the image processing device 300a (more specifically, the display detection unit 160) detects that a product has been returned to a location different from where it was taken, based on the detected change in the display state of the products and the detected person's flow line within the store. Therefore, the accuracy of detecting that a product has been returned to a location different from where it was taken can be improved, and the image processing device 300a can reduce sales opportunity losses and product disposal losses due to improper product display more than the image processing device 100.

[0204] [Fourth embodiment] Next, a fourth embodiment of the present invention will be described. In the fourth embodiment, an image processing device 400 will be described that has the configuration for generating customer flow line data, as described in the third embodiment, in addition to the configuration of the image processing device 200 described in the second embodiment. In other words, in this embodiment, an image processing device will be described that combines the configurations described in the second embodiment and the configurations described in the third embodiment.

[0205] The image processing device 400 of this embodiment is communicatively connected to a terminal 10, an imaging device 2, a shopping list management device 20, an output device 30, a payment device 40, and a person identification device 50, similar to the image processing device 100 shown in FIG. 1 described in the first embodiment.

[0206] FIG. 26 is an explanatory diagram showing an example configuration of an image processing device 400. The image processing device 400 of this embodiment includes a first acquisition unit 110, a second acquisition unit 210, a first change detection unit 120, a second change detection unit 220, a first storage unit 130, a second storage unit 230, a first relationship generation unit 340, a second relationship generation unit 440, a third acquisition unit 310, a movement line data generation unit 320, a relationship integration unit 250, and a display detection unit 160. As described above, the image processing device 400 of this embodiment has the same configuration as the image processing device 200 described in the second embodiment, and also includes the third acquisition unit 310 and the movement line data generation unit 320. In other words, the image processing device 400 of this embodiment includes a first relationship generation unit 340 instead of the first relationship generation unit 140 of the image processing device 200, and a second relationship generation unit 440 instead of the second relationship generation unit 240. Furthermore, the image processing device 400 includes a third acquisition unit 310 and a flow line data generation unit 320.

[0207] In the above, elements having the same functions as elements included in the drawings described in the first, second, and third embodiments are denoted by the same reference numerals. The following describes the configuration characteristic of this embodiment.

[0208] The second relationship generation unit 440 receives the classification result of the changed area and the position information of the changed area from the second change detection unit 220. The second relationship generation unit 440 also acquires movement line data from the movement line data generation unit 320. Then, the second relationship generation unit 440 generates relationship information between products and people by operating in the same manner as the first relationship generation unit 340, and supplies the generated relationship information to the relationship integration unit 250.

[0209] In this way, the second relevance generation unit 440 generates relevance information using flow line data, similar to the first relevance generation unit 340. The processes after the first relevance generation unit 340 and the second relevance generation unit 440 generate the relevance information are the same as those described in the second embodiment, and therefore detailed description will be omitted.

[0210] The above is an example of a characteristic configuration of the image processing device 400.

[0211] Next, the operation of the image processing device 400 of this embodiment will be described with reference to Fig. 27. Fig. 27 is a flowchart showing an example of the operation of the image processing device 400 of this embodiment.

[0212] 27, image processing device 400 performs the same processes as steps S1001 to S1008 shown in Fig. 11, whereby first change detection section 120 classifies changes (step S4001). Similarly, second change detection section 220 classifies changes (step S4002).

[0213] The third acquisition unit 310 acquires a captured image (third captured image) that is an RGB image from the video signal transmitted from the imaging device 5 that captured an image of the aisle of the store (step S4003). The third acquisition unit 310 supplies the acquired captured image to the flow line data generation unit 320.

[0214] The flow line data generation unit 320 generates flow line data of people in the store using RGB images captured by at least one imaging device (step S4004). Then, the flow line data generation unit 320 associates the generated flow line data with the time of the captured image of the flow line data, and supplies them to the first association generation unit 340.

[0215] The processes of steps S4003 and S4004 are similar to the processes of steps S3002 and S3003 described in the third embodiment. The processes of steps S4003 and S4004 may be executed in a reverse order with respect to the processes of steps S4001 and S4002, or may be executed simultaneously.

[0216] The first relation generation unit 340 generates relation information using the flow line data (Step S4005), and the second relation generation unit 440 generates relation information using the flow line data (Step S4006).

[0217] The processing in step S4005 and the processing in step S4006 are the same as the processing in step S3004 described in the third embodiment. In addition, the processing in step S4005 and the processing in step S4006 may be executed either first or simultaneously.

[0218] Thereafter, the image processing device 400 performs the processes from step S4007 to step S4009, which are the same as the processes from step S1010 to step S1012 shown in FIG.

[0219] As described above, the image processing device 400 of this embodiment includes a second relationship generation unit 440 instead of the second relationship generation unit 240 of the image processing device 200 of the second embodiment. Furthermore, the image processing device 400 includes a third acquisition unit 310 and a movement line data generation unit 320. Therefore, like the image processing device 200 of the second embodiment described above, the image processing device 400 of this embodiment can reduce sales opportunity losses and product waste caused by improper product display across a wide area of ​​a store. Furthermore, like the image processing device 300 of the third embodiment described above, the image processing device 400 can integrate relationship information between products and people with high accuracy. By integrating relationship information with high accuracy, the accuracy of detecting whether a product has been returned to a location different from where it was taken improves. Therefore, the image processing device 400 of this embodiment can reduce sales opportunity losses and product waste caused by improper product display compared to the image processing device 200.

[0220] [Fifth embodiment] Next, a fifth embodiment of the present invention will be described. In the fifth embodiment, an image processing device 500 will be described that has a notification unit 510 in addition to the configuration of the image processing device 100 described in the first embodiment. By having the notification unit 510, the image processing device 500 can notify a store clerk or the like when, for example, the display detection unit 160 detects that a product has been returned to a location different from the location from which it was taken.

[0221] The image processing device 500 of this embodiment is communicatively connected to a terminal 10, an imaging device 2, a shopping list management device 20, an output device 30, a payment device 40, and a person identification device 50, similar to the image processing device 100 shown in FIG. 1 described in the first embodiment.

[0222] 28 is an explanatory diagram showing an example of the configuration of an image processing device 500. The image processing device 500 of this embodiment includes a first acquisition unit 110, a first change detection unit 120, a first storage unit 130, a first relevance generation unit 140, a relevance integration unit 150, a display detection unit 160, and a notification unit 510. As such, the image processing device 500 of this embodiment has the same configuration as the image processing device 100 described in the first embodiment, and also includes the notification unit 510.

[0223] In the above, elements having the same functions as elements included in the drawings described in the first, second, third, and fourth embodiments are denoted by the same reference numerals. The following describes the configuration characteristic of this embodiment.

[0224] The display detection unit 160 receives the integrated relevance information from the relevance integration unit 150 and compares the location where the product was acquired with the location where the product was returned to detect that the product has been returned to a location different from where it was taken. The display detection unit 160, for example, supplies a signal indicating the detection result to the notification unit 510. The signal indicating the detection result may be, for example, 1 if it is detected that the product has been returned to a location different from where it was taken, and 0 if it is not detected that the product has been returned to a location different from where it was taken.

[0225] The notification unit 510 receives the detection result from the display detection unit 160. If the received detection result is a signal indicating that the product has been returned to a location different from the location from which it was taken, the notification unit 510 notifies, for example, a terminal (not shown) held by a store employee, that it has detected that the product has been returned to a location different from the location from which it was taken. If the received detection result is a signal indicating that the product has been returned to a location different from the location from which it was taken, the notification unit 510 may notify, for example, a point of sales (POS) terminal in the store, that it has detected that the product has been returned to a location different from the location from which it was taken. In this case, the notification unit 510 may notify a computer at the headquarters that it has detected that the product has been returned to a location different from the location from which it was taken. If the received detection result is a signal indicating that the product has been returned to a location different from the location from which it was taken, the notification unit 510 may store, in a storage medium installed in the store or the headquarters, information that it has detected that the product has been returned to a location different from the location from which it was taken. Furthermore, the notification unit 510 may use a predetermined lamp (not shown) or the like to notify that it has detected that the product has been returned to a location different from the location from which it was taken.

[0226] The notification unit 510 performs notification by any one of the above-mentioned methods or a combination of a plurality of methods. The notification unit 510 is realized by, for example, a CPU of a computer that operates according to a program.

[0227] The above is an example of a characteristic configuration of the image processing device 500.

[0228] Next, an example of the operation of the image processing device 500 of this embodiment will be described with reference to Fig. 29. Fig. 29 is a flowchart showing an example of the operation of the image processing device 500 of this embodiment.

[0229] 29, the image processing device 300 performs the same processes as steps S1001 to S1011 shown in Fig. 11, and the display detection unit 160 detects that a product has been returned to a location different from the location from which it was taken (step S5001). The display detection unit 160 supplies, for example, a signal indicating the detection result to the notification unit 510.

[0230] After step S5011 is completed, the notification unit 510 receives the detection result from the display detection unit 160. If the received detection result is a signal indicating that the product has been returned to a location different from the location from which it was taken, the notification unit 510 notifies the terminal carried by the store worker of this fact (step S5002).

[0231] Then, the image processing apparatus 500 executes step S5003, which is the same process as step S1012 shown in FIG.

[0232] As described above, the image processing device 500 of this embodiment has a configuration in which the notification unit 510 is added to the image processing device 100. With this configuration, when a product is returned to a location different from where it was taken, the store staff can be notified promptly. This makes it possible to reduce sales opportunity losses and product waste due to improper product display.

[0233] The notification unit 510 may be configured to postpone the timing of issuing a notification when the situation in the store or the like satisfies a predetermined condition.

[0234] The notification unit 510 may be configured to determine whether to withhold notification depending on, for example, the status of customer presence in the store. The notification unit 510 may withhold notification while it is determined, for example, based on captured images acquired by the imaging device 2, etc., that customers are lined up at the cash register (a predetermined number or more). Furthermore, the notification unit 510 may withhold notification while it is determined, for example, based on captured images acquired by the imaging device 2, etc., that a customer is in front of the product shelf 3 that is the target of work and that has detected that a product has been returned to a location different from where it was taken. Furthermore, if the image processing device 500 is capable of acquiring flow line data, the notification unit 510 may withhold notification when, for example, it is determined, based on the flow line data, that a customer will approach the product shelf 3 that is the target of work and that has detected that a product has been returned to a location different from where it was taken. In this way, the notification unit 510 may withhold notification when the status of customer presence in the store, such as the number of customers present and their locations, meets predetermined conditions. The notification unit 510 may issue a notification when there is no reason to defer or when the reason has disappeared.

[0235] The notification unit 510 may also determine the urgency of the notification based on the shelf 3 to which the product was returned, the type and characteristics of the product returned to a different location, and the relationship between the product returned to a different location and the shelf 3 to which it was returned, and control the timing of the notification based on the urgency of the notification. For example, if fresh food, frozen food, ice cream, or the like is returned to a non-frozen or non-refrigerated section, it is expected that the impact will be significant if a notification is not issued immediately. Also, if the relationship between the product and the shelf 3 to which the product was returned is unfavorable, such as when a non-food product such as detergent is returned to the perishable food section, it is desirable to correct it promptly. In this way, if it is determined that a notification is urgent based on the type of product returned to a different location, the notification unit 510 may issue a notification immediately. In other words, if there is an urgency in the notification as described above, for example, even if the above-mentioned hold conditions are met, the notification unit 510 will issue a notification without holding. On the other hand, if there is no urgency, the notification unit 510 may hold the notification until the hold conditions are no longer met. In this way, the alarm unit 510 may acquire information indicating the urgency of the alarm and control the timing of the alarm based on the acquired information. Note that the type and nature of a product that has been returned to a location different from where it was taken, such as a fresh food or a frozen food, may be determined by, for example, the area change classification unit 126 or the first extraction unit 124 based on the first featured image. For example, after the display detection unit 160 detects that a product has been returned to a location different from where it was taken, the display detection unit 160 may acquire a captured image from information about the captured image (such as an identifier and image capture time) added to the relevance information and determine the type and nature of the product based on the acquired captured image.

[0236] As described above, the notification unit 510 determines the urgency based on the customer's status in the store, the type and nature of the product, etc., and controls the timing of notification based on the urgency of the notification. In this case, the notification unit 510 may control the timing of notification based on information other than the above-mentioned examples.

[0237] In the present embodiment, the image processing device 100 described in the first embodiment includes the notification unit 510. However, each of the image processing units described above other than the image processing device 100 may include the notification unit 510. For example, the image processing device 200 described in the second embodiment, the image processing device 300 described in the third embodiment, and the image processing device 400 described in the fourth embodiment may include the notification unit 510. When the image processing device 200, the image processing device 300, or the image processing device 400 includes the notification unit 510, the processing performed by the notification unit 510 is the same as when the image processing device 100 includes the notification unit 510. In any of the above cases, the various modified examples described in the present embodiment can be adopted.

[0238] <Modification of foreground region detection unit> Next, a modified example of the processing of the foreground region detection section included in the first change detection section 120 or the second change detection section 220 of the image processing device of each of the above-described embodiments will be described.

[0239] In this modified example, the foreground area detection unit included in the first change detection unit 120 or the second change detection unit 220 further uses pre-registered shelf area information to identify that the object included in the change area is other than a product on the product shelf 3.

[0240] In this modification, a modification of the foreground region detection unit 121 of the image processing device 100 in the first embodiment will be described. However, this modification can be applied to any of the image processing device 200, image processing device 300, image processing device 400, image processing device 500, and devices other than the image processing device 100 that have the notification unit 510.

[0241] 30, 31, and 32 are explanatory diagrams showing an example of the operation of foreground region detection section 121 in this modified example to detect a foreground region.

[0242] For example, suppose that foreground region detection unit 121 detects changed regions by comparing the captured image supplied from first acquisition unit 110 with background information 131, and generates detection result 71, which is a binary image representing the changed regions, as shown in FIG. 30. This detection result includes three changed regions: changed region 72, changed region 73, and changed region 74. In such a case, foreground region detection unit 121 generates detection result 71A, detection result 71B, and detection result 71C, which are separate binary images for each changed region, from detection result 71 using a general labeling method.

[0243] That is, when the detection result includes a plurality of changed regions, foreground region detection section 121 generates a plurality of binary images in which each changed region is included in a separate binary image.

[0244] Then, the foreground area detection unit 121 determines whether or not the changed area is an area in which a change related to a change in a product has been detected, based on pre-registered shelf area information and each of the multiple binary images. Here, the shelf area information indicates the area on the product shelf 3 where the products are displayed.

[0245] The self-checkout system 1 monitors products on the product shelves 3. Therefore, the area where products are displayed, which is indicated by the shelf area information, can be referred to as the monitored area. The shelf area information can also be referred to as monitored area information. The shelf area information may be, for example, an image of the same size as the captured image acquired by the first acquisition unit 110, and may be a binary image in which the pixel value of the monitored area of ​​the product shelf 3 to be monitored is represented as 255 and the rest as 0. The shelf area information may include, for example, one or more monitored areas. The shelf area information may be stored in advance in the first storage unit 130, for example. The shelf area information includes information that identifies the product shelves 3 included in the captured image acquired by the first acquisition unit 110.

[0246] The foreground area detection unit 121 performs a logical AND operation for each corresponding pixel using shelf area information related to the product shelf 3 included in the captured image acquired by the first acquisition unit 110. For example, when using shelf area information 75 illustrated in Fig. 31, the foreground area detection unit 121 performs a logical AND operation for each corresponding pixel with detection result 71A, detection result 71B, or detection result 71C. In the example shown in Fig. 31, the areas to be monitored in the shelf area information 75 are represented in white, and therefore the shelf area information 75 includes six areas to be monitored.

[0247] 32 is the result of a logical AND operation between the shelf area information 75 and the detection result 71A. Furthermore, the calculation result 76B is the result of a logical AND operation between the shelf area information 75 and the detection result 71B. Furthermore, the calculation result 76C is the result of a logical AND operation between the shelf area information 75 and the detection result 71C.

[0248] Objects other than products, such as people and carts, span multiple shelf areas. Therefore, as a result of performing a logical AND operation on detection result 71A and shelf area information 75, the part (white part) representing the changed area, which has a pixel value of 255, is divided into multiple areas, as shown in operation result 76A illustrated on the left side of Fig. 32. On the other hand, the parts (white parts) representing the changed areas of operation results 76B and 76C are unchanged from detection results 71B and 71C, respectively, and are continuous areas (a pixel with a pixel value of 255, and a collection of pixels where any of the pixels adjacent to that pixel has a pixel value of 255).

[0249] Products displayed in the display area (monitored area) of product shelf 3 do not span multiple monitored areas. Therefore, when the changed area is divided into multiple areas as in calculation result 76A, foreground area detection unit 121 determines that the change to this changed area is due to something other than the products. In such a case, foreground area detection unit 121 does not include this change in the detection result supplied to foreground area tracking unit 123. In other words, foreground area detection unit 121 supplies detection result 71B and detection result 71C to foreground area tracking unit 123.

[0250] With this configuration, the foreground region tracking unit 123 can supply changes to products displayed on the product shelf 3 to the first extraction unit 124 and the second extraction unit 125. In other words, the region change classification unit 126 can perform classification processing of changes to products displayed on the product shelf 3, thereby preventing a decrease in classification accuracy of changes to products due to the influence of objects other than the products. Furthermore, because the foreground region detection unit 121 can classify changes in the changed region as changes caused by objects other than products before the classification processing of the region change classification unit 126 is performed, the processing load of the region change classification unit 126 can be reduced.

[0251] Next, specific examples of the present invention will be described. In the first specific example, an example of the operation of the self-checkout system when a customer carries the terminal 10 will be described, and in the second specific example, an example of the operation of the self-checkout system when a customer does not carry the terminal 10 will be described.

[0252] FIG. 33 is an explanatory diagram showing an example of the operation of a self-checkout system in a first specific example. First, when customer C enters the store, person identification device 50 identifies the person via terminal 10 carried by the customer (step S11). Thereafter, the person is uniquely managed using an identification ID that identifies that terminal 10. Then, shopping list generation means (shopping list generation unit 21 in the embodiment) generates a shopping list L corresponding to the identification ID (step S12). Notification unit 23 transmits the generated shopping list L to terminal 10 (step S13). Then, notification unit 23 stores the generated shopping list L in shopping list storage unit 24 (step S14).

[0253] Thereafter, the change detection means (first change detection unit 120 in the embodiment) detects a change in the display state of the products based on the captured image of the products (step S15). The shopping list generation means (shopping list update unit 22 in the embodiment) identifies the products based on the product shelf allocation information and performs a registration process to register the identified products in the shopping list L corresponding to the person (step S16). The notification unit 23 notifies the terminal 10 of the contents of the shopping list L in which the products have been registered (step S17).

[0254] Furthermore, the rearrangement detection means (display detection unit 160 in the embodiment) detects that a product has been returned to a location different from the location from which it was taken, based on the change in the display state of the product detected by the change detection means and the person included in the captured image or the person whose movement path within the store has been detected (step S18). Any method for detecting the movement path may be used, including a method in which the above-described image capture device 2 detects a person, a method in which the image capture device 5 detects a movement path, or a method in which the movement path detection device 6 detects a movement path. In this case, based on the detection result and the shelf allocation information, the shopping list generation means (shopping list update unit 22 in the embodiment) performs a deletion process to delete the product that has been returned to a location different from the location from which it was taken from the shopping list L corresponding to the person (step S19). The notification unit 23 notifies the terminal 10 of the contents of the shopping list L from which the product has been deleted (step S20). The notification unit 23 may also notify the terminal 10 of the shopping list L in which a deletion flag has been set for the product.

[0255] When the customer is about to leave the store, the payment device 40 performs payment processing based on the contents of the shopping list L corresponding to the customer (step S21). At this time, the output device 30 may display the contents of the shopping list L.

[0256] FIG. 34 is an explanatory diagram showing an example of the operation of a self-checkout system in a second specific example. First, when a customer enters the store, the person identification device 50 identifies person C from an image captured by the imaging device 2 (step S31). Thereafter, the person is identified to their location using an identification ID that identifies the person. Then, the shopping list generation means (shopping list generation unit 21 in the embodiment) generates a shopping list L corresponding to the identification ID (step S32). Then, the notification unit 23 stores the generated shopping list L in the shopping list storage unit 24 (step S33).

[0257] Thereafter, as in the first specific example, the change detection means (first change detection unit 120 in the embodiment) detects a change in the display state of the products based on the captured images in which the products are captured (step S34). The shopping list generation means (shopping list update unit 22 in the embodiment) identifies the products based on the product shelf allocation information and performs a registration process to register the identified products in the shopping list corresponding to the person (step S35). Specifically, the shopping list generation means adds the products to the shopping list L stored in the shopping list storage unit 24.

[0258] Also, as in the first specific example, the shopping list generation means (shopping list update unit 22 in the embodiment) performs a deletion process to delete the item that was returned to a location different from where it was taken from the shopping list L corresponding to the person (step S36). Specifically, the shopping list generation means deletes the item from the shopping list L stored in the shopping list storage unit 24.

[0259] When the customer is about to leave the store, the payment device 40 performs payment processing based on the contents of the shopping list L corresponding to person C (step S37). Specifically, the output device 30 displays the contents of the shopping list and the total amount, and the payment device 40 performs payment processing based on the deposit or card payment from that person.

[0260] <Hardware configuration> In each of the above-described embodiments, the components of each device (more specifically, the image processing devices 100 to 500 and the shopping list management device 20) are represented by, for example, functional blocks. Some or all of the components of each device are realized by any combination of an information processing device 600 and a program as exemplified in FIG. 35. FIG. 35 is a block diagram showing an example of the hardware configuration of the information processing device 600 that realizes the components of each device. The information processing device 600 includes, as an example, the following configuration. ·CPU(Central Processing Unit)601 ROM (Read Only Memory) 602 ·RAM(Random Access Memory)603 Programs 604 loaded into RAM 603 A storage device 605 for storing the programs 604 A drive device 606 that reads and writes data from a recording medium 610 external to the information processing device 600 A communication interface 607 that connects the information processing device 600 to an external communication network 611 Input / output interface 608 for inputting and outputting data Bus 609 connecting each component

[0261] The components of each device in each of the above-described embodiments are realized by the CPU 601 acquiring and executing a program group 604 that realizes these functions. The program group 604 that realizes the functions of the components of each device is stored in advance in, for example, the storage device 605 or the ROM 602, and is loaded into the RAM 603 and executed by the CPU 601 as needed. The program group 604 may be supplied to the CPU 601 via the communication network 611, or may be stored in advance in the recording medium 610, and the drive device 606 may read out the program and supply it to the CPU 601.

[0262] There are various variations in the method of realizing each device. Each device may be realized, for example, by any combination of a separate information processing device 600 and a program for each component. Furthermore, multiple components included in each device may be realized by any combination of a single information processing device 600 and a program.

[0263] In addition, some or all of the components of each image processing device can be realized by other general-purpose or dedicated circuits, processors, etc., or a combination of these. These may be configured by a single chip, or by multiple chips connected via a bus.

[0264] Some or all of the components of each device may be realized by a combination of the above-mentioned circuits and programs.

[0265] When some or all of the components of each device are realized by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. may be centrally or decentralized. For example, the information processing devices, circuits, etc. may be realized as a client-server system, a cloud computing system, or the like, in a form in which each device is connected via a communication network.

[0266] Next, an outline of the present invention will be described. Fig. 36 is a block diagram showing an outline of a self-checkout system according to the present invention. A self-checkout system 800 (e.g., self-checkout system 1) according to the present invention includes a change detection means 810 (e.g., first change detection unit 120) that detects changes in the display state of products based on an image in which the products are captured; a rearrangement detection means 820 (e.g., display detection unit 160) that detects that a product has been returned to a location other than where it was taken, based on the change in the display state of the products detected by the change detection means 810 and a person included in the captured image or a person whose movement path within the store has been detected; and a shopping list generation means 830 (e.g., shopping list update unit 22) that identifies a product whose display state has been detected to have changed due to a person picking it up, based on the shelf allocation information of the product shelf on which the product is placed, performs a registration process (e.g., product registration) to register the identified product in a shopping list corresponding to the person, and performs a deletion process (e.g., deleting the product, setting a deletion flag, etc.) to delete the product that has been returned to a location other than where it was taken from the shopping list corresponding to the person, based on the detection result by the rearrangement detection means 820 (i.e., that the product has been returned to a location other than where it was taken) and the shelf allocation information.

[0267] Such an arrangement allows for proper management of the products purchased by a customer, even if the products are returned to a location different from where they were taken.

[0268] Furthermore, the self-checkout system 800 may include an associating unit (for example, the first association generating unit 140 and the association integrating unit 150) that associates a change in the display state of products detected by the change detecting unit 810 with a person included in the captured image. Then, the rearrangement detecting unit 820 may detect that a product has been returned to a location different from where it was taken, based on the result of the association made by the associating unit.

[0269] The self-checkout system 800 may also include a notification means (e.g., notification unit 23) that notifies a person corresponding to a shopping list of the status of the shopping list. Then, as a deletion process, the shopping list generation means 830 may set a deletion flag, which identifies an item that has been returned to a location different from where it was taken, to the target item included in the shopping list, and the notification means may notify the person (e.g., terminal 10 carried by the person) of the item for which the deletion flag has been set.

[0270] Specifically, the shopping list may be associated with a device carried by the person (for example, the terminal 10), and the notification means may notify the device of the products for which a deletion flag has been set.

[0271] This configuration can prevent unauthorized purchases from being made.

[0272] Furthermore, the shopping list generation means 830 may receive an instruction indicating whether or not to delete an item for which a deletion flag has been set via a device (e.g., terminal 10) carried by the person who notified, and may delete the item from the shopping list when an instruction to delete the item is received.

[0273] Furthermore, if there is a product in the shopping list for which a deletion flag is set, the shopping list generating means 830 may halt the payment process based on that shopping list.

[0274] In addition, when, for the same person, one change in display state indicates a decrease in merchandise and the other change in display state indicates an increase in merchandise, if the shape of the area of ​​change matches the change in one display state and the change in the other display state, and the areas of change indicate different positions, the rearrangement detection means 820 may detect that a merchandise has been returned to a location different from where it was taken.

[0275] In addition, the rearrangement detection means 820 may detect products of the same type that are displayed in multiple locations based on the shelf allocation information, and even if a product is returned to a location different from where it was taken, it may detect that the product has been returned to the same location if that location is a location where products of the same type are displayed.

[0276] Meanwhile, the self-checkout system 800 may also include a flow line detection means (e.g., flow line detection device 6) that detects the flow lines of people within the store. In this case, the rearrangement detection means 820 may detect that a product has been returned to a location different from where it was taken, based on a change in the display state of products detected by the change detection means 810 and the flow lines of people detected by the flow line detection means. Such a configuration can improve the accuracy of detecting that a product has been returned to a location different from where it was taken, thereby reducing sales opportunity losses and product waste due to improper product display.

[0277] Specifically, the flow path detection means may authenticate a person at a predetermined location in the store (for example, the store entrance), track the authenticated person (for example, by short-range wireless communication, etc.) to detect the person's flow path, and the shopping list generation means 830 may generate a shopping list corresponding to the authenticated person when authentication is performed. Such a configuration makes it possible to appropriately associate a person with a shopping list.

[0278] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0279] (Supplementary Note 1) A self-checkout system comprising: a change detection means for detecting a change in the display state of a product based on an image in which the product is captured; a rearrangement detection means for detecting that a product has been returned to a location different from where it was taken based on the change in the display state of the product detected by the change detection means and a person included in the image or a person whose movement path within the store has been detected; a shopping list generation means for identifying a product whose display state has been detected to have changed due to the person picking up the product based on shelf allocation information for the product shelf on which the product is placed, performing a registration process to register the identified product in a shopping list corresponding to the person, and performing a deletion process to delete a product that has been returned to a location different from where it was taken from the shopping list corresponding to the person based on the detection result by the rearrangement detection means and the shelf allocation information.

[0280] (Appendix 2) A self-checkout system as described in Appendix 1, which is provided with an association means for associating a change in the display state of a product detected by the change detection means with a person included in the captured image, and a rearrangement detection means for detecting that a product has been returned to a location different from where it was taken, based on the association results obtained by the association means.

[0281] (Appendix 3) A self-checkout system as described in Appendix 1 or Appendix 2, which includes a notification means for notifying a person corresponding to a shopping list of the status of the shopping list, wherein the shopping list generation means sets a deletion flag for the target items included in the shopping list, identifying items that have been returned to a location different from where they were taken, as a deletion process, and the notification means notifies the person of the items for which the deletion flag has been set.

[0282] (Appendix 4) A self-checkout system according to appendix 3, wherein the shopping list is associated with a device carried by the person, and the notification means notifies the device of products for which a deletion flag has been set.

[0283] (Appendix 5) A self-checkout system as described in Appendix 4, wherein the shopping list generation means receives instructions indicating whether or not to delete an item for which a deletion flag has been set via a device carried by the person who sent the notification, and when an instruction to delete the item is received, deletes the item from the shopping list.

[0284] (Appendix 6) A self-checkout system described in any one of Appendices 3 to 5, wherein the shopping list generation means stops payment processing based on the shopping list if there is an item in the shopping list that has a deletion flag set.

[0285] (Appendix 7) A self-checkout system according to any one of Appendices 1 to 6, wherein the rearrangement detection means detects that an item has been returned to a location different from where it was taken if, for the same person, one change in display state indicates a decrease in the number of items and the other change in display state indicates an increase in the number of items, and if the shapes of the areas of change match between the one change in display state and the other change in display state, and the areas of change indicate different positions.

[0286] (Appendix 8) A self-checkout system according to any one of Appendices 1 to 7, wherein the rearrangement detection means detects the same type of product that is displayed in multiple locations based on shelf allocation information, and detects that the product has been returned to the same location even if the product is returned to a location different from where it was taken, if that location is a location where the same type of product is displayed.

[0287] (Appendix 9) A self-checkout system as described in Appendix 1, which is equipped with a movement path detection means for detecting the movement paths of people within the store, and a rearrangement detection means for detecting that a product has been returned to a location different from where it was taken, based on a change in the display state of the product detected by the change detection means and the movement paths of people detected by the movement path detection means.

[0288] (Appendix 10) A self-checkout system as described in Appendix 9, wherein the movement path detection means authenticates a person at a predetermined location within the store, tracks the authenticated person to detect the person's movement path, and the shopping list generation means generates a shopping list corresponding to the authenticated person when the authentication is performed.

[0289] (Appendix 11) A purchased item management method comprising: detecting a change in the display state of an item based on an image in which the item is captured; detecting that an item has been returned to a location different from where it was taken based on the detected change in the display state of the item and a person included in the image or a person whose movement within the store has been detected; identifying an item whose display state has been detected to have changed due to the person picking up the item based on shelf allocation information for the shelf on which the item is placed; performing a registration process to register the identified item in a shopping list corresponding to the person; and performing a deletion process to delete the item that has been returned to a location different from where it was taken from the shopping list corresponding to the person based on the detection result indicating that the item has been returned to a location different from where it was taken and the shelf allocation information.

[0290] (Appendix 12) A purchased item management program for causing a computer to execute a change detection process that detects a change in the display state of an item based on an image in which the item is captured, a rearrangement detection process that detects that an item has been returned to a location different from where it was taken based on the change in the display state of the item detected by the change detection process and a person included in the captured image or a person whose movement path within the store has been detected, and a shopping list generation process that identifies an item whose display state has been detected to have changed due to the person picking it up based on shelf allocation information for the product shelf on which the item is placed, performs a registration process to register the identified item in a shopping list corresponding to the person, and performs a deletion process to delete an item that has been returned to a location different from where it was taken from the shopping list corresponding to the person based on the detection result of the rearrangement detection process and the shelf allocation information.

[0291] The programs described in the above embodiments and appendices may be stored in a storage device or a computer-readable recording medium, such as a portable medium such as a flexible disk, an optical disk, a magneto-optical disk, or a semiconductor memory. [Explanation of symbols]

[0292] 1,4,4a Self-checkout system 2,5 Imaging device 3 Product shelves 6. Traffic flow detection device 10 devices 20 Shopping list management device 30 Output Devices 40 Payment Device 50 Personal Identification Device 71 Detection Results 72 Change Area 73 Change Area 74 Change Area 75 Shelf area information 76 Operation result 90 Classification results 91 Second featured image 92 First featured image 93 Types of Change 100 Image processing device 110 First acquisition part 120 First change detection unit 121 Foreground region detection unit 122 Background information update section 123 Foreground Region Tracking Unit 124 1st extraction part 125 Second extraction part 126 Area Change Classification Unit 130 1st memory section 131 Background information 132 Shelf change model 133 Foreground information 134 Person information 140 First relation generation unit 150 Relevance Integration Department 160 Display detection unit 200 Image processing device 210 Second Acquisition Department 220 Second change detection unit 230 2nd memory section 240 Second Relevance Generation Unit 250 Relevance Integration Department 300,300a Image processing device 310,310a 3rd acquisition part 320, 320a Flow line data generation unit 340 First Relevance Generation Unit 400 Image Processing Device 440 Second Relevance Generation Unit 500 Image Processing Device 510 Information Department 600 Information Processing Devices 601 CPU 602 ROM 603 RAM 604 Programs 605 Storage device 606 Drive unit 607 Communication Interface 608 Input / Output Interface 609 Bus 610 Recording Media 611 Communication Network

Claims

1. a change detection means for detecting a change area of ​​a product shelf on which the product is placed based on a captured image of the product and a background image; a tracking means for extracting an area where the amount of movement of the changed area is equal to or greater than a predetermined threshold as a person area; an association generating means for extracting people who intersect with the change area from people whose images were captured at a time prior to the image capturing time at which the change area was detected, based on the person area, and generating association information that associates the change area with people whose images were captured at a time closest to the image capturing time; a classification means for classifying a change from a state in a second image of interest to a state in a first image of interest, the second image of interest being an image obtained by extracting the changed region from the captured image, and a second image of interest being an image obtained by extracting the changed region from a background image; an output means for outputting the classified change details and the relevance information together with the first image of interest and the second image of interest; The classified changes are changes due to the product being taken away, changes in the product shelf environment, changes due to the product being placed on the product shelf, or changes due to a change in the appearance of the product; The change due to the change in appearance of the product includes a change in appearance due to the placement of a different product and a change in appearance due to a change in the position of the product. A system characterized by:

2. The output means outputs the image capture time together with the classified change content. The system of claim 1 .

3. The output means outputs the person's identifier together with the classified change content. The system according to claim 1 or claim 2.

4. The output means outputs information indicating the position of the changed area in the first target image together with the content of the classified change. A system according to any one of claims 1 to 3.

5. The computer detects a change area of ​​the product shelf on which the product is placed based on the captured image of the product and the background image, The computer extracts, as a person region, a region in which the amount of movement of the changed region is equal to or greater than a predetermined threshold; the computer extracts, based on the person area, people who intersect with the change area from people whose images were captured at a time before the image capture time at which the change area was detected, and generates association information that associates the change area with people whose images were captured at a time closest to the image capture time; the computer classifies a change from the second image of interest to a state in the first image of interest, based on a first image of interest that is an image obtained by extracting the changed region from the captured image and a second image of interest that is an image obtained by extracting the changed region from a background image; the computer outputs the classified change details and the relevance information together with the first image of interest and the second image of interest; The classified changes are changes due to the product being taken away, changes in the product shelf environment, changes due to the product being placed on the product shelf, or changes due to a change in the appearance of the product; The change due to the change in appearance of the product includes a change in appearance due to the placement of a different product and a change in appearance due to a change in the position of the product. A method characterized by:

6. On the computer, a change detection process for detecting a change area of ​​a product shelf on which the product is placed based on a captured image of the product and a background image; a tracking process for extracting, as a person region, a region in which the amount of movement of the change region is equal to or greater than a predetermined threshold; an association generation process for extracting people who intersect with the change area from people whose images were captured at a time before the image capture time at which the change area was detected based on the person area, and generating association information that associates the change area with people whose images were captured at a time closest to the image capture time; a classification process for classifying a change from a state in a first image of interest, which is an image obtained by extracting the changed region from the captured image, to a state in a second image of interest, which is an image obtained by extracting the changed region from a background image, based on the first image of interest, and executing an output process for outputting the classified change details and the relevance information together with the first target image and the second target image; The classified changes are changes due to the product being taken away, changes in the product shelf environment, changes due to the product being placed on the product shelf, or changes due to a change in the appearance of the product; The change due to the change in appearance of the product includes a change in appearance due to the placement of a different product and a change in appearance due to a change in the position of the product. A program characterized by:

Images