Commodity Quantity Specifying Device, Commodity Quantity Specifying Method, and Program

Abstract

This product quantity determination device (10) comprises an acquisition unit (110), an image processing unit (120), and a calculation unit (130). The acquisition unit (110) acquires a plurality of images. The imaging range of these images includes a subject region, which is a region in which products can be arranged. The image processing unit (120) carries out a product detection process for each of the plurality of images. The calculation unit (130) determines a first product, which is a product that has been detected from no fewer than a prescribed number of the images in the detection process, and sets, as a quantity of products for which a settlement is to be carried out, a quantity obtained by excluding the first product from the products detected by the detection process.

Description

Technical Field

[0001] The present invention relates to an article quantity specifying device, an article quantity specifying method, and Program the like.

Background Art

[0002] In recent years, when settling accounts for goods, it has been considered to acquire and process an image of the goods and utilize the processing result. For example, Patent Document 1 describes a recognition system that recognizes a target article using image processing. This recognition system includes a first detection means, an extraction means, a calculation means, a recognition means, and a selection means. The first detection means detects an article included in the image data captured by the imaging unit. The extraction means extracts a feature amount of the article detected by the first detection means from the image data. The calculation means calculates the similarity between the feature amount for verification stored in advance for each article with identification information and the article without the identification information, and the feature amount extracted by the extraction means. The recognition means recognizes the article detected by the first detection means based on this similarity. The selection means selects the article as the article captured by the imaging unit on the condition that the article recognized by the recognition means is an article without identification information.

[0003] Note that Patent Document 2 describes that by registering the feature amounts of general exclusion articles such as hands and arms in a dictionary file, the occurrence of misrecognition in which a hand is recognized as a good can be reduced.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] The inventor considered determining the possibility that products were not accurately registered by comparing the number of products registered in the product registration device with the number of products included in an image obtained by photographing the area including the product registration device. However, when performing this process, it is necessary to accurately detect the number of products included in the image. However, with the recognition system described in Patent Document 1 mentioned above, there is a possibility that the number of products cannot be accurately detected.

[0006] An example of the object of the present invention is, in view of the above problems, to provide a product number specifying device, a product number specifying method, and Program which can accurately detect the number of products included in an image.

Means for Solving the Problems

[0007] According to one aspect of the present invention, there is provided an acquisition means for acquiring a plurality of images including a target area, which is an area where products can be arranged, within a photographing range, image processing means for performing product detection processing on each of the plurality of images, calculation means for specifying a first object, which is a product detected from a predetermined number or more of the images in the detection processing, and setting the number of products excluding the first object among the products detected by the detection processing as a first product number, which is the number of products to be tallied, and a product number specifying device including the above is provided.

[0008] According to one aspect of the present invention, a computer acquires a plurality of images including a target area, which is an area where products can be arranged, within a photographing range, performs product detection processing on each of the plurality of images, specifies a first object, which is a product detected from a predetermined number or more of the images in the detection processing, and sets the number of products excluding the first object among the products detected by the detection processing as a first product number, which is the number of products to be tallied, and a product number specifying method is provided.

[0009] According to one aspect of the present invention, a computer An acquisition function that acquires a plurality of images including a target area, which is an area where a product can be placed, within a shooting range; An image processing function that performs product detection processing on each of the plurality of images; A calculation function that identifies a first object, which is a product detected from a predetermined number or more of the images in the detection processing, and sets the number obtained by excluding the first object from the products detected by the detection processing as a first product number, which is the number of products to be settled; And a program having the above. is Is provided.

Effect of the Invention

[0010] According to one aspect of the present invention, a product number specifying device, a product number specifying method, and Program Can be provided.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0013] FIG. 1 is a diagram showing an overview of a product number specifying device 10 according to an embodiment. The product number specifying device 10 includes an acquisition unit 110, an image processing unit 120, and a calculation unit 130. The acquisition unit 110 acquires a plurality of images. These images include a target area, which is an area where products can be arranged, within the shooting range. The image processing unit 120 performs product detection processing on each of the plurality of images. The calculation unit 130 identifies a first object, which is a product detected from a predetermined number or more of images, and sets the number of products excluding the first object among the products detected by the detection processing as the number of products to be settled. Hereinafter, this number of products will be referred to as the first product number.

[0014] Since the first object is a product detected from a predetermined number or more of images, it is highly likely that it is not a product to be settled, such as a human hand or an accessory device of the product number specifying device 10. Therefore, the calculation unit 130 sets the number of products excluding the first object among the products detected by the detection processing as the number of products to be settled. Accordingly, according to the product number specifying device 10, the number of products included in the image can be accurately detected.

[0015] Hereinafter, a detailed example of the product number specifying device 10 will be described.

[0016] FIG. 2 is a diagram for explaining the usage environment of the product number specifying device 10. In the example shown in this figure, the product number specifying device 10 is used as a device for registering and settling products. However, as will be described later, the product number specifying device 10 may be a device different from the device for registering and settling products, for example, a cloud server.

[0017] The merchandise quantity specifying device 10 is installed in a store or an office. When the merchandise quantity specifying device 10 is installed in a store, a customer who enters the store operates the merchandise quantity specifying device 10 when purchasing merchandise. On the other hand, when the merchandise quantity specifying device 10 is installed in an office, merchandise is displayed in a corner of the office. Then, a person working in the office operates the merchandise quantity specifying device 10 when purchasing this merchandise.

[0018] The merchandise quantity specifying device 10 is used together with a reading device 20, a photographing device 30, and a storage unit 40. In the example shown in this figure, the merchandise quantity specifying device 10 is used when a customer purchases merchandise and has a merchandise registration function and a settlement function. That is, the merchandise quantity specifying device 10 also functions as a merchandise registration device. Note that the merchandise quantity specifying device 10 does not necessarily have to have a settlement function. In this case, the merchandise quantity specifying device 10 transmits information indicating the registered merchandise to a settlement device.

[0019] The reading device 20 acquires merchandise identification information from the merchandise that a customer is about to purchase, that is, the merchandise to be settled. The reading device 20 may acquire the merchandise identification information by reading a code attached to the merchandise, such as a barcode or a two-dimensional code, or may acquire the merchandise identification information from a wireless communication tag attached to the merchandise, such as an RFID tag. The reading device 20 transmits the acquired merchandise identification information to the merchandise quantity specifying device 10. At this time, the reading device 20 also transmits information indicating the acquisition timing of the merchandise identification information, such as acquisition date and time information, to the merchandise quantity specifying device 10. Hereinafter, this information will be referred to as first timing information. The first timing information may be generated by the merchandise quantity specifying device 10. In this case, the merchandise quantity specifying device 10 generates the first timing information so as to indicate the timing at which the merchandise identification information was acquired from the reading device 20.

[0020] Note that the reading device 20 may be integrated with the merchandise quantity specifying device 10.

[0021] When the reading device 20 acquires product identification information from a product, the imaging device 30 captures an image of the product and generates a first image. The imaging device 30 may always generate an image, or may generate an image triggered by the reading device 20 acquiring product identification information. In the former case, the frame rate of the imaging device 30 is, for example, 1 pfs or more and 30 pfs or less, but is not limited thereto.

[0022] Note that the imaging device 30 may be attached to the product quantity specifying device 10, or may be attached above the product quantity specifying device 10, for example, to the ceiling.

[0023] The imaging range of the imaging device 30 includes the area where the product can be placed when the reading device 20 acquires product identification information, that is, the product reading area. The imaging range of the imaging device 30 may include the reading device 20. Also, the imaging range may include the area around the product reading area. In this case, it is preferable that the imaging range includes at least one of the area through which the product passes when the product moves toward the product reading area and the area through which the product passes when the product moves out of the product reading area. Then, the imaging device 30 transmits the generated image to the product quantity specifying device 10. At this time, the imaging device 30 also transmits information indicating the generation timing of the image, for example, generation date and time information, to the product quantity specifying device 10. Hereinafter, this information will be referred to as second timing information.

[0024] The storage unit 40 stores product identification information and prices for a plurality of products. The product quantity specifying device 10 uses the information stored in the storage unit 40 when performing the settlement process of the product. The storage unit 40 may be, for example, a server installed in the store where the product quantity specifying device 10 is arranged.

[0025] In the above description, the product quantity specifying device 10 is used when a product is purchased. However, the product quantity specifying device 10 may be used when a store clerk arranges products on a product shelf.

[0026] Note that the product quantity specifying device 10 does not necessarily perform the registration process and the settlement process for the target product. In this case, the product quantity specifying device 10 is a device different from a terminal that performs the registration process and the settlement process, such as a cloud server, for example, a POS terminal. And the reading device 20 communicates with this terminal. The product quantity specifying device 10 acquires, from this terminal, information indicating the number of products registered in the terminal, that is, the second product quantity described later.

[0027] FIG. 3 is a diagram for explaining a first example of the shooting range by the shooting device 30. In the example shown in this figure, the product quantity specifying device 10 is used when a product is purchased and is placed on the stand 50. The stand 50 is sufficiently larger than the product quantity specifying device 10, and a part thereof is the product display area 510.

[0028] As described with reference to FIG. 2, the shooting range of the shooting device 30 includes an area where products can be arranged. The area where products can be arranged includes at least one of an area where products are temporarily arranged and an area where products are arranged when the product identification information of the products is registered in the product quantity specifying device 10. An example of the former is an area on the stand 50 where products are temporarily placed. An example of the latter is an area (space) where the reading device 20 that reads the product identification information from the product can read the product identification information.

[0029] In the example shown in this figure, the shooting range of the shooting device 30 may further include the product quantity specifying device 10 and the product display area 510. Note that products that can be registered in the product quantity specifying device 10 may be displayed in places other than the product display area 510.

[0030] On the platform 50, at least one of the accessory devices 60 of the product quantity specifying device 10, for example, the reading device 20 shown in FIG. 2, a short-range wireless communication device that communicates with a mobile terminal, and a receipt printing device, is often arranged. Also, around the platform 50, an object 70 such as a trash can may be arranged. The imaging range includes the area where products are arranged when the reading device 20 reads product identification information, for example, includes the front surface of the reading device 20. Therefore, the products whose product identification information has been read by the reading device 20 are included in the images generated by the imaging device 30.

[0031] The imaging range of the imaging device 30 may include the accessory device 60 and the object 70. In this case, these accessory device 60 and object 70 may be misrecognized as products to be settled. However, since the accessory device 60 and the object 70 hardly move, they are included in a plurality of images generated by the imaging device 30. Therefore, in the product quantity specifying device 10, the accessory device 60 and the object 70 are recognized as the first object described above.

[0032] FIG. 4 is a diagram showing a modified example of FIG. 3. In the example shown in this figure, the imaging range includes the product display area 510 and the accessory device 60, but does not include the product quantity specifying device 10. Thus, the imaging range of the imaging device 30 only needs to include the area where products can be arranged.

[0033] FIG. 5 is a diagram for explaining a second example of the imaging range by the imaging device 30. In the example shown in this figure, the product quantity specifying device 10 is used when a store clerk takes out a product 92 from a container 90, for example, a foldable container or a cardboard box, and arranges it on the display shelf 80. In this case, the product quantity specifying device 10 specifies the number of products arranged on the display shelf 80 by the store clerk. In this example, the reading device 20 is a portable device operated by a store clerk. However, the reading device 20 may not be used.

[0034] In this example, the shooting range by the imaging device 30 includes at least one, preferably both, of the display shelf 80 and the area in front of the display shelf 80, that is, the location where the container 90 is arranged. Therefore, the product 92 displayed on the display shelf 80 is included in the image generated by the imaging device 30. The imaging device 30 is a surveillance camera arranged in the store, but it may be other cameras.

[0035] FIG. 6 is a diagram showing an example of the functional configuration of the product quantity specifying device 10. As described with reference to FIG. 1, the product quantity specifying device 10 includes an acquisition unit 110, an image processing unit 120, and a calculation unit 130. In the example shown in this figure, the product quantity specifying device 10 further includes an execution unit 140, a product registration unit 150, and a settlement unit 160. When the product quantity specifying device 10 corresponds to the example shown in FIG. 5, the product quantity specifying device 10 does not include the product registration unit 150 and the settlement unit 160.

[0036] The acquisition unit 110 acquires the image generated by the imaging device 30. At this time, the acquisition unit 110 also acquires the second timing information.

[0037] The image processing unit 120 performs a product detection process on each of the plurality of images generated by the imaging device 30. The image processing unit 120 may perform the product detection process using a model generated by machine learning, or may perform the product detection process by feature matching. The information required when the image processing unit 120 performs the product detection process is stored in the storage unit 40, for example.

[0038] Then, the image processing unit 120 calculates the total number of products to be settled using the results of these detection processes. Hereinafter, this total number of products is referred to as the first total number. At this time, the image processing unit 120 may track the products among the plurality of images and specify the first total number using the tracking results. Also, the image processing unit 120 may specify the number of products for each type of product. Hereinafter, these numbers for each type are referred to as the first individual numbers.

[0039] When the imaging device 30 generates an image triggered by the reading device 20 having acquired product identification information, the image acquired by the acquisition unit 110 is an image generated when the reading device 20 reads the product identification information. Then, it is preferable that the image processing unit 120 performs a product detection process on all of the plurality of images acquired by the acquisition unit 110.

[0040] On the other hand, when the imaging device 30 is always operating, the image processing unit 120 may perform a product detection process on all images, or may select an image that satisfies a predetermined condition as an object of the detection process. Examples of the "predetermined condition" are at least one of the following (1) to (4), for example.

[0041] (1) An image having second timing information closest to the first timing information. This example corresponds to the state shown in FIG. 3 or FIG. 4. The image selected here includes the product corresponding to the product identification information acquired by the acquisition unit 110. Here, when the first timing information and the second timing information indicate the same timing, when the first timing information is earlier, and when the second timing information is earlier, three cases are conceivable. In any case, the difference between the first timing information and the second timing information is, for example, 1 second or less. When the reading device 20 reads product identification information from each of a plurality of products, the acquisition unit 110 performs the above-described image selection process for each product identification information.

[0042] (2) The person shown in the image is using the reading device 20. This example also corresponds to the state shown in FIG. 3 or FIG. 4. The image processing unit 120 performs a person detection process on the image. Then, when the image processing unit 120 determines that a person can be detected and that person is causing the reading device 20 to read product identification information, the image processing unit 120 performs a product detection process on that image.

[0043] (3) An image generated while a product is registered in the product number specifying device 10. This example also corresponds to the state shown in FIG. 3 or FIG. 4. And the period during which products are registered in the product quantity specifying device 10 is, for example, the period from when information indicating the start of product registration is input to the product quantity specifying device 10 until information indicating the progress to the settlement process is input to the product quantity specifying device 10 in the product quantity specifying device 10. The image processing unit 120 specifies an image generated during the period when products are registered using the second timing information.

[0044] In addition, in this (3), the image processing unit 120 may further add at least one image generated before a product is registered in the product quantity specifying device 10 to the images to be processed. For example, the image processing unit 120 specifies the time when product registration starts. And the images generated from this time until a predetermined time before this time, for example, until 10 seconds before this time, are added to the images to be processed. The reason for this is that there is a high possibility that the first object is captured in these images.

[0045] (4) It is an image generated when a store clerk is performing an operation to arrange products on a display shelf This example corresponds to the state shown in FIG. 5. The image processing unit 120 performs a detection process of the store clerk on the image. And when the image processing unit 120 can detect the store clerk and determines that the store clerk is arranging product 92 on display shelf 80, the image processing unit 120 performs a product detection process on the image.

[0046] The product registration unit 150 acquires product identification information from the reading device 20. At this time, the product registration unit 150 also acquires the first timing information. The product registration unit 150 generates registration information indicating products to be settled. The registration information includes a list of product identification information of products to be settled, information indicating the total number of products to be settled, and information indicating the number of each product by product. Hereinafter, the total number of products based on the registration information is referred to as the second product quantity. Also, the number of each product by product based on the registration information is referred to as the second individual quantity.

[0047] As described with reference to FIG. 1, the calculation unit 130 identifies the first object, i.e., the product detected from a predetermined number or more of images in the detection process. The predetermined number is, for example, 3 or more, preferably 5 or more, but is not limited to these values. Then, the calculation unit 130 sets the number obtained by subtracting the number of the first objects from the first total number calculated by the image processing unit 120 as the number of the first products, i.e., the number of products to be settled.

[0048] Here, when a plurality of products are detected from at least one image, the calculation unit 130 may identify the first object by determining whether each of the plurality of products has been detected from other images. When a plurality of products are detected from one image, at least one of the plurality of products may be an object other than the product. In particular, when the imaging range by the imaging device 30 is relatively narrow and is, for example, 1 to 2 times larger than the product reading area by the reading device 20, this possibility is high. Therefore, the calculation unit 130 stores the feature amounts of the detected plurality of products in the storage unit 40, and sets each of the products having these feature amounts as a candidate for the first object. Then, the calculation unit 130 identifies the number of other images including these candidates for the first object, and uses this number to determine whether the candidate for the first object is actually the first object.

[0049] For example, when this number is large, it means that the candidate frequently enters the imaging range of the imaging device 30. In this case, this candidate is likely not a product that the customer actually intends to purchase, for example, a product or device placed around. Therefore, when the number described above, i.e., the number of other images including the candidate for the first object, is equal to or greater than the reference value, the calculation unit 130 determines that this candidate is an object other than the product, i.e., the first object. The reference value is, for example, 2, but may be 3 or more, or may be 1.

[0050] Note that the image processing unit 120 may store the feature amounts of the products detected from the plurality of images in the storage unit 40 for each of the plurality of images. In this case, the calculation unit 130 attaches a flag indicating that to the feature amount corresponding to the candidate for the first object.

[0051] Also, when the image processing unit 120 calculates the number of each type of product, that is, the first number of individuals, the calculation unit 130 may perform at least one of the processes shown in (A) and (B) below.

[0052] (A) The calculation unit 130 treats a product having a first number of individuals greater than the maximum value of the second number of individuals as a first object and calculates the first number of products. As described above, the second number of individuals is calculated based on the product identification information registered in the product registration unit 150 and indicates the number of each product. Here, if there is a product having a first number of individuals greater than the maximum value of the second number of individuals, that product is likely not actually a product. Specifically, the product having that first number of individuals is likely to be a human hand or the accessory device 60 or object 70 shown in FIG. 3. Therefore, when calculating the number of products, the calculation unit 130 treats the product having this first number of individuals as a first object, that is, an object different from a product.

[0053] For example, assume that the image processing unit 120 determines that "there are 2 products A", "there are 3 products B", and "there is 1 product C", and in the registration information generated by the product registration unit 150, "there are 2 rice balls" and "there is 1 bottled tea". In this case, the maximum value of the second number of individuals is "2", which is the number of rice balls. On the other hand, "product B" detected by the image processing unit 120 is 3, which is greater than the maximum value of the second number of individuals. In this case, the calculation unit 130 treats "product B" as an object different from a product and subtracts "3", which is the number of "product B", from the first total number.

[0054] (B) In any of the second numbers of individuals, when the type of the first product having the second number of individuals is less than the type of the second product having the same second number of individuals as the first number of individuals, the calculation unit 130 subtracts the value obtained by multiplying the difference in the number of types between the first product type and the second product type by the second number of individuals from the number of products detected by the detection process, that is, the first total number. For example, the image processing unit 120 determines that "there are 2 items of product A", "there are 2 items of product B", "there is 1 item of product C", and "there is 1 item of product D". Also, assume that in the registration information generated by the product registration unit 150, "there are 2 rice balls", "there is 1 piece of bread", and "there is 1 plastic bottle of tea". In this case, although there is 1 type of product with a quantity of 2, in the processing result of the image processing unit 120, there are 2 types of products with a quantity of 2. In this case, one of these 2 types of products is likely not actually a product. Therefore, the calculation unit 130 subtracts the value obtained by multiplying "1", which is "the difference between the number of the first product type and the number of the second product type", by the second quantity "2", that is, "2", from the first total quantity.

[0055] When the difference between the second product quantity, which is the total quantity of products registered by the product registration unit 150, and the first product quantity calculated by the calculation unit 130 is equal to or greater than a reference value, the execution unit 140 executes a predetermined process. This reference value is, for example, 1, but it may also be 2 or more. The fact that this difference is equal to or greater than the reference value indicates that there may be a difference between the total quantity of products registered in the product registration unit 150 and the total quantity of products that a customer or a person working in an office intends to purchase. And an example of the predetermined process is the output process of warning information. The execution unit 140 may output the warning information to a display or a speaker of the product quantity identification device 10, or may output the warning information to a terminal operated by an administrator who manages the sale of products, for example, a terminal operated by a store clerk.

[0056] When the execution unit 140 does not perform the predetermined process, that is, when it is highly likely that the total quantity of products registered by the product registration unit 150 and the total quantity of products that a customer or a person working in an office intends to purchase match, the settlement unit 160 performs a settlement process using the registration information generated by the product registration unit 150, that is, the list of product identification information of the products to be settled. At this time, the settlement unit 160 uses the information stored in the storage unit 40.

[0057] Incidentally, when the product quantity specifying device 10 is used as a device separate from a device such as a cloud server that performs product registration and settlement, the product quantity specifying device 10 does not have a product registration unit 150 and a settlement unit 160. In this case, separately from the product quantity specifying device 10, a device having a product registration unit 150 and a settlement unit 160, for example, a POS terminal, is arranged at the position of the product quantity specifying device 10 in FIG. 3 or FIG. 4, for example. Hereinafter, this device will be referred to as a registration and settlement device. Then, the product quantity specifying device 10 acquires information indicating the second quantity of individuals and the second quantity of products from this registration and settlement device. The execution unit 140 of the product quantity specifying device 10 transmits information indicating whether the difference between the second quantity of products and the first quantity of products is equal to or greater than a reference value to the registration and settlement device. When the difference between the second quantity of products and the first quantity of products is equal to or greater than the reference value, the registration and settlement device causes predetermined information to be displayed on the display of the device or output from the speaker of the device.

[0058] FIG. 7 is a diagram showing an example of the hardware configuration of the product quantity specifying device 10. The product quantity specifying device 10 includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0059] The bus 1010 is a data transmission path for the processor 1020, the memory 1030, the storage device 1040, the input / output interface 1050, and the network interface 1060 to transmit and receive data to and from each other. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0060] The processor 1020 is a processor realized by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0061] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0062] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a removable medium such as a memory card, or a read only memory (ROM), etc., and has a recording medium. The recording medium of the storage device 1040 stores program modules that implement the respective functions of the product quantity specifying device 10 (for example, the acquisition unit 110, the image processing unit 120, the calculation unit 130, the execution unit 140, the product registration unit 150, and the settlement unit 160). When the processor 1020 reads and executes these program modules onto the memory 1030, the respective functions corresponding to the program modules are realized. Also, the storage device 1040 may function as the storage unit 40.

[0063] The input / output interface 1050 is an interface for connecting the product quantity specifying device 10 and various input / output devices. For example, the product quantity specifying device 10 communicates with at least one of the reading device 20, the photographing device 30, and the storage unit 40 via the input / output interface 1050.

[0064] The network interface 1060 is an interface for connecting the product quantity specifying device 10 to a network. This network is, for example, a local area network (LAN) or a wide area network (WAN). The method by which the network interface 1060 connects to the network may be a wireless connection or a wired connection. The product quantity specifying device 10 may communicate with at least one of the photographing device 30 and the storage unit 40 via the network interface 1060.

[0065] FIG. 8 is a flowchart showing a first example of the processing performed by the product quantity specifying device 10. This figure corresponds to the example shown in FIG. 3.

[0066] When a customer or an office worker purchases a product, they cause a reading device 20 to read the product identification information of the product. The product registration unit 150 of the product quantity determination device 10 acquires this product identification information. When there are a plurality of products to be purchased, the product registration unit 150 acquires the product identification information of each of the plurality of products. Then, the product registration unit 150 calculates the second product quantity (step S10).

[0067] Also, the acquisition unit 110 acquires a plurality of images generated by the imaging device 30. Then, the image processing unit 120 processes these plurality of images to detect the products included in each of the plurality of images (step S20). A specific example of the processing performed here is as described with reference to FIG. 6.

[0068] Then, the calculation unit 130 calculates the first product quantity using the processing result of the image processing unit 120 (step S30). A specific example of the processing performed here is as described with reference to FIG. 6.

[0069] Then, when the difference between the second product quantity and the first product quantity is equal to or greater than a reference value (step S40: Yes), the execution unit 140 executes a predetermined process (step S50). An example of the predetermined process is a warning process. Thereafter, the product quantity determination device 10 returns to step S10.

[0070] On the other hand, when the difference between the second product quantity and the first product quantity is less than the reference value (step S40: No), the settlement unit 160 performs a settlement process (step S60).

[0071] FIG. 9 is a flowchart showing a second example of the processing performed by the product quantity determination device 10. This figure corresponds to the example shown in FIG. 5.

[0072] The store clerk moves the container 90 close to the display shelf 80 and starts stock replenishment (step S12). Specifically, the store clerk takes out the product 92 from the container 90 and displays it on the display shelf 80. The imaging device 30 repeatedly generates images while the store clerk is performing stock replenishment. The acquisition unit 110 acquires these images.

[0073] When the item picking is completed (step S22), the image processing unit 120 processes the image acquired by the acquisition unit 110 and detects the products included in each of the plurality of images (step S30). A specific example of the processing performed here is as described with reference to FIG. 6.

[0074] Then, the calculation unit 130 calculates the number of first products using the processing result of the image processing unit 120 (step S40). A specific example of the processing performed here is as described with reference to FIG. 6.

[0075] As described above, according to the present embodiment, the image processing unit 120 detects products by processing an image. Then, the calculation unit 130 identifies a first object, which is a product detected from a predetermined number or more of images in the detection process performed by the image processing unit 120, i.e., a first object. This first object is highly likely not to be a product, such as a human hand or an accessory device 60. Therefore, the calculation unit 130 sets the number obtained by excluding the first object from the products detected by the detection process as the number of first products that are the products to be settled. Therefore, when using the product number specifying device 10, the number of products included in the image can be accurately detected.

[0076] As described above, embodiments of the present invention have been described with reference to the drawings, but these are examples of the present invention, and various configurations other than the above can also be adopted.

[0077] Also, in the plurality of flowcharts used in the above description, a plurality of steps (processes) are described in order, but the execution order of the steps executed in each embodiment is not limited to the described order. In each embodiment, the order of the illustrated steps can be changed within a range that does not substantially affect the content. Also, at least one step may be performed by another operating entity, for example, another device or a person. Further, the above-described embodiments can be combined within a range where the contents do not conflict.

[0078] Some or all of the above embodiments can also be described as follows in the appended claims, but are not limited thereto. 1. An acquisition means for acquiring a plurality of images including a target area, which is an area where a product can be placed, within a shooting range; An image processing means for performing a detection process of the product on each of the plurality of images; A calculation means for identifying a first object, which is a product detected from a predetermined number or more of the images in the detection process, and setting the number obtained by excluding the first object from the products detected by the detection process as a first product number, which is the number of products to be settled; A product number specifying device comprising the above. 2. In the product number specifying device according to the above 1, The image processing means performs the detection process on the images that satisfy a predetermined condition, the product number specifying device. 3. In the product number specifying device according to the above 2, The predetermined condition is that a person shown in the image is using a reading device for reading product identification information, the product number specifying device. 4. In the product number specifying device according to the above 2, The predetermined condition is that the image was generated when a store clerk was performing an operation for arranging the product on a display shelf, the product number specifying device. 5. In the product number specifying device according to any one of the above 1 to 4, The shooting range includes at least one of a product display shelf and an area in front of the display shelf, the product number specifying device. 6. In the product number specifying device according to any one of the above 1 to 4, The shooting range includes an area where the product is placed when registering the product in a product registration device, the product number specifying device. 7. In the product number specifying device according to the above 6, The image processing means performs the detection process on the plurality of images generated while the product is being registered in the product registration device, the product number specifying device. 8. In the product number specifying device according to the above 7, The image processing means further performs the detection process on at least one of the images generated before the product is registered in the product registration device, the product number specifying device. 9. In the merchandise quantity specifying device according to 7 or 8 above, a merchandise quantity specifying device comprising execution means for executing predetermined processing when a difference between a second merchandise quantity, which is the quantity of the merchandise registered in the merchandise registration device, and the first merchandise quantity is equal to or greater than a reference value. 10. In the merchandise quantity specifying device according to any one of 7 to 9 above, the image processing means calculates a first individual quantity, which is the quantity of the merchandise, for each type of the merchandise, the merchandise registration device calculates a second individual quantity, which is the quantity of the merchandise, for each type of the merchandise, the calculating means calculates the first merchandise quantity with the merchandise having the first individual quantity greater than the maximum value of the second individual quantity as the first object. 11. In the merchandise quantity specifying device according to any one of 7 to 9 above, the image processing means calculates a first individual quantity, which is the quantity of the merchandise, for each type of the merchandise, the merchandise registration device calculates a second individual quantity, which is the quantity of the merchandise, for each type of the merchandise, when the calculating means calculates the first merchandise quantity, if, in any of the second individual quantities, the type of the first merchandise having the second individual quantity is less than the type of the second merchandise having the first individual quantity equal to the second individual quantity, a value obtained by multiplying the difference in the number of types between the type of the first merchandise and the type of the second merchandise by the second individual quantity is subtracted from the quantity of the merchandise detected by the detection processing. 12. In the merchandise quantity specifying device according to any one of 1 to 11 above, when a plurality of merchandise are detected from at least one of the images, the calculating means identifies the first object by determining whether each of the plurality of merchandise is detected from other images. 13. A computer acquires a plurality of images including a target area, which is an area where merchandise can be arranged, in a shooting range, performs merchandise detection processing on each of the plurality of images, A method for specifying the number of items, which specifies a first object that is an item detected from a predetermined number or more of the images in the detection process, and sets the number obtained by excluding the first object from the items detected by the detection process as a first item number that is the number of items to be settled. 14. In the method for specifying the number of items according to item 13 above, The computer performs the detection process on the images that satisfy predetermined conditions, the method for specifying the number of items. 15. In the method for specifying the number of items according to item 14 above, The predetermined condition is that a person shown in the image is using a reading device for reading item identification information, the method for specifying the number of items. 16. In the method for specifying the number of items according to item 14 above, The predetermined condition is that the image is generated when a store clerk is performing work for arranging the items on a display shelf, the method for specifying the number of items. 17. In the method for specifying the number of items according to any one of items 13 to 16 above, The imaging range includes at least one of a display shelf for items and an area in front of the display shelf, the method for specifying the number of items. 18. In the method for specifying the number of items according to any one of items 13 to 16 above, The imaging range includes an area where the item is placed when registering the item in an item registration device, the method for specifying the number of items. 19. In the method for specifying the number of items according to item 18 above, The computer performs the detection process on the plurality of images generated while the item is being registered in the item registration device, the method for specifying the number of items. 20. In the method for specifying the number of items according to item 19 above, The computer further performs the detection process on at least one of the images generated before the item is registered in the item registration device, the method for specifying the number of items. 21. In the method for specifying the number of items according to item 19 or 20 above, A method for specifying the number of products, wherein the computer executes a predetermined process when the difference between a second number of products, which is the number of the products registered in the product registration device, and the first number of products is equal to or greater than a reference value. 22. In the method for specifying the number of products according to any one of the above items 19 to 21, the computer calculates a first number of entities, which is the number of the products, for each type of the products, the product registration device calculates a second number of entities, which is the number of the products, for each type of the products, A method for specifying the number of products, wherein the computer calculates the first number of products by using, as a first object, the product having the first number of entities greater than the maximum value of the second number of entities. 23. In the method for specifying the number of products according to any one of the above items 19 to 21, the computer calculates a first number of entities, which is the number of the products, for each type of the products, the product registration device calculates a second number of entities, which is the number of the products, for each type of the products, When the computer calculates the first number of products, in any of the second numbers of entities, if the type of the first product having the second number of entities is less than the type of the second product having the same first number of entities as the second number of entities, a value obtained by multiplying the difference between the number of types of the first product and the number of types of the second product by the second number of entities is subtracted from the number of products detected by the detection process. A method for specifying the number of products. 24. In the method for specifying the number of products according to any one of the above items 13 to 23, A method for specifying the number of products, wherein when a plurality of products are detected from at least one of the images, the computer identifies the first object by determining whether each of the plurality of products is detected from the other images. 25. A computer An acquisition function for acquiring a plurality of images including a target area, which is an area where products can be arranged, in a shooting range, An image processing function for performing a detection process of products on each of the plurality of images, A calculation function that identifies a first object, which is a product detected from a predetermined number or more of the images in the detection process, and sets the number obtained by excluding the first object from the products detected by the detection process as a first product number, which is the number of products to be settled; A computer-readable recording medium storing a program having the above. 26. In the recording medium according to the above 25, The image processing function performs the detection process on the image that satisfies a predetermined condition. Recording medium. 27. In the recording medium according to the above 26, The predetermined condition is that a person shown in the image is using a reading device for reading product identification information. Recording medium. 28. In the recording medium according to the above 26, The predetermined condition is that the image was generated when a store clerk was performing an operation for arranging the product on a display shelf. Recording medium. 29. In the recording medium according to any one of the above 25 to 28, The shooting range includes at least one of a product display shelf and an area in front of the display shelf. Recording medium. 30. In the recording medium according to any one of the above 25 to 28, The shooting range includes an area where the product is placed when registering the product in a product registration device. Recording medium. 31. In the recording medium according to the above 30, The image processing function performs the detection process on the plurality of images generated while the product is being registered in the product registration device. Recording medium. 32. In the recording medium according to the above 31, The image processing function further performs the detection process on at least one of the images generated before the product is registered in the product registration device. Recording medium. 33. In the recording medium according to the above 31 or 32, The program causes the computer to A recording medium having an execution function for executing a predetermined process when a difference between a second product number, which is the number of the products registered in the product registration device, and the first product number is equal to or greater than a reference value. 34. In the recording medium according to any one of 31 to 33 above, The image processing function calculates a first individual number, which is the number of the products, for each type of the products. The product registration device calculates a second individual number, which is the number of the products, for each type of the products. The calculation function calculates the first product number using, as a first object, the product having the first individual number greater than the maximum value of the second individual numbers. 35. In the recording medium according to any one of 31 to 33 above, The image processing function calculates a first individual number, which is the number of the products, for each type of the products. The product registration device calculates a second individual number, which is the number of the products, for each type of the products. When calculating the first product number, the calculation function in any of the second individual numbers, if the type of the first product having the second individual number is less than the type of the second product having the same first individual number as the second individual number, subtracts a value obtained by multiplying the difference in the number of types between the first product type and the second product type by the second individual number from the number of products detected by the detection process. 36. In the recording medium according to any one of 25 to 35 above, When a plurality of products are detected from at least one of the images, the calculation function identifies the first object by determining whether each of the plurality of products is detected from other images. 37. The program according to any one of 25 to 36 above.

Explanation of Signs

[0079] 10 Product number specifying device 20 Reading device 30 Photographing device 40 Storage unit 110 Acquisition unit 120 Image Processing Unit 130 Calculation Unit 140 Execution Unit 150 Product Registration Unit 160 Settlement Unit

Claims

1. An acquisition means for acquiring a plurality of images including a target area where a product can be placed within a shooting range; An image processing means for performing a detection process of the product on each of the plurality of images; A calculation means for identifying a first object that is a product detected from a predetermined number or more of the images in the detection process, and setting the number obtained by excluding the first object from the products detected by the detection process as a first product number which is the number of products to be settled; A product number specifying device comprising the above.

2. In the product number specifying device according to Claim 1, The image processing means performs the detection process on the images that satisfy a predetermined condition, the product number specifying device.

3. In the product number specifying device according to Claim 2, The predetermined condition is that a person shown in the image is using a reading device for reading product identification information, the product number specifying device.

4. In the product number specifying device according to Claim 2, The predetermined condition is that the image was generated when a store clerk was performing an operation for placing the product on a display shelf, the product number specifying device.

5. In the product number specifying device according to any one of Claims 1 to 4, The shooting range includes at least one of a product display shelf and an area in front of the display shelf, the product number specifying device.

6. In the product number specifying device according to any one of Claims 1 to 4, The shooting range includes an area where the product is placed when registering the product in a product registration device, the product number specifying device.

7. In the product number specifying device according to Claim 6, The image processing means performs the detection process on the plurality of images generated while the product is being registered in the product registration device, the product number specifying device.

8. In the product number specifying device according to Claim 7, The image processing means further performs the detection process on at least one of the images generated before the product is registered in the product registration device, the product number specifying device.

9. A computer, Acquires a plurality of images including a target area where a product can be placed within a shooting range, Performs a detection process of the product on each of the plurality of images, Identifies a first object that is a product detected from a predetermined number or more of the images in the detection process, and sets the number obtained by excluding the first object from the products detected by the detection process as a first product number which is the number of products to be settled, the product number specifying method.

10. A computer is provided with: an acquisition function that acquires a plurality of images including a target area, which is an area where a product can be placed, within a shooting range; an image processing function that performs a detection process for a product on each of the plurality of images; a calculation function that identifies a first object, which is a product detected from a predetermined number or more of the images in the detection process, and sets the number obtained by excluding the first object from the products detected by the detection process as a first product number, which is the number of products to be settled; a program having the above functions.

Images