Product number specification device, product number specification method, and program
Patent Information
- Application Number
- JP2024510795
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2024-11-27
- Estimated Expiration
- 2042-03-29
AI Technical Summary
Existing image recognition systems for product identification struggle to accurately detect the number of products in an image, leading to potential errors in product counting during payment processing.
A product number identification device and method that acquires multiple images of a target area, performs image processing to detect products, and calculates the number of products by identifying a first object detected in a predetermined number of images, excluding it from the total count to ensure accurate product counting.
This approach enables accurate detection of the number of products in an image, reducing errors and ensuring precise product counting during payment processing.
Smart Images

Figure 2023187993000001
Abstract
Description
Product number determination device, product number determination method, and recording medium
[0001] The present invention relates to a product number specifying device, a product number specifying method, and a recording medium.
[0002] In recent years, it has been considered to acquire and process images of products and use the processing results when checking out products. For example, Patent Document 1 describes a recognition system that recognizes target products using image processing. This recognition system includes a first detection means, an extraction means, a calculation means, a recognition means, and a selection means. The first detection means detects products included in image data captured by an imaging unit. The extraction means extracts feature quantities of the products detected by the first detection means from the image data. The calculation means calculates the similarity between matching feature quantities pre-stored for each product, i.e., products with identification information and products without identification information, and the feature quantities extracted by the extraction means. The recognition means recognizes the products detected by the first detection means based on this similarity. The selection means selects the product recognized by the recognition means as the product captured by the imaging unit, provided that the product does not have identification information.
[0003] Patent Document 2 describes how the occurrence of erroneous recognition of a hand as a product can be reduced by registering the features of general excluded items such as hands and arms in a dictionary file.
[0004] JP 2020-17876 A JP 2018-101292 A
[0005] The inventors have considered determining whether or not there is a possibility that products have been registered inaccurately by comparing the number of products registered in a product registration device with the number of products included in an image captured of an area including the product registration device. However, when performing this process, it is necessary to accurately detect the number of products included in the image. However, the recognition system described in the above-mentioned Patent Document 1 may not be able to accurately detect the number of products.
[0006] In view of the above-mentioned problems, an example of an object of the present invention is to provide a product number specifying device, a product number specifying method, and a recording medium that can accurately detect the number of products included in an image.
[0007] According to one aspect of the present invention, there is provided a product quantity determination device comprising: an acquisition means for acquiring a plurality of images whose shooting range includes a target area, which is an area where products can be placed; an image processing means for performing product detection processing on each of the plurality of images; and a calculation means for identifying a first object, which is a product detected from a predetermined number or more of the images in the detection processing, and calculating the number of products detected by the detection processing excluding the first object as a first product quantity, which is the number of products to be checked out.
[0008] According to one aspect of the present invention, a method for determining the number of products is provided, in which a computer acquires a plurality of images whose shooting range includes a target area, which is an area where products can be placed, performs a product detection process on each of the plurality of images, identifies a first object, which is a product detected from a predetermined number or more of the images in the detection process, and sets the number of products detected by the detection process excluding the first object as a first product number, which is the number of products to be checked out.
[0009] According to one aspect of the present invention, there is provided a computer-readable recording medium having stored thereon a program that gives the computer the following functions: an acquisition function for acquiring a plurality of images whose shooting range includes a target area, which is an area where a product may be placed; an image processing function for performing product detection processing on each of the plurality of images; and a calculation function for identifying a first object, which is a product detected from a predetermined number or more of the images in the detection processing, and calculating the number of products detected by the detection processing excluding the first object as the first product number, which is the number of products to be checked out.
[0010] According to one aspect of the present invention, it is possible to provide a product number specifying device, a product number specifying method, and a recording medium that can accurately detect the number of products included in an image.
[0011] FIG. 1 is a diagram illustrating an overview of a product number specifying device according to an embodiment. FIG. 2 is a diagram illustrating a usage environment of the product number specifying device. FIG. 3 is a diagram illustrating a first example of an imaging range by a photographing device. FIG. 4 is a diagram illustrating a modified example of FIG. 3. FIG. 5 is a diagram illustrating a second example of an imaging range by a photographing device. FIG. 6 is a diagram illustrating an example of the functional configuration of a product number specifying device. FIG. 7 is a diagram illustrating an example of the hardware configuration of a product number specifying device. FIG. 8 is a flowchart illustrating a first example of processing performed by the product number specifying device. FIG. 9 is a flowchart illustrating a second example of processing performed by the product number specifying device.
[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.
[0013] FIG. 1 is a diagram illustrating an overview of a product quantity determination device 10 according to an embodiment. The product quantity determination device 10 includes an acquisition unit 110, an image processing unit 120, and a calculation unit 130. The acquisition unit 110 acquires multiple images. These images include a target area, which is an area where products may be placed, within their capture range. The image processing unit 120 performs product detection processing on each of the multiple images. The calculation unit 130 identifies first objects, which are products detected from a predetermined number or more of images in the detection processing, and determines the number of products detected by the detection processing excluding the first objects as the number of products to be checked out. Hereinafter, this number of products will be referred to as the first product count.
[0014] Since the first object is a product detected from a predetermined number of images or more, it is highly likely that it is not a product to be checked out, such as a human hand or an accessory to the product quantity identification device 10. Therefore, the calculation unit 130 determines the number of products to be checked out as the number of products detected by the detection process excluding the first object. Therefore, the product quantity identification device 10 can accurately detect the number of products included in the images.
[0015] A detailed example of the product number specifying device 10 will be described below.
[0016] 2 is a diagram illustrating the usage environment of the product quantity specifying device 10. In the example shown in this figure, the product quantity specifying device 10 is used as a device for registering and settling products. However, as will be described later, the product quantity specifying device 10 may be a device, such as a cloud server, separate from the device for registering and settling products.
[0017] The product number specifying device 10 is installed in a store or an office. When the product number specifying device 10 is installed in a store, customers who enter the store operate the product number specifying device 10 when purchasing products. On the other hand, when the product number specifying device 10 is installed in an office, products are displayed in a corner of the office. Then, a person working in the office operates the product number specifying device 10 when purchasing these products.
[0018] The product number specifying device 10 is used together with a reading device 20, a photographing device 30, and a storage unit 40. In the example shown in the figure, the product number specifying device 10 is used when a customer purchases products, and has a product registration function and a settlement function. In other words, the product number specifying device 10 also functions as a product registration device. Note that the product number specifying device 10 does not necessarily have to have a settlement function. In this case, the product number specifying device 10 transmits information indicating the registered products to the settlement device.
[0019] The reading device 20 acquires product identification information from the product that the customer is attempting to purchase, i.e., the product that is the subject of payment. The reading device 20 may acquire the product identification information by reading a code attached to the product, such as a barcode or two-dimensional code, or may acquire the product identification information from a wireless communication tag attached to the product, such as an RFID tag. The reading device 20 transmits the acquired product identification information to the product quantity specifying device 10. At this time, the reading device 20 also transmits information indicating the timing of acquisition of the product identification information, such as acquisition date and time information, to the product quantity specifying device 10. Hereinafter, this information will be referred to as first timing information. The first timing information may be generated by the product quantity specifying device 10. In this case, the product quantity specifying device 10 generates the first timing information to indicate the timing at which the product identification information was acquired from the reading device 20.
[0020] The reading device 20 may be integrated with the product number specifying device 10 .
[0021] The image capturing device 30 captures an image of a commodity and generates a first image when the reading device 20 acquires commodity identification information from the commodity. The image capturing device 30 may always generate an image, or may generate an image when the reading device 20 acquires commodity identification information. In the former case, the frame rate of the image capturing device 30 is, for example, not less than 1 pfs and not more than 30 pfs, but is not limited to this.
[0022] The photographing device 30 may be attached to the product number specifying device 10, or may be attached above the product number specifying device 10, for example, on the ceiling.
[0023] The imaging range of the imaging device 30 includes the area where products may be placed when the reading device 20 acquires the product identification information, i.e., the product reading area. The imaging range of the imaging device 30 may include the reading device 20. The imaging range may also include the area surrounding the product reading area. In this case, it is preferable that the imaging range includes at least one of the area through which the product passes when it moves toward the product reading area and the area through which the product passes when it moves out of the product reading area. The imaging device 30 then transmits the generated image to the product quantity identification device 10. At this time, the imaging device 30 also transmits information indicating the timing of generation of the image, for example, generation date and time information, to the product quantity identification device 10. Hereinafter, this information will be referred to as second timing information.
[0024] The storage unit 40 stores product identification information and prices for each of a plurality of products. The product quantity specifying device 10 uses the information stored in the storage unit 40 when processing the payment for the products. The storage unit 40 may be, for example, a server installed in the store where the product quantity specifying device 10 is located.
[0025] In the above description, the product number specifying device 10 is used when a product is purchased. However, the product number specifying device 10 may also be used when a store clerk displays products on a shelf.
[0026] The product number specifying device 10 does not necessarily have to perform the registration process and settlement process for the target products. In this case, the product number specifying device 10 is a device that performs the registration process and settlement process, such as a cloud server, separate from the terminal that performs the registration process and settlement process, such as a POS terminal. The reading device 20 communicates with this terminal. The product number specifying device 10 obtains from this terminal information indicating the number of products registered in the terminal, i.e., the second product number described below.
[0027] 3 is a diagram illustrating a first example of the imaging range of the imaging device 30. In the example shown in this figure, the product quantity specifying device 10 is used when purchasing products and is placed on a stand 50. The stand 50 is sufficiently larger than the product quantity specifying device 10, and a part of it serves as a product display area 510.
[0028] As explained using Figure 2, the imaging range of the imaging device 30 includes an area where products can be placed. The area where products can be placed includes at least one of an area where products are temporarily placed and an area where products are placed when the product identification information of the products is registered in the product number specification device 10. An example of the former is an area on the stand 50 where products can be temporarily placed. An example of the latter is an area (space) where the reading device 20, which reads product identification information from products, can read the product identification information.
[0029] In the example shown in the figure, the imaging range of the imaging device 30 may further include the product number determination device 10 and the product display area 510. Note that products that can be registered in the product number determination device 10 may be displayed in a location other than the product display area 510.
[0030] At least one of the accessory devices 60 of the product quantity determination device 10, such as the reading device 20 shown in FIG. 2 , a near-field wireless communication device that communicates with a mobile terminal, and a receipt printing device, is often placed on the platform 50. Objects 70, such as trash cans, may also be placed around the platform 50. The imaging range includes the area in which the products are placed when the reading device 20 reads the product identification information, such as the front of the reading device 20. Therefore, the products whose product identification information has been read by the reading device 20 are included in the image generated by the imaging device 30.
[0031] The photographing range of the photographing device 30 may include the accessory device 60 and the object 70. In this case, the accessory device 60 and the object 70 may be erroneously recognized as products to be checked out. However, because the accessory device 60 and the object 70 hardly move, they are included in the multiple images generated by the photographing device 30. Therefore, the product quantity specifying device 10 recognizes the accessory device 60 and the object 70 as the first object described above.
[0032] Fig. 4 is a diagram showing a modification of Fig. 3. In the example shown in this figure, the imaging range includes the product display area 510 and the accessory equipment 60, but does not include the product number specifying device 10. In this way, the imaging range of the imaging device 30 only needs to include an area where products can be placed.
[0033] 5 is a diagram illustrating a second example of the imaging range of the imaging device 30. In the example shown in this figure, the product number specifying device 10 is used when a store clerk removes products 92 from containers 90, such as collapsible containers or cardboard boxes, and displays them on display shelves 80. In this case, the product number specifying device 10 specifies the number of products that the store clerk has displayed on display shelves 80. In this example, the reading device 20 is a portable device operated by the store clerk. However, there are cases in which the reading device 20 is not used.
[0034] In this example, the range of photography by the camera 30 includes at least one of, and preferably both, the display shelf 80 and the area in front of the display shelf 80, i.e., the area where the containers 90 are placed. Therefore, the products 92 displayed on the display shelf 80 are included in the image generated by the camera 30. The camera 30 is a surveillance camera installed in the store, but it may be any other type of camera.
[0035] Fig. 6 is a diagram showing an example of the functional configuration of the product number specifying device 10. As described using Fig. 1, the product number specifying device 10 includes an acquisition unit 110, an image processing unit 120, and a calculation unit 130. In the example shown in this figure, the product number specifying device 10 further includes an execution unit 140, a product registration unit 150, and a settlement unit 160. Note that when the product number specifying device 10 corresponds to the example shown in Fig. 5, the product number specifying device 10 does not include the product registration unit 150 and the settlement unit 160.
[0036] The acquisition unit 110 acquires the image generated by the image capturing device 30. At this time, the acquisition unit 110 also acquires second timing information.
[0037] The image processing unit 120 performs product detection processing on each of the multiple images generated by the imaging device 30. The image processing unit 120 may perform product detection processing using a model generated by machine learning, or may perform product detection processing by feature matching. Information required for the image processing unit 120 to perform product detection processing is stored in, for example, the storage unit 40.
[0038] The image processing unit 120 then uses the results of these detection processes to calculate the total number of products to be checked out. Hereinafter, this total number of products will be referred to as the first total number. In this case, the image processing unit 120 may track the products across multiple images and use the tracking results to determine the first total number. The image processing unit 120 may also determine the number of products by product type. Hereinafter, these numbers by type will be referred to as the first individual number.
[0039] When the image capturing device 30 generates an image in response to the reading device 20 acquiring the commodity identification information, the image acquired by the acquiring unit 110 is the image generated when the reading device 20 reads the commodity identification information. The image processing unit 120 then preferably performs commodity detection processing on all of the multiple images acquired by the acquiring unit 110.
[0040] On the other hand, if the photographing device 30 is always operating, the image processing unit 120 may perform product detection processing on all images, or may select images that satisfy predetermined conditions as targets for the detection processing. Examples of the "predetermined conditions" include at least one of the following (1) to (4):
[0041] (1) The image has second timing information that is closest to the first timing information. This example corresponds to the state shown in FIG. 3 or 4. The image selected here includes a product corresponding to the product identification information acquired by the acquisition unit 110. Here, three cases are possible: the first timing information and the second timing information indicate the same timing; the first timing information is earlier; or the second timing information is earlier. In either case, the difference between the first timing information and the second timing information is, for example, one second or less. When the reading device 20 has read product identification information from each of multiple products, the acquisition unit 110 performs the above-mentioned image selection process for each product identification information.
[0042] (2) A person in the image is using the reading device 20. This example also corresponds to the state shown in Fig. 3 or 4. The image processing unit 120 performs a person detection process on the image. Then, if the image processing unit 120 can detect a person and determines that the person is causing the reading device 20 to read commodity identification information, it performs a commodity detection process on the image.
[0043] (3) The image is generated while the product is being registered in the product number specifying device 10. This example also corresponds to the state shown in Fig. 3 or 4. The period while the product is being registered in the product number specifying device 10 refers to, for example, the period after information indicating the start of product registration is input to the product number specifying device 10 and until information indicating that the checkout process will proceed is input to the product number specifying device 10. The image processing unit 120 uses the second timing information to specify the image generated while the product is being registered.
[0044] In this (3), the image processing unit 120 may further add at least one of the images generated before the products were registered in the product count identification device 10 to the images to be processed. For example, the image processing unit 120 identifies the time when product registration began. Then, the image processing unit 120 adds, to the images to be processed, images generated between this time and a predetermined time before that time, for example, up to 10 seconds before that time. The reason for this is that there is a high possibility that the first object is captured in these images.
[0045] (4) The image was generated while a store clerk was working to place products on the display shelves. This example corresponds to the state shown in Fig. 5. The image processing unit 120 performs a store clerk detection process on the image. If the image processing unit 120 can detect a store clerk and determines that the store clerk is displaying products 92 on the display shelves 80, it performs a product detection process on the image.
[0046] The product registration unit 150 acquires product identification information from the reading device 20. At this time, the product registration unit 150 also acquires first timing information. The product registration unit 150 generates registration information indicating the products to be settled. The registration information includes a list of product identification information for the products to be settled, information indicating the total number of products to be settled, and information indicating the number of such products for each product. Hereinafter, the total number of products based on the registration information will be referred to as the second product number. Furthermore, the number of such products for each product based on the registration information will be referred to as the second individual number.
[0047] 1, the calculation unit 130 identifies first objects, i.e., products detected from a predetermined number or more of images in the detection process. The predetermined number is, for example, 3 or more, preferably 5 or more, but is not limited to these values. The calculation unit 130 then subtracts the number of first objects from the first total number calculated by the image processing unit 120 to obtain the first product number, i.e., the number of products to be checked out.
[0048] Here, when multiple products are detected in at least one image, the calculation unit 130 may identify the first object by determining whether each of the multiple products is detected in other images. When multiple products are detected in one image, there is a possibility that at least one of the multiple products is an object other than a product. This possibility is particularly high when the imaging range of the image capture device 30 is relatively narrow, slightly larger than the product reading area of the reading device 20, for example, between one and two times larger. Therefore, the calculation unit 130 stores the feature amounts of each of the multiple detected products in the storage unit 40 and designates each product having these feature amounts as a candidate for the first object. The calculation unit 130 then identifies the number of other images containing these candidate first objects and uses this number to determine whether the candidate first object is actually the first object.
[0049] For example, if this number is large, the candidate frequently enters the imaging range of the imaging device 30. In this case, the candidate is likely not actually the product the customer is trying to purchase, but rather a nearby product or device. Therefore, if the number of other images containing the first object candidate is equal to or greater than a reference value, the calculation unit 130 determines that the candidate is an object other than a product, i.e., the first object. Note that the reference value is, for example, 2, but may be 3 or more, or even 1.
[0050] The image processing unit 120 may store the feature amounts of the products detected from the images for each of the multiple images in the storage unit 40. In this case, the calculation unit 130 sets a flag indicating that the feature amount corresponds to a candidate for the first object.
[0051] Furthermore, when the image processing unit 120 calculates the number of products by product type, i.e., the first number of products, the calculation unit 130 may perform at least one of the processes shown in (A) and (B) below.
[0052] (A) The calculation unit 130 treats products having a first number of individuals greater than the maximum value of the second number of individuals as first objects and calculates the first number of products. As described above, the second number of individuals is calculated based on the product identification information registered in the product registration unit 150, and indicates the number of products for each product. Here, if there is a product having a first number of individuals greater than the maximum value of the second number of individuals, there is a high possibility that the product is not actually a product. Specifically, there is a high possibility that the product having the first number of individuals is a human hand, or an accessory 60 or object 70 shown in FIG. 3 . Therefore, when calculating the number of products, the calculation unit 130 treats the product having this first number of individuals as a first object, i.e., an object different from a product.
[0053] For example, assume that the image processing unit 120 determines that there are "two units of product A," "three units of product B," and "one unit of product C," and that the registration information generated by the product registration unit 150 indicates that there are "two rice balls" and "one bottle of tea." In this case, the maximum value of the second number of individuals is "2," which is the number of rice balls. Meanwhile, the number of "product B" detected by the image processing unit 120 is three, which is greater than the maximum value of the second number of individuals. In this case, the calculation unit 130 treats "product B" as an object different from the product, and subtracts "3," which is the number of "product B," from the first total number.
[0054] (B) For any second number of items, if the number of first product types having the second number of items is fewer than the number of second product types having the same first number of items, the calculation unit 130 subtracts the difference between the number of first product types and the number of second product types multiplied by the second number of items from the number of items detected by the detection process, i.e., the first total number. For example, assume that the image processing unit 120 determines that there are "two items of product A," "two items of product B," "one item of product C," and "one item D," and that the registration information generated by the product registration unit 150 includes "two rice balls," "one item of bread," and "one bottled tea." In this case, even though there is only one type of product with two items, the processing result of the image processing unit 120 indicates that there are two types of product with two items. In this case, it is highly likely that one of these two types of products is not actually a product. Therefore, the calculation unit 130 subtracts "2", which is the value obtained by multiplying "1", which is the difference between the number of first product types and the number of second product types, by "2", which is the second number of individuals, from the first total number.
[0055] The execution unit 140 executes a predetermined process when the difference between the second product number, which is the total number of products registered by the product registration unit 150, and the first product number calculated by the calculation unit 130 is equal to or greater than a reference value. This reference value is, for example, 1, but may be 2 or greater. The difference being equal to or greater than the reference value indicates that the total number of products registered in the product registration unit 150 may differ from the total number of products that customers or office workers are attempting to purchase. An example of the predetermined process is outputting warning information. The execution unit 140 may output the warning information to a display or speaker provided in the product quantity identification device 10, or may output the warning information to a terminal operated by a manager who manages product sales, such as a terminal operated by a store clerk.
[0056] If the execution unit 140 does not perform the predetermined process, i.e., if there is a high possibility that the total number of products registered in the product registration unit 150 matches the total number of products that customers or office workers are trying to purchase, the settlement unit 160 performs settlement processing using the registration information generated by the product registration unit 150, i.e., the list of product identification information for the products to be settled. At this time, the settlement unit 160 uses the information stored in the storage unit 40.
[0057] Note that when the product number specifying device 10 is used as a device separate from the device that registers and settles products, such as a cloud server, the product number specifying device 10 does not include the product registration unit 150 and the settlement unit 160. In this case, a device having the product registration unit 150 and the settlement unit 160, such as a POS terminal, is placed in the position of the product number specifying device 10 in FIG. 3 or 4 , separate from the product number specifying device 10. Hereinafter, this device will be referred to as the registration settlement device. The product number specifying device 10 then acquires information indicating the second number of items and the second product count from the registration settlement device. The execution unit 140 of the product number specifying device 10 transmits information indicating whether the difference between the second product count and the first product count is equal to or greater than a reference value to the registration settlement device. When the difference between the second product count and the first product count is equal to or greater than the reference value, the registration settlement device displays predetermined information on the display of the device or outputs it from the speaker of the device.
[0058] 7 is a diagram illustrating an example of the hardware configuration of the product number identification device 10. The product number identification device 10 includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.
[0059] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.
[0060] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.
[0061] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.
[0062] The storage device 1040 is an auxiliary storage device realized by removable media such as a hard disk drive (HDD), a solid state drive (SSD), a memory card, or a read-only memory (ROM), and has a recording medium. The recording medium of the storage device 1040 stores program modules that realize each function of the product quantity identification device 10 (e.g., the acquisition unit 110, the image processing unit 120, the calculation unit 130, the execution unit 140, the product registration unit 150, and the settlement unit 160). The processor 1020 loads each of these program modules into the memory 1030 and executes them, thereby realizing each function corresponding to the program module. The storage device 1040 may also function as the memory unit 40.
[0063] The input / output interface 1050 is an interface for connecting the product number specifying device 10 to various input / output devices. For example, the product number specifying device 10 communicates with at least one of the reading device 20, the photographing device 30, and the storage unit 40 via the input / output interface 1050.
[0064] The network interface 1060 is an interface for connecting the product quantity specifying device 10 to a network. This network is, for example, a local area network (LAN) or a wide area network (WAN). The network interface 1060 may be connected to the network wirelessly or by wire. The product quantity specifying device 10 may communicate with at least one of the photographing device 30 and the storage unit 40 via the network interface 1060.
[0065] 8 is a flowchart showing a first example of the process performed by the product number specifying device 10. This figure corresponds to the example shown in FIG.
[0066] When a customer or an office worker purchases a product, the product identification information of the product is read by the reading device 20. The product registration unit 150 of the product number determination device 10 acquires this product identification information. If there are multiple products to be purchased, the product registration unit 150 acquires the product identification information for each of the multiple products. The product registration unit 150 then calculates the second product quantity (step S10).
[0067] The acquisition unit 110 acquires the images generated by the image capture device 30. The image processing unit 120 processes the images to detect the products included in each of the images (step S20). A specific example of the processing performed here is as described with reference to FIG. 6.
[0068] The calculation unit 130 then calculates the first product quantity using the processing result of the image processing unit 120 (step S30). A specific example of the processing performed here is as described with reference to FIG.
[0069] If the difference between the second product count and the first product count is equal to or greater than the reference value (Step S40: Yes), the execution unit 140 executes a predetermined process (Step S50). One example of the predetermined process is a warning process. Then, the product count specification device 10 returns to Step S10.
[0070] On the other hand, when the difference between the second product quantity and the first product quantity is less than the reference value (step S40: No), the settlement unit 160 performs settlement processing (step S60).
[0071] 9 is a flowchart showing a second example of the process performed by the product quantity specifying device 10. This figure corresponds to the example shown in FIG.
[0072] The store clerk moves the container 90 close to the display shelf 80 and starts stocking the products (step S12). Specifically, the clerk removes the products 92 from the container 90 and displays them on the display shelf 80. The image capturing device 30 repeatedly generates images while the clerk is stocking the products. The acquisition unit 110 acquires these images.
[0073] When the stocking is completed (step S22), the image processing unit 120 processes the images acquired by the acquisition unit 110 and detects the products included in each of the images (step S30). A specific example of the processing performed here is as described with reference to FIG. 6.
[0074] The calculation unit 130 then calculates the first product quantity using the processing result of the image processing unit 120 (step S40). A specific example of the processing performed here is as described with reference to FIG.
[0075] As described above, according to this embodiment, the image processing unit 120 detects products by processing images. The calculation unit 130 then identifies products, i.e., first objects, detected from a predetermined number or more of images in the detection process performed by the image processing unit 120. These first objects are likely not products, such as human hands or accessory devices 60. Therefore, the calculation unit 130 determines the number of products detected by the detection process excluding the first objects as the first product count, which is the number of products to be checked out. Therefore, by using the product number identification device 10, the number of products included in an image can be detected with high accuracy.
[0076] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations can also be adopted.
[0077] In addition, although the flowcharts used in the above description describe multiple steps (processes) in a sequential order, the execution order of the steps in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed as long as it does not interfere with the content. Furthermore, at least one step may be performed by another operating entity, such as another device or person. Furthermore, the above-described embodiments can be combined as long as the content does not conflict.
[0078] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes. 1. A product number specifying device comprising: an acquisition means for acquiring a plurality of images whose shooting range includes a target area, which is an area where products can be placed; an image processing means for performing a product detection process on each of the plurality of images; and a calculation means for identifying first objects, which are products detected from a predetermined number or more of the images in the detection process, and calculating the number of products detected by the detection process excluding the first objects, as a first product number, which is the number of products to be checked out. 2. The product number specifying device described in above 1, wherein the image processing means performs the detection process on the images that satisfy a predetermined condition. 3. The product number specifying device described in above 2, wherein the predetermined condition is that a person appearing in the image is using a reading device that reads product identification information. 4. The product number specifying device described in above 2, wherein the predetermined condition is that the image was generated while a store clerk was working to arrange the products on a display shelf. 5. 5. The product number specifying device according to any one of claims 1 to 4 above, wherein the imaging range includes at least one of the product display shelf and the area in front of the display shelf. 6. The product number specifying device according to any one of claims 1 to 4 above, wherein the imaging range includes the area where the products are placed when the products are registered in the product registration device. 7. The product number specifying device according to claim 6, wherein the image processing means performs the detection process on the plurality of images generated while the products are being registered in the product registration device. 8. The product number specifying device according to claim 7, wherein the image processing means further performs the detection process on at least one of the images generated before the products are registered in the product registration device. 9. The product number specifying device according to claim 7 or 8, wherein the product number specifying device comprises an execution means for executing a predetermined process when a difference between the first product number and a second product number, which is the number of products registered in the product registration device, is equal to or greater than a reference value.10. In the product number specifying device described in any one of items 7 to 9 above, the image processing means calculates a first number of items, which is the number of products, for each type of product, the product registration device calculates a second number of items, which is the number of products, for each type of product, and the calculation means calculates the first number of products by treating the products having the first number of items that is greater than the maximum value of the second number of items as the first object. 11. 11. A product number specifying device according to any one of claims 1 to 9 above, wherein the image processing means calculates a first number of items, which is the number of products, for each product type, and the product registration device calculates a second number of items, which is the number of products, for each product type, and when calculating the first product number, the calculation means subtracts the value obtained by multiplying the difference between the number of products of the first product type and the number of products of the second product type by the second number of items, from the number of products detected by the detection process, if, for any of the second numbers of items, a first product type having the second number of items is fewer than a second product type having the same first number of items as the second number of items. 12. A product number specifying device according to any one of claims 1 to 11 above, wherein, when a plurality of products are detected from at least one of the images, the calculation means specifies the first object by determining whether each of the plurality of products has been detected in the other images. A product number determination method in which a computer acquires a plurality of images that include, within a shooting range, a target area that is an area where products may be placed, performs a product detection process on each of the plurality of images, identifies first objects that are products detected from a predetermined number or more of the images in the detection process, and sets the number of products detected by the detection process excluding the first objects as a first product number that is the number of products to be checked out. 14. A product number determination method in accordance with claim 13, in which the computer performs the detection process on images that satisfy a predetermined condition. 15. A product number determination method in accordance with claim 14, in which the predetermined condition is that a person appearing in the image is using a reading device that reads product identification information.16. The method for specifying the number of products described in 14 above, wherein the predetermined condition is that the images were generated while a store clerk was working to arrange the products on a display shelf. 17. The method for specifying the number of products described in any one of 13 to 16 above, wherein the shooting range includes at least one of the display shelf on which the products are displayed and the area in front of the display shelf. 18. The method for specifying the number of products described in any one of 13 to 16 above, wherein the shooting range includes the area where the products are placed when they are registered in a product registration device. 19. The method for specifying the number of products described in 18 above, wherein the computer performs the detection process on the multiple images generated while the products are being registered in the product registration device. 20. The method for specifying the number of products described in 19 above, wherein the computer further performs the detection process on at least one of the images generated before the products are registered in the product registration device. 21. 21. The method for specifying the number of products described in 19 or 20 above, wherein the computer executes a predetermined process when a difference between a second product number, which is the number of products registered in the product registration device, and the first product number is equal to or greater than a reference value. 22. The method for specifying the number of products described in any one of 19 to 21 above, wherein the computer calculates a first number of products, which is the number of products, for each type of product, the product registration device calculates a second number of products, which is the number of products, for each type of product, and the computer calculates the first number of products, treating the products having the first number of products greater than the maximum value of the second number of products as the first objects.23. The method for specifying the number of products described in any one of claims 19 to 21 above, wherein the computer calculates a first number of items, which is the number of products, for each product type, and the product registration device calculates a second number of items, which is the number of products, for each product type, and when calculating the first number of items, the computer subtracts the value obtained by multiplying the difference between the number of products of the first product type and the number of products of the second product type by the second number of items, from the number of products detected by the detection process. 24. The method for specifying the number of products described in any one of claims 13 to 23, wherein, when multiple products are detected from at least one of the images, the computer specifies the first object by determining whether each of the multiple products is detected in the other images. 25. 26. A computer-readable recording medium storing a program that provides a computer with the following functions: an acquisition function that acquires multiple images whose shooting range includes a target area, which is an area where products can be placed; an image processing function that performs product detection processing on each of the multiple images; and a calculation function that identifies first objects, which are products detected from a predetermined number or more of the images in the detection processing, and calculates the number of products detected by the detection processing excluding the first objects as a first product number, which is the number of products to be checked out. 26. A recording medium as described in 25 above, wherein the image processing function performs the detection processing on the image that satisfies a predetermined condition. 27. A recording medium as described in 26 above, wherein the predetermined condition is that a person appearing in the image is using a reading device that reads product identification information. 28. A recording medium as described in 26 above, wherein the predetermined condition is that the image was generated while a store clerk was working to place the products on a display shelf. 29. 29. The recording medium according to any one of the above 25 to 28, wherein the photographing range includes at least one of a display shelf for merchandise and an area in front of the display shelf.30. The recording medium described in any one of claims 25 to 28 above, wherein the shooting range includes an area where the product is placed when the product is registered in the product registration device. 31. The recording medium described in claim 30 above, wherein the image processing function performs the detection process on the plurality of images generated while the product is being registered in the product registration device. 32. The recording medium described in claim 31 above, wherein the image processing function further performs the detection process on at least one of the images generated before the product is registered in the product registration device. 33. The recording medium described in claim 31 or 32 above, wherein the program provides the computer with an execution function for executing a predetermined process when the difference between the first product number and a second product number, which is the number of products registered in the product registration device, is equal to or greater than a reference value. 34. 34. A recording medium described in any one of claims 31 to 33 above, wherein the image processing function calculates a first number of individuals, which is the number of products, for each product type, and the product registration device calculates a second number of individuals, which is the number of products, for each product type, and the calculation function calculates the first product number by treating products having the first number of individuals greater than the maximum value of the second number of individuals as first objects. 35. A recording medium described in any one of claims 31 to 33 above, wherein the image processing function calculates a first number of individuals, which is the number of products, for each product type, and the product registration device calculates a second number of individuals, which is the number of products, for each product type, and when calculating the first product number, if, for any of the second numbers of individuals, the number of first product types having the second number of individuals is smaller than the number of second product types having the same first number of individuals as the second number of individuals, the calculation function subtracts the value obtained by multiplying the second number of individuals by the difference between the number of the first product type and the number of the second product type from the number of products detected by the detection process. 36. The recording medium according to any one of claims 25 to 35, wherein the calculation function, when a plurality of products are detected from at least one of the images, identifies the first object by determining whether each of the plurality of products is detected from another of the images.37. The program according to any one of 25 to 36 above.
[0079] REFERENCE SIGNS LIST 10 Product number identification device 20 Reading device 30 Photography device 40 Storage unit 110 Acquisition unit 120 Image processing unit 130 Calculation unit 140 Execution unit 150 Product registration unit 160 Payment unit
Claims
1. An acquisition means for acquiring a plurality of images including a target area, which is an area where a product may be placed, in a shooting range; an image processing means for performing a product detection process on each of the plurality of images; a calculation means for identifying a first object, which is a commodity detected from a predetermined number or more of the images in the detection process, and calculating a number of the commodities detected by the detection process excluding the first object as a first commodity number, which is the number of commodities to be checked out; A product number specifying device comprising:
2. 2. The device for specifying the number of products according to claim 1, The image processing means performs the detection process on the image that satisfies a predetermined condition.
3. 3. The device for specifying the number of products according to claim 2, The predetermined condition is that a person appearing in the image is using a reading device that reads product identification information.
4. 3. The device for specifying the number of products according to claim 2, The predetermined condition is that the image was generated while a store clerk was performing an operation to arrange the products on a display shelf.
5. In the device for specifying the number of products according to any one of claims 1 to 4, The photographing range of the product number specifying device includes at least one of the display shelf for the products and the area in front of the display shelf.
6. In the device for specifying the number of products according to any one of claims 1 to 4, The photographing range of the product number specifying device includes an area in which the product is placed when the product is registered in the product registration device.
7. 7. The device for specifying the number of products according to claim 6, The image processing means performs the detection process on the plurality of images generated while the products are being registered in the product registration device.
8. 8. The device for specifying the number of products according to claim 7, The image processing means further performs the detection process on at least one of the images generated before the product is registered in the product registration device.
9. The computer Acquire a plurality of images including a target area, which is an area where the product may be placed, in the photographing range; performing a product detection process on each of the plurality of images; A method for identifying the number of products, comprising: identifying a first object, which is a product detected from a predetermined number or more of the images in the detection process; and setting the number of products detected by the detection process excluding the first object as a first number of products, which is the number of products to be checked out.
10. On the computer, An acquisition function for acquiring a plurality of images including a target area, which is an area where a product may be placed, within a shooting range; an image processing function for performing a product detection process on each of the plurality of images; a calculation function of identifying a first object, which is a commodity detected from a predetermined number or more of the images in the detection process, and calculating a number of the commodities detected by the detection process excluding the first object as a first number of commodities, which is the number of commodities to be settled; A program that allows you to have
