Counting device and control method

Abstract

To perform specification of the number of commodities in a short period of time.SOLUTION: A counting device 2000 specifies identification information of an object commodity 10 from a mark 12 included in a first photographed image 32. The counting device 2000 acquires master information 50 corresponding to the object commodity 10. The counting device 2000 detects the object commodity 10 from a second photographed image 42 by analyzing the second photographed image 42 using the master information 50, and on the basis of the detection result, specifies the number of object commodities 10 contained in a commodity group 20.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004] ,

[0006] , , , , , ,

[0005] , , ,

[0003] , , , , ,

[0001] The present invention relates to the counting of products.

Background Art

[0002] In inspection work and the like, management of products to be inspected is carried out. For example, Patent Document 1 discloses a technique for analyzing an image of a package to be inspected to identify the name and number of stages of the package, identifying a stacking method corresponding to the name of the package, and identifying the number of packages based on the stacking method and the number of stages. The stacking method is predetermined for each item.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the invention of Patent Document 1, characteristic amounts of various packages are stored in a database in advance, and the name of the package is identified by comparing the characteristic amount obtained from the image of the package with the characteristic amounts of each package stored in the database. However, in this method, the time required to identify the name of the package becomes long. As a result, the time required to identify the number of packages also becomes long.

[0005] The present invention has been made in view of the above problems, and one of its objects is to provide a technique for identifying the number of products in a short time.

Means for Solving the Problems

[0006] The counting device of the present invention comprises: 1) an identification unit that identifies a product identified by a mark based on a mark detected from some of a plurality of products included in an image; and 2) a counting unit that identifies a target product that is determined to be the same product as the identified product from the image based on information about the image of the identified product, and counts the number of the identified target product.

[0007] The control method of the present invention is performed by a computer. The control method includes: 1) an identification step of identifying a product identified by a mark based on a mark detected from some of a plurality of products included in an image; and 2) a counting step of identifying a target product that is determined to be the same product as the identified product from the image based on information about the image of the identified product, and counting the number of the identified target product.

[0008] The program of the present invention causes a computer to execute each step of the control method of the present invention. [Effects of the Invention]

[0009] According to the present invention, a technology is provided for determining the number of items in a short amount of time. [Brief explanation of the drawing]

[0010] [Figure 1] This figure conceptually illustrates the operation of the counting device according to Embodiment 1. [Figure 2] This is a block diagram illustrating the functional configuration of a counting device. [Figure 3] This diagram illustrates a computer used to implement a counting device. [Figure 4] This is a flowchart illustrating the processing flow performed by the counting device of Embodiment 1. [Figure 5] This diagram illustrates a case where a group of products is imaged from one direction. [Figure 6]This diagram illustrates a case where a group of products is imaged from one direction, and a second camera generates a second image and a depth image. [Figure 7] This is a block diagram illustrating the functional configuration of the counting device according to Embodiment 2. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described below with reference to the drawings. In all drawings, similar components are denoted by the same reference numerals, and their descriptions are omitted as appropriate. Unless otherwise specified, in each block diagram, each block represents a functional unit configuration, not a hardware unit configuration. In the following description, unless otherwise specified, various predetermined values ​​(such as thresholds) are stored in advance in a storage device accessible from the functional unit that uses those values.

[0012] [Embodiment 1] <Overview> Figure 1 is a diagram conceptually illustrating the operation of the counting device 2000 according to Embodiment 1. The operation of the counting device 2000 described using Figure 1 is illustrative to facilitate understanding of the counting device 2000 and does not limit its operation. Details and variations of the operation of the counting device 2000 will be described later.

[0013] The counting device 2000 identifies the number of target products 10 included in the product group 20. A product group 20 is a collection of one or more target products 10 gathered in one place. For example, multiple target products 10 stacked on a pallet during inspection work for shipment or receipt of target products 10 are treated as a product group 20. The following outlines how the counting device 2000 identifies the number of target products 10 included in the product group 20.

[0014] The counting device 2000 acquires a first captured image 32. The first captured image 32 is generated by the first camera 30. The first camera 30 is provided such that the mark 12 enters its imaging range. The mark 12 is an arbitrary mark (e.g., barcode) representing the identification information of the target product 10. The counting device 2000 identifies the identification information of the target product 10 from the mark 12 included in the first captured image 32 by analyzing the first captured image 32.

[0015] The counting device 2000 acquires master information 50 associated with the identification information of the target product 10. The master information 50 is used for image analysis to detect the target product 10. The master information 50 is, for example, a template image of the target product 10, image features (feature amounts on the image) of the target product 10, and the like.

[0016] The counting device 2000 acquires a second captured image 42 including the product group 20. The second captured image 42 is generated by the second camera 40. The second camera 40 is provided such that the product group 20 is included in its imaging range. The counting device 2000 detects the target product 10 included in the second captured image 42 by analyzing the second captured image 42 using the master information 50. The counting device 2000 specifies the number of target products 10 included in the product group 20 based on this detection result.

[0017] Here, one second captured image 42 may include only some of the target products 10, not all of the target products 10 included in the product group 20. For example, when the target products 10 are stacked on a pallet, if the product group 20 is imaged only from one direction, some of the target products 10 may be hidden by other target products 10.

[0018] Therefore, for example, the counting device 2000 identifies the number of target products 10 included in the product group 20 based on the number of target products 10 detected from the second captured image 42 and a prior rule regarding the arrangement of the target products 10 in the product group 20. Additionally, for example, the counting device 2000 acquires the second captured image 42 from each of a plurality of second cameras 40 installed to capture the product group 20 from different directions, and identifies the number of target products 10 included in the product group 20 based on the number of target products 10 detected from each second captured image 42. A more specific method for identifying the number of target products 10 will be described later.

[0019] <An example of the operation and effect> The counting device 2000 of the present embodiment identifies the identification information of the target product 10 by using the mark 12 included in the first captured image 32. Further, the counting device 2000 acquires the master information 50 corresponding to the identified identification information of the target product 10, and detects the target product 10 from the second captured image 42 by using the master information 50. Then, the counting device 2000 identifies the number of target products 10 included in the product group 20 based on the detection result of the target product 10.

[0020] In this way, in the counting device 2000, when detecting the target product 10 from the second captured image 42 to identify the number of target products 10, the master information 50 corresponding to the identification information of the target product 10 is used. Therefore, when analyzing the second captured image 42, matching or the like using the master information of a product different from the target product 10 is not performed. Thus, compared with the invention disclosed in Patent Document 1 and the like, the time required for the process of detecting the target product 10 from the image is shortened. Therefore, the target product 10 can be counted in a shorter time. Further, using the master information 50 of the target product 10 also has the effect of reducing the occurrence of errors in the detection of products (for example, the product included in the second captured image 42 being determined to be a product other than the target product 10).

[0021] Hereinafter, the counting device 2000 will be described in more detail.

[0022] <Example of functional configuration> Figure 2 is a block diagram illustrating the functional configuration of the counting device 2000. The counting device 2000 includes a identification unit 2020, a master acquisition unit 2040, and a counting unit 2060. The identification unit 2020 identifies the identification information of the target product 10 represented by the mark 12 by analyzing the first captured image 32. The master acquisition unit 2040 acquires master information 50 corresponding to the identification information of the target product 10. The counting unit 2060 uses the master information 50 to detect the target product 10 from the second captured image 42 and, based on the detection result, identifies the number of target products 10 included in the product group 20.

[0023] <Example of hardware configuration for counting device 2000> Each functional component of the counting device 2000 may be implemented by hardware (e.g., hardwired electronic circuits) or by a combination of hardware and software (e.g., a combination of electronic circuits and programs that control them). The following will further explain the case where each functional component of the counting device 2000 is implemented by a combination of hardware and software.

[0024] Figure 3 illustrates a computer 1000 for realizing the counting device 2000. Computer 1000 is any computer. For example, computer 1000 could be a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, computer 1000 could be a portable computer such as a smartphone or tablet device.

[0025] Computer 1000 may be a dedicated computer designed to implement the counting device 2000, or it may be a general-purpose computer. In the latter case, for example, by installing a predetermined application on computer 1000, the functions of the counting device 2000 are implemented on computer 1000 (computer 1000 begins to operate as the counting device 2000). The above application consists of a program for implementing the functional components of the counting device 2000.

[0026] Computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120 to send and receive data from each other. However, the method of connecting the processor 1040 and other components is not limited to bus connection.

[0027] Processor 1040 is a variety of processors such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), and FPGA (Field-Programmable Gate Array). Memory 1060 is main memory implemented using RAM (Random Access Memory), etc. Storage device 1080 is auxiliary storage implemented using hard disk, SSD (Solid State Drive), memory card, or ROM (Read Only Memory), etc.

[0028] The input / output interface 1100 is an interface for connecting the computer 1000 with input / output devices. For example, input devices such as keyboards and output devices such as display devices are connected to the input / output interface 1100.

[0029] The network interface 1120 is an interface for connecting the computer 1000 to a communication network. This communication network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). For example, the first camera 30 and the second camera 40 are connected to the network interface 1120 via the communication network. However, the first camera 30 and the second camera 40 may be connected to the computer 1000 in a way that allows communication through a means other than the network interface 1120 (for example, via the input / output interface 1100), or they may not be connected to the computer 1000 in a way that allows communication. In the latter case, for example, each camera and the computer 1000 are made able to access a common storage device. The computer 1000 then acquires the first captured image 32 and the second captured image 42 via that storage device.

[0030] The storage device 1080 stores programs that implement each functional component of the counting device 2000 (programs that implement the aforementioned applications). The processor 1040 reads these programs into memory 1060 and executes them to implement each functional component of the counting device 2000.

[0031] <Processing flow> Figure 4 is a flowchart illustrating the processing flow performed by the counting device 2000 of Embodiment 1. The identification unit 2020 acquires the first captured image 32 (S102). The identification unit 2020 analyzes the first captured image 32 to identify the identification information of the target product 10 represented by the mark 12 (S104). The master acquisition unit 2040 acquires master information 50 corresponding to the identification information of the target product 10 (S106). The counting unit 2060 acquires the second captured image 42 (S108). The counting unit 2060 analyzes the second captured image 42 using the master information 50 to detect the target product 10 from the second captured image 42 (S110). Based on the detection result of the target product 10, the counting unit 2060 identifies the number of target products 10 included in the product group 20 (S110).

[0032] The processing flow performed by the counting device 2000 is not limited to that shown in Figure 4. For example, the timing of acquiring the second image 42 can be any timing before S110.

[0033] <Regarding the 10 target products and 20 product groups> Target product 10 is any product that is subject to counting. Here, target product 10 may be a single product that is sold as a unit, or it may be a package containing multiple products that are sold as units. For example, when multiple identical products are transported in one cardboard box, that cardboard box containing multiple products is treated as one target product 10.

[0034] The target product 10 is subjected to some kind of processing when multiple items are bundled together. Product group 20 is a collection of multiple target product 10 bundled together as a unit of processing. For example, multiple products may be placed on one pallet and inspected on a pallet basis. In this case, multiple target product 10 placed on one pallet are treated as product group 20. Furthermore, if multiple identical products are placed in one cardboard box, and multiple such cardboard boxes become a unit of processing (for example, if multiple cardboard boxes are stacked on a pallet and inspected on a pallet basis), then the bundle of multiple cardboard boxes is treated as product group 20.

[0035] For example, product 10 is a product that is subject to inspection during receiving and shipping operations. Inspection during receiving or shipping may be performed on multiple identical products as a single unit (for example, on a pallet). The counting device 2000 is used to identify how many product 10s are included in a product group 20, which is a single unit of product 10s.

[0036] The product group 20 can be transported in any way when it is being imaged by the first camera 30 or the second camera 40. For example, the product group 20 can be transported on a conveyor belt. Alternatively, the product group 20 can be transported by being placed on a vehicle such as a forklift or on a manually operated cart.

[0037] However, the first camera 30 and the second camera 40 may also image the stationary product group 20. In this case, for example, the product group 20 is transported by any means to the location where the first camera 30 and the second camera 40 are installed, and is temporarily stopped at that location. After imaging is performed by the first camera 30 and the second camera 40, it is then transported to another location.

[0038] <About Mark 12> Mark 12 represents the identification information of the target product 10 included in the product group 20. Here, any mark that can represent identification information and is also captureable by the camera can be used as Mark 12. For example, Mark 12 is one of various codes obtained by encoding the identification information of the target product 10 (e.g., a barcode or a two-dimensional code). Alternatively, Mark 12 may be a string of characters (including a sequence of numbers) that represents the identification information of the target product 10.

[0039] The location where Mark 12 is placed varies. For example, Mark 12 can be placed at any location on each target product 10. Alternatively, Mark 12 may be placed at any location on an object used to group multiple target products 10 into a single product group 20, such as a pallet.

[0040] However, the mark 12 must be placed in a position where it can be imaged by the first camera 30. For example, if each target product 10 has a mark 12, at least one target product 10 must be in a state where the mark 12 attached to it can be imaged by the first camera 30. Therefore, it is advisable to establish operational rules for the arrangement of target products 10 in the product group 20 and for the transportation of the product group 20 so that at least one mark 12 is included in the imaging range of the first camera 30. For example, an operational rule could be established stating that "when transporting the product group 20, at least one target product 10 must be positioned so that the mark 12 is visible (facing outwards)." The same applies when the mark 12 is attached to pallets, etc.

[0041] The method of attaching Mark 12 to the target product 10 or pallet, etc., is optional. For example, Mark 12 can be printed on a sticker that can be attached to the target product 10 or pallet, etc. In this case, Mark 12 is attached to the target product 10 or pallet, etc., by attaching the sticker to the target product 10 or pallet, etc. Alternatively, Mark 12 may be attached by directly printing it on the outside of the target product 10 or pallet, etc. Mark 12 may also be attached by drawing it by hand instead of printing it.

[0042] <Regarding Camera 1, No. 30> The first camera 30 is used to capture an image of the mark 12. In other words, the first camera 30 can be any camera that can satisfy the condition that "identification information of the target product 10 can be identified from the mark 12 contained in the first captured image 32". For example, cameras that satisfy this condition can be those used in barcode readers or QR code (registered trademark) readers. Alternatively, the first camera 30 may be a general digital camera (for example, a digital camera installed in a mobile device). The first camera 30 may be a still camera that captures still images or a video camera that captures moving images.

[0043] The first camera 30 is installed in a position where it can capture images of the mark 12. For example, when the product group 20 is being transported, the first camera 30 is installed along the transport route (the path along which the product group 20 moves). More specifically, it is installed near a conveyor belt or near a passage where a forklift or the like moves. Alternatively, the first camera 30 may be installed on a forklift or the like used to transport the product group 20.

[0044] The timing of the first camera 30's imaging is arbitrary. For example, the first camera 30 may repeat imaging at regular intervals, regardless of whether or not the mark 12 is included within its imaging range. In this case, as will be described later, the identification unit 2020 attempts to detect the mark 12 from each of the multiple first imaged images 32, and if the mark 12 is detected, it identifies the identification information of the target product 10 from that mark 12.

[0045] In addition, for example, the first camera 30 may be configured to recognize when the mark 12 enters its imaging range and to take an image at that time. For example, when a group of goods 20 is being transported, an object detection sensor (hereinafter referred to as an object detection sensor) is installed on the transport path of the group of goods 20. The object detection sensor is connected to the first camera 30 so as to transmit a signal to the first camera 30 when it detects the passage of an object. The first camera 30 is also configured to take an image and generate a first captured image 32 when it receives a signal from the object detection sensor. Furthermore, the position of the object detection sensor is predetermined so that the mark 12 is included in the imaging range of the first camera 30 at the time the group of goods 20 is detected by the object detection sensor. In this way, the first captured image 32 generated by the first camera 30 can be made to include the mark 12.

[0046] <Regarding the second camera 40> The second camera 40 is used to image the product group 20 and count the target product 10 included within it. Therefore, any camera capable of generating a second image 42 containing the target product 10 in a manner recognizable by image analysis can be used as the second camera 40. For example, a general digital camera, such as the camera installed in a mobile device, can be used as the second camera 40.

[0047] Furthermore, it is preferable that the second camera 40 can capture a wider area compared to the first camera 30. For example, the first camera 30 is installed relatively close to the product group 20 so that most of its field of view is occupied by the mark 12. On the other hand, the second camera 40 is installed relatively far away from the product group 20 so that it can capture the entire product group 20. However, the first camera 30 does not necessarily have to be installed close to the product group 20, and may be installed in the same way as the second camera 40 so as to capture the entire product group 20.

[0048] The second camera 40 is installed in a position where it can capture images of the product group 20. For example, when the product group 20 is being transported, the second camera 40 may be installed along the transport route, similar to the first camera 30, or it may be mounted on a forklift or the like used to transport the product group 20.

[0049] Furthermore, the master information 50 obtained using the first image 32 is used for the analysis of the second captured image 42. For this reason, it is preferable that the first camera 30 be positioned to capture the product group 20 at an earlier timing than the second camera 40. For example, if both the first camera 30 and the second camera 40 are located on the movement path of the product group 20, it is preferable that the first camera 30 be positioned in front of the second camera 40 with respect to the direction of movement of the product group 20.

[0050] The timing at which the second camera 40 takes images varies. For example, the second camera 40 may take images repeatedly at regular intervals, regardless of whether the product group 20 is included within its imaging range. In this case, as will be described later, the identification unit 2020 attempts to detect the product group 20 from each of the multiple second captured images 42, and if the product group 20 is detected, it counts the target products 10.

[0051] In addition, the second camera 40 may be configured to recognize the timing when the product group 20 enters its imaging range and to take an image at that time. For example, when the product group 20 is being transported, the difference between the timing when the mark 12 enters the imaging range of the first camera 30 and the timing when the product group 20 enters the imaging range of the second camera 40 is known (for example, when the product group 20 is moving at a constant speed). Furthermore, by using the object detection sensor mentioned above, the first camera 30 may take an image at the timing when the mark 12 enters the imaging range of the first camera 30. In this case, the second camera 40 is configured to detect that the first camera 30 has taken an image and to take an image at the time when the known difference mentioned above has elapsed since the first camera 30 took the image. The fact that the first camera 30 has taken an image can be recognized, for example, by having the first camera 30 send a predetermined signal to the second camera 40 in response to the first camera 30 taking an image, and the second camera 40 receiving that signal. Furthermore, the second camera 40 may receive a signal from the aforementioned object detection sensor and take an image at a predetermined time (such as the known difference mentioned above) that has elapsed since the reception of the signal. In addition, for example, a separate object detection sensor may be provided to determine the timing of imaging by the second camera 40, in addition to the object detection sensor that determines the timing of imaging by the first camera 30.

[0052] Furthermore, the second camera 40 may perform imaging at a timing that avoids the influence of imaging by the first camera 30. Specifically, the first camera 30 is configured to perform imaging when light is illuminating it. The light used to image the mark 12 in this way may be suitable for imaging the mark 12, but may not be suitable for imaging the product group 20. For example, the light illuminating for imaging by the first camera 30 may cause color saturation in the second image 42, potentially reducing the recognition accuracy of the target product 10 included in the second image 42.

[0053] Therefore, the second camera 40 is configured to take images at a time when the effect of the light irradiated for imaging by the first camera 30 has disappeared (such as when the light irradiation has ended or after a predetermined time has elapsed after the light irradiation has ended). In this way, for imaging by the first camera 30 This prevents the light used from affecting the imaging by the second camera 40, thereby improving the accuracy of the process of recognizing the target product 10 from the second captured image 42, and consequently, the accuracy of counting the target product 10.

[0054] For example, the second camera 40 is configured to take images a predetermined time after the first camera 30 has taken images. This predetermined time is set to a length of time at which the light used for imaging by the first camera 30 no longer has any effect. In this case, the installation position of the second camera 40 is determined so that the product group 20 is included in the imaging range at the time when the predetermined time has elapsed after the first camera 30 has taken images.

[0055] The method for avoiding the influence of light used for imaging by the first camera 30 is not limited to the method described above. For example, the second camera 40 may be installed to image a portion of the product group 20 that is not affected by the light irradiated for imaging by the first camera 30 (i.e., a portion that is not illuminated by light).

[0056] <Acquisition of the first image 32: S102> The identification unit 2020 acquires the first captured image 32 (S102). The method by which the identification unit 2020 acquires the first captured image 32 is arbitrary. For example, the identification unit 2020 acquires the first captured image 32 by receiving the first captured image 32 transmitted from the first camera 30. Alternatively, the identification unit 2020 may acquire the first captured image 32 by accessing the storage device in which the first captured image 32 is stored. The storage device in which the first captured image 32 is stored may be provided either inside or outside the first camera 30.

[0057] As mentioned above, the first camera 30 can generate a first captured image 32 even when the mark 12 is not included in the imaging range. In this case, for example, the identification unit 2020 attempts to detect the mark 12 from each first captured image 32 generated by the first camera 30. If the mark 12 is detected from the first captured image 32, the identification unit 2020 converts the detected mark 12 into identification information and identifies that identification information as the identification information of the target product 10 included in that first captured image 32.

[0058] <Identification of the identification information for target product 10: S104> The identification unit 2020 identifies the identification information of the target product 10 using the first captured image 32 (S104). More specifically, the identification unit 2020 identifies the identification information of the target product 10 by performing image analysis on the marks 12 included in the first captured image 32.

[0059] For example, Mark 12 is one of several codes obtained by encoding the identification information of the target product 10. In this case, the identification unit 2020 detects Mark 12 from the first captured image 32 and identifies the identification information of the target product 10 by decoding the detected Mark 12. Here, existing technologies can be used to analyze and decode barcodes and two-dimensional codes contained in the image to identify the identification information represented by them (for example, a JAN (Japan Article Number) code represented by a barcode).

[0060] In addition, for example, mark 12 is a string representing the identification information of the target product 10. In this case, the identification unit 2020 detects mark 12 from the first captured image 32 and performs optical character recognition (OCR) on the detected mark 12 to identify the identification information of the target product 10 represented by mark 12. Here, existing technologies can be used for detecting strings from an image and recognizing the detected strings.

[0061] If Mark 12 is a string of characters, the first captured image 32 may also contain strings of characters other than Mark 12 (for example, a product name). In this case, for example, conditions for a string of characters that represents the identification information of the target product 10 (such as having a predetermined number of characters, having a predetermined prefix or suffix, or being enclosed in a figure of a predetermined shape) are predetermined. The identification unit 2020 then identifies the string that satisfies these conditions from among the multiple strings detected from the first captured image 32 as representing the identification information of the target product 10.

[0062] <Retrieving Master Information 50: S106> The master acquisition unit 2040 acquires master information 50 corresponding to the identification information of the identified target product 10 (S106). To do this, the product identification information and the master information 50 for that product are associated and stored in the master information storage device 60 in advance. The master acquisition unit 2040 searches the master information storage device 60 using the identification information of the target product 10 identified by the identification unit 2020, and acquires the master information 50 stored in association with that identification information.

[0063] Here, multiple product groups 20 of the same target product 10 may be subject to processing by the counting device 2000. For example, this may occur when multiple pallets containing the same target product 10 are delivered and inspected sequentially. In such cases, the master acquisition unit 2040 may use the same master information multiple times. Therefore, the master information acquired from the master information storage device 60 may be stored in a cache, and when this master information is used again, it may be retrieved from the cache to improve the efficiency of master information acquisition. The storage device that implements the cache only needs to have a shorter access time from the counting device 2000 than the master information storage device 60, and may be provided either inside or outside the counting device 2000.

[0064] <Acquisition of the second image 42: S108> The counting unit 2060 acquires the second image 42. The method by which the counting unit 2060 acquires the second image 42 is the same as the method by which the identification unit 2020 acquires the first image 32.

[0065] As mentioned above, the second camera 40 may repeatedly take images regardless of whether the product group 20 is included in the imaging range. In this case, the counting unit 2060 attempts to detect the product group 20 from each of the multiple second imaging images 42, identifies the second imaging image 42 that contains the product group 20, and uses that second imaging image 42 to count the target products 10. For example, a second imaging image 42 that does not contain the target products 10, or a second imaging image 42 in which part of the target products 10 is cut off (i.e., a second imaging image 42 that only contains a part of the outer surface of the product group 20 that should be included in the second imaging image 42), is treated as a second imaging image 42 that does not contain the product group 20.

[0066] <Detection of target product 10: S110> The counting unit 2060 uses the master information 50 acquired by the master acquisition unit 2040 to detect the target product 10 from the second captured image 42 (S110). Here, existing technologies can be used for the technique of recognizing a specific object from an image using a template image or image features.

[0067] Furthermore, in cases where multiple product groups 20 sequentially pass through the imaging range of the first camera 30 and the second camera 40 (for example, when these product groups 20 are sequentially moving along a conveyor belt), identification information of the target product 10 included in each product group 20 is identified. Therefore, the counting unit 2060 needs to identify the product group 20 included in the second image 42 that it is about to analyze from the identification information of the target product 10 obtained for each of the multiple product groups 20. In other words, the counting unit 2060 needs to associate the first image 32 and the second image 42, in which the same product group 20 is captured.

[0068] For example, the counting unit 2060 associates first image 32 and second image 42 images that capture the same group of products 20 based on the generation time of each first image 32 and each second image 42. For example, the counting unit 2060 associates first image 32 and second image 42 images whose generation times differ by a small amount (below a predetermined value). Then, the counting unit 2060 uses the master information obtained using the identification information of the target product 10 obtained from the first image 32 associated with the second image 42 to perform image analysis of the second image 42.

[0069] In addition, for example, as mentioned above, the first camera 30 and the second camera 40 are configured such that the first camera 30 transmits a signal to the second camera 40 when it takes an image, and the second camera 40 takes an image when it receives that signal. In this case, the first image 32 generated by the image that triggered the transmission of the signal may be associated with the second image 42 generated by the image taken using that signal as a trigger.

[0070] <Identifying the number of target items 10: S112> The counting unit 2060 identifies the number of target products 10 included in the product group 20 based on the result of recognizing the target products 10 in the second captured image 42 (S112). However, not all of the target products 10 included in the product group 20 are necessarily included in the second captured image 42. For example, in the case of a product group 20 where "four target products 10 are arranged vertically and three horizontally, and stacked in three layers," the target products 10 hidden inside cannot be captured by the second camera 40.

[0071] For example, the counting unit 2060 identifies the number of target products 10 included in the product group 20 based on the number and arrangement of target products 10 recognized from the second image 42, and predetermined arrangement rules for the arrangement of target products 10 in the product group 20. The following describes a specific method for identifying the number of target products 10 included in the product group 20.

[0072] Figure 5 illustrates a case where the product group 20 is imaged from one direction. In Figure 5, the plane parallel to the pallet's mounting surface is the xy plane, and the vertical downward direction is the z direction. The imaging direction of the second camera 40 is the y direction. Therefore, the second image 42 captures the product group 20 viewed from the xz plane.

[0073] When imaging is performed only from the y-direction, it is difficult to determine from the second image 42 how many target products 10 are arranged in the y-direction. Therefore, a rule is predetermined for the arrangement of the target products 10: "N target products 10 are arranged in the y-direction."

[0074] The counting unit 2060 analyzes the second captured image 42 to determine how many target products 10 exist in the xz plane view. Then, the counting unit 2060 multiplies the number of target products 10 in the xz plane view by the number N of target products 10 in the y direction as defined by the placement rules to determine the number of target products 10 included in the product group 20.

[0075] Alternatively, a second camera 40, which has the added function of generating a depth image in addition to the second captured image 42, may be used to image the product group 20 from one direction. In this case, for example, the counting unit 2060 uses the second captured image 42 and the depth image to determine the number of target products 10 as follows.

[0076] Figure 6 illustrates a case in which the product group 20 is imaged from one direction, and the second camera 40 generates a second image 42 and a depth image 44. In Figure 6, the imaging direction of the second camera 40 is the z direction. Therefore, the second camera 40 can obtain a second image 42 showing the arrangement of the target products 10 in an xy-plane view, and a depth image 44 representing the depth of each position in the z direction. In the depth image 44, the intensity of the color of each pixel represents the depth (the distance to the object corresponding to that pixel).

[0077] The counting unit 2060 detects the target products 10 from the second image 42 to determine the arrangement of the target products 10 in an xy plane view. Furthermore, the counting unit 2060 uses the depth image 44 to determine how many target products 10 are stacked at each location where the target products 10 are placed. The counting unit 2060 then sums up the number of stacked target products 10 at each location to determine the total number of target products 10 included in the product group 20.

[0078] The method for determining how many objects are stacked at each location using the depth image 44 is as follows. First, by using the depth image 44, the distance D from the second camera 40 to the pallet's mounting surface and the distance a to the topmost target product 10 at a certain location P can be determined. In addition, the z-thickness b of the target product 10 is known in advance. Using this information, the number of target products 10 stacked at location P can be calculated as (Da) / b. Note that the information indicating the z-thickness of the target product 10 is stored in a memory device in association with the identification information of the target product 10, along with master information 50, for example.

[0079] Alternatively, instead of using depth images 44 to determine the number of target products 10 in the depth direction, multiple second cameras 40 may be installed, and multiple second images 42 taken from different directions of the product group 20 may be used. When two second cameras 40 are installed, for example, two cameras may be installed perpendicular to the side in the direction of movement of the product group 20, or two cameras may be installed diagonally in front of or behind the direction of movement of the product group 20. When three second cameras 40 are installed, for example, two second cameras 40 may be installed perpendicular to the side in the direction of movement of the product group 20, and one second camera 40 may be installed diagonally in front of or diagonally behind the direction of movement of the product group 20.

[0080] Here, the method of imaging the product group 20 from different directions using multiple second cameras 40 has the advantage of making it easier to detect situations where products other than the target product 10 are mistakenly mixed in, by recognizing the target product 10 in multiple second images 42 in which different surfaces of the product group 20 are captured. If it is detected that products other than the target product 10 are mixed in in this way, an error notification is output, for example, as will be explained in Embodiment 2 described later.

[0081] <How to utilize the results of identifying the number of target products (10 items)> There are various ways to utilize the information regarding the number of target products 10 included in the product group 20 identified by the counting unit 2060. The following are specific examples of how this information can be used.

[0082] <<Example of how to use it 1>> For example, the number of target products 10 included in product group 20 is used for managing product group 20. In this case, the counting unit 2060 stores the number of target products 10 included in product group 20 in a memory device, associating it with the identification information of product group 20 (e.g., pallet identification information). In this way, target products 10 that are received or shipped can be easily managed in units of product group 20 (e.g., pallet units).

[0083] The method for obtaining the identification information of product group 20 is arbitrary. For example, a mark representing the identification information of product group 20 can be placed at any location on product group 20 (for example, any location on the outer surface of the pallet). The counting device 2000 identifies the identification information of product group 20 from the mark representing the identification information of product group 20 in the same manner as the method for identifying the identification information of target product 10 from mark 12. The identification information of product group 20 may be captured by the first camera 30, by the second camera 40, or by another camera provided.

[0084] Furthermore, various types of information other than the number of target products 10 can be used as information managed in association with the identification information of product group 20. For example, the counting device 2000 further associates the identification information of the target products 10 included in product group 20 with the identification information of product group 20. In addition, for example, the counting device 2000 further acquires information regarding the attributes of the target products 10 included in product group 20 (hereinafter referred to as attribute information), associates this attribute information with the identification information of product group 20, and stores it in the memory device. As attribute information, any information can be used, such as the expiration date of the target product 10 (such as the best-before date or expiration date of food, or the expiration date of medicine).

[0085] For example, a mark (string of characters or code) representing the attribute information of the target product 10 is shown on the outer surface of the target product 10. The counting device 2000 identifies the attribute information of the target product 10 by performing image analysis on an image containing this mark. For example, if a string of characters representing the expiration date of the target product 10 is printed on the outer surface of the target product 10, the counting device 2000 identifies the expiration date of the target product 10 by analyzing an image containing that string of characters and recognizing it. The mark representing the attribute information of the target product 10 may be captured by the first camera 30, the second camera 40, or by any other image.

[0086] <<Example of how to use it 2>> When the number of target products 10 included in product group 20 is known, the counting device 2000 may use the number of target products 10 identified by the counting unit 2060 to check whether the number of target products 10 included in product group 20 is correct. In this case, the counting device 2000 obtains information indicating the number and range of target products 10 that should be included in product group 20, and compares that number with the number of target products 10 identified by the counting unit 2060 to determine whether the number of target products 10 identified by the counting unit 2060 is incorrect (does not match the correct number, or is outside the correct range). If the counting device 2000 determines that the number of target products 10 identified by the counting unit 2060 is incorrect, it outputs a notification indicating an error. Details regarding the notification indicating an error will be explained in Embodiment 2.

[0087] <Variation> In previous examples, the first camera 30 and the second camera 40 were provided as separate cameras. However, a single camera may be used as both the first camera 30 and the second camera 40. For example, the second camera 40 may be used as both the first camera 30 and the second camera 40 without providing a separate first camera 30. In this case, the identification unit 2020 identifies the identification information of the target product 10 from the mark 12 included in the second captured image 42. If multiple second cameras 40 are provided, the mark 12 should be included in the imaging range of at least one of the second cameras 40.

[0088] [Embodiment 2] Figure 7 is a block diagram illustrating the functional configuration of the counting device 2000 of Embodiment 2. Except for the points described below, the counting device 2000 of Embodiment 2 has the same functions as the counting device 2000 of Embodiment 1.

[0089] The counting device 2000 of Embodiment 2 has a notification unit 2080. The notification unit 2080 outputs a notification indicating an error (hereinafter referred to as an error notification). The types of errors output by the notification unit 2080 will be described below.

[0090] <Example of error notification 1> For example, the identification unit 2020 may not be able to identify the identification information of the target product 10 from the first captured image 32. In this case, the notification unit 2080 outputs an error notification indicating that the identification information of the target product 10 could not be identified. Here, the cases in which the identification information of the target product 10 cannot be identified are: 1) the case in which the mark 12 is not detected from the first captured image 32, and 2) the case in which the mark 12 detected from the first captured image 32 cannot be converted into the identification information of the target product 10. Therefore, the error notification may also include information indicating which of the two cases it is. The error notification may also include the first captured image 32.

[0091] Similar to the case where the identification information of the target product 10 could not be identified, an error notification may be output if the identification information of the target product 10 is invalid. For example, if it is known in advance that the target product 10 to be inspected is one of three specific types of products, and the identification information of a product that does not fall under any of those three types of products is identified by the identification unit 2020, then the identification information of the identified target product 10 can be said to be invalid. Therefore, the notification unit 2080 is configured to output an error notification even when the identification information of the target product 10 is invalid in this way.

[0092] <Example of error notification 2> The counting unit 2060 recognizes the target product 10 from the second image 42 using master information. However, it may not be able to recognize an object included in the product group 20 as the target product 10. This can happen, for example, if the counting unit 2060 uses the master information 50 of the target product 10 to calculate the probability that the product included in the second image 42 is the target product 10 (such as the similarity between the image features of the product included in the second image 42 and the image features shown in the master information 50 of the target product 10), and determines that the probability is below a predetermined value. This can occur if part of the second image 42 is unclear, or if an incorrect product is included in the product group 20.

[0093] Therefore, the notification unit 2080 outputs an error notification indicating that the product group 20 contains an object that cannot be recognized as the target product 10. It is preferable that this error notification include a second captured image 42 containing the object that could not be recognized as the target product 10. In addition, information that highlights the object that could not be recognized as the target product 10 (for example, a frame surrounding the object) may be added to this second captured image 42.

[0094] <Example of error notification 3> The counting device 2000 may acquire attribute information of the target product 10, as described above. In this case, the notification unit 2080 may output an error notification indicating that it was not possible to acquire attribute information of the target product 10 (for example, if an error occurred during OCR processing).

[0095] <Example of error notification 4> As described in "Example of Usage Method 2" of Embodiment 1, the counting device 2000 may also check whether the number of target products 10 identified by the counting unit 2060 is correct. In this case, if the notification unit 2080 determines that the number of target products 10 is incorrect, it outputs an error notification indicating that fact. This error notification may also indicate the number of target products 10 identified by the counting unit 2060 and a predetermined correct number of target products 10 or a range thereof.

[0096] <Regarding the output destination of error notifications> The destination of the error notification is arbitrary. For example, the error notification may be sent to the operator's terminal. For example, when the operator receives the above error notification, they can check the error status by viewing the notification. Alternatively, the operator may use images included in the error notification to provide correct information to the counting device 2000. For example, suppose an error notification is output indicating that the identification information and attribute information of target product 10 cannot be identified when it is represented as a string. In this case, the operator identifies the identification information and attribute information of target product 10 by visually checking the image included in the error notification. The operator then inputs the identified and correct identification information and attribute information of target product 10 into the terminal and sends it to the counting device 2000. If this information is managed in association with the identification information of product group 20, the information entered by the operator can be used to manage the product group 20 in association with the correct information.

[0097] The destination of the error notification is not limited to the operator's terminal; for example, it may be output to a display device connected to the counting device 2000, or stored as an error log in a storage device.

[0098] <Example of hardware configuration> The hardware configuration of the counting device 2000 in Embodiment 2 is similar to that of the counting device 2000 in Embodiment 1, as shown in Figure 3, for example. However, the storage device 1080 in Embodiment 2 stores programs that implement each of the functions of the counting device 2000 in Embodiment 2.

[0099] Some or all of the above embodiments may also be described as follows, but are not limited to the following: 1. An identification unit that identifies the identification information represented by the mark by analyzing a first image containing a mark representing the identification information of the target product, A master acquisition unit that acquires master information corresponding to the aforementioned identification information, A counting device comprising: a counting unit that analyzes a second image containing a group of products consisting of multiple target products using the acquired master information to detect the target products from the second image and, based on the detection result, determines the number of target products included in the product group. 2. The counting device according to 1, wherein the first captured image and the second captured image are the same image. 3. The first captured image is generated by the first camera. The counting device according to 1, wherein the second image is generated by a second camera different from the first camera. 4. The first camera and the second camera are installed on the movement path of the target product. The counting device according to 3, wherein the first camera is located on the front side with respect to the direction of movement of the target product, compared to the second camera. 5. The first camera is configured to perform imaging while light is illuminating it and generate the first image. The counting device according to 3. or 4., wherein the second camera is controlled to take images at times when the light is not illuminating the device. 6. A counting device according to any one of 1 to 5, further comprising a notification unit that outputs a notification indicating that the second image contains products in which the probability of being the target product is below a threshold, when such products are included in the second image. 7. The counting device according to any one of 1 to 6, wherein the counting unit identifies the number of the target products included in the product group based on the arrangement of the target products in the second captured image and the rules for the arrangement of products in the product group. 8. A control method performed by a computer, A first imaging image containing a mark representing the identification information of the target product is analyzed to identify the identification information represented by the mark, and A master acquisition step to acquire master information corresponding to the aforementioned identification information, A control method comprising: a counting step of detecting the target products from a second image by analyzing a second image containing a group of products consisting of multiple target products using the acquired master information, and determining the number of target products included in the product group based on the detection result. 9. The control method according to 8, wherein the first captured image and the second captured image are the same image. 10. The first captured image is generated by the first camera. The control method according to 8. wherein the second captured image is generated by a second camera different from the first camera. 11. The first camera and the second camera are installed on the movement path of the target product, The control method according to 10, wherein the first camera is located closer to the user with respect to the direction of movement of the target product compared to the second camera. 12. The first camera is configured to perform imaging while light is illuminating it and generate the first image. The control method according to 10. or 11., wherein the second camera is controlled to take images at a time when the light is not illuminating it. 13. A control method according to any one of 8 to 12, further comprising a notification step of outputting a notification that the second image contains products in which the probability of being the target product is below a threshold, if the products included in the second image contain products in which the probability is below a threshold. 14. The control method according to any one of 8 to 13, wherein in the counting step, the number of target products included in the product group is determined based on the arrangement of the target products in the second captured image and the rules for the arrangement of products in the product group. 15. A program that causes a computer to execute any one of the control methods described in 8. through 13. [Explanation of Symbols]

[0100] 10 Target Products 12 marks 20 product groups 30. Camera 1 32 First image 40. Second camera 42 Second image 44 Depth image 50 Master Information 60 Master Information Storage Device 1000 calculator 1020 Bus 1040 processor 1060 memory 1080 Storage Devices 1100 Input / Output Interface 1120 Network Interface 2000 Counting device 2020 Specific Department 2040 Master Data Acquisition Department 2060 Counting Department 2080 Notification Department

Claims

1. An identification unit identifies the product identified by the mark based on a mark detected on some of the products among multiple products included in the captured image, A counting unit identifies target products that are determined to be the same as the identified product from the captured image, and counts the number of identified target products. The system includes a notification unit that outputs a notification indicating an error if the captured image contains an object that could not be recognized as the target product. The counting unit is a counting device that counts the number of identified target products based on a template image of the product identified by the identification unit, or on the feature quantities on the image of the identified product.

2. The aforementioned mark is a barcode or a two-dimensional code. The counting device according to claim 1.

3. An identification unit that identifies the product identified by a mark based on a mark detected from some of the products among a plurality of products included in the captured image, A counting unit identifies target products that are determined to be the same as the identified product from the captured image, and counts the number of identified target products. The system includes a notification unit that outputs a notification indicating an error if the captured image contains an object that could not be recognized as the target product. The notification unit is a counting device that outputs a notification when the captured image contains products for which the certainty of being the target product is below a threshold.

4. The counting device according to any one of claims 1 to 3, wherein the notification unit outputs a notification indicating an error when there is a product among the products included in the captured image from which attribute information of the target product could not be obtained.

5. The counting device according to claim 3 or 4, wherein the notification unit outputs the notification to a terminal operated by the operator.

6. An identification unit that identifies the product identified by a mark based on a mark detected from some of the products among a plurality of products included in an image, A counting unit identifies target products that are determined to be the same as the identified product from the captured image, and counts the number of identified target products. The system includes a notification unit that outputs a notification indicating an error if the captured image contains an object that could not be recognized as the target product. The counting unit is a counting device that determines the number of target products included in a product group based on the arrangement of the target products in the captured image and the rules for the arrangement of products in a product group consisting of a plurality of target products.

7. A control method performed by a computer, A selection step of identifying the product identified by the mark based on a mark detected on some of the products among multiple products included in the captured image, A counting step in which, from the captured image, a target product is identified as the same product as the identified product, and the number of identified target products is counted. The system includes a notification step that outputs a notification indicating an error if the captured image contains an object that could not be recognized as the target product, A control method for counting the number of identified target products in the counting step, based on the template image of the product identified in the identification step, or on the feature quantities on the image of the identified product.

Images