Counting device, and control method

Abstract

To perform specification of the number of commodities in a short period of time.SOLUTION: A counting device 2000 specifies identification information of an object commodity 10 from a mark 12 included in a first photographed image 32. The counting device 2000 acquires master information 50 corresponding to the object commodity 10. The counting device 2000 detects the object commodity 10 from a second photographed image 42 by analyzing the second photographed image 42 using the master information 50, and on the basis of the detection result, specifies the number of object commodities 10 contained in a commodity group 20.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to item counting. [Background technology]

[0002] In inspection work, etc., products to be inspected are managed. For example, Patent Document 1 discloses a technology for analyzing an image of a package to be inspected to identify the package name and the number of layers, identifying a stacking method corresponding to the package name, and identifying the number of packages based on the stacking method and the number of layers. The stacking method is determined in advance for each product. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2013-067499 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the invention of Patent Document 1, the feature amounts of various packages are stored in a database in advance, and the name of the package is identified by comparing the feature amount obtained from an image of the package with the feature amount of each package stored in the database. However, this method takes a long time to identify the name of the package. As a result, the time required to identify the number of packages is also long.

[0005] The present invention has been made in consideration of the above-mentioned problems, and one of the objects thereof is to provide a technique for specifying the number of products in a short period of time. [Means for solving the problem]

[0006] The counting device of the present invention has 1) an identification unit that identifies products identified by marks based on marks detected from some of the multiple products included in an captured image, and 2) a counting unit that identifies target products from the captured image that are determined to be the same product as the identified product based on information about the images of the identified products, and counts the number of identified target products.

[0007] The control method of the present invention is executed by a computer and includes the steps of: 1) identifying products identified by marks based on marks detected from some of a plurality of products included in a captured image; and 2) identifying target products that are determined to be the same product as the identified product from the captured image based on information about the images of the identified products, and counting the number of the identified target products.

[0008] The program of the present invention causes a computer to execute each step of the control method of the present invention. Effect of the Invention

[0009] According to the present invention, a technique for identifying the number of products in a short period of time is provided. [Brief description of the drawings]

[0010] [Figure 1] 3A to 3C are diagrams conceptually illustrating an example of the operation of the counting device according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating a functional configuration of a counting device. [Diagram 3] FIG. 13 is a diagram illustrating a computer for implementing a counting device. [Figure 4] 4 is a flowchart illustrating a process executed by the counting device of the first embodiment. [Diagram 5] FIG. 13 is a diagram illustrating a case in which an image of a group of products is captured from one direction. [Figure 6]FIG. 13 is a diagram illustrating a case in which a group of products is imaged from one direction and a second camera generates a second captured image and a depth image. [Figure 7] FIG. 11 is a block diagram illustrating a functional configuration of a counting device according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all drawings, similar components are given similar reference numerals, and descriptions thereof will be omitted as appropriate. In addition, unless otherwise specified, in each block diagram, each block represents a configuration in functional units, not a configuration in hardware units. In the following description, unless otherwise specified, various predetermined values ​​(such as threshold values) are stored in advance in a storage device accessible from the functional configuration unit that uses the values.

[0012] [Embodiment 1] <Summary> Fig. 1 is a diagram conceptually illustrating an operation of the counting device 2000 according to the first embodiment. Here, the operation of the counting device 2000 described using Fig. 1 is an example for easily understanding the counting device 2000, and does not limit the operation of the counting device 2000. Details and variations of the operation of the counting device 2000 will be described later.

[0013] The counting device 2000 identifies the number of target products 10 included in a product group 20. A product group 20 is one or more target products 10 collected in one place. For example, a plurality of target products 10 stacked on a pallet during inspection work at the time of shipment or arrival of the target products 10 are treated as a product group 20. An overview of a method in which the counting device 2000 identifies the number of target products 10 included in the product group 20 will be described below.

[0014] The counting device 2000 acquires a first captured image 32. The first captured image 32 is generated by a first camera 30. The first camera 30 is provided so that the mark 12 is within its imaging range. The mark 12 is any mark (e.g., a barcode) that represents identification information of the target product 10. The counting device 2000 analyzes the first captured image 32 to identify the identification information of the target product 10 from the mark 12 included in the first captured image 32.

[0015] The counting device 2000 acquires master information 50 associated with the identification information of the target product 10. The master information 50 is used for image analysis to detect the target product 10. The master information 50 is, for example, a template image of the target product 10, image features of the target product 10 (feature amounts on the image), etc.

[0016] The counting device 2000 acquires a second captured image 42 including the group of products 20. The second captured image 42 is generated by the second camera 40. The second camera 40 is provided so that the imaging range of the second camera 40 includes the group of products 20. The counting device 2000 detects the target products 10 included in the second captured image 42 by analyzing the second captured image 42 using the master information 50. The counting device 2000 identifies the number of target products 10 included in the group of products 20 based on the detection result.

[0017] Here, one second captured image 42 may not include all of the target products 10 included in the product group 20, but may include only some of the target products 10. For example, if the target products 10 are stacked on a pallet, capturing an image of the product group 20 from only one direction may result in some of the target products 10 being hidden by other target products 10.

[0018] Therefore, for example, the counting device 2000 determines the number of target products 10 included in the product group 20 based on the number of target products 10 detected from the second captured image 42 and a priori rules regarding the arrangement of the target products 10 in the product group 20. As another example, the counting device 2000 acquires second captured images 42 from each of a plurality of second cameras 40 installed so as to capture images of the product group 20 from different directions, and determines the number of target products 10 included in the product group 20 based on the number of target products 10 detected from each second captured image 42. A more specific method for determining the number of target products 10 will be described later.

[0019] <Example of action and effect> The counting device 2000 of this embodiment identifies the identification information of the target products 10 by using the marks 12 included in the first captured image 32. The counting device 2000 also acquires master information 50 corresponding to the identification information of the identified target products 10, and detects the target products 10 from the second captured image 42 by using the master information 50. The counting device 2000 then identifies the number of target products 10 included in the product group 20 based on the detection results of the target products 10.

[0020] In this way, in the counting device 2000, when detecting the target products 10 from the second captured image 42 in order to identify the number of the target products 10, the master information 50 corresponding to the identification information of the target products 10 is used. Therefore, when analyzing the second captured image 42, matching using master information of products other than the target products 10 is not performed. Therefore, compared to the invention disclosed in Patent Document 1, the time required for the process of detecting the target products 10 from the image is shorter. Therefore, the target products 10 can be counted in a shorter time. In addition, using the master information 50 of the target products 10 has the effect of reducing the occurrence of errors in product detection (for example, a product included in the second captured image 42 being determined to be a product other than the target product 10).

[0021] The counting device 2000 will now be described in more detail.

[0022] <Example of functional configuration> 2 is a block diagram illustrating an example of a functional configuration of the counting device 2000. The counting device 2000 has an identification unit 2020, a master acquisition unit 2040, and a counting unit 2060. The identification unit 2020 analyzes the first captured image 32 to identify the identification information of the target product 10 represented by the mark 12. The master acquisition unit 2040 acquires master information 50 corresponding to the identification information of the target product 10. The counting unit 2060 detects the target product 10 from the second captured image 42 using the master information 50, and identifies the number of target products 10 included in the product group 20 based on the detection result.

[0023] <Example of hardware configuration of counting device 2000> Each functional component of the counting device 2000 may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). The case where each functional component of the counting device 2000 is realized by a combination of hardware and software will be further described below.

[0024] 3 is a diagram illustrating a computer 1000 for implementing the counting device 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 1000 may be a portable computer such as a smartphone or a tablet terminal.

[0025] The computer 1000 may be a dedicated computer designed to realize the counting device 2000, or may be a general-purpose computer. In the latter case, for example, by installing a specific application on the computer 1000, the computer 1000 realizes each function of the counting device 2000 (the computer 1000 operates as the counting device 2000). The application is constituted by a program for realizing the functional components of the counting device 2000.

[0026] The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection.

[0027] The processor 1040 is one of various processors such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, or a read only memory (ROM) or the like.

[0028] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0029] The network interface 1120 is an interface for connecting the computer 1000 to a communication network. This communication network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). For example, the first camera 30 and the second camera 40 are connected to the network interface 1120 via the communication network. However, the first camera 30 and the second camera 40 may be communicatively connected to the computer 1000 via a means other than the network interface 1120 (for example, via the input / output interface 1100), or may not be communicatively connected to the computer 1000. In the latter case, for example, each camera and the computer 1000 are set to be able to access a common storage device. Then, the computer 1000 acquires the first captured image 32 and the second captured image 42 via the storage device.

[0030] The storage device 1080 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the counting device 2000. The processor 1040 reads this program into the memory 1060 and executes it to realize each functional component of the counting device 2000.

[0031] <Processing flow> FIG. 4 is a flowchart illustrating the flow of a process executed by the counting device 2000 of the first embodiment. The identification unit 2020 acquires the first captured image 32 (S102). The identification unit 2020 analyzes the first captured image 32 to identify the identification information of the target product 10 represented by the mark 12 (S104). The master acquisition unit 2040 acquires master information 50 corresponding to the identification information of the target product 10 (S106). The counting unit 2060 acquires the second captured image 42 (S108). The counting unit 2060 analyzes the second captured image 42 using the master information 50 to detect the target product 10 from the second captured image 42 (S110). The counting unit 2060 identifies the number of target products 10 included in the product group 20 based on the detection result of the target products 10 (S110).

[0032] The flow of the process executed by the counting device 2000 is not limited to that shown in Fig. 4. For example, the timing for acquiring the second captured image 42 can be any timing before S110.

[0033] <About Target Product 10 and Product Group 20> The target product 10 is any product that is the subject of counting. Here, the target product 10 may be a single product that is a unit of sale, or may be a package of multiple products that are units of sale. For example, when multiple identical products are transported in a single cardboard box, the single cardboard box containing the multiple products is treated as one target product 10.

[0034] The target products 10 are grouped together and are subject to some kind of processing. A product group 20 is a collection of multiple target products 10 grouped together in this way as a processing unit. For example, multiple products may be placed on a single pallet, and inspection may be carried out on a pallet basis. Thus, multiple target products 10 placed on a single pallet in this way are treated as a product group 20. Note that when multiple identical products are placed in a single cardboard box, and the cardboard box is collectively used as a processing unit (for example, when multiple cardboard boxes are stacked on a pallet, and inspection is carried out on a pallet basis), such a group of multiple cardboard boxes is treated as a product group 20.

[0035] For example, the target product 10 is a product that is subject to inspection during receiving or shipping operations. Inspection operations during receiving or shipping may be performed with a group of the same products as one unit (for example, per pallet). The counting device 2000 is used to determine how many target products 10 are included in a product group 20, which is one unit of such target products 10 that are treated as one unit.

[0036] The product group 20 may be transported by any method when being imaged by the first camera 30 and the second camera 40. For example, the product group 20 may be transported on a belt conveyer. Alternatively, for example, the product group 20 may be transported by being placed on a vehicle such as a forklift or on a hand-powered cart.

[0037] However, the first camera 30 and the second camera 40 may capture images of the product group 20 in a stationary state. In this case, for example, the product group 20 is transported by any method to the location where the first camera 30 and the second camera 40 are installed, and stopped at that location for a while. Then, after the images are captured by the first camera 30 and the second camera 40, the product group 20 is transported again to another location.

[0038] <About Mark 12> The mark 12 represents the identification information of the target product 10 included in the product group 20. Any mark that can represent the identification information and can be captured by a camera can be used as the mark 12. For example, the mark 12 is a variety of codes (e.g., a barcode or a two-dimensional code) obtained by encoding the identification information of the target product 10. Alternatively, for example, the mark 12 may be a character string (including a numeric string) that represents the identification information of the target product 10.

[0039] The marks 12 may be attached in various locations. For example, the marks 12 may be attached to any location on each target product 10. Alternatively, for example, the marks 12 may be attached to any location on an object, such as a pallet, that is used to group multiple target products 10 into one product group 20.

[0040] However, the mark 12 must be affixed at a position that can be imaged by the first camera 30. For example, when the mark 12 is affixed to each target product 10, the mark 12 affixed to at least one target product 10 must be in a state in which it can be imaged by the first camera 30. Therefore, for example, it is publicly required to establish operational rules for the arrangement of the target products 10 in the product group 20 and the transportation of the product group 20 so that at least one mark 12 is included in the imaging range of the first camera 30. For example, an operational rule may be established that "when transporting the product group 20, at least one target product 10 must be arranged so that the mark 12 is visible (facing outward)." The same applies when the mark 12 is affixed to a pallet or the like.

[0041] Here, the method of attaching the mark 12 to the target product 10, a pallet, or the like is arbitrary. For example, the mark 12 is printed on a sticker that can be attached to the target product 10, a pallet, or the like. In this case, the mark 12 is attached to the target product 10, a pallet, or the like by attaching the sticker to the target product 10, a pallet, or the like. In addition, the mark 12 may be attached by directly printing the mark 12 on the outer surface of the target product 10, a pallet, or the like. Furthermore, the mark 12 may be attached by hand drawing instead of printing.

[0042] <Regarding the first camera 30> The first camera 30 is used to capture the mark 12. In other words, any camera that can satisfy the condition that "the identification information of the target product 10 can be specified from the mark 12 included in the first captured image 32" can be used as the first camera 30. For example, a camera that satisfies such a condition can be a camera used for a barcode reader or a QR code (registered trademark) reader. Alternatively, for example, a general digital camera (for example, a digital camera provided in a mobile terminal) may be used as the first camera 30. Note that the first camera 30 may be a still camera that captures still images, or a video camera that captures moving images.

[0043] The first camera 30 is installed at a position where it can capture an image of the mark 12. For example, when the group of products 20 is transported, the first camera 30 is installed on the transport path (the path along which the group of products 20 move). More specifically, the first camera 30 is installed near a belt conveyor or near an aisle along which a forklift or the like moves. The first camera 30 may also be installed on a forklift or the like used to transport the group of products 20.

[0044] The timing at which the first camera 30 captures images is arbitrary. For example, the first camera 30 may repeatedly capture images at regular intervals regardless of whether the mark 12 is included in its capture range. In this case, as described below, the identification unit 2020 attempts to detect the mark 12 from each of the multiple first captured images 32, and if the mark 12 is detected, identifies the identification information of the target product 10 from the mark 12.

[0045] Alternatively, for example, the first camera 30 may be configured to grasp the timing when the mark 12 enters its imaging range and capture an image at that timing. For example, when the product group 20 is transported, a sensor that detects the passage of an object (hereinafter, object detection sensor) is provided on the transport path of the product group 20. The object detection sensor is connected to the first camera 30 so as to transmit a signal to the first camera 30 when it detects the passage of an object. In addition, the first camera 30 is set to capture an image and generate a first captured image 32 when it receives a signal from the object detection sensor. Furthermore, the position of the object detection sensor is previously determined so that the mark 12 is included in the imaging range of the first camera 30 at the timing when the product group 20 is detected by the object detection sensor. In this way, the mark 12 can be included in the first captured image 32 generated by the first camera 30.

[0046] <Regarding the second camera 40> The second camera 40 is used to capture an image of the product group 20 and count the target products 10 contained therein. Therefore, any camera capable of generating a second captured image 42 in which the target products 10 are included in a manner that can be recognized by image analysis can be used as the second camera 40. For example, a general digital camera such as a camera provided in a mobile terminal can be used as the second camera 40.

[0047] It is preferable that the second camera 40 can capture a wider range than the first camera 30. For example, the first camera 30 is installed relatively close to the group of products 20 so that the marks 12 occupy most of its angle of view. On the other hand, the second camera 40 is installed relatively far from the group of products 20 so that it can capture an image of the entire group of products 20. However, the first camera 30 does not necessarily have to be installed close to the group of products 20, and may be installed so as to capture an image of the entire group of products 20, similar to the second camera 40.

[0048] The second camera 40 is installed in a position capable of capturing an image of the product group 20. For example, when the product group 20 is transported, the second camera 40, like the first camera 30, is installed on the transport route or on a forklift or the like used to transport the product group 20.

[0049] Note that master information 50 obtained using the first captured image 32 is used to analyze the second captured image 42. Therefore, it is preferable that the first camera 30 is provided at a position where it can capture an image of the commodity group 20 at an earlier timing than the second camera 40. For example, when both the first camera 30 and the second camera 40 are provided on the movement path of the commodity group 20, it is preferable that the first camera 30 is provided in front of the second camera 40 in the movement direction of the commodity group 20.

[0050] There are various timings for the second camera 40 to capture images. For example, the second camera 40 may repeatedly capture images at regular intervals regardless of whether the commodity group 20 is included within its capture range. In this case, as described below, the identification unit 2020 attempts to detect the commodity group 20 from each of the multiple second captured images 42, and counts the target commodities 10 when the commodity group 20 is detected.

[0051] Alternatively, for example, the second camera 40 may be configured to grasp the timing when the group of products 20 enters its imaging range and capture an image at that timing. For example, when the group of products 20 is transported, it is assumed that the difference between the timing when the mark 12 enters the imaging range of the first camera 30 and the timing when the group of products 20 enters the imaging range of the second camera 40 is known (for example, when the group of products 20 moves at a constant speed). In addition, it is assumed that the first camera 30 captures an image at the timing when the mark 12 enters the imaging range of the first camera 30 by using the object detection sensor described above. In this case, the second camera 40 is configured to detect that an image has been captured by the first camera 30 and capture an image at a timing when the known difference described above has elapsed from the time when the image was captured by the first camera 30. The fact that an image has been captured by the first camera 30 can be grasped, for example, by transmitting a predetermined signal from the first camera 30 to the second camera 40 in response to the first camera 30 capturing an image, and the second camera 40 receiving the signal. The second camera 40 may receive a signal from the object detection sensor described above, and capture an image when a predetermined time (such as the known difference described above) has elapsed since the signal was received. Alternatively, for example, an object detection sensor that determines the timing of image capture by the second camera 40 may be provided in addition to the object detection sensor that determines the timing of image capture by the first camera 30.

[0052] The second camera 40 may capture an image at a timing that avoids the influence of the image captured by the first camera 30. Specifically, the first camera 30 is configured to capture an image in a situation where light is irradiated. The light used to capture the image of the mark 12 in this manner may be suitable for capturing the image of the mark 12, but may not be suitable for capturing the image of the commodity group 20. For example, the light irradiated for capturing the image by the first camera 30 may cause color saturation in the second captured image 42, which may reduce the recognition accuracy of the target commodity 10 included in the second captured image 42.

[0053] Therefore, the second camera 40 is configured to capture an image when the influence of the light irradiated for the image capture by the first camera 30 is gone (when the light irradiation ends, or when a predetermined time has elapsed since the light irradiation ends, etc.). Since the light used can be prevented from affecting the imaging by the second camera 40, the accuracy of the process of recognizing the target products 10 from the second captured image 42, and ultimately the accuracy of counting the target products 10, can be improved.

[0054] For example, the second camera 40 is configured to capture an image a predetermined time after the first camera 30 captures an image. This predetermined time is set to a length that eliminates the influence of light used for capturing an image by the first camera 30. In this case, the installation position of the second camera 40 is determined so that the product group 20 is included in the capture range at the timing when the above-mentioned predetermined time has elapsed after the first camera 30 captures an image.

[0055] The method of avoiding the influence of light used for imaging by the first camera 30 is not limited to the above-mentioned method. For example, the second camera 40 may be installed so as to capture an image of a portion of the product group 20 that is not influenced by the light irradiated for imaging by the first camera 30 (i.e., a portion not irradiated with light).

[0056] <Acquisition of the first captured image 32: S102> The identification unit 2020 acquires the first captured image 32 (S102). The method by which the identification unit 2020 acquires the first captured image 32 is arbitrary. For example, the identification unit 2020 acquires the first captured image 32 by receiving the first captured image 32 transmitted from the first camera 30. Alternatively, for example, the identification unit 2020 may acquire the first captured image 32 by accessing a storage device in which the first captured image 32 is stored. Note that the storage device in which the first captured image 32 is stored may be provided either inside or outside the first camera 30.

[0057] Here, as described above, the first camera 30 may generate the first captured image 32 even when the mark 12 is not included in the imaging range. In this case, for example, the identification unit 2020 attempts to detect the mark 12 from each of the first captured images 32 generated by the first camera 30. Then, when the mark 12 is detected from the first captured image 32, the identification unit 2020 converts the detected mark 12 into identification information and identifies the identification information as the identification information of the target product 10 included in the first captured image 32.

[0058] <Identification of identification information of target product 10: S104> The identification unit 2020 identifies the identification information of the target product 10 by using the first captured image 32 (S104). More specifically, the identification unit 2020 identifies the identification information of the target product 10 by performing image analysis on the mark 12 included in the first captured image 32.

[0059] For example, the mark 12 is a variety of codes obtained by encoding the identification information of the target product 10. In this case, the identification unit 2020 detects the mark 12 from the first captured image 32 and identifies the identification information of the target product 10 by decoding the detected mark 12. Here, existing technology can be used for analyzing and decoding a barcode or two-dimensional code included in the image to identify the identification information represented by the code (for example, a JAN (Japan Article Number) code represented by a barcode).

[0060] Alternatively, for example, the mark 12 is a character string representing the identification information of the target product 10. In this case, the identification unit 2020 detects the mark 12 from the first captured image 32 and performs character string recognition processing (OCR: Optical Character Recognition) on the detected mark 12 to identify the identification information of the target product 10 represented by the mark 12. Here, existing technology can be used as a technology for detecting a character string from an image and recognizing the detected character string.

[0061] In addition, when the mark 12 is a character string, a character string other than the mark 12 (e.g., a product name) may also be included in the first captured image 32. In this case, for example, conditions for the character string representing the identification information of the target product 10 (such as a predetermined number of characters, having a predetermined prefix or suffix, or being surrounded by a figure of a predetermined shape) are determined in advance. Then, the identification unit 2020 identifies a character string that satisfies this condition among the multiple character strings detected from the first captured image 32 as one that represents the identification information of the target product 10.

[0062] <Getting master information 50: S106> The master acquisition unit 2040 acquires the master information 50 corresponding to the identification information of the identified target product 10 (S106). For this purpose, the identification information of the product and the master information 50 for that product are associated with each other and stored in advance in the master information storage device 60. The master acquisition unit 2040 searches the master information storage device 60 for the identification information of the target product 10 identified by the identification unit 2020, and acquires the master information 50 stored in association with the identification information.

[0063] Here, multiple product groups 20 for the same target product 10 may be processed by the counting device 2000. For example, multiple pallets loaded with the same target products 10 may be brought in and inspected one after another. In such a case, the master acquisition unit 2040 may use the same master information multiple times. Therefore, the master information acquired from the master information storage device 60 may be stored in a cache, and when the master information is used again, the master information may be acquired from the cache, thereby improving the efficiency of acquiring the master information. Note that the storage device realizing the cache may be one that requires less time for access from the counting device 2000 than the master information storage device 60, and may be provided either inside or outside the counting device 2000.

[0064] <Acquisition of second captured image 42: S108> The counting section 2060 acquires the second captured image 42. The method by which the counting section 2060 acquires the second captured image 42 is similar to the method by which the identification section 2020 acquires the first captured image 32.

[0065] Here, as described above, the second camera 40 may repeatedly capture images regardless of whether the commodity group 20 is included in the capture range. In this case, the counting unit 2060 attempts to detect the commodity group 20 from each of the multiple second captured images 42, identifies the second captured images 42 that include the commodity group 20, and counts the target commodities 10 using the second captured images 42. For example, a second captured image 42 that does not include the target commodities 10 or a second captured image 42 in which a part of the target commodities 10 is cut off (i.e., a second captured image 42 that includes only a part of the outer surface of the commodity group 20 that should be included in the second captured image 42) is treated as a second captured image 42 that does not include the commodity group 20.

[0066] <Detection of target product 10: S110> The counting unit 2060 detects the target product 10 from the second captured image 42 using the master information 50 acquired by the master acquisition unit 2040 (S110). Here, existing technology can be used for the technology of recognizing a specific object from an image using a template image, image features, or the like.

[0067] In cases where multiple product groups 20 pass sequentially through the imaging range of the first camera 30 and the second camera 40 (for example, where these product groups 20 are sequentially moving on a conveyer), the identification information of the target products 10 included in each product group 20 is identified. For this reason, the counting unit 2060 needs to identify the identification information of the target products 10 obtained for each of the multiple product groups 20 that has been identified for the product group 20 included in the second captured image 42 to be analyzed. In other words, the counting unit 2060 needs to associate the first captured image 32 and the second captured image 42 in which the same product group 20 has been captured.

[0068] For example, the counting unit 2060 associates the first captured image 32 and the second captured image 42 in which the same commodity group 20 is captured, based on the generation time of each first captured image 32 and the generation time of each second captured image 42. For example, the counting unit 2060 associates the first captured image 32 and the second captured image 42 in which the difference between the generation time of each is small (a predetermined value or less). Then, the counting unit 2060 uses master information obtained using the identification information of the target commodity 10 obtained from the first captured image 32 associated with the second captured image 42 for image analysis of the second captured image 42.

[0069] Alternatively, for example, as described above, the first camera 30 and the second camera 40 are configured so that the first camera 30 transmits a signal to the second camera 40 in response to capturing an image by the first camera 30, and the second camera 40 captures an image in response to receiving the signal. In this case, the first captured image 32 generated by capturing an image that triggers the transmission of the signal may be associated with the second captured image 42 generated by capturing an image triggered by the signal.

[0070] <Identifying the number of target products 10: S112> The counting unit 2060 identifies the number of target products 10 included in the product group 20 based on the result of recognizing the target products 10 in the second captured image 42 (S112). Here, all of the target products 10 included in the product group 20 are not necessarily included in the second captured image 42. For example, in the case of a product group 20 in which "four target products 10 are arranged vertically and three horizontally, stacked in three layers," the target products 10 hidden inside cannot be imaged by the second camera 40.

[0071] Therefore, for example, the counting unit 2060 identifies the number of target products 10 included in the product group 20 based on the number and arrangement of the target products 10 recognized from the second captured image 42 and predetermined arrangement rules for the arrangement of the target products 10 in the product group 20. A specific method for identifying the number of target products 10 included in the product group 20 will be described below.

[0072] Fig. 5 is a diagram illustrating a case where the group of products 20 is imaged from one direction. In Fig. 5, a plane parallel to the placement surface of the pallet is the xy plane, and the vertical downward direction is the z direction. The imaging direction of the second camera 40 is the y direction. Therefore, the second captured image 42 captures the group of products 20 viewed in the xz plane.

[0073] In this way, when imaging is performed only from the y direction, it is difficult to grasp how many target products 10 are lined up in the y direction from the second captured image 42. Therefore, a rule that "N target products 10 are lined up in the y direction" is determined in advance as a placement rule for the target products 10.

[0074] The counting unit 2060 identifies how many target products 10 are present in the xz planar view by analyzing the second captured image 42. Then, the counting unit 2060 identifies the number of target products 10 included in the product group 20 by multiplying the number of target products 10 in the xz planar view by the number N of target products 10 in the y direction defined by the arrangement rule.

[0075] The group of products 20 may be imaged from one direction by the second camera 40 that further has a function of generating a depth image in addition to the second captured image 42. In this case, for example, the counting unit 2060 identifies the number of target products 10 using the second captured image 42 and the depth image as follows.

[0076] Fig. 6 is a diagram illustrating a case where the product group 20 is imaged from one direction and the second camera 40 generates a second captured image 42 and a depth image 44. In Fig. 6, the image capturing direction of the second camera 40 is the z direction. Therefore, from the second camera 40, it is possible to obtain a second captured image 42 showing the arrangement of the target products 10 in the xy planar view and a depth image 44 showing the depth of each position in the z direction. In the depth image 44, the color density of each pixel represents the depth (the distance to the object corresponding to that pixel).

[0077] The counting unit 2060 detects the target products 10 from the second captured image 42, thereby identifying the arrangement of the target products 10 in the xy planar view. Furthermore, the counting unit 2060 uses the depth image 44 to identify how many target products 10 are stacked at the positions where each target product 10 is placed. Then, the counting unit 2060 totals the number of target products 10 stacked at each position to identify the number of target products 10 included in the product group 20.

[0078] The method of identifying how many objects are stacked at each position using the depth image 44 is, for example, as follows. First, by using the depth image 44, it is possible to identify the distance D from the second camera 40 to the placement surface of the pallet and the distance a to the target product 10 on the top shelf at a certain position P. In addition, the thickness b of the target products 10 in the z direction is known in advance. By using this information, the number of target products 10 stacked at position P can be calculated by (Da) / b. Note that information indicating the thickness of the target products 10 in the z direction is stored in a storage device, for example, together with the master information 50, in association with the identification information of the target products 10.

[0079] Furthermore, instead of using the depth image 44 to grasp the number of target products 10 in the depth direction, a plurality of second cameras 40 may be provided, and a plurality of second captured images 42 in which the product group 20 is captured from different directions may be used. When two second cameras 40 are provided, for example, two cameras may be provided perpendicularly to the side in the traveling direction of the product group 20, or two cameras may be provided diagonally forward or backward in the traveling direction of the product group 20. When three second cameras 40 are provided, for example, two second cameras 40 may be provided perpendicularly to the side in the traveling direction of the product group 20, and one second camera 40 may be provided diagonally upward in front or backward in the traveling direction of the product group 20.

[0080] Here, the method of capturing images of the product group 20 from different directions using the multiple second cameras 40 has the advantage that it becomes easier to detect a situation in which a product other than the target product 10 has been mistakenly mixed in by recognizing the target product 10 from multiple second captured images 42 in which different faces of the product group 20 are captured. When it is detected that a product other than the target product 10 has been mixed in in this manner, an error notification is output, for example, as described in a second embodiment below.

[0081] <How to use the results of identifying the number of target products> There are various methods for utilizing the information on the number of target products 10 included in the product group 20 identified by the counting unit 2060. Specific examples of such utilization methods will be described below.

[0082] <<Example of usage 1>> For example, the number of target products 10 included in a product group 20 is used to manage the product group 20. In this case, the counting unit 2060 stores the number of target products 10 included in the product group 20 in a storage device in association with identification information of the product group 20 (e.g., identification information of a pallet). In this way, it becomes possible to easily manage the target products 10 that are received or shipped in units of product groups 20 (e.g., pallet units).

[0083] The method of acquiring the identification information of the product group 20 is arbitrary. For example, a mark representing the identification information of the product group 20 is attached to an arbitrary position of the product group 20 (for example, an arbitrary position on the outer surface of the pallet). The counting device 2000 identifies the identification information of the product group 20 from the mark representing the identification information of the product group 20 using a method similar to the method of identifying the identification information of the target product 10 from the mark 12. The identification information of the product group 20 may be captured by the first camera 30, the second camera 40, or a separate camera prepared in addition to these.

[0084] Note that various information other than the number of target products 10 may be adopted as the information managed in association with the identification information of the product group 20. For example, the counting device 2000 further associates the identification information of the product group 20 with the identification information of the target products 10 included therein. As another example, the counting device 2000 further acquires information on the attributes of the target products 10 included in the product group 20 (hereinafter, attribute information), and stores the attribute information in association with the identification information of the product group 20 in the storage device. As the attribute information, for example, any information such as the expiration date of the target products 10 (such as the best-before date or expiration date of food, or the expiration date of medicine) may be adopted.

[0085] For example, a mark (character string or code) representing attribute information of the target product 10 is displayed on the outer surface of the target product 10. The counting device 2000 identifies the attribute information of the target product 10 by performing image analysis on an image including this mark. For example, if a character string representing the expiration date of the target product 10 is printed on the outer surface of the target product 10, the counting device 2000 identifies the expiration date of the target product 10 by analyzing an image including the character string and recognizing the character string. Note that the mark representing the attribute information of the target product 10 may be captured by the first camera 30, may be captured by the second camera 40, or may be captured by another image.

[0086] <<Example of usage 2>> In a case where the number of target products 10 included in the product group 20 is known, the counting device 2000 may use the number of target products 10 identified by the counting unit 2060 to check whether the number of target products 10 included in the product group 20 is correct. In this case, the counting device 2000 acquires information indicating the number and range of target products 10 that should be included in the product group 20, and compares the acquired number with the number of target products 10 identified by the counting unit 2060 to determine whether the number of target products 10 identified by the counting unit 2060 is incorrect (does not match the correct number or is outside the range of the correct number). When it is determined that the number of target products 10 identified by the counting unit 2060 is incorrect, the counting device 2000 outputs a notification indicating an error. Details of the notification indicating an error will be described in the second embodiment.

[0087] <Modification> In the examples described above, the first camera 30 and the second camera 40 are provided as separate cameras. However, one camera may be used as both the first camera 30 and the second camera 40. For example, the second camera 40 is used as the first camera 30 as well, without providing a separate first camera 30. In this case, the identification unit 2020 identifies the identification information of the target product 10 from the mark 12 included in the second captured image 42. Note that, when multiple second cameras 40 are provided, the mark 12 is included in the imaging range of at least one of the second cameras 40.

[0088] [Embodiment 2] 7 is a block diagram illustrating a functional configuration of a counting device 2000 according to embodiment 2. Except for the points described below, the counting device 2000 according to embodiment 2 has the same functions as the counting device 2000 according to embodiment 1.

[0089] The counting device 2000 of the second embodiment includes a notification unit 2080. The notification unit 2080 outputs a notification indicating an error (hereinafter, an error notification). The types of errors output by the notification unit 2080 will be described below.

[0090] <Error notification example 1> For example, it is considered that the identification unit 2020 cannot identify the identification information of the target product 10 from the first captured image 32. In this case, the notification unit 2080 outputs an error notification indicating that the identification information of the target product 10 could not be identified. Here, possible cases in which the identification information of the target product 10 cannot be identified include 1) a case in which the mark 12 is not detected from the first captured image 32, and a case in which the mark 12 detected from the first captured image 32 cannot be converted into the identification information of the target product 10. Therefore, the error notification may further include information indicating whether the case is 1) or 2). Moreover, the error notification may further include the first captured image 32.

[0091] Similar to the case where the identification information of the target product 10 could not be identified, an error notification may be output when the identification information of the target product 10 is invalid. For example, when it is known in advance that the target product 10 to be inspected is one of three specific types of products, if the identification unit 2020 identifies identification information of a product that does not fall into any of the three types of products, it can be said that the identification information of the identified target product 10 is invalid. Therefore, the notification unit 2080 is configured to output an error notification even when the identification information of the target product 10 is invalid in this way.

[0092] <Error notification example 2> The counting unit 2060 recognizes the target product 10 from the second captured image 42 using the master information. However, it may not be possible to recognize an object included in the product group 20 as the target product 10. This may occur, for example, when the counting unit 2060 uses the master information 50 of the target product 10 to calculate the probability that the product included in the second captured image 42 is the target product 10 (such as the similarity between the image features of the product included in the second captured image 42 and the image features shown in the master information 50 of the target product 10) and determines that the probability is equal to or lower than a predetermined value. This may occur when a part of the second captured image 42 is unclear or when the wrong product is mixed in with the product group 20.

[0093] Thus, when an object that cannot be recognized as the target product 10 is included in the product group 20, the notification unit 2080 outputs an error notification indicating that fact. This error notification preferably includes the second captured image 42 including the object that could not be recognized as the target product 10. Note that information emphasizing the object that could not be recognized as the target product 10 (for example, a frame surrounding the object) may be added to this second captured image 42.

[0094] <Error notification example 3> As described above, the counting device 2000 may acquire attribute information of the target product 10. In this case, if the counting device 2000 is unable to acquire the attribute information of the target product 10 (for example, if an error occurs in the OCR process), the notification unit 2080 may output an error notification indicating that fact.

[0095] <Error notification example 4> As described in "Example 2 of Application Method" of the first embodiment, the counting device 2000 may check whether the number of target products 10 identified by the counting unit 2060 is correct. In this case, if the notification unit 2080 determines that the number of target products 10 is incorrect, it outputs an error notification indicating that fact. Furthermore, this error notification may indicate the number of target products 10 identified by the counting unit 2060 and the correct number or range of target products 10 that is determined in advance.

[0096] <Error notification output destination> The output destination of the error notification is arbitrary. For example, the error notification is sent to the terminal of the operator. For example, when the operator receives the error notification, the operator checks the error status by looking at the error notification. The operator may also provide the counting device 2000 with correct information by using an image or the like included in the error notification. For example, when the identification information and attribute information of the target product 10 are expressed as a character string, an error notification is output indicating that they cannot be identified. In this case, the operator visually checks the image included in the error notification to identify the identification information and attribute information of the target product 10. Then, the operator inputs the identified correct identification information and attribute information of the target product 10 into the terminal and transmits it to the counting device 2000. When these pieces of information are managed in association with the identification information of the product group 20, the information input by the operator can be used to manage the correct information in association with the identification information of the product group 20.

[0097] The output destination of the error notification is not limited to the operator's terminal, and may be, for example, output to a display device connected to the counting device 2000, or stored as an error log in a storage device.

[0098] <Example of hardware configuration> The hardware configuration of the counting device 2000 of the second embodiment is shown in, for example, Fig. 3, similar to the hardware configuration of the counting device 2000 of the first embodiment. However, the storage device 1080 of the second embodiment stores a program for realizing each function of the counting device 2000 of the second embodiment.

[0099] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. 1. An identification unit that analyzes a first captured image including a mark that represents identification information of a target product, and identifies the identification information represented by the mark; a master acquisition unit that acquires master information corresponding to the identification information; a counting unit that detects the target products from a second captured image by analyzing the second captured image, which includes a product group consisting of a plurality of the target products, using the acquired master information, and identifies the number of the target products included in the product group based on the detection result. 2. The counting device described in 1., wherein the first captured image and the second captured image are the same image. 3. The first captured image is generated by a first camera; 1. The counting device described in 1., wherein the second captured image is generated by a second camera different from the first camera. 4. The first camera and the second camera are provided on a path along which the target product moves; The counting device described in 3., wherein the first camera is provided closer to the second camera in terms of the direction of movement of the target product. 5. The first camera is configured to capture an image while being irradiated with light to generate the first captured image; The counting device according to claim 3 or 4, wherein the second camera is controlled to capture an image when the light is not being irradiated. 6. A counting device described in any one of 1. to 5., having a notification unit that outputs a notification that the second captured image includes a product whose certainty as the target product is below a threshold value among the products included in the second captured image. 7. A counting device described in any one of 1. to 6., wherein the counting unit determines the number of target products included in the product group based on the arrangement of the target products in the second captured image and rules for product arrangement in the product group. 8. A computer-implemented control method comprising: A step of identifying the identification information represented by the mark by analyzing a first captured image including the mark representing the identification information of the target product; a master acquisition step of acquiring master information corresponding to the identification information; a counting step of detecting the target products from a second captured image by analyzing the second captured image, which includes a product group consisting of a plurality of the target products, using the acquired master information, and determining the number of the target products included in the product group based on the detection result. 9. The control method described in 8., wherein the first captured image and the second captured image are the same image. 10. The first captured image is generated by a first camera; 8. The control method according to claim 8, wherein the second captured image is generated by a second camera different from the first camera. 11. The first camera and the second camera are provided on a path along which the target product moves; The control method described in 10., wherein the first camera is provided closer to the second camera in terms of the direction of movement of the target product. 12. The first camera is configured to capture an image while being irradiated with light to generate the first captured image; The control method according to claim 10 or 11, wherein the second camera is controlled to capture an image when the light is not being irradiated. 13. A control method described in any one of 8. to 12., comprising a notification step of outputting a notification that the second captured image includes a product whose certainty as the target product is below a threshold value among the products included in the second captured image. 14. A control method described in any one of 8. to 13., wherein in the counting step, the number of target products included in the product group is determined based on the arrangement of the target products in the second captured image and rules for product arrangement in the product group. 15. A program for causing a computer to execute the control method described in any one of 8. to 13. [Explanation of symbols]

[0100] 10 Eligible Products 12 Marks 20 product groups 30 Camera No. 1 32 First captured image 40 Second Camera 42 Second captured image 44 Depth image 50 Master Information 60 Master information storage device 1000 calculator 1020 Bus 1040 Processor 1060 Memory 1080 Storage Device 1100 Input / Output Interface 1120 Network Interface 2000 Counting Device 2020 Specific Department 2040 Master Acquisition Department 2060 Counting Department 2080 Notification Department

Claims

1. an identification unit that identifies a product identified by a mark based on a mark detected from a part of a plurality of products included in a captured image; A counting device having a counting unit that identifies target products from the captured image, which are products that are determined to be the same product as the identified product, based on information regarding the image of the identified product, and counts the number of the identified target products.

2. The mark is a bar code or a two-dimensional code.

2. The counting device according to claim 1.

3. The information about the image is a template image of the identified product or a feature amount on the image of the identified product.

3. The counting device according to claim 1 or 2.

4. The counting device according to claim 1 , further comprising a notification unit that outputs a notification that the captured image includes a product whose certainty as the target product is below a threshold value among the products included in the captured image.

5. The counting device according to claim 1 , wherein the counting unit determines the number of the target products included in the product group based on the arrangement of the target products in the captured image and a rule for arranging products in the product group consisting of a plurality of the target products.

6. 1. A computer-implemented control method comprising: a step of identifying a product identified by a mark based on a mark detected from a part of a plurality of products included in a captured image; A control method comprising: a counting step of identifying target products from the captured image, which are products that are determined to be the same product as the identified product, based on information regarding the image of the identified product, and counting the number of the identified target products.

7. A program for causing a computer to execute the control method according to claim 6.

Images