Information processing system, control method, and program

Abstract

To provide technology to accurately determine the validity of articles recognized by a device that registers articles to be settled.SOLUTION: An information processing device having a function for registering an article to be settled includes: first acquisition means for acquiring article information relating to the article to be settled; second acquisition means for acquiring image information including the article; and determination means for determining, based on the output of the first and second acquisition means, whether or not there is any irregularity with respect to the article registration. The image information includes at least one of an image of the article when the article is placed on the registered article placement site or an image of the article placed on the registered article placement site.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for determining whether or not a product recognized by a device is a correct product. [Background technology]

[0002] There is a technology in which, when a device for registering products to be checked out recognizes a product to be checked out using a code for identifying the product, such as a barcode, a weight sensor is provided in a place where the recognized product is temporarily placed, and the validity of the product recognized by the device is judged. "Validity" here means that the device for registering products to be checked out recognizes the correct product as the product that the customer will purchase. This technology judges whether the weight measured by the weight sensor matches the weight of the product linked to the product identification code.

[0003] Another example of a technology for appropriately determining the validity of a product recognized by a device that registers products to be checked out is described, for example, in Patent Document 1. Patent Document 1 discloses a technology for acquiring an image of an object (product) to which a barcode identifying the product is attached and a basic figure of the product associated with the barcode in response to reading the barcode, and determining the validity of the recognized product based on a correlation value calculated using the shape of the object recognized from the image and the basic figure associated with the read barcode.

[0004] Patent Document 2 and Patent Document 3 disclose a technology for identifying a product by object recognition using an image. In detail, Patent Document 2 and Patent Document 3 disclose a technology in which a product ID is linked to a product image, and the product image and an object included in a captured image are matched based on features such as surface conditions such as surface color, pattern, and surface unevenness, and shape, and a candidate product corresponding to the object in the image is identified based on the similarity. Patent Document 3 also discloses a technology for detecting an area corresponding to a person's hand from an image using a thermo camera using far-infrared rays, and performing the above-mentioned feature matching by excluding the area. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2014-132501 A [Patent Document 2] JP 2013-182323 A [Patent Document 3] JP 2014-052806 A Summary of the Invention [Problem to be solved by the invention]

[0006] As described above, the higher the accuracy of the technology for determining the validity of the product recognized by the device that registers the product to be checked out, the better.

[0007] An object of the present invention is to provide a technique for accurately determining the validity of a product recognized by a device that registers the product to be checked out. [Means for solving the problem]

[0008] According to the present invention, A first acquisition means is provided in a container for storing and transporting the product, and acquires product information related to the product to be checked out; A second acquisition means is provided in the container and acquires image information of the product that the customer places in the container; an output means for outputting a warning when the product placed in the container by the customer is determined to be an illegal product using the information acquired by the first and second acquisition means; Equipped with The first acquisition means is different from the second acquisition means; An information processing system is provided.

[0009] According to the present invention, A computer provided in a container for storing and transporting a commodity, A first acquisition step of acquiring product information related to a product to be checked out; A second acquisition step of acquiring image information of a product that the customer places in the container; an output step of outputting a warning when the product placed in the container by the customer is determined to be an illegal product using the information acquired in the first and second acquisition steps; Run The first acquisition step is provided with a control method different from that of the second acquisition step.

[0010] According to the present invention, A computer provided in a container for storing and transporting a product, A first acquisition means for acquiring product information related to a product to be settled; A second acquiring means for acquiring image information of the product that the customer places in the container; an output means for outputting a warning when the product placed in the container by the customer is determined to be an illegal product using the information acquired by the first and second acquisition means; Functioning as a The first acquisition means is provided with a program different from that provided with the second acquisition means. Effect of the Invention

[0011] According to the present invention, it is possible to accurately determine the validity of merchandise recognized by a device that registers merchandise to be settled. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a diagram conceptually illustrating a processing configuration of an information processing device in the first embodiment. [Diagram 2] 11 is a diagram showing an example of information stored in a size / shape information storage unit; FIG. [Diagram 3] FIG. 2 is a diagram conceptually illustrating a hardware configuration of an information processing device. [Figure 4] 4 is a flowchart showing a flow of processing in the information processing apparatus of the first embodiment. [Diagram 5] FIG. 11 is a diagram showing an example of installation of an information processing device in the second embodiment. [Figure 6] FIG. 13 is a diagram conceptually illustrating a processing configuration of an information processing device in a third embodiment. [Figure 7] 13 is a flowchart showing a flow of processing in an information processing apparatus according to a third embodiment. [Figure 8] FIG. 13 is a diagram conceptually illustrating a processing configuration of an information processing device in a fourth embodiment. [Figure 9] 5 is a diagram illustrating an example of information stored in a weight information storage unit. FIG. [Figure 10] 13 is a flowchart showing a flow of processing in an information processing apparatus according to a fourth embodiment. [Figure 11] FIG. 13 is a diagram conceptually illustrating a processing configuration of an information processing device in a fifth embodiment. [Figure 12] 13 is a flowchart showing a process flow of an information processing apparatus according to a fifth embodiment. [Figure 13] A diagram conceptually showing the system configuration of a settlement system in a sixth embodiment. [Figure 14] A sequence diagram showing the processing flow of the settlement system of the sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all the drawings, the same components are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0014] [First embodiment] [Processing configuration] 1 is a diagram conceptually illustrating a processing configuration of an information processing device 10 in the first embodiment. The information processing device 10 is a product registration device such as a so-called Point of Sales (POS) terminal that has at least a function of registering products to be settled. The information processing device 10 may further have a function of settling the price of registered products. The information processing device 10 of this embodiment includes an image recognition unit 110, a product recognition unit 120 (a code acquisition unit 122, an imaging unit 124, and a thermography information acquisition unit 126), a size / shape information acquisition unit 130, a size / shape information storage unit 132, and a determination unit 140.

[0015] <Registering products to be paid for> In this embodiment, the products to be checked out are registered as follows.

[0016] The items to be checked out are moved as shown by the dotted lines in Fig. 1. That is, items that have not yet been registered as items to be checked out (hereinafter, such items will be referred to as "unregistered items") are removed from the location where unregistered items are temporarily placed, and moved to the position of item recognition unit 120. At this point, the unregistered items are registered as items to be checked out, as will be described in detail later. Then, items that have been registered as items to be checked out (hereinafter, such items will be referred to as "registered items") are moved to the location where registered items are temporarily placed.

[0017] For example, an unregistered product is registered as a product to be settled as follows. First, the code acquisition unit 122 of the product recognition unit 120 acquires the product identification code of the unregistered product held up to the product recognition unit 120 as the product identification code of the product to be settled. Here, the product identification code is data formed of letters, numbers, symbols, etc. that are predetermined to identify each product. For the product identification code, a PLU (Price Look Up) code (for example, a JAN (Japan Article Number) code, etc.), a Non-PLU code, etc. may be used.

[0018] The commodity identification code is obtained by detecting and analyzing the commodity information symbol from the commodity to be checked out. The commodity information symbol is a symbol that identifies information about the commodity. Here, the symbol is a barcode, a two-dimensional code (QR code (registered trademark), etc.), a character string symbol, etc. Note that the character string here also includes a numerical string. The commodity information symbol is a barcode in which information that identifies the commodity information (such as the ID of the commodity information) is encoded, or a character string symbol that represents information that identifies the commodity information. In this case, the commodity identification code is obtained from the commodity information symbol by using, for example, a code reader having a light source and a light receiving element such as a photodiode, or a code reader having an imaging element such as a lens and a CCD (Charge Coupled Device) image sensor. In addition, in the case of a code reader having a lens and an imaging element, a commodity identification code can also be obtained by performing a matching process by extracting a feature amount of the commodity (for example, color, shape, packaging, or a specific mark or character string that identifies the commodity) from an image including the commodity to be checked out, and referring to a storage unit (not shown) that stores the commodity identification code and the commodity feature amount in association with each other.

[0019] Then, using the product identification code acquired by the code acquisition unit 122, the product recognition unit 120 refers to a storage unit (not shown) that stores product identification codes linked to product information (e.g., product name, product unit price, whether or not a discount is applied, etc.), and reads out the product information linked to the acquired product identification code. Then, the product recognition unit 120 adds the read-out product information as information to be used in the settlement process executed by the information processing device 10. As a result, the pre-registration product is recognized on the information processing device 10 and registered as a product to be settled.

[0020] <Determining the validity of products recognized by the device> In this embodiment, the validity of the product registered as the payment target as described above is determined by each processing unit described below. Note that "valid" in this embodiment means that the information processing device 10 recognizes the correct product as the product to be purchased by the customer.

[0021] The imaging unit 124 is a so-called 3D camera or the like, and is configured to be able to generate an image by associating depth information indicating the distance in the depth direction of a commodity (unregistered commodity) included in the imaging range. The imaging unit 124 generates an image by associating the image with the depth information by a known method, such as a method using parallax in a stereo camera. The imaging unit 124 of this embodiment includes the vicinity of the commodity recognition unit 120 in the imaging range, and its orientation and position are adjusted so that an unregistered commodity moved to the commodity recognition unit 120 can be imaged. The imaging unit 124 may be provided integrally with the information processing device 10, such as a camera used for object recognition of a commodity, or may be installed around the information processing device 10, such as an external camera.

[0022] Furthermore, the imaging unit 124 preferably captures an image in accordance with the timing at which the code acquisition unit 122 reads the product identification code. For example, when the code acquisition unit 122 detects the presence of a product information symbol of a product, it sends a notification indicating this to the imaging unit 124. Upon receiving this notification, the imaging unit 124 can know the timing at which the code acquisition unit 122 reads the code. In this way, in the case of a product having the above-mentioned product information symbol or the like, if the imaging unit 124 captures an image at the timing at which the code acquisition unit 122 reads the product identification code, the posture of the product contained in the image is stabilized, and the change in posture for each acquired image is reduced. This is expected to have the effect of reducing the number of templates used in image matching and reducing the processing load, as well as the effect of improving the recognition accuracy by image matching.

[0023] The thermographic information acquisition unit 126 is a so-called thermographic camera that uses infrared rays, captures infrared radiation energy emitted by an object included in the imaging range, and generates thermographic information indicating the temperature distribution of the object. The thermographic information acquisition unit 126 is configured to acquire thermographic information in accordance with the imaging timing of the imaging unit 124. The position and orientation of the thermographic information acquisition unit 126 are adjusted so that the thermographic information acquisition unit 126 can acquire thermographic information of an imaging range that is the same as the imaging range of the imaging unit 124 or an imaging range that includes the imaging range of the imaging unit 124. The thermographic information acquisition unit 126 then aligns the positional relationship between the image generated by the imaging unit 124 and the thermographic information, for example, by a coordinate conversion process of the image space. In this way, the thermographic information acquisition unit 126 acquires thermographic information associated with the image generated by the imaging unit 124.

[0024] The image recognition unit 110 performs the following processing using the image generated by the imaging unit 124 and the thermographic information acquired by the thermographic information acquisition unit 126.

[0025] The image recognition unit 110 performs a known image recognition process on the image generated by the imaging unit 124 to detect, for example, the edges of the product in the image. Then, the image recognition unit 110 matches, for example, the shape indicated by the detected edges with preset template data of two-dimensional or three-dimensional shapes, and recognizes the shape with the highest similarity as the shape of the product. Furthermore, the image recognition unit 110 calculates, for example, the actual size of the product in the image (for example, height, width, depth width, area of ​​each surface, etc.) using the depth information associated with the image generated by the imaging unit 124 and the size of the area occupied by the product in the image. Since the actual distance from the imaging device to the product is known based on the depth information, the image recognition unit 110 can convert the size of the area occupied by the product in the image into the actual size of the product based on that distance.

[0026] Further, the image recognition unit 110 uses the thermography information acquired by the thermography information acquisition unit 126 to exclude an area estimated to be a person from the shape of the product recognized from the image generated by the imaging unit 124. For example, information indicating the range of a person's body temperature (for example, 35 to 40°C) is set in advance in the image recognition unit 110, and the image recognition unit 110 extracts an area indicating the set temperature from the thermography information. Then, based on the positional relationship between the thermography information acquired by the thermography information acquisition unit 126 and the image generated by the imaging unit 124, the image recognition unit 110 excludes an area corresponding to the above-mentioned extracted area from the target area of ​​the edge detection process, or excludes an area corresponding to the above-mentioned extracted area from the shape recognized based on the detected edge. This allows the image recognition unit 110 to recognize a more accurate area corresponding to the product from the image generated by the imaging unit 124, and as a result, the actual size of the product converted from the area can be calculated with higher accuracy.

[0027] The size / shape information acquisition unit 130 refers to the size / shape information storage unit 132 using the product identification code acquired by the code acquisition unit 122, and acquires size / shape information linked to the product identification code. The size / shape information is information indicating the size of each product (e.g., the height, width, and depth of the product, and the area of ​​each face calculated from these) and the external shape of each product. The size / shape information storage unit 132 stores the product identification code and the size / shape information in association with each other, as shown in FIG. 2, for example. FIG. 2 is a diagram showing an example of information stored in the size / shape information storage unit 132. The information stored in the size / shape information storage unit 132 is not limited to the example of FIG. 2.

[0028] The size / shape information acquisition unit 130 searches the size / shape information storage unit 132 to detect a product identification code identical to the product identification code acquired by the code acquisition unit 122, and acquires size / shape information linked to the detected product identification code. Note that, although the example in FIG. 1 shows an example in which the size / shape information storage unit 132 is provided in the information processing device 10, the size / shape information storage unit 132 may be provided in another device capable of communicating with the information processing device 10. In this case, the size / shape information acquisition unit 130 acquires the size / shape information linked to the product identification code acquired by the code acquisition unit 122 by communicating with the other device.

[0029] The determination unit 140 determines whether the size or shape recognized from the image generated by the imaging unit 124 and the size / shape information acquired using the product identification code satisfy a predetermined criterion. Here, the predetermined criterion is a criterion related to the size or shape of the product, and is a criterion that makes it possible to confirm that the product recognized from the image and the product identified from the product identification code indicate the same product. For example, the determination unit 140 calculates the similarity between the two according to a predetermined similarity calculation algorithm related to size and shape, and determines whether the similarity is equal to or greater than a predetermined criterion value (a threshold value previously set as a value for regarding the two products as the same product).

[0030] As a result of the determination, if the degree of match between the two is equal to or greater than a predetermined reference value, it is highly likely that the product moved to the position of the product recognition unit 120 and the product recognized by the information processing device 10 are the same, and the determination unit 140 can determine that the recognition result by the information processing device 10 is valid. On the other hand, if the degree of match between the two is less than a predetermined reference value, it is highly likely that the product moved to the position of the product recognition unit 120 and the product recognized by the information processing device 10 are different, and the determination unit 140 can determine that the recognition result by the information processing device 10 is invalid. In this case, it is considered that some kind of defect has occurred, such as a misreading of the code or an incorrect code being attached. Therefore, the determination unit 140 may be configured to display a message prompting the user to reread the product or a predetermined warning message on the display unit of the information processing device 10 or output it by voice when the degree of match between the two is less than a predetermined reference value.

[0031] In this embodiment, the size / shape information storage unit 132 may store a plurality of size / shape information items associated with each commodity identification code. Specifically, the size / shape information storage unit 132 may store size / shape information items indicating the orientation of the commodity as viewed from a plurality of directions, such as size / shape information items when viewed from the front, size / shape information items when viewed from above, and size / shape information items when viewed from diagonally above, associated with each commodity identification code. In this case, the determination unit 140 calculates the similarity between each of the plurality of size / shape information items associated with each commodity identification code and the size or shape recognized by the image recognition unit 110, and determines whether the degree of agreement between the two is equal to or greater than a predetermined reference value. In this way, it is possible to prevent the accuracy of the determination by the determination unit 140 from being reduced due to the orientation of the commodity included in the image generated by the imaging unit 124.

[0032] [Hardware configuration] Fig. 3 is a diagram conceptually showing a hardware configuration of the information processing device 10. As shown in Fig. 3, the information processing device 10 includes a CPU 101, a memory 102, a storage 103, an input / output interface (input / output I / F) 104, a communication module 105, and the like. The CPU 101, memory 102, storage 103, input / output interface 104, and communication module 105 are connected by a data transmission path for transmitting and receiving data to and from each other.

[0033] The memory 102 is, for example, a memory such as a random access memory (RAM) or a read only memory (ROM). The storage 103 is, for example, a storage device such as a hard disk, a solid state drive (SSD), or a memory card. The storage 103 stores program modules for implementing the functions of each processing unit including the image recognition unit 110, the commodity recognition unit 120, the code acquisition unit 122, the imaging unit 124, the thermography information acquisition unit 126, the size / shape information acquisition unit 130, and the determination unit 140 of the information processing device 10. The CPU 101 implements the functions of each processing unit by executing each of these program modules. When executing each of the above modules, the CPU 101 may read these modules onto the memory 102 before executing them, or may execute them without reading them onto the memory 102. The storage 103 also functions as a size / shape information storage unit 132.

[0034] The input / output interface 104 is connected to a display device 1041, an input device 1042, an imaging device 1043, a reading device 1044, and the like. The display device 1041 is a device that displays a screen corresponding to drawing data processed by the CPU 101, a GPU (Graphics Processing Unit) (not shown), and the like, such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube) display. One or more display devices 1041 may be connected to the input / output interface 104. The input device 1042 is a device that accepts input by an operator's operation, and is realized as, for example, a keyboard, a mouse, a touch sensor, and the like. The display device 1041 and the input device 1042 may be integrated and realized as a touch panel. The imaging device 1043 is a so-called 3D camera, and includes a monocular imaging module or a twin-lens imaging module (not shown). In this case, the imaging device 1043 may realize the function of the imaging unit 124 described above. The imaging device 1043 is a so-called thermography camera or the like, and includes an imaging module (not shown) that detects infrared energy radiated from an object and generates an image showing the temperature distribution of the object based on the amount of infrared energy. In this case, the imaging device 1043 can realize the function of the above-mentioned thermography information acquisition unit 126. The reading device 1044 is a device such as a code reader including a light source and a light receiving element such as a photodiode, or a code reader including a lens and an imaging element such as a CCD (Charge Coupled Device) image sensor. The reading device 1044 can realize the function of the above-mentioned code acquisition unit 122.

[0035] The communication module 105 is used to transmit and receive data to and from an external device, etc. There are various methods for connecting the information processing device 10 to an external device via the communication module 105. For example, this connection may be a bus connection via a bus line (e.g., a Universal Serial Bus (USB) line) or a network connection via a network line. The network line may be a wireless line or a wired line.

[0036] The hardware configuration of the information processing device 10 is not limited to the configuration shown in Fig. 3. For example, various devices (such as a money insertion device and a change dispensing device) necessary for the payment process of products may be connected to the input / output interface 104.

[0037] [Example of operation] An example of the operation of the information processing device 10 of this embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of processing of the information processing device 10 of the first embodiment.

[0038] When a product is moved to the position of the product recognition unit 120, the code acquisition unit 122 acquires the product identification code of the product (S101). In response to acquiring the product identification code, the imaging unit 124 generates an image of the product by associating it with depth information indicating the distance in the depth direction (S102), and the thermography information acquisition unit 126 acquires thermography information by associating it with the image generated by the imaging unit 124 (S103).

[0039] Then, the image recognition unit 110 recognizes the size or shape of the product using the generated image and the depth information and thermography information associated with the image (S104). For example, the image recognition unit 110 converts the size of the area that the product occupies in the image into the actual size of the product using the actual distance to the product obtained from the depth information. Also, for example, the image recognition unit 110 extracts the product area from the image using a known image recognition process such as edge processing, and obtains the shape of the area as the product shape.

[0040] In addition, the size / shape information acquisition unit 130 refers to the size / shape information storage unit 132 using the product identification code acquired by the code acquisition unit 122, and reads out the size / shape information linked to the product identification code from the size / shape information storage unit 132 (S105).

[0041] The determination unit 140 compares the size or shape of the product recognized by the image recognition unit 110 with the size / shape information linked to the product identification code, and determines whether or not a predetermined criterion is met (S106). For example, the determination unit 140 can calculate the similarity between the two according to a predetermined similarity calculation algorithm, and determine whether or not the two meet the predetermined criterion based on whether or not the similarity is equal to or greater than a predetermined threshold. If the determination unit 140 determines that the predetermined criterion is met (S106: YES), it determines that the product has been normally recognized by the information processing device 10, and does not execute the process of S107 described below. On the other hand, if the determination unit 140 determines that the predetermined criterion is not met (S106: NO), it outputs a predetermined output (for example, a message prompting reloading of the product, a warning message, a warning sound, etc.) via a display, a speaker, etc. (not shown) (S107).

[0042] [Functions and Effects of the First Embodiment] As described above, according to the present embodiment, for a product read by the product recognition unit 120, the similarity between the actual size or shape of the product calculated from the image and the size / shape information indicating the size or shape of the product linked to the product identification code of the product is calculated. Then, depending on whether the similarity between the two is equal to or greater than a predetermined threshold (a threshold value previously set as a value for regarding the product as the same product), it is determined whether the two satisfy a predetermined criterion. Then, if the sizes or shapes of the two are approximately the same and satisfy the predetermined criterion, it is highly likely that the product moved to the position of the product recognition unit 120 and the product recognized by the information processing device 10 are the same, and the determination unit 140 determines that the recognition result by the information processing device 10 is valid. On the other hand, if the sizes or shapes of the two do not match and do not satisfy the predetermined criterion, it is highly likely that the product moved to the position of the product recognition unit 120 and the product recognized by the information processing device 10 are different, and the determination unit 140 determines that the recognition result by the information processing device 10 is invalid. In this way, by comparing the actual size or shape of the product recognized from the image with the theoretical size or shape of the product linked to the product identification code, the validity of the product recognized by the information processing device 10 can be accurately determined.

[0043] [Second embodiment] There is also an operation method in which a device for reading products to be paid for is attached to a basket that a customer carries around to put the products they want to purchase, and the products are registered as products to be paid for when they are placed in the basket, and only the payment is settled at the cash register terminal. In such a case, by providing the basket with a device having a configuration similar to that of the information processing device 10 of the first embodiment, it is possible to obtain the same effect as that of the first embodiment.

[0044] [Processing configuration / hardware configuration] 1, the information processing device 10 of this embodiment includes an image recognition unit 110, a commodity recognition unit 120 (code acquisition unit 122, imaging unit 124, thermography information acquisition unit 126), a size / shape information acquisition unit 130, a size / shape information storage unit 132, and a determination unit 140. The information processing device 10 of this embodiment also has a hardware configuration similar to that of the first embodiment, and each processing unit is realized in the same manner as in the first embodiment. However, each hardware component of the information processing device 10 shown in FIG. 4 is made smaller than that of the first embodiment in order to be attached to the basket 20.

[0045] Fig. 5 is a diagram showing an example of installation of the information processing device 10 in the second embodiment. As shown in Fig. 5, the information processing device 10 is installed in a basket 20. As shown in Fig. 5, an unregistered product is placed in the basket 20 after being registered as a product to be checked out by the information processing device 10.

[0046] [Example of operation] As shown in FIG. 4, the information processing device 10 of this embodiment operates in the same manner as the information processing device 10 of the first embodiment.

[0047] [Functions and Effects of the Second Embodiment] As described above, the present embodiment can also provide the same functions and effects as the first embodiment.

[0048] [Third embodiment] This embodiment is similar to the first and second embodiments, except that the information processing device 10 further includes a number obtaining unit 150.

[0049] [Processing configuration] 6 is a diagram conceptually showing a processing configuration of the information processing device 10 in the third embodiment. As shown in FIG.

[0050] The number acquisition unit 150 acquires the number of products included in the imaging range of the image generated by the imaging unit 124. In detail, the number acquisition unit 150 acquires the number of product shapes recognized by the image recognition unit 110 as the number of products. Alternatively, the number acquisition unit 150 acquires the number of objects detected from within the image by pattern matching of pre-stored product features as the number of products.

[0051] In this embodiment, the determination unit 140 determines whether the number of product identification codes acquired by the code acquisition unit 122 matches the number of products acquired by the quantity acquisition unit 150. Note that "match" here means a perfect match. In this embodiment, when multiple product information symbols are present in an image, the code acquisition unit 122 performs, for example, image recognition processing to detect each product information symbol and acquire multiple product identification codes at approximately the same time. The code acquisition unit 122 in this embodiment further notifies the determination unit 140 of the number of product identification codes acquired here.

[0052] [Hardware configuration] Like the first and second embodiments, the information processing device 10 of this embodiment has a hardware configuration as shown in Fig. 4. The storage 103 further stores a program module that realizes the function of the number obtaining unit 150, and the number obtaining unit 150 is realized by the CPU 101 executing the program module.

[0053] [Example of operation] An example of the operation of the information processing device 10 of this embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the flow of processing of the information processing device 10 of the third embodiment. Note that, hereinafter, the steps (S201 to S203) different from the first and second embodiments will be mainly described.

[0054] When acquiring a product identification code, the code acquisition unit 122 counts the number of acquired product identification codes (S201). Then, the number acquisition unit 150 acquires the number of product shapes recognized by the image recognition unit 110 as the number of products included in the image generated by the imaging unit 124 (S202). Furthermore, the determination unit 140 compares the number of product identification codes counted in S201 with the number of products acquired in S202 to check whether they match (S203). If the number of products does not match the number of product identification codes (S203: NO), the determination unit 140 determines that some kind of error, such as a product reading error, has occurred, and displays a predetermined message, such as a message prompting the user to reread the product or a warning message, on the display or outputs it as audio information (S107). On the other hand, if the number of products and the number of product identification codes match (S203: YES), the judgment unit 140 determines that an error such as a misreading of the products has not occurred, and, as described in detail in the first embodiment, checks whether the size or shape obtained from the image and the size / shape information obtained from the product identification code satisfy a predetermined standard (S106).

[0055] [Functions and Effects of the Third Embodiment] As described above, in this embodiment, it is further determined whether or not the number of products recognized from the image matches the number of product identification codes actually recognized by the information processing device 10. As a result, according to this embodiment, it is possible to determine whether or not an error such as a misreading of a product has occurred, and if such an error has occurred, it is possible to prompt the information processing device 10 to accurately recognize the products.

[0056] [Fourth embodiment] In this embodiment, a form will be described in which the technology using a weight sensor is combined with the information processing device 10 described in each of the above embodiments.

[0057] When determining the validity of a product recognized by the information processing device 10 based on the weight measured by a weight sensor and the weight of the product linked to the product identification code, a correct determination result may not be obtained if the weight of the product is extremely light, or if there is variation in weight even for the same product, etc. When targeting such products, the information processing device 10 of this embodiment determines the validity of the recognition based on the size or shape of the product recognized from the image and the size / shape information acquired from the product identification code, as described in each of the above embodiments.

[0058] [Processing configuration] 8 is a diagram conceptually illustrating the processing configuration of the information processing device 10 in the fourth embodiment. The information processing device 10 in this embodiment is provided with weight sensors (160 and 162) at the placement locations of unregistered products and registered products, respectively, and basically determines the validity of the products recognized by the information processing device 10 based on the amount of change in weight measured by the weight sensors.

[0059] As shown in Fig. 8, the information processing device 10 of this embodiment further includes a weight information storage unit 142. The weight information storage unit 142 stores at least one of information indicating the weight of each product having the product identification code (hereinafter referred to as weight information) and information indicating the weight deviation of the product (hereinafter referred to as weight deviation information) in association with the product identification code. An example of information stored in the weight information storage unit 142 is shown in Fig. 9. Fig. 9 is a diagram showing an example of information stored in the weight information storage unit 142. In the example of Fig. 9, the weight information storage unit 142 stores weight information and weight deviation information in association with the product identification code.

[0060] The determination unit 140 of this embodiment acquires a product identification code from the code acquisition unit 122, and acquires weight information or weight deviation information linked to the product identification code acquired by the code acquisition unit 122 from the weight information storage unit 142 using the product identification code. When the weight information storage unit 142 stores information such as that shown in Fig. 9, the determination unit 140 can acquire weight information and weight deviation information. Then, based on the acquired weight information or weight deviation information, the determination unit 140 determines whether or not at least one of the following conditions is satisfied: (1) the weight linked to the acquired product identification code is equal to or greater than a reference value, (2) the weight linked to the acquired product identification code is not set, and (3) the weight deviation linked to the acquired product identification code is equal to or greater than a predetermined threshold value.

[0061] These conditions are used to determine whether or not the product with the product identification code acquired by the code acquisition unit 122 is a product that is not suitable for processing by a weight sensor. The above condition (1) is used to determine whether or not a product is such a product, for example, when the product is very light and the amount of change in weight that can be detected by a weight sensor is small, and therefore erroneous recognition is likely to occur when the product is processed by a weight sensor. The above condition (2) is used to determine whether or not a product is such a product, for example, when the weight is not set and therefore processing by a weight sensor is not possible in the first place. The above condition (3) is used to determine whether or not a product has a large variation in weight and therefore erroneous recognition is likely to occur when the product is processed by a weight sensor.

[0062] When the determination unit 140 of this embodiment determines that the item falls under any of the above (1) to (3), it executes the determination described in each of the above embodiments, that is, the process of determining the validity of the recognition by the information processing device 10 using the size or shape of the item. Specifically, when the determination unit 140 of this embodiment determines that the item falls under any of the above (1) to (3), it notifies the image recognition unit 110 and the size / shape information acquisition unit 130 of a request to execute the process described in each of the above embodiments. Upon receiving this notification, the image recognition unit 110 sends the size or shape of the item recognized from the image generated by the imaging unit 124 to the determination unit 140. Furthermore, upon receiving this notification, the code acquisition unit 122 sends the size / shape information linked to the item identification code acquired by the code acquisition unit 122 to the determination unit 140. The determination unit 140 executes the determination process as described in each of the above embodiments, using the size or shape of the item received from the image recognition unit 110 and the size / shape information received from the size / shape information acquisition unit 130.

[0063] [Hardware configuration] The information processing device 10 of this embodiment has a hardware configuration as shown in Fig. 4, similar to the first and second embodiments. The storage 103 further stores a program module that realizes the function of the determination unit 140 of this embodiment, and the determination unit 140 of this embodiment is realized by the CPU 101 executing the program module. The storage 103 can also realize the function of a weight information storage unit 142.

[0064] [Example of operation] An operation example of the information processing device 10 of this embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the flow of processing of the information processing device 10 of the fourth embodiment. Note that in the following description, the processing (S301 to S303) added in this embodiment will be mainly described based on the flowchart in Fig. 4.

[0065] The determination unit 140 refers to the weight information storage unit 142 using the product identification code acquired by the code acquisition unit 122 in S101, and acquires weight information or weight deviation information associated with the product identification code (S301). Then, the determination unit 140 uses the acquired weight information or weight deviation information to determine whether or not the product of the product identification code acquired by the code acquisition unit 122 in S101 is a product that is not suitable for determination processing using a weight sensor (S302). Specifically, the determination unit 140 determines whether or not at least one of the following conditions is satisfied: (1) the weight associated with the acquired product identification code is equal to or greater than a reference value, (2) the weight associated with the acquired product identification code is not set, and (3) the weight deviation associated with the acquired product identification code is equal to or greater than a predetermined threshold value.

[0066] If the product having the product identification code acquired by the code acquisition unit 122 in S101 is not suitable for the determination process using a weight sensor (S302: YES), the determination unit 140 notifies the image recognition unit 110 and the size / shape information acquisition unit 130 of that fact, and the process described in the first embodiment is executed (S102 to S107). On the other hand, if the product having the product identification code acquired by the code acquisition unit 122 in S101 is suitable for the determination process using a weight sensor (S302: NO), the determination process using a weight sensor is executed (S303).

[0067] [Actions and Effects of the Fourth Embodiment] As described above, in this embodiment, it is determined whether or not the product of the product identification code acquired by the code acquisition unit 122 is a product that is not suitable for determination processing using a weight sensor. Then, if it is determined that the product of the product identification code acquired by the code acquisition unit 122 is a product that is not suitable for determination processing using a weight sensor, a determination process is executed based on the size or shape of the product recognized from the image and the size / shape information linked to the product identification code. This makes it possible to prevent erroneous recognition that may occur when a weight sensor or the like is used, and improve the accuracy of determining the validity of the product recognized by the information processing device 10.

[0068] [Modification of the fourth embodiment] In this embodiment, the determination unit 140 basically performs a determination process based on the amount of change in weight measured by a weight sensor, and for a product that satisfies a predetermined condition, the determination unit 140 performs a determination process based on the size or shape recognized from an image. However, the determination unit 140 may be configured to perform a determination process based on the amount of change in weight measured by a weight sensor for a product that satisfies a condition different from the above-mentioned predetermined condition. Here, the "product that satisfies a different condition" is, for example, a thin product that has no thickness, and many feature points may be recognized depending on the orientation or posture, and is not suitable for image recognition processing. In such a case, the information processing device 10 stores the product identification code of the product that is not suitable for image recognition processing in a storage unit not shown, and when the code acquisition unit 122 acquires the same product identification code as the product identification code stored in the storage unit, the determination unit 140 judges the validity of the product recognized by the information processing device 10 based on the change in weight measured by the weight sensor.

[0069] [Fifth embodiment] In this embodiment, an information processing device 10 having a minimum configuration capable of obtaining the effects of the present invention will be described. The information processing device 10 is provided as a device (e.g., a server device) separate from a device for registering products to be settled (hereinafter, referred to as a product registration device) and is capable of communicating with the product registration device via, for example, a LAN (Local Area Network).

[0070] [Processing configuration] 11 is a diagram conceptually illustrating a processing configuration of the information processing device 10 in the fifth embodiment. As shown in FIG. 11, the information processing device 10 in this embodiment includes an image recognition unit 110, a code acquisition unit 122, a size / shape information acquisition unit 130, and a determination unit 140.

[0071] The image recognition unit 110 recognizes the size or shape of the product in the image by using an image associated with depth information indicating the depth direction distance of the product included in the imaging range. The code acquisition unit 122 and the size / shape information acquisition unit 130 acquire the size / shape information linked to the acquired product identification code. The determination unit 140 determines whether the size or shape of the product recognized from the image and the size / shape information acquired using the product identification code satisfy a predetermined criterion.

[0072] [Hardware configuration] The information processing device 10 of this embodiment has a hardware configuration as shown in Fig. 3. Note that the input / output interface 104 of the information processing device 10 of this embodiment may not be connected to the imaging device 1043 and the reading device 1044, and may be configured to be able to acquire images and product identification codes from other devices via the communication module 105. The image recognition unit 110, the code acquisition unit 122, the size / shape information acquisition unit 130, and the determination unit 140 of this embodiment are realized by the CPU 101 executing program modules that realize the functions of each processing unit, as in the above-mentioned embodiments.

[0073] [Example of operation] An example of the operation of the information processing device 10 of this embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart showing the flow of processing of the information processing device 10 of the fifth embodiment.

[0074] The code acquisition unit 122 communicates with the product registration device via a network such as a LAN, and acquires the product identification code of the product to be settled that has been read by the product registration device (S401). The size / shape information acquisition unit 130 then uses the product identification code acquired in S401 to read out size / shape information associated with the product identification code (S402). In this embodiment, the size / shape information is associated with the product identification code and stored in, for example, a storage unit (not shown) of another device capable of communicating with the information processing device 10.

[0075] The image recognition unit 110 also communicates with the product registration device via a network such as a LAN, and acquires an image generated by an imaging means or the like provided in the product registration device (S403). The imaging means provided in the product registration device is a so-called 3D camera, and can generate an image in association with depth information. The image recognition unit 110 then recognizes the shape of the product in the image using a known image recognition process, and calculates the actual size of the product based on the depth information associated with the image and the size of the product area in the image (S404).

[0076] The determination unit 140 determines whether or not the size / shape information acquired in S402 and the size and shape of the product acquired in S404 satisfy a predetermined standard (S405), and transmits the result to the product registration device (S406).

[0077] [Actions and Effects of the Fifth Embodiment] As described above, in this embodiment, the actual size or shape of the product recognized from the image is compared with the theoretical size or shape of the product associated with the product identification code, thereby making it possible to accurately determine the validity of the product recognized by the product registration device.

[0078] [Sixth embodiment] In this embodiment, a settlement system including a product registration device that performs a process of registering products to be settled, and a settlement device that performs a process of settling the prices of the products registered as the products to be settled will be described.

[0079] [System Configuration] FIG. 13 is a diagram conceptually illustrating a system configuration of the settlement system 1 in the sixth embodiment. As shown in FIG. 13, the settlement system 1 includes an information processing device 10 that functions as a product registration device, a settlement device 30 that performs a settlement process for the price of the product registered by the information processing device 10, and a server device 40. In this case, the information processing device 10, the settlement device 30, and the server device 40 are communicatively connected via a network such as a LAN. The information processing device 10 has a configuration similar to that of any one of the first to fourth embodiments. In the following description, a case in which the information processing device 10 of this embodiment has the configuration of FIG. 1 as in the first embodiment is illustrated. It should be noted that the example of FIG. 13 is not limited to the example, and the image recognition unit 110, the code acquisition unit 122, the size / shape information acquisition unit 130, and the determination unit 140 of the information processing device 10 may be provided as a device (e.g., the server device 40) separate from the product registration device and communicatively connected to the product registration device via, for example, a LAN, as described in the fifth embodiment. In this case, the product registration device has a processing unit that acquires a product identification code from a product information symbol or the like, in addition to the code acquisition unit 122 of the information processing device 10.

[0080] [Example of operation] The flow of processing in the settlement system 1 of this embodiment will be described with reference to Fig. 14. Fig. 14 is a sequence diagram showing the flow of processing in the settlement system 1 of the sixth embodiment.

[0081] As described in the first embodiment, the information processing device 10 executes the processes of S101 to S105 in the flowchart of Fig. 4. Then, the determination of S106 in the flowchart of Fig. 4 is performed. If the determination unit 140 determines that the predetermined criterion is not satisfied (S106: NO), the determination unit 140 performs a predetermined output (for example, an output to prompt the user to re-recognize the product) (S107), and the processes of S101 to S105 are executed again. On the other hand, if the determination unit 140 determines that the predetermined criterion is satisfied (S106: YES), the product recognition unit 120 determines that the product has been normally recognized by the information processing device 10, and reads out product information linked to the product identification code acquired by the code acquisition unit 122 (S501). Specifically, the product recognition unit 120 uses the product identification code acquired by the code acquisition unit 122 to refer to a storage unit (not shown) that associates and stores the product identification code with product information (e.g., product name, product unit price, whether or not a discount is applied, etc.), and reads out the product information associated with the acquired product identification code. Then, the product recognition unit 120 adds the read out product information as information to be used in the settlement process executed by the information processing device 10 (hereinafter referred to as settlement information) (S502). The settlement information includes at least the name of each product to be settled, the unit price and quantity of each product to be settled, the total amount of all products to be settled, etc. The processes of S101 to S107, S501, and S502 are repeated until the registration of products to be settled in one transaction is completed (S503: NO). The information processing device 10 detects an event such as pressing a "subtotal" button via an input device or the like provided on the information processing device 10, and detects that registration of products to be settled in one transaction has been completed.

[0082] When it is detected that the registration of the product to be settled in one transaction has been completed (S503: YES), the information processing device 10 transmits the settlement information of the transaction generated in S502 to the settlement device 30 via the server device 40 (S504). The information processing device 10 identifies the settlement device 30 to which the settlement information should be sent, for example, by an input operation by the operator of the information processing device 10, and transmits the settlement information. In this case, the information processing device 10 transmits the settlement information and information specifying the settlement device 30 to which the settlement information should be sent to the server device 40. The server device 40 then transmits the settlement information to the settlement device 30 specified by the information processing device 10, and the settlement device 30 performs settlement processing of the price using the settlement information transmitted from the server device 40 (S505). Furthermore, without being limited thereto, the information processing device 10 may transmit the settlement information to, for example, the server device 40, and may also dispense to the customer a portable medium that contains information identifying the settlement information and is readable by the settlement device 30, and the portable medium may be read into any settlement device 30 by the customer's operation. In this case, the settlement device 30 reads the settlement information from the server device 40 using the identification information included in the portable medium, and performs settlement processing of the fee (S505).

[0083] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various configurations other than those described above can also be adopted.

[0084] For example, in the first embodiment, the imaging unit 124 may be configured to acquire an image of an unregistered product being taken out of a placement location (e.g., a basket) in order to hold the product before registration over the product recognition unit 120, and the image recognition unit 110 may be configured to recognize the size or shape of the product from the image. In this case, it is possible to determine whether the unregistered product taken out of the placement location matches the product actually registered as a payment target. In addition, for example, the imaging unit 124 may be configured to acquire an image of a registered product being placed in the placement location, and the image recognition unit 110 may be configured to recognize the size or shape of the product from the image. In this case, it is possible to determine whether the registered product placed in the placement location matches the product actually registered as a payment target. In this way, it is possible to discover fraudulent acts such as product swapping at the time of product registration, and to determine whether the recognized product is the correct product.

[0085] In addition, in the multiple flowcharts or sequence diagrams used in the above description, multiple steps (processes) are described in order, but the order of execution of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the figures can be changed to the extent that does not cause any problems in terms of content. In addition, each of the above-mentioned embodiments can be combined to the extent that the content is not contradictory.

[0086] Below, examples of reference forms are given. 1. an image recognition means for recognizing a size or shape of a product in an image by using an image associated with depth information indicating a distance in a depth direction of the product included in an imaging range; A code acquisition means for acquiring a product identification code of a product to be checked out; A size / shape information acquisition means for acquiring size / shape information associated with the acquired product identification code; a determining means for determining whether or not the size or shape of the product recognized from the image and the size / shape information acquired using the product identification code meet a predetermined standard; A product registration device comprising: 2. The thermography information acquisition means further includes a thermography information acquisition unit that acquires thermography information in association with the image, The image recognition means includes: Using the acquired thermographic information, a portion estimated to be a person is excluded from the shape of the product recognized from the image. 1. A product registration device as described in claim 1. 3. A plurality of pieces of size / shape information are linked to the product identification code, the determining means determines whether or not the recognized size or shape and each of the plurality of size / shape information satisfy the predetermined criterion. 1. A product registration device according to 1 or 2. 4. The method further includes imaging means for capturing the image in response to a timing when the code acquisition means reads the product identification code. 4. A product registration device according to any one of 1 to 3. 5. The image capturing apparatus further includes a number acquisition means for acquiring the number of products included in the image capturing range, The determination means determines whether or not the number of the acquired product identification codes matches the number of the acquired products. 5. A product registration device according to any one of 1 to 4. 6. The product identification code is further associated with a weight or weight deviation of the product corresponding to the product identification code, the determination means executes the determination in at least one of the following cases: when the weight associated with the acquired product identification code is equal to or greater than a reference value, when the weight associated with the acquired product identification code is unset, or when the weight deviation associated with the acquired product identification code is equal to or greater than a predetermined threshold value. 6. A product registration device according to any one of 1 to 5. 7. The computer Recognizing a size or shape of a product in an image using an image associated with depth information indicating a depth direction distance of the product included in an imaging range; Obtain the product identification code of the product to be checked out, Obtain size / shape information associated with the obtained product identification code; Calculating a difference between the size or shape of the product recognized from the image and the size / shape information acquired using the product identification code, and determining whether or not the difference satisfies a predetermined criterion; The control method includes: 8. The computer, Acquiring thermographic information in association with the image; Using the acquired thermographic information, a portion estimated to be a person is excluded from the shape of the product recognized from the image. The control method according to claim 7, further comprising: 9. A plurality of pieces of size / shape information are linked to the product identification code, The computer, determining whether the recognized size or shape and each of the plurality of size / shape information pieces satisfy the predetermined criteria; 9. The control method according to claim 7 or 8, further comprising: 10. The computer, capturing the image in accordance with a timing for reading the product identification code; The control method according to any one of 7 to 9, comprising: 11. The computer, Obtaining the number of products included in the imaging range; determining whether the number of the acquired product identification codes matches the number of the acquired products; The control method according to any one of 7. to 10., 12. The product identification code is further associated with a weight or weight deviation of the product corresponding to the product identification code, The computer, Executing the determination in at least one of the following cases: when the weight associated with the acquired product identification code is equal to or greater than a reference value, when the weight associated with the acquired product identification code is unset, or when the weight deviation associated with the acquired product identification code is equal to or greater than a predetermined threshold value. The control method according to any one of 7. to 11., 13. Computer, an image recognition means for recognizing a size or shape of a product in an image by using an image associated with depth information indicating a distance in a depth direction of the product included in an imaging range; A code acquisition means for acquiring a product identification code of a product to be checked out; A size / shape information acquisition means for acquiring size / shape information linked to the acquired product identification code; a determination means for calculating a difference between the size or shape of the product recognized from the image and size / shape information acquired using the product identification code, and determining whether or not the difference satisfies a predetermined criterion; A program to function as a 14. The computer, a thermography information acquisition means for acquiring thermography information in association with the image; The image recognition means, a means for excluding a portion estimated to be a person from the shape of the product recognized from the image by using the acquired thermographic information; 13. The program for further functioning as described above in claim 13. 15. A plurality of pieces of size / shape information are linked to the product identification code, The computer, the determining means for determining whether or not the recognized size or shape and each of the plurality of size / shape information satisfy the predetermined criterion; 15. The program according to 13. or 14. for further functioning as a 16. The computer, an imaging means for capturing the image in response to a timing when the code acquisition means reads the product identification code; 16. The program according to any one of 13. to 15., for further functioning as a 17. The computer, a quantity acquisition means for acquiring the number of products included in the imaging range; the determining means for determining whether or not the number of the acquired product identification codes matches the number of the acquired products; 17. The program according to any one of 13. to 16., further functioning as a 18. The product identification code is further associated with a weight or weight deviation of the product corresponding to the product identification code, The computer, the determination means executing the determination in at least one of the following cases: when the weight linked to the acquired product identification code is equal to or greater than a reference value, when the weight linked to the acquired product identification code is unset, or when the weight deviation linked to the acquired product identification code is equal to or greater than a predetermined threshold value; 18. The program according to any one of 13. to 17. for causing the program to function as a [Explanation of symbols]

[0087] 1. Payment System 10. Information processing device 101 CPU 102 Memory 103 Storage 104 Input / Output Interface 1041 Display device 1042 Input Device 1043 Imaging device 1044 Reading device 105 Communication Module 110 Image Recognition Unit 120 Product recognition department 122 Code Acquisition Department 124 Imaging unit 126 Thermography information acquisition unit 130 Size / shape information acquisition unit 132 Size / shape information storage unit 140 Judgment section 142 Weight information storage section 150 Counting Department 170 Weight information acquisition section 20. Basket 30 Payment device 40 Server device

Claims

1. a determining means for determining whether a product placed in a container for storing and transporting the product is invalid based on a change in weight of the container; an output means for outputting a warning when the determining means determines that the product placed in the container by the customer is an illegal product; Equipped with The information processing system, wherein the warning includes a message prompting the user to reload the product.

2. a weight information acquiring means provided in the container for acquiring weight information of the commodity placed in the container; The determining means determines the validity of the product placed in the container using the weight information. The information processing system according to claim 1 .

3. a product information acquiring means provided in the container for acquiring product information related to the product to be paid for; image information acquisition means provided in the container for acquiring image information of a commodity that a customer places in the container; Further equipped with The information processing system according to claim 1 , wherein the output means outputs a warning when the product placed in the container by the customer is determined to be invalid using information acquired by the product information acquisition means and the image information acquisition means.

4. The information acquired by the product information acquisition means is further linked to the weight or weight deviation of the product, The information processing system of claim 3, wherein the determination means determines the validity of the product placed in the container in at least one of the following cases: when the weight linked to the information acquired by the product information acquisition means is below a reference value, when the weight linked to the information acquired by the product information acquisition means is not set, or when the weight deviation linked to the information acquired by the product information acquisition means is above a predetermined threshold.

5. A computer provided in a container for storing and transporting a commodity, a determining step of determining that the product placed in the container is unauthorized based on a change in weight of the container; an output step of outputting a warning when the product placed in the container by the customer is determined to be an illegal product in the determination step; Run The method of control, wherein the warning includes a message prompting the user to reload the product.

6. A computer provided in a container for storing and transporting a product, a determining means for determining that the product placed in the container is an illegal product based on a change in weight of the container; an output means for outputting a warning when the determining means determines that the product placed in the container by the customer is an illegal product; Function as a The warning includes a message prompting the user to reload the product.

Images