Article registration device, and cart POS system mounting article registration device
The article registration device adjusts exposure time and brightness uniformly across multiple cameras in cart POS systems, addressing exposure inconsistencies and improving product recognition speed and accuracy.
Patent Information
- Application Number
- JP2023071998
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-26
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2043-04-26
AI Technical Summary
Existing cart POS systems face issues with exposure differences in images captured by multiple cameras due to varying distances from the product, leading to inconsistent brightness and delayed exposure adjustments, which can hinder accurate product recognition.
An article registration device with imaging devices and illumination units that adjust exposure time based on the detected position of the product within the basket, ensuring uniform brightness across all cameras by controlling the lighting period and exposure timing.
This approach ensures consistent image brightness, allowing for faster and more accurate product recognition and registration, reducing overexposure and enhancing the efficiency of the cart POS system.
Smart Images

Figure 2024157591000001_ABST
Abstract
Description
[Technical field]
[0001] An embodiment of the present invention relates to an article registration device that recognizes and registers an article in an image without exposure difference, and a cart POS system equipped with the article registration device. [Background technology]
[0002] Usually, when purchasing an item, the item is registered and paid for at a cash register installed in the store. Cart POS (Point Of Sales) systems have been proposed for retail stores equipped with shopping carts. In such cart POS systems, the shopping cart is equipped with an information terminal equipped with a touch panel, a barcode scanner, etc. When a user, who is a customer using a shopping cart, places an item to be purchased in the shopping cart, the user has the barcode scanner read the identification code (barcode or two-dimensional barcode (QR Code (registered trademark))) of the item to be purchased and registers the item using barcode recognition technology.
[0003] Furthermore, within the category of this system, there is a cart POS system that utilizes AI (Artificial Intelligence), and is called a frictionless cart, AI cart, smart cart, intelligent cart, etc. This shopping cart uses package recognition technology, in which multiple imaging devices are placed around the basket, and when a product to be purchased is put into or taken out of the basket, a picture of the product is taken of each product, and the product package is recognized from the captured images, automatically identifying the type of product and automatically registering the product.
[0004] However, when multiple imaging devices capture images of one product at the same time, the brightness of the images captured by each imaging device may differ depending on the location where the product is placed, for example, the center of the basket or the corner of the basket. That is, an imaging device that is close to the product placed in the basket captures a bright image, and an imaging device that is farther away captures a darker image than an image captured at a closer distance. In addition, it takes time for each imaging device to automatically adjust the exposure time to prevent whiteout due to overexposure, and it is expected that there will be a delay before the recognition of the product begins. In addition, it may not be possible to automatically adjust the brightness to the optimal brightness, and the product may not be recognized. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] Patent No. 6537054 Summary of the Invention [Problem to be solved by the invention]
[0006] The problem that the embodiments of the present invention aim to solve is to provide an item registration device and a cart POS system equipped with an item registration device that, when photographing items to be purchased with multiple cameras, adjusts the exposure timing of each camera depending on the position of the item inserted, thereby obtaining images without exposure differences, and uses the images to register items by package recognition or identification code recognition. [Means for solving the problem]
[0007] An item registration device according to one embodiment includes at least one imaging device which photographs an item placed into a shooting area, at least one lighting unit which illuminates the shooting area with a lighting period having a set exposure amount, a position detection unit which divides the shooting area into a plurality of areas and detects the area into which the item was placed under the illumination of the lighting unit, and an information terminal which sets an exposure time under the illumination of the lighting unit for each imaging device so as to achieve the set exposure amount based on the brightness of the illumination according to the distance between the detected area and the imaging device, and causes the imaging device to photograph the item, and recognizes and registers the item in the photographed image. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a cart POS system equipped with an article registration device according to the first embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing the main data structure of the information terminal table. [Diagram 3] FIG. 3 is a block diagram showing an example of a circuit configuration of an information terminal. [Figure 4] FIG. 4 is a diagram showing the overall configuration of a shopping cart in which cameras and lighting units are arranged at the four corners of the basket in the first embodiment. [Figure 5A] FIG. 5A is a diagram showing the brightness of the area of the first camera in an example in which cameras and lighting units are arranged at the four corners of a cage. [Figure 5B] FIG. 5B is a diagram showing the brightness of the area captured by the second camera in an example in which cameras and lighting units are arranged at the four corners of the cage. [Figure 5C] FIG. 5C is a diagram showing the brightness of the area captured by the third camera in an example in which cameras and lighting units are arranged at the four corners of the cage. [Figure 5D] FIG. 5D is a diagram showing the brightness of the area of the fourth camera in an example in which cameras and lighting units are arranged at the four corners of the cage. [Figure 6] FIG. 6 is a diagram showing the exposure timing of each camera when a product is placed in area A in an example in which cameras and lighting units are arranged at the four corners of a basket. [Figure 7] FIG. 7 is a diagram showing the overall configuration of a shopping cart in which cameras and lighting units are arranged at the center of the four sides of the opening of the basket in the second embodiment. [Figure 8A] FIG. 8A is a diagram showing the brightness of the area captured by the first camera in an example in which the camera and the lighting unit are arranged in the center of the side of the cage. [Figure 8B] FIG. 8B is a diagram showing the brightness of the area captured by the second camera in an example in which the camera and lighting unit are placed in the center of the side of the cage. [Figure 8C] FIG. 8C is a diagram showing the brightness of the area captured by the third camera in an example in which the camera and the lighting unit are disposed in the center of the side of the cage. [Figure 8D] FIG. 8D is a diagram showing the brightness of the area of the fourth camera in an example in which the camera and the lighting unit are arranged in the center of the side of the cage. [Figure 9] FIG. 9 is a diagram showing the exposure timing of each camera when a product is placed in area A in an example in which the cameras and lighting units are arranged in the center of the sides of the basket. [Figure 10] FIG. 10 is a diagram showing the exposure timing of each camera when a product is placed in area A in an example in which cameras and lighting units are arranged at the four corners of a basket in a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, an embodiment of a cart POS system equipped with an article registration device will be described with reference to the drawings. The article registration device of this embodiment is configured to include an information terminal, a camera as an imaging device, and a lighting unit as a lighting device, and is incorporated into the cart POS system. In the following description, the identification code attached or printed on the product is assumed to be a barcode or a two-dimensional barcode (QR Code (registered trademark)). In addition, photographing a product with a camera as an imaging device is referred to as photographing or exposure, the shutter release time or the time that light is applied to the imaging element is referred to as exposure time, and the amount of light incident on the camera at that time is referred to as exposure amount. The brightness (or luminance value) of an area is considered to be equivalent to the brightness (exposure amount) of the light image (image) of the product incident on the camera when the product is photographed by the camera after being placed in an area under illumination.
[0010] [First embodiment] Fig. 1 is an overall configuration diagram of a cart POS system 1 according to a first embodiment. The cart POS system 1 includes a store server 2, a cart controller 3, an access point (indicated as AP in Fig. 1) 4, and a payment machine 5. The store server 2, the cart controller 3, the access point 4, and the payment machine 5 are connected to a network 6 such as a LAN (Local Area Network) or a wireless LAN. The cart POS system 1 includes a plurality of information terminals 7 connected to the network 6 by wireless communication via the access point 4.
[0011] The item registration device 100 is a device that allows a customer, who is a purchaser, to register an item to be purchased by himself / herself using an information terminal 7 and a camera 8. The information terminal 7, the camera 8 (corresponding to 8a to 8d in FIG. 3), and the lighting unit 9 (corresponding to 9a to 9d in FIG. 3) are provided in a plurality of shopping carts 10 that are used by customers when shopping in a store. In the following, the shopping cart 10 will be simply referred to as the cart 10. The plurality of carts 10 are an example of a carrier that transports the items purchased by the customers. Although not shown, each cart 10 is equipped with a rechargeable battery for driving the information terminal 7, the camera 8, and the lighting unit 9.
[0012] The information terminal 7 is an information processing device that functions as a user interface for operations related to registering products or canceling registration (hereinafter simply referred to as "cancellation"). The information terminal 7 is equipped with a wireless unit as a means for communicating with the access point 4. The access point 4 relays communications between the information terminal 7 and each device connected to the network 6, namely, the store server 2, the cart controller 3, and the payment machine 5. Although only one access point 4 is shown in FIG. 1, there may be two or more access points 4 depending on the size of the store, etc.
[0013] The accounting machine 5 is a device that allows a store clerk or a customer to make a payment (settlement) for registered products. When a store clerk makes a payment, for example, a well-known face-to-face POS terminal is used as the accounting machine 5, and when a customer makes a payment, a well-known self-service POS terminal is used. There is no particular limit to the number of accounting machines 5.
[0014] The store server 2 supports store operations such as accounting, purchasing, inventory management, sales, and customer service (customer support, complaint handling, etc.) To support these operations, the store server 2 manages a product database (DB) 11, a member database (DB) 12, etc.
[0015] The product database 11 stores product records that describe product data for each product sold in the store. The product records include product codes, prices, product names, etc. The product codes are product identification codes. The product database 11 also includes a sales database. The sales database stores sales records that describe sales data for each item, such as product, department, time period, date, week, month, etc. The sales data includes sales units, sales amounts, etc.
[0016] In the present embodiment, the product database 11 stores pre-trained learning data 13 for recognizing products stored in the product database 11 by AI (Artificial Intelligence). The learning data 13 is a collection of product-specific learning data related to information on each product sold in the store (including contents, external shape, package / wrapping pattern, and written product name (character form), etc.), and each product-specific learning data is incorporated into the learning data 13 in association with, for example, a product code. The learning data 13 can be updated according to an update of the product database 11 in response to the addition or replacement of a product sold in the store. The product-specific learning data for each product may be created by a provider of the product and provided to the store, or may be created independently by the store.
[0017] Member database 12 stores a member record that describes member data for each member. The member data includes a member ID, name, gender, email address, etc. The member ID is a code such as a unique member number that is assigned to each member to identify each member individually. Members own a recording medium on which the member ID is recorded. The recording medium may be, for example, a magnetic card, a contact type IC (Integrated Circuit) card, a contactless IC card, a smartphone, etc.
[0018] Although not shown in the figure, the store server 2 may further manage a member payment database. The member payment database stores a member payment record for each member that describes data necessary for payment. In addition to the member ID, the member payment record includes an electronic money identification code and an electronic money user ID as data necessary for payment.
[0019] The cart POS system 1 according to this embodiment allows the member to make a payment without using the accounting machine 5, i.e., without paying in cash. However, the payment method is limited to one type, electronic money. The electronic money identification code is an identification code of the electronic money registered in advance for use in payment by the member identified by the member ID. The electronic money user ID is identification information that the electronic money center EMC uses to identify the user of the electronic money, and is assigned by the electronic money center EMC.
[0020] In order to be able to pay with electronic money, a member registers the electronic money identification code and electronic money ID in advance, for example, when registering as a member. If electronic money has not been registered in advance, "NULL" or "0000" is stored as the electronic money identification code and electronic money ID in the accounting settlement record. When electronic money is made available in this way, the store server 2 has a communication server function, or the cart POS system 1 includes a server. The store server 2 or the communication server supports data communication processing carried out via an external communication network such as the Internet or a network dedicated to electronic money between the accounting machine 5 or the cart controller 3 and an electronic money center that supports payments with electronic money.
[0021] The cart controller 3 cooperates with the information terminal 7 to support the information terminal 7 to function as a POS terminal. The cart controller 3 has an information terminal table 14 corresponding to each information terminal 7. In this embodiment, an example is described in which the information terminal table 14 is provided in the cart controller 3, but a storage area may be provided in the auxiliary storage device 23 of the information terminal 7, and the information terminal table 14 may be set in each information terminal 7. In such a case, the store server 2 may manage the information terminal table 14 together with the information terminal 7. The information terminal table 14 is a part of the components of the item registration device 100.
[0022] FIG. 2 is a schematic diagram showing the main data structure of the information terminal table 14. As shown in FIG. The information terminal table 14 has a terminal ID area 14a, a member ID area 14b, one or more purchased product data areas 14c, and a subtotal area 14d. The terminal ID area 14a stores a terminal ID, which is a code such as a terminal number for uniquely identifying the information terminal 7 corresponding to the information terminal table 14. The member ID area 14b stores the member ID. The purchased product data area 14c creates a purchased product list each time a customer registers a product to be purchased on the information terminal 7, i.e., stores purchased product data related to the purchased product.
[0023] Alternatively, the purchased item list is updated each time a customer stops purchasing and cancels the registration of a product that has already been registered on the information terminal 7. The purchased item list includes the product code, unit price, sales points (quantity), product name, sales amount, tax amount, etc. of the purchased items purchased by the customer. The subtotal area 14d is amount data indicating the total amount of the sales amount and tax amount for all purchased items by the customer, calculated by aggregating the purchased item list stored in the purchased item data area 14c. This amount data, like the purchased item list, is updated each time a product to be purchased by the customer is registered or canceled.
[0024] FIG. 3 is a block diagram showing the main circuit configuration of the information terminal 7. As shown in FIG. 3, the information terminal 7 includes a processor 21, a main memory 22, an auxiliary storage device 23, a wireless unit 26, a touch panel 27, a speaker 28, and a system transmission path 29. The information terminal 7 may be a dedicated device or a general-purpose device such as a tablet computer. The system transmission path 29 includes an address bus, a data bus, a control signal line, a driving power supply line, and the like.
[0025] The information terminal 7 connects the processor 21, the main memory 22, the auxiliary storage device 23, the wireless unit 26, the touch panel 27, and the speaker 28 to a system transmission path 29. Furthermore, the information terminal 7 is connected to the reader 16, the cameras 8a to 8d, the lighting units 9a to 9d, the patrol lamp 17, and the position detection unit 18 by the system transmission path 29 via interfaces (not shown).
[0026] The information terminal 7 is a computer that is configured with a processor 21, a main memory 22, an auxiliary storage device 23, and a system transmission line 29 that connects these together. The processor 21 corresponds to the central part of the computer. The processor 21 controls each part of the information terminal 7 to realize various processes including product registration and cancellation processes, package recognition processes, and identification code recognition processes according to an operating system or application programs. The processor 21 controls the shooting timing (exposure timing) of the camera using a trigger signal, which is an arbitrarily set drive start pulse signal.
[0027] The processor 21 switches the photographing mode of the cameras 8a to 8d from a continuous image mode for package recognition, described later, to an identification code close-up mode for identifying code recognition, described later. Conversely, when switching is performed to return the application program from the identification code recognition process to the package recognition process, the photographing mode of the cameras 8a to 8d is also switched from the identification code close-up mode back to the continuous image mode.
[0028] As the photographing modes in the embodiment, the continuous image mode for package recognition is a photographing mode (moving image mode) in which one still image or one frame is photographed continuously, and the identification code close-up mode for identifying the identification code is a photographing mode in which the exposure time is shortened and the focal length is shortened (corresponding to shortening the reading depth (DOF: Depth Of Field) of a barcode reader) in comparison with the continuous image mode described above.
[0029] The processor 21 uses, for example, a central processing unit (CPU) as a device, but is not limited to this. In addition, the processor 21 may be multi-core / multi-threaded and can execute multiple processes in parallel. The processor 21 may also be equipped with an AI chip specialized for AI. Furthermore, the processor 21 may be realized in various other forms, including integrated circuits such as a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and a digital signal processor (DSP).
[0030] The main memory 22 corresponds to the main storage portion of the computer. The main memory 22 includes a nonvolatile memory area and a volatile memory area. The main memory 22 stores at least an operating system or an application program in the nonvolatile memory area. The main memory 22 may store data required for the processor 21 to execute processes for controlling each part in a nonvolatile or volatile memory area. The main memory 22 uses the volatile memory area as a work area where data is appropriately rewritten by the processor 21. The nonvolatile memory area is, for example, a ROM (Read Only Memory). The volatile memory area is, for example, a RAM (Random Access Memory).
[0031] The auxiliary storage device 23 corresponds to the auxiliary storage portion of the computer. For example, an EEPROM (registered trademark) (Electric Erasable Programmable Read-Only Memory), an HDD (Hard Disc Drive), or an SSD (Solid State Drive) can be the auxiliary storage device 23. The auxiliary storage device 23 stores data used by the processor 21 in performing various processes, or data created by the processes in the processor 21.
[0032] The auxiliary storage device 23 may store application programs that execute the above functions and processes. In this embodiment, as the application program, a terminal program 24 is saved for executing the functions of the information terminal 7, the cameras 8a-8d, and the lighting units 9a-9d as product registration according to the embodiment (product registration and cancellation functions, package recognition processing, and identification code recognition processing). The terminal program 24 may be stored in the main memory 22.
[0033] The auxiliary storage device 23 also stores learning data 25 learned for product recognition by AI for each product sold at the store that is stored in the product database 11 of the store server 2. This learning data 25 may be a copy of the learning data 13 included in the product database 11 of the store server 2. The timing for copying this learning data 13 as the learning data 25 may be a specific timing, such as each time the information terminal 7 is started up, or may be an arbitrary timing, such as when the learning data 25 is distributed from the store server 2 to each information terminal 7 when it is updated.
[0034] The wireless unit 26 wirelessly communicates data with the access point 4 in accordance with a wireless communication protocol. The touch panel 27 is a device that serves as both an input device and a display device for the information terminal 7. The touch panel 27 displays an image based on image data output from the processor 21, detects a touch position on the displayed image, and outputs the touch position information to the processor 21. As shown in Fig. 4, the information terminal 7 is attached to the cart 10 in an orientation and position that allows a customer to visually check the display of the touch panel 27 and easily perform a touch operation.
[0035] The speaker 28 is housed within the housing of the information terminal 7, and reproduces sounds based on the sound data output from the processor 21. The reproduced sounds include voice guidance, confirmation sounds, alert warning sounds, etc. The voice guidance, for example, instructs by voice to read the identification code of a product and perform identification code recognition when package recognition of the product fails. Also, information related to calling a store clerk (announcement that a store clerk is being called, the voice of the store clerk who responds, etc.) may be issued by voice.
[0036] The reader 16 reads data recorded on the recording medium and outputs the read data to the processor 21. The reader 16 is a magnetic card reader when the recording medium is a magnetic card, and is an IC card reader when the recording medium is a contact IC card. In the case of a recording medium using RFID (Radio Frequency Identification) such as a non-contact IC card or a smartphone, an RFID reader is used as the reader 16. The position where the reader 16 is attached to the cart 10 is not particularly specified. For example, as shown in FIG. 4, it can be attached adjacent to the information terminal 7.
[0037] The position detection unit 18 detects the position where the product was inserted (20 areas from area A to area T shown in Figs. 5A to 5D and Figs. 8A to 8D). The position detection unit 18 may be configured with a sensor element such as a distance sensor, or may calculate the position of the product from image data, for example, by using parallax of the same product in images taken by different cameras 8 at non-opposing positions to calculate the position of the product through arithmetic processing (triangulation, etc.). When the position of the product is calculated through arithmetic processing, the processing may be performed by the processor 21. In this case, the position detection unit 18 is provided within the processor 21.
[0038] 4, four cameras 8a to 8d, which are imaging devices, are provided on the cart 10 so as to capture images of the inside of a basket 19 of the cart 10 from the four upper corners. In this example, one camera 8 (8a to 8d) and two lighting units 9 (9a to 9d) are grouped together as one imaging unit 15 (15a to 15d), and one imaging unit is disposed at each corner so as to surround the edge of the square at the top of the basket 19. A processor 21 controls the operations of the cameras 8a to 8d and the lighting units 9a to 9d.
[0039] In this arrangement example, LED (Light Emitting Diode) lamps that serve as light sources for the illumination units 9a-9d are provided on both sides of the cameras 8a-8d. The illumination units 9a-9d are not always on, but are turned on when package recognition and identification code recognition are performed using the cameras 8a-8d. Here, an example is shown in which two illumination units 9a-9d are arranged on both sides of the cameras 8a-8d, but a configuration in which one illumination unit 9a-9d is arranged on either side of the cameras 8a-8d may also be used.
[0040] In this embodiment, the lighting units 9a to 9d are arranged adjacent to the cameras 8a to 8d on the sides thereof, but the positions of the lighting units 9a to 9d are not limited as long as the cameras 8a to 8d can illuminate the inside of the photographing areas (areas A to T shown in FIG. 5A and the like). For example, the lighting units 9a to 9d may be arranged above or below the cameras 8a to 8d at the opening of the basket 19. In the example shown in FIG. 4, the thickness of the upper edge of the basket 19 is increased and the basket 19 is configured to be accommodated within the upper edge and to eliminate any protruding parts, so that the products do not collide with the cameras 8a to 8d and the lighting units 9a to 9d when putting products in or taking them out of the basket 19. In this embodiment, a configuration example in which four cameras are arranged at the four corners has been described as an example, but the number of cameras is not limited to four, and it is possible to implement this embodiment using one or more cameras.
[0041] In this embodiment, the image used for identification code recognition needs to have as short an exposure time as possible to prevent blurring of the photographed bars. However, in order to capture a brighter image by shortening the exposure time, lighting is required to increase the exposure of the subject. In this embodiment, the illumination light of the illumination units 9a to 9d is set to be lit for an exposure period based on a predetermined exposure amount using, for example, LED lamps when a commodity is put into any area in the basket 19. This exposure amount is an exposure amount that allows the package recognition process and the identification code recognition process to be properly performed and does not cause whiteout. In addition, the exposure time is set for this exposure amount with the brightness of an area with the lowest luminance value described later as a reference value. In response to this lighting, the cameras 8a to 8d capture images with exposure times that obtain exposure amounts 31a to 31d, etc., described later. The processor 21 controls the light emission of the illumination units 9a to 9d, thereby making it possible to reduce power consumption of the battery mounted on the cart 10.
[0042] Furthermore, the information terminal 7 may be equipped with a patrol lamp 17 as indicated by a dashed line in FIG. The patrol lamp 17 is connected to the processor 21 by a system transmission line 29 via an interface (not shown). The operation of the patrol lamp 17 is controlled by the processor 21, and the patrol lamp 17 is a lamp for notifying customers using the cart 10 and store staff of the location of a customer who needs attention and for issuing a warning.
[0043] Next, it is determined whether or not a customer has put an item into or taken out of basket 19 as follows. Cameras 8a-8d take successive images, or continuous photographs, of items put into or taken out of basket 19 of cart 10. Here, for the successive images taken, the latest image data is compared with the oldest image data stored immediately before, as needed, to detect any changes in the image (composition) relating to the item.
[0044] When a change occurs in the captured image, it is necessary to determine whether the product has been put into or taken out of the basket 19 based on the direction of movement in the photographing area of the camera 8. For example, if it is determined that the product has crossed the photographing area in a downward direction approaching the bottom surface of the basket 19 in the vertical direction intersecting with the bottom surface of the basket 19, it is determined that the product has been "put in". On the other hand, if it is determined that the product has crossed the photographing area in a direction moving upward away from the bottom surface, it is determined that the product has been "taken out". Note that if the product does not cross the photographing area in the vertical direction but moves diagonally downward or horizontally for a short distance, it is considered that the product in the basket 19 has simply shifted position, and there has been no change in the detected image. Furthermore, in addition to determining based on the direction of movement of the product, a weight scale (not shown) may be placed on the bottom surface of the basket 19 to detect the putting in or taking out of the product based on the increase or decrease in weight.
[0045] Furthermore, when a product is put into the basket 19 or taken out of the basket 19, the processor 21 recognizes the package of the put-in or taken-out product using a package recognition process using AI. Specifically, the processor 21 sets a rectangular area called a bounding box for the product in the latest image that has been moved within the image, and performs image recognition of the product within the bounding box based on the learning data 25 stored in the auxiliary storage device 23, and recognizes which product that is.
[0046] Next, when the product is recognized by the package recognition process, the processor 21 registers the recognized product in the purchased product list in the information terminal table 14 described above, or cancels the registration, and updates the purchased product list. Specifically, a product registration notification is sent from the information terminal 7 to the cart controller 3. The cart controller 3, which has received the product registration notification, updates the purchased product list by adding or deleting purchased product data related to the recognized product to the purchased product list in the purchased product data area 14c of the information terminal table 14 associated with the member ID of the information terminal 7. The purchased product list is updated every time a product that the customer wishes to purchase is registered or cancelled on the information terminal 7.
[0047] On the other hand, if the processor 21 cannot recognize the package of the inserted or removed product, it switches to identification code recognition processing and photographs the product's identification code. The identification code contained in the photographed image is analyzed to recognize the product. If the product is recognized by such identification code recognition processing, the purchased product list in the information terminal table 14 is updated in the same manner as described above.
[0048] Next, when the customer touches the payment button on the product registration screen of the information terminal 7, the payment process is carried out by the payment machine. As part of the transaction process, for example, the processor 21 of the payment machine 5 generates a barcode based on the terminal ID assigned to the information terminal 7 and displays the barcode on the touch panel 27. The payment machine 5 reads the barcode and sends a transaction inquiry command including the terminal ID to the cart controller 3. The cart controller 3 generates payment data based on the purchased item list registered in the purchased item data area 14c of the information terminal table 14 that corresponds to the terminal ID included in the transaction inquiry command, and returns the payment data to the payment machine 5 that sent the transaction inquiry command. The payment machine 5 prompts the customer to make payment based on the received payment data, and makes payment for the purchased items.
[0049] With reference to Figs. 5A to 5D, the brightness of the area into which the product is placed with respect to each camera in an example in which cameras and lighting units are arranged at the four corners of the basket 19 will be described. First, Fig. 5A is a diagram showing the brightness (luminance value) of an area as seen by camera 8a under illumination light with cameras 8a-8d and lighting units 9a-9d placed at the four corners of basket 19 and all lighting units 9a-9d turned on. Here, the brightness of an area also means the amount of exposure light incident on camera 8a per unit time when photographing with camera 8a. As shown in the figure, the bottom surface of basket 19 into which products are placed is divided into, for example, 20 areas A-T. This number of divisions is an arbitrary number and may be set appropriately depending on the size of the bottom surface, etc.
[0050] If the brightness of the illumination light in each area as seen by the camera 8a is divided into five levels, from "1" (dark) to "5" (bright), the areas A to T have a luminance distribution (brightness per unit area as seen from above) or a light quantity distribution (amount of light per unit area) as shown in Fig. 5A. Note that the brightness when all the illumination units 9a to 9d are turned off is set to "0".
[0051] The brightness of each area A to T is proportional to the distance from the camera 8a, and the farther the distance, the darker it becomes. That is, with respect to the camera 8a, area A is at level "5", areas B, F, and G are at level "4", areas C, H, K, L, and M are at level "3", areas D, I, N, P, Q, R, and S are at level "2", and areas E, J, O, and T are at level "1".
[0052] Similarly, under the illumination light with all lighting units 9a to 9d turned on, Figure 5B shows the brightness of each of areas A to T as seen from camera 8b, Figure 5C shows the brightness of each of areas A to T as seen from camera 8c, and Figure 5D shows the brightness of each of areas A to T as seen from camera 8d.
[0053] Here, the amount of exposure incident on each of the cameras 8a to 8d from the area A when the camera 8 shoots with the same exposure time will be described. The amount of exposure incident on each of the cameras 8a to 8d is proportional to the brightness value, which is the brightness of the area. Therefore, in Figs. 5A to 5D, the level of the area A is "5" for the camera 8a, "2" for the camera 8b, and "1" for the cameras 8c and 8d. In order to make the brightness of the images shot by these cameras 8a to 8d the same, it is necessary to make the amount of exposure the same. Therefore, since the amount of illumination light irradiated by all the illumination units 9a to 9d cannot be increased or decreased linearly, it is adjusted by the exposure time of the camera. That is, under illumination in which all the illumination units 9a to 9d are turned on, the reference exposure time is set so that the exposure amount of the cameras 8c and 8d with the lowest level of "1" is appropriate, and the exposure times of the cameras 8a and 8b with the other levels are adjusted. As a result, the camera 8a needs to perform exposure control to shorten the exposure time so as to reduce the amount of incident light by four steps. Similarly, the camera 8b needs to perform exposure control to shorten the exposure time by dropping it by three steps.
[0054] Next, with reference to Fig. 5A to Fig. 5D and Fig. 6, the exposure control in the cameras 8a to 8d when the product to be purchased is placed in the area A will be described. Fig. 6 is a diagram showing the exposure timing and exposure amount of each of the cameras 8a to 8d photographing the area A under the illumination light with all the illumination units 9a to 9d turned on. Note that the exposure amounts 31a to 31d shown in Fig. 6 are the same value although the incidence periods are different because the amount of light per unit time incident on the cameras 8a to 8d is different.
[0055] In this embodiment, the exposure timing of each of the cameras 8a to 8d (the start time of the shutter opening, or the start time of light incidence on the image sensor) is initiated by the input of a trigger signal output by the processor 21. The exposure time of each of the cameras 8a to 8d (the shutter opening period, or the light receiving period of the image sensor) is preset to a fixed time common to all the cameras at the time of manufacture or shipment. Of course, the exposure time can also be changed by changing the settings of the application program in the processor 21.
[0056] All of the lighting units 9a to 9d are turned on and off simultaneously under the control of the processor 21. Of course, the processor 21 can also turn on and off the lighting units 9a to 9d individually. As described above, the lighting time (lighting period) under the control of the processor 21 is set so that when all of the lighting units 9a to 9d are turned on, the exposure amount of the cameras 8c, 8d at level "1" becomes the exposure amount 31c, 31d that can properly perform package recognition and identification code recognition.
[0057] Therefore, the exposure time of the cameras 8c and 8d and the lighting time of all the lighting units 9a to 9d are the same period. In addition, in the camera 8a, the exposure time (exposure period) is shortened to 1 / 5 in order to reduce the incident exposure amount 31a so that the brightness of the image drops by four steps with respect to the exposure amounts 31c and 31d of the cameras 8c and 8d as described above. Note that this 1 / 5 time is a time reduction for dropping the brightness by four steps, and if the exposure time and the attenuation of the brightness are not in a linear proportional relationship, the exposure time does not necessarily become 1 / 5 when dropping the brightness by four steps. Therefore, the processor 21 delays the trigger signal for starting the exposure of the camera 8a by 4 / 5 time with respect to the exposure start timing of the cameras 8c and 8d, which is the lighting start time of the lighting unit 9, and inputs it to the camera 8a. That is, the exposure start time is delayed by 4 / 5 time from the lighting start time of the lighting unit 9. Similarly, the trigger signal is delayed by ⅕ of the exposure start timing of the cameras 8c and 8d so as to lower the brightness by one step, and then input to the camera 8b.
[0058] Next, exposure control in cameras 8a to 8d when products put into basket 19 span multiple areas will be described. Here, for example, the center of all areas A to T is set as the judgment reference point, and in FIG. 5A, the center of the adjacent side of area H and area M is set as the judgment reference point. Also, a diagonal line in a direction crossing the front of the camera may be set as the judgment reference line. The positions of these judgment reference points and judgment reference lines can be set at any positions.
[0059] When the position of the product put into the basket 19 is an area farther away from the judgment reference point as seen from each of the cameras 8a to 8d, the brighter area is selected, and when the position of the product is an area closer to the judgment reference point, the darker area is selected. For example, when a product is put into a position straddling area A and area B, camera 8a selects area B on the darker side since the position is closer to the judgment reference point. Camera 8b selects either area A or area B since area A or area B has the same brightness. Furthermore, camera 8c selects area B on the brighter side since the position is farther away from the judgment reference point, and similarly, camera 8d selects area B. Of course, this setting method is only an example, and other setting methods may be used. For example, it is also possible to select the brightness of the area on the side where more than half of the products are hanging, using the ratio of products straddling multiple areas as a judgment criterion.
[0060] In this way, by delaying the timing of starting camera exposure according to the difference in the brightness (luminance) of the illumination light that differs for each area in the basket 19, the amount of light exposure received by each camera 8a to 8d can be made the same, and the brightness of the captured image can be made uniform at a level that allows image recognition. This also makes it possible to prevent blown-out highlights due to overexposure. Furthermore, compared to the conventional automatic adjustment of exposure time, the exposure time can be adjusted to a shorter time by delaying the input of the trigger signal, and the registration process for purchased items can be speeded up.
[0061] [Second embodiment] Next, a second embodiment will be described with reference to FIG. 7 and FIGS. 8A to 8D. The cart POS system 1 according to this embodiment is configured in the same manner as the configuration of the first embodiment described above, except for the arrangement of the cameras 8a-8d and the lighting units 9a-9d attached to the basket 19 of the cart 10. In the following description, the configuration units and reference symbols of the first embodiment described above will be used to simplify detailed description, and this embodiment will be described.
[0062] 7, in this embodiment, four cameras 8a to 8d, which are imaging devices, are provided on the cart 10 so as to capture images of the inside of a basket 19 of the cart 10 from all four sides above. In this example, one camera 8 (8a to 8d) and two lighting units 9 (9a to 9d) are treated as one imaging unit 15 (15a to 15d), and one imaging unit is disposed at the center of each of the four sides of the upper part of the basket 19. A processor 21 controls the operations of the cameras 8a to 8d and the lighting units 9a to 9d.
[0063] 8A to 8D, the brightness of the areas into which products are placed for each of the cameras 8a to 8d in an example in which the cameras 8a to 8d and the lighting units 9a to 9d are arranged in the center of the four sides of the basket 19 will be described. As shown in the figure, the bottom surface of the basket 19 into which products are placed is divided into, for example, 20 areas A to T. This number of divisions is an arbitrary number and may be set appropriately depending on the size of the bottom surface, etc.
[0064] First, FIG. 8A shows an example in which the cameras 8a-8d and the lighting units 9a-9d are arranged at the center of the sides of the cage 19, and is a diagram showing the brightness (luminance value) of the area as seen from the camera 8a arranged at the center of the long side under the illumination light with all the lighting units 9a-9d turned on. Here, the brightness of the area also means the amount of exposure light incident on the camera 8a per unit time when photographing with the camera 8a. When the brightness of the illumination light as seen from the camera 8a is divided into five levels from "1" (dark) to "5" (bright), the areas A to T have the brightness distribution (brightness per unit area as seen from above) or light amount distribution (amount of light per unit area) as shown in FIG. 8A. Note that the brightness when all the lighting units 9a-9d are turned off is set to "0".
[0065] The brightness of each area A to T is proportional to the distance from the camera 8a, and the farther the distance, the darker it becomes. That is, with respect to the camera 8a, area C is at level "5", areas B, D, GH, I, and J are at level "4", areas A, E, F, J, K, L, M, N, and O are at level "3", and areas P, Q, R, S, and T are at level "2".
[0066] 8B is a diagram showing the brightness (luminance value) of areas seen from camera 8b placed in the center of the short side under illumination light with all lighting units 9a to 9d turned on, in an example in which cameras 8a to 8d and lighting units 9a to 9d are placed in the center of the sides of cage 19. That is, with respect to camera 8b, areas F and K are at level "5", areas A, G, L and P are at level "4", areas B, H, M and Q are at level "3", areas C, I, N and R are at level "2", and areas D, E, J, O, S and T are at level "1".
[0067] Similarly, under illumination light with all lighting units 9a to 9d turned on, FIG. 8C shows the brightness of each of areas A to T as seen by camera 8c, and FIG. 8D shows the brightness of each of areas A to T as seen by camera 8d.
[0068] Next, the amount of exposure light incident on each of the cameras 8a to 8d from the area A when the cameras 8a to 8d have the same exposure time will be described. In area A of FIGS. 8A to 8D, camera 8a is at level "3", camera 8b is at level "4", camera 8c is at level "2", and camera 8d is at level "1". In order to make the brightness (exposure amount) of these cameras 8a to 8d the same level, the exposure time is adjusted as described above. That is, under illumination with all the illumination units 9a to 9d turned on, the reference exposure time is set so that the exposure amount of camera 8d with the lowest level "1" is appropriate, and the exposure times of the cameras 8a, 8b, and 8c with other levels are adjusted. As a result, camera 8a needs to perform exposure control to shorten the exposure time so as to reduce the amount of incident light by three steps. Similarly, camera 8b needs to perform exposure control to shorten the exposure time by two steps, and camera 8c needs to perform exposure control to shorten the exposure time by one step.
[0069] Next, with reference to Fig. 8A and Fig. 9, the exposure control in the cameras 8a to 8d when the product to be purchased is placed in the area A will be described. Fig. 9 is a diagram showing the exposure timing and exposure amount of each of the cameras 8a to 8d photographing the area A under the illumination light with all the illumination units 9a to 9d turned on. Note that the exposure amounts 32a to 32d shown in Fig. 9 are the same value although the incidence periods are different because the amount of light per unit time incident on the cameras 8a to 8d is different.
[0070] In this embodiment, the exposure timing of each of the cameras 8a to 8d (the start time of the shutter opening, or the start time of light incidence on the image sensor) is initiated by the input of a trigger signal output by the processor 21. The exposure time of each of the cameras 8a to 8d (the shutter opening period, or the light receiving period of the image sensor) is preset to a fixed time common to all the cameras at the time of manufacture or shipment. Of course, the exposure time can also be changed by changing the settings of the application program in the processor 21.
[0071] All the illumination units 9a to 9d are simultaneously turned on and off under the control of the processor 21. As described above, the lighting time (lighting period) under the control of the processor 21 is set so that when all the illumination units 9a to 9d are turned on, the exposure amount of the camera 8d at level "1" is an exposure amount that allows package recognition and identification code recognition to be performed appropriately. Therefore, the exposure time of the camera 8d and the lighting time of all the illumination units 9a to 9d are the same period.
[0072] In camera 8a, as described above, it is necessary to shorten the exposure time to 3 / 5 in order to reduce the incident exposure amount 32a so that the brightness of the image drops by two steps relative to the exposure amount 32d of camera 8d. Therefore, processor 21 delays the trigger signal for starting the exposure of camera 8a by 2 / 5 time with respect to camera 8d and inputs it to camera 8a. Note that this 2 / 5 time is a time reduction for lowering the brightness by three steps, and if the exposure time and the attenuation of the brightness are not in a linear proportional relationship, the exposure time does not necessarily become 2 / 5 when the brightness is lowered by three steps.
[0073] In addition, the trigger signal of the camera 8b is delayed by 3 / 5 of the time relative to the camera 8d so as to be three steps lower, and is input to the camera 8b. The trigger signal of the camera 8c is delayed by 1 / 5 of the time relative to the camera 8d so as to be one step lower, and is input to the camera 8c.
[0074] In this way, by delaying the timing of starting the exposure of the cameras according to the difference in the brightness (luminance) of the illumination light that differs for each area in the cage 19, the amount of light exposure received by each camera 8a to 8d due to the exposure can be made the same, and the brightness of the captured image can be made uniform at a level that allows image recognition. This also makes it possible to prevent blown-out highlights due to overexposure.
[0075] Next, a modified example will be described with reference to FIG. 10 is a diagram showing the exposure timing of each camera in an example in which cameras 8a-8d and lighting units 9a-9d are arranged at the four corners of basket 19 in a modified example, when a commodity is put into area A. In the above-described first and second embodiments, the exposure time of the camera is delayed to reduce the amount of incident light exposure so that the brightness of the image decreases.
[0076] In this modified example, all the lighting units 9a to 9d are turned on and shooting begins at an exposure timing where the exposure of each of the cameras 8a to 8d starts simultaneously, and the exposure ends after an exposure time according to the brightness of the area for each camera has elapsed. In this modification, the exposure time of the camera can be changed by changing the register value that sets the exposure time of the imaging element (image sensor). For example, as shown in Fig. 5A described above, the exposure time according to the brightness of each area A to T in each camera under illumination is calculated. An exposure time table that sets the register value corresponding to the exposure time for each area of each camera is created, and is rewritably stored in the auxiliary storage device 23 of the information terminal 7.
[0077] Processor 21 reads out the register value of the corresponding area from the exposure time table in auxiliary storage device 23 based on the area into which the product to be purchased, detected by position detection unit 18, is inserted, and changes the register value of the image sensor. After that, the product is photographed by the camera, and an image of the product is obtained.
[0078] In Fig. 10, as shown in Fig. 5A described above, the cameras 8c and 8d have a brightness level of "1", and in order to obtain the exposure amounts 33c and 33d, the exposure time and the lighting time of all the illumination units 9a to 9d are the same period. Also, the exposure time of the camera 8a is shortened to 1 / 5 in order to reduce the incident exposure amount 33a so that the brightness of the image drops by four steps with respect to the exposure amounts 31c and 31d of the cameras 8c and 8d. Similarly, the exposure time of the camera 8b is shortened to 4 / 5 in order to reduce the incident exposure amount 33b so that the brightness drops by one step. According to this modification, the exposure of each of the cameras 8a to 8d is completed within the period during which the illumination units 9a to 9d are turned on, so that shooting can be performed at a higher speed than in the first and second embodiments described above.
[0079] In addition, one embodiment of the present invention is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]
[0080] 1...Cart POS system, 2...Store server, 3...Cart controller, 4...Access point, 5...Accounting machine, 6...Network, 7...Information terminal, 8...Camera, 9...Lighting unit, 10...Shopping cart (cart), 11...Product database, 12...Member database, 13...Learning data, 14...Information terminal table, 14a...Terminal ID area, 14b...Member ID area, 14c...Purchased product data area, 14d...Subtotal area, 15...Basket, 16...Reader, 17...Patrol lamp, 21...Processor, 22...Main memory, 23...Auxiliary storage device, 24...Terminal program, 25...Learning data, 26...Wireless unit, 27...Touch panel, 28...Speaker, 29...System transmission path, 100...Item registration device.
Claims
1. At least one imaging device that captures an image of an item inserted into the imaging area; At least one illumination unit that illuminates the photographing area for a lighting period according to a set exposure amount; a position detection unit that divides the photographing area into a plurality of areas and detects the area into which the article is inserted under the illumination of the illumination unit; an information terminal that sets an exposure time under the illumination of the illumination unit for each of the imaging devices so that the exposure amount is set based on the brightness of the illumination corresponding to the distance between the detected area and the imaging devices, and recognizes and registers an item in the captured image; An article registration device comprising:
2. The information terminal includes: setting the lighting period based on the set exposure amount with the darkest area among the detected areas as a reference; The article registration device according to claim 1 , wherein an exposure start time of the imaging device is delayed from a lighting start time of the illumination unit so as to achieve the exposure time for each imaging device, and imaging is performed with the set amount of exposure.
3. The article registration device is mounted on a cart having a basket, The imaging device photographs the photographing area including the bottom surface of the basket into which the article is inserted from the upper opening of the basket, the position detection unit divides a bottom surface of the basket into the plurality of areas, and detects the area into which the article is inserted under illumination by the illumination unit. The article registration device according to claim 1.
4. The information terminal includes: A package recognition process for recognizing the item based on the package of the item included in the captured image; and and recognizing the article by any one of an identification code recognition process for analyzing an identification code of the article included in a captured image and recognizing the article. The article registration device according to claim 1.
5. A plurality of carts each equipped with the article registration device according to claim 1 and functioning as a POS (Point Of Sales) terminal; an accounting machine for processing the payment for the registered items; A cart POS system comprising:
6. A program for causing a processor to execute processing by each unit of the device according to any one of claims 1 to 5.
Citation Information
Patent Citations
Imaging device, imaging control method and program
JP2016058995A
Information processing system and information processing method
JP6537054B2
