Commodity registration device and storage medium

Abstract

The invention discloses a commodity registration device and a storage medium, which can properly carry out sorting in a shopping basket. The commodity registration device includes an acquisition unit that acquires a captured image from an image capturing unit that captures an image of a storage unit that stores a commodity, and a designation unit that designates a commodity to be put into the storage unit or a commodity to be taken out from the storage unit on the basis of the captured image. A registration unit that performs a process of registering the designated product or canceling the registered product on the basis of the result of the designation by the designation unit; and a control unit that performs control so as to invalidate the process performed by the registration unit on the condition that an operator has performed a predetermined operation.

Description

[0001] This application claims priority from Japanese patent application No. JP2023-177401, filed on October 13, 2023, and cites the contents of the above application, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] Embodiments of the present invention relate to a commodity registration device and a storage medium. Background Art

[0003] With the decrease in the working population, self-service checkout methods such as fully self-service POS and semi-self-service POS have been introduced in supermarkets and other stores as a measure to save manpower. The self-service checkout method is a method in which customers register the goods they want to purchase (hereinafter also referred to as product registration, sales registration) and settle the purchase of goods.

[0004] In addition, as a new form, a store mobile checkout method has been introduced in which customers can register products by themselves while moving in the store. For example, customers can register products by themselves by using a camera or a code reader of a terminal device installed on a shopping cart (hereinafter also referred to as a shopping cart) to read a barcode or other code symbol attached to the product while moving in the store. In addition, recently, a technology has been proposed to automatically register and cancel the registration of products by recognizing the category of products put into (stored in) or taken out of the shopping basket of the shopping cart from an image captured by a camera installed on the shopping cart.

[0005] By the way, during the shopping process, customers can move the goods stored in the shopping basket, or reorganize the goods in the shopping basket (hereinafter also referred to as "reorganizing in the shopping basket"). For example, customers can reorganize in the shopping basket by placing heavy goods at the bottom of the shopping basket, or placing goods such as eggs or bread that lose their value if deformed at the top of the shopping basket.

[0006] However, in the above-mentioned structure for registering or canceling a product through image recognition, when the camera captures a product that is moved during the sorting process in the shopping cart, the product registration or registration cancellation may be performed unintentionally. Summary of the invention

[0007] In view of the above problems, the present invention aims to provide a product registration device and a storage medium that can properly organize products in a shopping cart.

[0008] To solve the above-mentioned problem, the commodity registration device of the embodiment includes: an acquisition unit for acquiring a camera image from a camera unit that photographs a storage unit that stores commodities, a designation unit for designating commodities to be put into the storage unit or commodities to be taken out of the storage unit based on the camera image, a registration unit for registering designated commodities or canceling the processing of registered commodities based on the designation result of the designation unit, and a control unit for performing control based on the condition that an operator has performed a prescribed action so as to invalidate the processing performed by the registration unit.

[0009] According to the above-mentioned commodity registration device, it is possible to prevent commodity registration and registration cancellation from being performed unintentionally when arranging commodities in the storage section.

[0010] The above-mentioned commodity registration device further includes: a control for indicating that the processing performed by the registration unit is invalid, wherein the control unit controls to invalidate the processing performed by the registration unit based on the action of operating the control.

[0011] According to the above-mentioned commodity registration device, the processing performed by the registration unit can be invalidated by operating the control.

[0012] In the above-mentioned commodity registration device, the control unit performs control so as to invalidate the processing performed by the registration unit on the condition that it is detected that the operator has performed a predetermined action in the storage unit based on the captured image.

[0013] According to the above-mentioned commodity registration device, the processing performed by the registration section can be invalidated according to the action of the operator in the storage section.

[0014] In the above-mentioned commodity registration device, the control unit performs control so as to invalidate the processing performed by the registration unit on the condition that the operator performs an action of putting both hands into the storage unit.

[0015] According to the above-mentioned commodity registration device, the processing performed by the registration section can be invalidated by the operator's action of putting both hands into the storage section.

[0016] In the above-mentioned commodity registration device, the control unit performs control so as to invalidate the processing performed by the registration unit on the condition that the operator performs an action of moving the commodity in the storage unit in the horizontal direction.

[0017] According to the above-mentioned commodity registration device, the processing performed by the registration section can be invalidated according to the operator's action of moving the commodity in the horizontal direction within the storage section.

[0018] In the above-mentioned commodity registration device, the control unit causes the display unit to display a message notifying that the registration unit is in an invalid state while the processing performed by the registration unit is invalidated.

[0019] According to the above-mentioned commodity registration device, it is possible to notify the operator that the automatic registration is invalid.

[0020] In the above-mentioned commodity registration device, the control unit controls to enable the processing by the registration unit on condition that the control is operated in a state where the processing by the registration unit is disabled.

[0021] According to the above-mentioned commodity registration device, the processing performed by the registration unit can be invalidated by a simple operation.

[0022] In the above-mentioned commodity registration device, the control unit confirms the number of commodities registered so far before invalidating the processing performed by the registration unit.

[0023] According to the above-mentioned commodity registration device, the processing performed by the registration unit can be invalidated according to the commodity registration number.

[0024] In the above-mentioned commodity registration device, the control unit performs control so as not to invalidate the processing performed by the registration unit when the commodity quantity is below a prescribed value, and performs control so as to invalidate the processing performed by the registration unit when the commodity quantity is greater than the prescribed value.

[0025] According to the above-mentioned commodity registration device, it is possible to prevent the processing performed by the registration unit from being invalidated in order to conduct an unfair act.

[0026] A storage medium according to another aspect of the present invention stores a program for causing a computer of a commodity registration device to function as: an acquisition unit for acquiring a camera image from a camera unit for photographing a storage unit storing commodities; a designation unit for designating commodities to be put into or taken out of the storage unit based on the camera image; a registration unit for registering designated commodities or canceling registered commodities based on a designation result of the designation unit; and a control unit for performing control on the condition that an operator has performed a prescribed action so as to invalidate the processing performed by the registration unit.

[0027] According to the above-mentioned storage medium, it is possible to provide a computer with a function capable of preventing unintentional registration and cancellation of commodities when arranging commodities in a storage section. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a diagram showing an example of a checkout system according to the embodiment.

[0029] Figure 2 It is a side view showing an example of the structure of the shopping cart involved in the embodiment.

[0030] Figure 3 This is a diagram showing an example of the hardware configuration of a terminal device according to the embodiment.

[0031] Figure 4 This is a diagram showing an example of the data structure of the product master file involved in the embodiment.

[0032] Figure 5 This is a diagram showing an example of the data structure of the sales registration table involved in the embodiment.

[0033] Figure 6 This is a diagram showing an example of the functional structure of a terminal device involved in the embodiment.

[0034] Figure 7 This is a flowchart showing an example of sales registration processing performed by the terminal device of the embodiment.

[0035] Description of Reference Numerals

[0036] 1 Checkout system

[0037] 2 Terminal Devices

[0038] 3. Trolley (shopping cart)

[0039] 4 Server

[0040] 8 Checkout Devices

[0041] 200 Control Department

[0042] 201 Image acquisition unit

[0043] 202 Object Area Detection Unit

[0044] 203 Code Reading Unit

[0045] 204 Packaging Identification Department

[0046] 205 Registration Processing Department

[0047] 206 Registration Control Department

[0048] 207 Settlement Processing Department DETAILED DESCRIPTION

[0049] The following is a description of the commodity registration device and storage medium involved in this embodiment with reference to the accompanying drawings. In the following embodiment, a terminal device installed on a shopping cart (hereinafter also referred to as a trolley) used in a store for transporting commodities is described as an example of a commodity registration device. In addition, this application example does not limit the invention.

[0050] Figure 1 1 is a diagram showing an example of a checkout system 1 according to the embodiment. Figure 1As shown, the checkout system 1 includes a terminal device 2, a server 4, and a checkout device 8. The server 4 and the checkout device 8 are connected to each other via a network Na so as to be communicable.

[0051] The network Na can use the Internet, VPN (virtual private network), LAN (local area network), public communication network, mobile communication network, etc. alone or in appropriate combination. As an example, the Internet and mobile communication network are used in combination as the network Na.

[0052] The wireless access point 7 is connected to the network Na. The wireless access point 7 is connected to the terminal device 2 via a network Nb such as a wireless LAN. Therefore, the terminal device 2 can communicate with the server 4 and the checkout device 8.

[0053] The shopping cart 3 includes a storage portion 31 for storing commodities displayed in the store (see Figure 2 The upper part of the storage part 31 is open, and commodities are put into the storage part 31 from the upper part. Consumers (hereinafter also referred to as customers) who use the store borrow the cart 3 in the store, and store the commodities to be purchased from the displayed commodities in the storage part 31 while pushing the cart 3 to move in the store.

[0054] The terminal device 2 is a terminal device such as a tablet terminal operated by a customer. In the present embodiment, the terminal device 2 is mounted on the cart 3 and operated by a customer using the cart 3.

[0055] The terminal device 2 recognizes (hereinafter also referred to as specifying) the category of the commodity put into the storage unit 31 based on the image captured by the camera unit 34 described later, and performs sales registration processing related to the commodity. Here, the sales registration processing is also referred to as commodity registration, sales registration, etc., and refers to the processing of treating the commodity specified by the terminal device 2 as a purchase object. Specifically, the sales registration processing is the processing of registering information related to the specified commodity in the sales registration table 243 described later. Hereinafter, the information related to the commodity registered in the sales registration table 243 is also referred to as "sales registration data".

[0056] In addition, when the terminal device 2 receives an operation for performing a checkout (hereinafter also referred to as a settlement) of a commodity, it executes a process of transmitting the sales registration data related to a transaction registered in the sales registration process to the checkout device 8. For example, the terminal device 2 encodes the sales registration data into a code symbol such as a QR code and displays it. In this case, by reading the code symbol displayed on the terminal device 2 with the checkout device 8, the sales registration data related to a transaction that has been sold and registered with the terminal device 2 is obtained, and the checkout process is performed based on the sales registration data.

[0057] The server 4 is an example of a server device. The server 4 manages information (data) related to sales transactions of goods in a store using the terminal device 2. For example, the server 4 provides the terminal device 2 with a product master file described later. In addition, the server 4 can be a store server set up in the store or a server device set up outside the store. In addition, in this embodiment, it is assumed that the server 4 is implemented by a single device, but it can be a cloud server (or cloud system) implemented by multiple devices connected to the network.

[0058] The checkout device 8 is a checkout device installed in the store. The checkout device 8 performs the settlement processing of the goods based on the sales registration data. The settlement processing includes the processing of displaying the total amount and tax amount of the goods involved in a transaction, calculating and displaying the change based on the deposit deposited by the customer, and issuing a receipt printed with the product information and settlement information (total amount, deposit amount, change amount, etc.). In addition, in the present embodiment, the checkout device 8 is described as a self-service checkout device, which is a checkout device that allows customers who use the terminal device 2 to register sales to perform settlement processing through their own operations, but is not limited to this, and can be a checkout device operated by staff such as a clerk.

[0059] Next, the cart 3 will be described. Figure 2 1 is a side view showing an example of the structure of the cart 3. The cart 3 includes a basket-shaped storage portion 31 with an open top, rotatable casters 32 disposed below the storage portion 31, and a handle 33 connected to the storage portion 31 and the casters 32 and held by the customer when moving the cart 3.

[0060] The side and bottom surfaces of the storage section 31 are surrounded by a substantially rectangular mesh grid 311 and are open at the top, so that the products stored in the storage section 31 can be observed from the outside. In addition, the side and bottom surfaces may be formed by wall surfaces instead of the mesh grid 311.

[0061] In addition, the cart 3 includes a terminal device 2 near the handle portion 33. In addition, the cart 3 includes a camera unit 34 near the edge of the opening portion of the storage portion 31. The camera unit 34 is implemented by a camera device such as a digital camera. The camera unit 34 is set in a manner such that the camera direction faces the opening of the storage portion 31. In other words, the camera direction and camera range of the camera unit 34 are set to be able to capture the goods entering and leaving the storage portion 31. In addition, the camera unit 34 can also be a structure included in the terminal device 2.

[0062] The imaging unit 34 captures an image of commodities stored in the storage unit 31 or commodities taken out from the storage unit 31, and outputs the captured image to the terminal device 2. Here, the image is a still image or a moving image in units of frames.

[0063] In addition, the structure of the cart 3 is not limited to Figure 2 For example, the number of camera units 34 is not limited to one, and multiple cameras 34 may be provided. In addition, the terminal device 2 may be configured to be removable from the cart 3. In this case, the terminal device 2 may be a portable terminal (smartphone, tablet terminal, etc.) held by the customer or loaned by the store and installed on the cart 3.

[0064] Next, a configuration example of the main devices of the terminal device 2 is described. Figure 3 This is a diagram showing an example of the hardware configuration of the terminal device 2.

[0065] like Figure 3 As shown, the terminal device 2 includes a CPU (Central Processing Unit) 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a storage unit 24, a display unit 25, an operation unit 26, a communication unit 27, a connection unit 28, and the like.

[0066] The CPU 21 is an example of a processor, and generally controls the operation of the terminal device 2. The ROM 22 stores various programs. The RAM 23 is a work area for loading programs and various data.

[0067] The CPU 21, ROM 22, and RAM 23 are connected via a bus or the like, and constitute a computer-structured control unit 200. In the control unit 200, the CPU 21 operates using a program stored in the ROM 22 or the storage unit 24 and loaded into the RAM 23, thereby executing various processes.

[0068] The storage unit 24 is a storage medium including a HDD (Hard Disk Drive), a flash memory, etc., and retains the stored contents even when the power is turned off. The storage unit 24 stores programs executable by the CPU 21 and various setting information. For example, an application program for cooperating with the checkout system 1 is pre-stored (imported) in the storage unit 24.

[0069] The storage unit 24 also stores a commodity master file 241 and commodity identification data 242. The commodity master file 241 is a data table for storing and managing commodity information of commodities sold in a store. Figure 4 241 is a diagram showing an example of the data structure of the product master file 241. Figure 4 As shown, the product master file 241 stores product codes and product information in association with each other.

[0070] The product code is an example of identification information that can identify the type of products sold in a store. Product information is information about the product corresponding to the product code. Product information includes, for example, product name, price, etc.

[0071] The commodity identification data 242 is a data set used to identify and specify the category of the corresponding commodity based on the external characteristics of the commodity. Here, the external characteristics of the commodity include, for example, the design, pattern, color, texture, shape, etc. of the commodity or the external packaging. Hereinafter, the process of specifying a commodity code that can identify the category of the commodity based on the external characteristics of the commodity is also referred to as "package identification processing."

[0072] The commodity identification data 242 includes, for example, a feature extraction model 2421 and a comparison dictionary 2422 .

[0073] The feature extraction model 2421 is a learning model for extracting feature quantities (e.g., feature vectors) representing the external features of a product. When an image is input, the feature extraction model 2421 extracts and outputs feature quantities representing the external features of the item (product) shown in the image. The extraction of feature quantities can use known techniques such as CNN (Convolutional Neural Network).

[0074] The comparison dictionary 2422 is dictionary data for specifying the category of the commodity corresponding to the feature quantity based on the feature quantity of the commodity extracted from the image. For example, the comparison dictionary 2422 associates and stores the commodity code corresponding to the category of each commodity and the feature quantity representing the external feature of the commodity (hereinafter also referred to as the comparison feature quantity) for a plurality of commodities sold in a store.

[0075] In the package recognition process, the feature quantity of the commodity extracted from the image using the feature extraction model 2421 is compared with the comparison feature quantity of each commodity registered in the comparison dictionary 2422, thereby calculating the similarity (also called distance, possibility) between the feature quantities. Then, in the package recognition process, the commodity code of the commodity registered in the comparison dictionary 2422 whose feature quantity similarity exceeds a threshold value (for example, 90%) is designated as the category of the commodity being the photographed object.

[0076] Furthermore, the storage unit 24 stores a sales registration table 243. The sales registration table 243 is a data table for storing and managing commodities registered for sale on the terminal device 2 of the user. Figure 5 This is a diagram showing an example of the data structure of the sales registration table 243.

[0077] like Figure 5As shown, the sales registration table 243 stores the product code, product information, and the number of items in association with each other. Here, in the data structure of the sales registration table 243, for example, a set of product code, product name, product information, and the number of items corresponds to the sales registration data.

[0078] The product code is a product code that can identify the product that has been registered for sale, and corresponds to the product code registered in the product master file 241. The product information stores the product information (product name, price, etc.) of the product master file 241 corresponding to the product code of the product that has been registered for sale. The number of pieces refers to the number of products of the same category (same product code) that have been registered for sale.

[0079] In addition, the data structure of the sales registration table 243 is not limited to Figure 5 For example, the sales registration table 243 may be a data structure that does not store product information.

[0080] Return to Figure 3 The display unit 25 is a display device such as an LCD (Liquid Crystal Display). The display unit 25 displays various information under the control of the CPU 21. The operation unit 26 is an input device such as a keyboard and a pointing device.

[0081] The operation unit 26 outputs the operation content inputted via the input device to the CPU 21. For example, the operation unit 26 has a switch button 261 capable of switching the function related to the above-mentioned package identification process to be invalid. In addition, the operation unit 26 may be a touch panel provided on the display unit 25. In the case where the operation unit 26 is a touch panel structure, the switch button 261 may be, for example, a control displayed on the screen.

[0082] The communication unit 27 is a wired or wireless communication interface that can connect to the network Nb. The communication unit 27 is communicably connected to the wireless access point 7 via the network Nb. The control unit 200 communicates with external devices such as the server 4 and the checkout device 8 via the communication unit 27.

[0083] The connection unit 28 is an interface for connecting peripheral devices. The connection unit 28 is, for example, a wired or wireless interface that complies with a standard for short-range wireless communication such as USB (Universal Serial Bus) or Bluetooth (registered trademark). In this embodiment, the connection unit 28 is connected to the camera unit 34.

[0084] Next, the functional structure of the terminal device 2 will be described. Figure 6 This is a diagram showing an example of the functional structure of the terminal device 2.

[0085] like Figure 6As shown, the terminal device 2 includes an image acquisition unit 201, an object area detection unit 202, a code reading unit 203, a package identification unit 204, a registration processing unit 205, a registration control unit 206, and a settlement processing unit 207 as a functional structure.

[0086] The functional structure included in the terminal device 2 is a software structure implemented by the processor of the terminal device 2 acting according to the program stored in the ROM or storage unit, but is not limited to this. For example, part or all of the functional structure included in the terminal device 2 may also be implemented in the form of hardware such as a dedicated circuit.

[0087] The image acquisition unit 201 is an example of an acquisition unit. The image acquisition unit 201 acquires an image (hereinafter also referred to as a captured image) captured by the camera unit 34 via the connection unit 28. The captured image acquired by the image acquisition unit 201 is temporarily stored in the RAM 23 for processing in the code reading unit 203 and the registration processing unit 205.

[0088] The object region detection unit 202 detects a region where an object is shown from the camera image. Specifically, the object region detection unit 202 detects a region where a commodity is shown (hereinafter also referred to as a commodity region) from the image acquired by the image acquisition unit 201, and outputs the commodity region or information indicating the commodity region. In addition, the method of detecting the commodity region is not particularly limited, and a known object detection technology such as SSD (Single Shot multi box Detector) can be used.

[0089] Furthermore, the object region detection unit 202 detects the position and movement (for example, motion vector) of the product region from each image that is successively acquired in time sequence by the image acquisition unit 201 , thereby detecting the moving direction of the product.

[0090] For example, when a commodity is put into the storage section 31, the object region detection unit 202 detects that the commodity moves downward (hereinafter also referred to as the putting direction). In addition, when the commodity is taken out from the storage section 31, the object region detection unit 202 detects that the commodity moves upward (hereinafter also referred to as the taking-out direction).

[0091] Furthermore, the object region detection unit 202 detects a region showing a hand (hereinafter also referred to as a hand region) from the image acquired by the image acquisition unit 201 by using a known object detection technique, and outputs the hand region or information indicating the hand region. Furthermore, the object region detection unit 202 detects the position and movement of the hand region from the images acquired continuously in a time series by the image acquisition unit 201, thereby determining the direction of the hand movement.

[0092] The code reading unit 203 reads information from a code symbol such as a barcode or a two-dimensional code included in the captured image. Specifically, the code reading unit 203 performs a code reading process for reading the code symbol for the commodity area detected by the object area detection unit 202 .

[0093] In the code reading process, the code reading unit 203 extracts an image representing a code symbol from the commodity area and decodes the extracted code symbol to obtain information stored in the code symbol. For example, the code reading unit 203 obtains the commodity code of the photographed commodity by reading the code symbol attached to the commodity.

[0094] The package recognition unit 204 is an example of a designation unit. The package recognition unit 204 performs package recognition processing on the captured image using the commodity recognition data 242 to recognize the type of the commodity shown in the captured image.

[0095] Specifically, the package recognition unit 204 extracts (obtains) the feature quantity of the commodity shown in the commodity area from the commodity area by performing package recognition processing on the commodity area detected by the object area detection unit 202. Next, the package recognition unit 204 compares the extracted feature quantity with the comparison feature quantity of each commodity registered in the comparison dictionary 2422, thereby obtaining the commodity code of the commodity whose similarity exceeds the threshold from the comparison dictionary 2422 and outputs it as the recognition result.

[0096] The registration processing unit 205 is an example of a registration unit and performs sales registration processing or registration cancellation processing of the product based on the moving direction of the product determined by the object region detection unit 202 and the product code read by the code reading unit 203 or the product code recognized by the package recognition unit 204 .

[0097] When the commodity code is obtained by the code reading unit 203 or the package recognition unit 204 and the movement in the insertion direction is detected by the object region detection unit 202, the registration processing unit 205 performs a sales registration process of registering the commodity corresponding to the commodity code in the sales registration table 243. Specifically, the registration processing unit 205 reads the commodity information corresponding to the commodity code obtained by the code reading unit 203 or the package recognition unit 204 from the commodity master file 241, associates the commodity code with the number of pieces (initial value is 1) and registers it in the sales registration table 243. In addition, when the commodity code registered in the sales registration table 243 is obtained by the code reading unit 203 or the package recognition unit 204 and the movement in the insertion direction is detected by the object region detection unit 202, the registration processing unit 205 adds 1 to the number of pieces of the corresponding sales registration data.

[0098] In addition, when the commodity code is obtained by the code reading unit 203 or the package identification unit 204 and the movement in the insertion direction is detected by the object area detection unit 202, the registration processing unit 205 performs processing for canceling the sales registration of the commodity corresponding to the commodity code. Specifically, the registration processing unit 205 retrieves the sales registration data corresponding to the obtained commodity code from the sales registration table 243, and subtracts 1 from the number of sales registration data retrieved. In addition, when the number after subtraction is 0, the registration processing unit 205 deletes the sales registration data from the sales registration table 243.

[0099] Furthermore, each time the registration processing unit 205 updates the sales registration table 243, it causes the display unit 25 to display a screen based on the sales registration data registered in the sales registration table 243. For example, the registration processing unit 205 causes the display unit 25 to display a screen showing a list of sales-registered commodities.

[0100] Thus, in the checkout system 1 of the present embodiment, when a customer puts a commodity into the storage section 31 of the shopping cart 3, the sales registration of the commodity is automatically performed. In addition, in the checkout system 1, when the customer takes out a commodity from the storage section 31 of the shopping cart 3, the registration of the commodity is automatically cancelled. That is, the checkout system 1 is a store mobile type checkout system in which customers can register commodities by themselves while moving in the store.

[0101] Incidentally, during shopping, customers can perform operations such as moving the goods stored in the storage section 31 to reorganize the goods in the storage section 31 (hereinafter also referred to as shopping basket arrangement). For example, customers can perform shopping basket arrangement by placing heavy goods at the bottom of the storage section 31 or placing goods such as eggs and bread that lose their value if deformed at the top of the storage section 31.

[0102] However, in the configuration of registering or canceling the product by package recognition, it is possible to mistakenly recognize the product that was moved during the arrangement in the shopping basket. For example, when the product is taken out or put into the storage unit 31 due to the arrangement in the shopping basket, the customer may unintentionally register or cancel the product.

[0103] Therefore, the terminal device 2 of this embodiment includes the registration control unit 206, and thus performs control so as not to unintentionally register or cancel the sale of a product when a customer arranges the items in the shopping cart.

[0104] The registration control unit 206 is an example of a control unit. The registration control unit 206 controls to invalidate the function of sales registration / registration cancellation (hereinafter also referred to as automatic registration function) by the registration processing unit 205, on the condition that the customer who is the operator of the terminal device 2 performs a prescribed action. Specifically, the registration control unit 206 controls to invalidate the automatic registration function when an operation (or action) indicating the start of arrangement in the shopping cart is detected.

[0105] For example, the registration control unit 206 controls to disable the automatic registration function when detecting the operation of the switching button 261. Here, the control to disable the automatic registration function may be a control to stop or disable the functions (actions) of the code reading unit 203 and the package identification unit 204, or a control to stop or disable the functions (actions) of the registration processing unit 205.

[0106] When the customer operating the terminal device 2 arranges the commodities stored in the storage section 31, the customer can arrange the commodities in the shopping cart while disabling the automatic registration function by operating the switching button 261. Thus, even when arranging the commodities in the shopping cart, the terminal device 2 can prevent the customer from unintentionally registering or canceling the registration of commodities, thereby realizing an environment in which the commodities in the shopping cart can be properly arranged.

[0107] Here, while the automatic registration function is disabled, the registration control unit 206 preferably causes the display unit 25 to display a message notifying that the automatic registration function is disabled. That is, while the processing by the registration unit is disabled, the display unit displays a message notifying that the registration unit is disabled.

[0108] Thus, the customer can easily check whether the operation of the switch button 261 is reflected by viewing the screen of the display unit 25. In addition, the customer can easily check whether the automatic registration function is in an effective state. Therefore, the terminal device 2 can improve the convenience of the customer when arranging in the shopping cart.

[0109] In addition, when the registration control unit 206 receives the operation of the switch button 261 again in the state where the automatic registration function is invalid, it determines that the end of the arrangement in the shopping cart is detected, and the automatic registration function is enabled, thereby enabling the sale registration or registration cancellation of the product. The switch button 261 functions as a control that can switch the automatic registration function between enabling and disabling. That is, in the state where the processing performed by the registration unit is invalid, the processing performed by the registration unit is enabled under the condition of the action of operating the control. In addition, the control that indicates the end of the arrangement in the shopping cart is not limited to the switch button 261, and it can be a structure in which a control for enabling the automatic registration function is separately provided.

[0110] In addition, the registration control unit 206 is not limited to the case where a clear instruction action is performed via the switch button 261, and the automatic registration function can be disabled when an action indicating the start of arrangement in the shopping cart is detected from the camera image. Specifically, the registration control unit 206 cooperates with the object area detection unit 202 to control the automatic registration function to be disabled when an action indicating the start of arrangement in the shopping cart is detected from the camera image.

[0111] For example, the registration control unit 206 determines that the start of the arrangement in the shopping basket has been detected, based on the condition that two hand regions are detected, that is, both hands are detected entering the storage unit 31, when the commodity region is not detected by the object region detection unit 202. Since the arrangement in the shopping basket is usually performed with both hands, the start of the arrangement in the shopping basket can be detected by detecting that both hands have entered the storage unit 31.

[0112] Furthermore, as another example, the registration control unit 206 determines that the start of the arrangement in the shopping basket has been detected when the moving direction of the commodity area detected by the object area detection unit 202 has a horizontal vector component and the amount of movement in the horizontal direction exceeds a threshold value. Generally, when the arrangement in the shopping basket is performed, the commodities stored in the storage unit 31 are moved to another position in the storage unit 31. Therefore, by detecting the movement of the commodities in the horizontal direction, it is possible to detect that the arrangement in the shopping basket has started. In addition, the threshold value of the movement amount is preferably determined based on the amount of movement of the commodities that are usually moved during the actual arrangement in the shopping basket.

[0113] Furthermore, as another example, the registration control unit 206 determines that the start of the arrangement in the shopping basket has been detected when the amount of movement in any direction toward the commodity area detected by the object area detection unit 202 is continuously less than a threshold value (small) for a predetermined time, that is, when the commodity is in a substantially stationary state near the opening of the storage unit 31. During the arrangement in the shopping basket, when the commodity is moved to another position, the commodity is held in a state of being held, so by detecting that the commodity is in a substantially stationary state, it is possible to detect that the arrangement in the shopping basket has started. In addition, the thresholds for the amount of movement and the duration are preferably determined based on the amount of movement and the duration when the commodity is held and held during the actual arrangement in the shopping basket.

[0114] Thus, even when the terminal device 2 performs shopping cart arrangement without a clear instruction through the switch button 261, it is possible to prevent the customer from unintentionally registering or canceling the registration of the product, and to realize an environment in which the shopping cart arrangement can be properly performed. In addition, while the automatic registration function is disabled, the registration control unit 206 preferably causes the display unit 25 to display a message notifying that the automatic registration function is in an invalid state. That is, while the processing performed by the registration unit is disabled, the display unit is caused to display a message notifying that the registration unit is in an invalid state.

[0115] In addition, when the registration control unit 206 detects the completion of the arrangement in the shopping cart from the camera image, it enables the automatic registration function, thereby enabling the sale registration or registration cancellation of the product. Here, for example, the detection of the completion of the arrangement in the shopping cart can be conditioned on the customer's hand (both hands) being taken out of the storage unit 31. In addition, for example, it can be conditioned on the product (product area) no longer being detected.

[0116] The settlement processing unit 207 performs settlement processing for the sales registered commodity according to the customer's operation. Specifically, the settlement processing unit 207 performs processing for transmitting the sales registration data of the sales registered commodity to the checkout device 8, thereby enabling settlement using the checkout device 8.

[0117] For example, when the settlement processing unit 207 receives an operation indicating the end of sales registration or an operation indicating the start of settlement (hereinafter collectively referred to as a total operation) via the operation unit 26, the sales registration data registered in the sales registration table 243 is sent to the checkout device 8 via short-range wireless communication such as Bluetooth (registered trademark).

[0118] In addition, for example, the settlement processing unit 207 may also transmit the sales registration data registered in the sales registration table 243 to the checkout device 8 via the server 4. In this case, the settlement processing unit 207 transmits the sales registration data registered in the sales registration table 243 to the server 4 together with the identification ID that can identify the own terminal device 2 or the identification ID that can identify the transaction. In addition, the settlement processing unit 207 transmits the identification ID to the checkout device 8 at the checkout destination through short-range wireless communication. Then, by using the checkout device 8 to inquire the server 4 about the identification ID sent from the settlement processing unit 207, the sales registration data corresponding to the identification ID is obtained.

[0119] The method of notifying the checkout device 8 of the identification ID is not limited to short-range wireless communication. For example, the settlement processing unit 207 may encode the identification ID into a code symbol such as a QR code and display it on the display unit 25, thereby notifying the checkout device 8 of the identification ID via the code symbol. In this case, the checkout device 8 can read the code symbol displayed on the display unit 25 to obtain the identification ID from the terminal device 2.

[0120] In addition, for example, the settlement processing unit 207 may transmit the identification ID of its own terminal device 2 to the checkout device 8 via short-range wireless communication such as Bluetooth (registered trademark). In this case, the checkout device 8 obtains the sales registration data corresponding to the identification ID transmitted from the terminal device 2 from the server 4, and performs the checkout process of the commodity involved in the transaction registered for sale based on the obtained sales registration data.

[0121] In addition, the settlement of goods is not limited to the method using the checkout device 8, and can also be performed using the terminal device 2. For example, when the settlement processing unit 207 receives an operation indicating the start of settlement via the operating unit 26, it causes the display unit 25 to display a settlement method that can select electronic money settlement, credit card settlement, etc. When the settlement method is selected, the settlement processing unit 207 receives the input of settlement information for settlement with the selected settlement method, and cooperates with an external settlement server to perform settlement processing for the total amount of the sales and registration of the goods based on the input settlement information. The relevant settlement information can be, for example, a credit card number or a two-dimensional code for code settlement. In addition, the settlement processing unit 207 receives the input of a two-dimensional code for code settlement via the camera unit 34.

[0122] Next, an example of sales registration processing performed by the terminal device 2 will be described. Figure 7 This is a flowchart showing an example of sales registration processing performed by the terminal device 2.

[0123] First, the image acquisition unit 201 of the terminal device 2 sequentially acquires the images captured by the camera unit 34 (step S11). Next, the object region detection unit 202 determines whether the product region or the hand region is detected from the image captured in step S11 (step S12). Here, if neither the product region nor the hand region is detected (step S12; No), the process moves to step S25. In addition, if the product region or the hand region can be detected (step S12; Yes), the object region detection unit 202 detects the moving direction of the detected region from the images acquired continuously in time sequence (step S13).

[0124] Next, the registration control unit 206 determines whether the start of the arrangement in the shopping basket is detected based on the operation state of the switching button 261 or the processing results of steps S12 and S13 (step S14). In the case where the start of the arrangement in the shopping basket is detected (step S14; yes), the registration control unit 206 sets a flag value indicating that the automatic registration function is invalid in the control flag indicating the validity / invalidity of the automatic registration function (step S15), and transitions to step S25. Here, it is assumed that the control flag is stored in, for example, RAM 23, etc., and is set to a flag value indicating that the automatic registration function is valid in the default state. In addition, the registration control unit 206 may cause the display unit 25 to display a message notifying that the automatic registration function is invalid, etc., along with the processing of step S15. In addition, when the start of the arrangement in the shopping basket is detected, the registration control unit 206 confirms the number of goods registered in the sales registration table 243 (the number of goods registered before the start of the arrangement in the shopping basket is detected), and when the number of goods is small (for example, less than two pieces), the automatic registration function may not be invalidated. When the number of goods is small, it is not necessary to arrange the shopping basket. The instruction to invalidate the automatic registration function may be an action intentionally performed due to an erroneous operation, an erroneous detection, or an attempt to perform an unfair act. In order to prevent erroneous operations, erroneous detections, and unfair acts, whether the start instruction of the arrangement in the shopping cart is appropriate is determined based on the number of commodities registered in the sales registration table 243. Specifically, the threshold 2 for invalidating the automatic registration function is stored in the RAM 23. When the start of the arrangement in the shopping cart is detected, the commodity codes in the sales registration table 243 are counted. The counted value is compared with the threshold value, and a flag value indicating that the automatic registration function is invalid is set when the counted value exceeds the threshold value. If the counted value is less than or equal to the threshold value, the automatic registration function is maintained (the flag value indicating invalidity is not set in the control flag) and the process is transitioned to step 25. Step 15 is executed. That is, when the operator performs a prescribed action, the control unit confirms the number of commodities registered before the processing performed by the registration unit is invalidated, and controls the processing performed by the registration unit when the number of registered commodities is less than or equal to the prescribed value. In addition, when the number of registered commodities is greater than the prescribed value, control is performed to invalidate the processing performed by the registration unit. The prescribed value is arbitrarily determined based on the actual operation of the store.

[0125] Furthermore, when the start of the arrangement in the shopping cart is not detected in step S14 (step S14; No), the registration control unit 206 determines whether the end of the arrangement in the shopping cart has been detected based on the operation via the switching button 261 or the processing results in steps S12 and S13 (step S16). When the end of the arrangement in the shopping cart is detected (step S16; Yes), the registration control unit 206 sets a flag value indicating that the automatic registration function is enabled in the control flag (step S17), and transitions to step S25. In addition, the registration control unit 206 may delete the message indicating that the automatic registration function is disabled, etc., which is displayed on the display unit 25, along with the processing of step S17.

[0126] Furthermore, when the completion of the arrangement in the shopping cart is not detected in step S16 (step S16; No), the registration control unit 206 determines whether the automatic registration function is in a valid state based on the flag value of the control flag (step S18). Here, when the automatic registration function is invalid (step S18; No), the registration control unit 206 transitions the process to step S25. Furthermore, when the automatic registration function is valid (step S18; Yes), the registration control unit 206 transitions the process to step S19.

[0127] In the next step S19, the code reading unit 203 performs a code reading process (step S19) of reading a code symbol (commodity code) from the commodity area detected by the object area detection unit 202. In addition, the package recognition unit 204 performs a package recognition process based on the commodity area detected by the object area detection unit 202 (step S20).

[0128] Subsequently, the registration processing unit 205 determines whether the commodity code has been acquired in the processing of any one of step S19 and step S20 (step S21). If the commodity code has not been acquired (step S21; No), the processing transitions to step S25. In addition, if the commodity code has been acquired (step S21; Yes), the registration processing unit 205 determines whether the moving direction of the commodity area determined in step S13 is the input direction or the extraction direction (step S22).

[0129] Here, when the moving direction is the insertion direction (step S22; yes), the registration processing unit 205 performs a sales registration process (step S23) of registering the product corresponding to the product code in the sales registration table 243 based on the product code obtained in step S19 or step S20. In addition, when the moving direction is the removal direction (step S22; no), the registration processing unit 205 performs a registration cancellation process of deleting the product corresponding to the product code obtained in step S19 or step S20 from the sales registration table 243 (step S24).

[0130] Subsequently, the registration processing unit 205 (or the settlement processing unit 207) determines whether a totaling operation has been accepted (step S25). Here, in the case where a totaling operation has not been accepted (step S25; No), the process returns to step S11. In addition, in the case where a totaling operation has been accepted (step S25; Yes), the settlement processing unit 207 performs a process for settling the goods registered in the sales registration table 243 (step S26).

[0131] According to this, the terminal device 2 can use either the commodity code read from the code symbol or the commodity code specified from the external characteristics of the commodity to perform sales registration and registration cancellation processing of the commodity. In addition, in the terminal device 2, the control flag is set to invalid as the customer arranges the shopping cart, and the automatic registration function is invalid while the control flag is set to invalid, so that the customer can properly arrange the shopping cart.

[0132] As described above, the terminal device 2 of this embodiment obtains a camera image from the camera unit 34 that captures the storage unit 31 storing the commodities, and specifies the commodities put into the storage unit 31 or the commodities taken out from the storage unit based on the obtained camera image. Then, the terminal device 2 registers the commodities or cancels the registered commodities based on the specified result through the automatic registration function. In addition, the terminal device 2 controls the automatic registration function to be invalid under the condition that the customer has performed a prescribed action.

[0133] Thus, the terminal device 2 can prevent the automatic registration function from registering the sale of the product or cancelling the registration even if the shopping cart has been sorted after the customer has performed a predetermined action. Therefore, the terminal device 2 can realize an environment in which the shopping cart can be sorted appropriately.

[0134] In addition, the above-mentioned embodiment can be appropriately modified and implemented by changing a part of the structure or function of each of the above-mentioned devices. Therefore, below, several modified examples involved in the above-mentioned embodiment are described as other embodiments. In addition, the following will mainly describe the points different from the above-mentioned embodiment, and will omit the detailed description of the points common to the content already described. In addition, the modified examples described below can be implemented individually or in appropriate combination.

[0135] (Variation 1)

[0136] In the above embodiment, the terminal device 2 is described as being mounted on the cart 3, but the terminal device 2 is not limited thereto. For example, the terminal device 2 may be a portable terminal such as a smartphone held by a customer, or a portable terminal such as a smartphone or a tablet terminal lent to a customer by a store.

[0137] In this case, the terminal device 2 may be configured to cooperate with the cart 3 by communicating with the camera unit 34 by wire or wirelessly, or may be configured to be independent of the cart 3. In the latter case, the customer can take pictures of the product using the camera of the terminal device 2, thereby being able to register the sale of the product in the same manner as in the above-mentioned embodiment. In addition, when canceling the registration, the customer can cancel the registration by selecting the product to be canceled from the list of the sales-registered products displayed on the screen of the terminal device 2, thereby being able to cancel the registration in the same manner as in the above-mentioned embodiment.

[0138] (Variation 2)

[0139] In the above-mentioned embodiment, the terminal device 2 performs the package identification process, but the present invention is not limited thereto and may be a structure in which the package identification process is performed by an external device for the package identification process.

[0140] In this case, the package recognition unit 204 of the terminal device 2 cooperates with the external device for package recognition processing to achieve the same function as the above-mentioned embodiment. Specifically, the package recognition unit 204 sends the image of the image area detected by the object area detection unit 202 to the external device for package recognition processing, so that the external device performs the package recognition processing, and obtains the processing result of the package recognition processing, that is, the recognition result of the commodity, from the external device.

[0141] According to this, in the checkout system 1 according to this variation, since the external device performs the package recognition process, the processing load on the terminal device 2 can be reduced.

[0142] (Variant 3)

[0143] In the above embodiment, the code symbol (product code) attached to the product is read from the image captured by the camera unit 34, but the present invention is not limited to this. For example, a code reading device dedicated to reading the code symbol may be separately provided.

[0144] The code reading device can be a type of code reading device that the customer holds in his hand and uses, or a code reading device that is arranged facing the customer. When using these code reading devices, the customer puts the product into the storage unit 31 after the code reading device reads the code symbol attached to the product.

[0145] In this case, the registration processing unit 205 preferably invalidates the recognition result of the package recognition unit 204 even if the same product is recognized by the package recognition unit 204 within a predetermined time (e.g., 5 seconds) after the product is read by the code reader. Thus, even if the product read by the code reader is recognized by the package recognition processing, duplicate registration can be prevented.

[0146] Furthermore, when there is no possibility that the code reader can read the commodities in the storage unit 31, the registration control unit 206 does not invalidate the function of the code reader (code reader 203) but keeps it valid even if it detects the start of sorting in the shopping cart.

[0147] Therefore, in the checkout system 1 according to this modification, even if a code symbol is read using a code reading device, the same effects as those of the above-described embodiment can be obtained.

[0148] The program executed by each device of the above-mentioned embodiment (and modified example) is provided in a state of being pre-installed in a ROM, a storage unit, etc. The program executed in each device of the above-mentioned embodiment can be configured to be stored in a computer-readable storage medium such as a CD-ROM, a floppy disk (FD), a CD-R, a DVD (Digital Versatile Disk) and provided in a file in an installable or executable form.

[0149] Furthermore, the programs executed by the devices of the above embodiments may be stored in a computer connected to a network such as the Internet and provided by downloading via the network. Furthermore, the programs executed by the devices of the above embodiments may be provided or distributed via a network such as the Internet.

[0150] So far, the embodiments of the present invention have been described, but the embodiments are presented only as examples and are not intended to limit the scope of the invention. These novel embodiments and their variations can be implemented in various other forms, and can be omitted, replaced, changed and combined in various ways without departing from the scope of the invention. These embodiments and their variations are included in the scope and spirit of the invention, and are included in the invention described in the claims and their equivalents.

Claims

1. A commodity registration device, characterized in that: include: an acquisition unit that acquires a camera image from a camera unit that captures a picture of a storage unit storing commodities; a designating unit that designates commodities put into the storage unit or commodities taken out from the storage unit based on the captured image; a registering unit that performs processing of registering the designated commodity or cancelling the registered commodity based on the designation result of the designating unit; as well as The control unit performs control so as to invalidate the processing performed by the registration unit on the condition that the operator has performed a predetermined action.

2. The commodity registration device according to claim 1, wherein: Also includes: A control for indicating that the processing to be performed by the registration unit is invalid, wherein, The control unit performs control so as to invalidate the processing performed by the registration unit on the condition that the control has been operated.

3. The commodity registration device according to claim 1 or 2, wherein: The control unit performs control so as to invalidate the processing performed by the registration unit on the condition that it is detected that the operator has performed a predetermined action in the storage unit based on the captured image.

4. The commodity registration device according to claim 3, wherein: The control unit performs control so as to invalidate the processing performed by the registration unit on the condition that the operator performs an action of putting both hands into the storage unit.

5. The commodity registration device according to claim 3, wherein: The control unit performs control so as to invalidate processing performed by the registration unit on the condition that the operator performs an action of moving the commodity in the storage unit in the horizontal direction.

6. The commodity registration device according to claim 3, wherein: The control unit causes the display unit to display a message notifying that the registration unit is in an invalid state while the processing performed by the registration unit is invalidated.

7. The commodity registration device according to claim 3, wherein: The control unit controls to enable the processing performed by the registration unit on condition that the control is operated in a state where the processing performed by the registration unit is disabled.

8. The commodity registration device according to claim 1, wherein: The control unit checks the number of commodities registered before invalidating the processing performed by the registration unit.

9. The commodity registration device according to claim 8, wherein: The control unit performs control so as not to invalidate the processing performed by the registration unit when the number of commodities is less than a predetermined value, and performs control so as to invalidate the processing performed by the registration unit when the number of commodities is greater than the predetermined value.

10. A storage medium storing a program for causing a computer of a commodity registration device to function as the following units: an acquisition unit that acquires a camera image from a camera unit that captures a picture of a storage unit storing commodities; a designating unit that designates commodities put into the storage unit or commodities taken out from the storage unit based on the captured image; a registering unit that performs processing of registering the designated commodity or cancelling the registered commodity based on the designation result of the designating unit; as well as The control unit performs control so as to invalidate the processing performed by the registration unit on the condition that the operator has performed a predetermined action.

Images