Processing apparatus, processing method, and program

Abstract

To provide a store system that eliminates settlement processing (article registration, payment, etc.) at a checkout counter.SOLUTION: A processing apparatus has: a recognition part which processes an image of a customer who takes an article out of an article mount space to recognize customer identification information on the customer; a registration part which links and registers the customer identification information recognized by the recognition part and article identification information on the article taken out of the article mount space by the customer; a settlement part which performs settlement based upon details of the registration by the registration part on detecting the customer who has taken the article out performing a predetermined operation through the image processing; and a completion notification part which transmits settlement completion notice to a terminal of the customer after the settlement by the settlement part. The predetermined operation is a movement of the article mount space out of a reference range determined by a standard by the customer. The settlement completion notice at least includes the article name and price of the object article of the settlement.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a processing apparatus, a processing method, and a program.

Background Art

[0002] Non-Patent Document 1, Patent Documents 1 and 2 disclose a store system that eliminates settlement processing (such as product registration and payment) at a cash register.

[0003] Specifically, in the store system disclosed in Non-Patent Document 1, a gate is installed at the entrance and exit of the store, and customers entering the store are recognized based on information read by a reader installed on the gate. Then, the store system recognizes the products taken by the customers through image processing or the like, and performs settlement at the timing when the customers pass through the gate and leave the store.

[0004] In the purchase product settlement support system disclosed in Patent Document 1, a gate is installed, the entry and exit of customers passing through the gate are managed, and after the customers pass through the gate and leave the store, settlement processing of the products taken out of the store by the customers is performed.

[0005] In the vehicle settlement system disclosed in Patent Document 2, a product placement section is provided in the vehicle interior, and when a predetermined condition is satisfied after the movement of the product to the outside of the vehicle is detected, settlement processing of the product is performed.

[0006] Non-Patent Document 2 discloses a store that displays products such as snacks and drinks in a corner of an office, installs settlement means such as a cashless settlement terminal and a cash collection box, and allows employees of the office to purchase products through self-service.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Patent Document 2

[0008] [Non-Patent Document 1] Takuya Miyata, "Amazon Go's Mechanism: A Cashier-less Supermarket Realized with 'Cameras and Microphones'", [online], December 10, 2016, [Retrieved December 6, 2019], Internet<URL:https: / / www.huffingtonpost.jp / tak-miyata / amazon-go_b_13521384.html> [Non-Patent Document 2] "12 Recommended Office Convenience Stores: System, Reputation, and Benefits of Implementation," [online], November 18, 2019, [searched December 6, 2019], Internet <URL:https: / / boxil.jp / mag / a5253 / ?utm_source=google&utm_medium=cpc&utm_campaign=office_c-z&gclid=EAIaIQobChMIzJbdxKOg5gIVT3RgCh26uAb8EAAYAiAAEgKK-vD_BwE> [Overview of the project] [Problems that the invention aims to solve]

[0009] By eliminating the need for payment processing at the checkout counter using the technologies disclosed in Non-Patent Document 1, Patent Document 1, and Patent Document 2, user convenience is improved, and store operations are reduced. However, the technologies disclosed in Non-Patent Document 1 and Patent Document 1 require the presence of a gate and cannot be used in environments where a gate cannot be installed. Furthermore, the technology disclosed in Patent Document 1 requires the presence of a vehicle and cannot be used in environments where a vehicle cannot be used. For example, in a store located in a corner of an office, as disclosed in Non-Patent Document 2, gates and vehicles may not be usable due to cost and space issues.

[0010] The objective of this invention is to realize a store system that eliminates payment processing at the cash register (product registration and payment, etc.) using a novel and unprecedented method. [Means for solving the problem]

[0011] According to the present invention, A registration means that recognizes product identification information of a product removed from a product storage space using image processing, and registers the recognized product identification information in association with customer identification information, When image processing detects that a customer who has taken out the product has performed a predetermined action, a payment means performs payment based on the registered information from the registration means, A notification means that prompts confirmation of payment based on the registration details obtained by the aforementioned registration means, It has, The notification means includes a processing device that, when the elapsed time since the product identification information was linked to the customer identification information exceeds a threshold, sends a notification to the customer's terminal device prompting confirmation of payment based on the registered information.

[0012] Furthermore, according to the present invention, Computers A registration step involves using image processing to recognize product identification information of a product removed from a product storage space, and registering the recognized product identification information in association with customer identification information. When image processing detects that the customer who took out the product has performed a predetermined action, a payment step is performed to make a payment based on the registered information from the registration step. A notification step prompting confirmation of payment based on the registration details obtained in the aforementioned registration step, Execute, The notification step provides a processing method for sending a notification to the customer's terminal device prompting confirmation of payment based on the registered information if the elapsed time since the product identification information was linked to the customer identification information exceeds a threshold.

[0013] Furthermore, according to the present invention, Computers, Registration means for recognizing product identification information of a product taken out from a product placement space by image processing and registering the recognized product identification information linked to customer identification information. When it is detected that a customer who has taken out the product by image processing has performed a predetermined operation, settlement means for performing settlement based on the registration content by the registration means. Notification means for prompting confirmation of settlement based on the registration content by the registration means. Function as When the elapsed time since the product identification information was linked to the customer identification information exceeds a threshold value, a program is provided that transmits a notification prompting confirmation of settlement based on the registration content to the customer's terminal device.

Effect of the Invention

[0014] According to the present invention, a store system that eliminates settlement processing (product registration, payment, etc.) at a cash register is realized by a novel method that has never existed before.

Brief Description of the Drawings

[0015] The above-described object, as well as other objects, features, and advantages, will become even more apparent from the following preferred embodiments and the accompanying drawings described below.

[0016] [Figure 1] It is a diagram for explaining the outline of the processing device of the present embodiment. [Figure 2] It is a diagram for explaining the outline of the processing device of the present embodiment. [Figure 3] It is a diagram for explaining the outline of the processing device of the present embodiment. [Figure 4] It is a diagram for explaining the outline of the processing device of the present embodiment. [Figure 5] It is a diagram for explaining the outline of the processing device of the present embodiment. [Figure 6] It is a diagram showing an example of the hardware configuration of the processing device of the present embodiment. [Figure 7] It is an example of a functional block diagram of the processing device of the present embodiment. [Figure 8] This diagram schematically shows an example of the information processed by the processing device of this embodiment. [Figure 9] This diagram schematically shows an example of the information processed by the processing device of this embodiment. [Figure 10] This diagram schematically shows an example of the information processed by the processing device of this embodiment. [Figure 11] This flowchart shows an example of the processing flow of the apparatus in this embodiment. [Figure 12] This figure shows an example of the product placement space in this embodiment. [Figure 13] This figure shows an example of the product placement space in this embodiment. [Figure 14] This is an example of a functional block diagram of the processing apparatus of this embodiment. [Figure 15] This figure shows an example of a payment completion notification displayed on a customer's terminal device. [Modes for carrying out the invention]

[0017] <First Embodiment> First, the overall structure and overview of the processing device of this embodiment will be explained using Figures 1 to 5. The processing device of this embodiment is used, for example, in a store where products such as confectionery and beverages are displayed in a corner of an office, and the employees of that office purchase the products on a self-service basis.

[0018] As shown in Figure 1, a product placement space 2 is provided in a corner of an office or similar location for placing product 3. A camera 5 installed in frame 4 photographs customer 1 as they take product 3 from product placement space 2.

[0019] As shown in Figure 2, the image generated by camera 5 includes customer 1 located in front of product placement space 2. The processing unit analyzes the image and recognizes customer 1 included in the image.

[0020] Furthermore, as shown in Figure 3, the image generated by camera 5 includes product 3 being held by customer 1, who is positioned in front of product placement space 2. The processing device analyzes this image and recognizes product 3 being held by customer 1.

[0021] The processing device then analyzes the image and, upon detecting that customer 1, who is holding product 3, has performed a predetermined action, performs a payment process based on the recognition result in response to the detection. In this embodiment, the "predetermined action" is described as "customer 1 moving outside the reference range 6." More specifically, when the processing device detects that customer 1, who is holding product 3, has moved outside the reference range 6 (see Figures 4 and 5), it performs a payment process based on the recognition result in response to the detection. In this embodiment, the reference range 6 is defined with respect to the product placement space 2. For example, the reference range 6 is defined as a predetermined area in front of the product placement space 2 (an area where customer 1, who is considering purchasing product 3, is expected to move while holding product 3).

[0022] According to the processing device of this embodiment, a store system is realized that eliminates payment processing at the cash register counter (product registration and payment, etc.) using a novel method unlike anything before.

[0023] Next, we will describe the configuration of the processing unit in detail. First, we will describe an example of the hardware configuration of the processing unit.

[0024] Each functional unit of a processing unit is realized by any combination of hardware and software, centered around the CPU (Central Processing Unit) of any computer, memory, programs loaded into memory, storage units such as hard disks that store those programs (which can store programs pre-installed at the time of shipment, as well as programs downloaded from storage media such as CDs (Compact Discs) or from servers on the Internet), and a network connection interface. It will be understood by those skilled in the art that there are various modifications to the implementation method and the device.

[0025] Figure 6 is a block diagram illustrating the hardware configuration of a processing unit. As shown in Figure 6, the processing unit includes a processor 1A, memory 2A, input / output interface 3A, peripheral circuitry 4A, and bus 5A. Peripheral circuitry 4A includes various modules. The processing unit does not necessarily have peripheral circuitry 4A. The processing unit may consist of multiple physically and / or logically separated devices, or it may consist of a single physically and / or logically integrated device. If the processing unit consists of multiple physically and / or logically separated devices, each of the multiple devices may have the above hardware configuration.

[0026] Bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuits 4A, and input / output interface 3A to send and receive data to and from each other. Processor 1A is a processing unit such as a CPU or GPU (Graphics Processing Unit). Memory 2A is a memory such as RAM (Random Access Memory) or ROM (Read Only Memory). Input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. Input devices include, for example, keyboards, mice, microphones, physical buttons, touch panels, etc. Output devices include, for example, displays, speakers, printers, mailers, etc. Processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0027] Figure 7 shows an example of a functional block diagram of the processing unit 10. As shown in the figure, the processing unit 10 has an image acquisition unit 11, a registration unit 12, a settlement unit 13, and a storage unit 14. Note that the processing unit 10 does not necessarily have to have a storage unit 14. In this case, an external device configured to communicate with the processing unit 10 will have the storage unit 14.

[0028] The memory unit 14 stores various types of information. For example, the memory unit 14 stores customer information as shown in Figure 8 and product information as shown in Figure 9.

[0029] Customer information refers to information about customer 1 who uses a store located in a corner of an office or similar location. Customer information for each customer 1 is stored in the storage unit 14 after prior registration by customer 1 who uses the store.

[0030] The illustrated customer information includes customer identification information to distinguish multiple customers from each other, name, contact information, facial image, features extracted from the facial image, and payment method information, all of which are linked together. The payment method information is information used for making payments at stores, and examples include, but are not limited to, credit card information, bank account information, information indicating a desire for deduction from salary and the process for making such deductions, and user identification information for payment services (code payment services, electronic money services, point services, etc.). The illustrated customer information is merely an example, and may include information on other items, or may omit some of the items shown.

[0031] Product information refers to information about product 3 sold in the store. In the illustrated product information, product identification information for distinguishing multiple products from each other, product name, unit price, image, and feature quantities extracted from the image are linked together. Note that the image for each product 3 may include multiple images of product 3 taken from multiple angles.

[0032] The image acquisition unit 11 acquires images generated by the camera 5, which photographs the area in front of the product placement space 2 (the side from which the product 3 is removed). The processing unit 10 and the camera 5 are connected by wire and / or wireless and are configured to communicate with each other. The image acquisition unit 11 acquires the images generated by the camera 5 in real time.

[0033] Camera 5 is installed in a position and orientation that generates an image including "the face of customer 1 located in front of the product placement space 2" and "the product 3 being held by customer 1". In addition, camera 5 may be installed in a position and orientation that generates an image including, in addition to the above-mentioned target, at least one of "the boundary of the reference range 6" and "the product 3 placed on the product placement space 2".

[0034] In this embodiment, a single camera 5 generates an image that includes the subject being photographed as described above. Camera 5 may be equipped with a fisheye lens or a standard lens (for example, with a field of view of approximately 40° to 60°).

[0035] Camera 5 generates video. Camera 5 may record continuously or only for a predetermined period of time (e.g., while the motion sensor detects a person in front of the product placement space 2).

[0036] The registration unit 12 recognizes the customer identification information of customer 1 contained in the image. For example, if the registration unit 12 detects a person's face in the image, it may recognize customer 1 contained in the image by performing a matching process based on the feature quantities extracted from the detected person's face and the feature quantities of customer 1 stored in the storage unit 14 (see Figure 8).

[0037] Furthermore, the registration unit 12 recognizes the product identification information of the product 3 taken out of the product storage space 2 by image processing. For example, if the registration unit 12 detects that the recognized customer 1 is holding an object, it may recognize the product 3 that customer 1 is holding (the product 3 taken out of the product storage space 2) by performing a matching process based on the feature quantities extracted from that object and the feature quantities of product 3 stored in the storage unit 14 (see Figure 9).

[0038] The registration unit 12 then stores the recognition result in the storage unit 14. Figure 10 schematically shows an example of the information (recognition information) that the registration unit 12 stores in the storage unit 14. As shown in the figure, the registration unit 12 registers the recognized customer identification information as recognition information. The registration unit 12 then registers the recognized product identification information in the recognition information, linked to the recognized customer identification information. The registration unit 12 may also read the unit price, etc., linked to the recognized product identification information from the product information (see Figure 9) and register it in the recognition information.

[0039] Furthermore, the registration unit 12 detects, through image processing, that customer 1 has returned product 3, which they were holding, to product placement space 2. When the registration unit 12 detects that customer 1 has returned product 3 to product placement space 2, it updates the recognition information (see Figure 10) accordingly. Specifically, the registration unit 12 deletes the product identification information of product 3 that has been returned to product placement space 2 from the recognition information.

[0040] Here, we will explain an example of an algorithm for detecting when customer 1 returns product 3, which they are holding, to product placement space 2. For example, the registration unit 12 may determine that customer 1 has returned product 3, which they were holding, to product placement space 2 when it detects a change in state from "customer 1 is holding product 3" to "customer 1 is not holding product 3". However, with this algorithm, if customer 1 puts product 3, which they were holding, into their own bag, for example, it may be incorrectly determined that product 3 has been returned to product placement space 2. Therefore, in order to avoid such false detections, the detection may be performed using the following algorithm.

[0041] For example, when the registration unit 12 detects a state change from "customer 1 is holding product 3" to "customer 1 is not holding product 3," it may determine whether the movement trajectory of product 3 in the image before the state change satisfies predetermined conditions. If the predetermined conditions are met, it may determine that customer 1 has returned product 3, which was being held, to the product placement space 2.

[0042] The predetermined conditions for the movement trajectory include "movement toward product placement space 2," "movement toward product placement space 2 and reaching a predetermined position in the image," and "movement reaching a predetermined position in the image." The predetermined position in the image is product placement space 2 if it is included in the image, and if it is not included in the image, it is the position that appears just before product 3, which is moving toward product placement space 2, goes out of frame.

[0043] By determining predetermined conditions for such movement trajectories, it is possible to suppress the inconvenience of incorrectly updating recognition information caused by actions such as customer 1 putting product 3 into a bag or pouch, customer 1 eating product 3, or product 3 being in the camera 5's blind spot due to how it is held.

[0044] Returning to Figure 7, the settlement unit 13 detects through image processing that customer 1 has removed product 3 from product placement space 2 and has not returned product 3 to product placement space 2, and has moved outside the reference range 6 defined relative to product placement space 2. In response, the settlement unit 13 performs payment based on the registration details provided by the registration unit 12. More specifically, the settlement unit 13 detects through image processing that customer 1, for whom at least one product identification information is registered in the recognition information (see Figure 10), has moved outside the reference range 6.

[0045] The reference range 6 is the area in front of the product placement space 2 (the side from which product 3 is taken out) (the area where customer 1, who is considering purchasing product 3, is expected to move while holding product 3). The size and shape of the reference range 6 are design considerations. Information indicating the reference range 6 in the image may be stored in the storage unit 14 in advance.

[0046] For example, if camera 5 generates an image that includes the boundary of the reference range 6, the payment unit 13 may monitor whether a part of customer 1's body is within the reference range 6 through image processing. For example, the payment unit 13 may determine that customer 1 has moved outside the reference range 6 in response to detection that customer 1's feet have gone outside the reference range 6. Alternatively, the payment unit 13 may monitor whether a part of customer 1's body other than the feet is within a predetermined area in the image, and determine that customer 1 has moved outside the reference range 6 in response to detection that that part of customer 1's body has gone outside the predetermined area.

[0047] The settlement unit 13 settles the predetermined amount based on the payment method information (see Figure 8) that has been registered in advance for each customer.

[0048] In this embodiment, where one camera 5 is installed, the registration unit 12 and the payment unit 13 process images generated by the same camera 5 to recognize the customer 1 and product 3, and to detect when the customer 1 has moved outside the reference range 6.

[0049] Next, an example of the processing flow of the processing unit 10 will be explained using the flowchart in Figure 11.

[0050] When the image acquisition unit 11 acquires an image (S10), the registration unit 12 performs a process to detect a person's face from the image (S11). If no person's face is detected from the image (No in S11), the process proceeds to S21. On the other hand, if a person's face is detected from the image (Yes in S11), the registration unit 12 recognizes customer 1 included in the image through a customer recognition process that includes a matching process based on the feature quantities extracted from the detected person's face and the feature quantities of customer 1 stored in the storage unit 14 (see Figure 8) (S12).

[0051] Then, if the customer identification information of the recognized customer 1 is not registered in the recognition information (see Figure 10) (No in S13), the registration unit 12 registers the customer identification information in the recognition information (S14). At this time, other customer information may also be registered in the recognition information. Other customer information may include the person's payment method information. The payment method information may be stored in the storage unit 14 in advance through a registration process by the customer themselves via an app or website. The registration unit 12 may then register the payment method information stored in the storage unit 14 into the recognition information. Alternatively, the registration unit 12 may accept input of payment method information from the customer on the spot via a predetermined input device. The registration unit 12 may then register the payment method information entered on the spot into the recognition information. On the other hand, if the customer identification information of the recognized customer 1 is already registered in the recognition information (see Figure 10) (Yes in S13), the registration unit 12 does not update the recognition information.

[0052] Subsequently, the registration unit 12 determines, based on the image, whether customer 1 is holding an object (S15). If it does not detect that customer 1 is holding an object (No in S15), the process proceeds to S19. On the other hand, if it does detect that customer 1 is holding an object (Yes in S15), the registration unit 12 recognizes the product that customer 1 is holding (product 3 that customer 1 took out of the product placement space 2) by matching the feature quantities extracted from the object with the feature quantities of product 3 stored in the storage unit 14 (see Figure 9) (S16).

[0053] Then, if the product identification information of the recognized product 3 is not registered in the recognition information (see Figure 10) (No. in S17), the registration unit 12 registers the product identification information in the recognition information (S18). At this time, the registration unit 12 may read information (unit price, product name, etc.) associated with the product identification information from the product information (see Figure 9) and register it in the recognition information. On the other hand, if the product identification information of the recognized product 3 is already registered in the recognition information (see Figure 10) (Yes in S17), the registration unit 12 does not update the recognition information.

[0054] Furthermore, the unit price of a product may be dynamically determined based on at least one of the following: "customer information including gender, age group, and purchase history," and "inventory information regarding at least one of the current stock and sales performance of the product."

[0055] For example, the storage unit 14 may store a function that outputs a unit price when a value of a predetermined parameter included in customer information and / or inventory information is input, or a table that shows the correspondence between the values ​​of a predetermined parameter included in customer information and / or inventory information and the unit price. The storage unit 14 may also store information indicating the inventory and sales status of products. Furthermore, the storage unit 14 may store customer information as described above. The registration unit 12 may then determine the unit price for each customer based on this information.

[0056] The settlement unit 13 then uses image processing to determine whether customer 1 is within the reference range 6 (S19). If customer 1 is outside the reference range 6 (No in S19), the settlement unit 13 performs settlement processing based on the recognition information at that time (see Figure 10) (S20). That is, the settlement unit 13 identifies the payment amount calculated based on the recognition information. If the identified payment amount is not "0", the settlement unit 13 uses the payment method information that has been pre-registered and linked to customer 1 to perform payment processing for the identified payment amount. If the identified payment amount is "0", the detection unit 13 does not perform payment processing using the payment method information. If customer 1 is within the reference range 6 (Yes in S19), the settlement unit 13 does not perform settlement processing. The process then returns to S10 and the same process is repeated.

[0057] Next, the effects of the processing device 10 of this embodiment will be explained. According to the processing device 10 of this embodiment, a store system is realized that eliminates payment processing at the cash register counter (product registration and payment, etc.) using a novel method not previously available. As a result, even in stores where gates or vehicles cannot be used, the elimination of payment processing at the cash register counter (product registration and payment, etc.) improves convenience for users and reduces the workload for the store.

[0058] Furthermore, according to the processing device 10 of this embodiment, based on images generated by a single camera 5, it is possible to recognize the customer 1 and product 3, and to detect when the customer 1 has moved outside the reference range 6. This reduces the cost burden, such as equipment costs.

[0059] Now, a modified example of this embodiment will be described.

[0060] In the above example, the processing unit 10 was described on the premise that it would be used in a store where products such as confectionery and beverages are displayed in a corner of an office, and employees of that office purchase the products on a self-service basis. However, the usage scenarios of the processing unit 10 are not limited to this. The processing unit 10 may also be used in places other than offices.

[0061] Furthermore, the product placement space 2 only needs to be able to hold products, and instead of the stand shown in Figure 1, other items such as shelves, boxes, or chairs may be used to constitute the product placement space 2.

[0062] An example is shown in Figure 13. Figure 12 is an extracted view of frame 4 in Figure 13. Each of the two components that make up frame 4 is equipped with a camera 5 and lighting.

[0063] The light-emitting surface of the illumination extends in one direction and has a light-emitting part and a cover that covers the light-emitting part. The illumination mainly emits light in a direction perpendicular to the direction in which the light-emitting surface extends. The light-emitting part has a light-emitting element such as an LED and emits light in the direction not covered by the cover. In the case of an LED, multiple LEDs are arranged in the direction in which the illumination extends (up and down in the diagram).

[0064] Camera 5 is located at one end of the frame 4 component, which extends in a straight line, and its imaging range is the direction from which the illumination light is emitted. For example, in the left component of frame 4 in Figure 12, camera 5 has an imaging range of downward and diagonally downward to the right. Also, in the right component of frame 4 in Figure 12, camera 5 has an imaging range of upward and diagonally upward to the left.

[0065] As shown in Figure 13, the frame 4 is attached to the front frame (or the front of the side walls on both sides) of the product shelf that constitutes the product placement space. In this case, one component of the frame 4 is attached to one front frame with the camera 5 positioned downwards, and the other component of the frame 4 is attached to the other front frame with the camera 5 positioned upwards. The camera 5 attached to one component of the frame 4 captures images upwards and diagonally upwards so as to include the opening of the product shelf in its imaging range. On the other hand, the camera 5 attached to the other component of the frame 4 captures images downwards and diagonally downwards so as to include the opening of the product shelf in its imaging range. By using these two cameras 5, the entire range of the opening of the product shelf can be captured.

[0066] The processing unit 10 then generates various types of information as shown in the above embodiment by processing the images captured by the two cameras 5. For example, one of the cameras 5 captures images of the area above and diagonally above.

[0067] In addition, although camera 5 is installed on frame 4 in the example shown in Figure 1, camera 5 may be installed on other objects such as the ceiling, wall, or support column instead.

[0068] In the example above, one camera 5 was installed, and the target, including customer 1, product 3, and reference range 6, was captured with that single camera. However, multiple cameras 5 could be installed, and the target could be captured by each camera 5 separately. In this case, the target that each camera 5 needs to capture is limited (i.e., it is not necessary to capture all the target), making the installation of each camera 5 relatively easy.

[0069] Furthermore, in the above example, customer 1 was recognized by image-based facial recognition. However, customer 1 may be recognized by means other than image analysis. For example, the processing unit 10 may recognize customer 1 based on biometric information such as fingerprints, voiceprints, and iris scans entered via a predetermined input device (fingerprint sensor, microphone, etc.). In addition, the processing unit 10 may acquire customer identification information from the customer's portable device (IC card, smartphone, smartwatch, tablet terminal, mobile phone, etc.) via a reader, etc. In this example, the image generated by camera 5 does not have to include customer 1's face. This reduces the number of objects that camera 5 needs to photograph, thus reducing the number of cameras 5 required and making it easier to install camera 5.

[0070] Alternatively, the payment unit 13 may initiate payment when customer 1 moves out of the frame of the image from camera 5. This eliminates the need to pre-store information indicating the reference range 6 within the image in the storage unit 14, thereby reducing the processing burden on the operator.

[0071] In addition, the payment unit 13 may perform payment in response to detection that customer 1 has performed a predetermined action, such as detecting that "customer 1 has moved a predetermined number of steps in a predetermined direction (away from the product placement space)" or "customer 1 has moved in a predetermined direction (away from the product placement space) for a predetermined amount of time". In this way, it becomes unnecessary to store information indicating the reference range 6 in the image in the storage unit 14 in advance, thus reducing the processing burden on the operator.

[0072] <Second Embodiment> The area in which a customer 1 considering purchasing product 3 is likely to move while holding product 3 is likely to differ for each customer 1. That is, some customers 1 may remain still in front of the product placement space 2 while considering the purchase, while others may move to a location relatively far from the product placement space 2. In view of this situation, the processing device 10 of this embodiment is equipped with a function to set an appropriate reference range 6 for each customer 1. This will be explained in detail below.

[0073] Figure 14 shows an example of a functional block diagram of the processing device 10 of this embodiment. As shown in the figure, the processing device 10 includes an image acquisition unit 11, a registration unit 12, a settlement unit 13, a storage unit 14, a completion notification unit 15, and a reference range correction unit 16.

[0074] The memory unit 14 stores information indicating the reference range 6 for each customer 1 (information indicating the reference range 6 in the image). The payment unit 13 then identifies the reference range 6 for each customer 1 based on the information indicating the reference range 6 for each customer 1 and detects when each customer 1 moves outside of that reference range 6.

[0075] The completion notification unit 15 sends a payment completion notification to customer 1's terminal after payment is completed by the payment unit 13, and receives a response to the payment completion notification from customer 1's terminal device. The payment completion notification may be implemented, for example, through a predetermined application installed on customer 1's terminal, or by other means. Figure 15 shows an example of a payment completion notification displayed on customer 1's terminal device 7. The illustrated payment completion notification includes the name of the product 3 to be paid for, the price, the payment date, etc.

[0076] Although not shown in the diagram, terminal device 7 may accept the following responses from the payment completion notification screen: "Input that there is no problem" and "Input that product 3 has not been purchased." Terminal device 7 may then transmit these responses to processing device 10.

[0077] If the response to the payment completion notice indicates that the customer 1 has not purchased the product 3 indicated in the payment completion notice, the reference range modification unit 16 modifies the reference range 6 for that customer 1 in a direction that widens it, and stores information indicating the modified content in the storage unit 14.

[0078] Here, we will explain an example of an algorithm for modifying the reference range 6 for each customer. For example, the reference range modification unit 16 may identify the movement trajectory of customer 1 (customer 1 who made the "input that they did not purchase product 3") while holding product 3 (product 3 for which customer 1 has been inputted as not having purchased product 3) based on the image generated by camera 5. Then, the reference range modification unit 16 may determine the reference range 6 for customer 1 based on the identified movement trajectory. For example, the reference range modification unit 16 may determine the reference range 6 for the customer so as to encompass the identified movement trajectory. Alternatively, the reference range 6 for each customer 1 may be determined by machine learning using the identified movement trajectory as training data.

[0079] The other configurations of the processing apparatus 10 in this embodiment are the same as those in the first embodiment.

[0080] As described above, the processing device 10 of this embodiment achieves the same effects as the first embodiment. Furthermore, with the processing device 10 of this embodiment, an appropriate reference range 6 is set for each customer 1. Therefore, the inconvenience of mistakenly performing a payment process before the customer 1 decides to purchase the product 3 they are holding can be reduced.

[0081] <Third Embodiment> In this embodiment, the settlement unit 13 monitors the elapsed time since the product identification information was linked to the customer identification information, as long as it has not been detected that the product 3 has been returned to the product placement space 2 or that customer 1 has moved outside the reference range 6. If the elapsed time exceeds a threshold, the settlement unit 13 performs payment based on the registration details by the registration unit 12 without detecting that customer 1 has moved outside the reference range 6.

[0082] The other configurations of the processing apparatus 10 in this embodiment are the same as those in the first and second embodiments.

[0083] As described above, the processing device 10 of this embodiment achieves the same effects as those of the first and second embodiments. Furthermore, with the processing device 10 of this embodiment, even if it is not detected that customer 1 (customer 1 whose product identification information is registered in the recognition information (see Figure 10) linked to customer identification information) who is holding product 3 has moved outside the reference range 6 due to some malfunction, payment processing can be performed according to the elapsed time. As a result, payment omissions can be suppressed.

[0084] <Fourth Embodiment> In this embodiment, the settlement unit 13 detects changes in the orientation of the customer 1's body through image processing and performs settlement processing based on the detection result. For example, if the orientation of the body of customer 1 (customer 1 whose product identification information is registered in association with customer identification information in the recognition information (see Figure 10)) who is holding product 3 satisfies a predetermined condition (turning 180° to the opposite side from a state facing directly towards product placement space 2), the settlement unit 13 may perform settlement based on the registered content by the registration unit 12 without detecting that customer 1 has moved outside the reference range 6.

[0085] Customer 1, having decided to purchase item 3 which they are holding, is expected to change their body orientation and move away from the front of item placement space 2. On the other hand, Customer 1, who is still considering whether to purchase item 3 which they are holding, is expected to remain facing item placement space 2 and be positioned around item placement space 2. Therefore, the payment unit 13 may, as in this example, perform the payment when the orientation of customer 1's body meets predetermined conditions.

[0086] In addition, the settlement unit 13 may perform settlement based on the registered information by the registration unit 12 when it detects that the orientation of the body of customer 1 (customer 1 whose product identification information is registered in the recognition information (see Figure 10) linked to customer identification information) who is holding product 3 satisfies a predetermined condition (facing 180° away from a position directly facing product placement space 2) and that customer 1 has moved outside the reference range 6.

[0087] As described above, customer 1, who is considering purchasing the product 3 they are holding, is generally expected to remain facing the product placement space 2 and be positioned around the product placement space 2. However, some customers 1 may move outside the reference range 6 while still facing the product placement space 2 during their consideration. If payment is simply triggered when the customer moves outside the reference range 6, the inconvenience of making a payment while still considering a purchase may occur. Therefore, the payment unit 13 may perform payment when it detects that customer 1's body orientation satisfies a predetermined condition (facing 180° away from the product placement space 2) and that customer 1 has moved outside the reference range 6. By doing so, the occurrence of the above inconvenience can be suppressed.

[0088] The other configurations of the processing apparatus 10 in this embodiment are the same as those in the first to third embodiments.

[0089] The processing device 10 of this embodiment, as described above, achieves the same effects and advantages as the first to third embodiments. Furthermore, the processing device 10 of this embodiment allows for settlement at an appropriate timing.

[0090] <Fifth Embodiment> In this embodiment, the settlement unit 13 performs settlement in response to the detection that customer 1 has moved outside the reference range 6. However, the settlement is not triggered by the detection that customer 1 has moved outside the reference range 6, but rather by the detection that a predetermined process, which is triggered by the detection that customer 1 has moved outside the reference range 6, has been performed.

[0091] Specifically, the settlement unit 13, triggered by the detection that customer 1 has moved outside the reference range 6, transmits information indicating the settlement details determined based on the registration details by the registration unit 12 to customer 1's terminal device 7, and receives a response from customer 1's terminal device 7 indicating whether the settlement details are correct or not.

[0092] The information indicating the payment details includes the name and price of the product 3 to be paid for. The transmission of the payment details information to customer 1's terminal device 7 and the reception of the response from customer 1's terminal device 7 may be achieved, for example, through a predetermined application installed on customer 1's terminal, or by other means.

[0093] Then, the settlement unit 13, triggered by the receipt of a response confirming that the settlement details are correct, settles the payment indicated in the settlement details.

[0094] The other configurations of the processing apparatus 10 in this embodiment are the same as those in the first to third embodiments.

[0095] As described above, the processing device 10 of this embodiment achieves the same effects and advantages as the first to third embodiments. Furthermore, since the processing device 10 of this embodiment performs a confirmation process with the customer 1 before settlement, the inconvenience of making a settlement with incorrect information can be suppressed.

[0096] Although the present invention has been described above with reference to the embodiments (and examples), the present invention is not limited to the above embodiments (and examples). Various modifications to the structure and details of the present invention can be made, which can be understood by those skilled in the art within the scope of the present invention.

[0097] Some or all of the above embodiments may also be described as follows, but are not limited to these. 1. A registration means that recognizes the product identification information of a product removed from the product storage space by image processing, and registers the recognized product identification information linked to customer identification information. When image processing detects that a customer who has taken out the product has performed a predetermined action, a payment means performs payment based on the registered information from the registration means, A processing device. 2. The processing apparatus according to claim 1, wherein the registration means recognizes the customer identification information of the customer contained in the image, and registers the recognized product identification information in association with the recognized customer identification information. 3. The processing apparatus according to 1 or 2, wherein the registration means and the payment means process images generated by the same camera to recognize the product identification information and to detect that the customer has performed the predetermined action. 4. The aforementioned payment method is: Upon detection that the customer has performed the predetermined action, information indicating the payment details determined based on the registration details is transmitted to the customer's terminal device. The system receives a response from the customer's terminal device confirming whether the payment details are correct or not. A processing device according to any one of 1 to 3 that, upon receiving a response confirming that the payment details are correct, settles the payment amount indicated in the payment details. 5. The payment means is a processing device according to any one of 1 to 4, which performs payment based on the registered information without detecting that the customer has performed a predetermined action when the elapsed time since the product identification information was linked to the customer identification information exceeds a threshold. 6. The processing apparatus according to any one of 1 to 5, wherein the registration means registers the product identification information linked to the customer identification information identified by a facial image or iris. 7. The processing apparatus according to any one of 1 to 6, wherein the registration means detects by image processing that the customer has returned the product to the product placement space, and deletes the product identification information of the product returned to the product placement space from the information linking the product identification information and the customer identification information. 8. The payment means is a processing device according to any one of 1 to 7, which detects, by image processing, that the orientation of the customer's body meets a predetermined condition, that the customer has moved a predetermined number of steps in a predetermined direction, or that the customer has moved in a predetermined direction for a predetermined time as a predetermined action. 9. The payment means is a processing device according to any one of 1 to 8, wherein the payment means detects as a predetermined action that the customer has moved outside a reference range determined based on the product placement space. 10. The processing apparatus according to 9, wherein the reference range is the area in front of the product placement space. 11. The payment means identifies the respective reference range for each customer based on the information indicating the reference range for each customer, A completion notification means that, after payment by the payment means, sends a payment completion notification to the customer's terminal device and receives a response to the payment completion notification from the customer's terminal device, If the response to the payment completion notice indicates that the customer has not purchased the product indicated in the payment completion notice, the means for modifying the reference range to broaden the reference range of that customer, The apparatus according to 9 or 10, further comprising the same. 12. Computers, Image processing is used to recognize the product identification information of the product removed from the product storage space, and the recognized product identification information is registered in association with customer identification information. A processing method that, when it is detected by image processing that a customer who has taken out the product has performed a predetermined action, performs payment based on the registered contents of the product identification information. 13. Computers, A registration means that recognizes product identification information of a product removed from a product storage space using image processing, and registers the recognized product identification information in association with customer identification information. When image processing detects that a customer who has taken out the product has performed a predetermined action, a payment means performs payment based on the registered information from the registration means. A program that makes it function as such.

Claims

1. A recognition means that recognizes the customer identification information of the customer by processing an image of the customer taking a product from the product placement space, A registration means for registering the customer identification information recognized by the recognition means and the product identification information of the product taken out of the product storage space by the customer, When image processing detects that a customer who has taken out the product has performed a predetermined action, a payment means performs payment based on the registered information from the registration means, A completion notification means that sends a payment completion notification to the customer's terminal after payment has been made by the aforementioned payment means, It has, The aforementioned predetermined action is that the customer has moved outside the reference range defined based on the product placement space. The processing device is characterized in that the payment completion notification includes at least the name of the product to be paid and the price.

2. The processing apparatus according to claim 1, wherein the reference range is the area in front of the product placement space.

3. The processing apparatus according to claim 1 or 2, wherein the registration means registers the product identification information linked to the customer identification information identified by a facial image or iris.

4. The processing apparatus according to any one of claims 1 to 3, wherein the registration means detects by image processing that the customer has returned the product to the product placement space, and deletes the product identification information of the product returned to the product placement space from the information linking the product identification information and the customer identification information.

5. One or more computers, A recognition step involves processing an image of a customer taking a product from a product placement area to recognize the customer's identification information. A registration step involves linking the customer identification information recognized in the recognition step with the product identification information of the product taken out of the product storage space by the customer and registering the results. When image processing detects that the customer who took out the product has performed a predetermined action, a payment process is performed that carries out payment based on the registered information from the registration process. After payment is completed through the aforementioned payment process, a completion notification process is performed to send a payment completion notification to the customer's terminal. Execute, The aforementioned predetermined action is that the customer has moved outside the reference range defined based on the product placement space. The payment completion notification is characterized in that it includes at least the name of the product to be paid and the price.

6. Computers, A recognition means that recognizes the customer identification information of a customer by processing an image of a customer taking a product from a product placement space. A registration means for registering the customer identification information recognized by the recognition means and the product identification information of the product taken out of the product storage space by the customer, When image processing detects that a customer who has taken out the product has performed a predetermined action, a payment means performs payment based on the registered information from the registration means. A completion notification means that sends a payment completion notification to the customer's terminal after payment has been made by the aforementioned payment means. To make it function as, The aforementioned predetermined action is that the customer has moved outside the reference range defined based on the product placement space. The program is characterized in that the payment completion notification includes at least the name of the product to be paid and the price.

Images