Processing apparatus, processing method and program

Abstract

To provide a store system that eliminates settlement processing (article registration, payment, etc.) at a checkout counter.SOLUTION: A processing apparatus has: a recognition part which processes an image of a customer who takes an article out of an article mount space to recognize customer identification information on the customer; a registration part which links and registers the customer identification information recognized by the recognition part and article identification information on the article taken out of the article mount space by the customer; a settlement part which performs settlement based upon details of the registration by the registration part on detecting the customer who has taken the article out performing a predetermined operation through the image processing; and a completion notification part which transmits settlement completion notice to a terminal of the customer after the settlement by the settlement part. The predetermined operation is a movement of the article mount space out of a reference range determined by a standard by the customer. The settlement completion notice at least includes the article name and price of the object article of the settlement.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a processing device, a processing method, and a program. [Background technology]

[0002] Non-Patent Document 1 and Patent Documents 1 and 2 disclose store systems that eliminate the need for payment processing (item registration, payment, etc.) at checkout counters.

[0003] Specifically, the store system disclosed in Non-Patent Document 1 installs a gate at the entrance of the store and recognizes customers who enter the store based on information read by a reader installed at the gate. The store system then uses image processing or the like to recognize the products picked up by the customers and completes payment when the customers pass through the gate and exit the store.

[0004] In the purchase settlement support system disclosed in Patent Document 1, a gate is installed to manage the entry and exit of customers who pass through the gate, and after passing through the gate and leaving the store, payment processing for products that the customer takes out of the store is carried out.

[0005] In the vehicle payment system disclosed in Patent Document 2, a product placement section is provided inside the vehicle's interior, and payment processing for the product is carried out when a predetermined condition is met after the movement of the product outside the vehicle is detected.

[0006] Non-patent document 2 discloses a store in which products such as snacks and drinks are displayed in a corner of an office, and payment methods such as cashless payment terminals and cash collection boxes are installed, allowing office employees to purchase products on a self-service basis. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2019-86832 [Patent Document 2] Japanese Patent Application Publication No. 2019-117473 [Non-patent literature]

[0008] [Non-Patent Document 1] Takuya Miyata, "Amazon Go's cashierless supermarket realized with cameras and microphones," [online], December 10, 2016, [Retrieved December 6, 2019], Internet<URL:https: / / www.huffingtonpost.jp / tak-miyata / amazon-go_b_13521384.html> [Non-patent document 2] “12 Recommended Office Convenience Stores: How They Work, Reviews, and Benefits,” [online], November 18, 2019, [Retrieved December 6, 2019], Internet <URL:https: / / boxil.jp / mag / a5253 / ?utm_source=google&utm_medium=cpc&utm_campaign=office_c-z&gclid=EAIaIQobChMIzJbdxKOg5gIVT3RgCh26uAb8EAAYAiAAEgKK-vD_BwE> Summary of the Invention [Problem to be solved by the invention]

[0009] Eliminating payment processing at the cash register counter using the technologies disclosed in Non-Patent Document 1, Patent Document 1, and Patent Document 2 improves convenience for users and reduces the workload of stores. However, the technologies disclosed in Non-Patent Document 1 and Patent Document 1 require the presence of a gate and cannot be used in environments where a gate cannot be installed. Furthermore, the technology disclosed in Patent Document 1 requires the presence of a vehicle and cannot be used in environments where a vehicle cannot be used. For example, in a store located in a section of an office, such as that disclosed in Non-Patent Document 2, it may not be possible to use a gate or a vehicle due to cost or space constraints.

[0010] The object of the present invention is to realize a store system that eliminates payment processing (product registration, payment, etc.) at the checkout counter using a new, unprecedented method. [Means for solving the problem]

[0011] According to the present invention, a registration means for recognizing, by image processing, the commodity identification information of a commodity removed from the commodity placement space, and registering the recognized commodity identification information in association with customer identification information; a payment means for performing payment based on the registered details when it is detected by image processing that the customer who has taken out the product has performed a predetermined action; a notification means for prompting confirmation of payment based on the registered contents by the registration means; and The notification means is provided with a processing device that sends a notification to the customer's terminal device prompting confirmation of payment based on the registered details when the elapsed time since the product identification information was linked to the customer identification information exceeds a threshold.

[0012] Further, according to the present invention, The computer a registration step of recognizing product identification information of a product removed from the product placement space by image processing, and linking the recognized product identification information to customer identification information and registering it; a payment step of performing payment based on the registered details in the registration step when it is detected by image processing that the customer who has taken out the product has performed a predetermined action; a notification step of prompting confirmation of payment based on the registered content in the registration step; Run In the notification step, a processing method is provided in which, if the elapsed time since the product identification information was linked to the customer identification information exceeds a threshold, a notification prompting confirmation of payment based on the registered content is sent to the customer's terminal device.

[0013] Further, according to the present invention, Computer, a registration means for recognizing, by image processing, the commodity identification information of a commodity removed from the commodity placement space, and registering the recognized commodity identification information in association with customer identification information; a payment means for performing payment based on the registered details when it is detected by image processing that the customer who has taken out the product has performed a predetermined action; a notification means for prompting confirmation of payment based on the registered contents by the registration means; It functions as The notification means is provided with a program that sends a notification to the customer's terminal device prompting confirmation of payment based on the registered details when the elapsed time since the product identification information was linked to the customer identification information exceeds a threshold. [Effects of the Invention]

[0014] According to the present invention, a store system is realized that eliminates the need for payment processing (item registration, payment, etc.) at a checkout counter using a new, unprecedented method. [Brief explanation of the drawings]

[0015] The above-mentioned objects, as well as other objects, features and advantages, will become more apparent from the preferred embodiments described below and the accompanying drawings.

[0016] [Figure 1] FIG. 1 is a diagram for explaining an overview of a processing apparatus according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram for explaining an overview of a processing apparatus according to an embodiment of the present invention. [Figure 3] FIG. 1 is a diagram for explaining an overview of a processing apparatus according to an embodiment of the present invention. [Figure 4] FIG. 1 is a diagram for explaining an overview of a processing apparatus according to an embodiment of the present invention. [Figure 5] FIG. 1 is a diagram for explaining an overview of a processing apparatus according to an embodiment of the present invention. [Figure 6] FIG. 2 is a diagram illustrating an example of a hardware configuration of a processing device according to the present embodiment. [Figure 7] FIG. 2 is a functional block diagram of a processing apparatus according to an embodiment of the present invention; [Figure 8] FIG. 2 is a diagram schematically illustrating an example of information processed by the processing device of the present embodiment. [Figure 9] FIG. 2 is a diagram schematically illustrating an example of information processed by the processing device of the present embodiment. [Figure 10] FIG. 2 is a diagram schematically illustrating an example of information processed by the processing device of the present embodiment. [Figure 11] 10 is a flowchart showing an example of a processing flow of the processing device of the present embodiment. [Figure 12] FIG. 2 is a diagram showing an example of a product placement space according to the present embodiment. [Figure 13] FIG. 2 is a diagram showing an example of a product placement space according to the present embodiment. [Figure 14] FIG. 2 is a functional block diagram of a processing apparatus according to an embodiment of the present invention; [Figure 15] FIG. 10 is a diagram showing an example of a payment completion notice displayed on a customer's terminal device. DETAILED DESCRIPTION OF THE INVENTION

[0017] First Embodiment First, an overview and summary of the processing device of this embodiment will be described with reference to Figures 1 to 5. The processing device of this embodiment is used, for example, in a store where products such as snacks and drinks are displayed in a corner of an office and employees of the office purchase the products on a self-service basis.

[0018] As shown in Fig. 1, a product placement space 2 for placing a product 3 is provided in a section of an office or the like. A camera 5 installed in a frame 4 photographs a customer 1 taking out a product 3 from the product placement space 2.

[0019] 2, the image generated by the camera 5 includes the customer 1 positioned in front of the product placement space 2. The processing device analyzes the image and recognizes the customer 1 included in the image.

[0020] 3, the image generated by the camera 5 includes the product 3 being held by the customer 1 who is positioned in front of the product placement space 2. The processing device analyzes the image and recognizes the product 3 being held by the customer 1.

[0021] The processing device then analyzes the image, and upon detecting that the customer 1 holding the product 3 has performed a predetermined action, performs payment processing based on the recognition result in response to the detection. In this embodiment, the "predetermined action" is described as "the customer 1 moving out of the reference range 6." More specifically, upon detecting that the customer 1 holding the product 3 has moved out of the reference range 6 (see FIGS. 4 and 5), the processing device performs payment processing based on the recognition result in response to the detection. In this embodiment, the reference range 6 is defined based on the product placement space 2. For example, a predetermined area in front of the product placement space 2 (an area through which the customer 1 considering purchasing the product 3 is expected to move while holding the product 3) is defined as the reference range 6.

[0022] According to the processing device of this embodiment, a store system that eliminates the need for payment processing (item registration, payment, etc.) at the checkout counter can be realized using a new and unprecedented method.

[0023] Next, the configuration of the processing device will be described in detail. First, an example of the hardware configuration of the processing device will be described.

[0024] Each functional unit of the processing device is realized by any combination of hardware and software, centered on a computer's CPU (Central Processing Unit), memory, programs loaded into the memory, a storage unit such as a hard disk that stores the programs (this can store programs that are pre-loaded when the device is shipped, as well as programs downloaded from storage media such as CDs (Compact Discs) or servers on the Internet), and a network connection interface. Those skilled in the art will understand that there are many variations in the methods and devices for realizing this.

[0025] FIG. 6 is a block diagram illustrating an example of the hardware configuration of a processing device. As shown in FIG. 6, the processing device has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The processing device does not necessarily have the peripheral circuit 4A. Note that the processing device may be composed of multiple physically and / or logically separated devices, or may be composed of a single physically and / or logically integrated device. When the processing device is composed of multiple physically and / or logically separated devices, each of the multiple devices may have the above hardware configuration.

[0026] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to transmit and receive data to and from each other. The processor 1A is an arithmetic processing device such as a CPU or a GPU (Graphics Processing Unit). The memory 2A is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The input / output interface 3A includes an interface for acquiring information from an input device, an external device, an external server, an external sensor, a camera, etc., and an interface for outputting information to an output device, an external device, an external server, etc. Examples of input devices include a keyboard, a mouse, a microphone, physical buttons, a touch panel, etc. Examples of output devices include a display, a speaker, a printer, a mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0027] 7 shows an example of a functional block diagram of the processing device 10. As shown in the figure, the processing device 10 has an image acquisition unit 11, a registration unit 12, a settlement unit 13, and a storage unit 14. Note that the processing device 10 does not necessarily have to have the storage unit 14. In this case, an external device configured to be able to communicate with the processing device 10 has the storage unit 14.

[0028] The storage unit 14 stores various types of information. For example, the storage unit 14 stores customer information such as that shown in FIG. 8 and product information such as that shown in FIG.

[0029] The customer information is information about customers 1 who use a store located in a section of an office, etc. The customer information of each customer 1 is stored in the storage unit 14 through advance registration by the customers 1 who use the store.

[0030] The illustrated customer information links together customer identification information for distinguishing multiple customers from one another, their name, contact information, a facial image, features extracted from the facial image, and payment method information. Payment method information is information used to make payments at stores, and examples include, but are not limited to, credit card information, account information, a request for payroll deduction and information for implementing that process, and user identification information for payment services (such as code payment services, electronic money services, and point services). Note that the illustrated customer information is merely an example, and may include information on other items, or may not include some of the items illustrated.

[0031] The product information is information about the products 3 sold in the store. In the product information shown in the figure, product identification information for distinguishing between multiple products, product names, unit prices, images, and feature amounts extracted from the images are linked together. Note that the images of each product 3 may include multiple images of the product 3 taken from multiple angles.

[0032] The image acquisition unit 11 acquires images generated by a camera 5 that captures an area in front of the product placement space 2 (the side where the product 3 is removed). The processing device 10 and the camera 5 are connected by wire and / or wirelessly and are configured to be able to communicate with each other. The image acquisition unit 11 acquires the images generated by the camera 5 by real-time processing.

[0033] The camera 5 is installed in a position and orientation that generates an image that includes "the face of the customer 1 positioned in front of the product placement space 2" and "the product 3 being held by the customer 1." The camera 5 may also be installed in a position and orientation that generates an image that includes at least one of "the boundary of the reference range 6" and "the product 3 placed in the product placement space 2" in addition to the above-mentioned subject.

[0034] In this embodiment, one camera 5 generates an image including the subject as described above. The camera 5 may be equipped with a fisheye lens or a standard lens (for example, with an angle of view of approximately 40° to 60°).

[0035] The camera 5 generates moving images. The camera 5 may take pictures all the time, or may take pictures only for a predetermined period of time (for example, while a human presence sensor detects a person in front of the product placement space 2).

[0036] The registration unit 12 recognizes customer identification information of the customer 1 included in the image. For example, when the registration unit 12 detects a person's face from the image, the registration unit 12 may recognize the customer 1 included in the image by performing a matching process based on the feature amount extracted from the detected person's face and the feature amount of the customer 1 stored in the storage unit 14 (see FIG. 8).

[0037] Furthermore, the registration unit 12 uses image processing to recognize the product identification information of the product 3 taken out of the product placement space 2. For example, when the registration unit 12 detects that the recognized customer 1 is holding an object, it may recognize the product 3 held by the customer 1 (the product 3 taken out of the product placement space 2) by performing a matching process based on the feature amount extracted from the object and the feature amount of the product 3 stored in the memory unit 14 (see FIG. 9 ).

[0038] The registration unit 12 then stores the recognition result in the memory unit 14. FIG. 10 schematically shows an example of information (recognition information) that the registration unit 12 stores in the memory unit 14. As shown in the figure, the registration unit 12 registers the recognized customer identification information in the recognition information. The registration unit 12 then links the recognized product identification information to the recognized customer identification information and registers it in the recognition information. Note that the registration unit 12 may further read out the unit price, etc. linked to the recognized product identification information from the product information (see FIG. 9) and register it in the recognition information.

[0039] Furthermore, the registration unit 12 detects through image processing that the customer 1 has returned the product 3 being held by him to the product placement space 2. When the registration unit 12 detects that the customer 1 has returned the product 3 being held by him to the product placement space 2, it updates the recognition information (see FIG. 10 ) accordingly. Specifically, the registration unit 12 deletes the product identification information, etc. of the product 3 being returned to the product placement space 2 from the recognition information.

[0040] Here, an example of an algorithm for detecting when the product 3 held by the customer 1 has been returned to the product placement space 2 will be described. For example, when the registration unit 12 detects a state change from "customer 1 is holding the product 3" to "customer 1 is not holding the product 3," it may determine that the product 3 held by the customer 1 has been returned to the product placement space 2. However, this algorithm may erroneously detect that the product 3 has been returned to the product placement space 2 if, for example, customer 1 puts the product 3 held by the customer 1 into his or her bag. Therefore, to avoid such erroneous detection, the following algorithm may be used for the detection.

[0041] For example, when detecting a state change from "customer 1 is holding product 3" to "customer 1 is not holding product 3," the registration unit 12 may determine whether the movement trajectory of product 3 in the image before the state change satisfies a predetermined condition. If the predetermined condition is satisfied, the registration unit 12 may determine that customer 1 has returned the product 3 he is holding to the product placement space 2.

[0042] The predetermined conditions for the movement trajectory are "movement towards the product placement space 2," "movement towards the product placement space 2 and reaching a predetermined position in the image," "movement reaching a predetermined position in the image," etc. The predetermined position in the image is the product placement space 2 if the image includes the product placement space 2, and if the image does not include the product placement space 2, it is a position where the product 3 heading towards the product placement space 2 appears just before it leaves the frame, etc.

[0043] By determining the predetermined conditions of such a movement trajectory, it is possible to prevent the inconvenience of erroneously updating the recognition information due to an action such as customer 1 putting product 3 into a bag or purse, customer 1 eating product 3, or the way product 3 is held causing product 3 to be in the blind spot of camera 5.

[0044] 7, the payment unit 13 performs payment based on the details registered by the registration unit 12 in response to detecting through image processing that a customer 1 who has removed a product 3 from the product placement space 2 but has not returned the product 3 to the product placement space 2 has moved outside of a reference range 6 defined based on the product placement space 2. More specifically, the payment unit 13 performs payment based on the details registered by the registration unit 12 in response to detecting through image processing that a customer 1 who has at least one product identification information registered in the recognition information (see FIG. 10) has moved outside of the reference range 6.

[0045] The reference range 6 is an area in front of the product placement space 2 (the side where the product 3 is removed) (an area where a customer 1 considering purchasing the product 3 is expected to move while holding the product 3). The size and shape of the reference range 6 are design factors. Information indicating the reference range 6 in the image may be stored in advance in the memory unit 14.

[0046] For example, when the camera 5 generates an image including the boundary of the reference range 6, the payment unit 13 may use image processing to monitor whether or not a part of the body of the customer 1 is within the reference range 6. For example, the payment unit 13 may determine that the customer 1 has moved outside the reference range 6 in response to detecting that the feet of the customer 1 have moved outside the reference range 6. The payment unit 13 may also monitor whether or not a part of the body of the customer 1 other than the feet is within a predetermined area in the image, and determine that the customer 1 has moved outside the reference range 6 in response to detecting that part of the body of the customer 1 has moved outside the predetermined area.

[0047] The settlement unit 13 settles the predetermined amount based on the settlement method information (see FIG. 8) registered in advance for each customer 1.

[0048] In this embodiment where one camera 5 is installed, the registration unit 12 and the payment unit 13 process the images generated by the same camera 5 to recognize the customer 1 and the product 3, and to detect whether the customer 1 has moved outside the reference range 6.

[0049] Next, an example of the processing flow of the processing device 10 will be described with reference to the flowchart of FIG.

[0050] When the image acquisition unit 11 acquires an image (S10), the registration unit 12 performs a process to detect a person's face from the image (S11). If a person's face is not detected from the image (No in S11), the process proceeds to S21. On the other hand, if a person's face is detected from the image (Yes in S11), the registration unit 12 recognizes the customer 1 included in the image by a customer recognition process including a matching process based on the feature amount extracted from the detected person's face and the feature amount of the customer 1 stored in the storage unit 14 (see FIG. 8) (S12).

[0051] If the customer identification information of the recognized customer 1 is not registered in the recognition information (see FIG. 10) (No in S13), the registration unit 12 registers the customer identification information in the recognition information (S14). At this time, other customer information may be registered in the recognition information. The other customer information may include the person's payment method information. The payment method information may be stored in the memory unit 14 in advance through a registration process performed by the customer themselves via an app, a website, or the like. The registration unit 12 may then register the payment method information, etc. stored in the memory unit 14 in the recognition information. Alternatively, the registration unit 12 may accept input of payment method information from the customer on the spot via a predetermined input device. The registration unit 12 may then register the payment method information, etc. input on the spot in the recognition information. On the other hand, if the customer identification information of the recognized customer 1 is already registered in the recognition information (see FIG. 10) (Yes in S13), the registration unit 12 does not update the recognition information.

[0052] The registration unit 12 then determines whether the customer 1 is holding an object based on the image (S15). If it does not detect that the customer 1 is holding an object (No in S15), the process proceeds to S19. On the other hand, if it detects that the customer 1 is holding an object (Yes in S15), the registration unit 12 recognizes the product being held by the customer 1 (the product 3 that the customer 1 has taken out of the product placement space 2) by performing a matching process based on the feature amount extracted from the object and the feature amount of the product 3 stored in the memory unit 14 (see FIG. 9) (S16).

[0053] Then, if the product identification information of the recognized product 3 is not registered in the recognition information (see FIG. 10) (No in S17), the registration unit 12 registers the product identification information in the recognition information (S18). At this time, the registration unit 12 may read information linked to the product identification information (unit price, product name, etc.) from the product information (see FIG. 9) and register it in the recognition information. On the other hand, if the product identification information of the recognized product 3 has already been registered in the recognition information (see FIG. 10) (Yes in S17), the registration unit 12 does not update the recognition information.

[0054] In addition, the unit price of a product may be dynamically determined based on at least one of "customer information including at least one of gender, age group, and purchasing history" and "inventory information regarding at least one of the product's inventory and sales at that time."

[0055] For example, the storage unit 14 may store a function that outputs a unit price when a value of a predetermined parameter included in the customer information and / or inventory information is input, or a table showing the correspondence between the value of a predetermined parameter included in the customer information and / or inventory information and the unit price. Information showing product inventory and sales may also be stored in the storage unit 14. The above-mentioned customer information may also be stored in the storage unit 14. The registration unit 12 may then determine a unit price for each customer based on this information.

[0056] The settlement unit 13 then uses image processing to determine whether the customer 1 is within the reference range 6 (S19). If the position of the customer 1 is outside the reference range 6 (No in S19), the settlement unit 13 performs settlement processing based on the recognition information at that time (see FIG. 10) (S20). That is, the settlement unit 13 identifies the payment amount calculated based on the recognition information. If the identified payment amount is not "0," the settlement unit 13 performs payment processing for the identified payment amount using payment method information that has been registered in advance in association with the customer 1. If the identified payment amount is "0," the detection unit 13 does not perform payment processing using the payment method information. Note that if the position of the customer 1 is within the reference range 6 (Yes in S19), the settlement unit 13 does not perform settlement processing. Thereafter, the process returns to S10, and the same processing is repeated.

[0057] Next, the effects of the processing device 10 of this embodiment will be described. The processing device 10 of this embodiment realizes a store system that eliminates payment processing (product registration, payment, etc.) at the checkout counter using a new, unprecedented method. Therefore, even in stores where gates or vehicles cannot be used, eliminating payment processing (product registration, payment, etc.) at the checkout counter improves convenience for users and reduces the workload of the store.

[0058] Furthermore, the processing device 10 of this embodiment can recognize the customer 1 and the product 3, and detect whether the customer 1 has moved outside the reference range 6, based on an image generated by a single camera 5. This reduces the cost burden, such as equipment costs.

[0059] A modification of this embodiment will now be described.

[0060] In the above example, the processing device 10 has been described on the assumption that it is used in a store where products such as snacks and drinks are displayed in a section of an office or the like and employees of the office purchase the products on a self-service basis. However, the use of the processing device 10 is not limited to this. The processing device 10 may also be used in places other than an office.

[0061] Furthermore, the product placement space 2 may be configured with other items such as shelves, boxes, chairs, etc., instead of the tables shown in FIG. 1, as long as the product placement space 2 is capable of placing products thereon.

[0062] An example is shown in Fig. 13. Fig. 12 is a diagram of the frame 4 extracted from Fig. 13. Each of the two components that make up the frame 4 is provided with a camera 5 and lighting.

[0063] The light emitting surface of the lighting fixture extends in one direction and has a light emitting unit and a cover that covers the light emitting unit. The lighting fixture mainly emits light in a direction perpendicular to the extension direction of the light emitting surface. The light emitting unit has a light emitting element such as an LED, and emits light in a direction that is not covered by the cover. When the light emitting element is an LED, multiple LEDs are lined up in the direction in which the lighting fixture extends (up and down in the figure).

[0064] The camera 5 is provided at one end of the linearly extending frame 4 component, and its imaging range is in the direction in which the illumination light is emitted. For example, in the frame 4 component on the left side of Fig. 12, the imaging range of the camera 5 is downward and diagonally downward to the right. In addition, in the frame 4 component on the right side of Fig. 12, the imaging range of the camera 5 is upward and diagonally upward to the left.

[0065] As shown in FIG. 13, the frame 4 is attached to the front frame (or the front of both side walls) of the product shelf that constitutes the product placement space. In this case, one of the components of the frame 4 is attached to one of the front frames with the camera 5 facing downward, and the other of the components of the frame 4 is attached to the other front frame with the camera 5 facing upward. The camera 5 attached to one of the components of the frame 4 captures images above and diagonally upward so that the opening of the product shelf is included in the imaging range. On the other hand, the camera 5 attached to the other component of the frame 4 captures images below and diagonally downward so that the opening of the product shelf is included in the imaging range. By using two cameras 5 in this way, it is possible to capture images of the entire area of the opening of the product shelf.

[0066] The processing device 10 generates the various types of information shown in the above embodiment by processing the images captured by the two cameras 5. For example, one of the cameras 5 captures images above and obliquely above.

[0067] In the example shown in FIG. 1, the camera 5 is installed on the frame 4, but instead of this configuration, the camera 5 may be installed on other objects such as a ceiling, a wall, or a support.

[0068] Furthermore, in the above example, one camera 5 was installed, and the one camera photographed the object to be photographed, including the customer 1, the product 3, the reference range 6, etc. However, multiple cameras 5 may be installed, and the objects to be photographed may be divided among the multiple cameras 5. In this case, the objects to be photographed by each camera 5 are limited (i.e., it is not necessary to photograph all of the objects), and therefore installation of each camera 5 becomes relatively easy.

[0069] In the above example, customer 1 is recognized using facial recognition based on an image. However, customer 1 may also be recognized by means other than image analysis. For example, processing device 10 may recognize customer 1 based on biometric information such as a fingerprint, voiceprint, or iris input via a predetermined input device (fingerprint sensor, microphone, etc.). Alternatively, processing device 10 may acquire customer identification information from the customer's mobile device (IC card, smartphone, smartwatch, tablet terminal, mobile phone, etc.) via a reader or the like. In this example, the image generated by camera 5 does not need to include the face of customer 1. This reduces the number of subjects that camera 5 needs to capture, thereby reducing the number of cameras 5 required and making camera 5 installation easier.

[0070] Furthermore, the settlement unit 13 may perform settlement when the customer 1 leaves the frame of the image from the camera 5. This eliminates the need to store information indicating the reference range 6 in the image in advance in the storage unit 14, thereby reducing the processing burden on the operator.

[0071] Alternatively, the settlement unit 13 may perform settlement in response to detection of the customer 1 performing a predetermined action, such as detection that "customer 1 has moved a predetermined number of steps in a predetermined direction (away from the product placement space)" or detection that "customer 1 has moved in a predetermined direction (away from the product placement space) for a predetermined period of time." In this way, it becomes unnecessary to store information indicating the reference range 6 in the image in the storage unit 14 in advance, thereby reducing the processing burden on the operator.

[0072] <Second embodiment> The area into which a customer 1 considering purchasing a product 3 is expected to move while holding the product 3 is thought to differ for each customer 1. That is, while considering a purchase, some customers 1 may stay still in front of the product placement space 2, while other customers 1 may move to a position relatively far away from the product placement space 2. In consideration of this situation, the processing device 10 of this embodiment has a function to set an appropriate reference range 6 for each customer 1. This will be explained in detail below.

[0073] 14 shows an example of a functional block diagram of the processing device 10 of this embodiment. As shown in the figure, the processing device 10 has an image acquisition unit 11, a registration unit 12, a settlement unit 13, a storage unit 14, a completion notification unit 15, and a reference range correction unit 16.

[0074] The memory unit 14 stores information indicating the reference range 6 of each customer 1 (information indicating the reference range 6 in the image). Then, the settlement unit 13 identifies the reference range 6 of each customer 1 based on the information indicating the reference range 6 of each customer 1, and detects when each customer 1 moves outside the reference range 6.

[0075] After the payment is made by the payment unit 13, the completion notification unit 15 sends a payment completion notification to the terminal of the customer 1 and receives a response to the payment completion notification from the terminal device of the customer 1. The payment completion notification may be realized, for example, via a predetermined application installed on the terminal device of the customer 1, or may be realized by other means. Figure 15 shows an example of a payment completion notification displayed on the terminal device 7 of the customer 1. The payment completion notification shown in the figure includes the name of the product 3 to be paid for, the price, the payment date, etc.

[0076] Although not shown, the terminal device 7 may accept, as a response to the payment completion notification screen, an input indicating that there is no problem and an input indicating that the product 3 was not purchased. Then, the terminal device 7 may transmit the response to the processing device 10.

[0077] If the response to the settlement completion notice indicates that the product 3 indicated in the settlement completion notice has not been purchased, the reference range correction unit 16 corrects the reference range 6 of the customer 1 in a wider direction, and stores information indicating the corrected content in the memory unit 14.

[0078] Here, an example of an algorithm for processing to correct the reference range 6 for each customer 1 will be described. For example, the reference range correction unit 16 may identify the movement trajectory of the customer 1 (the customer 1 who "input that he did not purchase the product 3") while holding the product 3 (the product 3 for which an input was made that he did not purchase the product 3) based on an image generated by the camera 5. Then, the reference range correction unit 16 may determine the reference range 6 for the customer 1 based on the identified movement trajectory. For example, the reference range correction unit 16 may determine the reference range 6 for the customer so as to include the identified movement trajectory. Note that the reference range 6 for each customer 1 may be determined by machine learning using the identified movement trajectory as learning data.

[0079] Other configurations of the processing apparatus 10 of this embodiment are the same as those of the first embodiment.

[0080] As described above, the processing device 10 of this embodiment achieves the same effects as those of the first embodiment. Furthermore, the processing device 10 of this embodiment sets an appropriate reference range 6 for each customer 1. This reduces the inconvenience of mistakenly performing payment processing before deciding to purchase the product 3 that the customer 1 is holding.

[0081] <Third embodiment> In this embodiment, the settlement unit 13 monitors the time elapsed since the product identification information was linked to the customer identification information, as long as it has not detected that the product 3 has been returned to the product placement space 2 or that the customer 1 has moved out of the reference range 6. If the elapsed time exceeds a threshold, the settlement unit 13 performs settlement based on the details registered by the registration unit 12, without detecting that the customer 1 has moved out of the reference range 6.

[0082] Other configurations of the processing apparatus 10 of this embodiment are the same as those of the first and second embodiments.

[0083] As described above, the processing device 10 of this embodiment achieves the same effects as those of the first and second embodiments. Furthermore, the processing device 10 of this embodiment can perform payment processing after a predetermined time has elapsed, even if it is not detected that a customer 1 holding a product 3 (a customer 1 whose product identification information is registered in association with customer identification information in the recognition information (see FIG. 10)) has moved outside the reference range 6 due to some malfunction. As a result, payment omissions can be reduced.

[0084] <Fourth embodiment> The payment unit 13 of this embodiment detects changes in the body orientation of the customer 1 through image processing and performs payment processing based on the detection results. For example, if the body orientation of the customer 1 holding the product 3 (the customer 1 whose product identification information is registered in association with the customer identification information in the recognition information (see FIG. 10 )) satisfies a predetermined condition (facing the product placement space 2 by 180°), the payment unit 13 may perform payment based on the details registered by the registration unit 12 without detecting that the customer 1 has moved out of the reference range 6.

[0085] It is considered that the customer 1 who has decided to purchase the product 3 that he / she is holding will turn his / her body and move away from the product placement space 2. On the other hand, it is considered that the customer 1 who is considering whether to purchase the product 3 that he / she is holding will generally remain facing the product placement space 2 and be located near the product placement space 2. Therefore, as in this example, the payment unit 13 may perform payment when the orientation of the customer 1's body satisfies a predetermined condition.

[0086] In addition, the payment unit 13 may perform payment based on the registered details by the registration unit 12 when the body orientation of the customer 1 holding the product 3 (customer 1 whose product identification information is registered in association with customer identification information in the recognition information (see Figure 10)) satisfies a predetermined condition (facing 180 degrees in the opposite direction from facing the product placement space 2) and when it is detected that the customer 1 has moved outside the reference range 6.

[0087] As described above, a customer 1 considering whether to purchase the product 3 they are holding is likely to be positioned near the product placement space 2 while generally facing the product placement space 2. However, some customers 1 may move outside the reference range 6 while still facing the product placement space 2 during their consideration. If payment is made simply on the condition that the customer 1 has moved outside the reference range 6, this could result in the inconvenience of payment being made while the customer is considering purchasing a product. Therefore, the payment unit 13 may make payment when the orientation of the customer 1's body satisfies a predetermined condition (facing the product placement space 2 and facing 180 degrees in the opposite direction) and when it is detected that the customer 1 has moved outside the reference range 6. This can prevent the above-mentioned inconvenience from occurring.

[0088] Other configurations of the processing apparatus 10 of this embodiment are the same as those of the first to third embodiments.

[0089] As described above, the processing device 10 of this embodiment achieves the same effects as those of the first to third embodiments. Furthermore, the processing device 10 of this embodiment can perform settlement at an appropriate timing.

[0090] <Fifth embodiment> The settlement unit 13 of this embodiment makes a settlement in response to the detection that customer 1 has moved outside the reference range 6, but the settlement is not triggered by the detection that customer 1 has moved outside the reference range 6, but by the detection that a predetermined process has been performed, which is triggered by the detection that customer 1 has moved outside the reference range 6.

[0091] Specifically, the settlement unit 13 is triggered by detecting that the customer 1 has moved outside the reference range 6, and sends information indicating the settlement details determined based on the registration details by the registration unit 12 to the terminal device 7 of the customer 1, and receives a response from the terminal device 7 of the customer 1 as to whether the settlement details are correct or not.

[0092] The information indicating the payment details includes the name of the product 3 to be paid for, the price, etc. The transmission of the information indicating the payment details to the terminal device 7 of the customer 1 and the reception of the response from the terminal device 7 of the customer 1 may be realized, for example, via a predetermined application installed on the terminal of the customer 1, or may be realized by other means.

[0093] Then, the settlement unit 13 is triggered by receiving a response indicating that the settlement details are correct, and settles the amount indicated in the settlement details.

[0094] Other configurations of the processing apparatus 10 of this embodiment are the same as those of the first to third embodiments.

[0095] As described above, the processing device 10 of this embodiment achieves the same effects as those of the first to third embodiments. Furthermore, the processing device 10 of this embodiment performs a process of confirming with the customer 1 before settlement, thereby reducing the inconvenience of making a settlement with incorrect details.

[0096] Although the present invention has been described above with reference to the embodiments (and examples), the present invention is not limited to the above-described embodiments (and examples). Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0097] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. 1. A registration means for recognizing product identification information of a product removed from a product placement space by image processing, and registering the recognized product identification information in association with customer identification information; a payment means for performing payment based on the registered details when it is detected by image processing that the customer who has taken out the product has performed a predetermined action; A processing device having: 2. The processing device described in 1, wherein the registration means recognizes the customer identification information of the customer contained in the image and registers the recognized customer identification information by linking it to the recognized product identification information. 3. A processing device according to 1 or 2, in which the registration means and the payment means process images generated by the same camera to recognize the product identification information and detect that the customer has performed the specified action. 4. The payment method is: In response to detecting that the customer has performed the predetermined action, transmitting information indicating the settlement details determined based on the registered details to the customer's terminal device; receiving a response from the customer's terminal device indicating whether the payment details are correct; 4. The processing device according to any one of 1 to 3, which, upon receiving a response indicating that the payment details are correct, settles the amount indicated in the payment details. 5. A processing device described in any one of 1 to 4, wherein the payment means performs payment based on the registered details without detecting that the customer has performed a specified action if the elapsed time since the product identification information was linked to the customer identification information exceeds a threshold. 6. A processing device according to any one of 1 to 5, wherein the registration means registers the product identification information by linking it to the customer identification information identified by a face image or iris. 7. A processing device described in any one of 1 to 6, wherein the registration means detects through image processing that the customer has returned the product to the product placement space, and deletes the product identification information of the product returned to the product placement space from information linking the product identification information and the customer identification information. 8. A processing device according to any one of 1 to 7, wherein the payment means detects, by image processing, that the orientation of the customer's body satisfies a predetermined condition, that the customer has moved a predetermined number of steps in a predetermined direction, or that the customer has moved in a predetermined direction for a predetermined period of time as the predetermined action. 9. A processing device described in any one of 1 to 8, wherein the payment means detects, as the specified action, that the customer has moved outside a reference range defined based on the product placement space. 10. The processing device described in 9, wherein the reference range is an area in front of the product placement space. 11. The settlement means identifies the reference range for each of the customers based on the information indicating the reference range for each of the customers, completion notification means for transmitting a payment completion notification to the customer's terminal device after the payment is made by the payment means and receiving a response to the payment completion notification from the customer's terminal device; a criterion range correcting means for widening the criterion range of the customer when the reply to the payment completion notice indicates that the customer has not purchased the product indicated in the payment completion notice; 11. The processing device according to claim 9 or 10, further comprising: 12. The computer Recognizing product identification information of a product removed from the product placement space by image processing, and registering the recognized product identification information in association with customer identification information; A processing method in which, when it is detected by image processing that the customer who has taken out the product has performed a predetermined action, payment is made based on the registered content of the product identification information. 13. Computer, a registration means for recognizing, by image processing, the commodity identification information of a commodity removed from the commodity placement space, and registering the recognized commodity identification information in association with customer identification information; a payment means for performing payment based on the registered details when it is detected by image processing that the customer who has taken out the product has performed a predetermined action; A program that functions as a

Claims

1. a recognition means for recognizing customer identification information of a customer by processing an image of the customer removing the product from the product placement space; a registration means for registering the customer identification information recognized by the recognition means and commodity identification information of the commodity removed from the commodity placement space by the customer in association with each other; a payment means for performing payment based on the registered details when it is detected by image processing that the customer who has taken out the product has performed a predetermined action; a completion notification means for sending a payment completion notification to the customer's terminal after the payment is made by the payment means; and the predetermined action is that the customer moves out of a reference range defined based on the product placement space, The processing device is characterized in that the payment completion notice includes at least the product name and price of the item to be paid.

2. The processing device according to claim 1 , wherein the reference range is an area in front of the product placement space.

3. The processing device according to claim 1 or 2, wherein the registration means registers the commodity identification information in association with the customer identification information specified by a face image or an iris.

4. The processing device described in any one of claims 1 to 3, wherein the registration means detects through image processing that the customer has returned the product to the product placement space, and deletes the product identification information of the product returned to the product placement space from information linking the product identification information and the customer identification information.

5. One or more computers a recognition step of recognizing customer identification information of a customer by processing an image of the customer removing the product from the product placement space; a registration step of linking and registering the customer identification information recognized in the recognition step with commodity identification information of the commodity removed from the commodity placement space by the customer; a payment step of performing payment based on the registered details when it is detected by image processing that the customer who has taken out the product has performed a predetermined action; a completion notification step of sending a payment completion notification to the customer's terminal after the payment is made by the payment step; Run the predetermined action is that the customer moves out of a reference range defined based on the product placement space, The processing method is characterized in that the payment completion notice includes at least the product name and price of the item to be paid.

6. Computer, a recognition means for recognizing customer identification information of a customer by processing an image of the customer removing the product from the product placement space; a registration means for registering the customer identification information recognized by the recognition means in association with commodity identification information of the commodity removed by the customer from the commodity placement space; a payment means for performing payment based on the registered details when it is detected by image processing that the customer who has taken out the product has performed a predetermined action; a completion notification means for sending a payment completion notification to the customer's terminal after the payment is made by the payment means; It functions as the predetermined action is that the customer moves out of a reference range defined based on the product placement space, The program is characterized in that the payment completion notice includes at least the name and price of the product to be paid.

Images