Processing device, processing method, and program

Abstract

To achieve a shop system that eliminates settlement processing (such as commodity registration and payment) at a register counter.SOLUTION: A processing device of the present invention includes: a registration unit that recognizes commodity identification information on a commodity taken out from a commodity placement space by image processing, and registers the recognized commodity identification information in association with customer identification information; a processing unit that when it is detected by image processing that a customer having taken out the commodity performs a predetermined operation, performs predetermined processing based on content registered by the registration unit; and a notification unit that prompts confirmation of the predetermined processing based on the content registered by the registration unit. The notification unit transmits, to a terminal device of the customer, a notification that prompts confirmation of the predetermined processing based on the registered content when a time period elapsed from association of the commodity identification information with the customer identification information exceeds a threshold value.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a processing device, a processing method, and a program.

Background Art

[0002] Non-Patent Document 1, Patent Documents 1 and 2 disclose a store system that eliminates settlement processing (such as product registration and payment) at a cash register.

[0003] Specifically, in the store system disclosed in Non-Patent Document 1, a gate is installed at the entrance and exit of the store, and based on the information read by a reader installed at the gate, a customer who enters the store is recognized. Then, the store system recognizes the products taken by the customer through image processing or the like, and performs settlement at the timing when the customer exits the store through the gate.

[0004] In the purchase product settlement support system disclosed in Patent Document 1, a gate is installed, the entry and exit of customers passing through the gate are managed, and after the customers pass through the gate and leave the store, settlement processing for the products taken out of the store by the customers is performed.

[0005] In the vehicle settlement system disclosed in Patent Document 2, a product placement section is provided in the vehicle interior of the vehicle, and when a predetermined condition is satisfied after the movement of the product to the outside of the vehicle is detected, settlement processing for the product is performed.

[0006] Non-Patent Document 2 discloses a store that displays products such as snacks and drinks in a part of an office, installs settlement means such as a cashless settlement terminal and a cash collection box, and allows employees of the office to purchase products through self-service.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Patent Document 2

Non-Patent Document

[0008]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0009] By eliminating the settlement process at the cashier counter using the technologies disclosed in Non-Patent Document 1, Patent Document 1, and Patent Document 2, the convenience for users is improved, and the labor of the store can also be saved. However, the technologies disclosed in Non-Patent Document 1 and Patent Document 1 are technologies that require the presence of gates and cannot be used in environments where gates cannot be installed. In addition, the technology disclosed in Patent Document 1 is a technology that requires the presence of vehicles and cannot be used in environments where vehicles cannot be utilized. For example, in a store installed in a corner of an office as disclosed in Non-Patent Document 2, due to cost and space issues, gates and vehicles may not be available for use.

[0010] An object of the present invention is to realize a store system that eliminates settlement processing (such as product registration and payment) at a cash register by a novel method not found in the prior art.

Means for Solving the Problems

[0011] According to the present invention, registration means for recognizing product identification information of a product taken out from a product placement space by image processing and registering the recognized product identification information in association with customer identification information; settlement means for performing settlement based on the registration content by the registration means when it is detected that a customer who has taken out the product by image processing has performed a predetermined operation; notification means for prompting confirmation of settlement based on the registration content by the registration means; and the notification means is provided with a processing device that transmits a notification prompting confirmation of settlement based on the registration content to a customer's terminal device when the elapsed time since the product identification information was associated with the customer identification information exceeds a threshold value.

[0012] Also, according to the present invention, a computer performs a registration step of recognizing product identification information of a product taken out from a product placement space by image processing and registering the recognized product identification information in association with customer identification information; a settlement step of performing settlement based on the registration content by the registration step when it is detected that a customer who has taken out the product by image processing has performed a predetermined operation; a notification step of prompting confirmation of settlement based on the registration content by the registration step; and in the notification step, when the elapsed time since the product identification information was associated with the customer identification information exceeds a threshold value, a processing method is provided for transmitting a notification prompting confirmation of settlement based on the registration content to a customer's terminal device.

[0013] Also, according to the present invention, a computer Registration means for recognizing product identification information of a product taken out from a product placement space by image processing and registering the recognized product identification information by associating it with customer identification information. Settlement means for performing settlement based on the registration content by the registration means when it is detected that a customer who has taken out the product by image processing has performed a predetermined operation. Notification means for prompting confirmation of settlement based on the registration content by the registration means. Function as The notification means is provided with a program for transmitting a notification prompting confirmation of settlement based on the registration content to the customer's terminal device when the elapsed time since the product identification information was associated with the customer identification information exceeds a threshold value.

Effect of the Invention

[0014] According to the present invention, a store system that eliminates settlement processing (product registration, payment, etc.) at a cash register is realized by a novel method that has never existed before.

Brief Description of the Drawings

[0015] The above-described object, as well as other objects, features, and advantages, will become even more apparent from the following preferred embodiments described below and the accompanying drawings.

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Embodiment for Carrying Out the Invention

[0017] <First Embodiment> First, the overall image and outline of the processing device of the present embodiment will be described with reference to FIGS. 1 to 5. The processing device of the present embodiment is used, for example, in a store where products such as snacks and drinks are displayed in a section of an office, and employees of the office purchase products through self-service.

[0018] As shown in FIG. 1, a product placement space 2 for placing product 3 is provided in a section of an office or the like. A camera 5 installed on the frame 4 photographs a customer 1 who takes out product 3 from the product placement space 2.

[0019] As shown in FIG. 2, the image generated by the camera 5 includes the customer 1 located in front of the product placement space 2. The processing device analyzes the image and recognizes the customer 1 included in the image.

[0020] Also, as shown in FIG. 3, the image generated by the camera 5 includes the product 3 held by the customer 1 located in front of the product placement space 2. The processing device analyzes the image and recognizes the product 3 held by the customer 1.

[0021] Then, when the processing device analyzes the image and detects that the customer 1 holding the product 3 has performed a predetermined operation, it performs a settlement process based on the recognition result in response to the detection. In the present embodiment, the "predetermined operation" will be described as "the customer 1 has gone out of the reference range 6". More specifically, when the processing device detects that the customer 1 holding the product 3 has gone out of the reference range 6 (see FIGS. 4 and 5), it performs a settlement process based on the recognition result in response to the detection. In the present embodiment, the reference range 6 is defined based on the product placement space 2. For example, a predetermined area in front of the product placement space 2 (an area where it is considered that the customer 1 considering purchasing the product 3 moves while holding the product 3) is defined as the reference range 6.

[0022] According to the processing device of such a present embodiment, a store system that eliminates settlement processes (such as product registration and payment) at the cash register is realized by a novel method that has never existed before.

[0023] Next, the configuration of the processing device will be described in detail. First, an example of the hardware configuration of the processing device will be described.

[0024] Each functional part of the processing device is realized by an arbitrary combination of hardware and software centered around the CPU (Central Processing Unit) of an arbitrary computer, memory, program loaded into the memory, storage unit such as a hard disk for storing the program (in addition to the program stored in advance at the stage of shipping the device, programs downloaded from storage media such as CD (Compact Disc) and servers on the Internet can also be stored), and network connection interface. And it is understood by those skilled in the art that there are various modification examples for the realization method and device.

[0025] FIG. 6 is a block diagram illustrating the hardware configuration of a processing device. As shown in FIG. 6, the processing device includes a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The processing device may not have the peripheral circuit 4A. Note that the processing device may be composed of a plurality of physically and / or logically divided devices, or may be composed of one physically and / or logically integrated device. When the processing device is composed of a plurality of physically and / or logically divided devices, each of the plurality of devices can include the above hardware configuration.

[0026] The bus 5A is a data transmission path for the processor 1A, the memory 2A, the peripheral circuit 4A, and the input / output interface 3A to transmit and receive data to and from each other. The processor 1A is an arithmetic processing device such as a CPU or a GPU (Graphics Processing Unit). The memory 2A is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The input / output interface 3A includes an interface for acquiring information from an input device, an external device, an external server, an external sensor, a camera, etc., and an interface for outputting information to an output device, an external device, an external server, etc. The input device is, for example, a keyboard, a mouse, a microphone, a physical button, a touch panel, etc. The output device is, for example, a display, a speaker, a printer, a mailer, etc. The processor 1A can issue commands to each module and perform operations based on their operation results.

[0027] FIG. 7 shows an example of a functional block diagram of a processing device 10. As shown in the figure, the processing device 10 includes an image acquisition unit 11, a registration unit 12, a settlement unit 13, and a storage unit 14. Note that the processing device 10 may not include the storage unit 14. In this case, an external device configured to be communicable with the processing device 10 includes the storage unit 14.

[0028] The storage unit 14 stores various types of information. For example, the storage unit 14 stores customer information as shown in FIG. 8 and product information as shown in FIG. 9.

[0029] The customer information is information regarding customer 1 who uses a store provided in a section such as an office. Through a prior registration operation by customer 1 who uses the store, the customer information of each customer 1 is stored in the storage unit 14.

[0030] In the illustrated customer information, customer identification information for identifying a plurality of customers from each other, name, contact information, face image, feature amount extracted from the face image, and payment means information are linked to each other. The payment means information is information for performing payment at the store. For example, credit card information, account information, information indicating the desire for direct debit from salary and information for realizing the process, user identification information of a payment service (QR code payment service, electronic money service, point service, etc.), etc. are exemplified, but are not limited thereto. Note that the illustrated customer information is merely an example, and may include information of other items, or may not include some of the illustrated items.

[0031] The product information is information regarding product 3 handled at the store. In the illustrated product information, product identification information for identifying a plurality of products from each other, product name, unit price, image, and feature amount extracted from the image are linked to each other. Note that the image of each product 3 may include a plurality of images taken of product 3 from a plurality of directions.

[0032] The image acquisition unit 11 acquires an image generated by a camera 5 that photographs an area in front of the product placement space 2 (the side where product 3 is taken out). The processing device 10 and the camera 5 are connected by wire and / or wirelessly and are configured to be able to communicate with each other. The image acquisition unit 11 acquires the image generated by the camera 5 in real-time processing.

[0033] The camera 5 is installed at a position and in an orientation that generates an image including "the face of the customer 1 located in front of the product placement space 2" and "the product 3 held by the customer 1". Note that, in addition to the above-described imaging target, the camera 5 may be installed at a position and in an orientation that generates an image including at least one of "the boundary of the reference range 6" and "the product 3 placed in the product placement space 2".

[0034] In this embodiment, one camera 5 generates an image including the above-described imaging target. The camera 5 may be equipped with a fish-eye lens or a standard lens (for example, an angle of view of around 40° to around 60°).

[0035] The camera 5 generates a moving image. The camera 5 may perform continuous shooting or may perform shooting only for a predetermined period (for example, while a human presence sensor detects a person in front of the product placement space 2).

[0036] The registration unit 12 recognizes the customer identification information of the customer 1 included in the image. For example, when the registration unit 12 detects a person's face from the image, it may recognize the customer 1 included in the image by performing a collation process based on the feature amount extracted from the detected person's face and the feature amount of the customer 1 stored in the storage unit 14 (see FIG. 8).

[0037] In addition, the registration unit 12 recognizes the product identification information of the product 3 taken out from the product placement space 2 by image processing. For example, when the registration unit 12 detects that the recognized customer 1 is holding an object, it may recognize the product 3 held by the customer 1 (the product 3 taken out from the product placement space 2) by performing a collation process based on the feature amount extracted from the object and the feature amount of the product 3 stored in the storage unit 14 (see FIG. 9).

[0038] Then, the registration unit 12 causes the storage unit 14 to store the recognition result. FIG. 10 schematically shows an example of the information (recognition information) stored by the registration unit 12 in the storage unit 14. As shown in the figure, the registration unit 12 registers the recognized customer identification information in the recognition information. Then, the registration unit 12 registers the recognized product identification information in the recognition information by associating it with the recognized customer identification information. Note that the registration unit 12 may further read out the unit price and the like associated with the recognized product identification information from the product information (see FIG. 9) and register them in the recognition information.

[0039] In addition, the registration unit 12 detects, by image processing, that the product 3 held by the customer 1 has been returned to the product placement space 2. Then, when the registration unit 12 detects that the product 3 held by the customer 1 has been returned to the product placement space 2, it updates the recognition information (see FIG. 10) accordingly. Specifically, the registration unit 12 deletes the product identification information and the like of the product 3 returned to the product placement space 2 from the recognition information.

[0040] Here, an example of an algorithm for detecting that the product 3 held by the customer 1 has been returned to the product placement space 2 will be described. For example, when the registration unit 12 detects a state change from the state where the customer 1 is holding the product 3 to the state where the customer 1 is not holding the product 3, it may be determined that the product 3 held by the customer 1 has been returned to the product placement space 2. However, in this algorithm, for example, when the customer 1 stores the product 3 held by him / her in his / her bag, it may be erroneously detected that the product 3 has been returned to the product placement space 2. Therefore, in order to avoid such an erroneous detection, the detection may be performed by the following algorithm.

[0041] For example, when the registration unit 12 detects a state change from the state where the customer 1 is holding the product 3 to the state where the customer 1 is not holding the product 3, it may determine whether the movement trajectory of the product 3 in the image before the state change satisfies a predetermined condition. And when the predetermined condition is satisfied, it may be determined that the product 3 held by the customer 1 has been returned to the product placement space 2.

[0042] The predetermined conditions for the movement trajectory include "movement towards the product placement space 2", "movement towards the product placement space 2 and reaching a predetermined position within the image", "movement reaching a predetermined position within the image", etc. The predetermined position within the image is the product placement space 2 when the product placement space 2 is included in the image, and is a position where the product 3 moving towards the product placement space 2 is about to be framed out just before when the product placement space 2 is not included in the image, etc.

[0043] By determining such predetermined conditions for the movement trajectory, it is possible to suppress the inconvenience of erroneously updating the recognition information due to the action of the customer 1 putting the product 3 into a bag or a sack, the action of the customer 1 eating the product 3, the event that the product 3 becomes a blind spot of the camera 5 due to the way of holding the product 3, etc.

[0044] Returning to FIG. 7, the settlement unit 13 performs settlement based on the registration content by the registration unit 12 in response to detecting by image processing that the customer 1 who has taken out the product 3 from the product placement space 2 and has not returned the product 3 to the product placement space 2 has moved outside the reference range 6 defined based on the product placement space 2. More specifically, the settlement unit 13 performs settlement based on the registration content by the registration unit 12 in response to detecting by image processing that the customer 1 in whom at least one product identification information is registered in the recognition information (see FIG. 10) has moved outside the reference range 6.

[0045] The reference range 6 is an area in front of the product placement space 2 (the side where the product 3 is taken out), an area where it is considered that the customer 1 considering purchasing the product 3 moves while holding the product 3. The size and shape of the reference range 6 are matters of design. Information indicating the reference range 6 in the image may be stored in the storage unit 14 in advance.

[0046] For example, when the camera 5 generates an image including the boundary of the reference range 6, the settlement unit 13 may monitor, by image processing, whether a part of the body of the customer 1 is within the reference range 6. For example, the settlement unit 13 may determine that the customer 1 has moved outside the reference range 6 in response to detecting that the foot of the customer 1 has extended outside the reference range 6. Note that the settlement unit 13 monitors whether a part of the customer 1's body other than the foot is within a predetermined area in the image, and may determine that the customer 1 has moved outside the reference range 6 in response to detecting that the part of the customer 1's body has extended outside the predetermined area.

[0047] The settlement unit 13 performs settlement of a predetermined amount based on the settlement means information (see FIG. 8) registered in advance for each customer 1.

[0048] Note that in the case of this embodiment in which one camera 5 is installed, the registration unit 12 and the settlement unit 13 process the image generated by the same camera 5 to recognize the customer 1 and the product 3, and to detect that the customer 1 has moved outside the reference range 6.

[0049] Next, an example of the processing flow of the processing device 10 will be described using the flowchart of FIG. 11.

[0050] When the image acquisition unit 11 acquires an image (S10), the registration unit 12 performs processing to detect a person's face from the image (S11). If a person's face is not detected from the image (No in S11), the process proceeds to S21. On the other hand, when a person's face is detected from the image (Yes in S11), the registration unit 12 recognizes the customer 1 included in the image by customer recognition processing including collation processing based on the feature amount extracted from the detected person's face and the feature amount of the customer 1 stored in the storage unit 14 (see FIG. 8) (S12).

[0051] Then, when the customer identification information of the recognized Customer 1 is not registered in the recognition information (see FIG. 10) (No in S13), the registration unit 12 registers the customer identification information in the recognition information (S14). At this time, other customer information may also be registered in the recognition information. Other customer information is, for example, the payment means information of that person. The payment means information may be stored in the storage unit 14 by a registration process by the customer himself / herself in advance via an app, a homepage, or the like. And the registration unit 12 may register the payment means information and the like stored in the storage unit 14 in the recognition information. Alternatively, the registration unit 12 may receive an input of the payment means information from the customer via a predetermined input device on the spot. And the registration unit 12 may register the payment means information and the like input on the spot in the recognition information. On the other hand, when the customer identification information of the recognized Customer 1 is already registered in the recognition information (see FIG. 10) (Yes in S13), the registration unit 12 does not update the recognition information.

[0052] After that, the registration unit 12 determines, based on the image, whether Customer 1 is holding an object (S15). When it is not detected that Customer 1 is holding an object (No in S15), the process proceeds to S19. On the other hand, when it is detected that Customer 1 is holding an object (Yes in S15), the registration unit 12 recognizes the product held by Customer 1 (Product 3 taken out by Customer 1 from the product placement space 2) by a collation process based on the feature amount extracted from the object and the feature amount of Product 3 stored in the storage unit 14 (see FIG. 9) (S16).

[0053] Then, when the product identification information of the recognized Product 3 is not registered in the recognition information (see FIG. 10) (No in S17), the registration unit 12 registers the product identification information in the recognition information (S18). At this time, the registration unit 12 may read out the information (unit price, product name, etc.) associated with the product identification information from the product information (see FIG. 9) and register it in the recognition information. On the other hand, when the product identification information of the recognized Product 3 is already registered in the recognition information (see FIG. 10) (Yes in S17), the registration unit 12 does not update the recognition information.

[0054] Note that the unit price of the product may be dynamically determined based on at least one of "customer information including at least one of gender, age group, and purchase history" and "inventory information related to at least one of the inventory and sales of the product at that time".

[0055] For example, a function that outputs the unit price when the values of predetermined parameters included in the customer information and / or inventory information are input, a table showing the correspondence between the values of the predetermined parameters included in the customer information and / or inventory information and the unit price, etc. may be stored in the storage unit 14. Also, information indicating the inventory and sales of the product may be stored in the storage unit 14. Further, customer information as described above may be stored in the storage unit 14. And the registration unit 12 may determine the unit price for each customer based on this information.

[0056] Then, the settlement unit 13 determines whether the customer 1 is within the reference range 6 by image processing (S19). If the position of the customer 1 is outside the reference range 6 (No in S19), the settlement unit 13 performs settlement processing based on the recognition information at that time (see FIG. 10) (S20). That is, the settlement unit 13 identifies the payment amount calculated based on the recognition information. And when the identified payment amount is not "0", the settlement unit 13 performs payment processing for the identified payment amount using the payment means information registered in advance in association with the customer 1. When the identified payment amount is "0", the detection unit 13 does not perform payment processing using the payment means information. Note that when the position of the customer 1 is within the reference range 6 (Yes in S19), the settlement unit 13 does not execute settlement processing. Thereafter, it returns to S10 and the same processing is repeated.

[0057] Next, the operation and effect of the processing device 10 of the present embodiment will be described. According to the processing device 10 of the present embodiment, a store system that eliminates settlement processing (such as product registration and payment) at the cash register is realized by a novel method that has not existed in the past. For this reason, even in a store where gates and vehicles cannot be used, the convenience of users is improved by eliminating settlement processing (such as product registration and payment) at the cash register, and the labor of the store is omitted.

[0058] Moreover, according to the processing device 10 of the present embodiment, based on the images generated by one camera 5, it is possible to recognize the customer 1 and the product 3, detect that the customer 1 has moved outside the reference range 6, etc. Therefore, the cost burden such as equipment costs is reduced.

[0059] Here, a modified example of the present embodiment will be described.

[0060] In the above example, the processing device 10 was described on the premise that it is used in a store where products such as snacks and drinks are displayed in a corner of an office or the like, and the employees in the office or the like purchase the products through self-service. However, the usage scenario of the processing device 10 is not limited to this. The processing device 10 may be used in places other than the office.

[0061] Moreover, the product placement space 2 only needs to be able to place products. Instead of the table shown in FIG. 1, the product placement space 2 may be constituted by other articles such as a shelf, a box, and a chair.

[0062] An example is shown in FIG. 13. FIG. 12 is a diagram in which the frame 4 of FIG. 13 is extracted. A camera 5 and lighting are provided on each of the two components constituting the frame 4.

[0063] The light emission surface of the lighting extends in one direction and has a light emitting part and a cover covering the light emitting part. The lighting mainly emits light in a direction orthogonal to the extending direction of the light emission surface. The light emitting part has a light emitting element such as an LED and emits light in a direction not covered by the cover. When the light emitting element is an LED, a plurality of LEDs are arranged in the direction in which the lighting extends (the vertical direction in the figure).

[0064] And the camera 5 is provided on one end side of the component of the frame 4 extending linearly, and the direction in which the light of the lighting is emitted is set as the imaging range. For example, in the component of the left frame 4 in FIG. 12, the camera 5 has the downward and diagonally downward right directions as the imaging range. Also, in the component of the right frame 4 in FIG. 12, the camera 5 has the upward and diagonally upward left directions as the imaging range.

[0065] As shown in FIG. 13, the frame 4 is attached to the front frame of the product shelf (or the front surfaces of the side walls on both sides) that constitutes the product placement space. At this time, one of the components of the frame 4 is attached to one front frame in a direction where the camera 5 is positioned downward, and the other component of the frame 4 is attached to the other front frame in a direction where the camera 5 is positioned upward. Then, the camera 5 attached to one of the components of the frame 4 images upward and obliquely upward so as to include the opening of the product shelf in the imaging range. On the other hand, the camera 5 attached to the other component of the frame 4 images downward and obliquely downward so as to include the opening of the product shelf in the imaging range. By using two cameras 5 in this way, the entire range of the opening of the product shelf can be photographed.

[0066] Then, the processing device 10 generates various types of information shown in the above embodiment by processing the images captured by the two cameras 5. For example, one camera 5 images upward and obliquely upward.

[0067] Also, in the example shown in FIG. 1, the camera 5 is installed on the frame 4. However, instead of this configuration, the camera 5 may be installed on other objects such as the ceiling, wall, or pillar.

[0068] Also, in the above example, one camera 5 is installed, and one camera photographs the imaging objects including the customer 1, the product 3, the reference range 6, etc. However, a plurality of cameras 5 may be installed, and the imaging objects may be photographed separately by the plurality of cameras 5. In this case, since the imaging objects to be photographed by each camera 5 are limited (that is, it is not necessary to photograph all the imaging objects), the installation of each camera 5 and the like becomes relatively easy.

[0069] Also, in the above example, Customer 1 was recognized by face authentication based on an image. However, Customer 1 may also be recognized by means other than image analysis. For example, the processing device 10 may recognize Customer 1 based on biometric information such as fingerprints, voiceprints, and irises input via a predetermined input device (fingerprint sensor, microphone, etc.). Additionally, the processing device 10 may obtain customer identification information from the customer's mobile device (IC card, smartphone, smartwatch, tablet terminal, mobile phone, etc.) via a reader or the like. In the case of this example, the image generated by the camera 5 may not include the face of Customer 1. As a result, the number of objects to be photographed by the camera 5 is reduced, so the number of required cameras 5 may decrease or the installation of the camera 5 may become easier.

[0070] Further, the settlement unit 13 may perform settlement triggered by Customer 1 timing out from the image of the camera 5. In this way, it is not necessary to previously store in the storage unit 14 information indicating the reference range 6 in the image, thus reducing the processing burden on the operator.

[0071] Additionally, the settlement unit 13 may perform settlement in response to detection of Customer 1 performing a predetermined action, such as detection of "Customer 1 moving a predetermined number of steps in a predetermined direction (a direction away from the product placement space)" or detection of "Customer 1 moving for a predetermined time in a predetermined direction (a direction away from the product placement space)". In this way, it is not necessary to previously store in the storage unit 14 information indicating the reference range 6 in the image, thus reducing the processing burden on the operator.

[0072] <Second Embodiment> The area where Customer 1, who is considering purchasing Product 3, is thought to move while holding Product 3 is considered to be different for each Customer 1. That is, among Customers 1 considering a purchase, there may be those who stand still in front of the product placement space 2, and there may also be those who move to a position relatively far from the product placement space 2. In view of this situation, the processing device 10 of the present embodiment has a function of setting an appropriate reference range 6 for each Customer 1. This will be described in detail below.

[0073] FIG. 14 shows an example of a functional block diagram of the processing device 10 of the present embodiment. As shown in the figure, the processing device 10 includes an image acquisition unit 11, a registration unit 12, a settlement unit 13, a storage unit 14, a completion notification unit 15, and a reference range correction unit 16.

[0074] The storage unit 14 stores information indicating the reference range 6 of each customer 1 (information indicating the reference range 6 in the image). Then, based on the information indicating the reference range 6 of each customer 1, the settlement unit 13 identifies the reference range 6 of each customer 1 and detects that each customer 1 has moved outside the reference range 6.

[0075] After the settlement by the settlement unit 13, the completion notification unit 15 transmits a settlement completion notification to the terminal of the customer 1 and receives a response to the settlement completion notification from the terminal device of the customer 1. The settlement completion notification may be realized, for example, via a predetermined application installed on the terminal of the customer 1, or may be realized by other means. FIG. 15 shows an example of the settlement completion notification displayed on the terminal device 7 of the customer 1. The illustrated settlement completion notification includes the name of the product 3 to be settled, the price, the settlement date, etc.

[0076] Although not shown, the terminal device 7 may receive, as a response to this settlement completion notification screen, an input indicating "no problem" and an input indicating "not purchased the product 3" from the customer 1. Then, the terminal device 7 may transmit the response to the processing device 10.

[0077] When the response to the settlement completion notification indicates that the product 3 shown in the settlement completion notification has not been purchased, the reference range correction unit 16 corrects the reference range 6 of the customer 1 in a widening direction and stores information indicating the corrected content in the storage unit 14.

[0078] Here, an example of an algorithm for processing to modify the reference range 6 for each customer 1 will be described. For example, the reference range modification unit 16 may identify the movement trajectory of the customer 1 (the customer 1 who has made an input indicating that they have not purchased the product 3) while holding the product 3 (the product 3 for which an input indicating that it has not been purchased has been made) based on the image generated by the camera 5. Then, the reference range modification unit 16 may determine the reference range 6 of the customer 1 based on the identified movement trajectory. For example, the reference range modification unit 16 may determine the reference range 6 of the customer so as to enclose the identified movement trajectory. Note that the reference range 6 for each customer 1 may be determined by machine learning using the identified movement trajectory as learning data.

[0079] Other configurations of the processing device 10 of the present embodiment are the same as those of the first embodiment.

[0080] As described above, according to the processing device 10 of the present embodiment, the same operational effects as those of the first embodiment are realized. Further, according to the processing device 10 of the present embodiment, an appropriate reference range 6 is set for each customer 1. Therefore, it is possible to reduce the inconvenience of performing the settlement process erroneously before determining the purchase of the product 3 held by the customer 1.

[0081] <Third Embodiment> The settlement unit 13 of the present embodiment monitors the elapsed time since the product identification information was linked to the customer identification information while it has not been detected that the product 3 has been returned to the product placement space 2 or that the customer 1 has moved outside the reference range 6 after the product identification information has been linked to the customer identification information. Then, when the elapsed time exceeds the threshold value, the settlement unit 13 performs settlement based on the registration content by the registration unit 12 without detecting that the customer 1 has moved outside the reference range 6.

[0082] Other configurations of the processing device 10 of the present embodiment are the same as those of the first and second embodiments.

[0083] According to the processing device 10 of the present embodiment described above, the same operational effects as those of the first and second embodiments are achieved. Further, according to the processing device 10 of the present embodiment, even when it is not detected that a customer 1 who has grasped the product 3 (in the recognition information (see FIG. 10), the customer 1 in which product identification information is registered in association with the customer identification information) has moved outside the reference range 6, the settlement process can be performed according to the elapse of a predetermined time. As a result, settlement omission can be suppressed.

[0084] <Fourth Embodiment> The settlement unit 13 of the present embodiment detects a change in the orientation of the body of the customer 1 by image processing, and performs a settlement process based on the detection result. For example, when the orientation of the body of the customer 1 who is grasping the product 3 (in the recognition information (see FIG. 10), the customer 1 in which product identification information is registered in association with the customer identification information) satisfies a predetermined condition (faces the 180° opposite side from the state of facing the product placement space 2 directly), the settlement unit 13 may perform settlement based on the registration content by the registration unit 12 without detecting that the customer 1 has moved outside the reference range 6.

[0085] It is considered that the customer 1 who has decided to purchase the grasped product 3 changes the orientation of the body and moves away from in front of the product placement space 2. On the other hand, it is considered that the customer 1 who is considering purchasing the grasped product 3 is generally located around the product placement space 2 while facing the product placement space 2. Therefore, the settlement unit 13 may perform settlement according to the fact that the orientation of the body of the customer 1 satisfies a predetermined condition as in this example.

[0086] In addition, the settlement unit 13 may perform settlement based on the registration content by the registration unit 12 according to the fact that the orientation of the body of the customer 1 who is grasping the product 3 (in the recognition information (see FIG. 10), the customer 1 in which product identification information is registered in association with the customer identification information) satisfies a predetermined condition (faces the 180° opposite side from the state of facing the product placement space 2 directly) and it is detected that the customer 1 has moved outside the reference range 6.

[0087] As described above, customer 1 who is considering purchasing the held product 3 is considered to be located around the product placement space 2 while generally facing the product placement space 2. However, depending on customer 1, during consideration, while facing the product placement space 2, they may go outside the reference range 6. If settlement is performed simply on the condition that they have moved outside the reference range 6, it may cause the inconvenience of performing settlement while considering purchasing the product. Therefore, the settlement unit 13 may perform settlement in response to the detection that the orientation of customer 1's body satisfies a predetermined condition (facing the side opposite 180° from the state of facing the product placement space 2 directly) and that customer 1 has moved outside the reference range 6. By doing so, the occurrence of the above inconvenience can be suppressed.

[0088] Other configurations of the processing device 10 of the present embodiment are the same as those of the first to third embodiments.

[0089] As described above, according to the processing device 10 of the present embodiment, the same operational effects as those of the first to third embodiments are realized. Further, according to the processing device 10 of the present embodiment, settlement can be performed at an appropriate timing.

[0090] <Fifth Embodiment> The settlement unit 13 of the present embodiment performs settlement in response to the detection that customer 1 has moved outside the reference range 6. However, instead of performing settlement triggered by the detection that customer 1 has moved outside the reference range 6, settlement is performed triggered by the detection that a predetermined process performed triggered by the detection that customer 1 has moved outside the reference range 6 has been performed.

[0091] Specifically, the settlement unit 13 transmits, triggered by the detection that customer 1 has moved outside the reference range 6, information indicating the settlement content determined based on the registration content by the registration unit 12 to the terminal device 7 of customer 1, and receives, from the terminal device 7 of customer 1, an answer as to whether the settlement content is correct.

[0092] The information indicating the settlement details includes the name of the product 3 to be settled, the price, etc. The transmission of the information indicating the settlement details to the terminal device 7 of the customer 1 and the reception of the response from the terminal device 7 of the customer 1 may be realized, for example, via a predetermined application installed on the terminal of the customer 1, or may be realized by other means.

[0093] Then, the settlement unit 13 performs the settlement of the price indicated by the settlement details upon receiving a response indicating that the settlement details are correct.

[0094] Other configurations of the processing device 10 in the present embodiment are the same as those in the first to third embodiments.

[0095] According to the processing device 10 of the present embodiment described above, the same operational effects as those in the first to third embodiments are achieved. Further, according to the processing device 10 of the present embodiment, since the process of confirming with the customer 1 in advance before settlement is performed, the inconvenience of performing settlement with incorrect content can be suppressed.

[0096] Although the present invention has been described with reference to the embodiments (and examples), the present invention is not limited to the above embodiments (and examples). Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0097] Some or all of the above embodiments may be described as follows in the following supplementary notes, but are not limited thereto. 1. Registration means for recognizing product identification information of a product taken out from a product placement space by image processing and registering the recognized product identification information in association with customer identification information; Settlement means for performing settlement based on the registration content by the registration means when it is detected that a customer who has taken out the product by image processing has performed a predetermined operation; A processing device having the above. 2. The processing device according to 1, wherein the registration means recognizes customer identification information of the customer included in the image and registers the recognized product identification information in association with the recognized customer identification information. 3. The registration means and the settlement means process an image generated by the same camera to perform the recognition of the product identification information and the detection of the customer having performed the predetermined operation, according to the processing apparatus of 1 or 2 above. 4. The settlement means in response to the detection of the customer having performed the predetermined operation, transmits information indicating the settlement content determined based on the registration content to the customer's terminal device, receives a response from the customer's terminal device as to whether the settlement content is correct, and performs the settlement of the price indicated by the settlement content in response to receiving a response indicating that the settlement content is correct, according to the processing apparatus of any one of 1 to 3 above. 5. The settlement means performs settlement based on the registration content without detecting that the customer has performed a predetermined operation when the elapsed time since the product identification information was linked to the customer identification information exceeds a threshold value, according to the processing apparatus of any one of 1 to 4 above. 6. The registration means registers by linking the product identification information to the customer identification information specified by a face image or iris, according to the processing apparatus of any one of 1 to 5 above. 7. The registration means detects, by image processing, that the customer has returned the product to the product placement space, and deletes the product identification information of the product returned to the product placement space from the information linking the product identification information and the customer identification information, according to the processing apparatus of any one of 1 to 6 above. 8. The settlement means detects, as the predetermined operation, by image processing that the orientation of the customer's body satisfies a predetermined condition, that the customer has moved a predetermined number of steps in a predetermined direction, or that the customer has moved for a predetermined time in a predetermined direction, according to the processing apparatus of any one of 1 to 7 above. 9. The settlement means detects, as the predetermined operation, that the customer has moved outside a reference range defined based on the product placement space, according to the processing apparatus of any one of 1 to 8 above. 10. The reference range is an area in front of the product placement space, according to the processing apparatus of 9 above. 11. The settlement means identifies the reference range of each customer based on the information indicating the reference range of each customer, after settlement by the settlement means, a completion notification means for transmitting a settlement completion notification to the customer's terminal device and receiving a response to the settlement completion notification from the customer's terminal device; when the response to the settlement completion notification indicates that the product shown in the settlement completion notification has not been purchased, a reference range correction means for widening the reference range of the customer; The processing device according to 9 or 10, further comprising. 12. A computer recognizes the product identification information of the product taken out from the product placement space by image processing, associates the recognized product identification information with the customer identification information and registers it, A processing method for performing settlement based on the registration content of the product identification information when it is detected that a customer who has taken out the product by image processing has performed a predetermined operation. 13. A computer a registration means for recognizing the product identification information of the product taken out from the product placement space by image processing and associating the recognized product identification information with the customer identification information and registering it, a settlement means for performing settlement based on the registration content by the registration means when it is detected that a customer who has taken out the product by image processing has performed a predetermined operation, A program that functions as.

Claims

1. Registration means for recognizing product identification information of a product taken out from a product placement space by image processing and registering the recognized product identification information by associating it with customer identification information; Processing means for performing predetermined processing based on the registration content by the registration means when it is detected that a customer who has taken out the product by image processing has performed a predetermined operation; Notification means for prompting confirmation of the predetermined processing based on the registration content by the registration means; comprising When the elapsed time since the product identification information was associated with the customer identification information exceeds a threshold value, the notification means transmits a notification prompting confirmation of the predetermined processing based on the registration content to the customer's terminal device; The predetermined processing is settlement; The processing means A processing device that performs the predetermined processing when it is detected that the customer has framed out from the image.

2. The registration means The processing device according to claim 1, wherein it is detected that the customer has returned the product to the product placement space based on the movement trajectory of the product in an image before the customer changes from a state of holding the product to a state of not holding the product.

3. The registration means The processing device according to claim 2, wherein it is detected that the customer has returned the product to the product placement space based on at least one of the moving direction indicated by the movement trajectory of the product, the position passed through, and the position where the customer moved into immediately before framing out.

4. The registration means The unit price of the product is determined based on at least one of the gender, age group, purchase history of the customer, the inventory of the product, and the sales performance of the product, and the unit price determined by associating it with the customer identification information and the product identification information is registered. The processing device according to any one of claims 1 to 3.

5. The processing means The processing device according to any one of claims 1 to 4, wherein when it is detected that the customer has moved outside a reference range determined for each customer with respect to the product placement space, the predetermined processing is performed.

6. The processing means The processing device according to claim 5, wherein the reference range for each customer is determined based on the movement trajectory of each customer when the customer has picked up the product but has not purchased it.

7. One or more computers A registration step of recognizing product identification information of a product taken out from a product placement space by image processing and registering the recognized product identification information by associating it with customer identification information; When it is detected that a customer who has taken out the product by image processing has performed a predetermined operation, a processing step of performing a predetermined process based on the registration content in the registration step; A notification step of prompting confirmation of the predetermined process based on the registration content in the registration step; are executed, In the notification step, when the elapsed time since the product identification information was linked to the customer identification information exceeds a threshold value, a notification prompting confirmation of the predetermined process based on the registration content is transmitted to the customer's terminal device. The predetermined process is settlement. In the processing step, When it is detected that the customer has framed out from the image, a processing method for performing the predetermined process.

8. A computer, A registration means for recognizing product identification information of a product taken out from a product placement space by image processing and registering the recognized product identification information linked to customer identification information; A processing means for performing a predetermined process based on the registration content by the registration means when it is detected that a customer who has taken out the product by image processing has performed a predetermined operation; A notification means for prompting confirmation of the predetermined process based on the registration content by the registration means; function as, The notification means transmits a notification prompting confirmation of the predetermined process based on the registration content to the customer's terminal device when the elapsed time since the product identification information was linked to the customer identification information exceeds a threshold value. The predetermined process is settlement. The processing means, When it is detected that the customer has framed out from the image, a program for performing the predetermined process.

Images