Registration confirmation device, control method, and program

Abstract

This registration checking device (2000) determines, with respect to a subject transaction, whether a relation between the number of times scan operations are performed and the number of commodities registered as a settlement target meets a prescribed condition. The scan operations are detected by analyzing a moving image (12). The registration checking device (2000) performs prescribed output processing when the relation does not meet the prescribed condition. Here, the prescribed condition is met when the number of registered commodities is equal to or greater than the number of the detected scan operations.

Description

[Technical field]

[0001] The present invention relates to a technique for registering a product as a payment target. [Background technology]

[0002] When purchasing an item at a store, the item is registered as a payment target by having a scanner (such as a barcode reader) read the item. A device used to register items is called a product registration device.

[0003] A technology has been developed to detect fraudulent operations when scanning products in such product registration devices. Patent Document 1 discloses a technology that detects fraud by comparing the number of scanned products with the number of passing products detected by an optical sensor installed near the scanner. More specifically, when a user causes a scanner to scan products (hereinafter, a scanning operation), two sensors with a reflective optical detection structure consisting of a light emitter and a light receiver are installed at a distance from each other at a location where the user's hand passes. Then, when the two sensors detect objects in a predetermined order, the number of passing products is counted. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 07-210757 Summary of the Invention [Problem to be solved by the invention]

[0005] In the invention of Patent Document 1, because the passing of a commodity is detected by a sensor consisting of a light emitter and a light receiver, it is considered that detection misses and erroneous detections are likely to occur. For example, when a user moves his / her body, if the user's body happens to pass in front of two sensors, the movement will be erroneously detected as a scanning motion.

[0006] The present invention has been made in consideration of the above-mentioned problems, and one of its objectives is to provide a technique for accurately confirming that a product has been registered as a payment target. [Means for solving the problem]

[0007] The first registration confirmation device of the present invention has a scanning motion detection unit that detects scanning motions by analyzing a video including a user's hands, a determination unit that determines whether the relationship between the number of scanning motions performed and the number of items registered as items to be settled for a target transaction satisfies a specified condition, and an output unit that performs a specified output processing if the relationship does not satisfy the specified condition. The predetermined condition is a condition that is met when the number of registered products is equal to or greater than the number of detected scanning operations.

[0008] The second registration confirmation device of the present invention has a scanning motion detection unit that detects a scanning motion by analyzing a video including a user's hands, a determination unit that determines whether or not a product has been registered as a checkout target in accordance with the scanning motion, and an output unit that performs a predetermined output processing if the product has not been registered as a checkout target in accordance with the scanning motion. In the video, scanning motion is detected when a hand enters the reading area of ​​the scanner and then moves out of the reading area.

[0009] The first control method of the present invention is executed by a computer and includes a scanning motion detection step of detecting a scanning motion by analyzing a video including a user's hand, a determination step of determining whether or not a relationship between the number of scanning motions performed and the number of products registered as items to be settled for a target transaction satisfies a predetermined condition, and an output step of performing a predetermined output process if the relationship does not satisfy the predetermined condition. The predetermined condition is a condition that is met when the number of registered products is equal to or greater than the number of detected scanning operations.

[0010] A second control method of the present invention is executed by a computer and includes a scan motion detection step of detecting a scan motion by analyzing a video including a user's hand, a determination step of determining whether or not the product is registered as a payment target according to the scan motion, and an output step of performing a predetermined output process when the product is not registered as a payment target according to the scan motion. In the video, scanning motion is detected when a hand enters the reading area of ​​the scanner and then moves out of the reading area. Effect of the Invention

[0011] According to the present invention, a technique is provided for accurately confirming that a product has been registered as a payment target. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram for explaining an overview of a registration confirmation device according to a first embodiment. [Diagram 2] 2 is a block diagram illustrating a functional configuration of a registration confirmation device according to the first embodiment. FIG. [Diagram 3] FIG. 13 is a diagram illustrating a computer for realizing a registration confirmation device. [Figure 4] 4 is a flowchart illustrating a process flow executed by the registration confirmation device of the first embodiment. [Diagram 5] 11 is a diagram illustrating a flow of user operations on the product registration device; [Figure 6] 11A and 11B are diagrams illustrating screen transitions of the product registration device. [Figure 7] FIG. 13 is a diagram conceptually illustrating the association of scanning operations with products. [Figure 8] FIG. 11 is a diagram for explaining an overview of a registration confirmation device according to a second embodiment. [Figure 9] FIG. 11 is a block diagram illustrating a functional configuration of a registration confirmation device according to a second embodiment. [Figure 10] 10 is a flowchart illustrating a process executed by a registration confirmation device of the second embodiment. [Figure 11] 11A and 11B are diagrams illustrating screen transitions of the product registration device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all drawings, similar components are given similar reference numerals, and descriptions thereof will be omitted as appropriate. In addition, unless otherwise specified, in each block diagram, each block represents a configuration in functional units, not a configuration in hardware units. In the following description, unless otherwise specified, various predetermined values ​​(such as threshold values) are stored in advance in a storage device accessible from the functional configuration unit that uses the values.

[0014] [Embodiment 1] <Summary> Fig. 1 is a diagram for explaining an overview of the registration confirmation device 2000 of this embodiment. Note that Fig. 1 is an example for facilitating understanding of the registration confirmation device 2000, and the functions of the registration confirmation device 2000 are not limited to those shown in Fig. 1.

[0015] The registration confirmation device 2000 is used to confirm whether or not a product has been registered as a payment target (as a payment target) by performing an operation of scanning the product with the scanner 22 of the product registration device 20 (hereinafter, the scanning operation) in the store where the product is purchased. The product registration device 20 is a device used to register a product as a payment target. The scanner 22 is a reader that reads a predetermined symbol such as a barcode attached to the product, or a camera that captures an image of the product. Hereinafter, a product registered as a payment target is also referred to as a registered product.

[0016] Here, in a store where the registration confirmation device 2000 is operated, the device used to register products (i.e., the product registration device 20) and the device used to settle (pay for) the registered products may be provided as one unit or as separate devices. The device used to settle the payment for registered products is called a settlement device, etc. The user who operates the product registration device 20 to register products may be a store clerk or a customer.

[0017] Here, if a product is successfully scanned by each scanning operation, one or more products are registered each time a scanning operation is performed. Therefore, if products are successfully scanned by each of all scanning operations performed for one transaction, a predetermined relationship will be established between the number of registered products registered for that transaction and the number of scanning operations performed for that transaction. Specifically, the number of registered products will be greater than or equal to the number of scanning operations. Note that the number of registered products will be greater than the number of scanning operations when multiple products are scanned in one scanning operation. Also, "one transaction" refers to a transaction for one or more products that are settled together at one time (a transaction for one or more products purchased by one customer at one time).

[0018] In consideration of the relationship between the number of scan operations and the number of registered products described above, the registration confirmation device 2000 performs the following processing to confirm whether products have been registered correctly in all scan operations in a certain transaction. Here, the transaction that is the subject of confirmation of whether products have been registered correctly is called the target transaction. The registration confirmation device 2000 detects scan operations by analyzing video 12 obtained from a camera 10 that captures the user's actions. The registration confirmation device 2000 determines whether the relationship between the detected number of scan operations and the number of registered products satisfies a specified condition. The specified condition is a condition that is met when the number of registered products is equal to or greater than the number of scan operations.

[0019] If the above-mentioned predetermined condition is not satisfied (i.e., if there is a scan operation in which the product has not been registered), the registration confirmation device 2000 performs a predetermined output process. For example, this output process is the output of a message that makes the user aware of the existence of the product that has not been registered.

[0020] <Example of action and effect> According to the registration confirmation device 2000 of this embodiment, it is determined whether or not the relationship between the number of scanning operations and the number of registered products satisfies a predetermined condition. If the predetermined condition is not satisfied, a predetermined output process is performed. Here, the registration confirmation device 2000 detects the scanning operation by analyzing the video 12 in which the scanning operation is recorded. Therefore, compared to a case in which the scanning operation is detected using a sensor that senses the passage of an object, the scanning operation can be detected with high accuracy. Furthermore, the number of scanning operations can be accurately counted. Therefore, it is possible to accurately confirm whether or not a product has been registered as a payment target by the scanning operation.

[0021] This embodiment will be described in further detail below.

[0022] <Example of functional configuration> 2 is a diagram illustrating an example of the functional configuration of the registration confirmation device 2000 of the first embodiment. The registration confirmation device 2000 has a scan motion detection unit 2020, a first determination unit 2040, and a first output unit 2060. The scan motion detection unit 2020 detects scan motions by analyzing the video 12. The first determination unit 2040 determines whether the number of detected scan motions and the number of registered products satisfy a predetermined condition. The first output unit 2060 performs a predetermined output process if the predetermined condition is not satisfied.

[0023] <Example of Hardware Configuration of Registration Verification Device 2000> Each functional component of the registration confirmation device 2000 may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a further explanation will be given of the case where each functional component of the registration confirmation device 2000 is realized by a combination of hardware and software.

[0024] FIG. 3 is a diagram illustrating a computer 1000 for realizing the registration confirmation device 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, for example, the computer 1000 may be a portable computer such as a smartphone or a tablet terminal. Note that when the registration confirmation device 2000 and the product registration device 20 are provided integrally, the computer 1000 may be any computer capable of realizing the product registration device 20 (such as a POS (Point of Sales) terminal).

[0025] The computer 1000 may be a dedicated computer designed to realize the registration confirmation device 2000, or may be a general-purpose computer. In the latter case, for example, the functions of the registration confirmation device 2000 are realized in the computer 1000 by installing a specific application in the computer 1000. The application is composed of a program for realizing each functional component of the registration confirmation device 2000.

[0026] The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection.

[0027] The processor 1040 is one of various processors such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, or a read only memory (ROM) or the like.

[0028] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0029] The network interface 1120 is an interface for connecting the computer 1000 to a communication network. This communication network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). In FIG. 3, the network interface 1120 is connected to the camera 10 and the product registration device 20.

[0030] The method of communicatively connecting the camera 10 and the product registration device 20 to the computer 1000 is not limited to a method of connecting via a network. Furthermore, the camera 10 and the product registration device 20 do not have to be communicatively connected to the computer 1000. In this case, information is exchanged between these devices, for example, via a storage device commonly accessible from these devices. Furthermore, the registration confirmation device 2000 and the product registration device 20 may be provided as an integrated unit. In this case, the computer 1000 functions not only as the registration confirmation device 2000 but also as the product registration device 20.

[0031] The storage device 1080 stores program modules (program modules that realize the above-mentioned applications) that realize the various functional components of the registration confirmation device 2000. The processor 1040 reads these program modules into the memory 1060 and executes them to realize the functions corresponding to the program modules.

[0032] <Processing flow> FIG. 4 is a flowchart illustrating the flow of processing executed by the registration confirmation device 2000 of the first embodiment. The scan operation detection unit 2020 analyzes the video 12 for the target transaction and detects scan operations from the video 12 to identify the number of scan operations performed for the target transaction (S102). The first determination unit 2040 determines whether the number of scan operations and the number of registered products for the target transaction satisfy a predetermined condition (S104). If the predetermined condition is not satisfied (S104: NO), the first output unit 2060 performs a predetermined output process (S106). If the predetermined condition is satisfied (S104: YES), the processing of FIG. 4 ends.

[0033] The first output unit 2060 may perform some output processing even when the predetermined condition is satisfied. However, different output processing is performed when the predetermined condition is satisfied and when it is not satisfied. For example, when the predetermined condition is not satisfied, a message prompting the user to redo the product registration is output, and when the predetermined condition is satisfied, a message indicating that the product registration has been completed normally is output. Further details of the output processing will be described later.

[0034] <Operation flow for the product registration device 20> In order to facilitate understanding of the usage of the registration confirmation device 2000, an example of the flow of user operations performed by a user on the product registration device 20 in a target transaction will be described. The flow of user operations described here is also the same as that in the second embodiment described later.

[0035] 5 is a diagram illustrating an example of a flow of user operations on the product registration device 20. The user performs an operation on the product registration device 20 indicating the start of product registration (S202). In response to this operation, the product registration device 20 transitions to a state in which the product registration device 20 accepts product registration. For example, this operation is an operation of pressing a button that displays "Start product scanning" or the like.

[0036] The user scans each product to be purchased and has the scanner 22 read each product (S204, S206: NO). When the user has finished scanning all products (S206: YES), the user performs an operation to indicate completion of product registration (S208). In response to this operation, the state of the product registration device 20 transitions to a state in which product registration is not accepted. In addition, the product registration device 20 transmits a notification to the registration confirmation device 2000 indicating that product registration has been completed. However, if the product registration device 20 and the registration confirmation device 2000 are provided as an integrated unit, this notification is not necessary.

[0037] When the registration of the product is completed, the registration confirmation device 2000 compares the number of scanning operations with the number of registered products (S104). It is preferable that the detection of the scanning operations and the counting of the number of scanning operations are performed in parallel while the product registration is being performed.

[0038] Note that imaging by camera 10 may be performed at all times, or may be performed only during the period from when product registration starts (registration start time) to when product registration ends (registration end time). In FIG. 5, the registration start time and registration end time are times S202 and S208, respectively. When imaging is performed only during the period from the registration start time to the registration end time, for example, camera 10 receives from product registration device 20 a notification indicating that an operation indicating the start of product registration has been performed, and a notification indicating that an operation indicating the end of product registration has been performed. Camera 10 starts imaging in response to receiving the former notification, and ends imaging in response to receiving the latter notification.

[0039] <Getting Video 12> The scan motion detection unit 2020 analyzes the video 12 corresponding to the target transaction and detects a scan motion from the video 12. To do so, the scan motion detection unit 2020 acquires the video 12. Here, the scan motion detection unit 2020 may acquire all of the images (video frames) that make up the video 12 all at once, or may acquire the video frames multiple times. In the latter case, it is preferable for the scan motion detection unit 2020 to perform, in parallel, a process of repeatedly acquiring video frames and a process of analyzing already acquired video frames to detect a scan motion.

[0040] There are various methods for acquiring video frames constituting video 12 in multiple batches. For example, the scanning motion detection unit 2020 acquires a new video frame every time the camera 10 generates the new video frame. Alternatively, for example, the scanning motion detection unit 2020 may acquire unacquired video frames periodically (e.g., once per second) in batches. For example, when a 30 fps (frames / second) camera is used as the camera 10, when video frames are acquired once per second, 30 video frames are acquired at a time.

[0041] The scan motion detection unit 2020 may acquire the video 12 from the camera 10, or may acquire the video 12 from a source other than the camera 10. In the latter case, for example, the scan motion detection unit 2020 acquires the video 12 by accessing a storage device in which the video 12 is stored by the camera 10.

[0042] The scan motion detection unit 2020 uses, from among the videos generated by the camera 10, a video from the start of registration to the end of registration as the video 12. For example, the scan motion detection unit 2020 acquires information indicating the start of registration and the end of registration from the merchandise registration device 20. Then, the scan motion detection unit 2020 acquires, from among the videos generated by the camera 10, a portion between the start of registration and the end of registration as the video 12. Alternatively, for example, as described above, the camera 10 may capture images only between the start of registration and the end of registration. In this case, the scan motion detection unit 2020 acquires, as the video 12, the entire video generated by the camera 10 for the target transaction.

[0043] <Scanning motion detection: S102> The scan motion detection unit 2020 detects a scan motion from the video 12 (S102). For example, the scan motion detection unit 2020 has a detection model that detects a scan motion from a video in which a scan motion is recorded in response to input of the video. Here, "a video in which a scan motion is recorded" means that the video is generated by capturing the scan motion with a camera.

[0044] For example, the detection model sequentially accepts time-series video frames constituting a video, and when a scanning motion is detected from one or more of the accepted video frames, outputs an indication that a scanning motion has been detected. The detection model then initializes its internal state and further accepts subsequent video frames. In this manner, one or more scanning motions included in the video 12 are detected in sequence. Note that, when a scanning motion is detected from a video, the detection model preferably outputs information indicating the start and end points of the scanning motion (for example, the frame numbers of the video frames at the start and end points of the scanning motion).

[0045] The detection model is trained in advance using a video in which a scanning motion is recorded. This training is performed using at least a video in which a scanning motion is recorded as training data of a positive example. In addition, negative example data (i.e., a video in which a scanning motion is not recorded) may also be used for training.

[0046] Various videos can be used as training data. For example, a video including a series of events, such as 1) a hand being positioned outside the reading area of ​​the scanner 22 (i.e., an area where a product can be scanned), 2) the hand entering the reading area, and 3) the hand leaving the reading area, is used as training data for a positive example (i.e., a video in which a scanning operation is recorded).

[0047] However, only actions performed while holding an object in the hand may be treated as scanning actions. That is, a video including a series of actions, 1) a hand holding an object is positioned outside the reading area of ​​the scanner 22, 2) the hand enters the reading area while holding the object, and 3) the hand leaves the reading area while holding the object, is used as training data of a positive example. In this case, a video in which an action in which an object is not being held in any of 1) to 3) is recorded is treated as training data of a negative example (i.e., a video in which a scanning action is not recorded).

[0048] Also, only actions performed with the hand in a predetermined shape may be treated as scanning actions. That is, a video including a series of actions, 1) a hand in a predetermined shape is positioned outside the reading area of ​​the scanner 22, 2) the hand enters the reading area while maintaining that shape, and 3) the hand leaves the reading area while maintaining that shape, is used as training data of a positive example. In this case, a video in which a hand action that is not in the predetermined shape in any of 1) to 3) is recorded is treated as training data of a negative example (i.e., a video in which a scanning action is not recorded).

[0049] Furthermore, by combining the above two examples, only actions performed in a state where "an object is being held by a hand in a specific shape" may be treated as a scanning action. In this case, if an object is being held but the hand shape is not in a specific shape, or if the hand shape is in a specific shape but an object is not being held, it will not be treated as a scanning action.

[0050] Other than that, for example, only actions that satisfy conditions such as the hand moving in a predetermined direction or the hand remaining in the reading area of ​​the scanner 22 for a predetermined period of time may be treated as scanning actions. Also, if the palm of the hand is captured in the image, it may not be treated as a scanning action.

[0051] The method of detecting a scanning motion from video 12 is not limited to using a trained model, and various techniques for detecting a predetermined motion from a video can be used.

[0052] <Comparison of the number of scans and the number of registered products: S104> The first determination unit 2040 compares the number of registered scan operations and the number of registered products for the target transaction to determine whether a predetermined condition is satisfied for the relationship between them (S104). The predetermined condition is a condition that is met when the number of registered products is equal to or greater than the number of scan operations. For example, the condition is "the number of registered products is equal to or greater than the number of scan operations" or "the value obtained by subtracting the number of scan operations from the number of registered products is equal to or greater than 0." In addition, in a case where the number of products that can be scanned in one scan operation is limited to one (i.e., the scanner 22 can scan only one product at a time), it is preferable that the predetermined condition is a condition that is met when the number of registered products and the number of scan operations match (such as "the number of registered products matches the scan operations" or "the value obtained by subtracting the number of scan operations from the number of registered products is 0").

[0053] In order to determine whether the above-mentioned predetermined condition is satisfied, the first determination unit 2040 obtains the number of scan operations and the number of registered products for the target transaction (i.e., for the currently ongoing transaction). In other words, the first determination unit 2040 identifies the number of scan operations performed during the period from the start of registration to the end of registration for the target transaction, and the number of products registered during that period.

[0054] In this regard, as described above, the scanning motion detection unit 2020 analyzes the video 12 from the start of registration to the end of registration to detect scanning motions contained therein. The first determination unit 2040 then counts the number of detected scanning motions to identify the number of scanning motions. In addition, the number of registered products can be obtained from the product registration device 20. The first determination unit 2040 then obtains the number of registered products from the product registration device 20.

[0055] <Output when the specified conditions are not met: S106> If the relationship between the number of scan operations and the number of registered products for the target transaction does not satisfy a predetermined condition (S104: NO), the first output unit 2060 performs a predetermined output process (S106). Various processes can be used as the predetermined output process. Below, several examples of the predetermined output process are given.

[0056] <<Example 1 of a given output process>> For example, the first output unit 2060 outputs a message to the product registration device 20 to make the user aware of the existence of the unregistered product. For example, this message may be displayed on a display device of the product registration device 20 or output as sound from a speaker of the product registration device 20.

[0057] Here, the messages output to the product registration device 20 will be illustrated in more detail, together with an explanation of the screen transitions in the product registration device 20. Fig. 6 is a diagram illustrating the screen transitions in the product registration device 20. Note that each screen in Fig. 6 has been simplified for the purpose of explaining the registration confirmation device 2000, and various information not shown here may be displayed on each screen during actual operation. Also, Fig. 6 assumes a case in which the product registration device 20 and a settlement device are provided as an integrated unit.

[0058] Screen 30 is a screen that is displayed on the display device of the product registration device 20 before product registration begins. When the user presses button 32, the screen displayed on the display device transitions to screen 40. Screen 40 is displayed on the display device of the product registration device 20 while product registration is being performed. Note that screen 40 may be configured to display both the number of detected scanning operations and the number of registered products. In this case, the registration confirmation device 2000 counts the number of scanning operations in real time. By displaying the number of scanning operations and the number of registered products on the screen in this manner, the user can recognize that the scanning operations are being monitored, which is expected to have a crime prevention effect.

[0059] When the user has finished scanning all the products, he or she presses button 42. In response to the pressing of button 42, the registration confirmation device 2000 determines whether or not the relationship between the number of scanning operations and the number of registered products satisfies a predetermined condition.

[0060] If the predetermined conditions are met, the target transaction will move from product registration to payment. Therefore, the screen displayed on the display device will transition to a payment screen (a screen for making payment for the product). Here, the configuration of the payment screen is arbitrary.

[0061] On the other hand, if the predetermined condition is not satisfied, the screen displayed on the display device transitions to screen 50. Screen 50 includes a message prompting the user to redo the product registration. To redo the product registration, the user presses button 52. When button 52 is pressed, the state of product registration device 20 transitions to a state in which product registration is accepted again. Therefore, the screen displayed on the display device transitions to screen 40.

[0062] However, the screen displayed when redoing product registration may be a screen different from screen 40. For example, as described below, in the case where only unregistered products are redoed, it is necessary for the user to know which products were not registered. For this reason, information for knowing which products were not registered is displayed on the screen. The information for knowing which products were not registered will be described later.

[0063] If the user does not want to redo the product registration, he or she presses button 54. When button 54 is pressed, the target transaction moves from product registration to settlement. Therefore, the screen displayed on the display device transitions to the settlement screen. Note that button 54 may not be displayed, and the user may be forced to redo the product registration whenever a predetermined condition is not met.

[0064] Here, "re-registering products" may mean re-registering all products, or may mean re-registering only those products that have not been registered (in other words, registering additional products). In the former case, when button 52 is pressed, product registration device 20 clears all information on already registered products and transitions to a state in which product registration is accepted. The user then performs a scan operation on all products again.

[0065] On the other hand, when the user wants to redo the registration of only the products that were not registered, the product registration device 20 transitions to a state in which the registration of products is accepted without clearing the information of the already registered products when the button 52 is pressed. Then, the user performs a scan operation again only on the products that were not registered.

[0066] When redoing registration only for products that were not registered, the user needs to know which products were not registered. Therefore, the first output unit 2060 outputs information for the user to know which products were not registered. For example, the first output unit 2060 presents the user with an image in which the scanning operation performed on the products that were not registered is recorded. By looking at this image, the user can know which products were not registered.

[0067] Images of the scanning action of a product that was not registered can be identified by comparing the time when each product was registered with the period during which each scanning action was performed. Specifically, when a product is registered by a certain scanning action, the registration of the product is performed within the period from the start to the end of the scanning action (hereinafter, the period of the scanning action). Therefore, if the time when a certain product was registered is within the period of a certain scanning action, it can be determined that the product was registered by that scanning action.

[0068] Therefore, the first output unit 2060 compares the period of each scan operation with the time when each product was registered, thereby associating the scan operation with the product registered by the scan operation. More specifically, the first output unit 2060 associates the scan operation with the product registered within the period of the scan operation. According to this process, the product is associated with the scan operation with which the product is registered, while the product is not associated with the scan operation with which the product is not registered. Therefore, the first output unit 2060 identifies the scan operation with which the product is not associated as the scan operation with which the product is not registered. Note that the period of each scan operation can be identified as the period from the generation time of the video frame detected as the start point of the scan operation to the generation time of the video frame detected as the end point of the scan operation, for example.

[0069] Fig. 7 is a diagram conceptually illustrating the association of scanning actions with products. In Fig. 7, three scanning actions, X to Z, are performed. However, two products, A and B, are registered. Therefore, there is one scanning action for which no product has been registered.

[0070] Here, products A and B are registered within the periods of scan motions X and Z, respectively. Therefore, the first output unit 2060 associates products A and B with scan motions X and Z, respectively. Furthermore, no products are associated with scan motion Y. Therefore, the first output unit 2060 identifies scan motion Y as a scan motion for which no products have been registered.

[0071] The first output unit 2060 outputs information about the scan operation in which the product was not registered. For example, this information is an image in which the scan operation is recorded. This image may be a plurality of video frames (i.e., a video) from the start point to the end point of the scan operation, or a part of video frames included in the video (for example, one video frame).

[0072] In the latter case, it is preferable to output an image that allows the user to easily grasp the product. Therefore, for example, the first output unit 2060 identifies the video frame with the largest image area representing the product from among the video frames that constitute the video in which the scan operation in which the product was not registered was recorded, and outputs the video frame.

[0073] Here, when the scanner 22 is realized by a camera, an image generated by the scanner 22 may be output instead of or together with the image in which the scanning operation is recorded. Compared to the image in which the scanning operation is recorded, the image generated by the scanner 22 is considered to capture the product larger, making it easier to grasp the product.

[0074] For example, the first output unit 2060 outputs, among multiple images (video) generated by the scanner 22, those generated during a period of scanning operation in which the product was not registered. As another example, the first output unit 2060 may analyze the scanning operation recorded in the video 12 to identify the point in time at which the product passed through the center of the reading range of the scanner 22 during that scanning operation, and output the image generated by the scanner 22 at that point in time.

[0075] Furthermore, each output image may not be the entire image generated by the camera, but may be an image in which only a part of the image area including the product is cut out from the entire image. Existing technology can be used to cut out only a part of the image area including the product from the image. For example, the image area including the product can be obtained as an image area of ​​a predetermined size including a hand. Alternatively, for example, the image area including the product can be obtained by using a model trained to detect only the image area representing the product from the input image.

[0076] The information for identifying the products that were not registered is not limited to information regarding the scan operation in which the products were not registered. For example, the first output unit 2060 may present a list of registered products to the user. In this case, the user can identify which products were not registered by comparing the presented list with the product that the user is trying to purchase. For example, assume that products A and B are shown in the presented list, and the products that the user is trying to purchase are A, B, and C. In this case, the user can identify that product C was not registered, and can re-register product C.

[0077] Alternatively, for example, the first output unit 2060 may analyze an image in which a scan operation in which a product was not registered is recorded, thereby identifying the unregistered product and outputting the identification information (such as the name) of the product. Here, existing technology can be used as a technology for identifying a product included in an image by image analysis.

[0078] <<Example 2 of a given output process>> For example, the first output unit 2060 may provide an output that enables a person related to the store, such as a store clerk or a security guard, to understand that a predetermined condition has not been satisfied with respect to the relationship between the number of scan operations and the number of registered products (i.e., an output that enables a person related to the store to understand that there was a scan operation that did not register a product). For example, the first output unit 2060 transmits predetermined output information to a mobile terminal carried by the store clerk or security guard. In another example, when the product registration device 20 in which a scan operation that did not register a product was detected is the product registration device 20 operated by a customer, the first output unit 20 may output the above-mentioned output information to the product registration device 20 operated by the store clerk.

[0079] Various information may be included in the output information. For example, the output information may include identification information of the product registration device 20 where a scanning operation in which a product was not registered was performed, and an image (particularly, a facial image) of the user who performed the scanning operation. For example, the image of the user may be obtained from a camera installed so as to be able to capture an image of the user's face, such as a surveillance camera. In addition, if the camera 10 is installed so as to be able to capture an image of the user's face, as well as an image of the user's hands, the first output unit 2060 may obtain an image of the user from the camera 10.

[0080] The predetermined output information may include an image recording a scanning operation that did not result in the product being registered. By viewing this image, the store clerk or the like can ascertain to some extent whether the scanning operation that did not result in the product being registered was an operation that may have been performed intentionally to prevent the product from being registered, or whether it was simply a scanning error. The method for identifying an image recording a scanning operation that did not result in the product being registered is as described above.

[0081] The first output unit 2060 may also use sound, light, or the like to enable a store clerk or the like to know that the predetermined condition has not been met. For example, if the merchandise registration device 20 is provided with a lamp visible from the surroundings, the first output unit 2060 outputs an indication that the predetermined condition has not been met in the target transaction in the merchandise registration device 20 by turning on or blinking the lamp in a predetermined manner. Alternatively, for example, a predetermined sound may be output from a speaker provided in the merchandise registration device 20.

[0082] <<Changing output processing according to the characteristics of scanning operations>> Scanning actions that do not result in the registration of a product can be classified as unintentional (a scanning action that is performed with the intention of registering the product, but the product is not registered) or intentional (a scanning action that is performed without the intention of registering the product). Hereinafter, the former will be referred to as a non-fraudulent scan action, and the latter as a fraudulent scan action. The registration confirmation device 2000 may determine whether a scanning action that does not result in the registration of a product is a non-fraudulent scan action or a fraudulent scan action, and may change the output processing depending on the determination result.

[0083] For example, if the scan action that did not register a product is a non-illegal scan action, the first output unit 2060 performs the output process of Example 1 described above. On the other hand, if the scan action is an illegal scan action, the first output unit 2060 performs the output process of Example 2 described above, or both the output processes of Examples 1 and 2. In this way, if the user has not performed an illegal operation, the system is limited to prompting the user to re-register the product, but if the user has performed an illegal operation, a store clerk or the like is notified, so that measures such as setting up an anti-crime system can be taken.

[0084] To realize such processing, scan behavior detection unit 2020 is configured to be able to distinguish between non-illegal scan behavior and unauthorized scan behavior. For example, a detection model that detects scan behavior is provided with a function to distinguish whether the scan behavior is a non-illegal scan behavior or an unauthorized scan behavior in addition to detecting the scan behavior. In this case, the detection model is trained using both a video in which a non-illegal scan behavior is recorded and a video in which an unauthorized scan behavior is recorded.

[0085] Actions that are treated as unauthorized scanning actions are determined, for example, by an administrator of the registration confirmation device 2000. For example, if a user checks the position of the barcode of a product but performs a scanning action while covering the barcode with their hand, this action is considered to be an unauthorized scanning action. Other actions that are considered to be unauthorized scanning actions include a scanning action in which a product is moved to avoid the reading area of ​​the scanner 22, or a scanning action in which products are held in the hand so that they are stacked and moved so that the product captured by the camera 10 is different from the product to be read by the scanner 22.

[0086] In addition, for example, whether or not a scan is fraudulent can also be determined based on the degree to which information necessary for registering a product is obtained. For example, a scan in which only a small portion of the product's barcode is obscured can be predicted as a non-fraudulent scan. On the other hand, a scan in which most or all of the product's barcode is obscured can be predicted as a fraudulent scan. If the scanner 22 is a camera that captures images of products, the degree to which the product is obscured can be used to predict whether or not a scan is fraudulent.

[0087] Therefore, the degree to which the products and their barcodes are hidden is changed between the video for learning non-fraudulent scanning behavior and the video for learning fraudulent scanning behavior, so that the detection model can distinguish between non-fraudulent scanning behavior and fraudulent scanning behavior depending on the degree to which the products and their barcodes are hidden.

[0088] <Processing when the specified conditions are met> When a predetermined condition is satisfied with respect to the relationship between the number of scan operations and the number of registered products, the registration confirmation device 2000 may perform any process. For example, the registration confirmation device 2000 notifies the product registration device 20 that the product registration has been completed normally. In response to receiving this notification, the product registration device 20 performs a process to complete the product registration. Here, various existing processes may be adopted as the process performed by the product registration device 20 in response to the completion of the product registration. For example, when the product registration device 20 and the settlement device are provided integrally, the state of the product registration device 20 transitions to a state in which payment of the price is accepted. On the other hand, when the product registration device 20 and the settlement device are provided separately, for example, the product registration device 20 displays a message or the like on a display device to encourage the user to make payment at the settlement device. Note that when the product registration device 20 and the registration confirmation device 2000 are provided integrally, notification from the registration confirmation device 2000 to the product registration device 20 is not necessary.

[0089] <Monitoring other than scanning operations> In a store where the registration confirmation device 2000 is operated, monitoring other than the scanning operation may also be performed. For example, the operation of a customer putting an unregistered product into a bag or the operation of a store clerk putting money taken from a drawer of a payment device into his pocket may be monitored. The monitoring of these operations is performed by analyzing video obtained from a camera that captures the operations of the customer and the store clerk. This camera may be the camera 10 used to monitor the scanning operation, or it may be another camera.

[0090] [Embodiment 2] <Summary> Fig. 8 is a diagram for explaining an overview of the registration confirmation device 2000 of this embodiment. Note that Fig. 8 is an example for facilitating understanding of the registration confirmation device 2000, and the functions of the registration confirmation device 2000 are not limited to those shown in Fig. 8. Furthermore, unless otherwise specified, the registration confirmation device 2000 of the second embodiment has the same functions as the registration confirmation device 2000 of the first embodiment.

[0091] The registration confirmation device 2000 of the second embodiment checks whether or not the registration of the product has been performed normally in response to each scan operation, not for each transaction, but for each scan operation. Specifically, the registration confirmation device 2000 detects a scan operation from the video 12, and further determines whether or not the product has been registered as a payment target in response to the scan operation. Then, the registration confirmation device 2000 performs a predetermined output process when the product has not been registered in response to the scan operation (i.e., when the scan operation has been detected but the product has not been registered). For example, the predetermined output process is a notification to make the user aware that the product has not been registered.

[0092] <Example of action and effect> According to the registration confirmation device 2000 of this embodiment, it is determined whether or not a product has been registered as a payment target in accordance with a scanning operation, and if the product has not been registered, a predetermined output process is performed. Here, the registration confirmation device 2000 detects the scanning operation by analyzing the video 12 in which the scanning operation is recorded. Therefore, the scanning operation can be detected with high accuracy compared to a case in which the scanning operation is detected using a sensor that senses the passage of an object. Therefore, it is possible to accurately confirm whether or not a product has been registered as a payment target by the scanning operation.

[0093] <Example of Functional Configuration of Registration Confirmation Device 2000> 9 is a diagram illustrating an example of a functional configuration of a registration confirmation device 2000 of the second embodiment. The registration confirmation device 2000 of the second embodiment has a scan motion detection unit 2020, a second determination unit 2080, and a second output unit 2100. The scan motion detection unit 2020 of the second embodiment is similar to the scan motion detection unit 2020 of the first embodiment. The second determination unit 2080 determines whether or not a product has been registered in response to the detected scan motion. The second output unit 2100 performs a predetermined output process when a product has not been registered in response to the detected scan motion.

[0094] <Example of hardware configuration> The hardware configuration of the registration confirmation device 2000 of this embodiment is shown in Fig. 3, similar to the hardware configuration of the registration confirmation device 2000 of embodiment 1. However, the storage device 1080 of this embodiment stores a program for realizing the registration confirmation device 2000 of this embodiment.

[0095] <Processing flow> Fig. 10 is a flowchart illustrating a process flow executed by the registration confirmation device 2000 of the second embodiment. The scan motion detection unit 2020 detects a scan motion from the video 12 (S302). The second determination unit 2080 determines whether or not a product has been registered in response to the scan motion (S304). If a product has not been registered in response to the scan motion (S304: NO), the second output unit 2100 performs a predetermined output process. The process of Fig. 10 is repeatedly executed from the start to the end of product registration for the target transaction.

[0096] <Determine whether the product has been registered: S304> The second determination unit 2080 determines whether or not a product has been registered in response to the scan operation (S304). For example, when the scan operation detection unit 2020 detects a scan operation, the second determination unit 2080 determines whether or not a new product has been registered in the product registration device 20 since the previous determination was made by the second determination unit 2080. If a new product has been registered since the previous determination, the second determination unit 2080 determines that a product has been registered in response to the scan operation. On the other hand, if no new product has been registered since the previous determination, the second determination unit 2080 determines that a product has not been registered in response to the scan operation. Note that information regarding product registration can be obtained from the product registration device 20.

[0097] Alternatively, for example, the second determination unit 2080 may determine whether or not a registered product exists during the period of the scanning operation detected by the scanning operation detection unit 2020, and if a registered product exists during the period of the scanning operation, determine that the product has been registered in response to the scanning operation. Also, if a registered product does not exist during the period of the scanning operation, determine that the product has not been registered in response to the scanning operation.

[0098] <Output Processing by the Second Output Unit 2100: S306> If the product is not registered in response to the scanning operation (S304: NO), the second output unit 2100 performs a predetermined output process. As the predetermined output process, various processes can be adopted. Some examples of the predetermined output process are described below.

[0099] <<Example 1 of a given output process>> For example, the second output unit 2100 outputs a message to the product registration device 20 to let the user know that the product has not been registered. For example, this message may be displayed on a display device of the product registration device 20 or output as sound from a speaker of the product registration device 20.

[0100] Here, the messages output to the product registration device 20 will be illustrated in more detail, together with an explanation of the screen transitions in the product registration device 20. Fig. 11 is a diagram illustrating the screen transitions in the product registration device 20. Note that each screen in Fig. 11 has been simplified for the purpose of explaining the registration confirmation device 2000, and various information not shown here may be displayed on each screen during actual operation. Also, Fig. 11 assumes a case in which the product registration device 20 and a settlement device are provided as an integrated unit.

[0101] Screen 30 and screen 40 are the same as those described in the first embodiment (screen 30 and screen 40 in FIG. 6). However, in this embodiment, the screen transition after screen 40 is displayed is different from that in the first embodiment. When screen 40 is displayed on the display device, the user performs a scan operation to register the product. The registration confirmation device 2000 analyzes the video 12 to detect the scan operation, and determines whether the product has been registered in response to the scan operation. If the product has been registered in response to the scan operation, the product registration device 20 continues to accept scans of the product. Therefore, screen 40 continues to be displayed on the display device of the product registration device 20.

[0102] On the other hand, if the product is not detected in response to the scanning operation, the screen displayed on the display device transitions to screen 60. Screen 60 includes a message that prompts the user to retry registering the product. The user then performs a scanning operation again for the product that the product registration device 20 attempted to scan in the previous scanning operation, in order to attempt to register the product.

[0103] The registration confirmation device 2000 analyzes the video 12 to detect the scanning action, and determines whether or not the product has been registered in response to the scanning action. If the product has been registered in response to the scanning action, the screen displayed on the display device transitions to screen 70. On the other hand, if the product has not been registered in response to the scanning action, screen 60 continues to be displayed on the display device.

[0104] A message is displayed on screen 70 indicating that the product has been registered successfully. To end the product registration, the user presses button 72. When button 72 is pressed, the screen displayed on the display device transitions to a checkout screen. To further register the product, the user performs a scan operation to attempt to register the product. If the product is registered in response to the scan operation, the screen displayed on the display device transitions to screen 40. On the other hand, if the product is not registered in response to the scan operation, the screen displayed on the display device transitions to screen 60.

[0105] In the example of Fig. 11, the message indicating that the product registration was successful is displayed only when the product registration is redone. However, this message may be displayed when the product is registered regardless of whether the product registration is redone.

[0106] In this embodiment, a determination is made as to whether or not a product has been registered each time a scanning operation is performed, so when "re-registering a product," it is only necessary to redo the scanning operation for the product that was attempted to be registered in the previous scanning operation.

[0107] <<Example 2 of a given output process>> For example, the second output unit 2100 may output information that enables a store associate, such as a store clerk or a security guard, to understand that the product was not registered in response to the scan operation. For example, the second output unit 2100 transmits predetermined output information to a mobile terminal carried by the store clerk or security guard. In another example, if the product registration device 20 in which the product was not registered in response to the scan operation is a product registration device 20 operated by a customer, the second output unit 2100 may output the above-mentioned output information to the product registration device 20 operated by the store clerk.

[0108] The output information may include various information. For example, the output information may include identification information of the product registration device 20 where a product was not registered in response to a scanning operation, and an image of the user who performed the scanning operation. The method of acquiring the image of the user is as described above. Furthermore, the predetermined output information may include an image in which a scanning operation where a product was not registered is recorded.

[0109] Here, output information may be provided to a store clerk or the like only when a situation occurs in the target transaction where the product is not registered in response to a scan operation a predetermined number of times (two or more times). For example, if an abnormality occurs in the scanner 22, a situation may arise where the product cannot be registered normally even if the user repeatedly tries to register the product. By notifying a store clerk or the like when a situation occurs multiple times where the product is not registered in response to a scan operation, the store clerk or the like can easily grasp such a situation requiring attention.

[0110] The second output unit 2100 may use sound or light to enable a store clerk or the like to know that the product was not registered in response to the scanning operation. For example, if the product registration device 20 is provided with a lamp visible from the surroundings, the second output unit 2100 outputs a message indicating that the product was not registered in response to the scanning operation by turning on or blinking the lamp in a predetermined manner. Alternatively, for example, a predetermined sound may be output from a speaker provided in the product registration device 20.

[0111] As described in the first embodiment, the registration confirmation device 2000 may determine whether the scan action is a non-fraudulent scan action or a fraudulent scan action, and may change the output process depending on the determination result. For example, similar to the first embodiment, when the scan action for which the product was not registered is a non-fraudulent scan action, the second output unit 2100 performs the output process of example 1. On the other hand, when the scan action is a fraudulent scan action, the second output unit 2100 performs the output process of example 2, or both the output processes of examples 1 and 2.

[0112] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and it is also possible to adopt combinations of the above-described embodiments or various configurations other than those described above.

[0113] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. 1. A scanning motion detection unit that detects scanning motions by analyzing a video including a user's hands; a determination unit that determines whether or not a relationship between the number of times the scanning operation has been performed and the number of products registered as items to be settled for a target transaction satisfies a predetermined condition; an output unit that performs a predetermined output process when the relationship does not satisfy the predetermined condition, The predetermined condition is a condition that is satisfied when the number of the registered products is equal to or greater than the number of the detected scanning operations. 2. The registration confirmation device described in 1., wherein a scanning motion is detected when, in the video, a hand enters a reading area of ​​the scanner and then moves out of the reading area. 3. A registration confirmation device as described in 1. or 2., in which the output unit identifies scanning operations for which products were not registered by comparing the period during which each scanning operation was performed with the time at which each product was registered, and outputs an image including that scanning operation. 4. A scanning motion detection unit that detects a scanning motion by analyzing a video including a user's hand; a determination unit that determines whether or not the product is registered as a payment target in response to the scanning operation; an output unit that performs a predetermined output process when the product is not registered as a payment target in response to the scanning operation, An enrollment verification device in which a scanning motion is detected when, in the video, a hand enters a reading area of ​​a scanner and then the hand moves out of the reading area. 5. A registration confirmation device described in any one of 2 to 4, in which a scanning action is detected when, in the video, a hand holding a product enters the scanner's reading area and then moves out of the reading area. 6. A registration confirmation device described in any one of 2 to 5, in which a scanning action is detected when, in the video, a hand with a shape that meets a predetermined criterion enters the scanner's reading area and then moves out of the reading area. 7. The scanning motion detection unit detects the scanning motion using a detection model that detects a scanning motion from an input video; A registration confirmation device described in any one of 1 to 6, wherein the detection model is a model trained using video obtained by capturing a scanning operation with a camera. 8. A registration confirmation device described in any one of 1 to 7, wherein the output unit outputs a message prompting the user to rescan the product. 9. The scan operation detection unit determines whether the scan operation is an unauthorized scan operation or a non-illegal scan operation; A registration confirmation device described in any one of 1 to 8, wherein the output unit performs different output processing when the scanning operation that did not result in the product being registered is an unauthorized scanning operation and when the scanning operation that did not result in the product being registered is a non-invalid scanning operation. 10. A computer-implemented control method comprising: a scanning motion detection step of detecting a scanning motion by analyzing a video including a user's hand; a determination step of determining whether or not a relationship between the number of times the scanning operation has been performed and the number of products registered as items to be settled for the target transaction satisfies a predetermined condition; and an output step of performing a predetermined output process when the relationship does not satisfy the predetermined condition, A control method, wherein the predetermined condition is a condition that is satisfied when the number of the registered products is equal to or greater than the number of the detected scanning operations. 11. The control method described in 10., wherein a scanning motion is detected when, in the video, a hand enters a reading area of ​​the scanner and then moves out of the reading area. 12. A control method described in 10. or 11., in which, in the output step, a scanning operation for which a product was not registered is identified by comparing the period during which each scanning operation was performed with the time at which each product was registered, and an image including that scanning operation is output. 13. A computer-implemented control method comprising: a scanning motion detection step of detecting a scanning motion by analyzing a video including a user's hand; a determination step of determining whether or not the product has been registered as a payment target in response to the scanning operation; and an output step of performing a predetermined output process when the product is not registered as a payment target in response to the scanning operation, A control method in which a scanning motion is detected when, in the video, a hand enters a reading area of ​​the scanner and then the hand moves out of the reading area. 14. A control method described in any one of 11 to 13, in which a scanning action is detected when, in the video, a hand holding a product enters the scanner's reading area and then moves out of the reading area. 15. A control method described in any one of 11 to 14, in which a scanning action is detected when, in the video, a hand with a shape that meets a predetermined criterion enters the scanner's reading area and then moves out of the reading area. 16. In the scanning motion detection step, the scanning motion is detected using a detection model for detecting a scanning motion from an input video; A control method described in any one of 10 to 15, wherein the detection model is a model trained using video obtained by capturing the scanning operation with a camera. 17. A control method according to any one of claims 10 to 16, wherein in the output step, a message is output prompting the user to rescan the product. 18. In the scan operation detection step, determining whether the scan operation is an unauthorized scan operation or a non-illegal scan operation; A control method described in any one of 10 to 17, wherein in the output step, different output processing is performed when the scanning operation for which a product was not registered is an invalid scanning operation and when the scanning operation for which a product was not registered is a non-invalid scanning operation. 19. A program for causing a computer to execute the control method described in any one of 10 to 18.

[0114] This application claims priority based on Japanese Patent Application No. 2020-035052, filed on March 2, 2020, the disclosure of which is incorporated herein in its entirety. [Explanation of symbols]

[0115] 10. Camera 12 videos 20 Product registration device 22 Scanner 30 screens 32 Buttons 40 screens 42 Buttons 50 screens 52 Buttons 54 Buttons 60 screens 70 screens 72 Buttons 1000 calculator 1000 Computers 1020 Bus 1040 Processor 1060 Memory 1080 Storage Device 1100 Input / Output Interface 1120 Network Interface 2000 Registration Verification Device 2020 Scanning motion detector 2040 1st Judgment Department 2060 1st output section 2080 Second Judgment Department 2100 2nd output section

Claims

1. a scanning motion detection unit that detects a scanning motion by analyzing a video including a user's hand; a determination unit that determines whether or not a relationship between a total number of times the scanning operation has been performed and the number of products registered as items to be settled for one target transaction for one or more products to be settled at once satisfies a predetermined condition; an output unit that performs a predetermined output process when the relationship does not satisfy the predetermined condition, the predetermined condition is a condition that is satisfied when the number of the registered products is equal to or greater than the total number of the detected scanning operations; The determination unit counts the number of times the scanning operation is performed, The output unit outputs the number of times the scanning operation is performed and the number of registered products to be displayed on a display device while the products are being registered. Registration verification device.

2. 2. The enrollment verification device of claim 1, wherein a scanning motion is detected when, in the video, a hand enters a reading area of ​​a scanner and then the hand moves out of the reading area.

3. The registration confirmation device according to claim 1 or 2, wherein the output unit identifies a scanning operation for which a product was not registered by comparing a period during which each scanning operation was performed with a time when each product was registered, and outputs an image including that scanning operation.

4. 4. The registration confirmation device according to claim 2, wherein a scanning action is detected when, in the video, a hand holding a product enters a reading area of ​​a scanner and then the hand moves out of the reading area.

5. The registration confirmation device according to claim 2 , wherein a scanning motion is detected when, in the video, a hand whose shape meets a predetermined criterion enters a reading area of ​​a scanner and then moves out of the reading area.

6. The scanning motion detection unit detects the scanning motion using a detection model that detects a scanning motion from an input video; The registration confirmation device according to claim 1 , wherein the detection model is a model trained using a video obtained by capturing a scanning operation with a camera.

7. The registration confirmation device according to claim 1 , wherein the output unit outputs a message prompting the user to rescan the product.

8. The scanning motion detection unit determines whether the scanning motion is an unauthorized scanning motion or a non-unauthorized scanning motion using a detection model trained using both a video in which a non-unauthorized scanning motion is recorded and a video in which an unauthorized scanning motion is recorded; The registration confirmation device according to any one of claims 1 to 7, wherein the output unit performs different output processing when the scanning operation that did not result in the product being registered is an unauthorized scanning operation and when the scanning operation that did not result in the product being registered is a non-invalid scanning operation.

9. 1. A computer-implemented control method comprising: a scanning motion detection step of detecting a scanning motion by analyzing a video including a user's hand; a determination step of determining whether or not a relationship between the total number of times the scanning operation is performed and the number of products registered as items to be settled for one target transaction for one or more products to be settled at once satisfies a predetermined condition; and an output step of performing a predetermined output process when the relationship does not satisfy the predetermined condition, the predetermined condition is a condition that is satisfied when the number of the registered products is equal to or greater than the total number of the detected scanning operations; Counting the number of times the scanning operation is performed; and and outputting the number of times the scanning operation has been performed and the number of registered products to be displayed on a display device while the products are being registered. Control methods.

10. A program for causing a computer to execute a control method, The control method includes: a scanning motion detection step of detecting a scanning motion by analyzing a video including a user's hand; a determination step of determining whether or not a relationship between the total number of times the scanning operation is performed and the number of products registered as items to be settled for one target transaction for one or more products to be settled at once satisfies a predetermined condition; and an output step of performing a predetermined output process when the relationship does not satisfy the predetermined condition, the predetermined condition is a condition that is satisfied when the number of the registered products is equal to or greater than the total number of the detected scanning operations; The control method includes: Counting the number of times the scanning operation is performed; and and outputting the number of times the scanning operation has been performed and the number of registered products to be displayed on a display device while the products are being registered. program.

Images