Fraud detection programs, fraud detection devices, and fraud detection systems

Abstract

The system uses captured images to detect malicious actions related to scanning operations with high accuracy. [Solution] The memory unit stores region information indicating the positions of multiple image regions set in a captured image taken of the front area of ​​a self-checkout terminal equipped with a scanner. The processing unit recognizes the product from the captured image. Based on the movement path of the recognized product through multiple image regions and the time the product stayed in the first image region closest to the scanner among the multiple image regions, the processing unit detects any unauthorized operation related to the scanning operation that causes the scanner to scan the product information attached to the product.

Description

Technical Field

[0001] The present invention relates to an unauthorized detection program, an unauthorized detection device, and an unauthorized detection system.

Background Art

[0002] In stores where products are sold, the spread of self-checkout terminals where users themselves perform barcode reading and settlement operations of products is progressing. When using a self-checkout terminal, the user himself / herself causes the barcode attached to the product to be read by a barcode scanner provided in the self-checkout terminal.

[0003] Here, as a method for determining whether the scanning operation has been appropriately performed, there are the following proposals. For example, when a product barcode is read by a scanner, an image of the product at the scanning position at that time is extracted, and by tracing the captured image and checking until the corresponding product is taken out from the unregistered product stand, a POS (Point Of Sale) terminal for confirming that the barcode has been appropriately scanned has been proposed.

[0004] Also, when recognizing the shopping basket and the products taken out from it from the captured image, a self-checkout terminal device that determines whether a product has been scanned and gives an error warning when it has not been scanned has been proposed.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] There is a problem in that it is difficult to accurately detect fraudulent activity related to scanning operations using only images captured by a camera, without linking with self-checkout terminals or POS systems. In one aspect, the present invention aims to provide a fraud detection program, fraud detection device, and fraud detection system capable of detecting fraudulent operations related to scanning using captured images with high accuracy. [Means for solving the problem]

[0007] One proposal involves providing a fraud detection program that instructs a computer to recognize products from images taken of the front area of ​​a self-checkout terminal equipped with a scanner, and to perform a process to detect fraudulent activity related to scanning operations that cause the scanner to scan product information attached to the products, based on the product's movement path through multiple image regions set in the image and the time the product spends in the first image region closest to the scanner among the multiple image regions.

[0008] In another proposal, a fraud detection device is provided that performs the same processing as the fraud detection program described above. Furthermore, one proposal provides a fraud detection system having the above-mentioned fraud detection device and camera. [Effects of the Invention]

[0009] In one respect, the captured images can be used to detect fraudulent actions related to scanning operations with high accuracy. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example configuration and processing example of the fraud detection system according to the first embodiment. [Figure 2] This figure shows an example configuration of a self-checkout monitoring system according to the second embodiment. [Figure 3] This figure shows an example of the hardware configuration of a fraud detection device. [Figure 4]It is a diagram showing an overview of a method for detecting skipped scans. [Figure 5] It is a diagram showing a configuration example of a processing function provided in a fraud detection device. [Figure 6] It is a diagram showing an example of setting a determination area. [Figure 7] It is a diagram showing a data configuration example of area setting information. [Figure 8] It is a flowchart showing an example of processing by an operation extraction unit. [Figure 9] It is a diagram showing a data configuration example of a product position information DB. [Figure 10] It is a flowchart showing an example of processing by an area passage detection unit. [Figure 11] It is a diagram showing a data configuration example of an area passage information DB. [Figure 12] It is a flowchart showing an example of processing by a determination unit. [Figure 13] It is a diagram showing an example of a movement trajectory of a product when a normal scan operation is performed. [Figure 14] It is a diagram showing an example of a movement trajectory of a product when an improper operation is performed. [Figure 15] It is a diagram for explaining the detection of skipped scans based on the movement trajectory. [Figure 16] It is a flowchart showing an example of determination processing based on the movement trajectory. [Figure 17] It is a flowchart showing an example of determination processing based on area passage information and the movement trajectory. [[ID=E40]] [Figure 18] It is a flowchart showing an example of processing by a determination unit in a second modification example. <000E<4000090>It is a diagram showing a configuration example of a processing function provided in a fraud detection device according to a third embodiment. [Figure 20] It is a flowchart showing an example of processing by a determination unit in the third embodiment. [Figure 21] It is a flowchart showing a modification example of the determination processing shown in FIG. 20. [Figure 22] It is a flowchart showing an example of processing by a notification unit in the third embodiment. [Figure 23] It is a diagram showing a first modification of the process shown in FIG. 22. [Figure 24] It is a diagram showing a second modification of the process shown in FIG. 22.

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. 〔First Embodiment〕 FIG. 1 is a diagram showing a configuration example and a processing example of a fraud detection system according to the first embodiment. The fraud detection system shown in FIG. 1 includes a fraud detection device 10 and a camera 1 connected to the fraud detection device 10.

[0012] The camera 1 photographs the front area of the self-checkout terminal 3 including the scanner 2 and transmits the data of the photographed image to the fraud detection device 10. The scanner 2 is a device that scans the barcode attached to the product to be purchased, and is arranged at a position where the purchaser of the product can perform a scanning operation by approaching the product from the front side of the self-checkout terminal 3. The camera 1 photographs the front area of the self-checkout terminal 3, for example, from above the self-checkout terminal 3.

[0013] The fraud detection device 10 has a storage unit 11 and a processing unit 12. The storage unit 11 is a storage area secured in a storage device (not shown) provided in the fraud detection device 10. The processing unit 12 is, for example, a processor. The processes described below are realized, for example, when the processing unit 12, which is a processor, executes a predetermined program.

[0014] The storage unit 11 stores area information 13 indicating the positions of a plurality of image areas set in the photographed image. The processing unit 12 recognizes the product from the photographed image. Then, the processing unit 12 detects an illegal operation related to the scanning operation based on the movement path of the product in a plurality of image areas of the recognized product and the staying time of the product in the first image area closest to the scanner 2 among the plurality of image areas.

[0015] In the captured image 4 shown in Figure 1, three image regions Ra to Rc are set as an example. The following describes an example of processing using these image regions Ra to Rc. The processing unit 12 recognizes product 5 from the captured image 4. Then, the processing unit 12 analyzes the movement path of product 5 within the image region Ra to Rc. For example, it analyzes which image regions the product moved through and in what order.

[0016] The image regions Ra to Rc described above are arranged sequentially along the path that product 5 moves when a buyer performs a normal scan operation on the captured image 4. In the example in Figure 1, when a normal scan operation is performed, product 5 moves from left to right on the captured image 4. In this case, when a normal scan operation is performed, product 5 moves from image region Ra to image region Rb as shown by arrow L1 and the scan operation is performed, and then product 5 moves from image region Rb to image region Rc as shown by arrow L2.

[0017] However, even if product 5 moves in the order of image regions Ra, Rb, and Rc as described above, there are cases where the scan operation is not performed. This occurs when product 5 moves relatively quickly near scanner 2 in order to prevent scanner 2 from scanning, or to make it appear as if the purchaser has performed a scan. To detect such fraudulent behavior, the processing unit 12 considers not only the movement path described above, but also the time product 5 spends in image region Rb, which is closest to scanner 2, in order to detect fraudulent behavior.

[0018] For example, suppose product 5 moves in image region Rb along the path indicated by arrow L3. Along this path, product 5 enters image region Rb at entry position P1 and exits image region Rb at exit position P2. In this case, processing unit 12 obtains the time T1 when product 5 moves to entry position P1 and the time T2 when product 5 moves to exit position P2, and calculates the time product 5 stays in image region Rb by (T2-T1).

[0019] The processing unit 12 then detects any unauthorized actions based on the analysis results of the movement path and the time spent in each area. For example, suppose the processing unit 12 determines that product 5 moved in the order of image regions Ra, Rb, and Rc. In this case, the processing unit 12 determines that no unauthorized action occurred if the time spent in image region Rb is greater than or equal to a predetermined threshold. However, if the time spent in each area is less than the threshold, the processing unit 12 determines that an unauthorized action occurred.

[0020] Through this process, the fraud detection device 10 can detect fraudulent operations related to scanning with high accuracy using the images 4 captured by the camera 1. [Second Embodiment] Figure 2 shows an example of the configuration of a self-checkout monitoring system according to the second embodiment. The self-checkout monitoring system shown in Figure 2 is a system for monitoring the purchasing actions of users (customers) in stores where goods are sold, and includes a fraud detection device 100 and a camera 101 connected to the fraud detection device 100. The fraud detection device 100 is an example of the fraud detection device 10 shown in Figure 1.

[0021] The fraud detection device 100 is a computer device such as a personal computer. The camera 101 is installed inside the store where the self-checkout terminal 50 is installed. The self-checkout terminal 50 is a terminal device included in a POS system and is a self-service cash register device in which the user performs the payment operation themselves. Such a self-checkout terminal 50 is also called a self-checkout terminal.

[0022] The self-checkout terminal 50 includes a barcode scanner 51, a display 52, and a deposit / withdrawal unit 53. The barcode scanner 51 reads the barcode attached to the product, which indicates the product code. The display 52 displays the price of the product whose barcode has been read, the total amount of the purchased items, and the amount of change. The deposit / withdrawal unit 53 accepts deposits from the user and dispenses change.

[0023] In this embodiment, a pre-scan item storage area 54 where items before scanning are placed and a post-scan item storage area 55 where scanned items are placed are positioned opposite each other with the self-checkout terminal 50 in between. For example, the user places the items to be purchased in the pre-scan item storage area 54. In some cases, the items to be purchased may be placed in a shopping basket, and that basket may be placed in the pre-scan item storage area 54. The user picks up each item placed in the pre-scan item storage area 54 (or in the basket) one by one and performs a "scan operation" by bringing the item close to the barcode scanner 51 to read the barcode.

[0024] Once the user has finished scanning all items, they perform a "settlement operation" to request payment. For example, if the display 52 is a touch panel, the user can perform the settlement operation by pressing the settlement button on the touch panel. After performing the settlement operation, the user deposits the purchase amount into the deposit / withdrawal unit 53 according to the information displayed on the display 52, and receives any change from the deposit / withdrawal unit 53.

[0025] Camera 101 photographs the front area of ​​the self-checkout terminal 50 (particularly around the barcode scanner 51) so as to capture the user's purchasing actions using the self-checkout terminal 50. In this embodiment, camera 101 is positioned above the self-checkout terminal 50, and the area in the vicinity of the front of the self-checkout terminal 50 is photographed from above by camera 101. The fraud detection device 100 can determine from the image captured by camera 101 whether the user performed a correct purchasing action, and can issue a warning if it is determined that an abnormal purchasing action was performed.

[0026] Figure 3 shows an example of the hardware configuration of a fraud detection device. The fraud detection device 100 is implemented as a computer, for example, as shown in Figure 3. The fraud detection device 100 shown in Figure 3 includes a processor 111, RAM (Random Access Memory) 112, HDD (Hard Disk Drive) 113, GPU (Graphics Processing Unit) 114, input interface (I / F) 115, reading device 116, network interface (I / F) 117, and communication interface (I / F) 118.

[0027] The processor 111 (processor circuit) comprehensively controls the entire fraud detection device 100. The processor 111 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or a PLD (Programmable Logic Device). Alternatively, the processor 111 may be a combination of two or more elements from among the CPU, MPU, DSP, ASIC, and PLD.

[0028] RAM 112 is used as the main memory of the fraud detection device 100. At least a portion of the OS (Operating System) program and application programs to be executed by the processor 111 are temporarily stored in RAM 112. In addition, various data necessary for processing by the processor 111 are stored in RAM 112.

[0029] HDD113 is used as an auxiliary storage device for the fraud detection device 100. The HDD113 stores the OS program, application programs, and various data. Other types of non-volatile storage devices, such as SSDs (Solid State Drives), can also be used as auxiliary storage devices.

[0030] A display device 114a is connected to the GPU 114. The GPU 114 displays images on the display device 114a according to instructions from the processor 111. The display device can be an LCD or an OLED (Electroluminescent) display.

[0031] An input device 115a is connected to the input interface 115. The input interface 115 transmits signals output from the input device 115a to the processor 111. Examples of input devices 115a include keyboards and pointing devices. Examples of pointing devices include mice, touch panels, tablets, touchpads, and trackballs.

[0032] A portable recording medium 116a is attached to and detached from the reading device 116. The reading device 116 reads the data recorded on the portable recording medium 116a and transmits it to the processor 111. The portable recording medium 116a can be an optical disc, a semiconductor memory, or the like.

[0033] The network interface 117 transmits and receives data with other devices via the network 117a. The communication interface 118 transmits and receives data to and from the camera 101.

[0034] The processing functions of the fraud detection device 100 can be realized with the hardware configuration described above. The processor 111 is an example of the processing unit 12 shown in Figure 1, and the memory area allocated in RAM 112 or HDD 113 is an example of the storage unit 11 shown in Figure 1.

[0035] Incidentally, self-checkout terminals, which are operated by the user themselves, are rapidly becoming widespread for purposes such as addressing labor shortages caused by population decline. However, there is a risk of fraudulent activity where users intentionally avoid scanning product barcodes at self-checkout terminals, and there is a need for technology to detect such fraudulent activity.

[0036] In this embodiment, the fraud detection device 100 detects a type of fraud called "scan skipping," in which a user pretends to scan a product's barcode with the barcode scanner 51 but does not actually scan it. Here, one possible method for the fraud detection device 100 to detect scan skipping is for the fraud detection device 100 to work in conjunction with the self-checkout terminal 50 or POS system. As an example, the fraud detection device 100 can use information obtained from the self-checkout terminal 50 or POS system, such as scan operation detection information and information on scanned products, in addition to images captured by the camera 101, to make the detection.

[0037] On the other hand, there is a problem in that it is difficult to accurately detect scan skips using only images captured by camera 101. If the fraud detection device 100 can detect scan skips using only images captured by camera 101, it will not need to be linked with the self-checkout terminal 50 or the POS system. Therefore, it will be possible to develop and install the fraud detection device 100 regardless of the specifications of the self-checkout terminal 50 or the POS system. Consequently, the versatility of the fraud detection device 100 will be improved, and the ease of installation will also increase.

[0038] In this embodiment, the fraud detection device 100 accurately detects scan skips using only images captured by the camera 101 in the manner shown in Figure 4. Figure 4 shows an overview of the method for detecting scan skips. Images 201-203 in Figure 4 are images taken from above by camera 101 of the vicinity of the front of the self-checkout terminal 50 (the vicinity of the front of the barcode scanner 51). Images 201-203 also show the user 204 grasping the product 205 to be purchased.

[0039] In this embodiment, as shown in Figure 4, when viewed from above, the pre-scan item placement area 54 is located to the left of the self-checkout terminal 50, and the post-scan item placement area 55 is located to the right of the self-checkout terminal 50. In this case, the item 205 held by the user 204 moves from left to right in the captured image when scanned. However, the pre-scan item placement area 54 and the post-scan item placement area 55 may be located on opposite sides of the self-checkout terminal 50, in which case the direction of movement of the item 205 will also be reversed.

[0040] Image 201 is an example of an image captured when the system is operating normally by user 204. In Image 201, the product 205 held by user 204 is moved from the pre-scan product storage area 54 to the barcode scanner 51, and then to the post-scan product storage area 55.

[0041] Images 202 and 203 are examples of images captured when a scan skip occurs. Possible patterns of operation when a scan skip occurs include, for example, the first type of malfunction shown in Image 202 and the second type of malfunction shown in Image 203.

[0042] In image 202, the product 205, held by user 204, is removed from the pre-scan product storage area 54, moves to the right while moving away from the barcode scanner 51, and is placed in the post-scan product storage area 55. In image 202, product 205 starts from the pre-scan product storage area 54, moves upwards and to the right, passes in front of the barcode scanner 51, then turns downwards and to the right to move to the post-scan product storage area 55.

[0043] In this first type of fraudulent activity, product 205 is clearly moved away from the barcode scanner 51 so that its barcode cannot be scanned. As a result, there is a clear difference in the movement path of product 205 compared to when normal operation is performed as shown in image 201. Therefore, the fraud detection device 100 can detect the first type of fraudulent activity based on the movement path of product 205 in the captured image.

[0044] On the other hand, in the second fraudulent operation, user 204 moves product 205 at high speed in the vicinity of barcode scanner 51. This allows user 204 to move product 205 along a path similar to normal operation so as not to notice that a scan has been skipped, while preventing the barcode of product 205 from being scanned. For example, as shown in image 203, product 205, held by user 204, is taken out of the pre-scan product storage area 54 and then moves almost in a straight line to the post-scan product storage area 55 without being brought close to the barcode scanner 51. Also, as in image 201, product 205 may be brought close to the barcode scanner 51 before moving to the post-scan product storage area 55.

[0045] Thus, in the second fraudulent operation, the movement path of product 205 is similar to that of normal operation, so the fraud detection device 100 cannot detect that the second fraudulent operation has occurred solely from the movement path of product 205 in the captured image. However, the fraud detection device 100 can detect the second fraudulent operation based on the movement speed of product 205 in the vicinity of the barcode scanner 51.

[0046] In light of the above, the fraud detection device 100 detects scan skips based on the movement path and movement speed of the product 205. This makes it possible to detect scan skips with high accuracy using only the images captured by the camera 101.

[0047] Figure 5 shows an example of the configuration of the processing functions provided by the fraud detection device. The fraud detection device 100 includes a storage unit 120 and a control unit 130. The storage unit 120 is a storage area reserved in a storage device provided by the fraud detection device 100, such as a RAM 112 or an HDD 113. The storage unit 120 stores area setting information 121, product location information DB (database) 122, and area passage information DB 123.

[0048] The area setting information DB 121 stores location information for multiple judgment areas set on the image captured by the camera 101. The product location information DB 122 stores the position of the product held by the user in the captured image, frame by frame. The area passage information DB 123 stores information indicating the passage status of the product through each set judgment area.

[0049] The processing of the control unit 130 is achieved, for example, by the processor 111 executing a predetermined application program. The control unit 130 includes an image input unit 131, a region setting unit 132, an action extraction unit 133, a region passage detection unit 134, a determination unit 135, and a notification unit 136.

[0050] The image input unit 131 receives data of the captured image taken by the camera 101. The region setting unit 132 sets multiple judgment regions on the captured image and registers the position information of the set judgment regions in the region setting information 121.

[0051] The motion extraction unit 133 performs image recognition processing on the captured image to recognize the product being held by the person (user) from the captured image, tracks the position of the product in the captured image, and registers the position information indicating the movement path of the product in the product position information DB 122.

[0052] The area passage detection unit 134 detects the passage status of products in each set judgment area based on the location information of products registered in the product location information DB 122, and registers information indicating the passage status in the area passage information DB 123. At this time, the area passage information DB 123 also registers the entry time and exit time of products in a particular judgment area.

[0053] The determination unit 135 detects the occurrence of a scan skip based on the information registered in the product location information DB 122 and the area passage information DB 123. When a scan skip is detected, the notification unit 136 notifies the administrator of the self-checkout terminal 50 that a scan skip has been detected, for example, using the display device 114a.

[0054] Figure 6 shows an example of setting the judgment area. When the camera 101 is installed, the area setting unit 132 sets multiple judgment areas within the image captured by the camera 101 to determine if a scan skip has occurred, in response to an input operation by the administrator.

[0055] The captured image 211 shown in Figure 6 is an example of an image captured by camera 101. Camera 101 is installed to capture the area in the vicinity of the front of the self-checkout terminal 50 from above the self-checkout terminal 50. More specifically, camera 101 is configured so that its shooting area includes at least the area in front of the barcode scanner 51, the top surface of the pre-scan item placement area 54, and the top surface of the post-scan item placement area 55.

[0056] As shown in Figure 2, in the self-checkout terminal 50, a display 52 is provided above the barcode scanner 51, and a cash deposit / withdrawal unit 53 is provided below the barcode scanner 51. Therefore, as shown in Figure 6, in the captured image 211, the display 52 is captured in the area below the barcode scanner 51, and the cash deposit / withdrawal unit 53 is captured above the barcode scanner 51.

[0057] For such a captured image 211, as shown in the lower part of Figure 6, the above-mentioned determination area is set to include a product retrieval area R1, a scan area R2, and a product removal area R3. The product retrieval area R1 is set to encompass the upper surface area of ​​the product storage area 54 before scanning. The scan area R2 is set in the front area of ​​the barcode scanner 51 so that the product to be scanned is captured. The product removal area R3 is set to encompass the upper surface area of ​​the product storage area 55 after scanning.

[0058] With the judgment areas set as described above, if a normal scan operation is performed, the product will move in the following order: product retrieval area R1, scan area R2, and product removal area R3. In other words, it is desirable that the multiple judgment areas be arranged along the product's movement path when a normal scan operation is performed.

[0059] Figure 7 shows an example of the data structure of area setting information. Area setting information 121 registers the area ID and coordinates for each of the following areas: product retrieval area R1, scan area R2, and product removal area R3.

[0060] The area ID indicates the identification number of the judgment area. In the example in Figure 7, area IDs "1", "2", and "3" represent the product retrieval area R1, the scanning area R2, and the product removal area R3, respectively. The coordinates are coordinate information indicating the position of the judgment area in the captured image. In this embodiment, as illustrated in Figure 6, each judgment area is rectangular. In this case, the coordinates field registers the coordinates of the four vertices of each judgment area in a predetermined order. Note that when the judgment area is rectangular in this way, the coordinates field may also register, for example, the coordinates of one vertex of each judgment area and the coordinates of the vertex opposite that vertex (for example, the coordinates of the top-left and bottom-right vertices in the captured image). Furthermore, the shape of each judgment area is not limited to a rectangle.

[0061] Figure 8 is a flowchart showing an example of the processing in the motion extraction unit. [Step S11] The motion extraction unit 133 detects a user from the captured image through person recognition processing.

[0062] [Step S12] The motion extraction unit 133 detects from the captured image that the user is grasping the product through object recognition processing. In steps S11 and S12, the user and product may be detected using object or human body recognition technology such as YOLO (You Only Look Once). Alternatively, the user detection in step S11 and the product detection in step S12 may be performed simultaneously using HOID (Human-Object Interaction Detection) technology. In this case, the captured image is input to the HOID learning model (neural network), and HOID information indicating the interaction between the person and object recognized from the captured image is output. The HOID information includes information about the recognized person, information about the recognized object, and an action ID indicating the person's action toward that object. In steps S11 and S12, if HOID information including an action ID indicating "holding an object" as the type of action is output, the user grasping the product is detected, and the positional information of the person (user) and object (product) in the captured image is obtained.

[0063] The position of the product is detected, for example, as a bounding box, which is a rectangular area surrounding the product in the captured image. [Step S13] After the gripping of the product is detected in step S12, the motion extraction unit 133 tracks the position of the product from each image frame of the input captured image.

[0064] [Step S14] The motion extraction unit 133 determines whether tracking has finished. Tracking finishes when the product moves away from the user or when the product is no longer recognized in the captured image. If tracking has not finished, the process proceeds to step S13 and tracking continues. On the other hand, if tracking has finished, the processing for that product ends.

[0065] Figure 9 shows an example of the data structure of the product location information database. The product location information database 122 registers a record for each product that has been recognized as being picked up by a user. Each record includes the track ID, pick-up start time, pick-up end time, location information, and image data.

[0066] The Track ID is an identification number assigned to each product. The Grasp Start Time indicates the first time that a user is detected to have grasped a product. When a product grasped by a user is detected in step S12 of Figure 8, a record is added to the product location information DB122, a unique Track ID is registered for the added record, and the time at that time is registered as the Grasp Start Time.

[0067] The gripping completion time indicates the time when the user has finished gripping the product. When tracking of the product is completed in step S14 of Figure 8, that time is registered as the gripping completion time.

[0068] The location information is coordinate information indicating the position of the product in the captured image. For each image frame from the start time of gripping to the end time of gripping, coordinate information indicating the position of the product in that image frame is registered. If the position of the product is detected as a bounding box as described above, then the location information only needs to be information representing the position of the bounding box. For example, the location information registered would be a numerical sequence (x0,y0,x1,y1) indicating the coordinates of the top-left vertex (x0,y0) and the bottom-right vertex (x1,y1) of the bounding box.

[0069] The image data indicates the filename of the image data of the captured image (image frame) in which the product was detected. From the start time of gripping to the end time of gripping, the filenames of the image data of the captured images in which the tracked product is recognized are sequentially registered in the image data field, and the coordinates indicating the position of the product in the captured images are sequentially registered in the location information field.

[0070] Figure 10 is a flowchart showing an example of the processing performed by the area passage detection unit. The processing in Figure 10 is performed for each image frame of the captured image. Figure 10 also shows an example of the processing when product Xm with track ID=m passes through the judgment area Rn with area ID=n. Note that area ID=1 indicates the product retrieval area R1, area ID=2 indicates the scan area R2, and area ID=3 indicates the product removal area R3. The area passage detection unit 134 performs the processing in Figure 10 for each judgment area with area ID=1 to 3.

[0071] [Step S21] The area passage detection unit 134 obtains the location information of product Xm in the current frame (most recent image frame) and the location information of product Xm in the previous frame (the image frame before this one) from the record of product Xm (track ID=m) in the product location information DB122.

[0072] [Step S22] The region passage detection unit 134 determines whether the position of product Xm in the previous frame was outside the determination region Rn, and whether the position of product Xm in the current frame is inside the determination region Rn. If both of these conditions are met, the process proceeds to step S23; if at least one of the conditions is not met, the process proceeds to step S24.

[0073] [Step S23] The region passage detection unit 134 determines that product Xm has entered the determination region Rn. After this, the process proceeds to step S21, and processing for the next image frame is executed.

[0074] Furthermore, if the position of product Xm is within the determination area Rn when the gripping of product Xm is detected in step S12 of Figure 8, the process in step S23 will also be executed. [Step S24] The region passage detection unit 134 determines whether the position of product Xm in the previous frame is within the determination region Rn, and whether the position of product Xm in the current frame is outside the determination region Rn. If both of these conditions are met, the process proceeds to step S25. On the other hand, if at least one of the conditions is not met, the process proceeds to step S21, and processing for the next image frame is performed.

[0075] Furthermore, if tracking of product Xm is completed in the current frame (corresponding to step S14: Yes in Figure 8), the process in step S24 will also be executed. Here, in steps S22 and S24, for example, if even a part of the bounding box of product Xm is included in the determination area Rn, it may be determined to be within the area, and if the entire bounding box is not included in the determination area Rn, it may be determined to be outside the area. Alternatively, as another example, if the center point of the bounding box of product Xm is included in the determination area Rn, it may be determined to be within the area, and if the center point of the bounding box is not included in the determination area Rn, it may be determined to be outside the area.

[0076] [Step S25] The area passage detection unit 134 determines that product Xm has exited the determination area Rn. Figure 11 shows an example of the data structure of the area passage information database. The area passage information database 123 registers records (area passage information) for each product. Each record registers the track ID, each judgment area (product retrieval area R1, scan area R2, product removal area R3), entry time, and exit time.

[0077] The Track ID is an identification number assigned to each product. Each judgment area item registers information indicating the item's status within that area. The judgment area items are initially set to "NONE". In step S23 of Figure 10, if it is determined that the item has entered the judgment area, the value of the corresponding item is updated to "IN". Also, in step S25 of Figure 10, if it is determined that the item has exited the item retrieval area R1 or the item removal area R3, the value of the corresponding item is updated to "OUT". On the other hand, in step S25, if it is determined that the item in the scan area R2 has exited from the "IN" state, the value of the corresponding item is updated to "THROUGH". In other words, "THROUGH" indicates that the item entered the judgment area and then exited it.

[0078] The entry time indicates the time when the product entered the scan area R2. If it is determined in step S23 of Figure 10 that the product has entered the scan area R2, that time is registered as the entry time. The exit time indicates the time when the product left the scan area R2. If it is determined in step S25 of Figure 10 that the product has left the scan area R2, that time is registered as the exit time.

[0079] Figure 12 is a flowchart showing an example of the processing in the judgment unit. In the processing shown in Figure 12 below, the count value V of the fraud counter used for fraud detection is used. [Step S31] When a user is detected from the captured image in step S11 of Figure 8, the determination unit 135 initializes the count value V of the fraud counter to 0. After this, when the grasping of an item is detected in step S12 of Figure 8, the processing in step S32 is executed with that item as the target of processing.

[0080] [Step S32] The determination unit 135 obtains the product's position information in the current frame from the product position information DB 122 and also obtains the current area passage information (information of the record corresponding to the product) from the area passage information DB 123. If the product exit area R3 is "IN" in the area passage information, the process proceeds to step S33. On the other hand, if the product exit area R3 is not "IN", the process in step S32 is executed for the next image frame.

[0081] [Step S33] The determination unit 135 determines, based on the area passage information, whether the product has exited the product retrieval area R1, passed through the scan area R2, and entered the product removal area R3. If the area passage information indicates that the product retrieval area R1 is "OUT", the scan area R2 is "THROUGH", and the product removal area R3 is "IN", it is determined that the above conditions have been met, and the process proceeds to step S34. On the other hand, if the above conditions have not been met, the process proceeds to step S36.

[0082] [Step S34] The determination unit 135 obtains the entry time and exit time for scan area R2 from the area passage information. The determination unit 135 calculates the time the product stays in scan area R2 (the time it took to pass through the area) by calculating the difference between the exit time and the entry time.

[0083] The determination unit 135 determines whether the calculated stay time is less than a predetermined threshold TH1. If the stay time is less than the threshold TH1, the process proceeds to step S35; if the stay time is equal to or greater than the threshold TH1, the process proceeds to step S38.

[0084] In step S34, for example, the movement speed of the product in the scan area R2 may be compared with a predetermined threshold. For example, the total distance of the product's movement trajectory in the scan area R2 may be calculated based on the location information registered in the product location information DB122, and the movement speed may be calculated by dividing the total distance by the dwell time mentioned above. In this case, if the movement speed is greater than the threshold, the process proceeds to step S34, and if it is less than or equal to the threshold, the process proceeds to step S38.

[0085] [Step S35] The determination unit 135 increments the count value V of the fraud counter by 1. [Step S36] The determination unit 135 determines, based on the area passage information, whether the product has exited the product retrieval area R1 and entered the product removal area R3 without passing through the scan area R2. If the area passage information shows that the product retrieval area R1 is "OUT", the scan area R2 is "NONE", and the product removal area R3 is "IN", it is determined that the above conditions have been met, and the process proceeds to step S37. On the other hand, if the above conditions have not been met, the process proceeds to step S38.

[0086] [Step S37] The determination unit 135 increments the count value V of the fraud counter by 1. [Step S38] The determination unit 135 determines whether the count value V of the fraud counter is equal to or greater than a predetermined threshold TH2. If the count value V of the fraud counter is equal to or greater than the threshold TH2, the process proceeds to step S39; if it is less than the threshold TH2, the process proceeds to step S40.

[0087] [Step S39] The determination unit 135 determines that there is a high possibility that a scan skip has occurred and causes the notification unit 136 to execute a process to issue a warning to that effect. The notification unit 136 may, for example, display image information on the display device 114a indicating that a scan skip has occurred, and notify the administrator that a scan skip has occurred. An audio warning may also be issued. For example, the notification unit 136 may output a warning sound that a scan skip has occurred to a speaker connected to the fraud detection device 100. Furthermore, if the administrator or store employee is wearing earphones that allow them to hear audio via wireless communication, the notification unit 136 may transmit audio information warning that a scan skip has occurred and output the sound to the earphones.

[0088] Furthermore, the notification unit 136 may send notification information indicating that a scan skip has occurred to a management device within the store or a management server that oversees multiple stores. Such notification information may include, for example, the date and time of occurrence, an identification number indicating the self-checkout register where the incident occurred, and information about the details of the incident indicating that a scan skip has occurred.

[0089] [Step S40] The determination unit 135 determines whether the user has left. When the user is no longer detected in the captured image, it is determined that the user has left, and the processing for this user shown in Figure 12 is terminated. On the other hand, if a new product is detected being grasped in step S12 of Figure 8, the processing in step S32 is re-executed with that product as the target of processing.

[0090] According to the processing of the determination unit 135 described above, if "Yes" is determined in step S36, it is determined that there is a high probability that the first fraudulent operation shown in Figure 4 has occurred. Also, even if "Yes" is determined in step S33 (when the product moves from the product retrieval area R1 to the product removal area R3 via the scanning area R2), if "Yes" is determined in step S34, it is determined that there is a high probability that the second fraudulent operation shown in Figure 4 has occurred. In other words, by making a determination using the dwell time in step S34, the determination unit 135 can detect not only the first fraudulent operation but also the second fraudulent operation as a scan skip. For this reason, it is possible to detect scan skips with high accuracy using only images captured by the camera 101, without coordinating with the self-checkout terminal 50 or POS system.

[0091] Note that the threshold TH2 in step S38 can be set to "1". However, setting an integer of "2" or greater, such as "2", can reduce the possibility of detection errors due to skipped scans.

[0092] Next, a modified example in which part of the processing of the determination unit 135 is altered will be described. <First variation of the judgment process> In the first modified example, the determination unit 135 detects a scan skip based on the movement trajectory of the product in the captured image.

[0093] Figure 13 shows an example of the product's movement trajectory when a normal scanning operation is performed. Figure 14 shows an example of the product's movement trajectory when an unauthorized operation is performed.

[0094] Images 221-223 in Figure 13 and images 231-233 in Figure 14 are images taken from above by camera 101 of the vicinity of the front of the self-checkout terminal 50 (the vicinity of the front of the barcode scanner 51). Images 221-223 and 231-233 also show the movement trajectories 221a-223a and 231a-233a of the product held by the user, respectively. In all of images 221-223 and 231-233, the pre-scan product placement area 54 is located to the left of the self-checkout terminal 50, and the post-scan product placement area 55 is located to the right of the self-checkout terminal 50. Therefore, the product held by the user moves from left to right in images 221-223 and 231-233.

[0095] The movement trajectories 221a to 223a shown in images 221 to 223 are all examples of the trajectory when the product's barcode is correctly scanned by the barcode scanner 51. According to these movement trajectories 221a to 223a, the product picked up from the pre-scan product storage area 54 is first pulled towards the user, then brought closer to the barcode scanner 51. After scanning by the barcode scanner 51, the product is pulled towards the user again, then moved further away from the user, and placed in the post-scan product storage area 55. Therefore, the position of the product in the image first moves diagonally upward to the right, then diagonally downward to the right, after scanning is performed, it moves diagonally upward to the right again, and then diagonally downward to the right.

[0096] Thus, when a normal scanning operation is performed, the movement trajectory of the product generally follows the shape of the letter M. In particular, in the vicinity of the barcode scanner 51, the movement trajectory of the product generally follows the shape of the letter V.

[0097] On the other hand, the movement trajectories 231a to 233a shown in images 231 to 233 are all examples of trajectories when an unauthorized action occurs in which the product's barcode is not scanned by the barcode scanner 51. According to these movement trajectories 231a to 233a, the product picked up from the pre-scan product storage area 54 is similar to the example in Figure 13 in that it is initially drawn towards the user, but after that it moves almost in a straight line to the right or to the lower right.

[0098] Thus, a clear difference in shape arises between the movement trajectory when normal operation occurs and the movement trajectory when abnormal operation occurs. In particular, in the vicinity of the barcode scanner 51, the movement trajectory when normal operation occurs traces a V-shape (i.e., a clear change in slope occurs along the way). In contrast, the movement trajectory when abnormal operation occurs traces a generally straight line, and no clear change in slope is observed.

[0099] Therefore, the determination unit 135 analyzes the characteristics of the product's movement trajectory in the scan area R2 set near the barcode scanner 51, and detects a skipped scan based on the analysis results.

[0100] Figure 15 is a diagram illustrating the detection of scan skips based on movement trajectories. Figure 15 shows image 221, which is an example of a normal scan operation as shown in Figure 13, and image 231, which is an example of a scan skip as shown in Figure 14.

[0101] The determination unit 135 performs the analysis of the product's movement trajectory in the scan area R2 in the following procedure. Based on the product's position information registered in the product position information DB 122, the determination unit 135 obtains the product's entry position coordinates (x1, y1) into the scan area R2, the lowest point position coordinates (x2, y2) of the product in the scan area R2, and the product's exit position coordinates (x3, y3) from the scan area R2. The lowest point position coordinates are the coordinates of the position when the y-coordinate is at its minimum in the scan area R2.

[0102] The determination unit 135 determines whether a clear change in the inclination of the movement trajectory occurs at the lowest point, and determines whether or not a scan skip has occurred based on the determination result. Furthermore, the determination unit 135 determines whether or not a scan skip has occurred based on the difference in the length of the movement trajectory within the scan area R2 before and after the lowest point. As described above, if a scan skip has occurred, no clear change in the inclination of the movement trajectory occurs, so the lowest point is likely to be significantly off from the center of the movement trajectory within the scan area R2, and as a result, the above difference is likely to be large.

[0103] As an example, in Figure 16 below, the presence or absence of a scan skip is determined using the vector vec1:(x2-x1,y2-y1) from the entry position to the exit position and the vector vec2:(x3-x2,y3-y2) from the lowest point to the exit position.

[0104] Figure 16 is a flowchart showing an example of a judgment process based on the movement trajectory. The process in Figure 16 is executed, for example, when the product leaves the scan area R2. [Step S51] The determination unit 135 obtains the entry position coordinates (x1, y1) of the product relative to the scan area R2, the exit position coordinates (x3, y3) of the product from the record of the product in the product location information DB, and the lowest end position coordinates (x2, y2) of the product in the scan area R2.

[0105] [Step S52] The determination unit 135 calculates vector vec1:(x2-x1,y2-y1) and vector vec2:(x3-x2,y3-y2). [Step S53] The determination unit 135 determines whether the vectors vec1 and vec2 satisfy at least one of the following determination conditions C1 and C2.

[0106] (Judgment condition C1) TH3<(|vec1| / |vec2|) <TH4 (Decision criterion C2) (cosine similarity between vec1 and vec2) <TH5 If criterion C2 is met, it indicates that the inclination of the movement trajectory before and after the lowest point is greater than a certain angle. If criterion C1 is met, it indicates that the difference in length of the movement trajectory before and after the lowest point in the scan region R2 is small. If neither criterion C1 nor C2 is met, it is determined that the shape of the movement trajectory in the scan region R2 is not similar to a V-shape and is likely to be similar to a straight line. Therefore, it is determined that there is a high possibility that a scan skip occurred. For example, TH3=0.5, TH4=1.5, and TH5=0.6 can be set.

[0107] If at least one of the determination conditions C1 and C2 is met, the process proceeds to step S54; if neither of the determination conditions C1 nor C2 is met, the process proceeds to step S55. [Step S54] The determination unit 135 determines that no scan skipping has occurred.

[0108] [Step S55] The determination unit 135 determines that a scan skip has occurred. In this way, the determination unit 135 detects skipped scans by analyzing the movement trajectory of the product in the scan area R2 and estimating its shape characteristics. This makes it possible to detect skipped scans with high accuracy using only images captured by the camera 101, without coordinating with the self-checkout terminal 50 or POS system.

[0109] Furthermore, as shown in Figure 17, the determination unit 135 can also detect scan skips using the aforementioned area passage information in addition to the product's movement trajectory. Figure 17 is a flowchart showing an example of a determination process based on area passage information and movement trajectory. In Figure 17, processing steps that perform the same process as in Figure 12 are indicated with the same reference numerals, and their explanation is omitted.

[0110] In the process shown in Figure 17, if "Yes" is determined in step S34 of Figure 12, the process in step S34a is executed. In step S34a, the determination unit 135 determines whether at least one of the determination conditions C1 and C2 in step S53 of Figure 16 is satisfied. If at least one of the determination conditions C1 and C2 is satisfied, it is determined that no scan skipping has occurred, and the process proceeds to step S38. On the other hand, if neither of the determination conditions C1 and C2 is satisfied, it is determined that a scan skip has occurred, and the process proceeds to step S35.

[0111] In this way, by using the judgment conditions in both steps S34 and S34a to determine whether a scan has been skipped, the accuracy of the determination can be improved. As another example, if "Yes" is determined in step S34, the process may proceed to step S35, and even if "No" is determined in step S34, the process may proceed to step S35 if "No" is determined in step S34a.

[0112] <Second variation of the judgment process> Figure 18 is a flowchart showing an example of the processing of the determination unit in the second modified example. In Figure 18, the same reference numerals are used to indicate processing steps that are the same as those in Figure 12, and their explanations are omitted.

[0113] The process in Figure 18 differs from Figure 12 in that it uses two count values, V1 and V2, as the count values ​​for the fraud counter. Count value V1 is incremented when the condition in step S34 is met. Count value V2 is incremented when the condition in step S36 is met. Then, weighted addition is performed using count values ​​V1 and V2, and the presence or absence of a scan skip is determined based on the added value.

[0114] In other words, in Figure 18, steps S31b, S35b, S37b, and S38b are executed instead of steps S31, S35, S37, and S38 in Figure 12, respectively. [Step S31b] When a user is detected from the captured image, the determination unit 135 initializes the count values ​​V1 and V2 of the fraud counter to 0. After this, when the grasping of an item is detected in step S12 in Figure 8, the processing in step S32 is executed with that item as the target of processing.

[0115] [Step S35b] The determination unit 135 increments the count value V1 of the fraud counter by 1. [Step S37b] The determination unit 135 increments the count value V2 of the fraud counter by 1.

[0116] [Step S38b] The determination unit 135 calculates α·V1 + β·V2 using the weight coefficients α and β, and determines whether the calculation result is greater than or equal to a predetermined threshold TH6. If the calculation result is greater than or equal to the threshold TH6, the process proceeds to step S39; if it is less than the threshold TH6, the process proceeds to step S40.

[0117] Furthermore, it is considered that the probability of a scan skip detection error is lower when the conditions of step S36 are met compared to the conditions of step S34. For this reason, it is desirable to set the weight coefficients α and β such that α < β. For example, α = 0.8 and β = 1.2. Also, while the threshold TH6 can be set to an integer greater than or equal to (α + β), setting an integer greater than (α + β) can reduce the possibility of a scan skip detection error.

[0118] According to the processing of the determination unit 135 described above, a count value is used for each determination condition for skipping a scan, and the presence or absence of a skipped scan is determined based on the result of a weighting calculation using these count values. This makes it possible to improve the accuracy of detecting skipped scans.

[0119] For example, count values ​​V1 and V2 can also be used in the processing shown in Figure 17. Specifically, count value V1 is incremented in step S35, and count value V2 is incremented in step S37. Then, the judgment in step S38b is performed in step S38. In this case, since count value V1 is incremented based on the judgment result of step S34a in addition to step S34, the probability of a scan skip detection error is considered to be lower compared to the case in Figure 18. Therefore, it is possible to set the value of the weight coefficient α applied to count value V1 to a higher value than in the case in Figure 18.

[0120] In addition, in the second embodiment and the first and second modifications described above, the presence or absence of a scan sound emitted when the self-checkout terminal 50 scans a barcode may also be used to determine whether or not a scan skip has occurred. Since the scan sound is unique to each self-checkout terminal 50, it can be defined in advance for each self-checkout terminal 50 to be monitored. The fraud detection device 100 can pick up the scan sound using a microphone and determine whether the picked up sound is a predefined scan sound. For example, the determination unit 135 may determine whether or not a scan sound was detected during the period in which the product is included in the scan area R2, and if a scan sound is detected, it may unconditionally determine that no scan skip has occurred.

[0121] [Third Embodiment] Next, a fraud detection device according to a third embodiment will be described. This fraud detection device is capable of detecting multiple types of fraudulent activities, including the scan skipping described above.

[0122] Figure 19 shows an example of the configuration of processing functions provided by the fraud detection device according to the third embodiment. In the fraud detection device 100a shown in Figure 19, the control unit 130 includes, in addition to the image input unit 131, area setting unit 132, motion extraction unit 133, area passage detection unit 134, determination unit 135, and notification unit 136 shown in Figure 5, a scan detection unit 137, a product image acquisition unit 138, and an operation status acquisition unit 139.

[0123] When a barcode is scanned by the self-checkout terminal 50, the scan detection unit 137 receives a scan notification from the self-checkout terminal 50 indicating that a barcode has been scanned. At this time, the scan detection unit 137 also receives product information about the scanned product. This product information includes product identification information and data of an image (product image) showing the appearance of the product.

[0124] The product image acquisition unit 138 extracts product images from the product information received by the scan detection unit 137. The operation status acquisition unit 139 acquires information indicating the user's operation status from the self-checkout terminal 50. For example, the operation status acquisition unit 139 can acquire a payment start notification from the self-checkout terminal 50 indicating that the payment process for the goods has begun. The payment start notification is sent from the self-checkout terminal 50, for example, when a user who has finished scanning a barcode touches the payment start button displayed on the display 52 of the self-checkout terminal 50 to initiate the payment process.

[0125] The determination unit 135 detects multiple types of fraudulent activity based on information obtained from the captured image (for example, area passage information and time spent in the scan area R2), the product image, and whether or not a payment start notification has been received. In this embodiment, the determination unit 135 is capable of detecting "product retention" and "barcode forgery" as fraudulent activity, in addition to the "scan skipping" described above. Product retention is the action of intentionally leaving products that have not been scanned in the pre-scan product storage area 54 or post-scan product storage area 55 to avoid scanning those products. Barcode forgery is the action of changing the barcode attached to a product to the barcode of another product with a lower price, and then having the barcode scanner 51 scan the changed barcode.

[0126] Furthermore, the notification unit 136 issues warnings in different ways depending on the type of fraudulent activity detected and the timing at which the fraudulent activity was detected (whether or not settlement has started). Figure 20 is a flowchart showing an example of the processing of the determination unit in the third embodiment.

[0127] [Step S61] The determination unit 135 determines whether the product has been placed in the product storage area 55 after scanning. For example, if the product take-out area R3 is set to encompass the product storage area 55 after scanning, the determination unit 135 determines that the product has been placed in the product storage area 55 after scanning when the user has finished grasping the product while the product's position is within the product take-out area R3. If the product has been placed in the product storage area 55 after scanning, the process proceeds to step S62; otherwise, the process proceeds to step S67.

[0128] [Step S62] The determination unit 135 determines whether the barcode was scanned between the time the user started and finished picking up the product. If a scan notification is received from the self-checkout terminal 50 during the relevant period, it is determined that the product was scanned. If the product was scanned during the relevant period, the process proceeds to step S64; otherwise, the process proceeds to step S63.

[0129] [Step S63] The determination unit 135 determines that a scan skip has occurred. [Step S64] The determination unit 135 obtains a product image from the product information received from the self-checkout terminal 50 along with the scan notification.

[0130] [Step S65] The determination unit 135 compares the image of the product captured in the image (the product that was grasped in step S61) with the product image acquired in step S64 and determines whether these images are similar. For example, the determination unit 135 calculates the feature quantities of each image using CLIP (Contrastive Language-Image Pre-training) or the like, and determines that the images are similar if the similarity of the feature quantities between the images is above a predetermined threshold. As an example, images are determined to be similar if the cosine similarity is 0.75 or higher. If the images are similar, the process proceeds to step S67; otherwise, the process proceeds to step S66.

[0131] [Step S66] The determination unit 135 determines that barcode forgery has occurred. [Step S67] The determination unit 135 determines whether payment has started at the self-checkout terminal 50. If a payment start notification has been received from the self-checkout terminal 50 before the processing in step S61, it is determined that payment has started, and the process proceeds to step S68. On the other hand, if payment has not started, the process proceeds to step S61.

[0132] [Step S68] The determination unit 135 determines whether there are any products remaining in the pre-scan product storage area 54, product basket, or product cart. For example, corresponding image areas are set in the captured image, and if products are present in each image area, it is determined that products remain. If products remain, the process proceeds to step S69; if no products remain, the process shown in Figure 20 ends.

[0133] [Step S69] The determination unit 135 determines that product waste has occurred. Furthermore, the determination unit 135 may determine, for example, that an item has been left behind even before receiving a payment commencement notification, if scanning is not performed for a certain period of time after the item has been picked up, or if scanning is not performed for a certain period of time after the item in the pre-scan item storage area 54, the shopping basket, or the shopping cart has been recognized.

[0134] Figure 21 is a flowchart showing a modified version of the determination process shown in Figure 20. In Figure 21, the same reference numerals are used to indicate processing steps that are the same as those in Figure 20, and their explanations are omitted.

[0135] The determination of whether a scan has been skipped may be performed using the method shown in the second embodiment. For example, in Figure 21, step S62a is executed instead of step S62 in Figure 20. In step S62a, if "Yes" is determined in step S34 or step S36 in Figure 12, the process proceeds to step S63, and if the above conditions are not met, the process proceeds to step S64. Alternatively, the process may proceed to step S63 and it may be determined that a scan has been skipped if the count value V of the fraud counter in Figure 12 becomes equal to or greater than the threshold TH2.

[0136] Figure 22 is a flowchart showing an example of the processing of the notification unit in the third embodiment. [Step S71] The notification unit 136 obtains the result of the determination unit 135's determination of the malfunction. In other words, the subsequent processing is executed when the determination unit 135 detects any type of malfunction.

[0137] [Step S72] The notification unit 136 determines whether payment has started at the self-checkout terminal 50. If a payment start notification has been received from the self-checkout terminal 50 before the determination unit 135 detects fraudulent activity, it is determined that payment has started, and the process proceeds to step S75. On the other hand, if payment has not started, the process proceeds to step S73.

[0138] [Step S73] The notification unit 136 determines the type of fraudulent activity detected. If a scan skip or product retention is detected, the process proceeds to step S74. If barcode forgery is detected, the process proceeds to step S75.

[0139] [Step S74] The notification unit 136 displays a notification image on the display 52 of the self-checkout terminal 50 to inform the user of the situation regarding the detected fraudulent activity. For example, if a skipped scan is detected, a notification image is displayed to inform the user that the item has not been scanned. If an item is left unscanned is detected, a notification image is displayed to inform the user that there is an unscanned item remaining. In all of these cases, the user is able to operate the self-checkout terminal 50. By notifying the user of the situation as described above, the notification unit 136 prompts the user to scan the item in question.

[0140] [Step S75] The notification unit 136 displays a warning image on the display 52 of the self-checkout terminal 50 to alert the user. For example, the notification unit 136 makes it impossible for the user to operate the self-checkout terminal 50 and notifies the user that operation is impossible with a message such as "Please wait a moment." In addition, if the notification unit 136 detects a skipped scan or leftover items, it may further notify the user that there are unscanned items remaining. On the other hand, if barcode forgery is detected, the notification unit 136 may issue a stronger warning to the user.

[0141] In addition to displaying the image on the self-checkout terminal 50, the notification unit 136 also notifies the administrator of the self-checkout terminal 50 that it has detected unauthorized operation. For example, the notification unit 136 outputs an audio message to the administrator's earphones informing them that unauthorized operation has been detected.

[0142] In situations where a scan skip or leftover item is detected, it cannot necessarily be said that the user intentionally failed to scan the item; it is possible that the user simply forgot to scan it. Therefore, in step S74, instead of issuing a strong warning to the user, the system prompts the user to scan the item in question.

[0143] On the other hand, barcode forgery is a highly likely instance of malicious activity intentionally performed by the user. Therefore, in step S75, a stronger warning is issued to the user, and the administrator is also notified.

[0144] Furthermore, even if a skipped scan or leftover items are detected, if the user initiates the payment process, it is more likely that the user did so intentionally. Therefore, even in such cases corresponding to "Yes" in step S72, the process in step S75 is performed to issue a stronger warning to the user and also notify the administrator.

[0145] Figure 23 shows a first modified example of the process shown in Figure 22. In this first modified example, a count value V indicating the number of times a scan skip has been detected is counted, as shown in Figure 12. Then, if "Yes" is determined in step S73 of Figure 22, the process in step S81 is executed.

[0146] In step S81, the notification unit 136 determines whether the count value V is equal to or greater than a predetermined threshold TH7. If the count value V is less than the threshold TH7, the process proceeds to step S74; if the count value V is equal to or greater than the threshold TH7, the process proceeds to step S75.

[0147] The higher the count value V, the more likely it is that the user intentionally skipped a scan. Therefore, in the process shown in Figure 23, even if settlement has not started when a scan skip is detected, if the count value V, which indicates the number of detections, is above a certain value, a stronger warning is issued to the user compared to when it is below that value.

[0148] Figure 24 shows a second modified version of the process shown in Figure 22. In the second modified version, the storage unit 120 of the fraud detection device 100a stores a price database in which the price of each product is associated with each of the images of multiple products. Then, if "No" is determined in step S73 of Figure 22 (i.e., barcode forgery is detected), the process in step S82 is executed.

[0149] In step S82, the notification unit 136 extracts an image of the product in which barcode forgery has been detected from the captured image and extracts the price corresponding to the product image from the price database. The notification unit 136 recognizes the extracted price as the amount of damage caused by barcode forgery and determines whether the amount of damage is equal to or greater than a predetermined threshold TH8. If the amount of damage is equal to or greater than the threshold TH8, the process proceeds to step S75, a strong warning is issued to the user, and the administrator is notified. On the other hand, if the amount of damage is less than the threshold TH8, no notification or warning is given to the user.

[0150] Furthermore, the processing functions of the devices shown in each of the above embodiments (for example, the fraud detection devices 10, 100, and 100a) can be implemented by a computer. In this case, a program describing the processing content of the functions that each device should have is provided, and by executing that program on a computer, the above processing functions are implemented on the computer. The program describing the processing content can be recorded on a computer-readable recording medium. Computer-readable recording media include magnetic storage devices, optical discs, and semiconductor memory. Magnetic storage devices include hard disk drives (HDDs) and magnetic tapes. Optical discs include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (BD, registered trademark).

[0151] When distributing a program, portable recording media such as DVDs and CDs containing the program are sold. Alternatively, the program can be stored in the storage device of a server computer and transferred from the server computer to other computers via a network.

[0152] A computer executing a program stores programs, for example, those recorded on a portable storage medium or transferred from a server computer, in its own memory. The computer then reads the program from its memory and executes the processing according to the program. Alternatively, the computer can directly read the program from the portable storage medium and execute the processing according to that program. Furthermore, the computer can sequentially execute the processing according to the programs received from a server computer connected via a network, each time a program is transferred. [Explanation of Symbols]

[0153] 1 Camera 2 Scanners 3 Self-checkout terminals 4. Photo Gallery 5 products 10. Fraud detection device 11 Storage section 12 Processing Units 13 Area information L1~L3 Arrows P1 approach position P2 exit position Ra~Rc image area T1,T2 time

Claims

1. On the computer, The scanner recognizes products from images taken of the area in front of the self-checkout terminal. Based on the movement path of the product in multiple image regions set in the captured image and the time the product stays in the first image region closest to the scanner among the multiple image regions, the system detects any unauthorized actions related to the scanning operation that causes the scanner to scan product information attached to the product. An anti-fraud program that executes a process.

2. The plurality of image regions include a second image region and a third image region arranged opposite to the first image region, In detecting the aforementioned unauthorized operation, it is determined that the unauthorized operation has occurred if the product moves from the second image area through the first image area to the third image area, and the time the product spends in the first image area is shorter than a predetermined threshold. The fraud detection program according to claim 1.

3. The aforementioned captured image is an image taken from above of the front area of ​​the self-checkout terminal. The fraud detection program according to claim 2.

4. Each time it is recognized from the captured image that the same person has grasped each of the multiple products, the system performs detection of the fraudulent activity based on the movement path and the time spent there, and when the number of detected fraudulent activities reaches a predetermined value, it notifies the system of the detection of the fraudulent activity. The fraud detection program according to claim 1, which causes the computer to perform the processing.

5. Whenever it is recognized from the captured image that the same person is grasping each of the multiple items, it is determined whether the recognized item satisfies a first condition based on the movement path and the time spent there, and a second condition based on the movement path. If the first condition is met, the first count value is incremented. If the second condition is met, the second count value is incremented. When the result of a weighted addition calculation using the first and second count values ​​with a predetermined weight coefficient reaches a predetermined value, the detection of the unauthorized operation is notified. The fraud detection program according to claim 1, which causes the computer to perform the processing.

6. The first condition indicates that all of the plurality of image regions are passed through in a predetermined order, and the time spent in the first image region is shorter than a predetermined threshold. The second condition indicates that the image passed through one of the plurality of image regions without passing through the first image region. The fraud detection program according to claim 5.

7. In detecting the aforementioned fraudulent activity, the fraudulent activity is detected based on the movement path, the duration of stay, and the characteristics of the shape of the movement trajectory of the product in the first image region. The fraud detection program according to claim 1.

8. In detecting the aforementioned unauthorized movement, the characteristics of the shape of the movement trajectory are determined based on the position where the product enters the first image area, the position where the product is closest to the scanner in the first image area, and the position where the product exits the first image area. The fraud detection program according to claim 7.

9. Based on the aforementioned travel path and the aforementioned stay time, a first fraudulent operation related to the scanning operation is detected; and based on the recognition result of the product in the captured image and information regarding the product or the scanning operation of the product obtained from the self-checkout terminal, a second fraudulent operation related to the scanning operation is detected. When the first malicious operation is detected and when the second malicious operation is detected, the process for notifying the detection of malicious activity is executed in different ways. The fraud detection program according to claim 1, which causes the computer to perform the processing.

10. When a payment start operation is performed on the self-checkout terminal to complete the scanning operation and begin payment, the self-checkout terminal receives a notification indicating that the payment start operation has been performed. When the aforementioned fraudulent activity is detected, the process of notifying the fraudulent activity is executed in a different manner depending on whether the settlement start operation has been performed or not. The fraud detection program according to claim 1, which causes the computer to perform the processing.

11. A storage unit that stores region information indicating the positions of multiple image regions set in a captured image of the front area of ​​a self-checkout terminal equipped with a scanner, A processing unit that recognizes a product from the captured image, detects any unauthorized actions related to a scan operation that causes the scanner to scan product information attached to the product, based on the movement path of the recognized product in the plurality of image regions and the time the product stays in the first image region closest to the scanner among the plurality of image regions. A fraud detection device having the following features.

12. A camera that photographs the area in front of the self-checkout terminal equipped with a scanner, An unauthorized detection device that recognizes a product from an image captured by the camera, detects unauthorized actions related to a scan operation that causes the scanner to scan product information attached to the product, based on the movement path of the product in a plurality of image regions set in the captured image and the time the product stays in the first image region closest to the scanner among the plurality of image regions, A fraud detection system having the following features.

Images