Operation detection system, operation detection method and program

Abstract

To provide an operation detection system, an operation detection method and a non-temporary computer readable medium, capable of favorably detecting a fraudulent act related to a checkout machine.SOLUTION: An operation detection system (10) includes: an operation specification unit (18) configured to start identifying at least one operation ID according to at least part of skeleton information extracted from image data imaging a user in response to a detection of a visit by the user to a prescribed checkout machine; a determination unit (20) configured to determine whether or not an operation sequence including the identified at least one operation ID corresponds to a pre-registered registration operation sequence; and a processing control unit (21) configured to execute prescribed processing according to a determination result.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a motion detection system, a motion detection method, and a program. [Background technology]

[0002] In recent years, self-checkout registers, which allow customers to check out by scanning the barcodes of products, have become widespread. In stores with few or no staff, there is a need to automatically detect fraudulent behavior by operators of self-checkout registers. The need to automatically detect fraudulent behavior by operators is not limited to self-checkout registers, but also applies to other devices that handle money.

[0003] For example, Patent Document 1 discloses a monitoring system that compares the actions of an operator of a cash center sorter with videos of fraudulent activity patterns to determine whether or not fraud has occurred, and transmits the facts of the detected fraud and evidence thereof to an administrator terminal. The management server of the monitoring system analyzes the stored video files and checks whether the video files contain image data that corresponds to the fraudulent activity patterns. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-080082 A Summary of the Invention [Problem to be solved by the invention]

[0005] If it is necessary to detect fraudulent activity by operators in real time, it is necessary to constantly analyze the video from the surveillance camera to check for the presence or absence of fraudulent activity. However, there is a problem in that continuous analysis processing increases the processing load and the amount of data to be processed becomes enormous.

[0006] In view of the above-mentioned problems, an object of the present disclosure is to provide a motion detection system, a motion detection method, and a non-transitory computer-readable medium that can effectively detect fraudulent activity related to a payment machine. [Means for solving the problem]

[0007] A motion detection system according to one aspect of the present disclosure includes: a motion identification means for starting to identify at least one motion ID based on at least a part of skeletal information extracted from video data of the user in response to detection that the user has visited a predetermined payment machine; A determination means for determining whether or not the identified motion sequence including the at least one motion ID corresponds to a registered motion sequence that has been registered in advance; A process control means for executing a predetermined process according to the result of the determination; Equipped with.

[0008] A motion detection method according to one aspect of the present disclosure includes: In response to detecting that a user has visited a predetermined payment machine, starting to identify at least one motion ID based on at least a part of skeletal information extracted from video data capturing the user; determining whether the identified motion sequence including the at least one motion ID corresponds to a registered motion sequence that has been registered in advance; A predetermined process is executed depending on the result of the determination.

[0009] According to one aspect of the present disclosure, there is provided a non-transitory computer-readable medium, comprising: a motion identification process that starts identifying at least one motion ID based on at least a part of skeletal information extracted from video data of the user in response to detecting that the user has visited a predetermined payment machine; a determination process for determining whether or not the identified motion sequence including the at least one motion ID corresponds to a registered motion sequence that has been registered in advance; A process control process for executing a predetermined process according to the result of the determination. A program for causing a computer to execute the above is stored. Effect of the Invention

[0010] The present disclosure provides a motion detection system, a motion detection method, and a non-transitory computer-readable medium that suitably detect fraudulent activity associated with a payment machine. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of a motion detection system according to a first embodiment. [Diagram 2] 4 is a flowchart showing a flow of a motion detection method according to the first embodiment. [Diagram 3] FIG. 11 is a diagram showing an overall configuration of a motion detection system according to a second embodiment. [Figure 4] FIG. 11 is a block diagram showing the configurations of a server and a terminal device according to the second embodiment. [Diagram 5] FIG. 11 is a diagram showing skeletal information extracted from a frame image included in video data according to the second embodiment. [Figure 6] FIG. 11 is a diagram showing skeletal information extracted from a frame image included in video data according to the second embodiment. [Figure 7] 10 is a flowchart showing the flow of a method for transmitting video data by a terminal device according to a second embodiment. [Figure 8] 13 is a flowchart showing the flow of a method for registering a registration action ID and a registration action sequence by a server according to the second embodiment. [Figure 9] FIG. 11 is a diagram for explaining a registration operation according to the second embodiment. [Figure 10] FIG. 11 is a diagram for explaining a normal operation sequence according to the second embodiment. [Figure 11] FIG. 11 is a diagram for explaining an unauthorized operation sequence according to the second embodiment. [Figure 12]10 is a flowchart showing the flow of a motion detection method by a server according to a second embodiment. [Figure 13] FIG. 13 is a block diagram showing the configuration of a server and a terminal device according to a third embodiment. [Figure 14] FIG. 11 is a diagram for explaining an unauthorized operation sequence according to the third embodiment. [Figure 15] FIG. 13 is a block diagram showing the configuration of a server and a terminal device according to a fourth embodiment; [Figure 16] 13 is a flowchart showing the flow of a motion detection method by a server according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the present disclosure will be described through embodiments, but the disclosure according to the claims is not limited to the following embodiments. In addition, all of the configurations described in the embodiments are not necessarily essential as means for solving the problems. In each drawing, the same elements are given the same reference numerals, and duplicated explanations are omitted as necessary.

[0013] <Embodiment 1> First, a first embodiment of the present disclosure will be described. FIG. 1 is a block diagram showing a configuration of a motion detection system 10 according to the first embodiment. The motion detection system 10 is a computer system that detects fraudulent behavior related to a settlement machine by a user who visits the settlement machine in a store. The settlement machine is a device that executes monetary processing based on the operation of an operator. The motion detection system 10 includes a motion identification unit 18, a determination unit 20, and a process control unit 21.

[0014] The action identification unit 18 is also called action identification means. In response to detecting that the user has visited a predetermined checkout machine, the action identification unit 18 starts identifying an action ID based on skeletal information extracted from video data of the user. The action ID is information for identifying an action. One action ID may identify an individual action such as "taking a product out of the basket," "scanning a product," or "putting a product into the basket." One action ID may also identify a series of actions that combine multiple individual actions, such as "taking a product out of the basket, scanning the product, and putting the product into the basket." The action ID identified by the action identification unit 18 from the video data may be one or multiple. The action identification unit 18 may identify the action ID based on all of the skeletal information extracted from the video data, or may identify the action ID based on a part of the skeletal information.

[0015] Here, "a user visiting the checkout machine" is also called a start trigger. The detection of the start trigger may be the detection of a person area from an image captured by a camera capturing the scenery around the checkout machine. In particular, the detection of the start trigger may be the detection of a person area recognized as that of the same person for a predetermined period of time. The start trigger may also be the detection of a basket containing products being placed on a predetermined platform from the image captured by the camera or the measurement value of a load sensor. The detection of the start trigger may also be the detection of an operation signal indicating that a checkout start button has been pressed. The detection of the start trigger may also be the acquisition of a voice indicating the start of checkout.

[0016] Then, the action identification unit 18 may end the identification of the action ID in response to detecting that the user has left the settlement machine. "The user has left the settlement machine" is also called an end trigger. The end trigger may be detected when a person area is no longer detected from an image of a camera that captures the scenery around the settlement machine. In particular, the end trigger may be detected when a person area that was recognized as the same person is no longer detected for a predetermined period of time after the start trigger. The end trigger may also be detected when an unscanned commodity is detected from an image of the camera or a measurement value of a load sensor that is detected from a basket placed on a predetermined placement table. The end trigger may also be detected when an operation signal indicating that a settlement end button has been pressed is detected. The end trigger may also be detected when a voice indicating the completion of settlement is obtained.

[0017] The judgment unit 20 is also called a judgment means. The judgment unit 20 judges whether or not an operation sequence including at least one identified operation ID corresponds to a registration operation sequence. A registration operation sequence defines normal actions related to a settlement machine, and its configuration is registered in advance in an operation sequence table (not shown). A registration operation sequence includes one or more registration operation IDs, which are operation IDs registered in advance in an operation database (DB) (not shown). In addition, when a registration operation sequence includes multiple registration operation IDs, the registration operation sequence includes a combination of the registration operation IDs and information on the chronological order in which each registration operation ID is performed. For example, a registration operation sequence is configured to include multiple registration operation IDs in chronological order.

[0018] The process control unit 21 is also called a process control means. The process control unit 21 executes a predetermined process according to the result of the above-mentioned determination. The predetermined process may be to notify the user of warning information, to notify the store staff of warning information, or to record that the above-mentioned operation sequence does not correspond to the registered operation sequence.

[0019] FIG. 2 is a flowchart showing the flow of the motion detection method according to the first embodiment. First, the motion detection system 10 judges whether or not the user has visited the settlement machine (S10). That is, the motion detection system 10 judges whether or not the start trigger has been detected. Then, when it is judged that the user has visited the settlement machine (Yes in S10), the motion identification unit 18 starts identifying the motion ID based on the skeletal information extracted from the video data (S11). The determination unit 20 judges whether or not the motion sequence including the motion ID identified in S11 corresponds to the registered motion sequence (S12). When the determination unit 20 judges that the motion sequence corresponds to the registered motion sequence (Yes in S12), it determines that a normal action has been performed and ends the process. On the other hand, when the determination unit 20 does not judge that the motion sequence corresponds to the registered motion sequence (No in S12), it determines that a fraudulent action has been performed and advances the process to S13. Then, the process control unit 21 executes a predetermined process (S13) and ends the process.

[0020] Thus, according to the first embodiment, the behavior detection system 10 detects fraudulent behavior related to the payment machine by determining whether the flow of the user's behavior when visiting the payment machine is normal. Therefore, even in a store with few or no staff, the fraudulent behavior of the user can be automatically detected.

[0021] Here, in the motion detection system 10, motion identification is performed after waiting for the user to visit the settlement machine, which reduces the processing load and the amount of data to be processed. In addition, the motion detection system 10 ends motion identification when the user leaves the settlement machine, which makes the above-mentioned effects even more pronounced. This allows the motion detection system 10 to effectively detect fraudulent behavior related to the settlement machine.

[0022] In addition, since the motion detection system 10 uses skeletal information to identify motion, there is no need to store specific pixel information, and therefore privacy can be ensured.

[0023] <Embodiment 2> Next, a second embodiment of the present disclosure will be described. Fig. 3 is a diagram showing an overall configuration of a motion detection system 1 according to the second embodiment. The motion detection system 1 is a computer system that monitors a user U who visits a settlement machine 50 and executes a predetermined process in response to detection of fraudulent activity.

[0024] As an example, the normal flow when a user U makes a payment at a cash register 50 in a store is as follows. (1) First, user U places products taken from the store's shelves in a basket and visits the payment machine 50 at the time of payment. (2) User U places the basket on the placement platform of the pre-scan basket 60. (3) User U removes the products from the pre-scan basket 60. (4) User U holds the hand scanner S of the payment machine 50 over the barcode of the product to scan it. (5) User U places the scanned products into the scanned basket 70. (6) User U repeats steps (3) to (5) until all pre-scanned products remain in the pre-scan basket 60. (7) After completing the scan, user U presses the end scan button. (8) User U pays for the scanned products. The place where the scanned product is placed is not limited to the scanned basket 70. For example, the user U may place the scanned product in a personal bag (a so-called my bag) or in any free space. The user U may also return the scanned product to the unscanned basket 60.

[0025] Here, the motion detection system 1 includes a server 100, a terminal device 200, and a skeleton camera 300. The server 100 and the terminal device 200 are communicatively connected via a network N. The network N may be wired or wireless.

[0026] The skeleton camera 300 is a camera that photographs the user U standing in front of the settlement machine 50 and monitors the user U. The skeleton camera 300 is disposed at a position and angle that allows it to photograph at least a part of the body of the user U standing in front of the settlement machine 50. In this embodiment 2, the skeleton camera 300 is configured to photograph the upper body of the user U.

[0027] The terminal device 200 acquires video data from the skeleton camera 300 and transmits the video data to the server 100 via the network N. The terminal device 200 also receives warning information indicating that the server 100 has detected a fraudulent act by the user U, and outputs the warning information using the display unit 203 or the audio output unit 204. The display unit 203 of the terminal device 200 may be installed in a position where it is easily visible to the user U or the store staff. The audio output unit 204 of the terminal device 200 may be installed in a position where it is easily heard by the user U or the store staff.

[0028] The server 100 is a computer device that detects fraudulent activity by the user U in relation to the settlement machine 50, based on the video data received from the terminal device 200. When the server 100 detects fraudulent activity, the server 100 transmits warning information to the terminal device 200 via the network N.

[0029] FIG. 4 is a block diagram showing the configuration of the server 100 and the terminal device 200 according to the second embodiment.

[0030] (Terminal device 200) The terminal device 200 includes a communication unit 201 , a control unit 202 , a display unit 203 , and an audio output unit 204 .

[0031] The communication unit 201 is also called a communication means. The communication unit 201 is a communication interface with the network N. The communication unit 201 is also connected to the skeleton camera 300, and acquires video data from the skeleton camera 300 at predetermined time intervals.

[0032] The control unit 202 is also called a control means. The control unit 202 controls the hardware of the terminal device 200. For example, when the control unit 202 detects a start trigger, it starts transmitting the video data acquired from the skeleton camera 300 to the server 100. The detection of the start trigger refers to the above-mentioned "detection that the user has visited the settlement machine." Also, for example, when the control unit 202 detects an end trigger, it stops transmitting the video data acquired from the skeleton camera 300 to the server 100. The detection of the end trigger refers to the above-mentioned "detection that the user U has left the settlement machine 50."

[0033] When the communication unit 201 receives warning information from the server 100, the control unit 202 causes the display unit 203 to display the warning information. The control unit 202 may also cause the audio output unit 204 to output the warning information.

[0034] The display unit 203 is a display device. The audio output unit 204 is an audio output device including a speaker.

[0035] (Server 100) The server 100 includes a registration information acquisition unit 101, a registration unit 102, an action DB 103, an action sequence table 104, an image acquisition unit 105, an extraction unit 107, an action identification unit 108, a generation unit 109, a determination unit 110, and a process control unit 111.

[0036] The registration information acquisition unit 101 is also called a registration information acquisition means. The registration information acquisition unit 101 acquires a plurality of registration video data in response to an action registration request from the terminal device 200 or an operation by an administrator of the server 100. In the present embodiment 2, each registration video data is video data showing an individual action (e.g., an action of taking a product out of a basket) included in a person's normal action or fraudulent action. Note that, in the present embodiment 2, the registration video data is a video including a plurality of frame images, but may be a still image (one frame image).

[0037] Furthermore, the registration information acquisition unit 101 acquires a plurality of registered action IDs and information on the chronological order in which the actions are performed in a series of actions in response to a sequence registration request from the terminal device 200 or an operation by the administrator of the server 100 .

[0038] The registration information acquisition unit 101 supplies the acquired information to the registration unit 102 .

[0039] The registration unit 102 is also called a registration means. First, the registration unit 102 executes an action registration process in response to an action registration request. Specifically, the registration unit 102 supplies registration video data to an extraction unit 107 (described later), and acquires skeleton information extracted from the registration video data from the extraction unit 107 as registered skeleton information. The registration unit 102 then registers the acquired registered skeleton information in the action DB 103 in association with a registered action ID.

[0040] Next, the registration unit 102 executes a sequence registration process in response to the sequence registration request. Specifically, the registration unit 102 arranges the registration operation IDs in chronological order based on the chronological order information to generate a registration operation sequence. At this time, if the sequence registration request is for a normal operation, the registration unit 102 registers the generated registration operation sequence in the operation sequence table 104 as a normal operation sequence NS. On the other hand, if the sequence registration request is for an unauthorized operation, the registration unit 102 registers the generated registration operation sequence in the operation sequence table 104 as an unauthorized operation sequence IS.

[0041] The action DB 103 is a storage device that stores registered skeleton information corresponding to each action included in normal actions in association with a registered action ID. The action DB 103 may also store registered skeleton information corresponding to each action included in fraudulent actions in association with a registered action ID.

[0042] The operation sequence table 104 stores a normal operation sequence NS and an unauthorized operation sequence IS. In the second embodiment, the operation sequence table 104 stores a plurality of normal operation sequences NS and a plurality of unauthorized operation sequences IS.

[0043] The image acquisition unit 105 is also called image acquisition means. The image acquisition unit 105 acquires video data captured by the frame camera 300 from the terminal device 200 during operation of the settlement machine 50. That is, the image acquisition unit 105 acquires video data in response to detection of a start trigger. The image acquisition unit 105 supplies the frame images included in the acquired video data to the extraction unit 107.

[0044] The extraction unit 107 is also called an extraction means. The extraction unit 107 detects an image area (body area) of a person's body from a frame image included in the video data, and extracts (e.g., cuts out) it as a body image. The extraction unit 107 then uses a skeleton estimation technique using machine learning to extract skeleton information of at least a part of the person's body based on the characteristics of the person's joints and the like recognized in the body image. The skeleton information is information composed of "key points" that are characteristic points of joints and the like, and "bones (bone links)" that indicate links between the key points. The extraction unit 107 may use a skeleton estimation technique such as OpenPose. The extraction unit 107 supplies the extracted skeleton information to the motion identification unit 108.

[0045] The action identification unit 108 is an example of the action identification unit 18 described above. The action identification unit 108 converts the skeleton information extracted from the video data acquired during operation into an action ID using the action DB 103. In this way, the action identification unit 108 identifies the action. Specifically, first, the action identification unit 108 identifies registered skeleton information having a similarity to the skeleton information extracted by the extraction unit 107 equal to or greater than a predetermined threshold from the registered skeleton information registered in the action DB 103. Then, the action identification unit 108 identifies the registered action ID associated with the identified registered skeleton information as the action ID corresponding to the person included in the acquired frame image.

[0046] Here, the motion identification unit 108 may identify one behavior ID based on skeletal information corresponding to one frame image, or may identify one behavior ID based on time-series data of skeletal information corresponding to each of a plurality of frame images. When identifying one behavior ID using a plurality of frame images, the motion identification unit 108 may extract only skeletal information with large movements and compare the extracted skeletal information with registered skeletal information in the motion DB 103. Extracting only skeletal information with large movements may mean extracting skeletal information with a difference of a predetermined amount or more between skeletal information of different frame images included in a predetermined period. Since only a small amount of comparison is required in this way, the calculation load can be reduced and the amount of registered skeletal information can be reduced. In addition, since the duration of a motion varies depending on a person, only skeletal information with large movements is subject to comparison, so that motion detection can be made robust.

[0047] In addition to the above-mentioned method, various other methods can be considered for identifying the action ID. For example, there is a method of estimating the action ID from the target video data using a motion estimation model trained on video data that has been corrected by the action ID as training data. However, it is difficult to collect this training data, and the cost is high. In contrast, in this embodiment 2, skeletal information is used to estimate the action ID, and the action DB 103 is used to compare it with pre-registered skeletal information. Therefore, in this embodiment 2, the server 100 can more easily identify the action ID.

[0048] The generating unit 109 is also called a generating means. The generating unit 109 generates an action sequence based on the multiple action IDs identified by the action identifying unit 108. The action sequence is configured to include the multiple action IDs in time series. The generating unit 109 supplies the generated action sequence to the determining unit 110.

[0049] The determination unit 110 is an example of the above-mentioned determination unit 20. The determination unit 110 determines whether the generated operation sequence matches (corresponds to) any of the normal operation sequences NS registered in the operation sequence table 104.

[0050] The process control unit 111 is an example of the above-mentioned process control unit 21. The process control unit 111 outputs warning information to the terminal device 200 when it is determined that the generated operation sequence does not correspond to any of the normal operation sequences NS.

[0051] If the determination unit 110 determines that the operation sequence does not correspond to any of the normal operation sequences NS, it may determine that the operation sequence corresponds to any of the unauthorized operation sequences. In this case, the process control unit 111 may output information that is predetermined according to the type of the unauthorized operation sequence to the terminal device 200. As an example, the display mode (font, color, thickness, or blinking of characters) when displaying warning information may be changed according to the type of the unauthorized operation sequence, or the volume or sound itself when outputting the warning information as sound may be changed. This allows the store staff to recognize the content of the unauthorized act and to deal with the unauthorized act promptly and appropriately. The process control unit 111 may also record the time, place, and video when the unauthorized act was performed as history information together with information on the type of the unauthorized operation sequence. This allows the store staff to recognize the content of the unauthorized act and to take appropriate preventive measures against the unauthorized act.

[0052] Fig. 5 is a diagram showing skeletal information extracted from a frame image 400 included in video data according to the second embodiment. The frame image 400 includes an image area of ​​the upper body of a user U captured from the front while the user U scans a product P1 using a hand scanner S. The skeletal information shown in Fig. 5 also includes a plurality of key points and a plurality of bones detected from the upper body. As an example, Fig. 5 shows the right ear A11, the left ear A12, the right eye A21, the left eye A22, the nose A3, the neck A4, the right shoulder A51, the left shoulder A52, the right elbow A61, the left elbow A62, the right hand A71, and the left hand A72 as key points.

[0053] The server 100 compares such skeletal information with registered skeletal information corresponding to the upper body and determines whether they are similar to each other, thereby identifying each motion. For example, in identifying a scanning motion, it is important whether the right hand and the left hand are close to each other, and in identifying a motion of "taking a product out of a basket" or "putting a product into a basket", the positions of the right hand and the left hand in the frame image 400 are important. Therefore, the server 100 may calculate the similarity by weighting the positions of the right hand A71 and the left hand A72. The server 100 may also calculate the similarity by weighting the right shoulder A51, the left shoulder A52, the right elbow A61, and the left elbow A62 in addition to the right hand A71 and the left hand A72.

[0054] The skeleton camera 300 may capture at least the hand region of the user U from above. FIG. 6 is a diagram showing skeleton information extracted from a frame image 500 according to the second embodiment. The frame image 500 includes an image area of ​​the hand region of the user U captured from above while the user U is scanning the product P1 using the hand scanner S. As an example, FIG. 6 shows a right hand A71 and a left hand A72 as key points. The server 100 may compare the skeleton information extracted from the frame image 500 with the registered skeleton information corresponding to the hand region and determine whether they are similar to each other to determine each motion.

[0055] 7 is a flowchart showing a flow of a method for transmitting video data by the terminal device 200 according to the second embodiment. First, the control unit 202 of the terminal device 200 judges whether or not a start trigger is detected (S20). When the control unit 202 judges that a start trigger is detected (Yes in S20), it starts transmitting the video data acquired from the skeleton camera 300 to the server 100 (S21). On the other hand, when the control unit 202 does not judge that a start trigger is detected (No in S20), it repeats the process shown in S20.

[0056] Next, the control unit 202 of the terminal device 200 judges whether or not an end trigger is detected (S22). When the control unit 202 judges that an end trigger is detected (Yes in S22), it ends the transmission of the video data acquired from the skeleton camera 300 to the server 100 (S23). On the other hand, when the control unit 202 does not judge that an end trigger is detected (No in S22), it repeats the process shown in S22 while transmitting the video data.

[0057] In this way, by limiting the transmission period of video data to the period between a predetermined start trigger and end trigger, the amount of communication data can be minimized. Also, outside the period, the motion detection process in the server 100 can be omitted, thereby saving computational resources.

[0058] FIG. 8 is a flowchart showing a flow of a method for registering a registration motion ID and a registration motion sequence by the server 100 according to the second embodiment. First, the registration information acquisition unit 101 of the server 100 receives a motion registration request including registration video data and a registration motion ID from the terminal device 200 (S30). Next, the registration unit 102 supplies the registration video data to the extraction unit 107. The extraction unit 107, which has acquired the registration video data, extracts a body image from a frame image included in the registration video data (S31). Next, the extraction unit 107 extracts skeletal information from the body image (S32). Next, the registration unit 102 acquires skeletal information from the extraction unit 107, and registers the acquired skeletal information as registered skeletal information in the motion DB 103 in association with the registration motion ID (S33). Note that the registration unit 102 may register all skeletal information extracted from the body image as registered skeletal information, or may register only a part of the skeletal information (for example, skeletal information of a shoulder, elbow, and hand) as registered skeletal information.

[0059] FIG. 9 is a diagram for explaining the registration operations according to the second embodiment. As an example, the operation DB 103 may store registered skeleton information of five registered operations having registered operation IDs "A" to "E". The registered operation "A" is an operation of removing a product from the pre-scan basket 60. The registered operation "B" is an operation of scanning a product using the hand scanner S. The registered operation "C" is an operation of putting a product into the scanned basket 70. The registered operation "D" is an operation of pressing the scan end button. The registered operation "E" is an operation of inputting the quantity into the settlement machine.

[0060] Returning to Fig. 8, the explanation will be continued. Next, the registration information acquisition unit 101 receives a sequence registration request including a plurality of registration action IDs and information on the chronological order of each action from the terminal device 200 (S34). Next, the registration unit 102 registers a registration action sequence (normal action sequence NS or incorrect action sequence IS) in which the registration action IDs are arranged based on the information on the chronological order in the action sequence table 104 (S35). Then, the server 100 ends the process.

[0061] FIG. 10 is a diagram for explaining a normal operation sequence NS according to the second embodiment. As an example, the operation sequence table 104 may include at least four normal operation sequences NS having normal operation sequence IDs of "11" to "14". The normal operation sequence "11" is a sequence (A→B→C→D) in which the scan is completed after one scan. The normal operation sequence "12" is a sequence (A→B→C→A→B→C→D) in which the scan is completed after two scans. The normal operation sequence "13" is a sequence (A→B→C→E→D) in which the quantity is input after one scan. The normal operation sequence "14" is a sequence (A→B→C→E→A→B→C→D) in which the quantity is input after one scan, and then the scan is completed after another scan.

[0062] FIG. 11 is a diagram for explaining the unauthorized operation sequence IS according to the second embodiment. The operation sequence table 104 may include at least two unauthorized operation sequences NS having the unauthorized operation sequence IDs "21" and "22". The unauthorized operation sequence "21" is a sequence (? → A → C → ?) including an operation of placing an item from the unscanned basket 60 to the scanned basket 70 without scanning it. Note that "?" indicates any operation. The unauthorized operation sequence "22" is a sequence in which none of the registered operation IDs are identified between the start trigger and the end trigger, and indicates, for example, a case in which a user U stops by the settlement machine 50 but leaves without doing anything.

[0063] FIG. 12 is a flowchart showing a flow of the action detection method by the server 100 according to the second embodiment. First, when the image acquisition unit 105 of the server 100 starts acquiring video data from the terminal device 200 (Yes in S40), the extraction unit 107 extracts a body image from a frame image included in the video data (S41). Next, the extraction unit 107 extracts skeletal information from the body image (S42). The action identification unit 108 calculates the similarity between at least a part of the extracted skeletal information and each registered skeletal information registered in the action DB 103, and identifies, as the action ID, a registered action ID associated with registered skeletal information whose similarity is equal to or greater than a predetermined threshold (S43). Next, the generation unit 109 adds the action ID to the action sequence. Specifically, in the first cycle, the generation unit 109 sets the action ID identified in S43 as the action sequence, and in the next and subsequent cycles, adds the action ID identified in S43 to the already generated action sequence. The server 100 then determines whether the scan has ended or the acquisition of the video data has ended (S45). Note that the server 100 may determine that the scan has ended if the action identified in S43 of the current cycle is the action of registered action ID "D". If the server 100 determines that the scan has ended or the acquisition of the video data has ended (Yes in S45), the process proceeds to S46, and if not (No in S45), the process returns to S41 and repeats the process of adding the action sequence.

[0064] In S46, the determination unit 110 determines whether or not the operation sequence corresponds to any normal operation sequence NS in the operation sequence table 104. If the operation sequence corresponds to a normal operation sequence NS (Yes in S46), the determination unit 110 proceeds to S49, and if not (No in S46), the determination unit 110 proceeds to S47.

[0065] In S47, the determination unit 110 determines the type of unauthorized behavior by determining whether the operation sequence corresponds to any of the unauthorized operation sequences IS in the operation sequence table 104. Then, the process control unit 111 transmits warning information according to the type of unauthorized behavior to the terminal device 200 (S48). Then, the server 100 advances the process to S49.

[0066] In S49, the server 100 determines whether or not the acquisition of the video data has been completed. When the server 100 determines that the acquisition of the video data has been completed (Yes in S49), the server 100 ends the process. On the other hand, when the server 100 does not determine that the acquisition of the video data has been completed (No in S49), the server 100 returns the process to S41 and repeats the additional process of the operation sequence. By returning the process to S41, it is possible to monitor the operation from the end of the scan until the user U leaves the settlement machine 50.

[0067] Thus, according to the second embodiment, the server 100 judges whether the actions of the user U who visited the settlement machine 50 are normal or not by comparing the action sequence showing the flow of the actions of the user U who visited the settlement machine 50 with the normal action sequence NS. In this way, by registering in advance a plurality of normal action sequences NS that correspond to the flow of operations using the settlement machine 50, it is possible to realize detection of fraudulent actions that correspond to the actual situation. Note that the second embodiment also achieves the same effects as the first embodiment.

[0068] <Embodiment 3> Next, a third embodiment of the present disclosure will be described. The third embodiment is characterized in that audio data is used in addition to video data to identify a predetermined motion. For example, the predetermined motion is a scan motion.

[0069] 13 is a block diagram showing the configurations of the server 100a and the terminal device 200a according to the third embodiment. The terminal device 200a differs from the terminal device 200 in that it includes a sound collection unit 205. The sound collection unit 205 collects sounds around the settlement machine 50. When transmitting video data to the server 100a, the control unit 202 also transmits sound data to the server 100a.

[0070] The server 100a includes a voice acquisition unit 112 and a motion identification unit 108a instead of the motion identification unit 108. The voice acquisition unit 112 acquires voice data from the terminal device 200 and supplies the voice data to the motion identification unit 108a.

[0071] The action identification unit 108a identifies an action ID based on the skeleton information extracted by the extraction unit 107, the registered skeleton information in the action DB 103, and the voice data acquired by the voice acquisition unit 112. For example, when the similarity between the skeleton information extracted by the extraction unit 107 and the registered skeleton information of the registered action ID indicating a predetermined action is equal to or greater than a predetermined threshold, the action identification unit 108a determines whether or not the voice data includes a predetermined voice. When the voice data includes a predetermined voice, the action identification unit 108a identifies the registered action ID indicating the above action as the action ID. On the other hand, when the predetermined voice is not included, the action identification unit 108a does not identify the registered action ID indicating the above action as the action ID.

[0072] For example, the predetermined action may be an action of scanning a product. In this case, if the similarity between the extracted skeleton information and the registered skeleton information of the scanning action is equal to or greater than a predetermined threshold, the action identifying unit 108a determines whether or not the voice data contains an electronic sound generated when scanning is performed normally. If the electronic sound is not contained, the action identifying unit 108a does not identify the registered action ID of the scanning action as the action ID. In this case, the action identifying unit 108a may identify an action "B'" in which the user U only pretends to scan but does not actually scan. This action includes, for example, an action of holding the hand scanner S over a surface of a product that does not have a barcode.

[0073] Fig. 14 is a diagram for explaining an unauthorized operation sequence according to the third embodiment. In Fig. 14, an unauthorized operation sequence "23" is added in addition to the unauthorized operation sequences "21" to "22". The unauthorized operation sequence "23" is a sequence (?→A→B'→C→?) including an operation "B'" in which the user U pretends to scan but does not actually scan.

[0074] In this way, the server 100a can specify a behavior by combining skeletal information and voice, thereby increasing the variety of unauthorized behaviors to be detected.

[0075] <Embodiment 4> Next, a fourth embodiment of the present disclosure will be described. The fourth embodiment is characterized in that the server determines the action sequence based on the number of items (products) that the user U has brought out.

[0076] 15 is a block diagram showing the configuration of the server 100b and the terminal device 200 according to the fourth embodiment. The terminal device 200 acquires video data from the commodity camera 350 in addition to the skeleton camera 300. The commodity camera 350 is a camera that photographs the commodity brought out by the user U. For example, the commodity camera 350 may be disposed in a position where it can photograph at least one of the unscanned basket 60 and the scanned basket 70 from above.

[0077] The server 100b differs from the server 100 in that, instead of the determining unit 110, the server 100b includes a product number specifying unit 113 and a determining unit 110b.

[0078] The product number determination unit 113 is also called a product number determination means. The product number determination unit 113 acquires the image of the product camera 350 from the terminal device 200, and determines the number of products by an object detection method based on the image of the product camera 350. For example, when the product camera 350 is a camera that takes an image from above the basket 60 before scanning, the product number determination unit 113 may determine the number of products from the image of the product camera 350 acquired in response to a start trigger. Also, for example, when the product camera 350 is a camera that takes an image from above the basket 70 after scanning, the product number determination unit 113 may determine the number of products from the image of the product camera 350 acquired in response to an end trigger. In this case, the product number determination unit 113 may use the image of the product camera 350 acquired when the weight of the basket 60 before scanning becomes less than a predetermined threshold, instead of the end trigger, to determine the number of products.

[0079] The determination unit 110b determines whether the operation sequence generated by the generation unit 109 corresponds to a registered operation sequence corresponding to the number of products among the registered operation sequences. For example, if the number of products is two, the determination unit 110b compares the generated operation sequence with a normal operation sequence NS corresponding to the number of products of two (normal operation sequences "12" or "13" in FIG. 10) and determines whether there is a normal operation sequence NS corresponding to the generated operation sequence.

[0080] 16 is a flowchart showing the flow of a motion detection method by the server 100b according to embodiment 4. The steps shown in FIG. 16 include S50 to S51 instead of S46 shown in FIG.

[0081] If the server 100b determines in S45 that the scan has ended or that the acquisition of video data has ended (Yes in S45), the process proceeds to S50. In S50, the product number determination unit 113 determines the number of products based on the video of the product camera 350, and determines a normal operation sequence NS corresponding to the determined number of products from among the normal operation sequences NS included in the operation sequence table 104. Then, in S51, the determination unit 110b determines whether or not the operation sequence corresponds to the determined normal operation sequence NS. If the operation sequence corresponds to the determined normal operation sequence NS (Yes in S51), the determination unit 110b proceeds to S49, and if not (No in S51), the process proceeds to S47.

[0082] As described above, according to the fourth embodiment, the server 100b uses the number of products to determine the operation sequence, which enables more detailed determination and improves the accuracy of the determination. As a result, even if the user U holds multiple products in his / her hand and scans only one of them, it is possible to detect the operation as an unauthorized operation.

[0083] The present disclosure is not limited to the above-described embodiments, and may be modified as appropriate without departing from the spirit and scope of the present disclosure. For example, the third and fourth embodiments may be combined.

[0084] Also, for example, in the above-mentioned second to fourth embodiments, the registration information acquisition unit 101 acquires registration video data indicating individual actions when registering an action, and the registration unit 102 registers skeletal information and an action ID for each registration video data in the action DB 103. However, the registration information acquisition unit 101 may acquire registration video data indicating a series of actions including a plurality of individual actions (for example, actions of taking a product out of a basket, scanning the product, and putting the product into the basket) and information on the order of appearance of each individual action. Then, the registration unit 102 may extract skeletal information for each individual action included in the registration video data, and register the skeletal information in the action DB 103 together with an action ID corresponding to each individual action.

[0085] In the above-described second to fourth embodiments, the servers 100, 100a, and 100b perform the extraction process, the action identification process, the sequence generation process, and the determination process. However, some or all of the processes may be performed by the terminal device 200, or may be performed by an external device (not shown) connected to the network N.

[0086] Although the above-described embodiment has been described as a hardware configuration, the present disclosure is not limited to this. Any process of the present disclosure can be realized by causing a processor to execute a computer program.

[0087] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals. [Explanation of symbols]

[0088] 1,1b,10 Motion Detection System 18 Operation specific part 20 Judgment section 21 Processing control section 50 Payment machine 60 Scanning Basket 70 scanned baskets 100,100a,100b servers 101 Registration Information Acquisition Department 102 Registration Department 103 Operation DB 104 Operation sequence table 105 Image acquisition unit 107 Extraction part 108,108a Operation specific part 109 Generation part 110,110b Judgment section 111 Processing control section 112 Voice Acquisition Unit 113 Product number specification section 200,200a Terminal equipment 201 Communications Department 202 Control section 203 Display section 204 Audio output section 205 Sound pickup section 300 Skeleton Camera 350 Product Camera 400,500 frame images S Hand Scanner N Network

Claims

1. a motion identification means for starting to identify at least one motion ID based on at least a part of skeletal information extracted from video data of the user in response to detection that the user has visited a predetermined payment machine; A determination means for determining whether or not the action sequence including the at least one identified action ID corresponds to a registered action sequence that has been registered in advance; A process control means for executing a predetermined process according to a result of the determination, When the operation sequence does not correspond to the registered operation sequence, the processing control means outputs information predetermined according to the type of the operation sequence in a manner according to the type of the operation sequence. Motion detection system.

2. The manner includes at least one of a display manner and an audio manner. The motion detection system of claim 1 .

3. When the operation sequence does not correspond to the registered operation sequence, the processing control means records data regarding the operation sequence as history information. The motion detection system according to claim 1 or 2.

4. The action identification means ends the identification of the at least one action ID in response to detecting that the user has left the settlement machine. The motion detection system according to claim 1 or 2.

5. The action identification means, in response to detecting that the user has visited the payment machine, starts identifying a plurality of action IDs based on at least a part of skeletal information extracted from video data capturing the user; The determination means determines whether or not an action sequence including the identified plurality of action IDs in chronological order corresponds to the registered action sequence; The process control means executes the predetermined process when the operation sequence does not correspond to the registered operation sequence. The motion detection system according to claim 1 or 2.

6. In response to detecting that a user has visited a predetermined payment machine, starting to identify at least one motion ID based on at least a part of skeletal information extracted from video data capturing the user; determining whether or not the action sequence including the identified at least one action ID corresponds to a registered action sequence that has been registered in advance; Executing a predetermined process according to the result of the determination; When the operation sequence does not correspond to the registered operation sequence, information predetermined according to a type of the operation sequence is output in a display mode according to the type of the operation sequence. Motion detection methods.

7. a motion identification process that starts identifying at least one motion ID based on at least a part of skeletal information extracted from video data of the user in response to detection that the user has visited a predetermined payment machine; a determination process for determining whether or not the action sequence including the at least one action ID identified corresponds to a registered action sequence that has been registered in advance; A process control process for executing a predetermined process according to the result of the determination. an output process of outputting information predetermined according to a type of the operation sequence in a display mode according to the type of the operation sequence when the operation sequence does not correspond to the registered operation sequence; A program for causing a computer to execute the following.

Images