Estimation program, estimation method, and estimation device

Abstract

To solve a problem in which, because images captured by surveillance cameras in retail stores are two-dimensional images, a fraud detection system cannot estimate a scale of a person if a positional relationship between the person and a cart is unknown, such that the fraud detection system may not be able to correctly recognize a person's behavior from the image captured by the surveillance camera.SOLUTION: An estimation program instructs a computer to perform the processing of: identifying a first person using a first cart from a first image captured in a store, generating skeletal information of the first person; obtaining, using the skeletal information, a first interval at which the first person grasps a grasping portion of the first cart; and estimating a first scale of the first person based on the first interval and the length information of the first cart.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to an estimation program, an estimation method, and an estimation device. [Background technology]

[0002] In order to reduce congestion at cash registers, retail stores are increasingly introducing systems in which customers themselves scan and register products and then check out. Furthermore, in recent years, systems have begun to be introduced in which customers use terminals provided by retail stores or applications installed on their own terminals to scan products at locations other than cash registers, such as the sales floor where they pick up each product. In such systems in which customers themselves scan products, it is important to correctly recognize and detect customer actions, such as the actions of customers picking up products and putting them in a shopping cart and the actions of scanning the products, in order to detect fraudulent actions such as shoplifting. Note that a shopping cart is sometimes simply called a "cart."

[0003] Meanwhile, systems have been developed to detect fraudulent behavior by customers in retail stores, for example, systems that use surveillance cameras in the store to detect suspicious behavior by customers and fraudulent behavior such as shoplifting. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2014-524071 [Patent Document 2] JP 2020-173816 A [Patent Document 3] US Patent Application Publication No. 2020 / 0118401 Summary of the Invention [Problem to be solved by the invention]

[0005] However, since the images (strictly speaking, images) captured by security cameras are two-dimensional, if the positional relationship between the person and the cart is unknown, the fraud detection system cannot estimate the scale of the person. Therefore, the fraud detection system may not be able to correctly recognize the behavior of the person from the images captured by the security cameras.

[0006] According to one aspect, an object of the present invention is to provide an estimation program, an estimation method, and an estimation device that can more accurately estimate the scale of a person from a captured image. [Means for solving the problem]

[0007] In one embodiment, the estimation program causes a computer to execute a process of identifying a first person using a first cart from a first image taken of the interior of the store, generating skeletal information of the first person, obtaining a first distance at which the first person is holding the handle of the first cart using the skeletal information, and estimating a first scale of the first person based on the first distance and information on the length of the first cart. Effect of the Invention

[0008] In one aspect, the scale of a person can be more accurately estimated from a captured image. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration example of the fraud detection system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of product purchase by self-scanning according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating another example of purchasing a product by self-scanning according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the fraud detection process according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of object detection and skeleton detection according to the first embodiment. [Figure 6]FIG. 6 is a diagram showing an example of skeleton information. [Figure 7] FIG. 7 is a diagram for explaining whole body posture determination. [Figure 8] FIG. 8 is a diagram for explaining problems that arise when behavior is recognized from a captured image. [Figure 9] FIG. 9 is a diagram illustrating an example of a configuration of the estimation device 10 according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the scale estimation process according to the first embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of the three-dimensional position and orientation acquisition process of the cart according to the first embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of a 2D coordinate / 3D coordinate correspondence relationship for calculating a homography according to the first embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of setting the reference position according to the first embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of a positional relationship between 2D / 3D coordinate systems according to the first embodiment. [Figure 15] FIG. 15 is a diagram illustrating an example of absolute 3D skeleton position estimation by world coordinate mapping of a 3D skeleton according to the first embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of the operation detection process according to the first embodiment. [Figure 17] FIG. 17 is a diagram illustrating another example of the operation detection process according to the first embodiment. [Figure 18] FIG. 18 is a flowchart of the scale estimation process according to the first embodiment. [Figure 19] FIG. 19 is a flowchart of the operation detection process according to the first embodiment. [Figure 20] FIG. 20 is a diagram illustrating an example of the hardware configuration of the estimation device 10. As shown in FIG. [Figure 21] FIG. 21 is a diagram illustrating an example of the hardware configuration of the information processing terminal 1000. As shown in FIG. [Figure 22] FIG. 22 is a diagram illustrating an example of the hardware configuration of the self-checkout terminal 400. As shown in FIG. [Diagram 23] FIG. 23 is a diagram illustrating an example of the hardware configuration of the gate reader 500. As shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Below, examples of the estimation program, estimation method, and estimation device according to the present embodiment will be described in detail with reference to the drawings. Note that the present embodiment is not limited to these examples. In addition, each example can be appropriately combined within a range that does not cause inconsistency. EXAMPLES

[0011] First, a fraud detection system for implementing this embodiment will be described. Fig. 1 is a diagram showing a configuration example of the fraud detection system according to Example 1. As shown in Fig. 1, the fraud detection system 1 is a system in which an estimation device 10 and user terminals 100-1 to 100-n (n is an arbitrary integer; hereinafter, collectively referred to as "user terminals 100") are connected to each other via a network 50 so as to be able to communicate with each other.

[0012] The estimation device 10 is also connected to camera devices 200-1 to 200-m (m is an arbitrary integer; hereinafter, collectively referred to as "camera devices 200") and a store clerk terminal 300 via a network 50 so as to be able to communicate with each other.

[0013] Network 50 may be any of a variety of communication networks, whether wired or wireless, such as an intranet used within a retail store. Network 50 may not be a single network, but may be configured, for example, by connecting an intranet and the Internet via a network device such as a gateway or other device (not shown). The interior of a retail store is not limited to indoors, and may include outdoors within the premises.

[0014] The estimation device 10 is, for example, an information processing device such as a desktop PC (Personal Computer), a notebook PC, or a server computer that is installed in a retail store and used by store staff, managers, and the like.

[0015] The estimation device 10 receives a plurality of images captured by the camera device 200 of a predetermined imaging range, such as the inside of a retail store or on the premises, from the camera device 200. Note that the plurality of images, strictly speaking, are images captured by the camera device 200, i.e., a series of frames of a video.

[0016] Furthermore, the estimation device 10 uses existing object detection technology to identify customers (hereinafter, sometimes simply referred to as "people") staying in a store, shopping baskets (hereinafter, sometimes simply referred to as "baskets") and shopping carts carried by the people, and the user terminal 100 from the captured image. Furthermore, the estimation device 10 uses existing skeleton detection technology to generate skeleton information of the identified people from the captured image to estimate the position, posture, and scale of the people, and detect actions such as holding a cart and putting products into the basket or cart. Here, the scale is, for example, a measure of the height of a person, and the estimation device 10 estimates the scales of various people appearing in the captured image based on information such as the length of the cart appearing in the captured image. As a result, the estimation device 10 can estimate, for example, the positional relationship between a person and a cart based on the scale of the person, and can improve the accuracy of person behavior recognition.

[0017] 1, the estimation device 10 is illustrated as a single computer, but may be a distributed computing system configured with a plurality of computers. The estimation device 10 may also be a cloud computing device managed by a service provider that provides a cloud computing service.

[0018] The user terminal 100 is an information processing terminal for a customer to scan the barcode of a product and register the purchased product in order to purchase the product. The user terminal 100 may be a mobile terminal such as a smartphone or tablet PC (Personal Computer) owned by the customer, or may be a dedicated terminal rented out in a store. An application for scanning and registering products is pre-installed in the user terminal 100.

[0019] The camera device 200 is, for example, a surveillance camera installed inside a retail store or on the premises. Although multiple camera devices 200 are shown in FIG. 1, for example, in a small store, only one camera device may be used. The video captured by the camera device 200 is transmitted to the estimation device 10.

[0020] The clerk terminal 300 may be a mobile terminal such as a smartphone or tablet PC carried by a retail store clerk, or may be an information processing device such as a desktop PC or notebook PC installed at a predetermined position in the store. The clerk terminal 300 receives an alert from the estimation device 10 when the estimation device 10 detects fraudulent behavior of a customer, such as failure to scan a product. Note that, for example, there may be multiple clerk terminals 300 for each clerk in the store, but the terminal to which the alert is notified may be limited to, for example, a terminal carried by a clerk in charge of security near the exit.

[0021] Next, a method in which a customer scans and registers a product by himself / herself (hereinafter, may be referred to as "self-scanning") and purchases the product will be described with reference to Fig. 2 and Fig. 3. Fig. 2 is a diagram showing an example of product purchase by self-scanning according to the first embodiment.

[0022] As shown in Fig. 2, first, the customer selects the store to be visited via an application displayed on the user terminal 100. Next, the customer picks up the product to be purchased and reads, for example, a barcode for each product attached to the product or a product shelf using the user terminal 100 (hereinafter, sometimes referred to as "product scanning"). As a result, the product to be purchased is registered in the application.

[0023] Next, the customer scans the payment code displayed on the display unit of the self-checkout terminal 400 or the like. The customer then pays the amount displayed on the checkout screen of the self-checkout terminal 400 to complete the purchase of the product. The customer can then leave the store by having the payment completion code displayed on the user terminal 100 read by a gate reader 500 or the like installed at the exit of the store. Although not shown in FIG. 1, the self-checkout terminal 400 and the gate reader 500 are connected to the estimation device 10 via a network 50 so as to be able to communicate with each other.

[0024] Next, another example of purchasing a product by self-scanning will be described below. Fig. 3 is a diagram showing another example of purchasing a product by self-scanning according to the first embodiment.

[0025] As shown in Fig. 3, first, the customer logs in to an application displayed on the user terminal 100 and selects the store to be visited. Next, the customer picks up the product to be purchased and uses the user terminal 100 to read, for example, a barcode for each product affixed to the product to be purchased or to a product shelf.

[0026] Next, the customer places the basket containing the purchased items at a checkpoint in the store and completes the payment for the purchased items by pressing a "checkout button" or a "purchase button" displayed on the user terminal 100. Payment for the purchased items can be made by electronic money, credit card, or the like via an application displayed on the user terminal 100. The customer can then leave the store by having a gate reader 500 installed at the store's exit or the like read the payment completion code displayed on the user terminal 100.

[0027] The above describes the purchase of goods by self-scanning, using FIG. 2 and FIG. 3. However, with self-scanning, for example, a customer can put goods in a basket without scanning the goods, and avoid paying without going through a self-checkout. Or, a customer can scan only some of the purchased goods and pay for only the scanned goods at the self-checkout, and avoid paying for some of the goods. In particular, fraudulent behavior when the number of goods is small is easily detected by a store clerk, but it is difficult for a store clerk to detect fraudulent behavior when the number of goods is large and some of the goods are not scanned, for example. Therefore, the fraud detection system recognizes the behavior of the customer from the image captured by the camera device 200 and detects fraudulent behavior. A fraud detection process by the fraud detection system will be described.

[0028] FIG. 4 is a diagram illustrating an example of fraud detection processing according to the first embodiment. A captured image 250 illustrated in FIG. 4 is an example of a captured image of the inside of a retail store captured by the camera device 200. The estimation device 10 identifies a person or an object from the captured image 250. Next, based on the person and cart identified from the captured image 250, the estimation device 10 determines the person's behavior, such as whether the person put an item into a cart or scanned an item. Then, the estimation device 10 detects a person who is engaging in fraudulent behavior, such as not scanning an item, as a target, and notifies an alert to the store clerk terminal 300.

[0029] Here, a method for identifying a person or an object from a captured image will be specifically described with reference to FIG. 5. FIG. 5 is a diagram showing an example of object detection and skeleton detection according to the first embodiment. As shown in the upper right of FIG. 5, the estimation device 10 detects and identifies a person or a cart from a captured image 251 using, for example, an existing object detection algorithm. Here, the existing object detection algorithm is, for example, an object detection algorithm using deep learning such as Faster R-CNN (Convolutional Neural Network). Alternatively, the object detection algorithm may be, for example, YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector).

[0030] In addition to people and baskets, for example, products, the user terminal 100, sales areas for products such as sales floors and shelves, and people's clothing may be detected from the captured image. This allows the estimation device 10 to detect, for example, that a user has the user terminal 100 but does not perform a self-scan. The estimation device 10 can also determine the age of a person identified from a captured image using, for example, an existing algorithm, and identify a group relationship between people, such as a parent-child relationship. This allows, for example, in the case of a parent-child relationship, it may be determined that no product has been missed if one of the target people scans the product.

[0031] 5, the estimation device 10 detects the skeleton of the person identified from the captured image 251 by using, for example, an existing skeleton estimation algorithm. Here, the existing skeleton estimation algorithm is, for example, a skeleton estimation algorithm using deep learning such as HumanPoseEstimation such as DeepPose and OpenPose.

[0032] For example, the estimation device 10 obtains skeletal information by inputting image data (each frame) into a trained machine learning model. FIG. 6 is a diagram showing an example of skeletal information. The skeletal information can use 18 definition information (0 to 17) in which each joint specified by a known skeletal model is numbered. For example, the right shoulder joint (SHOULDER_RIGHT) is assigned the number 7, the left elbow joint (ELBOW_LEFT) is assigned the number 5, the left knee joint (KNEE_LEFT) is assigned the number 11, and the right hip joint (HIP_RIGHT) is assigned the number 14. Therefore, the estimation device 10 can obtain the coordinate information of the 18 skeletons shown in FIG. 6 from the image data. For example, the estimation device 10 obtains "X coordinate=X7, Y coordinate=Y7, Z coordinate=Z7" as the position of the right shoulder joint number 7. For example, the Z axis can be defined as the distance direction from the imaging device toward the target, the Y axis as the height direction perpendicular to the Z axis, and the X axis as the horizontal direction.

[0033] In addition, the estimation device 10 can determine the posture of the whole body, such as standing, walking, crouching, sitting, and lying down, by using a machine learning model that has been trained in advance on a skeletal pattern. For example, the estimation device 10 can determine the closest posture of the whole body by using a machine learning model that has been trained with Multilayer Perceptron on some joints and angles between joints, such as the skeletal information and the beauty and technique diagram in FIG. 6. FIG. 7 is a diagram for explaining the posture determination of the whole body. As shown in FIG. 7, the estimation device 10 can detect the posture of the whole body by acquiring the joint angle (a) between "HIP_LEFT" at No. 10 and "KNEE_LEFT" at No. 11, the joint angle (b) between "HIP_RIGHT" at No. 14 and "KNEE_RIGHT" at No. 15, the angle (c) of "KNEE_LEFT" at No. 11, and the angle (d) of "KNEE_RIGHT" at No. 15, etc.

[0034] In addition, the estimation device 10 may estimate the posture using a machine learning model such as a Multi Layer Perceptron, which is generated by machine learning using some joints and angles between the joints as features and whole-body postures such as standing and crouching as correct labels.

[0035] The estimation device 10 may use, as a posture estimation algorithm, 3D Pose Estimation such as VNect that estimates a three-dimensional posture from one captured image. The estimation device 10 may also estimate a posture from three-dimensional joint data using, for example, a 3d-pose-baseline that generates three-dimensional joint data from two-dimensional skeletal information.

[0036] Furthermore, the estimation device 10 may estimate the posture of a person by identifying the motion of each body part based on the orientation of each body part, such as the face, arms, elbows, etc., and the angle when the body part is bent. Note that the algorithm for posture estimation and skeleton estimation is not limited to one type, and the posture and skeleton may be estimated in a composite manner using multiple algorithms.

[0037] However, since the image captured by the camera device 200 is a two-dimensional image, if the positional relationship between the person and the cart is unknown, the estimation device 10 cannot estimate the scale of the person, and therefore cannot correctly recognize the behavior of the person from the captured image.

[0038] Fig. 8 is a diagram for explaining problems in behavior recognition from a captured image. As shown in Fig. 8, for example, when a person 150 and a cart 160 are overlapped in a captured image 251, the positional relationship between the person 150 and the cart 160 is unclear, and therefore, it may not be correctly recognized whether the person 150 is holding the cart 160 or not. In other words, assuming that the captured image 251 is captured from the front and viewed from the side, as shown in the lower part of Fig. 8, it may not be recognized from the captured image whether the person 150 is right behind the cart 160 or behind the cart 160.

[0039] Here, information on the length of the carts 160 handled in the store is known in advance and can be stored in the estimation device 10, so if the scale of the person 150 is known, the estimation device 10 can estimate the positional relationship between the person 150 and the cart 160. Therefore, one of the objectives of this embodiment is to more accurately estimate the scale of a person from an image captured by the camera device 200 of the inside of a retail store or on the premises, and ultimately to more accurately recognize the behavior of the person.

[0040] [Functional configuration of the estimation device 10] Next, a functional configuration of the estimation device 10 which is an execution subject of the present embodiment will be described. Fig. 9 is a diagram illustrating an example of the configuration of the estimation device 10 according to Example 1. As illustrated in Fig. 9, the estimation device 10 includes a communication unit 20, a storage unit 30, and a control unit 40.

[0041] The communication unit 20 is a processing unit that controls communication with other devices such as the user terminal 100 and the camera device 200, and is, for example, a communication interface such as a USB (Universal Serial Bus) interface or a network interface card.

[0042] The storage unit 30 has a function of storing various data and programs executed by the control unit 40, and is realized by a storage device such as a memory or a hard disk. For example, the storage unit 30 stores a human skeleton 3D model for estimating three-dimensional skeleton information of a person, an object 3D model for estimating the posture of an object, and the like.

[0043] The storage unit 30 also stores a plurality of captured images, which are a series of frames captured by the camera device 200. The storage unit 30 can also store position information in the captured images of people and objects identified from the captured images. The storage unit 30 also stores two-dimensional skeletal information of people identified from the captured images captured by the camera device 200. The storage unit 30 also stores length information of the cart, such as the length of the handle of the cart.

[0044] The above information stored in the storage unit 30 is merely an example, and the storage unit 30 can store various information other than the above information.

[0045] The control unit 40 is a processing unit, such as a processor, that controls the entire estimation device 10. The control unit 40 includes an image acquisition unit, an object detection unit, an object pose estimation unit, a human region detection unit, a 2D skeleton estimation unit, a scale estimation unit, a relative 3D skeleton estimation unit, an absolute 3D skeleton mapping unit, a motion determination unit, etc. Each processing unit is an example of an electronic circuit included in a processor or an example of a process executed by a processor.

[0046] The image acquisition unit acquires from the camera device 200 a plurality of captured images, which are a series of frames captured by the camera device 200 .

[0047] The object detection unit uses existing technology such as YOLO to detect an object from an image captured by the camera device 200. The detected object may be represented by a bounding box.

[0048] The object posture estimation unit inputs, for example, a partial image within a bounding box of the cart in the captured image and CAD (Computer Aided Design) data of the cart to a machine learning model to acquire three-dimensional position and posture data of the cart. Note that the machine learning model is a machine learning model generated by machine learning using, for example, a captured image of the cart and the CAD data of the cart as feature quantities and the three-dimensional position and posture data of the cart as correct answer labels.

[0049] The human region detection unit identifies a person from an image captured by the camera device 200, and detects a bounding box that is the region of the person.

[0050] The 2D skeleton estimation unit uses existing technologies such as CPN (Cascaded Pyramid Network) to estimate the posture of a person identified from a captured image from a partial image within the bounding box of the person, and generates two-dimensional skeleton information.

[0051] The scale estimation unit obtains the distance between the wrists of a person holding the handle of a cart with both hands using two-dimensional skeletal information of the person identified from the captured image, and estimates the scale of the person based on the distance and the information on the length of the handle of the cart. This is because the distance between the wrists of a person holding the handle of a cart with both hands generally coincides with the length of the handle of the cart.

[0052] It may be determined that a person is holding the handle of the cart when the variance of the distance between the central coordinates of the bounding box of the person and the bounding box of the cart in multiple captured images captured continuously by the camera device 200 is equal to or less than a predetermined threshold. Alternatively, it may be determined that a person is holding the handle of the cart when the moving speed and direction of the bounding box of the person and the bounding box of the cart are within a predetermined range and are similar in multiple captured images captured continuously.

[0053] Furthermore, the process of estimating the scale of a person may include a process of calculating an average value of each scale estimated from a plurality of consecutively captured images, and estimating the average value as the scale of the person.

[0054] The relative 3D skeleton estimation unit uses existing technology to estimate the 3D skeleton coordinates of each part of the person's 2D skeleton information relative to a reference position such as the waist. Note that the 3D skeleton coordinates estimated by the relative 3D skeleton estimation unit are relative coordinates normalized from a reference position, and the dimensions between each part are not actual size, so in this embodiment, the scale of the person is estimated. Also, since the 3D skeleton coordinates estimated by the relative 3D skeleton estimation unit are relative coordinates from a reference position, in this embodiment, absolute coordinates relative to world coordinates are calculated.

[0055] The absolute 3D skeleton mapping unit converts the relative 3D skeleton coordinates of the person into absolute 3D skeleton information relative to the world coordinates, for example, using the estimated scale and a homography transformation matrix. The homography transformation matrix may be calculated based on the coordinates of four different points in an image captured inside a store and the world coordinates corresponding to each of the four points. The absolute 3D skeleton information may be calculated for several parts, such as the waist and right foot, and the absolute coordinates of other parts may be calculated using the calculated absolute coordinates of the waist and right foot and the estimated scale.

[0056] The movement determination unit determines and detects the movement of the person, for example, whether or not the person has put a product into the cart, based on the absolute three-dimensional skeleton information of the person and the three-dimensional position and orientation data of the cart.

[0057] [Function details] Next, each process executed by the estimation device 10 as the operating subject will be described in detail with reference to Figs. 10 to 17. Fig. 10 is a diagram showing an example of the scale estimation process according to the first embodiment. The person scale estimation process shown in Fig. 10 may be repeatedly executed every time a person is detected from a captured image. At that time, the estimation device 10 assigns an identifier to each person and stores information of the same person in association with each other.

[0058] 10 focuses on the fact that the distance between the wrists of a person holding a cart handle with both hands roughly corresponds to the length of the handle, and estimates the scale from the distance between the handle of the cart in the captured image and the wrists of the person holding the cart. Note that information on the length of the handles of carts handled in stores is known in advance and is stored in estimation device 10.

[0059] First, as described with reference to FIG. 5, the estimation device 10 detects and identifies a person and a cart from an image captured by the camera device 200 using an existing technique such as YOLO. Next, the estimation device 10 further identifies a person holding the identified cart from among the identified people. For example, if the variance of the distance between the center coordinates of the bounding box of the person and the bounding box of the cart is equal to or less than a predetermined threshold, the estimation device 10 determines that the person is holding the cart. Then, as described with reference to FIGS. 5 to 7, the estimation device 10 estimates the posture of the person and detects two-dimensional skeleton information from a partial image within the bounding box of the person identified as the person holding the cart using an existing technique such as CPN.

[0060] Next, the estimation device 10 uses existing technology to estimate the relative 3D skeletal coordinates of each part of the person's 2D skeletal information with respect to a reference position such as the waist. As shown in the lower part of Fig. 10, the estimation of the 3D skeletal coordinates involves (1) inputting an image showing a person, (2) finding true values ​​of each part with respect to the position of the waist as a reference, and (3) estimating the relative 3D coordinates of each part from the waist.

[0061] Then, the estimation device 10 calculates the scale r by using the estimated relative three-dimensional coordinates of both wrists and the length of the gripping portion of the cart, for example, according to the following formula (1).

[0062]

number

[0063] Here, in formula (1), X rh and X lhare the relative three-dimensional coordinates of the estimated right and left wrists, respectively, and L is the length of the grip part of the cart. The length of the grip part of the cart may be measured in advance using a tape measure or the like. In addition, the estimation device 10 may identify the same person holding the cart from a series of captured images, which are frames captured by the camera device 200, estimate the scale for each captured image, and set the average value of the estimated scales as the scale r.

[0064] In this way, the estimation device 10 estimates the scale of the person holding the cart, and by using the estimated scale when recognizing the behavior of the person, it is possible to more accurately recognize the behavior of the person. Next, the person behavior recognition process will be described. The person behavior recognition according to this embodiment estimates the three-dimensional position and posture of the person and the cart identified from the captured image, and recognizes the behavior of the person from the estimated three-dimensional information.

[0065] First, a process of acquiring three-dimensional position and orientation data of a cart identified from an image captured by the camera device 200 will be described. Fig. 11 is a diagram illustrating an example of a process of acquiring three-dimensional position and orientation of a cart according to the first embodiment. The process of acquiring three-dimensional position and orientation shown in Fig. 11 may be executed only once during person behavior recognition.

[0066] 11, the estimation device 10 inputs a partial image within the bounding box of the cart in the captured image and the CAD of the cart into a machine learning model to obtain a three-dimensional rectangular area that is a rectangular parallelepiped area indicating the three-dimensional position and posture of the cart. The machine learning model used here is a machine learning model that is generated by collecting captured images of the cart from multiple directions, using the captured images and the CAD data of the cart as feature quantities, and using the three-dimensional rectangular area of ​​the cart in the captured images as a correct answer label.

[0067] Next, a process of acquiring 3D position and posture data of a person identified from an image captured by the camera device 200 will be described. First, as described with reference to FIG. 10, the estimation device 10 estimates the relative 3D skeletal coordinates of each part of the person's 2D skeletal information with respect to a reference position such as the waist, using existing technology. Furthermore, by multiplying the coordinates of each part in the relative 3D skeletal coordinates by a scale value, relative 3D skeletal information of the real scale with respect to a reference position such as the waist can be obtained. Then, the estimation device 10 converts the relative 3D skeletal coordinates of the real scale of the person into absolute 3D skeletal information with respect to the world coordinates, using the estimated scale and a homography transformation matrix.

[0068] Here, a method for calculating a homography transformation matrix used when converting the relative 3D skeleton coordinates of a person in real scale into absolute 3D skeleton information will be described. FIG. 12 is a diagram showing an example of a 2D coordinate-3D coordinate correspondence relationship for homography calculation according to the first embodiment. FIG. 12 shows a correspondence relationship between the 2D coordinates of four points on a captured image 252 and the 3D coordinates, which are the world coordinates of four points on a layout drawing 260 corresponding to the four points. The layout drawing 260 is a layout drawing of the imaging location at the time of construction, etc., and shows the layout of the imaging location shown in the captured image 252 when viewed from above. The same markers on the captured image 252 and the layout drawing 260 each indicate the same location.

[0069] The estimation device 10 calculates a homography transformation matrix using, for example, four two-dimensional coordinate points on the captured image 252 and four corresponding three-dimensional coordinate points on the layout diagram 260, using an existing technique such as a Direct Linear Transformation (DLT) method. The homography transformation matrix is ​​expressed by the following formula (2).

[0070]

number

[0071] Here, in equation (2), u and v indicate the input two-dimensional coordinates, and x, y, and 1 (z value) indicate the converted three-dimensional coordinates. The three-dimensional coordinates calculated by equation (2) are three-dimensional coordinates on the ground with the z value being 0. Therefore, in this embodiment, it is assumed that the x and y coordinates of the feet and the waist are the same when the person is standing, and the z coordinate is calculated based on the leg length h. leg can be replaced by.

[0072] Leg length h leg A method of calculating the leg length h will be described. FIG. 13 is a diagram showing an example of setting the reference position according to the first embodiment. FIG. 13 shows a case where the hip center and the ankle center are set as reference positions in the whole body skeletal information shown in FIG. 7. The leg length h is calculated from the coordinate distance in the height direction between the hip center and the ankle center, that is, the z-axis direction. leg is calculated.

[0073] 14 is a diagram illustrating an example of a positional relationship in a 2D / 3D coordinate system according to the first embodiment. As shown on the right side of FIG. 14, if the three-dimensional coordinates of the center of the feet are (x', y', 0), the three-dimensional coordinates of the center of the waist are (x, y, h leg ) can be expressed as h leg is obtained by multiplying the three-dimensional distance between the center of the waist and the center of the feet by the scale. Then, the three-dimensional coordinates (x, y, h leg ) is the two-dimensional coordinate of the center of the waist (u ch ,v ch ) and the homography transformation matrix expressed by the following equation (3).

[0074]

number

[0075] Equation (3) is obtained by replacing the two-dimensional and three-dimensional coordinate parts of equation (2) with the coordinates of the waist center. In addition, the third line of the three-dimensional coordinates (x, y, 1) of the waist center in equation (3), i.e., the z coordinate part, is 1, which means that the values ​​of x and y are divided so that this becomes 1. And the z coordinate part is calculated by dividing the leg length h as shown in the following equation (4). leg can be replaced by.

[0076]

number

[0077] In addition, the three-dimensional coordinates (x', y', 0) of the center of the foot are the two-dimensional coordinates (u cf ,v cf ) and the homography transformation matrix expressed by the following equation (5).

[0078]

number

[0079] Equation (5) is obtained by replacing the two-dimensional and three-dimensional coordinates of equation (2) with the coordinates of the center of the foot. Also, the three-dimensional coordinates of the center of the foot (x', y', 1) have the z-coordinate replaced with the ground coordinate value 0, as shown in the following equation (6).

[0080]

number

[0081] Using the above formulas (3) to (6), the estimation device 10 can calculate the absolute three-dimensional coordinates of the waist center and the foot center. The absolute three-dimensional coordinates of other parts can be calculated using the relative three-dimensional coordinates and absolute three-dimensional coordinates of the waist center and the foot center.

[0082] Fig. 15 is a diagram showing an example of absolute 3D skeleton position estimation by world coordinate mapping of a 3D skeleton according to Example 1. As shown on the left side of Fig. 15, for example, the relative three-dimensional coordinates of the foot center, the waist center, and the head are (1,3,0), (1,3,1), and (1,3,3), respectively. Also, as shown on the right side of Fig. 15, for example, the absolute three-dimensional coordinates of the foot center and the waist center are calculated from the relative three-dimensional coordinates using the estimated scale and homography transformation matrix, and are (5,6,0) and (5,6,4), respectively.

[0083] Based on the relative three-dimensional coordinates of the center of the feet and the center of the waist, the vector from the center of the waist to the center of the feet is (1,3,0)-(1,3,1)=(0,0,-1). Similarly, the vector from the center of the waist to the center of the head is (1,3,3)-(1,3,1)=(0,0,2). In the example of FIG. 15, the scale is four times, so the absolute three-dimensional coordinates of the head are (5,6,4)+(0,0,2)×4=(5,6,12). Strictly speaking, the rotation direction must be taken into consideration depending on the body part, but as explained with reference to FIG. 15, the absolute three-dimensional coordinates of other body parts can be calculated from the absolute three-dimensional coordinates of the two reference body parts.

[0084] As described above, the estimation device 10 calculates three-dimensional information of a person or cart identified from a captured image. Then, the estimation device 10 uses the calculated three-dimensional information of the person or cart to detect the motion of the person and recognize the action.

[0085] Fig. 16 is a diagram illustrating an example of the motion detection process according to the embodiment 1. On the left side of Fig. 16, the skeleton information of the person is calculated absolute three-dimensional coordinates as described with reference to Fig. 15 and the like, and the rectangular parallelepiped region surrounding the cart is a three-dimensional rectangular region indicating the three-dimensional position and posture of the cart as described with reference to Fig. 11.

[0086] Then, as shown on the right side of FIG. 16, the estimation device 10 determines that the person has put his / her hand into the cart when, for example, the absolute three-dimensional coordinate position of the person's right wrist or left wrist is inside the three-dimensional coordinate position of the rectangular parallelepiped area of ​​the cart. This allows the estimation device 10 to detect, for example, the person's action of putting a product into the cart. Furthermore, by using the absolute three-dimensional information, the estimation device 10 can detect the person's action and perform behavior recognition without being influenced by the angle of the captured image. Then, the estimation device 10 can store the product purchasing behavior and the like by recognizing the person's behavior, determine the presence or absence of fraudulent behavior, and detect fraudulent behavior.

[0087] Furthermore, by measuring in advance the three-dimensional coordinate positions of not only carts but also product shelves and other items in a store and storing them in the estimation device 10, the types of person's actions that can be recognized can be increased. FIG. 17 is a diagram showing another example of the action detection process according to the first embodiment. For example, the three-dimensional coordinate positions of a rectangular parallelepiped area of ​​a product shelf are stored in advance in the estimation device 10. Then, as shown in FIG. 17, the estimation device 10 determines that the person has put his / her hand into the product shelf when, for example, the absolute three-dimensional coordinate position of the person's right wrist or left wrist is inside the three-dimensional coordinate position of the rectangular parallelepiped area of ​​the product shelf. In this way, the estimation device 10 can detect, for example, an action of the person removing a product from the product shelf.

[0088] [Processing flow] Next, a flow of a person scale estimation process executed by the estimation device 10 will be described. Fig. 18 is a flowchart showing the flow of the scale estimation process according to the first embodiment. The scale estimation process shown in Fig. 18 may be executed at regular time intervals or every time a captured image is received from the camera device 200, for example.

[0089] 18, the estimation device 10 acquires captured images of a predetermined imaging range, such as the inside of a retail store or on the premises, captured by the camera device 200 from the storage unit 30 (step S101). In this embodiment, the captured images captured by the camera device 200, strictly speaking, surveillance footage, are processed in real time, so the captured images are transmitted from the camera device 200 at any time and stored in the storage unit 30.

[0090] Next, the estimation device 10 detects the cart from the captured image acquired in step S101 using an existing object detection algorithm (step S102).

[0091] Next, the estimation device 10 detects a person from the captured image acquired in step S101 using an existing object detection algorithm, and further detects two-dimensional skeletal information of the detected person using existing posture estimation and skeleton estimation algorithms (step S103). Note that the order of execution of steps S102 and S103 may be reversed, or they may be executed in parallel. Furthermore, if multiple people are detected from the captured image, the processes from step S103 onwards are executed for each detected person.

[0092] Next, the estimation device 10 uses existing technology to estimate the relative three-dimensional skeletal coordinates of each part of the two-dimensional skeletal information detected in step S103 with respect to a reference position such as the waist (step S104).

[0093] Next, the estimation device 10 determines whether or not the detected person is holding the cart based on the bounding box of the person detected from the captured image and the bounding box of the cart (step S105). If the detected person is not holding the cart (step S106: No), the scale estimation process shown in FIG. 18 ends.

[0094] On the other hand, if the detected person is holding a cart (step S106: Yes), the estimation device 10 estimates the scale of the person (step S107). The estimation of the person's scale in step S107 is performed by acquiring the distance between the person's wrists based on the relative 3D skeletal coordinates estimated in step S104, and based on this distance and information on the length of the grip part of the cart. After step S107 is performed, the scale estimation process shown in FIG. 18 ends.

[0095] Next, a flow of a person's motion detection process executed by the estimation device 10 will be described. Fig. 19 is a flowchart showing the flow of the motion detection process according to the first embodiment. The motion detection process shown in Fig. 19 may be executed only once during person behavior recognition, for example.

[0096] First, as shown in FIG. 19, the estimation device 10 acquires from the storage unit 30 a captured image of a predetermined imaging range, such as the inside of a retail store or on the premises, captured by the camera device 200 (step S201).

[0097] Next, the estimation device 10 detects the cart from the captured image acquired in step S201 using an existing object detection algorithm (step S202).

[0098] Next, the estimation device 10 inputs a partial image within the bounding box of the cart detected in step S202 and the CAD of the cart into a machine learning model, and detects a three-dimensional rectangular area, which is a rectangular parallelepiped area indicating the three-dimensional position and orientation of the cart (step S203).

[0099] Next, the estimation device 10 detects a person from the captured image acquired in step S201 using an existing object detection algorithm, and further detects two-dimensional skeleton information of the detected person using existing posture estimation and skeleton estimation algorithms (step S204). Note that the order of execution of steps S202 and S203 and step S204 may be reversed, or they may be executed in parallel.

[0100] Next, the estimation device 10 estimates the three-dimensional skeletal coordinates of each part of the person's two-dimensional skeletal information relative to a reference position such as the waist (step S205).

[0101] Next, the estimation device 10 multiplies the coordinates of each part of the 3D skeletal coordinates estimated in step S205 by the scale estimated in step S107 to convert them, thereby obtaining relative 3D skeletal information of the real scale with respect to a reference position such as the waist (step S206).

[0102] Next, the estimation device 10 converts the relative three-dimensional skeleton coordinates acquired in step S206 into absolute three-dimensional skeleton information by mapping them to world coordinates using the scale estimated in step S107 and the homography transformation matrix (step S207).

[0103] Next, the estimation device 10 detects the person's motion, such as the motion of the person putting a product into the cart, based on the absolute 3D skeleton information acquired in step S207 and the 3D position and orientation of the cart detected in step S203 (step S208). After step S208 is executed, the motion detection process shown in Fig. 19 ends, but if an unauthorized action of the person is recognized by the motion detection in step S208, the estimation device 10 notifies an alert to the store clerk terminal 300 or the like.

[0104] [effect] As described above, the estimation device 10 identifies a first person using a first cart from a first image captured inside the store, generates skeletal information of the first person, obtains a first distance at which the first person is holding the handle of the first cart using the skeletal information, and estimates a first scale of the first person based on the first distance and information on the length of the first cart.

[0105] In this way, the estimation device 10 generates skeletal information of a person using a cart identified from a captured image, and estimates the scale of the person from the cart gripping interval and cart length information acquired from the skeletal information, which allows the estimation device 10 to more accurately estimate the scale of the person from a captured image.

[0106] In addition, the estimation device 10 determines that the first person is holding the handle when the variance of the distance between the central coordinates of the bounding box of the first person and the bounding box of the first cart in multiple first images captured in succession is less than a predetermined threshold.

[0107] This enables the estimation device 10 to more accurately determine, from the captured image, that a person is holding a cart.

[0108] Furthermore, the process of estimating the first scale, which is executed by the estimation device 10, includes a process of calculating an average value of the first scales estimated from each of the multiple first images, and estimating the average value as the first scale.

[0109] This enables the estimation device 10 to reduce the scale error between captured images and estimate the scale more accurately.

[0110] In addition, the process of generating skeletal information executed by the estimation device 10 includes a process of generating two-dimensional skeletal information of the first person, and the estimation device 10 converts the two-dimensional skeletal information into three-dimensional skeletal information using a homography transformation matrix and a first scale.

[0111] This enables the estimation device 10 to more accurately estimate three-dimensional skeletal information of a person from a captured image.

[0112] Furthermore, the estimation device 10 calculates a homography transformation matrix based on the coordinates of four different predetermined points in a second image capturing the interior of the store and the world coordinates corresponding to each of the four points.

[0113] This enables the estimation device 10 to more accurately estimate three-dimensional skeletal information of a person from a captured image.

[0114] In addition, the process of converting the two-dimensional skeletal information into three-dimensional skeletal information executed by the estimation device 10 includes a process of converting the coordinates of two specific parts of the two-dimensional skeletal information into the coordinates of two corresponding parts of the three-dimensional skeletal information using a homography transformation matrix and a first scale, and calculating the coordinates of other parts in the three-dimensional skeletal information using the two-dimensional skeletal information, the coordinates of the two corresponding parts of the three-dimensional skeletal information, and the first scale.

[0115] This enables the estimation device 10 to more accurately estimate three-dimensional skeletal information of a person from a captured image.

[0116] Furthermore, the estimation device 10 recognizes a predetermined behavior of the first person based on the three-dimensional skeleton information.

[0117] This enables the estimation device 10 to more accurately recognize the behavior of a person from a captured image.

[0118] In addition, the estimation device 10 inputs a partial image within the bounding box of the first cart in the first image and the CAD data of the first cart into a machine learning model generated by machine learning using an image of the cart and CAD data of the cart as features and the three-dimensional position and orientation data of the cart as correct answer labels, and acquires the three-dimensional position and orientation data of the first cart.

[0119] This enables the estimation device 10 to more accurately estimate three-dimensional information of the cart from the captured images.

[0120] In addition, the process of recognizing a predetermined action executed by the estimation device 10 includes a process of detecting the action of the first person putting a product into the cart based on the three-dimensional skeleton information and the three-dimensional position and orientation data of the first cart.

[0121] This enables the estimation device 10 to more accurately recognize the behavior of a person from a captured image.

[0122] [system] The information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings may be changed arbitrarily unless otherwise specified. In addition, the specific examples, distributions, numerical values, etc. described in the embodiments are merely examples and may be changed arbitrarily.

[0123] Furthermore, the specific form of distribution or integration of the components of each device is not limited to that shown in the figures. In other words, all or part of the components may be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions of each device may be realized by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or may be realized as hardware using wired logic.

[0124] [Hardware] Fig. 20 is a diagram illustrating an example of a hardware configuration of the estimation device 10. As shown in Fig. 20, the estimation device 10 includes a communication interface 10a, a hard disk drive (HDD) 10b, a memory 10c, and a processor 10d. The components shown in Fig. 20 are connected to each other via a bus or the like.

[0125] The communication interface 10a is a network interface card or the like, and performs communication with other information processing devices. The HDD 10b stores programs and data for operating the functions shown in FIG.

[0126] The processor 10d is a hardware circuit that operates a process that executes each function described in FIG. 9 and the like by reading a program that executes the same processing as each processing unit shown in FIG. 9 from the HDD 10b or the like and expanding the program in the memory 10c. That is, this process executes a function similar to that of each processing unit of the estimation device 10. Specifically, the processor 10d reads a program having a function similar to that of the image acquisition unit, the object detection unit, and the like from the HDD 10b or the like. Then, the processor 10d executes a process that executes a process similar to that of the image acquisition unit, the object detection unit, and the like.

[0127] In this way, the estimation device 10 operates as an information processing device that executes operation control processing by reading and executing a program that executes the same processing as each processing unit shown in Fig. 9. The estimation device 10 can also realize functions similar to those of the above-mentioned embodiment by reading a program from a recording medium using a medium reading device and executing the read program. Note that the program in these other embodiments is not limited to being executed by the estimation device 10. For example, this embodiment may be similarly applied to a case where another computer or server executes a program, or a case where these execute a program in cooperation with each other.

[0128] A program that executes the same processes as those of each processing unit shown in Fig. 9 can be distributed via a network such as the Internet. This program can be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD), and can be executed by being read out from the recording medium by a computer.

[0129] Fig. 21 is a diagram for explaining an example of the hardware configuration of an information processing terminal 1000. Since the user terminal 100 and the clerk terminal 300 may have the same configuration, in Fig. 21 the user terminal 100 and the clerk terminal 300 are collectively shown as the information processing terminal 1000 as an example of the hardware configuration of the user terminal 100 and the clerk terminal 300. As shown in Fig. 21, the information processing terminal 1000 has a communication interface 1000a, a HDD 1000b, a memory 1000c, a processor 1000d, an input unit 1000e, and a display unit 1000f. In addition, each unit shown in Fig. 21 is connected to each other by a bus or the like.

[0130] The communication interface 1000a is a network interface card or the like, and communicates with other information processing devices. The HDD 1000b stores programs and data for operating each function of the information processing terminal 1000.

[0131] The processor 100d is a hardware circuit that reads out a program for executing the processing of each function of the information processing terminal 1000 from the HDD 1000b or the like and loads it in the memory 1000c, thereby operating a process for executing each function of the information processing terminal 1000. In other words, this process executes the same function as each processing unit of the information processing terminal 1000.

[0132] In this way, the information processing terminal 1000 operates as an information processing device that executes operation control processing by reading and executing a program that executes processing of each function of the information processing terminal 1000. The information processing terminal 1000 can also realize each function of the information processing terminal 1000 by reading a program from a recording medium using a medium reading device and executing the read program. Note that the program in this other embodiment is not limited to being executed by the information processing terminal 1000. For example, this embodiment may be similarly applied to a case where another computer or server executes a program, or a case where these execute a program in cooperation with each other.

[0133] Furthermore, the program for executing the processing of each function of the information processing terminal 1000 can be distributed via a network such as the Internet. Moreover, this program can be recorded on a computer-readable recording medium such as a hard disk, a FD, a CD-ROM, an MO, or a DVD, and can be executed by being read out from the recording medium by a computer.

[0134] The input unit 1000e detects various input operations by the user, such as an input operation to a program executed by the processor 100d. The input operations include, for example, a touch operation and inserting an earphone jack into the information processing terminal 1000. Here, the touch operation refers to various contact operations on the display unit 1000f, such as tapping, double tapping, swiping, pinching, and the like. The touch operation also includes an operation of bringing an object, such as a finger, close to the display unit 1000f. The input unit 1000e may be, for example, a button, a touch panel, a proximity sensor, or the like.

[0135] The display unit 1000f displays various types of visual information under the control of the processor 1000d. The display unit 1000f may be a liquid crystal display (LCD), an organic light emitting diode (OLED), or a so-called organic electroluminescence (EL) display.

[0136] Fig. 22 is a diagram illustrating an example of the hardware configuration of the self-checkout terminal 400. As shown in Fig. 22, the self-checkout terminal 400 has a communication interface 400a, an HDD 400b, a memory 400c, a processor 400d, an input unit 400e, and an output unit 400f. The units shown in Fig. 22 are connected to each other via a bus or the like.

[0137] The communication interface 400a is a network interface card or the like, and communicates with other information processing devices. The HDD 400b stores programs and data for operating each function of the self-checkout terminal 400.

[0138] The processor 400d is a hardware circuit that reads out a program for executing the processing of each function of the self-checkout terminal 400 from the HDD 400b or the like and loads it into the memory 400c, thereby operating a process for executing each function of the self-checkout terminal 400. In other words, this process executes the same functions as each processing unit of the self-checkout terminal 400.

[0139] In this way, the self-checkout terminal 400 operates as an information processing device that executes operation control processing by reading and executing a program that executes processing of each function of the self-checkout terminal 400. The self-checkout terminal 400 can also realize each function of the self-checkout terminal 400 by reading a program from a recording medium using a media reading device and executing the read program. Note that the program in this other embodiment is not limited to being executed by the self-checkout terminal 400. For example, this embodiment may be similarly applied to a case where another computer or server executes a program, or a case where these execute a program in cooperation with each other.

[0140] Furthermore, the program for executing the processing of each function of the self-checkout terminal 400 can be distributed via a network such as the Internet. Moreover, this program can be recorded on a computer-readable recording medium such as a hard disk, FD, CD-ROM, MO, or DVD, and executed by a computer by reading it from the recording medium.

[0141] The input unit 400e detects various input operations by the user, such as an input operation to a program executed by the processor 400d. The input operations include, for example, a touch operation. In the case of a touch operation, the self-checkout terminal 400 further includes a display unit, and the input operation detected by the input unit 400e may be a touch operation to the display unit. The input unit 400e may be, for example, a button, a touch panel, a proximity sensor, or the like.

[0142] The output unit 400f outputs data output from a program executed by the processor 400d via an external device, such as an external display device, connected to the self-checkout terminal 400. If the self-checkout terminal 400 includes a display unit, the self-checkout terminal 400 does not need to include the output unit 400f.

[0143] Fig. 23 is a diagram illustrating an example of the hardware configuration of the gate reader 500. As shown in Fig. 23, the gate reader 500 has a USB interface 500a, an image sensor 500b, a light emitting unit 500c, and a processor 500d. The units shown in Fig. 23 are connected to each other via a bus or the like.

[0144] The USB interface 500a communicates with other information processing devices.

[0145] The image sensor 500b receives light emitted or reflected by an object to be read by the gate reader 500, and converts the brightness of the light into electrical information.

[0146] The light emitting unit 500c is an illumination light source such as a high-brightness LED, and emits light to make it easier to read an object to be read by the gate reader 500. When the object to be read by the gate reader 500 or a device displaying the object emits light, the gate reader 500 does not need to include the light emitting unit 500c.

[0147] The processor 500d controls the light emitting unit 500c to irradiate the object with light, and controls the image sensor 500b to convert the object into electrical information and read it in. The processor 500d also transmits the electrical information of the object that has been read in to another information processing device via the USB interface 500a.

[0148] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0149] (Appendix 1) Identifying a first person using a first cart from a first image captured inside the store; Generate skeletal information of the first person; obtaining a first interval at which the first person is gripping a gripping portion of the first cart using the skeleton information; Estimating a first scale of the first person based on the first distance and information about the length of the first cart. An estimation program that causes a computer to execute a process.

[0150] (Appendix 2) The estimation program according to appendix 1, characterized in that the computer is caused to execute a process of determining that the first person is holding the grip part when the variance of the distance between the central coordinates of the bounding box of the first person and the bounding box of the first cart in multiple first images captured consecutively is equal to or less than a predetermined threshold.

[0151] (Supplementary Note 3) The estimation program according to Supplementary Note 1, characterized in that the process of estimating the first scale executed by the computer includes a process of calculating an average value of the first scales estimated from each of a plurality of the first images and estimating the average value as the first scale.

[0152] (Supplementary Note 4) The process of generating the skeletal information executed by the computer includes a process of generating two-dimensional skeletal information of the first person, The estimation program according to claim 1, characterized in that the computer is caused to execute a process of converting the two-dimensional skeleton information into three-dimensional skeleton information using a homography transformation matrix and the first scale.

[0153] (Appendix 5) The estimation program according to appendix 4, characterized in that the computer is caused to execute a process of calculating a homography transformation matrix based on the coordinates of four different predetermined points in a second image capturing the interior of the store and the world coordinates corresponding to each of the four points.

[0154] (Appendix 6) The process of converting the two-dimensional skeletal information into the three-dimensional skeletal information includes: transforming the coordinates of two predetermined parts of the two-dimensional skeleton information into the coordinates of two corresponding parts of the three-dimensional skeleton information using the homography transformation matrix and the first scale; Calculating coordinates of other parts in the three-dimensional skeleton information using the two-dimensional skeleton information, the coordinates of the two corresponding parts in the three-dimensional skeleton information, and the first scale. 5. The estimation program according to claim 4, further comprising the steps of:

[0155] (Supplementary Note 7) The estimation program according to Supplementary Note 4, characterized in that the computer is caused to execute a process of recognizing a predetermined behavior of the first person based on the three-dimensional skeleton information.

[0156] (Appendix 8) The estimation program according to appendix 7, characterized in that a partial image within a bounding box of the first cart in the first image and the CAD data of the first cart are input into a machine learning model generated by machine learning using an image of a cart and CAD data of the cart as features and the 3D position and orientation data of the cart as correct answer labels, and the computer is caused to execute a process of acquiring the 3D position and orientation data of the first cart.

[0157] (Appendix 9) The estimation program according to appendix 8, wherein the process of recognizing the specified behavior executed by the computer includes a process of detecting the action of the first person putting a product into a cart based on the three-dimensional skeletal information and the three-dimensional position and posture data of the first cart.

[0158] (Appendix 10) Identifying a first person using a first cart from a first image captured inside the store; Generate skeletal information of the first person; obtaining a first interval at which the first person is gripping a gripping portion of the first cart using the skeleton information; Estimating a first scale of the first person based on the first distance and information about the length of the first cart. An estimation method characterized in that the processing is executed by a computer.

[0159] (Appendix 11) The estimation method according to appendix 10, characterized in that the computer executes a process of determining that the first person is holding the grip part when the variance of the distance between the central coordinates of the bounding box of the first person and the bounding box of the first cart in multiple first images captured consecutively is equal to or less than a predetermined threshold.

[0160] (Appendix 12) The estimation method according to appendix 10, wherein the process of estimating the first scale executed by the computer includes a process of calculating an average value of the first scales estimated from each of the multiple first images and estimating the average value as the first scale.

[0161] (Supplementary Note 13) The process of generating skeletal information executed by the computer includes a process of generating two-dimensional skeletal information of the first person; The estimation method according to claim 10, characterized in that the computer is caused to execute a process of converting the two-dimensional skeleton information into three-dimensional skeleton information using a homography transformation matrix and the first scale.

[0162] (Appendix 14) The estimation method according to appendix 13, characterized in that the computer executes a process of calculating a homography transformation matrix based on the coordinates of four different predetermined points in a second image taken of the interior of the store and the world coordinates corresponding to each of the four points.

[0163] (Appendix 15) The process of converting the two-dimensional skeleton information into the three-dimensional skeleton information includes: transforming the coordinates of two predetermined parts of the two-dimensional skeleton information into the coordinates of two corresponding parts of the three-dimensional skeleton information using the homography transformation matrix and the first scale; Calculating coordinates of other parts in the three-dimensional skeleton information using the two-dimensional skeleton information, the coordinates of the two corresponding parts in the three-dimensional skeleton information, and the first scale. 14. The estimation method according to claim 13, further comprising the steps of:

[0164] (Supplementary Note 16) The estimation method according to Supplementary Note 13, characterized in that the computer executes a process of recognizing a predetermined behavior of the first person based on the three-dimensional skeleton information.

[0165] (Appendix 17) The estimation method according to appendix 16, characterized in that the computer executes a process of inputting a partial image within a bounding box of the first cart in the first image and the CAD data of the first cart into a machine learning model generated by machine learning using an image of a cart and CAD data of the cart as features and the 3D position and orientation data of the cart as correct answer labels, and acquiring the 3D position and orientation data of the first cart.

[0166] (Appendix 18) The estimation method described in Appendix 17, characterized in that the process of recognizing the specified behavior executed by the computer includes a process of detecting the action of the first person putting a product into a cart based on the three-dimensional skeletal information and the three-dimensional position and posture data of the first cart.

[0167] (Appendix 19) Identifying a first person using a first cart from a first image captured inside the store; Generate skeletal information of the first person; obtaining a first interval at which the first person is gripping a gripping portion of the first cart using the skeleton information; Estimating a first scale of the first person based on the first distance and information about the length of the first cart. An estimation device comprising a control unit that executes processing.

[0168] (Appendix 20) The estimation device described in Appendix 19, characterized in that the control unit executes a process of determining that the first person is holding the grip part when the variance of the distance between the central coordinates of the bounding box of the first person and the bounding box of the first cart in multiple first images captured consecutively is equal to or less than a predetermined threshold.

[0169] (Supplementary Note 21) The estimation device described in Supplementary Note 19, characterized in that the process of estimating the first scale executed by the control unit includes a process of calculating an average value of the first scales estimated from each of a plurality of the first images and estimating it as the first scale.

[0170] (Supplementary Note 22) The process of generating the skeletal information executed by the control unit includes a process of generating two-dimensional skeletal information of the first person, 20. The estimation device according to claim 19, wherein the control unit is configured to execute a process of converting the two-dimensional skeleton information into three-dimensional skeleton information using a homography transformation matrix and the first scale.

[0171] (Appendix 23) The estimation device described in Appendix 22, characterized in that the control unit executes a process of calculating a homography transformation matrix based on the coordinates of four different predetermined points in a second image taken of the interior of the store and world coordinates corresponding to each of the four points.

[0172] (Appendix 24) The process of converting the two-dimensional skeletal information into the three-dimensional skeletal information includes: transforming the coordinates of two predetermined parts of the two-dimensional skeleton information into the coordinates of two corresponding parts of the three-dimensional skeleton information using the homography transformation matrix and the first scale; Calculating coordinates of other parts in the three-dimensional skeleton information using the two-dimensional skeleton information, the coordinates of the two corresponding parts in the three-dimensional skeleton information, and the first scale. 23. The estimation device of claim 22, further comprising a process.

[0173] (Supplementary Note 25) The estimation device according to Supplementary Note 22, characterized in that the control unit executes a process of recognizing a predetermined behavior of the first person based on the three-dimensional skeleton information.

[0174] (Appendix 26) The estimation device described in Appendix 25, characterized in that the control unit executes a process of inputting a partial image within a bounding box of the first cart in the first image and the CAD data of the first cart into a machine learning model generated by machine learning using an image of a cart and CAD data of the cart as features and the 3D position and orientation data of the cart as correct answer labels, and acquiring the 3D position and orientation data of the first cart.

[0175] (Appendix 27) The estimation device described in Appendix 26, characterized in that the process of recognizing the specified behavior executed by the control unit includes a process of detecting the action of the first person putting a product into a cart based on the three-dimensional skeletal information and the three-dimensional position and orientation data of the first cart.

[0176] (Appendix 28) A processor; a memory operatively connected to the processor; An estimation apparatus comprising: Identifying a first person using a first cart from a first image captured inside the store; Generate skeletal information of the first person; obtaining a first interval at which the first person is gripping a gripping portion of the first cart using the skeleton information; Estimating a first scale of the first person based on the first distance and information about the length of the first cart. An estimation device that executes a process. [Explanation of symbols]

[0177] 1. Fraud Detection System 10 Estimation device 10a Communication Interface 10b HDD 10c Memory 10d Processor 20 Communications Department 30 Storage section 40 Control section 50 Network 100 User terminals 200 Camera Equipment 250, 251, 252 Captured images 260 Layout diagram 300 Store clerk terminal 400 Self-checkout terminals 400a communication interface 400b HDD 400c Memory 400d processor 400e Input section 400f Output section 500 Gate Reader 500a USB interface 500b image sensor 500c Light emitting part 500d Processor 1000 Information processing terminal 1000a communication interface 1000b HDD 1000c Memory 1000d Processor 1000e Input section 1000f display section

Claims

1. Identifying a first person using a first cart from a first image captured inside the store; Generate skeletal information of the first person; using the skeleton information to obtain a first interval at which the first person grips a gripping portion of the first cart; Estimating a first scale of the first person based on the first distance and the length of the first cart. An estimation program that causes a computer to execute a process.

2. The estimation program according to claim 1, characterized in that the computer is caused to execute a process of determining that the first person is holding the grip part if a variance of the distance between the central coordinates of a bounding box of the first person and a bounding box of the first cart in a plurality of first images captured in succession is equal to or less than a predetermined threshold.

3. 2. The estimation program according to claim 1, wherein the process of estimating the first scale executed by the computer includes a process of calculating an average value of the first scales estimated from each of the plurality of first images and estimating the average value as the first scale.

4. The process of generating skeleton information, which is executed by the computer, includes a process of generating two-dimensional skeleton information of the first person; The estimation program according to claim 1 , further comprising: causing the computer to execute a process of converting the two-dimensional skeleton information into three-dimensional skeleton information using a homography transformation matrix and the first scale.

5. The estimation program according to claim 4, characterized in that the computer is caused to execute a process of calculating a homography transformation matrix based on the coordinates of four different predetermined points in a second image capturing the interior of the store and world coordinates corresponding to each of the four points.

6. The process of converting the two-dimensional skeleton information into the three-dimensional skeleton information includes: transforming coordinates of two predetermined parts of the two-dimensional skeleton information into coordinates of two corresponding parts of the three-dimensional skeleton information using the homography transformation matrix and the first scale; Calculating coordinates of other parts in the three-dimensional skeleton information using the two-dimensional skeleton information, the coordinates of the two corresponding parts in the three-dimensional skeleton information, and the first scale.

5. The estimation program according to claim 4, further comprising a process for:

7. 5. The estimation program according to claim 4, further comprising a step of causing the computer to execute a process of recognizing a predetermined action of the first person based on the three-dimensional skeleton information.

8. The estimation program according to claim 7, further comprising inputting a partial image within a bounding box of the first cart in the first image and the CAD data of the first cart into a machine learning model generated by machine learning using an image of a cart and CAD (Computer Aided Design) data of the cart as features and three-dimensional position and orientation data of the cart as correct answer labels, and causing the computer to execute a process of acquiring three-dimensional position and orientation data of the first cart.

9. The estimation program according to claim 8, characterized in that the process of recognizing the specified behavior executed by the computer includes a process of detecting the action of the first person putting a product into a cart based on the three-dimensional skeletal information and the three-dimensional position and orientation data of the first cart.

10. Identifying a first person using a first cart from a first image captured inside the store; Generate skeletal information of the first person; using the skeleton information to obtain a first interval at which the first person grips a gripping portion of the first cart; Estimating a first scale of the first person based on the first distance and the length of the first cart. An estimation method characterized in that the processing is executed by a computer.

11. Identifying a first person using a first cart from a first image captured inside the store; Generate skeletal information of the first person; using the skeleton information to obtain a first interval at which the first person grips a gripping portion of the first cart; Estimating a first scale of the first person based on the first distance and the length of the first cart. An estimation device comprising a control unit that executes processing.

Images