Setting program, setting method and setting device

Abstract

To detect taking in and out a product.SOLUTION: An information processor acquires video from a camera that is set in a store having a storage section in which a product is stored. The information processor identifies a depth indicating a distance from the camera for each of constituent elements of the video by inputting the acquired video from the camera into a machine learning model. The information processor generates a three-dimensional in-store model that is configured with the identified depth indicating the distance from the camera for each of the constituent elements of the video. The information processor generates skeleton information on a person who moves inside the store from the video acquired from the camera. The information processor sets a range and a direction of an aisle in the store in the generated three-dimensional in-store model on the basis of a transition of the generated skeleton information. The information processor sets a detection line for detecting that the person has extended his / her hand to a product, in the storage section on the basis of the range and the direction of the aisle in the store.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a setting program and the like. [Background technology]

[0002] In recent years, technology that recognizes customers taking products in and out of shelves has been attracting attention in stores. By using such technology, it is possible to introduce unmanned cash registers and perform shoplifting detection, etc. Technologies for recognizing the taking in and out of products include a method that uses cameras and image recognition technology, a method that assigns RFID (Radio Frequency Identification) to each product, and a method that installs a weight sensor on each shelf.

[0003] Compared to methods that attach RFID tags to each product or that install weight sensors on each shelf, methods that use cameras and image recognition technology have the advantage of being easy to implement.Furthermore, methods that use cameras and image recognition technology can reduce additional costs by using existing surveillance cameras, and are therefore in high demand.

[0004] Here, an example of a conventional technology that uses a camera and image recognition technology to recognize the taking in and out of products will be described. FIG. 14 is a diagram for explaining the conventional technology. The conventional technology uses a camera 6 installed above a shelf 5. The shelf 5 on the left side is a view of the shelf 5 from the front, and the shelf 5 on the right side is a view of the shelf 5 from above. In the conventional technology, a detection line 5a is set on the shelf 5 in advance.

[0005] In the prior art, the image from the camera 6 is analyzed to detect a person's hand, and when the person's hand passes through the detection line 5a, the image of the area around the hand is used to recognize whether a product is being taken in or out. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-107989 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the above-mentioned conventional technology has a problem in that it may erroneously detect the insertion or removal of a commodity.

[0008] For example, when using a surveillance camera, the positional relationship between the camera and the shelf differs for each combination of camera and shelf, making it difficult to use a uniformly predefined detection line as in conventional technology.

[0009] In particular, depending on the shooting conditions, the appropriate detection line will differ depending on the position of a person, even on the same shelf. FIG. 15 is a diagram showing an example of an appropriate detection line. For example, the appropriate detection line for the position of person C1 is detection line 7a on shelf 7. The appropriate detection line for the position of person C2 is detection line 7b on shelf 7. Because people move in front of shelf 7, using a preset detection line as in the prior art will result in erroneous detection of products being taken in or out.

[0010] In one aspect, the present invention aims to provide a setting program, a detection program, a setting method, and a setting device that can detect the insertion and removal of a product. [Means for solving the problem]

[0011] In the first proposal, a computer is caused to perform the following process. The computer acquires video from a camera set up inside a store having a storage unit in which products are stored. The computer inputs the acquired camera video into a machine learning model to identify depths indicating the distance from the camera for each component of the video. The computer generates a three-dimensional store model composed of the depths indicating the distance from the camera for each identified component of the video. The computer generates skeletal information of a person moving inside the store from the camera video. Based on the transitions in the generated skeletal information, the computer sets the range and direction of aisles inside the store in the generated three-dimensional store model. Based on the range and direction of aisles inside the store, the computer sets detection lines in the storage unit to detect when a person reaches for a product. [Effects of the Invention]

[0012] It can detect the insertion and removal of products. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing a system according to this embodiment. [Figure 2] FIG. 2 is a diagram (1) for explaining the processing of the information processing device according to the present embodiment. [Figure 3] FIG. 3 is a diagram (2) for explaining the processing of the information processing device according to the present embodiment. [Figure 4] FIG. 4 is a diagram for supplementing the processing of the information processing device according to the present embodiment. [Figure 5] FIG. 5 is a diagram for explaining an example of erroneous detection by the conventional technology. [Figure 6] FIG. 6 is a functional block diagram showing the configuration of an information processing device according to this embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of the data structure of the movement trajectory table. [Figure 8] FIG. 8 is a diagram illustrating an example of skeleton information. [Figure 9]FIG. 9 is a diagram for explaining the process of detecting the behavior of taking out a product. [Figure 10] FIG. 10 is a diagram for explaining other processes of the determination unit. [Figure 11] FIG. 11 is a flowchart showing the presetting process executed by the information processing device. [Figure 12] FIG. 12 is a flowchart showing the detection process executed by the information processing device. [Figure 13] FIG. 13 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. [Figure 14] FIG. 14 is a diagram for explaining the prior art. [Figure 15] FIG. 15 shows an example of a suitable detection line. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the setting program, detection program, setting method, and setting device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0015] Fig. 1 is a diagram showing a system according to this embodiment. As shown in Fig. 1, this system includes cameras 10a, 10b, and 10c and an information processing device 100. The information processing device 100 is an example of a setting device and a detection device. The cameras 10a to 10c and the information processing device 100 are connected to each other via a network 15.

[0016] Cameras 10a to 10c are cameras installed inside a store that has storage areas such as shelves for storing products. Cameras 10a to 10c capture video including the shelves installed inside the store and transmit information about the captured video to information processing device 100. In the following description, video information will be referred to as "video information." Video information includes time-series images (still images). Cameras 10a to 10c will be collectively referred to as "camera 10."

[0017] The information processing device 100 sets a detection line for detecting the taking in and out of products stored on the shelves based on the video information received from the camera 10. Furthermore, the information processing device 100 uses the set detection line to detect the taking in and out of products by the user.

[0018] 2 and 3 are diagrams for explaining the processing of the information processing device according to this embodiment. First, FIG. 2 will be explained. The information processing device 100 generates a depth image Im2 by inputting video information Im1 received from the camera 10 into a trained learning model that performs monocular depth estimation. The depth image Im2 has multiple pixels, and distance information is set for each pixel. Each pixel of the depth image Im2 corresponds to each pixel at the same coordinates in the video information (image) Im1. The distance information set for each pixel of the depth image Im2 indicates the distance between the camera 10 and the area inside the store corresponding to the pixel in the video information Im1.

[0019] When the information processing device 100 generates the depth image Im2, it is assumed that the video information Im1 does not include the user. That is, the information processing device 100 generates the depth image Im2 using the video information Im1 captured by the camera 10 during a time period when the user is not included.

[0020] The information processing device 100 generates skeletal information of the user moving around the store based on the video information Im1, and identifies a movement trajectory based on the transition of the feet in the generated skeletal information. When the information processing device 100 identifies a movement trajectory, it is assumed that the user is included in the video information Im1. That is, the information processing device 100 identifies the user's movement trajectory using the video information Im1 captured by the camera 10 during a time period in which the user is included.

[0021] 2 shows an example in which the information processing device 100 identifies movement trajectories tr1, tr2, tr3, and tr4. The information processing device 100 identifies a passage area f1 and a passage direction d1 based on the movement trajectories tr1, tr2, tr3, and tr4. For example, the information processing device 100 determines the movement range of the movement trajectories tr1 to tr4 as the passage area f1. The information processing device 100 determines the average vector of the displacement vectors of the movement trajectories tr1 to tr4 as the passage direction d1.

[0022] The information processing device 100 generates a 3D model of the inside of the store in a three-dimensional space based on the depth image Im2, the aisle region f1, and the path direction d1. For example, the information processing device 100 converts the distance information of the depth image Im2, the coordinates of the aisle region f1, and the coordinates of the path direction d1 into three-dimensional coordinates on the inside of the store 3D model and combines them.

[0023] Now, let us move on to the explanation of Fig. 3. For example, as shown in Fig. 3, shelves 8a and 8b, an aisle region f1, and an aisle direction d1 are set in the store 3D model M1. The information processing device 100 estimates skeletal information of the user C3 based on the video information Im3 including the user C3, and identifies the position p1 of the user C3's feet.

[0024] The information processing device 100 converts the position p1 into a position p1' on the store 3D model M1. The information processing device 100 sets a plane H1 on the store 3D model M1 based on the position p1' and the aisle direction d1. The line segment l1 at the bottom of the plane H1 passes through the position p1' and is perpendicular to the aisle direction d1.

[0025] The information processing device 100 utilizes the fact that product shelves 8a, 8b are installed perpendicular to the aisle direction d1 to set a detection line L10a connecting a point a1 where line segment l1 and shelf 8a meet and the top end b1 of shelf 8a. The information processing device 100 sets a detection line L10b connecting a point a2 where line segment l1 and shelf 8b meet and the top end b2 of shelf 8b. Each time user C1 moves and position p1 changes, the information processing device 100 executes the above process and resets the detection lines L10a, L10b. For example, the information processing device 100 calculates detection lines for multiple representative positions in advance, determines a representative position close to position p1, and uses the detection line corresponding to position p1. Alternatively, the information processing device 100 sets a detection line by integrating detection lines for multiple representative positions close to position p1 according to the distance between position p1 and the representative position.

[0026] The information processing device 100 utilizes the fact that the user C3 faces the shelf 8a (8b) when taking out an item from the shelf 8a (8b), and detects the taking out or putting in of an item by the user C3 using the detection lines L10a and L10b.

[0027] FIG. 4 is a diagram for supplementing the processing of the information processing device according to this embodiment. FIG. 4 shows the in-store 3D model M1 shown in FIG. 3 as viewed from above. As shown in FIG. 4, product shelves 8a are installed so as to face perpendicular to the aisle direction d1. For example, when a user stands at position p1' and takes a product from shelf 8a, the user faces shelf 8a. In other words, line segment l1 shown in FIG. 3 is perpendicular to the aisle direction d1.

[0028] As described above, the information processing device 100 generates an in-store 3D model M1 based on a depth image obtained from video information from the camera 10 and the user's movement trajectory, and sets shelf detection lines for each user's standing position on the floor for use in detection. By setting detection lines for each user's standing position in this way, it is possible to detect the taking in and out of products with higher accuracy than when detection lines are set in advance as in conventional technology.

[0029] FIG. 5 is a diagram illustrating an example of erroneous detection in the prior art. Image Im4-1 in FIG. 5 includes shelves 9A and 9B, and detection lines L9A and L9B are set in advance. Detection line L9A is used to detect the insertion and removal of products stored on shelf 9A. Detection line L9B is used to detect the insertion and removal of products stored on shelf 9B. Unlike the information processing device 100 of this embodiment, the detection lines L9A and L9B are not set according to the user's position. Therefore, even if user C4 moves in front of shelf 9A and does not insert or remove products from shelf 9A, the hand of user C4 may cross the detection line L9A, resulting in erroneous detection.

[0030] Image Im4-2 in FIG. 5 includes shelf 9C, and the outline of the shelf is set in advance as detection line L9C. Detection line L9C is used when detecting the insertion and removal of products stored on shelf 9C. This detection line L9C is not set according to the user's position, as in the information processing device 100 of this embodiment. Therefore, even if user C5 moves in front of shelf 9C and does not insert or remove products stored on shelf 9C, the hand of user C5 may cross detection line L9C, which may result in erroneous detection.

[0031] Next, a configuration example of an information processing device 100 according to this embodiment will be described. Fig. 6 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Fig. 6, this information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0032] The communication unit 110 transmits and receives information to and from the camera 10 via the network 15. The communication unit 110 receives video information from the camera 10. For example, the communication unit 110 is realized by a NIC (Network Interface Card) or the like.

[0033] The input unit 120 is realized using input devices such as a keyboard and a mouse, and inputs various information to the control unit 150 in response to input operations by an administrator or the like.

[0034] The display unit 130 is realized by a display device such as a liquid crystal display, etc. For example, the display unit 130 may display the processing results of the control unit 150.

[0035] The storage unit 140 includes a video buffer 141, depth image information 142, a movement trajectory table 143, and a store 3D model M1. The storage unit 140 is realized by, for example, a semiconductor memory element such as a flash memory, or a storage device such as a hard disk or an optical disk.

[0036] Video buffer 141 is a buffer that holds video information transmitted from camera 10. For example, video buffer 141 holds video information Im1 described in FIG. 2, video information Im3 described in FIG.

[0037] The depth image information 142 has multiple pixels, and distance information is set for each pixel. The multiple pixels of the video (image) correspond to multiple components. The distance information indicates the distance between the area inside the store corresponding to the pixel of the video information and the camera 10. The depth image information 142 corresponds to the depth image Im2 etc. described in FIG. 2.

[0038] Movement trajectory table 143 is a table that holds information on the movement trajectory of a user. FIG. 7 is a diagram showing an example of the data structure of the movement trajectory table. As shown in FIG. 7, this movement trajectory table 143 associates a user ID (Identification) with a movement trajectory. The user ID is information that identifies a user identified from the video information of camera 10. The movement trajectory indicates the movement trajectory of a user identified from the video information, and position information for each predetermined time interval is set.

[0039] The store 3D model M1 is a three-dimensional model of the store interior described in Fig. 3. For example, the store 3D model M1 defines the three-dimensional coordinates of each corner of the shelves 8a and 8b, the three-dimensional coordinates of each corner of the aisle area f1, and the three-dimensional coordinates of the start and end points indicating the aisle direction d1.

[0040] Returning to the description of Fig. 6, the control unit 150 has a receiving unit 151, a depth identification unit 152, a posture estimation unit 153, a movement trajectory calculation unit 154, a passage estimation unit 155, a detection line setting unit 156, and a determination unit 157. The control unit 150 is realized by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The control unit 150 may also be executed by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0041] The receiving unit 151 receives video information from the camera 10. The receiving unit 151 stores the received video information in the video buffer 141.

[0042] The depth determination unit 152 generates depth image information 142 by inputting video information received from the camera 10 during times when no users are in the store into a trained first learning model that performs monocular depth estimation. For example, this learning model is trained based on training data that uses video information as input and distance information for each pixel of the video information as a correct answer label. The depth determination unit 152 stores the depth image information 142 in the storage unit 140.

[0043] The posture estimation unit 153 estimates the user's skeletal information by inputting the video information received from the camera 10 during the time period when the user is in the store into a trained second learning model that estimates the user's skeletal information. For example, the posture estimation unit 153 inputs time-series images included in the video information into the second learning model in order, and estimates the skeletal information of each image in order. The posture estimation unit 153 outputs the time-series skeletal information to the movement trajectory calculation unit 154.

[0044] When detecting whether the user has reached out for a product, the posture estimation unit 153 outputs time-series skeletal information to the detection line setting unit 156 and the determination unit 157.

[0045] For example, the second learning model is a trained machine learning model that receives an image including a user's region as input and outputs skeletal information. For example, the second learning model is constructed using an existing algorithm such as DeepPose or OpenPose. FIG. 8 is a diagram illustrating an example of skeletal information. The skeletal information can use 18 pieces of definition information (numbered from 0 to 17) in which each joint identified by a known skeletal model is numbered. For example, the right shoulder joint (SHOULDER_RIGHT) is assigned number 7, the left elbow joint (ELBOW_LEFT) is assigned number 5, the left knee joint (KNEE_LEFT) is assigned number 11, and the right hip joint (HIP_RIGHT) is assigned number 14. Therefore, the coordinate information of the 18 skeletal joints shown in FIG. 8 can be acquired from the image data. For example, "X coordinate = X7, Y coordinate = Y7" is acquired as the position of the right shoulder joint numbered 7. The X axis can be defined as the horizontal direction of the image, and the Y axis can be defined as the vertical direction of the image.

[0046] The movement trajectory calculation unit 154 calculates the movement trajectory of the user based on the transition of the time-series skeletal information acquired from the posture estimation unit 153. The movement trajectory calculation unit 154 assigns a user ID to the movement trajectory, associates the user ID with the information of the user's movement trajectory, and stores the information in the movement trajectory table 143. For example, the movement trajectory calculation unit 154 identifies the position of the foot in the time-series skeletal information, and calculates the transition of the time-series foot position (for example, two-dimensional coordinates) as the movement trajectory. The position of the foot in the skeletal information is the coordinate of No. 17 (FOOT_RIGHT) or the coordinate of No. 13 (FOOT_LEFT) in the skeletal information described in FIG. 8, or the average value of No. 17 and No. 13, etc.

[0047] When the foot positions are not observed, the movement trajectory calculation unit 154 may estimate the foot positions from each joint in the remaining observed skeletal information and use the estimated foot positions as the foot positions.

[0048] The passage estimation unit 155 estimates the passage area and the passage direction based on the information of each movement trajectory stored in the movement trajectory table 143. The passage estimation unit 155 outputs the information of the estimated passage area and the passage direction to the detection line setting unit 156.

[0049] An example of the process of estimating a passage area by the passage estimation unit 155 will be described. The passage estimation unit 155 calculates a polygon that contains the foot positions of each movement trajectory, and estimates the calculated polygon as the passage area. For example, the information on the passage area estimated by the passage estimation unit 155 corresponds to the passage area f1 shown in FIG. 2.

[0050] An example of the process of estimating the aisle direction by the aisle estimation unit 155 will be described below. The average vector of the foot displacement vectors of each movement trajectory is estimated as the aisle direction. For example, the aisle direction information estimated by the aisle estimation unit 155 corresponds to the aisle direction d1 shown in FIG. 2.

[0051] The detection line setting unit 156 sets a detection line on the shelf to detect when a user reaches out for a product. For example, the detection line setting unit 156 performs a process to set the detection line after performing a process to generate the in-store 3D model M1.

[0052] An example of a process for generating a store 3D model M1 by the detection line setting unit 156 will be described below. The detection line setting unit 156 generates a store 3D model M1 in a three-dimensional space based on the depth image information 142, the aisle area f1, and the route direction d1.

[0053] For example, the detection line setting unit 156 converts the distance information of the depth image information 142 into three-dimensional coordinates on the in-store 3D model M1 based on the distance information of the depth image information 142 and the first definition table. The first definition table is a table that defines the relationship between the distance information and the three-dimensional coordinates on the in-store 3D model M1.

[0054] The detection line setting unit 156 converts the coordinates of the passage area f1 and the coordinates of the path direction d1 into three-dimensional coordinates on the in-store 3D model M1 based on the coordinates of the passage area f1 and the coordinates of the path direction d1 and the second definition table. The second definition table defines the relationship between the coordinates of the passage area f1 and the coordinates of the path direction d1 and the three-dimensional coordinates on the in-store 3D model M1.

[0055] For example, the detection line setting unit 156 identifies, as shelves, objects in an area other than the aisle area f1 among objects on the store 3D model M1 obtained from the distance information of the depth image information 142. In the example described in Fig. 3, the detection line setting unit 156 identifies, in the store 3D model M1, objects other than the aisle area f1 as shelves 8a and 8b.

[0056] Next, an example of the process by which the detection line setting unit 156 sets a detection line will be described. For example, when detecting whether a user has reached out for a product, the detection line setting unit 156 acquires skeletal information from the posture estimation unit 153 and calculates the position (three-dimensional coordinates) of the user's feet. For example, the foot position calculated by the detection line setting unit 156 is set to position p1' shown in FIG. 3. The detection line setting unit 156 places the surface H1 in the in-store 3D model M1 so that the line segment l1 at the bottom end of the surface H1 passes through position p1' and is perpendicular to the aisle direction d1. For example, the coordinates of each joint in the skeletal information estimated by the posture estimation unit 153 are two-dimensional coordinates, and the detection line setting unit 156 calculates the three-dimensional coordinates using a conversion table or the like that converts two-dimensional coordinates into three-dimensional coordinates.

[0057] The detection line setting unit 156 sets a detection line L10a that connects the point a1 where the line segment l1 and the shelf 8a contact each other, and the top end b1 of the shelf 8a. The detection line setting unit 156 sets a detection line L10b that connects the point a2 where the line segment l1 and the shelf 8b contact each other, and the top end b2 of the shelf 8b. The detection line setting unit 156 outputs information about the set detection lines L10a and L10b to the determination unit 157.

[0058] The detection line setting unit 156 executes the above process each time the user moves and the position p1′ changes, resets the detection lines L10a and L10b, and outputs information on the reset detection lines L10a and L10b to the determination unit 157.

[0059] The determination unit 157 detects that the user has reached out for a product and determines whether or not the product has been taken from the shelf, based on the detection line acquired from the detection line setting unit 156. An example of the processing of the determination unit 157 will be described below.

[0060] Determination unit 157 acquires skeletal information from posture estimation unit 153 and calculates the position (two-dimensional coordinates) of the user's hand. For example, determination unit 157 calculates the coordinate of No. 6 (WRIST_RIGHT) or the coordinate of No. 9 (WRIST_LEFT) in the skeletal information described in FIG. 8 as the position of the user's hand. In the following description, coordinate No. 6 (WRIST_RIGHT) is referred to as the position of the right hand, and coordinate No. 9 (WRIST_LEFT) is referred to as the position of the left hand.

[0061] The processing of the determination unit 157 will be described using the detection line L10a. The determination unit 157 compares the position of the detection line L10a with the positions of the right hand and the left hand, and determines the hand that is closer to the position of the detection line L10a as the hand of interest. Here, the description will be given assuming that the right hand is the hand of interest.

[0062] The determination unit 157 calculates the distance between the position of the detection line L10a and the position of the right hand, and if the distance is less than a threshold, determines that the user has reached for a product on the shelf. If the determination unit 157 determines that the user has reached for a product on the shelf, it performs the following process to detect the user's action of taking out the product.

[0063] 9 is a diagram for explaining the process of detecting the behavior of taking out a product. The determination unit 157 acquires an image (video information) of the area 25 set near the detection line L10a from the video buffer 141. As described above, the position of the detection line L10a changes depending on the position of the user C1, and therefore, the position of the area 25 dynamically changes depending on the position of the detection line L10a.

[0064] For example, an image of region 25 when user C1 reaches his / her hand out to shelf 20a is defined as image 25a. An image of region 25 when user C1 removes his / her hand from shelf 20a is defined as image 25b. Determination unit 157 determines whether or not a product is present in user C1's hand using identification model 26. Identification model 26 is a model that identifies whether or not a product is present in the user's hand, and may be a learning model based on HOID (Human Object Interaction Detection) or the like.

[0065] The judgment unit 157 detects that the user C1 has taken the action of taking out a product when the image 25a is input into the identification model 26 and is identified as "no product" and the image 25b is input into the identification model 26 and is identified as "product present."

[0066] On the other hand, when image 25a is input into identification model 26 and is identified as "product present," and when image 25b is input into identification model 26 and is identified as "product absent," judgment unit 157 detects that user C1 has taken the action of returning the product.

[0067] Here, when using video information (image) from camera 10, the background to the user may be a shelf, and the same product as the product held by the user may be captured in the background, reducing the accuracy of determining whether the user is holding the product. Determination unit 157 may improve the accuracy of image recognition by comparing an image (background image) from a time period before user C appeared with the current image.

[0068] Fig. 10 is a diagram for explaining other processing by the determination unit. As shown in Fig. 10, when determination unit 157 retrieves input image Im5-1 (video information) from video buffer 141 and performs image processing, it generates a foreground mask Im5-3 by subtracting a background image Im5-2 prepared in advance from input image Im5-1. Determination unit 157 compares input image Im5-1 with foreground mask Im5-3 and generates image Im5-4 by extracting the area other than mask A1. Since image Im5-4 does not include a background image, it is possible to improve the accuracy when executing the detection processing described in Fig. 9.

[0069] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 11 is a flowchart showing a pre-setting process executed by the information processing device. As shown in Fig. 11, the receiving unit 151 of the information processing device 100 receives video information from the camera 10 and stores it in the video buffer 141 (step S101).

[0070] The depth determination unit 152 of the information processing device 100 inputs the video information into the first learning model and generates depth image information 142 (step S102). The posture estimation unit 153 of the information processing device 100 inputs the video information into the second learning model and generates skeletal information (step S103).

[0071] The movement trajectory calculation unit 154 of the information processing device 100 generates movement trajectory information based on the time-series skeleton information and stores it in the movement trajectory table 143 (step S104). The passage estimation unit 155 of the information processing device 100 estimates the passage area and the passage direction based on the movement trajectory information stored in the movement trajectory table 143 (step S105).

[0072] The detection line setting unit 156 of the information processing device 100 generates a store interior 3D model M1 based on the depth image information 142, the aisle area, and the aisle direction (step S106).

[0073] Fig. 12 is a flowchart showing the detection process executed by the information processing device. As shown in Fig. 12, the receiving unit 151 of the information processing device 100 receives video information from the camera 10 and stores it in the video buffer 141 (step S201).

[0074] The posture estimation unit 153 of the information processing device 100 inputs the video information into a second learning model and generates skeletal information (step S202). The detection line setting unit 156 of the information processing device 100 calculates the foot positions based on the skeletal information (step S203). The detection line setting unit 156 sets a detection line based on the in-store 3D model M1 and the foot positions (step S204).

[0075] The determination unit 157 of the information processing device 100 calculates the position of the hand based on the skeletal information (step S205). The determination unit 157 calculates the distance between the position of the detection line and the position of the hand (step S206). If the distance is less than the threshold (step S207, Yes), the determination unit 157 determines that the hand has reached out for a product on the shelf, and detects the behavior of taking out the product (step S208).

[0076] On the other hand, if the distance is not less than the threshold value (step S207, No), the determination unit 157 ends the process.

[0077] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 generates an in-store 3D model M1 based on depth image information 142 obtained from video information from the camera 10 and the user's movement trajectory, and sets shelf detection lines for each user's standing position on the floor for use in detection. By setting detection lines for each user's standing position in this way, it is possible to detect the taking in and out of products with higher accuracy than when detection lines are set in advance as in conventional technology.

[0078] The information processing device 100 identifies the transition of the user's foot position based on the user's chronological skeletal information, and identifies a polygon that contains the identified transition of the foot position as an aisle area. This makes it possible to identify the aisle area and the 3D positions of the shelves in the in-store 3D model M1.

[0079] The information processing device 100 identifies transitions in the user's foot positions based on the user's time-series skeletal information, and identifies the aisle direction based on a displacement direction vector based on the identified transitions in the foot positions. By placing a plane that intersects the identified aisle direction at a right angle in the in-store 3D model M1, the detection line can be set with high accuracy.

[0080] The information processing device 100 sets a detection line for each position where the user is standing, identifies the body part corresponding to the user's hand based on the user's skeletal information, and detects that the user has reached out for a product stored on a shelf based on the positional relationship between the detection line and the hand part. This makes it possible to detect the user's action of taking a product from the shelf.

[0081] Furthermore, according to the information processing device 100, by setting the in-store 3D model M1 in advance and identifying only the position of the user's feet, it is possible to set a detection line and reduce the processing load of the information processing device 100. Furthermore, it is possible to detect the behavior of a user taking a product from a shelf without using a high-resolution camera.

[0082] Next, an example of the hardware configuration of a computer that realizes the same functions as the information processing device 100 described in the above embodiment will be described. Fig. 13 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0083] 13, computer 300 includes CPU 301 for executing various types of arithmetic processing, input device 302 for receiving data input from a user, and display 303. Computer 300 also includes communication device 304 for transmitting and receiving data to and from external devices via a wired or wireless network, and interface device 305. Computer 300 also includes RAM 306 for temporarily storing various types of information, and hard disk drive 307. Devices 301 to 307 are connected to bus 308.

[0084] The hard disk drive 307 stores a reception program 307a, a depth determination program 307b, a posture estimation program 307c, a movement trajectory calculation program 307d, and a passage estimation program 307e. The hard disk drive 307 also stores a detection line setting program 307f and a determination program 307g. The CPU 301 reads out each of the programs 307a to 307g and loads them into the RAM 306.

[0085] The receiving program 307a functions as a receiving process 306a. The depth determination program 307b functions as a depth determination process 306b. The attitude estimation program 307c functions as an attitude estimation process 306c. The movement trajectory calculation program 307d functions as a movement trajectory calculation process 306d. The passage estimation program 307e functions as a passage estimation process 306e. The detection line setting program 307f functions as a detection line setting process 306f. The judgment program 307g functions as a judgment process 306g.

[0086] The processing of the reception process 306a corresponds to the processing of the reception unit 151. The processing of the depth determination process 306b corresponds to the processing of the depth determination unit 152. The processing of the posture estimation process 306c corresponds to the processing of the posture estimation unit 153. The processing of the movement trajectory calculation process 306d corresponds to the processing of the movement trajectory calculation unit 154. The processing of the passage estimation process 306e corresponds to the processing of the passage estimation unit 155. The processing of the detection line setting process 306f corresponds to the processing of the detection line setting unit 156. The processing of the determination process 306g corresponds to the processing of the determination unit 157.

[0087] It should be noted that each of the programs 307a to 307g does not necessarily have to be stored in the hard disk drive 307 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 300. Then, the computer 300 may read and execute each of the programs 307a to 307g.

[0088] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0089] (Appendix 1) Obtaining images from cameras installed in a store having a storage section where products are stored, The acquired image of the camera is input into a machine learning model to identify a depth indicating a distance from the camera for each component of the image; generating a three-dimensional model of the store interior constructed from depths indicating the distances from the camera of each of the identified components of the video; Generate skeletal information of a person moving within the store from the image captured by the camera; Based on the transition of the generated skeleton information, a range and a direction of an aisle in the store are set in the generated three-dimensional store model; A detection line for detecting when a person reaches for the product is set in the storage unit based on the range and direction of the aisle in the store. A setting program that causes a computer to execute a process.

[0090] (Appendix 2) The setting program described in Appendix 1 is characterized in that the process of setting the range and direction of the aisles within the store identifies the transition in foot position contained in the person's skeletal information, and sets a polygon that contains the identified transition in foot position as the range of the aisle.

[0091] (Appendix 3) The setting program described in Appendix 1 or 2, characterized in that the process of setting the range and direction of the aisles in the store identifies the transition of foot position contained in the skeletal information of the person, and sets the direction based on a displacement direction vector based on the transition of the foot position.

[0092] (Appendix 4) In a three-dimensional store model composed of depths indicating the distance from the camera for each component of the image of a camera set up in the store, setting information is specified that sets a detection line for detecting when a person reaches for a product based on the range and direction of the aisles in the store, in a storage unit arranged in the store, Identifying a part of the person corresponding to the hand based on skeletal information of the person generated from the video of the inside of the store; Based on the positional relationship between the identified part corresponding to the person's hand and the detection line included in the setting information, it is detected that the person has reached out for a commodity contained in the container. A detection program that causes a computer to execute a process.

[0093] (Appendix 5) Obtaining images from cameras installed in a store having a storage section where products are stored, The acquired image of the camera is input into a machine learning model to identify a depth indicating a distance from the camera for each component of the image; generating a three-dimensional model of the store interior constructed from depths indicating the distances from the camera of each of the identified components of the video; Generate skeletal information of a person moving within the store from the image captured by the camera; Based on the transition of the generated skeleton information, a range and a direction of an aisle in the store are set in the generated three-dimensional store model; A detection line for detecting when a person reaches for the product is set in the storage unit based on the range and direction of the aisle in the store. A setting method characterized in that the processing is executed by a computer.

[0094] (Appendix 6) The setting method described in Appendix 5 is characterized in that the process of setting the range and direction of the aisles within the store identifies the transition of foot position contained in the skeletal information of the person, and sets a polygon that contains the identified transition of foot position as the range of the aisle.

[0095] (Appendix 7) The setting method described in Appendix 5 or 6, characterized in that the process of setting the range and direction of the aisles in the store identifies the transition of foot position contained in the skeletal information of the person, and sets the direction based on a displacement direction vector based on the transition of the foot position.

[0096] (Appendix 8) In a three-dimensional store model composed of depths indicating the distance from the camera for each component of the image of a camera set in the store, setting information is specified that sets a detection line for detecting when a person reaches for a product based on the range and direction of the aisles in the store, in a storage unit arranged in the store, Identifying a part of the person corresponding to the hand based on skeletal information of the person generated from the video of the inside of the store; Based on the positional relationship between the identified part corresponding to the person's hand and the detection line included in the setting information, it is detected that the person has reached out for a commodity contained in the container. A detection method characterized in that the processing is performed by a computer.

[0097] (Appendix 9) Obtaining images from cameras installed in a store having a storage section where products are stored, The acquired image of the camera is input into a machine learning model to identify a depth indicating a distance from the camera for each component of the image; generating a three-dimensional model of the store interior constructed from depths indicating the distances from the camera of each of the identified components of the video; Generate skeletal information of a person moving within the store from the image captured by the camera; Based on the transition of the generated skeleton information, a range and a direction of an aisle in the store are set in the generated three-dimensional store model; A detection line for detecting when a person reaches for the product is set in the storage unit based on the range and direction of the aisle in the store. A setting device having a control unit that executes processing.

[0098] (Appendix 10) The process of setting the range and direction of the aisles within the store is characterized in that the setting device described in Appendix 9 identifies the transition of foot position contained in the skeletal information of the person, and sets a polygon that contains the identified transition of foot position as the range of the aisle.

[0099] (Appendix 11) The process of setting the range and direction of the aisles in the store is characterized in that the setting device described in Appendix 9 or 10 identifies the transition of foot position contained in the skeletal information of the person, and sets the direction based on a displacement direction vector based on the transition of the foot position.

[0100] (Appendix 12) In a three-dimensional store model composed of depths indicating the distance from the camera for each component of the image of a camera set in the store, setting information is specified that sets a detection line for detecting when a person reaches for a product based on the range and direction of the aisles in the store, in a storage unit arranged in the store, Identifying a part of the person corresponding to the hand based on skeletal information of the person generated from the video of the inside of the store; Based on the positional relationship between the identified part corresponding to the person's hand and the detection line included in the setting information, it is detected that the person has reached out for a commodity contained in the container. A detection device having a control unit that performs processing. [Explanation of symbols]

[0101] 100 Information processing device 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 Video Buffer 142 Depth image information 143 Movement trajectory table 150 control section 151 Receiving unit 152 Depth identification part 153 Posture estimation section 154 Movement trajectory calculation unit 155 Passage estimation part 156 Detection line setting section 157 Judgment section

Claims

1. Acquire an image from a camera installed in a store having a storage section in which the product is stored; The acquired image of the camera is input into a machine learning model to identify a depth indicating a distance from the camera for each component of the image; generating a three-dimensional model of the interior of the store composed of depths indicating the distances from the camera of each of the identified components of the video; Generate skeletal information of a person moving within the store from the image captured by the camera; based on the transition of the generated skeleton information, set the range and direction of the aisles in the store in the generated three-dimensional store model; A detection line for detecting when a person reaches for the product is set in the storage unit based on the range and direction of the aisle in the store. A setting program that causes a computer to execute a process.

2. The setting program according to claim 1, characterized in that the process of setting the range and direction of the aisles in the store identifies the transition of foot positions contained in the skeletal information of the person, and sets a polygon that contains the identified transition of foot positions as the range of the aisles.

3. The setting program according to claim 1 or 2, characterized in that the process of setting the range and direction of the aisles in the store identifies transitions in foot position contained in the person's skeletal information, and sets the directions based on displacement direction vectors based on the transitions in foot position.

4. Acquire an image from a camera installed in a store having a storage section in which the product is stored; The acquired image of the camera is input into a machine learning model to identify a depth indicating a distance from the camera for each component of the image; generating a three-dimensional model of the interior of the store composed of depths indicating the distances from the camera of each of the identified components of the video; Generate skeletal information of a person moving within the store from the image captured by the camera; based on the transition of the generated skeleton information, set the range and direction of the aisles in the store in the generated three-dimensional store model; A detection line for detecting when a person reaches for the product is set in the storage unit based on the range and direction of the aisle in the store. A setting method characterized in that the processing is executed by a computer.

5. Acquire an image from a camera installed in a store having a storage section in which the product is stored; The acquired image of the camera is input into a machine learning model to identify a depth indicating a distance from the camera for each component of the image; generating a three-dimensional model of the interior of the store composed of depths indicating the distances from the camera of each of the identified components of the video; Generate skeletal information of a person moving within the store from the image captured by the camera; based on the transition of the generated skeleton information, set the range and direction of the aisles in the store in the generated three-dimensional store model; A detection line for detecting when a person reaches for the product is set in the storage unit based on the range and direction of the aisle in the store. A setting device having a control unit that executes processing.

Images