Purchase analysis device, purchase analysis method, and program

Abstract

Provided are a purchase analysis device, a purchase analysis method, and the like for analyzing in detail the behavior of a customer concerning a commodity or the like. A purchase analysis device (100a) comprises: a movement specification unit (107a) which analyzes movement of a customer in a selling area included in captured video data, and specifies the movement of the customer in accordance with a stored movement pattern; a commodity related information specification unit (108a) which specifies information related to commodity to which the customer is showing interest, on the basis of the specified movement of the customer or the specified position of the customer; and an association unit (109a) which associates the specified movement with the specified commodity related information.

Description

[Technical Field]

[0001] The present disclosure relates to a purchasing analysis device, a purchasing analysis method, and a non-transitory computer-readable medium. [Background technology]

[0002] Technologies for analyzing customer behavior in retail stores have been developed using POS (Point of Sale) and other methods.

[0003] For example, Patent Document 1 discloses a behavioral analysis device that analyzes customer behavior within a store, and includes a posture estimation unit that estimates the posture of the customer within the store based on images taken within the store to obtain posture information, a product fluctuation information extraction unit that analyzes the display state of products within the store from the images and extracts product fluctuation information that indicates changes in the display state, and a purchasing behavior determination unit that determines the purchasing behavior of the customer based on the posture information and the product fluctuation information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-211891 Summary of the Invention [Problem to be solved by the invention]

[0005] However, it is not possible to analyze in detail customer behavior regarding products, such as customer behavior regarding products that were not purchased.

[0006] In view of the above-mentioned problems, an object of the present disclosure is to provide a purchase analysis device, a purchase analysis method, and a non-transitory computer-readable medium that perform a more detailed analysis of customer behavior regarding products in a sales area. [Means for solving the problem]

[0007] A purchase analysis device according to one aspect of the present disclosure includes: a movement identification means for analyzing the movements of customers in the sales area included in the captured video data and identifying the movements of the customers in accordance with the stored movement patterns; a product-related information specifying means for specifying product-related information in which the customer is interested based on the specified customer's behavior or location; an association means for associating the identified action with the product-related information; Equipped with.

[0008] A purchase analysis method according to one aspect of the present disclosure includes: Analyzing the customer's movements in the sales area contained in the captured video data, and identifying the customer's movements according to the stored movement patterns; Identifying product-related information in which the customer is interested based on the identified customer behavior or location; The identified action is associated with the product-related information.

[0009] According to one aspect of the present disclosure, there is provided a non-transitory computer-readable medium, comprising: Analyzing the customer's movements in the sales area contained in the captured video data, and identifying the customer's movements according to the stored movement patterns; Identifying product-related information in which the customer is interested based on the identified customer behavior or location; The storage device stores a program for causing a computer to execute a purchase analysis method for associating the identified behavior with the product-related information. [Effects of the Invention]

[0010] The present disclosure can provide a purchase analysis device, a purchase analysis method, and a non-transitory computer-readable medium that perform detailed analysis of customer behavior regarding products and the like. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of a purchase analysis device according to a first embodiment. [Figure 2] 1 is a flowchart showing the flow of a purchase analysis method according to the first embodiment. [Figure 3] FIG. 10 is a block diagram showing the configuration of a purchase analysis device according to a second embodiment. [Figure 4] 10 is a flowchart showing the flow of a purchase analysis method according to a second embodiment. [Figure 5] FIG. 10 is a block diagram showing the configuration of a purchase analysis device according to a third embodiment. [Figure 6] 10 is a flowchart showing the flow of a purchase analysis method according to a third embodiment. [Figure 7] FIG. 10 is a diagram showing the overall configuration of a purchase analysis device according to a fourth embodiment. [Figure 8] FIG. 10 is a block diagram showing detailed configurations of a purchase analysis device 100 and a POS management device 200 according to a fourth embodiment. [Figure 9] FIG. 11 is a diagram showing skeletal information of a customer extracted from a frame image included in video data according to the fourth embodiment. [Figure 10] FIG. 11 is an enlarged view of a customer's hand included in a frame image according to a fourth embodiment. [Figure 11] FIG. 11 is an enlarged view of a customer's hand included in a frame image according to a fourth embodiment. [Figure 12] FIG. 10 is a diagram showing frame images included in video data according to the fourth embodiment. [Figure 13] 13 is a flowchart showing the flow of a method for registering a registration operation ID and a registration operation sequence by a server according to a fourth embodiment. [Figure 14] 10A to 10C are diagrams for explaining various registration operations according to the fourth embodiment. [Figure 15] FIG. 10 is a diagram for explaining an operation sequence with a high probability of purchase according to the fourth embodiment. [Figure 16] FIG. 10 is a diagram for explaining a sequence with a low probability of purchase according to the fourth embodiment. [Figure 17] 10 is a flowchart showing the flow of a purchase analysis method by the purchase analysis device 100 according to the fourth embodiment. [Figure 18] FIG. 10 is a block diagram showing the configuration of an imaging device according to a fifth embodiment. [Figure 19] FIG. 2 is a block diagram showing a hardware configuration of the purchase analysis device. DETAILED DESCRIPTION OF THE INVENTION

[0012] The present disclosure will be described below through embodiments, but the disclosure according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations are omitted as necessary.

[0013] <Embodiment 1> FIG. 1 is a block diagram showing the configuration of a purchase analysis device 100a according to a first embodiment. The purchase analysis device 100a may be realized by a computer server or the like equipped with a processor and a memory. The purchase analysis device 100a may be used to analyze and identify customer behavior in video data and provide analysis information that associates the behavior with information about products and the like (also referred to as product-related information). Specifically, the purchase analysis device 100a includes: a behavior identification unit 107a that analyzes customer behavior in a sales floor included in captured video data and identifies the customer behavior according to stored behavior patterns; a product-related information identification unit 108a that identifies product-related information in which the customer is interested based on the identified customer behavior or customer location; and an association unit 109a that associates the identified behavior with the product-related information.

[0014] The behavior identification unit 107a can identify characteristic purchasing behaviors of customers from video data captured in various sales areas such as supermarkets, home improvement stores, and convenience stores. Characteristic purchasing behaviors can be various behaviors such as picking up a product, returning a product to a shelf, or comparing products in front of a shelf. The customer behaviors that can be identified can be one or more customer behaviors, or a series of consecutive customer behaviors.

[0015] The product-related information that is likely to interest the customer is information associated with the product or the like in which the customer is interested, and can include, for example, at least one or all of the following: product (e.g., product number, product name, etc.), product classification (e.g., chocolate, sweets, beverages, etc.), product shelf, and floor map information associated with these pieces of information. Note that floor map information, also called a floor guide, can be information that allows customers to find where various products are located within the sales floor.

[0016] The product-related information identification unit 108a can identify product-related information that the customer is likely to be interested in by comprehensively determining the customer's standing position and the customer's behavior. For example, if a customer picks up a product, it can be determined that the customer is likely to be interested in that product. Also, if a customer stands in front of a product shelf and directs their gaze or face toward a specific product for a predetermined period of time or longer, it can be determined that the customer is likely to be interested in that specific product. In some embodiments, the product-related information identification unit 108a may identify the product, product shelf, etc. that is closest in distance to the location of the customer who performed the identified behavior as the product-related information.

[0017] The associating unit 109a can associate the identified motion with the product-related information in various formats. The various formats may be any format that allows an analyst to recognize the association between the identified motion and the product-related information, and may be, for example, an association between a video or image and a code indicating the product-related information, or an association in the form of a table between the content (type) of the motion and a code indicating the product-related information.

[0018] FIG. 2 is a flowchart showing the flow of the purchase analysis method according to the first embodiment. The purchase analysis method according to the first embodiment includes the following steps: The behavior identification unit 107a analyzes the behavior of customers in the sales area included in the captured video data, and identifies the behavior of the customers according to the stored behavior patterns (step S101a); The product-related information identification unit 108a identifies product-related information in which the customers are interested, based on the identified behavior of the customers or their locations (step S102a); and the association unit 109a associates the identified predetermined behavior with the product-related information (step S103a).

[0019] According to the first embodiment described above, it is possible to provide information for analyzing in detail customer behavior regarding products and the like.

[0020] <Embodiment 2> FIG. 3 is a block diagram showing the configuration of a purchase analysis device 100b according to the second embodiment. The basic configuration of the purchase analysis device 100b according to the second embodiment is the same as that of the first embodiment, and detailed description thereof will be omitted. The storage unit 103b of the purchase analysis device 100b according to the second embodiment stores a movement pattern LP with a relatively low purchase likelihood and a movement pattern HP with a high purchase likelihood. The movement identification unit 107b according to the second embodiment can identify customer movement according to the movement pattern LP with a relatively low purchase likelihood and the movement pattern HP with a high purchase likelihood. The association unit 109b according to the second embodiment can associate some product-related information with the movement of customers with a high purchase likelihood, and associate some other product-related information with the movement of customers with a low purchase likelihood.

[0021] In some other embodiments, the storage unit 103b stores at least the behavior pattern LP with a relatively low purchasing likelihood. The behavior identification unit 107b identifies a behavior with a relatively low purchasing likelihood of a customer based on the stored behavior patterns with a relatively low purchasing likelihood. The association unit 109b associates the behavior with a relatively low purchasing likelihood of a specific customer with the identified product-related information.

[0022] FIG. 4 is a flowchart showing the flow of the purchase analysis method according to the second embodiment. The purchase analysis method according to the second embodiment includes the following steps: The behavior identification unit 107b analyzes the behavior of customers in the sales area included in the captured video data, and identifies the behavior of the customers according to stored behavior patterns associated with a relatively high purchase likelihood or a relatively low purchase likelihood (step S101b). The product-related information identification unit 108b identifies product-related information in which the customers are interested, based on the identified customer behavior or customer location (step S102b). The association unit 109b associates the identified customer behavior with the product-related information (step S103b).

[0023] In some embodiments, the behavior identifying unit 107b may identify a behavior of the customer with a relatively low purchasing likelihood based on the stored behavior patterns with a relatively low purchasing likelihood, and the associating unit 109b may associate the identified behavior with a relatively low purchasing likelihood with product-related information corresponding to the identified behavior.

[0024] This makes it possible to obtain related information indicating the likelihood of purchase of product-related information, thereby enabling detailed purchase analysis. Specifically, products can be classified into those associated with actions that are relatively likely to be purchased and those associated with actions that are relatively unlikely to be purchased. In particular, it is possible to obtain information indicating whether or not customers are interested in product-related information such as products or product categories that have not been purchased or are likely not to be purchased, which was not possible with previous POS systems, thereby enabling more detailed purchase analysis.

[0025] <Embodiment 3> FIG. 5 is a block diagram showing the configuration of a purchase analysis device 100c according to a third embodiment. The purchase analysis device 100c can be used to analyze and identify customer behaviors in video data and provide information associating the behaviors with information about products and other items (also referred to as product-related information) and sales information for those products. Specifically, the purchase analysis device 100c includes: a behavior identification unit 107c that analyzes customer behaviors in a sales floor included in captured video data and identifies the customer's behavior according to stored behavior patterns; a product-related information identification unit 108c that identifies product-related information in which the customer is interested based on the identified customer behavior or customer location; an association unit 109c that associates the identified behaviors with the product-related information; and a POS linkage unit 110c that acquires sales information for the identified products or product-related information from the POS management device 200 based on the identified behaviors. The POS linkage unit 110c links with the POS terminal device and the POS management device 200c to support the association by the association unit 109c. The storage unit 103c stores various customer behavior patterns.

[0026] FIG. 5 is a flowchart showing the flow of the purchase analysis method according to the third embodiment. The purchase analysis method according to the third embodiment includes the following steps: The behavior identification unit 107c analyzes the behavior of customers in the sales area included in the captured video data and identifies the behavior of the customers according to various stored behavior patterns (step S101c). The product-related information identification unit 108c identifies product-related information in which the customers are interested based on the identified behavior of the customers or their locations (step S102c). The associating unit 109c associates the identified predetermined behavior with the product-related information (step S103c). The associated information is linked to sales information from the POS management device (step S104c).

[0027] The POS linking unit 110c can recognize whether the predetermined action identified by the action identifying unit 107c and the predetermined product identified by the product-related information identifying unit 108c have actually been purchased. Therefore, the associating unit 109c, in cooperation with the POS linking unit 110c, can obtain information indicating whether the customer is interested in a product that was not purchased, or what purchasing action the customer is taking, and can perform a more detailed purchase analysis.

[0028] <Embodiment 4> Next, a fourth embodiment of the present disclosure will be described. 7 is a diagram showing the overall configuration of the purchase analysis system 1 according to the embodiment 4. As an example, the general flow of a purchase made by a customer C in a sales floor 50 of a store is as follows. (1) First, customer C grabs an item from a shelf in the sales area and places it in a basket, or moves away from the shelf with the item. (2) Customer C goes to another shelf, grabs another product with his / her hand, and either places the product in his / her basket or moves away from the shelf with the product, repeating this action. (3) Customer C places all the items he or she wishes to purchase into a basket or carries them away from the shelves and proceeds to the cash register. (4) Customer C completes the payment for all items using the POS system.

[0029] Next, various examples will be described in which customer C performs non-purchasing actions in the sales area 50 of the store that do not ultimately result in a purchase. Customer C grabs an item from the shelf, but then returns it to the shelf. Customer C stops in front of a shelf and looks at the products for a certain period of time, then moves on to another location. Customer C stops in front of a shelf, reaches for an item, then pulls his hand back and moves on to another location. Customer C stops in front of a shelf, picks up several products, compares them, but puts only one in his basket and returns the others to the shelf. The above non-purchasing actions are merely examples, and various other non-purchasing actions may be used.

[0030] Using POS (Point of Sale) has made it possible to recognize and analyze products that have led to purchases, but it has not been possible to analyze various purchasing behaviors, including non-purchasing behaviors such as those mentioned above.

[0031] The present disclosure relates to analyzing products and the like by associating various purchasing behaviors, including non-purchasing behaviors, with product-related information. As shown in FIG. 7, the purchase analysis system 1 is a computer system that monitors a customer C who visits a sales floor 50 using one or more cameras 300, detects predetermined behaviors of the customer C, and analyzes products and the customer's purchasing behavior by linking with a POS system. As used herein, non-purchasing behavior or behavior refers to a customer's behavior or behavior that ultimately does not result in a purchase or is likely not to result in a purchase.

[0032] Here, the purchase analysis system 1 comprises a purchase analysis device (server) 100, a POS management device 200, one or more cameras 300 in a sales floor 50, and one or more POS terminal devices 400 in the sales floor. The components are connected to each other via a network N. The network N may be wired or wireless.

[0033] Although only one camera 300 is shown in Fig. 7, multiple cameras 300 may be provided and installed in various locations in the sales floor 50 to capture images of customer C and monitor the purchasing behavior of customer C. The camera 300 may be disposed at a position and angle that allows it to capture an image of at least a part of the body of customer C standing in front of a product shelf. Preferably, the camera 300 may be disposed at a position and angle that allows it to recognize the relationship between customer C standing in front of the product shelf and the product shelf.

[0034] The purchase analysis device 100 acquires video data of the sales floor from the camera 300 via the network N. Based on the video data received from the camera 300, the purchase analysis device 100 detects purchasing behavior by customer C related to product-related information of the sales floor 50. The product-related information may include products in which customer C has shown interest, or product shelves or floor map information of the sales floor that may be associated with the products. In other words, the purchase analysis device 100 can perform purchase analysis by obtaining information that associates product-related information with customer behavior in the sales floor. In some embodiments, the customer's purchasing behavior may be behavior near product shelves.

[0035] The POS management device 200 can aggregate sales data for each product sent from one or more POS terminal devices 400 (also called POS registers) installed in the sales floor 50 of the store, and perform sales analysis or inventory management. In some embodiments, the POS management device 200 can receive analysis information, which is a combination of the purchasing behavior of customer C and product-related information, from the purchase analysis device 100, and associate the analysis information with the sales data. The POS management device 200 can receive this analysis information from the purchase analysis device 100 and display it using the display unit 203. In other embodiments, the purchase analysis device 100 and the POS management device 200 may be configured as a single unit. In yet another embodiment, the camera 300 and the purchase analysis device 100 may be configured as a single unit.

[0036] FIG. 8 shows an embodiment 4 1 is a block diagram showing detailed configurations of a purchase analysis device 100 and a POS management device 200. FIG.

[0037] (Purchase analysis device 100) The purchase analysis device 100 includes a registration information acquisition unit 101, a registration unit 102, an action DB 103, an action sequence table 104, a video acquisition unit 105, a customer identification unit 106, an action identification unit 107, a product-related information identification unit 108, an association unit 109, a POS linkage unit 110, and a processing control unit 111.

[0038] The registration information acquisition unit 101 is also called a registration information acquisition means. The registration information acquisition unit 101 acquires multiple pieces of registration video data from the camera 300 or another camera through operations by an administrator of the purchase analysis device 100 or the like. In the fourth embodiment, each piece of registration video data is video data showing past individual actions included in the purchasing behavior of a customer in a sales area (for example, the action of taking a product from a shelf, the action of putting a product in a cart, etc.). The registration video data is reference data for identifying the purchasing behavior of a customer in the video data acquired from the camera 300 during operation. Note that in the fourth embodiment, the registration video data is a video including multiple frame images, but in some embodiments, it may be a still image (one frame image).

[0039] Furthermore, the registration information acquisition unit 101 acquires information on a plurality of registered action IDs and the chronological order in which the actions are performed in a series of actions, through operations by an administrator or the like of the purchase analysis device 100. The registration information acquisition unit 101 supplies the acquired information to the registration unit 102.

[0040] The registration unit 102 is also referred to as a registration means. First, the registration unit 102 executes a motion registration process in response to a motion registration request. Specifically, the registration unit 102 supplies registration video data to the motion identification unit 107 (described later) and acquires skeleton information extracted from the registration video data from the motion identification unit 107 as registration skeleton information. The registration unit 102 then registers the acquired registration skeleton information in the motion DB 103 in association with a registration motion ID. The registration unit 102 can classify and register the extracted motion patterns, for example, according to purchase likelihood. The registration unit 102 can classify and register, for example, motion patterns (HP) with a relatively high purchase likelihood and motion patterns (LP) with a relatively low purchase likelihood. In another embodiment, the registration unit 102 can classify and register, for example, motion patterns (HP) with a relatively high purchase likelihood, motion patterns (MP) with a medium purchase likelihood, and motion patterns (LP) with a relatively low purchase likelihood. Note that these classifications are merely examples, and various modifications are possible.

[0041] Next, the registration unit 102 executes a sequence registration process in response to the sequence registration request. Specifically, the registration unit 102 generates a registration action sequence by chronologically arranging the registration action IDs based on the chronological order information. At this time, if the sequence registration request is for an action with a relatively high purchase possibility, the registration unit 102 registers the generated registration action sequence in the action sequence table 104 as an action sequence HS with a relatively high purchase possibility. On the other hand, if the sequence registration request is for an action with a relatively low purchase possibility, the registration unit 102 registers the generated registration action sequence in the action sequence table 104 as an action sequence LS with a relatively low purchase possibility.

[0042] The behavior DB 103 is a storage device that stores registered framework information corresponding to each behavior included in a purchasing behavior in association with a registered behavior ID. The behavior DB 103 may also store registered framework information corresponding to each behavior included in a behavior pattern (HP) with a relatively high purchase possibility and a behavior pattern (LP) with a relatively low purchase possibility in association with a registered behavior ID.

[0043] The action sequence table 104 stores action sequences HS with a relatively high purchase likelihood and action sequences LS with a relatively low purchase likelihood. In some embodiments, the action sequence table 104 can store multiple action sequences HS with a relatively high purchase likelihood and multiple action sequences LS with a relatively low purchase likelihood. In some embodiments, in addition to the action sequences HS and LS, the action sequence table 104 may also store multiple action sequences MS with a medium purchase likelihood.

[0044] The video acquisition unit 105 is also referred to as image acquisition means or video acquisition means. The video acquisition unit 105 acquires video data captured by the multiple cameras 300 in the sales floor 50 during operation. In other words, the video acquisition unit 105 acquires video data in response to the detection of a start trigger. The video acquisition unit 105 supplies frame images included in the acquired video data to the action identification unit 107. The start trigger may be, for example, when a customer enters the sales floor or when a customer approaches a product shelf.

[0045] The customer identification unit 106 is also called a customer identification means. The customer identification unit 106 identifies the same customer by, for example, known face recognition technology or image recognition technology. This makes it possible to identify a series of actions performed by the same customer, as will be described later. It is also possible to determine whether the same customer ultimately purchased a specific product.

[0046] The customer identification unit 106 also functions as a position determination unit. The customer identification unit 106 determines the customer's position within the sales floor (e.g., the customer's position near a shelf or a cash register). For example, because the camera's angle of view is fixed to the sales floor, the correspondence between the customer's position in the captured image and the customer's position on the sales floor can be defined in advance, and the position within the image can be converted to a position within the sales floor based on this definition. More specifically, in the first step, the height, azimuth, and elevation of the camera capturing the image of the sales floor, as well as the focal length of the camera (hereinafter referred to as camera parameters) are estimated from the captured image using existing technology. These may be measured or may be based on specifications. In the second step, the position of a person's feet is converted from two-dimensional coordinates on the image (hereinafter referred to as image coordinates) to three-dimensional coordinates in the real world (hereinafter referred to as world coordinates) using existing technology based on the camera parameters. Note that the conversion from image coordinates to world coordinates is usually not uniquely determined, but a unique conversion can be achieved by fixing the height coordinate value of the feet to, for example, zero. In the third step, a three-dimensional map of the transportation area is prepared in advance, and the customer's location within the sales floor can be identified by projecting the world coordinates obtained in the second step onto the map.

[0047] The motion identification unit 107 is also called a motion identification means. The motion identification unit 107 detects an image area (body area) of the customer's body from a frame image included in the video data and extracts (e.g., cuts out) it as a body image. Then, the motion identification unit 107 uses a skeletal estimation technique using machine learning to extract skeletal information of at least a part of the customer's body based on the characteristics of the customer's joints and the like recognized in the body image. The skeletal information is information composed of "key points," which are characteristic points of the joints and the like, and "bones (bone links)," which indicate the links between the key points. The motion identification unit 107 may use a skeletal estimation technique such as OpenPose.

[0048] Furthermore, the action identification unit 107 converts the skeleton information extracted from the video data acquired during operation into an action ID using the action DB 103. In this way, the action identification unit 107 identifies various actions of the customer. Specifically, first, the action identification unit 107 identifies registered skeleton information registered in the action DB 103, whose similarity to the extracted skeleton information is equal to or greater than a predetermined threshold. Then, the action identification unit 107 identifies the registered action ID associated with the identified registered skeleton information as the action ID corresponding to the customer included in the acquired frame image.

[0049] Here, the motion identification unit 107 may identify one behavior ID based on skeletal information corresponding to one frame image, or may identify one behavior ID based on time-series data of skeletal information corresponding to each of multiple frame images. When identifying one behavior ID using multiple frame images, the motion identification unit 107 may extract only skeletal information with large movements and compare the extracted skeletal information with registered skeletal information in the motion DB 103. Extracting only skeletal information with large movements may mean extracting skeletal information in which the difference between skeletal information of different frame images included within a predetermined period is equal to or greater than a predetermined amount. This reduced comparison reduces the computational load and the amount of registered skeletal information. Furthermore, since the duration of motions varies depending on the person, only skeletal information with large movements is compared, thereby making motion detection more robust.

[0050] In addition to the above-mentioned method, various other methods are possible for identifying the action ID. For example, there is a method of estimating the action ID from the target video data using an action estimation model trained on video data that has been labeled with the action ID as training data. However, collecting this training data is difficult and expensive. In contrast, in this fourth embodiment, skeletal information is used to estimate the action ID, and the action DB 103 is used to compare it with pre-registered skeletal information. Therefore, in this fourth embodiment, the purchase analysis device 100 can more easily identify the action ID.

[0051] In the above-described similarity determination, the motion identification unit 107 detects unit motions by calculating the degree of similarity between the shapes of elements constituting the skeletal data. Skeletal data has pseudo joint points or skeletal structures set as its components to indicate the posture of the body. The shape of the elements constituting the skeletal data can be, for example, the relative geometric relationship of the positions, distances, angles, etc. of other key points or bones when a certain key point or bone is used as a reference. Alternatively, the shape of the elements constituting the skeletal data can be, for example, the shape of a single integrated form formed by multiple key points or bones.

[0052] The movement identification unit 107 analyzes whether the relative shapes of the constituent elements are similar between the two pieces of skeletal data being compared. At this time, the movement identification unit 107 calculates the similarity between the two pieces of skeletal data. When calculating the similarity, the movement identification unit 107 may calculate the similarity using, for example, feature amounts calculated from the constituent elements of the skeletal data. The movement identification unit 107 can also recognize the type of a unit movement by detecting the start movement and end movement of the unit movement.

[0053] The product-related information identification unit 108 is also referred to as a product-related information identification means. The product-related information identification unit 108 identifies product-related information based on the customer's location identified by the customer identification unit 106 or the customer's predetermined behavior identified by the behavior identification unit 107. The product-related information is information associated with a specific product and may include, for example, at least one or all of the following: product (e.g., product number, product name, etc.), product classification (e.g., chocolate, sweets, beverages, etc.), product shelf, and floor map information. The product-related information identification unit 108 identifies product-related information in which the customer is interested based on the customer's behavior. For example, when a customer grabs a product with their hand, the product-related information identification unit 108 may recognize and identify the product itself using known image recognition technology or product position information within the camera's field of view. Furthermore, for example, when a customer stands in front of a product shelf where a specific type of product is placed and focuses their gaze on the specific product, the product-related information identification unit 108 may identify the specific type of product, product classification, or product shelf. For example, the product-related information identifying unit 108 may identify the product or product shelf in which the customer is interested based on the position in the image where the customer is located from the floor map information.

[0054] The associating unit 109 is also referred to as an associating means. The associating unit 109 associates the purchase likelihood corresponding to the identified customer behavior with the product-related information identified based on the identified customer behavior. This makes it possible to obtain related information indicating the purchase likelihood for the product-related information, thereby enabling purchase analysis. Specifically, products can be classified into those associated with behaviors with a relatively high purchase likelihood and those associated with behaviors with a relatively low purchase likelihood. In particular, information indicating whether a customer is interested in products or product categories that have not been purchased or are unlikely to be purchased, which was not available in previous POS systems, can be obtained, allowing for more detailed purchase analysis.

[0055] The POS linkage unit 110 is also called a POS linkage means. The POS linkage unit 110 supports the above-mentioned association in cooperation with the POS terminal device 400 and the POS management device 200 external to the purchase analysis device 100. The POS linkage unit 110 can acquire sales information of a product identified based on the identified behavior of an identified customer. The POS linkage unit 110 can recognize whether a specific product identified based on a specific behavior identified by the behavior identification unit 107 has actually been purchased. Therefore, the association unit 109, in cooperation with the POS linkage unit 110, can obtain information indicating whether a customer is interested in a product that was not purchased, or what behavior the customer is performing, thereby enabling more detailed purchase analysis.

[0056] The process control unit 111 is also called a process control means. The process control unit 111 can output the above-mentioned association information (i.e., analysis information) to the POS management device 200 or the like and display it on the display unit 203 of the POS management device 200. In some embodiments, the process control unit 111 may display the above-mentioned association information (i.e., analysis information) on a display unit (not shown) of the purchase analysis device 100.

[0057] If the associating unit 109 determines that the action sequence does not correspond to any of the action sequences HS with a high purchase possibility, it may determine whether the action sequence corresponds to any of the action sequences LS with a low purchase possibility. In this case, the process control unit 111 may output predetermined information corresponding to either an action sequence with a low purchase possibility or an action sequence with a high purchase possibility to the POS management device 200. As an example, the display mode (such as the font, color, thickness, or blinking of characters) may be changed depending on whether the action sequence is a low purchase possibility or a high purchase possibility. This allows store staff or managers to recognize the content of the purchase behavior and analyze the customer's purchase in detail. Furthermore, the process control unit 111 may record the time, location, and video of the purchase behavior as history information, along with information on the type of action sequence (whether it is a low purchase possibility or a high purchase possibility). This allows store staff or managers to recognize the content of the purchase behavior and perform a more detailed analysis.

[0058] (POS management device 200) The POS management device 200 includes a communication unit 201 , a control unit 202 , a display unit 203 , and a data management unit 204 .

[0059] The communication unit 201 is also called a communication means. The communication unit 201 is a communication interface with the network N. The communication unit 201 is also connected to the purchase analysis device 100, and transmits sales data to the purchase analysis device 100 (the POS linkage unit 110).

[0060] The control unit 202 is also called a control means. The control unit 202 controls the hardware of the POS management device 200. When the communication unit 201 receives association information or analysis information from the purchase analysis device 100, the control unit 202 causes the display unit 203 to display the association information or analysis information.

[0061] The display unit 203 is a display device. The data management unit 204 manages sales data for each product compiled by the POS system, and association or analysis information from the purchase analysis device 100 as history.

[0062] FIG. 9 is a diagram showing skeletal information of a customer extracted from a frame image 60 included in video data according to the fourth embodiment. The frame image 60 is an image captured from the side of a customer C1 holding a product P1 in his left hand, grabbing a product P2 from a product shelf 51, and comparing the product P1 with the product P2. The skeletal information shown in FIG. 9 also includes multiple key points and multiple bones detected throughout the body. As an example, FIG. 9 shows the following key points: a left ear A12, a right eye A21, a left eye A22, a nose A3, a right shoulder A51, a left shoulder A52, a right elbow A61, a left elbow A62, a right hand A71, a left hand A72, a right hip A81, a left hip A82, a right knee A91, a left knee A92, a right ankle A101, and a left ankle A102.

[0063] The purchase analysis device 100 compares such skeletal information of the customer with registered skeletal information corresponding to the entire body and determines whether they are similar, thereby identifying each action. For example, when identifying the action of comparing two products, the direction of the right and left hands and the face (gaze) are important. As another example, when identifying the action of "taking a product out of a basket" or "putting a product into a basket," the positions of the right and left hands in the frame image are important.

[0064] The camera 300 may capture at least the hand region of the customer C1 from above. FIG. 10 is a partially enlarged view showing skeletal information extracted from the right hand of a frame image according to the fourth embodiment. The partially enlarged view of the frame image shows the hand region of the customer C1 when the customer C1 is photographed from above while grabbing the product P3 from a product shelf. As an example, FIG. 10 shows the right hand A71 as a key point. The purchase analysis device 100 may then identify each action by comparing the skeletal information extracted from the series of frame images with the registered skeletal information corresponding to the hand region and determining whether they are similar.

[0065] FIG. 11 is a partially enlarged view showing skeletal information extracted from the right hand of a frame image according to the fourth embodiment. The partially enlarged view of the frame image shows the hand region of customer C1 when photographing the customer C1 from above as he returns product P3 to the product shelf and withdraws his hand. As an example, FIG. 11 shows the right hand A71 as a key point. The purchase analysis device 100 may then identify each action by comparing skeletal information extracted from multiple frame images (e.g., the frames in FIGS. 10 and 11) with registered skeletal information corresponding to the hand region and determining whether they are similar. In this way, if the action of returning product P3 to the product shelf and withdrawing his hand is identified, the purchase analysis device 100 can determine that there is a high possibility that customer C1 did not purchase product P3. Furthermore, such a behavior pattern may be pre-registered in the behavior DB 103 as a behavior pattern (LP) with a relatively low purchase likelihood.

[0066] FIG. 12 shows a frame image 70 included in the video data according to the fourth embodiment. For simplicity, the frame image 70 does not show skeletal information about customer C2. The frame image 70 shows customer C2 paying for product P4 at a cash register. For example, customer C2 displays electronic money (QR code (registered trademark)) on his / her smartphone 500, and the store staff member SP scans the displayed electronic money with the scanner 401 to complete the payment. The POS terminal device 400 transmits sales data for the purchase of product P4 to the POS management device 200 ( FIG. 8 ). When the behavior of the customer who paid for product P4 at the cash register is identified in this way, the purchase analysis device 100 can determine that customer C2 purchased product P4 (or that there is a high possibility that customer C2 purchased product P4). Furthermore, such behavior patterns can be pre-registered in the behavior DB 103 as behavior patterns (HP) with a relatively high purchase probability.

[0067] As described above, the purchase analysis device 100 may determine the possibility of purchasing a specific product from the customer's actions contained in multiple frame images, but it may also work in cooperation with the POS management device 200 to determine whether a specific product has definitely been purchased from sales data.

[0068] 13 is a flowchart showing the flow of a method for registering a registration action ID and a registration action sequence by the purchase analysis device 100 according to the fourth embodiment. First, the registration information acquisition unit 101 of the purchase analysis device 100 receives an action registration request including registration video data and a registration action ID from the user interface of the purchase analysis device 100 (operation by the administrator) (S30). Next, the registration unit 102 supplies the registration video data to the action identification unit 107. Having acquired the registration video data, the action identification unit 107 extracts a body image from a frame image included in the registration video data (S31). Next, the action identification unit 107 extracts skeletal information from the body image (S32). Next, the registration unit 102 acquires skeletal information from the action identification unit 107 and registers the acquired skeletal information as registration skeletal information in the action DB 103 in association with the registration action ID (S33). The registration unit 102 may register all of the skeletal information extracted from the body image as the registered skeletal information, or may register only part of the skeletal information (for example, skeletal information of the shoulders, elbows, and hands). Next, the registration information acquisition unit 101 receives a sequence registration request including multiple registered action IDs and information on the chronological order of each action from the user interface of the purchase analysis device 100 (operation by the administrator) (S34). Next, the registration unit 102 registers a registered action sequence (for example, an action sequence HS with a high purchase possibility or an action sequence LS with a low purchase possibility) in which the registered action IDs are arranged based on the chronological order information in the action sequence table 104 (S35). Then, the purchase analysis device 100 ends the processing.

[0069] FIG. 14 is a diagram for explaining registered actions according to the fourth embodiment. As an example, the action DB 103 may store registered framework information for eight registered actions having registered action IDs "A" to "H." These registered actions are also called unit actions. Registered action "A" is an action of taking a product from a product shelf (see, for example, FIG. 9). Registered action "B" is an action of putting a product into a basket. Registered action "C" is an action of moving the product taken from the product shelf or the basket containing the product to another location (e.g., another product shelf). Registered action "D" is an action of moving the product itself or the basket to a cash register (see, for example, FIG. 12). Registered action "E" is an action of returning a product to the product shelf (see, for example, FIG. 11). Registered action "F" is an action of stopping in front of a product shelf and looking at the product (see, for example, FIG. 9). Registered action "G" is an action of comparing multiple products (see, for example, FIG. 9). The registered action "H" is an action of moving from the product shelf to another location without carrying anything.

[0070] FIG. 15 is a diagram illustrating an action sequence HS with a high purchase possibility according to the fourth embodiment. An action sequence includes one or more unit actions. As an example, the action sequence table 104 may include at least four action sequences HS with a high purchase possibility having purchase action sequence IDs of "11" to "14." The action sequence with a high purchase possibility "11" is a sequence (A→B→C→D) in which a customer purchases a product in a single purchase action. The action sequence with a high purchase possibility "12" is a sequence (A→E→C→A→B→C→D) in which a customer finally purchases a product in a single non-purchase action and a single purchase action. In this case, the identified non-purchase action is associated with the first product, and the identified purchase action is associated with the last product. The action sequence with a high purchase possibility "13" is a sequence (A→B) in which a customer is likely to purchase a product in a single purchase action. The action sequence with a high purchase possibility "14" is a sequence (A→B→C) in which a customer is likely to purchase a product in a single purchase action. Thus, in some embodiments, the action of placing an item in a basket (registration action "B") and the action of taking the item from the shelf or taking the basket containing the item to another location (e.g., another shelf) (registration action "C") may be determined to be an action sequence with a relatively high likelihood of purchase.

[0071] FIG. 16 is a diagram illustrating an action sequence LS with a low purchase possibility according to the fourth embodiment. The action sequence table 104 may include at least two action sequences LS with a low purchase possibility having action sequence IDs of "21" and "22." The action sequence "21" with a low purchase possibility is a sequence (? → A → E → ?) in which a customer picks up a product from a product shelf but returns the product to the product shelf. "?" indicates any action. In this case, the product can also be associated with this action sequence. Furthermore, the action sequence "22" with a low purchase possibility is a action sequence (? → F → H → ?) in which a customer stops in front of a product shelf but leaves without doing anything. In this case, the customer's action sequence can also be associated with the product shelf where the customer stopped.

[0072] FIG. 17 is a flowchart showing the flow of a purchase analysis method by the purchase analysis device 100 according to the fourth embodiment. First, the video acquisition unit 105 of the purchase analysis device 100 acquires video data from the camera 300 (S401). The customer identification unit 106 then identifies the customer using, for example, known face recognition or image recognition technology (S402). For example, the customer identification unit 106 recognizes customer attribute information such as the customer's height, clothing, face, hairstyle, body type, gender, and age group (e.g., teens or younger, 20s to 50s, 60s or older, etc.), and stores this information in a storage unit. This allows the identification unit 106 to subsequently identify a series of actions being performed by the same identified person. For example, in the example of FIG. 9, customer C1 can be identified as a man in his 20s with a medium build.

[0073] Thereafter, or in parallel with this, the action identification unit 107 extracts a body image from a frame image included in the video data (S403). Next, the action identification unit 107 extracts skeletal information from the body image (S404). The action identification unit 107 calculates the similarity between at least a part of the extracted skeletal information and each registered skeletal information registered in the action DB 103, and identifies, as the action ID, a registered action ID associated with registered skeletal information whose similarity is equal to or greater than a predetermined threshold (S405). Next, in some embodiments, the action identification unit 10 7 In the first cycle, the action identification unit 107 sets the action ID identified in S405 as the action sequence, and in the next and subsequent cycles, it adds the action ID identified in S405 to the action sequence that has already been generated.

[0074] The product-related information identification unit 108 identifies product-related information in which the customer is interested, based on the customer's position identified by the customer identification unit (position identification unit) 106 and the customer's behavior identified by the behavior identification unit 107 (S406). For example, when the customer picks up a product, the product-related information identification unit 108 may recognize and identify the product itself using a known image recognition technology. Furthermore, for example, when the customer stands in front of a product shelf on which a specific type of product is placed and focuses his or her gaze on the product shelf, the product-related information identification unit 108 may identify the specific type of product, product category, or product shelf as product-related information. For example, the product-related information identification unit 108 may identify the product or product shelf in which the customer is interested, based on the position in the image where the customer identified by the customer identification unit (position identification unit) 106 is located, from floor map information.

[0075] The associating unit 109 associates the purchase likelihood corresponding to the identified predetermined behavior with the product-related information identified based on the identified predetermined behavior of the customer (S407). The associating unit 109 may also associate this information with the customer attribute information described above. For example, in the example of FIG. 9, it is determined from subsequent frame images that customer C1 subsequently returned product P2 to the shelf and then moved away with product P1. In this case, product P2 is associated with a behavior associated with a low purchase likelihood of a male in his twenties, and product P1 is associated with a behavior associated with a high purchase likelihood of a male in his twenties. This association information may be transmitted to the POS management device 200. This allows for obtaining related information indicating the purchase likelihood of the product-related information, thereby enabling detailed purchase analysis. Specifically, products can be classified into those associated with behaviors associated with a relatively high purchase likelihood and those associated with a relatively low purchase likelihood. In particular, information indicating whether a customer is interested in products or product categories that were not purchased or are likely not to be purchased, which was not available in previous POS systems, can be obtained, enabling more detailed purchase analysis.

[0076] The POS linkage unit 110 supports the above-described association in cooperation with the POS terminal device 400 and the POS management device 200 (S408). The POS linkage unit 110 can acquire sales information for the identified product based on the identified behavior for the identified customer. The POS linkage unit 110 can recognize whether a specific product identified based on the customer's behavior identified by the behavior identification unit 107 was actually purchased. Therefore, the association unit 109, in cooperation with the POS linkage unit 110, can obtain information indicating whether the customer is interested in a product or what behavior the customer is performing for a product that the customer did not purchase, thereby enabling more detailed purchase analysis. For a product that the customer ultimately purchased, information indicating what purchasing behavior the customer performed can be obtained. For example, in the example of FIG. 9, it is determined from subsequent frame images that customer C1 subsequently returned product P2 to the shelf and then moved away with product P1. In this case, product P2 is associated with a behavior indicating a low likelihood of purchase, and product P1 is associated with a behavior indicating a high likelihood of purchase. However, if sales of either product P1 or P2 are not recorded in the sales information, the POS linkage unit 110 may associate not only the action of returning product P2 to the shelf, but also the action of moving with product P1 as actions that ultimately did not result in a purchase, with each product.

[0077] The process control unit 111 outputs the above-mentioned association information (that is, analysis information) to the POS management device 200 or the like (S409), and can display it on the display unit 203 of the POS management device 200.

[0078] Thus, according to the fourth embodiment, the purchase analysis device 100 determines the purchase possibility of customer C's actions regarding a specific product, etc., by comparing the action sequence showing the flow of actions of customer C who visited the sales floor 50 with the action pattern HP or action sequence HS having a high purchase possibility and the action pattern LP or action sequence LS having a low purchase possibility. Furthermore, by linking this related information with POS sales information, it is possible to more accurately analyze customer C's actions regarding a specific product, etc.

[0079] Although the flowchart of Figure 17 shows a specific order of execution, the order of execution may differ from that depicted. For example, the order of execution of two or more steps may be swapped relative to the order shown. Also, two or more steps shown as successive in Figure 17 may be performed concurrently or with partial concurrence. Furthermore, in some embodiments, one or more steps shown in Figure 17 may be skipped or omitted.

[0080] <Embodiment 5> FIG. 18 is an exemplary block diagram showing the configuration of an imaging device. The imaging device 300b is also called an intelligent camera and may include a registration information acquisition unit 101, a registration unit 102, an action database 103, an action sequence table 104, a camera 105b, a customer identification unit 106, an action identification unit 107, a product-related information identification unit 108, an association unit 109, a POS linkage unit 110, and a process control unit 111. The configuration of the imaging device 300b is basically the same as that of the purchase analysis device 100 described above, and therefore a detailed description will be omitted. However, the imaging device 300b differs from the purchasing analysis device 100 in that it includes a built-in camera 105b. The camera 105b includes an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor. Furthermore, captured video data created by the camera 105b is stored in the action database 103b. The configuration of the imaging device 300b is not limited to this, and various modifications are possible.

[0081] In some embodiments, the imaging device 300b (intelligent camera) according to the fifth embodiment and the purchase analysis device 100 according to the fourth embodiment may share some of their functions to achieve the object of the present disclosure.

[0082] FIG. 19 is a block diagram showing the hardware configuration of the purchase analysis device. FIG. 19 is a block diagram showing an example of the hardware configuration of a purchasing analysis device 100, 100a to 100c (hereinafter referred to as the purchasing analysis device 100, etc.). Referring to FIG. 19, the purchasing analysis device 100, etc. includes a network interface 1201, a processor 1202, and a memory 1203. The network interface 1201 is used to communicate with other network node devices constituting a communication system. The network interface 1201 may be used for wireless communication. For example, the network interface 1201 may be used for wireless LAN communication defined in the IEEE 802.11 series or mobile communication defined in the 3GPP (3rd Generation Partnership Project). Alternatively, the network interface 1201 may include, for example, a network interface card (NIC) conforming to the IEEE 802.3 series.

[0083] The processor 1202 reads and executes software (computer programs) from the memory 1203 to perform the processes of the purchase analysis device 100 and the like described using flowcharts or sequences in the above-described embodiments. The processor 1202 may be, for example, a microprocessor, an MPU (Micro Processing Unit), or a CPU (Central Processing Unit). The processor 1202 may include multiple processors.

[0084] The memory 1203 is configured by a combination of volatile memory and non-volatile memory. The memory 1203 may include storage located remotely from the processor 1202. In this case, the processor 1202 may access the memory 1203 via an I / O interface (not shown).

[0085] 19, the memory 1203 is used to store a group of software modules. The processor 1202 reads and executes these software modules from the memory 1203, thereby performing the processing of the purchase analysis device 100 and the like described in the above-described embodiment.

[0086] As explained using the flowchart above, each of the processors of the purchasing analysis device 100, etc. executes one or more programs including a group of instructions for causing a computer to perform the algorithm explained using the drawings.

[0087] Although the above-described embodiments have been described as hardware configurations, the present disclosure is not limited to such configurations. Any processing in the present disclosure can also be realized by causing a processor to execute a computer program.

[0088] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0089] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. (Appendix 1) a movement identification means for analyzing the movements of customers in the sales area included in the captured video data and identifying the movements of the customers in accordance with the stored movement patterns; a product-related information specifying means for specifying product-related information in which the customer is interested based on the specified customer's behavior or location; an association means for associating the identified action with the product-related information; A purchasing analysis device comprising: (Appendix 2) A purchasing analysis device as described in Appendix 1, wherein the identified actions include various customer actions performed near product shelves in a sales area. (Appendix 3) A purchasing analysis device as described in Appendix 1 or 2, wherein the stored behavior patterns are behaviors associated with the product-related information and include at least behavior patterns with a relatively high likelihood of purchase and behavior patterns with a relatively low likelihood of purchase. (Appendix 4) The method further comprises a storage unit that stores at least a relatively low purchase probability behavior pattern based on a plurality of consecutive image frames, the behavior identification means identifies a behavior of the customer with a relatively low purchasing possibility based on the stored behavior patterns with a relatively low purchasing possibility; 4. The purchase analysis device according to any one of appendices 1 to 3, wherein the associating means associates the identified behavior with a relatively low purchase possibility with the identified product-related information. (Appendix 5) The system further includes a POS linkage means for acquiring sales information of the identified product or product-related information from the POS management device based on the identified operation, the associating means associates the identified action with the product or the product-related information when sales information about the identified product or product-related information is available; A purchasing analysis device described in any one of Appendices 1 to 4, wherein the associating means associates the identified action with the product or the product-related information if there is no sales information for the identified product or product-related information. (Appendix 6) The purchase analysis device according to claim 3, wherein the behavior pattern with a relatively low likelihood of purchase is a behavior in which a customer returns an item to a shelf. (Appendix 7) 7. The purchase analysis device according to any one of appendices 1 to 6, wherein the identified actions include an action of the customer grabbing a product. (Appendix 8) 8. The purchase analysis device according to any one of appendices 1 to 7, wherein the identified behavior includes a behavior of a customer comparing a plurality of products. (Appendix 9) The purchase analysis device according to any one of appendices 1 to 8, wherein the identified behavior includes a behavior in which the customer stops in front of a product or a product shelf for a predetermined period of time or more. (Appendix 10) The purchase analysis device according to any one of appendices 1 to 9, further comprising a customer identification means for identifying a customer, wherein the association means groups and associates a series of actions performed by the identified customer. (Appendix 11) The customer identification means identifies the location of the customer; The purchase analysis device according to claim 10, wherein the product-related information specifying means specifies product-related information based on the specified location of the customer. (Appendix 12) 11. The purchase analysis device according to any one of appendices 1 to 10, wherein the product-related information includes at least one of product, product classification, product shelf, and floor map information. (Appendix 13) 12. The purchase analysis device according to any one of appendices 1 to 11, wherein the action identification means sets the physical feature points and pseudo skeleton of the customer based on video data. (Appendix 14) Analyzing the customer's movements in the sales area contained in the captured video data, and identifying the customer's movements according to the stored movement patterns; Identifying product-related information in which the customer is interested based on the identified customer behavior or location; The identified behavior is associated with the product-related information. (Appendix 15) Analyzing the customer's movements in the sales area contained in the captured video data, and identifying the customer's movements according to the stored movement patterns; Identifying product-related information in which the customer is interested based on the identified customer behavior or location; A non-transitory computer-readable medium storing a program that causes a computer to execute a purchasing analysis method that associates the identified actions with the product-related information. [Explanation of symbols]

[0090] 1. Purchasing analysis system 50 sales floor 51 Product shelf 60 frame images 70 frame images 100, 100a, 100b, 100c Purchase analysis device (server) 101 Registration Information Acquisition Department 102 Registration Department 103 Operation DB 103b, 103c storage section 104 Operation Sequence Table 105 Video acquisition unit 106 Customer Identification Unit (Location Identification Unit) 107,107a,107b,107c Operation specific part 108, 108a, 108b, 108c Product related information identification section 109, 109a, 109b, 109c Association section 110,110c POS cooperation department 111 Processing control unit 200,200a POS management device 201 Communications Department 202 Control section 203 Display section 204 Data Management Department 300 cameras 300b Imaging device 400 POS terminals 401 Scanner 500 smartphones C Customer N Network

Claims

1. a motion identification means for analyzing the motions of customers in the sales area included in the captured video data, and identifying, according to the motion patterns associated with the stored product-related information, motions with a relatively low purchase possibility including at least a motion pattern of the customer grabbing a product and a motion pattern with a relatively low purchase possibility including at least a motion pattern of the customer returning the product to the shelf, the motions of the customer including a motion of the customer comparing multiple products with the right and left hands and the direction of the customer's face during the comparison motion; a product-related information specifying means for specifying product-related information in which the customer is interested based on the specified customer's behavior or location; POS linkage means for acquiring sales information of the identified product or product-related information from the POS management device based on the identified operation; an association means for associating the identified action with the product-related information, which associates the identified action with the product or the product-related information when sales information about the identified product or the product-related information is available; an association means for associating the specified action with the product or the product-related information when there is no sales information for the specified product or product-related information; A purchasing analysis device comprising:

2. The purchase analysis device according to claim 1 , wherein the identified actions include various actions of customers performed near product shelves in a sales area.

3. The method further comprises a storage unit that stores at least a relatively low purchase probability behavior pattern based on a plurality of consecutive image frames, the behavior identification means identifies a behavior of the customer with a relatively low purchasing possibility based on the stored behavior patterns with a relatively low purchasing possibility; 3. The purchase analysis device according to claim 1, wherein the associating means associates the identified behavior with a relatively low purchase possibility with the identified product-related information.

4. A purchase analysis method executed by a computer, comprising: The system analyzes the customer's movements in the sales floor contained in the captured video data, and identifies the customer's movements that are relatively less likely to purchase, the customer's movements that are relatively more ... comparing multiple products with their right and left hands, and the customer's facial orientation during the comparison movements, according to the movement patterns associated with the stored product-related information, the movement patterns including at least a movement pattern in which the customer grabs a product and a movement pattern in which the customer returns the product to the shelf and a movement pattern in which the customer has a relatively low likelihood of purchasing, and the customer's movements that are relatively more likely to purchase, the customer's movements comparing multiple products with their right and left hands, and the facial orientation of the customer during the comparison movements; Acquire sales information of the identified product or product-related information based on the identified operation from the POS management device; Identifying product-related information in which the customer is interested based on the identified customer behavior or location; Associating the identified action with the product-related information, If there is sales information for the identified product or product-related information, the identified action is associated with the product or the product-related information; A purchase analysis method, wherein if there is no sales information for the identified product or product-related information, the identified behavior is associated with the product or the product-related information.

5. The system analyzes the customer's movements in the sales floor contained in the captured video data, and identifies the customer's movements that are relatively less likely to purchase, the customer's movements that are relatively more ... comparing multiple products with their right and left hands, and the customer's facial orientation during the comparison movements, according to the movement patterns associated with the stored product-related information, the movement patterns including at least a movement pattern in which the customer grabs a product and a movement pattern in which the customer returns the product to the shelf and a movement pattern in which the customer has a relatively low likelihood of purchasing, and the customer's movements that are relatively more likely to purchase, the customer's movements comparing multiple products with their right and left hands, and the facial orientation of the customer during the comparison movements; Acquire sales information of the identified product or product-related information based on the identified operation from the POS management device; Identifying product-related information in which the customer is interested based on the identified customer behavior or location; Associating the identified action with the product-related information, If there is sales information for the identified product or product-related information, the identified action is associated with the product or the product-related information; A program that causes a computer to execute a purchase analysis method that associates the identified behavior with the product or the product-related information if there is no sales information for the identified product or product-related information.

Images