Information processing apparatus, program, and behavior analysis system

Abstract

To provide an information processing terminal device, a program, and a behavior analysis system for easily improving determination accuracy for estimating customer behavior in a store.SOLUTION: In a behavior analysis system 1, the behavior analysis device 30 as an information processing device extracts characteristic information (for example, a face image) of a specific customer from the imaging data acquired from a camera 10, and the imaging data containing feature information are arranged in chronological order, and the imaging data showing one action from the customer reaching out to the product display shelf until returning it is extracted from the imaging data containing the feature information. The information processing device 50 compares the purchased product in one transaction estimated by the behavior analysis device 30 with the purchased product included in the transaction information actually traded in the one transaction, and if the two are different, the behavior analysis is performed to correct the behavioral information estimated by the behavior analysis device 30.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing apparatus, a program, and a behavior analysis system.

Background Art

[0002] Conventionally, detection of customer behavior in a retail store or the like has been performed (for example, Patent Document 1). The store can execute various measures for expanding sales based on the detected customer behavior. For example, the store can change the arrangement of products based on the detected movement trajectory (flow line) of the customer, or distribute recommendations or coupons for other products related to the product when it is detected that the customer has taken out the product from the product display shelf.

[0003] In recent years, efforts have been made to estimate human behavior using machine learning. In machine learning, improving the determination accuracy of the machine learning model becomes important. In order to improve the determination accuracy of the machine learning model, training data associating sensor values in the actual environment with correct labels is required, but in reality, it takes time and cost to collect the training data.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to provide an information processing apparatus, a program, and a behavior analysis system that can easily improve the determination accuracy of a machine learning model for estimating customer behavior in a store.

Means for Solving the Problems

[0005] The information processing device of the embodiment includes: a purchase estimation information acquisition unit that acquires purchase estimation information which associates behavior information indicating customer behavior estimated based on sensor information from sensors installed in the store, product identification information which identifies products that are estimated to have been purchased by the customer in one transaction based on the behavior information, and transaction identification information which can identify the one transaction; a transaction information acquisition unit that acquires transaction information relating to the store's transactions which associates the product identification information of products traded in one transaction with the transaction identification information; a comparison unit that compares the product identification information included in the purchase estimation information with the product identification information included in the transaction information with respect to one transaction identified by the transaction identification information; a modification unit that modifies the behavior information based on the transaction information if, as a result of the comparison by the comparison unit, the product identification information included in the purchase estimation information and the product identification information included in the transaction information do not match; and an output unit that outputs the behavior information modified by the modification unit. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 is a schematic diagram of a behavioral analysis system including an information processing device according to an embodiment. [Figure 2] Figure 2 is a block diagram showing the hardware configuration of the information processing device according to the embodiment. [Figure 3] Figure 3 shows the data structure of the purchase estimation information file stored in the memory section of the information processing device of the embodiment. [Figure 4] Figure 4 is a diagram showing the data structure of the data file to be estimated, which is stored in the memory section of the information processing device of the embodiment. [Figure 5] Figure 5 is a diagram showing the data structure of the transaction information file stored in the memory section of the information processing device of the embodiment. [Figure 6] Figure 6 is a diagram showing the data structure of the correction data file stored in the memory section of the information processing device of the embodiment. [Figure 7] Figure 7 is a block diagram showing the functional configuration of the control unit in the information processing apparatus of the embodiment. [Figure 8]Figure 8 is a flowchart showing the processing flow by the control unit of the information processing device in this embodiment. [Figure 9] Figure 9 shows the action information modification screen in the information processing device of the embodiment. [Modes for carrying out the invention]

[0007] The following describes an information processing device, program, and behavioral analysis system of an embodiment with reference to the drawings. However, the invention is not limited to the embodiments described below. For example, the embodiment described below describes an example of applying the behavioral analysis system to a supermarket, but it can be broadly applied to various types of stores that sell goods.

[0008] Figure 1 is a schematic diagram of a behavioral analysis system including an information processing device. The behavioral analysis system 1 of this embodiment is applied to supermarkets (hereinafter also referred to as stores) that sell food products, daily necessities, etc. The behavioral analysis system 1 comprises multiple cameras 10, multiple POS (Point of Sales) terminals 20, a behavioral analysis device 30, a store server 40, and an information processing device 50. These multiple cameras 10, multiple POS terminals 20, behavioral analysis device 30, store server 40, and information processing device 50 are connected to each other so as to be able to communicate with one another via a network such as a LAN (Local Area Network).

[0009] Multiple cameras 10 are placed in the sales area of ​​the store. Each camera 10 is positioned corresponding to each product display shelf in the sales area and is capable of capturing images of customer movements, etc., at the corresponding product display shelf. Each camera 10 outputs the captured image data to the behavior analysis device 30 in real time. The image data is associated with image data captured by camera 10, information indicating the time the image data was captured, and information identifying the camera that captured the image. Camera 10 is an example of a sensor installed in the store. The image data output by camera 10 is also an example of sensor information.

[0010] Multiple POS terminals 20 are installed at the checkout counter in the store's sales area. The POS terminals 20 register product information for items purchased by customers (hereinafter also referred to as product registration) and perform accounting processing for the registered products. Accounting processing is the process of paying for the products purchased by the customer, and includes processes such as calculating the total amount, calculating change, sending and receiving information with the payment server in cashless payment, and issuing receipts. The POS terminals 20 may be POS terminals operated by store employees or self-service POS terminals operated by customers. The POS terminals 20 are an example of accounting equipment that performs accounting processing for transactions in a store. The accounting equipment may also be accounting equipment in a so-called semi-self-service POS system, where product registration and accounting processing are performed by separate devices.

[0011] The POS terminal 20 outputs transaction information to the store server 40. Transaction information is information that associates product identification information (e.g., product code and product name) that identifies the products traded in one transaction with transaction identification information (e.g., payment (settlement) time and POS terminal No.). The store server 40 stores the acquired transaction information in a storage unit (not shown). The POS terminal 20 may output transaction information to the store server 40 at any time, or it may output it periodically (e.g., at a predetermined time every day).

[0012] The behavior analysis device 30 estimates customer behavior based on image data acquired from the camera 10. For example, the behavior analysis device 30 estimates customer behavior such as the action of taking an item from the product display shelf (hereinafter also referred to as the acquisition action), the action of returning the item taken from the product display shelf to the same product display shelf (hereinafter also referred to as the return action), and the action of paying at the POS terminal 20 (hereinafter also referred to as the payment action). The estimation of customer behavior is performed using a machine learning model. In addition, the behavior analysis device 30 estimates the items purchased by the customer in a single transaction based on the estimated customer behavior.

[0013] Here, we will explain the functions of the behavioral analysis device 30 in detail. The processes performed by the behavioral analysis device 30 include generating data to be estimated, estimating customer behavior, generating purchase estimation information, outputting information to systems that utilize the estimation results, and outputting information to the store server 40. These processes will be explained below.

[0014] (Generation of data to be estimated) The data to be estimated is data to be input into the machine learning model, and is image data generated by image processing based on the image data acquired from the camera 10. The behavioral analysis device 30 stores layout data of the store's sales floor. The layout data includes information such as the arrangement of product display shelves and the correspondence between each product display shelf and the products displayed on that shelf. As a preprocessing step for inputting data into the machine learning model, the behavioral analysis device 30 generates the data to be estimated based on the image data acquired from the camera 10 and the layout data.

[0015] The behavioral analysis device 30 extracts characteristic information of a specific customer (e.g., facial image) from the image data acquired from the camera 10 and arranges the image data containing this characteristic information in chronological order. The behavioral analysis device 30 also extracts image data from the image data containing the characteristic information that shows a single action from when the customer reaches for a product display shelf until they return it. Each image data is associated with information that identifies the camera 10 that captured the image (camera No.). Each camera 10 corresponds to each product display shelf, and since each product display shelf is associated with the products displayed on that shelf by layout data, the behavioral analysis device 30 can recognize which product the customer indicated in the image data reached for. The behavioral analysis device 30 also extracts image data from when the identified customer approaches the POS terminal 20 until they leave. It is also possible to perform the above extraction of image data using a machine learning model. Estimation target data is generated by extracting image data in this way.

[0016] The image data extracted by the behavioral analysis device 30, showing a single action of a specific customer from reaching for a product on a shelf to returning it, and the image data from standing at the POS terminal 20 to leaving, are examples of data to be estimated. By arranging this data to be estimated in a time series, data is generated to estimate the behavior of a specific customer in a single transaction.

[0017] (Estimating customer behavior) Customer behavior is estimated using a machine learning model. The machine learning model is generated based on training data consisting of image data and correct labels for the behaviors contained in the image data. The machine learning model is also trained using behavior information obtained from the information processing device 50 as training data, and its parameters are adjusted. This improves the accuracy of the machine learning model's judgments. The behavior information obtained from the information processing device 50 will be described later.

[0018] The machine learning model is input with data to be estimated. For example, if the machine learning model is input with image data showing a single action of a specific customer from reaching for a product on a shelf to returning it, it will determine whether the customer action included in the image data is an acquisition action or a return action. The machine learning model extracts the customer action, and if the probability value of determining that the extracted action was performed is greater than or equal to a predetermined value (e.g., 70 percent), it will estimate that the action was performed. The machine learning model is input with data to be estimated regarding the customer's actions during a single transaction, and the machine learning model extracts the actions included in each piece of data to be estimated, and determines whether or not the extracted action was performed. For example, the machine learning model determines whether or not actions such as acquisition actions, return actions, and settlement actions were performed during a single transaction. The machine learning model outputs information indicating the actions of the extracted transaction (hereinafter also referred to as action information). The action information includes, for example, the name of the action such as acquisition, return, or settlement, the result of the determination of whether or not the action was performed, and the time at which the action was estimated to have been performed. The data to be estimated input to the machine learning model and the action information output from the machine learning model are assigned the same determination ID and are associated with each other.

[0019] (Generation of Purchase Estimation Information) The behavior analysis device 30 generates purchase estimation information based on each piece of behavior information output from the machine learning model at any time. The purchase estimation information includes behavior information indicating the behavior of one transaction output by the machine learning model, product identification information for identifying the product estimated to be purchased by the customer in one transaction based on the behavior information, and information in which transaction identification information capable of specifying the one transaction is associated. In the present embodiment, the purchase estimation information further includes information indicating the number of each product estimated to be purchased in one transaction. The product identification information is, for example, a product name or a product code. The transaction identification information is composed of the time estimated to be settled by the POS terminal 20 and the POS terminal number in the present embodiment. The time estimated to be settled by the POS terminal 20 may be set, for example, as the time when the behavior of receiving a receipt during the settlement operation is estimated. Also, the time when the customer is located at the POS terminal 20 and the time when the customer leaves the POS terminal 20 after settlement may be set as a set and treated as the settlement time.

[0020] (Output of Information to a System Utilizing Estimation Results) The behavior analysis device 30 outputs the behavior information output by the machine learning model or the purchase estimation information estimated based on the behavior information to an external system. The external system develops various services based on the information acquired from the behavior analysis device 30. As an example of the external system, a coupon issuance system or the like can be mentioned. When the coupon issuance system acquires behavior information indicating that a customer has taken out a certain product from a product display shelf from the behavior analysis device 30, it can perform processes such as distributing a coupon for a product related to the product to the customer's mobile terminal. Note that the external system is not limited to the coupon issuance system.

[0021] (Output of Information to the Store Server 40) The behavior analysis device 30 outputs the generated estimated target data and purchase estimated information to the store server 40 as needed. The estimated target data and purchase estimated target information also include a judgment ID. The store server 40 stores the acquired estimated target data and purchase estimated information in a storage unit (not shown). In this embodiment, the estimated target data and purchase estimated target information output by the behavior analysis device 30 to the store server 40 includes not only actions that the machine learning model determined to have been performed by the customer, but also actions that were extracted but were not determined to have been performed by the customer. However, the estimated target data and purchase estimated target information output by the behavior analysis device 30 to the store server 40 may consist only of actions that the machine learning model determined to have been performed by the customer.

[0022] Next, the store server 40 will be described. The store server 40 acquires transaction information from each POS terminal 20 and stores this transaction information. The store server 40 also acquires purchase estimation information and estimated target data from the behavioral analysis device 30 and stores this purchase estimation information and estimated target data. The store server 40 then outputs transaction information, estimated target data, and purchase estimation information for a predetermined period (e.g., one month) to the information processing device 50, for example, in response to an output request from the information processing device 50. The store server 40 may also output transaction information, estimated target data, and purchase estimation information for a predetermined period to the information processing device 50 at a pre-set timing, rather than in response to an output request from the information processing device 50.

[0023] The store server 40 also stores a product master that associates product codes with product information (product name, price, etc.) for the products handled by the store. Since the products handled by the store change daily, the product master is updated as needed. The store server 40 transmits the product master to each POS terminal 20.

[0024] The information processing device 50 provides the behavior analysis device 30 with training data for training the machine learning model of the behavior analysis device 30. The information processing device 50 compares the purchased goods in a transaction estimated by the behavior analysis device 30 with the purchased goods included in the transaction information of the actual transaction in that transaction. If the two are different, the information processing device 50 corrects the behavior information estimated by the behavior analysis device 30. For example, the information processing device 50 corrects the behavior information using the actual transaction information as the ground truth data. The information processing device 50 then outputs the corrected behavior information to the behavior analysis device 30. The behavior analysis device 30 can improve the accuracy of its machine learning model's judgment by having the machine learning model train using the behavior information obtained from the information processing device 50 as training data. As the information processing device 50 functions as described above, it can also be called a training data generation device.

[0025] Next, the hardware configuration of the information processing device 50 will be described. Figure 2 is a block diagram showing the hardware configuration of the information processing device 50. The information processing device 50 comprises a control unit 500, a memory unit 510, a display device 520, an input device 530, and a communication unit 540. The control unit 500, the memory unit 510, the display device 520, the input device 530, and the communication unit 540 are connected to each other via a bus 550, etc.

[0026] The control unit 500 consists of a computer equipped with a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 550.

[0027] The CPU 501 controls the overall operation of the information processing device 50. The ROM 502 stores various programs and data, such as the program used to drive the CPU 501. The RAM 503 is used as the work area for the CPU 501 and loads the various programs and data stored in the ROM 502 and the memory unit 510. The control unit 500 executes various control processes of the information processing device 50 by having the CPU 501 operate according to the control programs stored in the ROM 502 and the memory unit 510 and loaded into the RAM 503.

[0028] The memory unit 510 is composed of a storage medium such as an HDD (Hard Disk Drive) or flash memory, and retains its contents even when the power is cut off. The memory unit 510 stores the control program 511, the purchase estimation information file 512, the estimation target data file 513, the transaction information file 514, and the correction data file 515. The purchase estimation information file 512, the estimation target data file 513, the transaction information file 514, and the correction data file 515 may also be stored in external memory.

[0029] The control program 211 is a program that causes the information processing device 50 to function as a device for generating training data to be provided to the machine learning model of the behavior analysis device 30.

[0030] The purchase estimation information file 512 is a file that manages purchase estimation information acquired from the store server 40. Purchase estimation information is data for each transaction and is transmitted from the behavior analysis device 30 to the store server 40 as needed. The information processing device 50 acquires, for example, one month's worth of purchase estimation information from the store server 40 once a month. Each piece of information included in the purchase estimation information is information estimated by the behavior analysis device 30. Figure 3 shows the data structure of the purchase estimation information file 512. Each piece of data registered in the purchase estimation information file 512 is associated with information indicating the settlement time, POS terminal No., product name, behavior, time, judgment, probability value, and judgment ID.

[0031] The "Settlement Time" field registers information indicating the estimated time when settlement occurred in a single transaction. In other words, the information registered in "Settlement Time" is information indicating the settlement time estimated by the behavioral analysis device 30 based on the image data from the camera 10. The "POS Terminal No." field registers a unique number that identifies the POS terminal 20 to which settlement is estimated. By identifying the settlement time and the number of the POS terminal 20 to which settlement occurred, it is possible to identify a single transaction. In other words, the combination of settlement time and POS terminal No. is an example of transaction identification information. Note that if there is only one accounting machine such as a POS terminal 20 installed in the store, a single transaction can be identified by the settlement time alone, so only the settlement time becomes transaction identification information.

[0032] The "Product Name" field registers information indicating the name of the product that the customer is presumed to have purchased. The information registered in the "Product Name" field is an example of product identification information. The purchase estimation information file 512 may also register the product code as product identification information. The "Behavior" field registers information indicating the customer's behavior extracted by the behavior analysis device 30. The information registered in the "Behavior Name" field is an example of behavior information. The "Time" field registers information indicating the start and end times of the imaging data used to estimate the behavior. The "Time" field may, for example, register only the start time or only the end time of the imaging data used to estimate the behavior.

[0033] The "Judgment" field registers information indicating whether the customer's actions extracted by the behavioral analysis device 30 are correct or incorrect. A "○" in the "Judgment" field indicates that the corresponding action was determined to have occurred. The "Probability Value" field registers information indicating the probability that the machine learning model of the behavioral analysis device 30 determined the extracted action to be correct. In the example shown in Figure 3, in a transaction settled at POS terminal 20 with POS terminal No. 1 at 12:15 on March 3, 2021, the probability that product A was picked up during the action between 10:38 and 10:39 is 90 percent, and the behavioral analysis device determined that product A was picked up and purchased. The "Judgment ID" field registers information that associates the input (estimated target data) and output (behavioral information) in the machine learning model of the behavioral analysis device 30.

[0034] The estimation target data file 513 is a file that manages the estimation target data acquired from the store server 40. The estimation target data consists of image data for each individual action generated by the behavior analysis device 30, which is transmitted from the behavior analysis device 30 to the store server 40 as needed. The information processing device 50 acquires the estimation target data from the store server 40 at the time it acquires purchase estimation information. Figure 4 shows the data structure of the estimation target data file 513. Each data entry registered in the estimation target data file 513 is associated with information indicating time, camera number, image data, product name, and judgment ID.

[0035] The "Time" field registers information indicating the start and end times of the imaging data used to estimate the behavior. The "Camera No." field registers a unique number identifying the camera that captured the imaging data. The "Imaging Data" field registers the imaging data captured by the camera corresponding to the time and camera no. The "Product Name" field registers information indicating the name of the product that is the subject of the imaging data, in other words, the product that is the target of the behavior extracted by the behavior analysis device 30. As described above, the "Judgment ID" field registers information that links the input (estimated target data) and output (behavioral information) in the machine learning model of the behavior analysis device 30.

[0036] The transaction information file 514 is a file that manages transaction information obtained from the store server 40. Transaction information is sent to the store server 40 from each POS terminal 20 as it occurs. The information processing device 50 obtains transaction information from the store server 40 at the time it obtains purchase estimation information. Figure 5 shows the data structure of the transaction information file 514. Each data entry registered in the transaction information file 514 is associated with information indicating the settlement time, POS terminal No., and product name.

[0037] The "Settlement Time" field registers information indicating the settlement time recorded by POS terminal 20. The "POS Terminal No." field registers a unique number that identifies POS terminal 20. By identifying the settlement time and the number of the POS terminal 20 used for settlement, it is possible to identify a single transaction. As described above, the combination of settlement time and POS terminal No. is one example of transaction identification information.

[0038] The corrected data file 515 is a file that manages the corrected data to be output as training data to the behavior analysis device 30. The corrected data is generated by the information processing device 50. For each transaction, the corrected data is generated based on a comparison between the purchased items estimated based on the output of the machine learning model and the purchased items based on the transaction information settled at the POS terminal 20. Figure 6 shows the data structure of the corrected data file 515. Each data registered in the corrected data file 515 is associated with information indicating the behavior, corrected judgment, time, camera number, and imaging data.

[0039] The "Behavior" field registers information indicating the customer's behavior extracted by the behavior analysis device 30, similar to the behavior field in the purchase estimation information file 512. The "Revised Judgment" field registers the judgment result corrected by the information processing device 50. In the example in Figure 6, "×" is registered in the "Revised Judgment" field, indicating that the "○" judgment result of the acquired action by the machine learning model was incorrect and has been corrected to "×". The information registered in the "Behavior" field and the information registered in the "Revised Judgment" field are examples of corrected behavior information.

[0040] The "Time" field registers information indicating the start and end times of the imaging data used to estimate the behavior. The "Camera No." field registers a unique number identifying the camera that captured the imaging data. The "Imaging Data" field registers the imaging data captured by the camera corresponding to the time and camera no. The modified data file 515 can be said to contain the modified behavior information (information registered in each of the behavior and modified judgment fields) and the estimated target data corresponding to that behavior information (information registered in each of the time, camera no., and imaging data fields).

[0041] Let's return to Figure 2 for explanation. The display device 520 displays various information, such as the correction screen which will be described later. The input device 530 inputs information to the control unit 500 and consists of a touch panel and keyboard provided on the surface of the display device 520. The communication unit 540 is an interface for communicating with external devices such as the behavior analysis device 30 and the store server 40. The control unit 500 is connected to the external device via the communication unit 540, enabling it to send and receive information (data) with the external device.

[0042] Next, the functional configuration of the information processing device 50 will be described. Figure 7 is a block diagram showing the functional configuration of the control unit 500 of the information processing device 50. The control unit 500 functions as a purchase estimation information acquisition unit 5001, an imaging data acquisition unit 5002, a transaction information acquisition unit 5003, a storage processing unit 5004, a comparison unit 5005, a correction unit 5006, a display processing unit 5007, a reception unit 5008, and an output unit 5009, with the CPU 501 operating according to the control program stored in the ROM 502 and the memory unit 510. Note that each of these functions may be configured with hardware such as dedicated circuits.

[0043] The purchase estimation information acquisition unit 5001 acquires purchase estimation information that associates behavioral information indicating customer behavior estimated based on sensor information from sensors installed in the store, product identification information that identifies products estimated to have been purchased by the customer in one transaction based on the behavioral information, and transaction identification information that can identify the said transaction. The behavioral information is, for example, information indicating customer behavior estimated by the machine learning model of the behavior analysis device 30 based on imaging data from the camera 10. The product identification information is, for example, the product name that is estimated to have been purchased by the customer in one transaction based on the estimation results of the machine learning model of the behavior analysis device 30. The transaction identification information is, for example, the settlement time and POS terminal No. estimated by the behavior analysis device 30. The purchase estimation information acquisition unit 5001 acquires purchase estimation information from the store server 40.

[0044] The imaging data acquisition unit 5002 acquires imaging data from the camera 10. More specifically, the imaging data acquisition unit 5002 acquires the estimated target data generated by the behavior analysis device 30 from the store server 40. In other words, the imaging data acquisition unit 5002 acquires the imaging data input to the machine learning model of the behavior analysis device 30 from the store server 40.

[0045] The transaction information acquisition unit 5003 acquires transaction information relating to store transactions, which associates product identification information and transaction identification information of the goods traded in a single transaction. Product identification information is, for example, the product name, which indicates the name of the product. Transaction identification information is, for example, the settlement time recorded by the POS terminal 20 and the number of the POS terminal 20. The transaction information acquisition unit 5003 acquires transaction information from the store server 40.

[0046] The purchase estimation information acquisition unit 5001, the image data acquisition unit 5002, and the transaction information acquisition unit 5003 each acquire information from the store server 40 at the same time once a month, for example. The purchase estimation information acquisition unit 5001, the image data acquisition unit 5002, and the transaction information acquisition unit 5003 may actively acquire information by outputting an output request to the store server 40, or they may passively acquire information by waiting for an output from the store server 40.

[0047] The memory processing unit 5004 stores the information acquired by the purchase estimation information acquisition unit 5001, the imaging data acquisition unit 5002, and the transaction information acquisition unit 5003 in the memory unit 510. Specifically, the memory processing unit 5004 registers the purchase estimation information acquired by the purchase estimation information acquisition unit 5001 in the purchase estimation information file 512. The memory processing unit 5004 also registers the estimated target data acquired by the imaging data acquisition unit 5002 in the estimated target data file 513. Similarly, the memory processing unit 5004 registers the transaction information acquired by the transaction information acquisition unit 5003 in the transaction information file 514. Furthermore, the memory processing unit 5004 registers the behavior information corrected by the correction unit 5006 in the correction data file 515.

[0048] The comparison unit 5005 compares the product identification information included in the purchase estimation information with the product identification information included in the transaction information for each transaction identified by the transaction identification information. For example, for each transaction, the comparison unit 5005 compares the product name and quantity of the product estimated by the behavioral analysis device 30 to have been purchased by the customer with the product name and quantity of the product actually paid for by the customer at the POS terminal 20.

[0049] If the comparison by the comparison unit 5005 reveals that the product identification information included in the purchase estimation information does not match the product identification information included in the transaction information, the modification unit 5006 modifies the behavior information based on the transaction information. Specifically, if the comparison by the comparison unit 5005 reveals that the product name and quantity of the product estimated to have been purchased by the customer by the behavior analysis device 30 do not match the product name and quantity of the product actually purchased by the customer at the POS terminal 20, the modification unit 5006 modifies the behavior information based on the transaction information from the POS terminal 20.

[0050] For example, in a particular transaction, the behavioral analysis device 30 might estimate that a customer purchased a certain product based on a machine learning model's estimation that the customer acquired that product, but the transaction information output from the POS terminal 20 might not include that product. In this case, the correction unit 5006 corrects the behavioral information indicating that the product was acquired by the machine learning model based on the correction input entered into the reception unit 5008 after the operator has confirmed the image data. The correction unit 5006 may also correct the behavioral information automatically. For example, in the above example, the correction unit 5006 might determine that the transaction information is correct data and that the machine learning model's estimation that the product was acquired is incorrect, and correct the behavioral information indicating the acquisition without operator confirmation.

[0051] The display processing unit 5007 displays a modified screen on the display device 520 that can display the image data acquired by the image data acquisition unit 5002, the behavior information and product identification information acquired by the purchase estimation information acquisition unit 5001, and the product identification information acquired by the transaction information acquisition unit 5003, all related to one transaction identified by the transaction identification information. Specifically, the display processing unit 5007 displays the estimated target data, the estimated customer behavior information, and the product name, which are information output from the behavior analysis device 30, on the display device 520. The display processing unit 5007 also displays the number of products identified by the product name. In addition, the display processing unit 5007 displays the product name and the number of products purchased by the customer in the above transaction, which are information output from the POS terminal 20, on the display device 520.

[0052] The reception unit 5008 accepts input for corrections to behavioral information. For example, it accepts input for corrections to behavioral information that an operator operating the information processing device 50 enters into the input device 530 while viewing a confirmation screen displayed on the display device 520.

[0053] The output unit 5009 outputs the behavioral information corrected by the correction unit 5006. Specifically, the output unit 5009 outputs the behavioral information corrected by the correction unit 5006 to the behavioral analysis device 30. The corrected behavioral information is used as training data for the machine learning model of the behavioral analysis device 30.

[0054] Next, we will explain the processing performed by the control unit 500 of the information processing device 50. Figure 8 is a flowchart showing the processing flow by the control unit 500 of the information processing device 50.

[0055] First, trigger information is input to the control unit 500 (S1). The trigger information is information that instructs the start of generating training data for the machine learning model, and may be input by an operator or by the store server 40 at a pre-set date and time. When the trigger information is input, the purchase estimation information acquisition unit 5001 acquires purchase estimation information from the store server 40, and the storage processing unit 5004 stores the purchase estimation information in the purchase estimation information file 512 (S2).

[0056] In addition, the imaging data acquisition unit 5002 acquires the estimated target data from the store server 40, and the storage processing unit 5004 stores the estimated target data in the estimated target data file 513 (S3). Furthermore, the transaction information acquisition unit 5003 acquires transaction information from the store server 40, and the storage processing unit 5004 stores the transaction information in the transaction information file 514 (S4). The information acquired by the purchase estimation information acquisition unit 5001, the imaging data acquisition unit 5002, and the transaction information acquisition unit 5003 is data from the same period (for example, the same one month).

[0057] Next, the comparison unit 5005 extracts and associates purchase estimation information, estimated data, and transaction information for one transaction from the information stored in the purchase estimation information file 512, the estimated target data file 513, and the transaction information file 514 (S5). The association between purchase estimation information and transaction information is performed using transaction identification information (settlement time and POS terminal No.) as the key (see Figures 3 and 5). Note that the settlement time included in the purchase estimation information is an estimated time and may not match the settlement time included in the transaction information. In this case, the comparison unit 5005 compares the purchase estimation information and transaction information and associates the two transactions that have matching POS terminal No. and the closest settlement times. The association between purchase estimation information and estimated target data is performed using the judgment ID as the key (see Figures 3 and 4).

[0058] Next, the comparison unit 5005 compares the product names and quantities included in the corresponding estimated purchase information and transaction information (S6). In other words, the comparison unit 5005 compares the products that the customer is estimated to have purchased by the behavioral analysis device 30 with the products that the customer actually paid for at the POS terminal 20.

[0059] The control unit 500 determines, based on the comparison result of the comparison unit 5005, whether there is a discrepancy between the product names and quantities of the products included in the purchase estimation information and the product names and quantities of the products included in the transaction information (S7). If there is a discrepancy, i.e., if there is at least one product name that does not match (Y in S7), the display processing unit 5007 displays the correction screen 521 on the display device 520 (S8).

[0060] The control unit 500 determines whether the reception unit 5008 has received the input for modification of the behavior information (S9). Details of the modification screen and the input for modification of behavior information will be described later. If the modification input is not received (N in S9), the control unit 500 returns to the process in S9. If the modification input is received (Y in S9), the modification unit 5006 modifies the behavior information based on the modification input (S10). For example, the modification unit 5006 modifies behavior information that is estimated to indicate that the customer acquired a certain product to behavior information that indicates the customer did not acquire that product.

[0061] Next, the memory processing unit 5004 stores the corrected behavior information in the corrected data file 515 of the memory unit 510 (S11). The control unit 500 determines whether the processing in S5 to S11 has been completed for all transactions of the acquired information (S12). If it has not been completed (N in S12), the control unit 500 returns to the processing in S5. If it has been completed (Y in S12), the output unit 5009 outputs the corrected data (corrected behavior information stored in the corrected data file 515) to the behavior analysis device 30. Then, the control unit 500 terminates the processing. If there is no discrepancy in the processing in S7 (N in S7), the control unit 500 skips the processing in S8 to S11 and proceeds to the processing in S12.

[0062] Through the above process, the behavioral information estimated by the machine learning model of the behavioral analysis device 30 is corrected based on the transaction information of the POS terminal 20, and the corrected behavioral information is output to the behavioral analysis device 30 as training data.

[0063] Next, the modification screen and the input of modification information for action information will be explained. Figure 9 shows an example of the modification screen 521. The modification screen 521 includes an image data display area 522, a purchase estimation information display area 523, and a transaction information display area 524.

[0064] A display frame 5221 is formed in the image data display area 522. The display frame 5221 displays the data to be estimated that has been input into the machine learning model of the behavior analysis device 30. The data to be estimated is the camera image, i.e., the image data captured by the camera 10.

[0065] The purchase estimation information display area 523 displays a play button 5231, a time display unit 5232, a judgment action information display unit 5233, a probability value display unit 5234, a modification action information display unit 5235, a modification button 5236, a delete button 5237, an item addition button 5238, and a confirmation button 5239. The play button 5231, time display unit 5232, judgment action information display unit 5233, probability value display unit 5234, modification action information display unit 5235, modification button 5236, and delete button 5237 are associated with each data to be estimated, in other words, with each action. The purchase estimation information display area 523 displays information for one transaction.

[0066] The play button 5231 is a button that plays back the estimated target data input to the machine learning model of the behavior analysis device 30. The time display unit 5232 displays the start time of imaging of the estimated target data. The judgment behavior information display unit 5233 displays behavior information and judgment results corresponding to the estimated target data. Specifically, the judgment behavior information display unit 5233 displays the product, behavior, and judgment result of the behavior by the machine learning model of the behavior analysis device 30 extracted from the estimated target data. In the first row example of Figure 9, the machine learning model of the behavior analysis device 30 extracted that the customer picked up one item A from the product display shelf, and it is shown that the extracted behavior was judged to be correct.

[0067] The probability value display unit 5234 displays the probability value of the action extracted by the machine learning model of the behavior analysis device 30 that was determined to have been performed. The corrected behavior information display unit 5235 displays the corrected behavior information and the judgment result. For behavior information whose judgment result is not corrected, the corrected behavior information display unit 5235 displays it in the same way as the judged behavior information display unit 5233. In addition, the corrected behavior information display unit 5235 highlights the corrected behavior information and the judgment result. The highlighting is done by changing the color or boldness of the text compared to other displays. In the second row example of Figure 9, the machine learning model of the behavior analysis device 30 determined that the action of a customer picking up product B from the product display shelf was correct (performed), but this was corrected to indicate that product B was not picked up from the product display shelf.

[0068] The edit button 5236 is a button for correcting behavior information. When the edit button 5236 is pressed, a dialog box will pop up, for example, allowing the user to input the corrected content. The delete button 5237 is a button for deleting the corresponding line displayed in the purchase estimation information display area 523, i.e., the delete button 5237 itself, as well as the play button 5231, time display unit 5232, judgment behavior information display unit 5233, probability value display unit 5234, corrected behavior information display unit 5235, and edit button 5236 that correspond to the delete button 5237.

[0069] One instance in the purchase estimation information display area 523 where a line of data is deleted might occur if the estimated target data played back by the play button 5231 does not show a person, if it shows a person other than the identified customer, if it shows multiple instances of the same action, or if it shows the customer doing nothing (not performing an acquisition or payment action). These are all cases where the process of generating the estimated target data from the camera 10's image data, performed by the behavior analysis device 30, was not carried out properly. Therefore, if the estimated target data generation process in the behavior analysis device 30 also uses a machine learning model, outputting the data related to the deleted actions from the purchase estimation information display area 523 as training data for the estimated target data generation process to the behavior analysis device 30 can improve the accuracy of the estimated target data generation process.

[0070] The item addition button 5238 is for the operator to manually add an item. The confirmation button 5239 is for saving the corrected action information. Alternatively, the corrected data may be automatically saved once the corrections have been entered (for example, when the pop-up display for entering the corrected content is closed), in which case the confirmation button 5239 is unnecessary.

[0071] The transaction information display area 524 displays the corresponding receipt information display unit 5241. The corresponding receipt information display unit 5241 displays the product name and quantity of the items included in the transaction information settled at the POS terminal 20 for one transaction corresponding to the information for one transaction displayed in the purchase estimation information display area 523.

[0072] This section explains how to modify behavioral information using the modification screen 521. First, as mentioned above, if there is a discrepancy in the comparison results of the comparison unit 5005, the modification screen 521 is displayed. In other words, for one transaction identified in the transaction identification information, if the goods (including quantity) estimated by the behavioral analysis device 30 to have been purchased by the customer do not match the goods (including quantity) that were paid for at the POS terminal 20, the modification screen 521 is displayed.

[0073] In the example shown in Figure 9, the products included in the purchase estimation information output by the behavioral analysis device 30 are product A, product B, and product C, while the products included in the transaction information output by the POS terminal 20 are product A and product C.

[0074] The purchase estimation information display area 523 displays behavioral information that forms the basis for the behavioral analysis device 30's estimation that products A, B, and C were purchased. For product A, the behavioral analysis device 30 estimates that product A was purchased based on the machine learning model's estimation of the acquisition action (see the first row of the judgment behavior information display unit 5233). Similarly, for product B, the behavioral analysis device 30 estimates that product B was purchased based on the machine learning model's estimation of the acquisition action (see the second row of the judgment behavior information display unit 5233).

[0075] For product C, the behavior analysis device 30 generates two data points for estimation, and the behavior is determined by a machine learning model. Specifically, for product C, the machine learning model does not estimate one acquisition action (see the third row of the determined behavior information display unit 5233), and estimates one acquisition action (see the fifth row of the determined behavior information display unit 5233). Based on the estimation results of this machine learning model, the behavior analysis device 30 estimates that product C was purchased. The information displayed in the third row of the determined behavior information display unit 5233 is an example of information showing customer behavior extracted based on sensor information that was not estimated to have been performed by the customer.

[0076] For product D, the behavior analysis device 30 generates two data points for estimation, and the behavior is determined by a machine learning model. Specifically, for product D, the machine learning model estimates one action as an acquisition action (see the fourth row of the determined behavior information display unit 5233) and the other action as a return action (see the sixth row of the determined behavior information display unit 5233). Based on the estimation results of this machine learning model, the behavior analysis device 30 estimates that product D was taken from the product display shelf by the customer, but was returned to the same shelf and not purchased by the customer.

[0077] The operator operating the information processing device 50 corrects the behavioral information regarding product B, which is included in the purchase estimation information output by the behavioral analysis device 30 but not in the transaction information output by the POS terminal 20. In making the correction, the operator operates the second-row play button 5231 corresponding to the behavioral information regarding product B and visually checks the estimated target data displayed in the display frame 5221. After visually checking the estimated target data, if the operator confirms that product B has not been picked up from the product display shelf by the customer, they operate the second-row correct button 5236 to correct the acquisition operation judgment result from "○" to "×". This corrects the behavioral information.

[0078] Furthermore, the information processing device 50 may automatically correct the behavioral information without displaying a correction screen. For example, in the example shown in Figure 9, there is only one piece of behavioral information regarding product B, which is included in the purchase estimation information but not in the transaction information, so this behavioral information can be automatically corrected. Also, for example, if the number of product B items estimated to have been purchased in the purchase estimation information is 2, but the number of product B items included in the transaction information is 1, the behavioral information with a low probability value may be automatically corrected from "○" to "×" as an error.

[0079] However, as in this embodiment, by displaying a correction screen and allowing the operator to confirm and correct the behavioral information, accurate corrected data can be provided to the behavioral analysis device 30. This is because, even if a customer takes an item from the display shelf, it may not be paid for due to shoplifting by the customer or the store clerk forgetting to register the item. Furthermore, the correction screen is only displayed when the product name and quantity of the item included in the purchase estimation information do not match the product name and quantity of the item included in the transaction information, so the burden on the operator will not become unnecessarily large.

[0080] As described above, the information processing device 50 of the embodiment includes: a purchase estimation information acquisition unit 5001 that acquires purchase estimation information which associates behavior information indicating customer behavior estimated based on sensor information from sensors installed in the store, product identification information which identifies products that are estimated to have been purchased by the customer in one transaction based on the behavior information, and transaction identification information which can identify the one transaction; a transaction information acquisition unit 5003 that acquires transaction information relating to the store's transactions, which associates the product identification information of products traded in one transaction with the transaction identification information; a comparison unit 5005 that compares the product identification information included in the purchase estimation information with the product identification information included in the transaction information with respect to one transaction identified by the transaction identification information; a modification unit 5006 that modifies the behavior information based on the transaction information if, as a result of the comparison by the comparison unit 5005, the product identification information included in the purchase estimation information and the product identification information included in the transaction information do not match; and an output unit 5009 that outputs the behavior information modified by the modification unit 5006.

[0081] This allows transaction information to be used to modify behavioral information estimated by machine learning models, and the modified behavioral information can be provided to the machine learning model as training data. Therefore, it becomes easier to improve the accuracy of machine learning models that estimate customer behavior within stores.

[0082] Furthermore, the information processing device 50 of the embodiment includes a purchase estimation information acquisition unit 5001 which acquires behavioral information indicating customer behavior extracted based on sensor information that was not estimated to have been performed by the customer.

[0083] This makes it possible to correct behavioral information that the machine learning model did not predict, or in other words, behavior that the machine learning model determined was not performed. As a result, the information processing device 50 can provide the machine learning model with more corrected behavioral information, thereby improving the accuracy of the machine learning model's decisions.

[0084] Furthermore, the information processing device 50 of this embodiment includes, where the sensor information is image data from the camera 10, an image data acquisition unit 5002 that acquires the image data, a display processing unit 5007 that displays the image data acquired by the image data acquisition unit 5002 on the display device 520, and a reception unit 5008 that receives input for correction of behavior information.

[0085] This allows the operator of the information processing device 50 to review the image data capturing the customer's actions and then correct the behavioral information. Therefore, the behavioral information can be corrected more accurately.

[0086] In addition, the information processing device 50 of the embodiment displays a modified screen on the display device, which can display imaging data acquired by the imaging data acquisition unit 5002, behavioral information and product identification information acquired by the purchase estimation information acquisition unit 5001, and product identification information acquired by the transaction information acquisition unit 5003, all related to one transaction identified by the transaction identification information.

[0087] This improves the ease of use for operators modifying behavioral information, thereby reducing the workload on operators.

[0088] In the above embodiment, the control programs executed by the behavioral analysis device 30, the store server 40, and the information processing device 50 may be provided by recording them on a computer-readable recording medium such as a CD-ROM. Alternatively, the control programs executed by each of the above devices in the above embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading them via the network, or they may be provided via a network such as the Internet.

[0089] Although embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0090] 1. Behavioral Analysis System 10 Cameras 20 POS terminals (accounting devices) 30 Behavior analysis device 40 Store Servers 50 Information Processing Devices 5001 Purchase Estimate Information Acquisition Unit 5002 Imaging data acquisition unit 5003 Transaction Information Acquisition Department 5004 Memory Processing Unit 5005 Comparison Section 5006 Correction Department 5007 Display Processing Unit 5008 Reception Department 5009 Output section [Prior art documents] [Patent Documents]

[0091] [Patent Document 1] Japanese Patent Publication No. 2020-27313

Claims

1. a purchase estimation information acquisition unit that acquires purchase estimation information that associates behavior information indicating customer behavior estimated based on sensor information from sensors installed in the store, product identification information that identifies products estimated to have been purchased by the customer in one transaction based on the behavior information, and transaction identification information that can identify the one transaction; a transaction information acquisition unit that acquires transaction information relating to transactions at the store, the transaction information being information relating to the product identification information of products traded in one transaction and the transaction identification information; a comparison unit that compares, for one transaction identified by the transaction identification information, the product identification information included in the purchase estimation information with the product identification information included in the transaction information; a correction unit that corrects the behavioral information based on the transaction information when the comparison by the comparison unit shows that the product identification information included in the purchase estimation information does not match the product identification information included in the transaction information; an output unit that outputs the behavioral information corrected by the correction unit; An information processing device comprising:

2. the purchase estimation information acquisition unit acquires behavior information indicating behavior of the customer extracted based on the sensor information but not estimated to have been performed by the customer; The information processing device according to claim 1 .

3. The sensor information is image data of a camera, an imaging data acquisition unit that acquires the imaging data; a display processing unit that displays the imaging data acquired by the imaging data acquisition unit on a display device; and a receiving unit that receives an input for correcting the behavioral information.

3. The information processing device according to claim 1.

4. the display processing unit displays, on the display device, a correction screen capable of displaying the imaging data acquired by the imaging data acquisition unit, the behavior information and product identification information acquired by the purchase estimation information acquisition unit, and the product identification information acquired by the transaction information acquisition unit, relating to one transaction identified by the transaction identification information. The information processing device according to claim 3 .

5. A program for controlling an information processing device by a computer, The computer a purchase estimation information acquisition unit that acquires purchase estimation information that associates behavior information indicating customer behavior estimated based on sensor information from sensors installed in the store, product identification information that identifies products estimated to have been purchased by the customer in one transaction based on the behavior information, and transaction identification information that can identify the one transaction; a transaction information acquisition unit that acquires transaction information relating to transactions at the store, the transaction information being information relating to the product identification information of products traded in one transaction and the transaction identification information; a comparison unit that compares, for one transaction identified by the transaction identification information, the product identification information included in the purchase estimation information with the product identification information included in the transaction information; a correction unit that corrects the behavioral information based on the transaction information when the comparison by the comparison unit shows that the product identification information included in the purchase estimation information does not match the product identification information included in the transaction information; an output unit that outputs the behavioral information corrected by the correction unit; A program that makes it work.

6. A behavior analysis system comprising: a behavior analysis device that estimates customer behavior in a store; a checkout device that processes transactions in the store; and an information processing device that corrects information indicating customer behavior estimated by the behavior analysis device based on transaction information of transactions processed by the checkout device, The information processing device includes: a purchase estimation information acquisition unit that acquires purchase estimation information that associates behavioral information indicating customer behavior estimated based on sensor information from sensors installed in the store, which is information output by the behavior analysis device, product identification information that identifies products estimated to have been purchased by the customer in a single transaction based on the behavioral information, and transaction identification information that can identify the single transaction; a transaction information acquisition unit that acquires transaction information relating to transactions at the store output by the accounting device, the transaction information associating the commodity identification information of the commodity traded in one transaction with the transaction identification information; a comparison unit that compares, for one transaction identified by the transaction identification information, the product identification information included in the purchase estimation information with the product identification information included in the transaction information; a correction unit that corrects the behavioral information based on the transaction information when the comparison by the comparison unit shows that the product identification information included in the purchase estimation information does not match the product identification information included in the transaction information; an output unit that outputs the behavioral information corrected by the correction unit to the behavior analysis device; A behavior analysis system comprising:

Images