Information processing program, information processing method, and information processing device

Abstract

To obtain a machine learning model capable of generating effective product information for customers.SOLUTION: An information processing device acquires a video obtained by shooting the interior of a store and analyzes the acquired video to identify a first action of a person for a product displayed in the store. The information processing device inputs the video to a machine learning model to generate content. The information processing device outputs the generated content to a terminal placed in the store. The information processing device analyzes the video after the content is output to the terminal to identify a second action of the person for the product. The information processing device retrains the machine learning model on the basis of the transition from the first action to the second action, the generated content, and an image of the product being a target of the identified second action.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing program and the like. [Background technology]

[0002] In stores, various methods are used to encourage customers to purchase more products. For example, pre-set product information is displayed on the cash register, and sales staff serve customers. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-165483 A Summary of the Invention [Problem to be solved by the invention]

[0004] As mentioned above, it is difficult to increase a customer's willingness to buy by simply displaying product information. On the other hand, if a customer shows interest in a certain product, it is possible to increase the customer's willingness to buy if useful product information can be presented to the customer.

[0005] For this reason, it is desirable to generate a machine learning model that can obtain useful product information for customers.

[0006] In one aspect, the present invention aims to provide an information processing program, an information processing method, and an information processing device that can obtain a machine learning model that can generate useful product information for customers. [Means for solving the problem]

[0007] In the first proposal, a computer executes the following process. The computer acquires a video of the inside of a store. The computer analyzes the acquired video to identify a first action of a person with respect to a product placed in the store. The computer generates content by inputting the video into a machine learning model. The computer outputs the generated content to a terminal placed in the store. The computer analyzes the video after the content is output to the terminal to identify a second action of the person with respect to the product. The computer retrains the machine learning model based on the transition from the first action to the second action, the generated content, and an image of the product that is a target of the identified second action. Effect of the Invention

[0008] It is possible to obtain a machine learning model that can generate useful product information for customers. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a system according to the present embodiment. [Diagram 2] FIG. 2 is a diagram (1) for explaining the processing of the information processing device according to the present embodiment. [Diagram 3] FIG. 3 is a diagram (2) for explaining the process of the information processing device according to the present embodiment. [Figure 4] FIG. 4 is a diagram (3) for explaining the process of the information processing device according to the present embodiment. [Diagram 5] FIG. 5 is a diagram explaining the machine learning of HOID. [Figure 6] FIG. 6 is a diagram showing an example of specifying a relationship using a scene graph. [Figure 7] FIG. 7 is a functional block diagram illustrating a configuration of an information processing device according to the present embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a data structure of the model table according to the present embodiment. [Figure 9]FIG. 9 is a diagram illustrating an example of a data structure of the display device management table according to the present embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a data structure of the transition definition table according to the present embodiment. [Figure 11] FIG. 11 is a flowchart (1) showing a processing procedure of the information processing device according to the present embodiment. [Figure 12] FIG. 12 is a flowchart (2) showing the processing procedure of the information processing device according to the present embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an information processing program, an information processing method, and an information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments. EXAMPLES

[0011] A system according to the present embodiment will be described. Fig. 1 is a diagram showing an example of a system according to the present embodiment. As shown in Fig. 1, this system has cameras 10a, 10b, and 10c, display devices 15a, 15b, and 15c, and an information processing device 100. The cameras 10a to 10c and the information processing device 100 are connected to each other via a network. In addition, the display devices 15a to 15c and the information processing device 100 are connected to each other via a network.

[0012] For convenience of explanation, FIG. 1 shows only cameras 10a to 10c and display devices 15a to 15c, but the system according to this embodiment may include other cameras and other display devices.

[0013] The cameras 10a to 10c are installed at predetermined positions in the store. A plurality of products are arranged in the store. The positions (coordinates) at which the cameras 10a to 10c are installed are different positions. In the following description, the cameras 10a to 10c are referred to as "camera 10" unless otherwise distinguished.

[0014] Camera 10 captures video of the inside of the store and transmits data of the captured video to information processing device 100. In the following description, the video data transmitted by camera 10 to information processing device 100 will be referred to as "video data."

[0015] The video data includes a plurality of image frames in a time series. A frame number is assigned to each image frame in ascending chronological order. One image frame is a still image captured by the camera 10 at a certain timing. Time data may be assigned to each image frame. Camera identification information for identifying the camera 10 that captured the video data is set in the video data.

[0016] The display devices 15a to 15c are installed at predetermined positions in the store, for example, around the products. The positions (coordinates) at which the display devices 15a to 15c are installed are different positions. In the following description, the display devices 15a to 15c are referred to as "display device 15" unless otherwise distinguished. The display device 15 displays product information output from the information processing device 100, etc.

[0017] The information processing device 100 acquires video data of the inside of the store from the camera 10, and by analyzing the acquired video data, identifies a first area including a target customer who is to purchase a product in the store, a second area including the product, and a relationship that identifies an interaction between the customer and the product. For example, the relationships include "looking," "touching," "sitting," "holding," etc. The relationship that identifies an interaction between the customer and the product corresponds to "a person's behavior toward the product."

[0018] The information processing device 100 selects a machine learning model from among a plurality of machine learning models stored in the storage unit based on the identified relationship. Each machine learning model is a trained machine learning model that outputs product advertisement information when an image of a second region including a product is input. The machine learning model is a neural network (NN) or the like. In the following description, the selected machine learning model is appropriately referred to as a "selected learning model." The product advertisement information output from the selected learning model is referred to as "content."

[0019] The information processing device 100 inputs an image of a second area including a product in the video data (image frame) into a selection learning model to generate content, and outputs the generated content to the display device 15 for display.

[0020] After outputting the content to the display device 15, the information processing device 100 acquires video data of the target customer from the camera 10, and analyzes the acquired video data to identify a first area including the target customer, a second area including the product, and a relationship that identifies an interaction between the customer and the product. Here, the relationship that identifies an interaction between the customer and the product before the content is output to the display device 15 is referred to as a "first relationship." In contrast, the relationship that identifies an interaction between the customer and the product after the content is output to the display device 15 is referred to as a "second relationship."

[0021] When the transition from the first relationship to the second relationship corresponds to a predefined transition, the information processing device 100 generates training data using an image of a second region including a product in the video data (image data) as input data and the content output from the selection learning model as a correct answer label, and registers the training data in a training data table. Note that when the transition from the first relationship to the second relationship does not correspond to a predefined transition, the information processing device 100 skips the process of generating training data.

[0022] For example, when the first relationship "looking" transitions to the second relationship "holding," it can be said that the target customer became interested in the product by referring to the content displayed on the display device 15, and useful product information was provided to the customer. In other words, the content output when an image of the second area including the product is input to the selection learning model corresponding to the first relationship is appropriate content for increasing the customer's willingness to purchase.

[0023] The information processing device 100 repeatedly executes the above process to register pairs (training data) of input data and correct labels in the training data table. The information processing device 100 retrains the selection learning model based on the input data and correct labels registered in the training data table. For example, the information processing device 100 retrains the selection learning model using backpropagation (backpropagation method). This makes it possible to obtain a machine learning model that can generate information on products that are effective for customers.

[0024] 2, 3, and 4 are diagrams for explaining the processing of the information processing device according to the present embodiment. First, FIG. 2 will be explained. For example, the information processing device 100 analyzes the video data (image frame) 20 captured by the camera 10a to identify a first area 20a including a customer, a second area 20b including a product, and a relationship (first relationship) between the customer and the product. In the example shown in FIG. 2, the first relationship is "looking at". Note that a display device 15a is installed near the product included in the second area 20b. Also, the display device 15a is the display device closest to the camera 10a among the multiple display devices 15.

[0025] We now move on to the description of FIG. 3. The information processing device 100 has a plurality of machine learning models. FIG. 3 shows machine learning models 30a, 30b, and 30c. For example, the machine learning model 30a is a machine learning model dedicated to the relationship "looking". The machine learning model 30b is a machine learning model dedicated to the relationship "touching". The machine learning model 30c is a machine learning model dedicated to the relationship "holding". The machine learning models 30a to 30c are NNs (Neural Networks) or the like.

[0026] The machine learning model 30a is trained in advance by a plurality of first training data corresponding to the relationship "looking". For example, the input of the first training data is image data of a product, and the output (correct answer label) is product information. The product information of the first training data is "advertising information of the product" or the like.

[0027] The machine learning model 30b is trained in advance by a plurality of second training data corresponding to the relationship "touching". For example, the input of the second training data is image data of a product, and the output (correct label) is product information. The product information of the second training data is "information explaining the advantages of the product", "information explaining the popularity of the product", etc.

[0028] The machine learning model 30c is trained in advance by a plurality of third training data corresponding to the relationship "held". For example, the input of the third training data corresponding to the relationship "held" is image data of a product, and the output (correct label) is product information. The product information of the third training data is "information explaining a benefit to be obtained by purchasing a product" or the like.

[0029] The information processing device 100 selects a machine learning model corresponding to the first relationship identified by the process described in Fig. 2 from the machine learning models 30a to 30c. For example, when the identified relationship is "watching", the information processing device 100 selects the machine learning model 30a in Fig. 3 as the selected learning model.

[0030] The information processing device 100 generates product information (content) of the products included in the second area 20b by inputting image data of the second area 20b including the products to the machine learning model 30a. The information processing device 100 outputs and displays the generated content on the display device 15a arranged near the products included in the second area, allowing customers to refer to the content.

[0031] We now move on to the description of Fig. 4. After outputting and displaying the content on the display device 15a, the information processing device 100 analyzes the video data (image frames) 21 captured by the camera 10a to identify a first area 21a including the customer, a second area 21b including the product, and a relationship between the customer and the product (second relationship). In the example shown in Fig. 4, the second relationship is set to "grasping".

[0032] The information processing device 100 executes the following process when the transition from the first relationship "looking" to the second relationship "holding" is a pre-specified relationship transition. The pre-specified relationship transition is a transition from the first relationship to the second relationship that is closer to the purchasing behavior of the product.

[0033] The information processing device 100 uses the image of the product in the second area 20b shown in Figure 2 as input data, generates first training data in which the content output when the image of the product in the second area 20b is input into the machine learning model 30a is used as the correct label, and registers the first training data in a first training data table for the machine learning model 30a.

[0034] 2 to 4, the case where the information processing device 100 selects the machine learning model 30a as the selected learning model has been described, but similar processing is performed even when the machine learning models 30b and 30c are selected. Note that the table that stores the second training data for the machine learning model 30b is referred to as the second training data table. The table that stores the third training data for the machine learning model 30c is referred to as the third training data table. In the following description, the first training data, the second training data, and the third training data are collectively referred to as training data as appropriate. The first training data table, the second training data table, and the third training data table are collectively referred to as the training data table.

[0035] The information processing device 100 repeatedly executes the above process to register a pair (training data) of input data and a correct label in a training data table corresponding to the selection learning model. The information processing device 100 retrains the selection learning model based on the input data and the correct label registered in the training data table. This makes it possible to obtain a machine learning model that can generate content that is effective for customers.

[0036] Meanwhile, the information processing device 100 uses HOID (Human Object Interaction Detection) to identify a first area including a customer, a second area including a product, and the relationship between the first area and the second area. The information processing device 100 inputs video data (time-series image frames) to HOID, and outputs information on the first area, the second area, and the relationship.

[0037] Here, a description will be given of an example of a learning process of the HOID executed by the information processing device 100. The information processing device 100 uses a plurality of training data to train the HOID that identifies a first class indicating a person, a second class indicating an object, and the relationship between the first class and the second class.

[0038] Each training data has image data (image frame) that is input data, and correct answer information set for the image data.

[0039] The correct answer information includes the classes of the human and object to be detected, a class indicating the interaction between the human and object, and a Bbox (Bounding Box) indicating the area of ​​each class. For example, the correct answer information includes area information of the Something class indicating an object, area information of the Human class indicating a user, and a relationship indicating the interaction between the Something class and the Human class.

[0040] In addition, multiple classes and multiple interactions can be set in the training data, and the trained HOID can also recognize multiple classes and multiple interactions.

[0041] Generally, if you create a Something class using normal object recognition, you will detect all backgrounds, clothing, small items, and other things that are not related to the task. And because they are all Something, all that is recognized in the image data is a large number of Bboxes, and nothing is known. In the case of HOID, you can tell that it is a special relationship between a person and an object (it could be something else, such as holding, sitting, or manipulating), so you can use it as meaningful information for the task.

[0042] Fig. 5 is a diagram for explaining machine learning of HOID. As shown in Fig. 5, the information processing device 100 inputs input data of training data to HOID and obtains an output result of HOID. This output result includes the human class, object class, and interaction between the human and object detected by HOID. Then, the information processing device 100 calculates error information between the correct answer information of the training data and the output result of HOID, and executes machine learning of HOID by error backpropagation so as to reduce the error.

[0043] Next, an example of the identification process using HOID will be described. The information processing device 100 inputs each image frame of the video data captured by the camera 10 to HOID and acquires the output result of HOID. The output result of HOID includes the Bbox of the person, the Bbox of the object, the probability value of the interaction between the person and the object (the probability value of each relationship), and the class name. The Bbox of the person corresponds to the above-mentioned first area. The Bbox of the object corresponds to the above-mentioned second area. The information processing device 100 specifies the relationship based on the output result of HOID. For example, the information processing device 100 specifies the relationship with the maximum probability value as the relationship between the first area and the second area.

[0044] As described above, the information processing device 100 can specify the first area, the second area, and the relationship by inputting the video data to the HOID. Note that the information processing device 100 may store the machine-learned HOID in advance in the storage unit, and specify the first area, the second area, and the relationship by using the HOID.

[0045] In addition, the information processing device 100 may use a technique other than HOID when identifying the first region, the second region, and the relationship. For example, the information processing device 100 may use an existing detection algorithm to detect objects including a person, estimate the relationship between each object, and generate a scene graph that represents each object and the relationship between them, that is, the context. Here, the existing detection algorithm is, for example, YOLO (YOU Only Look Once), SSD (Single Shot Multibox Detector), RCNN (Region Based Convolutional Neural Networks), etc.

[0046] FIG. 6 is a diagram showing an example of relationship specification by a scene graph. In the example shown in FIG. 6, at least two men represented by Bbox (Bounding box) 51 and 52, a woman represented by Bbox 53, a box represented by Bbox 54, and a shelf represented by Bbox 55 are detected from an image frame 50. The information processing device 100 cuts out the Bbox region of each object from the image frame 50, extracts the feature amount of each region, estimates the relationship of each object from the feature amount of the pair of objects (Subject, Object), and generates a scene graph 59. In FIG. 6, the scene graph 59 shows the relationship that the man represented by Bbox 51 is standing on the shelf represented by Bbox 55. In addition, the relationship to the man represented by Bbox 51 represented by the scene graph 59 is not limited to one. 6, the scene graph 59 indicates all the estimated relationships, such as a shelf, a person being behind a man indicated by Bbox 52, and a person holding a box indicated by Bbox 54. In this way, the information processing device 100 can specify the relationships between objects and people included in a video (image frame) by generating a scene graph.

[0047] Next, a configuration example of an information processing device 100 that executes the processes described in Fig. 2 to Fig. 4 will be described. Fig. 7 is a functional block diagram showing the configuration of an information processing device according to this embodiment. As shown in Fig. 7, the information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0048] The communication unit 110 executes data communication between the camera 10, the display device 15, an external device, etc., via a network. The communication unit 110 is a NIC (Network Interface Card) or the like. For example, the communication unit 110 receives video data from the camera 10.

[0049] The input unit 120 is an input device that inputs various information to the control unit 150 of the information processing device 100. For example, the input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0050] The display unit 130 is a display device that displays information output from the control unit 150.

[0051] The storage unit 140 has an HOID 141, an image buffer 142, a model table 143, and a display device management table 144. The storage unit 140 also has a first training data table 145a, a second training data table 145b, a third training data table 145c, and a transition definition table 146. The storage unit 140 is a storage device such as a memory.

[0052] HOID 141 is the HOID described in FIG. 5 and the like. By inputting an image frame of video data to HOID 141, a first area on the image frame, a second area, and a relationship between the first area (a person included in the first area) and the second area (an object included in the second area) are output. Such a relationship corresponds to the behavior of the person included in the first area with respect to the object included in the second area.

[0053] Video buffer 142 holds video data captured by camera 10. For example, video buffer 142 holds video data in association with camera identification information.

[0054] The model table 143 holds information on the multiple machine learning models 30a to 30c described in FIG. 3. FIG. 8 is a diagram showing an example of a data structure of the model table according to this embodiment. As shown in FIG. 8, this model table 143 associates model identification information, relationships, and machine learning models. The model identification information is information that uniquely identifies a machine learning model. The relationships indicate relationships corresponding to the machine learning models. The machine learning model is a NN that receives image data (image frames) as input and outputs product information.

[0055] For example, model identification information "M30a" indicates machine learning model 30a. Machine learning model 30a is a machine learning model corresponding to the relationship "looking". Model identification information "M30b" indicates machine learning model 30b. Machine learning model 30b is a machine learning model corresponding to the relationship "touching". Model identification information "M30c" indicates machine learning model 30c. Machine learning model 30c is a machine learning model corresponding to the relationship "holding".

[0056] The display device management table 144 holds information about the display devices 15 arranged in the store. Fig. 9 is a diagram showing an example of the data structure of the display device management table according to this embodiment. As shown in Fig. 9, this display device management table 144 associates display device identification information, positions, and camera identification information.

[0057] The display device identification information is information that uniquely identifies the display device 15. For example, the display device identification information of the display devices 15a, 15b, and 15c are A15a, A15b, and A15c, respectively. The position indicates the position (coordinates) of the display device 15. The camera identification information is information that identifies the camera 10 that is closest to the display device 15. For example, the camera identification information C10a, C10b, and C10c correspond to the cameras 10a, 10b, and 10c shown in FIG. 1.

[0058] For example, in FIG. 9, a display device 15a with display device identification information "A15a" is installed at a position "(x1, y1)", and information is registered that the camera 10 closest to the display device 15a is the camera 10a with camera identification information "C10a".

[0059] Returning to the description of Fig. 7, the first training data table 145a holds first training data for retraining the machine learning model 30a. As will be described later, the first training data is generated by the control unit 150 and registered in the first training data table 145a.

[0060] The second training data table 145b holds second training data for retraining the machine learning model 30b. As will be described later, the second training data is generated by the control unit 150 and registered in the second training data table 145b.

[0061] The third training data table 145c holds third training data for retraining the machine learning model 30c. As will be described later, the third training data is generated by the control unit 150 and registered in the third training data table 145c.

[0062] The transition definition table 146 is used when determining whether or not to register a pair of product image data and content as training data. When the transition from the first relationship to the second relationship is a transition defined in the transition definition table 146, the control unit 150 determines that the pair of product image data and content is to be registered as training data.

[0063] FIG. 10 is a diagram showing an example of a data structure of a transition definition table according to the present embodiment. As shown in FIG. 10, the transition definition table 146 sets item numbers and transitions from a first relationship to a second relationship. For example, the transition defined in item number (1) is a transition from the first relationship "looking" to the second relationship "holding". The transition defined in item number (2) is a transition from the first relationship "looking" to the second relationship "touching". The transition defined in item number (3) is a transition from the first relationship "touching" to the second relationship "holding". It can be said that the transitions from item number (1) to item number (3) show that the customer is more willing to purchase in the second relationship after the transition than in the first relationship before the transition. Although only the transitions from item number (1) to item number (3) are shown in FIG. 10, it is assumed that transitions of other item numbers are also defined.

[0064] The control unit 150 includes an acquisition unit 151, a generation unit 152, and a learning processing unit 153. The control unit 150 is a central processing unit (CPU), a graphics processing unit (GPU), or the like.

[0065] Acquiring unit 151 acquires video data from camera 10. As described above, the video data is set with camera identification information of camera 10 that captured the video data. Acquiring unit 151 stores the video data (time-series image frames) in video buffer 142 in association with the camera identification information.

[0066] The generation unit 152 generates training data for retraining the machine learning models 30a to 30c based on the video data captured by the camera 10. For example, the generation unit 152 executes a first analysis process, a content generation process, a second analysis process, and a training data generation process. The first analysis process, the content generation process, the second analysis process, and the training data generation process executed by the generation unit 152 will be described below in this order.

[0067] First, an example of the "first analysis process" executed by the generating unit 152 will be described. The generating unit 152 acquires video data from the video buffer 142, and identifies a first region, a second region, and a relationship by analyzing the video data. The relationship is a relationship between a "first object included in the first region" and a "second object included in the second region." For example, the generating unit 152 inputs time-series image frames (image data) included in the video data to the HOID 141, and identifies the first region, the second region, and the relationship on each image frame.

[0068] The generating unit 152 repeatedly executes the above process to identify the first area, the second area, and the relationship for each image frame in the time series. When repeatedly identifying the first area, the second area, and the relationship, the generating unit 152 tracks the customer included in the first area and the product included in the second area. The generating unit 152 generates information on the analysis result of the above process. In the following description, the information on the analysis result is referred to as "first analysis result information." For example, the first analysis result information is associated with image data of the first area including the person being tracked, image data of the second area including the product being tracked, the first relationship, and the camera identification information of the camera 10 that captured the video data (image frame). In the following description, the image data of the second area including the product being tracked is referred to as "product image data."

[0069] Next, an example of the "content generation process" executed by the generation unit 152 will be described. The generation unit 152 selects a machine learning model corresponding to the first relationship included in the first analysis result information from a plurality of machine learning models registered in the model table 143. The generation unit 152 inputs product image data to the selected machine learning model (selected learning model) and acquires content output from the selected learning model.

[0070] For example, when the relationship included in the first analysis result information is "looking", the generation unit 152 selects the machine learning model 30a corresponding to "looking" as the selected learning model from the model table 143. The generation unit 152 generates content by inputting product image data to the selected machine learning model 30a.

[0071] Based on the camera identification information and display device identification information set in the first analysis result information, the generation unit 152 selects the display device 15 located closest to the camera 10 corresponding to the camera identification information, and outputs and displays the generated content. For example, if the camera identification information set in the first analysis result information is "C10a", the display device 15 located closest to the camera 10a with the camera identification information "C10a" is the display device 15a with the display device identification information "A15a". In this case, the generation unit 152 outputs and displays the content on the display device 15a.

[0072] Next, an example of the "second analysis process" executed by the generation unit 152 will be described. For convenience of explanation, the selected learning model is assumed to be the machine learning model 30a. The generation unit 152 displays content generated by inputting product image data to the machine learning model 30a on the display device 15a installed near the camera 10a.

[0073] After displaying the content on the display device 15a, the generation unit 152 acquires the video data captured by the camera 10a and analyzes the video data to identify the first area, the second area, and the relationship (second relationship). For example, the generation unit 152 inputs time-series image frames (image data) included in the video data to the HOID 141 and identifies the first area, the second area, and the second relationship on each image frame.

[0074] The generating unit 152 generates second analysis result information based on the above processing results. The second analysis result information includes the first relationship, the second relationship, the product image data, and the content. The generating unit 152 generates multiple pieces of second analysis result information by repeatedly executing the above processing.

[0075] Next, an example of the "training data generation process" executed by the generation unit 152 will be described. The generation unit 152 determines whether or not the relationship between the first relationship and the second relationship in the second analysis result information (the transition from the first relationship to the second relationship) is registered in the transition definition table 146. For example, the relationship between the first relationship "looking" and the second relationship "holding" in the second analysis result information corresponds to the transition of item number (1) in the transition definition table 146.

[0076] When the relationship between the first relationship and the second relationship in the second analysis result information is registered in the transition definition table 146, the generation unit 152 executes the following process to generate training data. For example, the generation unit 152 generates training data in which the product image data included in the second analysis result information is used as input data and the content is used as a correct answer label. The generation unit 152 registers the training data in a corresponding training data table based on the first relationship included in the second analysis result information.

[0077] If the first relationship included in the second analysis result information is "looking", the generation unit 152 registers the generated training data in the first training data table 145a. If the first relationship included in the second analysis result information is "touching", the generation unit 152 registers the generated training data in the second training data table 145b. If the first relationship included in the second analysis result information is "holding", the generation unit 152 registers the generated training data in the third training data table 145c.

[0078] The generating unit 152 repeatedly executes the above process for each piece of second analysis result information. Note that, if the relationship between the first relationship and the second relationship of the second analysis result information is not registered in the transition definition table 146, the generating unit 152 skips the process of generating training data based on the corresponding second analysis result information, and executes the process for the next piece of second analysis result information.

[0079] As described above, the generation unit 152 executes the first analysis process, the content generation process, the second analysis process, and the training data generation process in that order to generate training data and register the training data in a corresponding training data table.

[0080] The learning processing unit 153 executes retraining of each machine learning model using backpropagation based on the training data table. For example, the learning processing unit 153 retrains the machine learning model 30a using the first training data registered in the first training data table 145a. The learning processing unit 153 updates the parameters of the machine learning model 30a so that the error between the output result when the input data of the first training data is input to the machine learning model 30a and the correct label of the first training data is reduced.

[0081] The learning processing unit 153 updates the parameters of the machine learning model 30b so as to reduce the error between the output result when the input data of the second training data is input to the machine learning model 30b and the correct label of the second training data.

[0082] The learning processing unit 153 updates the parameters of the machine learning model 30c so as to reduce the error between the output result when the input data of the third training data is input to the machine learning model 30c and the correct label of the third training data.

[0083] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 11 and Fig. 12 are flowcharts showing the processing procedure of the information processing device according to this embodiment. In Fig. 11, the acquisition unit 151 of the information processing device 100 acquires video data and stores it in the video buffer 142 (step S101).

[0084] The generation unit 152 of the information processing device 100 inputs each image frame of the video data to the HOID 141, and identifies a first region, a second region, and a first relationship for each image frame (step S102). The generation unit 152 selects a machine learning model from the model table 143 based on the first relationship (step S103).

[0085] The generating unit 152 generates content by inputting the product image data into the selection learning model (step S104). The generating unit 152 identifies the display device 15 based on the display device management table 144 (step S105). The generating unit 152 outputs the generated content to the display device 15 (step S106).

[0086] The generation unit 152 acquires video data from the camera 10 located near the display device that has output the content (step S107). The generation unit 152 inputs an image frame of the video data after the content has been output to the HOID, identifies a second relationship (step S108), and proceeds to step S109 in FIG.

[0087] 12 will now be described. The generating unit 152 of the information processing device 100 determines whether or not the transition from the first relationship to the second relationship is a transition set in the transition definition table 146 (step S109). If the transition is not a set transition (step S110, No), the generating unit 152 proceeds to step S113.

[0088] On the other hand, if the transition is a set transition (Yes at Step S110), the generating unit 152 proceeds to Step S111. The generating unit 152 generates training data with the product image data as input data and the content as a correct label (Step S111).

[0089] The generating unit 152 registers the generated training data in a training data table corresponding to the first relationship (Step S112).

[0090] If the generation unit 152 continues the process of generating training data (step S113, Yes), the process proceeds to step S101 in Fig. 11. On the other hand, if the generation unit 152 does not continue the process of generating training data (step S113, No), the process proceeds to step S114.

[0091] The learning processing unit 153 of the information processing device 100 retrains the machine learning model based on the training data table (step S114).

[0092] Next, the effect of the information processing device 100 according to the present embodiment will be described. The information processing device 100 identifies a first relationship by analyzing in-store video data, and displays content obtained by inputting product image data into a selection learning model corresponding to the first relationship on the display device 15. The information processing device 100 identifies a second relationship by analyzing the video data after the content is output, and retrains the selection learning model based on the transition from the first relationship to the second relationship, the content, and the product image data.

[0093] For example, if a transition from a first relationship to a second relationship is a transition that increases a customer's purchasing motivation, the content output when product image data is input to the selection learning model is appropriate content for increasing the customer's purchasing motivation. Therefore, by retraining the selection learning model using training data in which the product image data is input data and the content is used as a correct answer label, a machine learning model that can generate product information that is effective for customers can be obtained. Note that the first relationship corresponds to a first behavior, and the second relationship corresponds to a second behavior.

[0094] The information processing device 100 identifies the first relationship, selects a machine learning model (selected learning model) corresponding to the identified first relationship, inputs image data of the product into the selected learning model, generates content, and displays the content on the display device 15. In this way, content can be generated by a machine learning model suitable for the first relationship, and it is possible to stimulate customers' purchasing desire.

[0095] The information processing device 100 inputs video data (time-series image frames) to the HOID 141 to identify a first area including a person, a second area including a product, and the association between the first area and the second area. This makes it possible to select a machine learning model suited to the association of people.

[0096] In addition, the information processing device 100 according to the present embodiment has been described in the case where the first area of ​​the person, the second area of ​​the product, and the relationship are specified using the HOID 141, but the present embodiment is not limited thereto. The information processing device 100 may generate the scene graph described in FIG. 6 to specify the first area of ​​the person, the second area of ​​the product, and the relationship. In addition, as a related technique, a bounding box (Bbox) that encloses an area including an object or person in a rectangle from a video using a machine learning model is extracted, and the behavior of the person is identified based on the positional relationship between both Bboxes. However, in the related technique, the positional relationship between the Bboxes extracted from the video is based on a two-dimensional space, so that, for example, the depth between the Bboxes cannot be analyzed. Therefore, the related technique cannot identify the behavior of the person. On the other hand, the information processing device 100 according to the present embodiment can identify the behavior of the person by specifying the first area of ​​the person, the second area of ​​the product, and the relationship.

[0097] Next, a description will be given of an example of a hardware configuration of a computer that realizes the same functions as the above-mentioned information processing device 100. Fig. 13 is a diagram showing an example of a hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0098] 13, computer 300 includes CPU 301 for executing various arithmetic processes, input device 302 for receiving data input from a user, and display 303. Computer 300 also includes communication device 304 for transmitting and receiving data to and from external devices via a wired or wireless network, and interface device 305. Computer 300 also includes RAM 306 for temporarily storing various information, and hard disk device 307. Each of devices 301 to 307 is connected to bus 308.

[0099] The hard disk device 307 has an acquisition program 307a, a generation program 307b, and a learning processing program 307c. The CPU 301 reads out each of the programs 307a to 307c and loads them in the RAM 306.

[0100] The acquisition program 307a functions as an acquisition process 306a. The generation program 307b functions as a generation process 306b. The learning processing program 307c functions as a learning processing process 306c.

[0101] The processing of the acquisition process 306a corresponds to the processing of the acquisition unit 151. The processing of the generation process 306b corresponds to the processing of the generation unit 152. The processing of the learning processing process 306c corresponds to the processing of the learning processing unit 153.

[0102] It is not necessary that each of the programs 307a to 307c is stored in the hard disk device 307 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 300. Then, the computer 300 may read and execute each of the programs 307a to 307c.

[0103] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0104] (Appendix 1) We obtained footage of the inside of the store, By analyzing the captured video, the system identifies the primary behavior of people with respect to products placed in the store, generating content by inputting the video into a machine learning model; outputting the generated content to a terminal disposed within the store; analyzing the video after the content is output to the terminal to identify a second action of the person with respect to the product; Retraining the machine learning model based on the transitions from the first behavior to the second behavior, the generated content, and images of the products that are subject to the identified second behavior. An information processing program that causes a computer to execute a process.

[0105] (Appendix 2) The computer further executes a process of identifying the machine learning model based on a first behavior of the identified person; The generating process generates the content by inputting the video into the machine learning model identified by the identifying process; The information processing program according to claim 1, wherein the retraining process retrains the machine learning model identified by the identifying process.

[0106] (Appendix 3) The method further comprises: causing the computer to execute a process of identifying a first area including a person from within the video, a second area including the product, and a relationship that identifies an interaction between the person and the product by analyzing the video; The information processing program described in Appendix 2, characterized in that the process of identifying the machine learning model identifies a machine learning model related to the product from a plurality of machine learning models based on the relationship.

[0107] (Additional Note 4) The process of identifying includes referring to a storage unit in which the relationship is associated with a plurality of machine learning models in which product information has been learned, and identifying a machine learning model corresponding to the identified relationship from the plurality of machine learning models; The generating process generates content related to product information by inputting an image of the product that is the target of the identified first behavior into a specified machine learning model; The information processing program according to claim 3, characterized in that it causes a computer to execute a process of displaying content related to the identified product information on a terminal used by the person.

[0108] (Appendix 5) We obtained footage of the inside of the store, By analyzing the captured video, the system identifies the primary behavior of people with respect to products placed in the store, generating content by inputting the video into a machine learning model; outputting the generated content to a terminal disposed within the store; analyzing the video after the content is output to the terminal to identify a second action of the person with respect to the product; Retraining the machine learning model based on the transitions from the first behavior to the second behavior, the generated content, and images of the products that are subject to the identified second behavior. An information processing method characterized in that the processing is executed by a computer.

[0109] (Supplementary Note 6) The computer further executes a process of identifying the machine learning model based on the first behavior of the identified person, The generating process generates the content by inputting the video into the machine learning model identified by the identifying process; The information processing method according to claim 5, wherein the retraining process retrains the machine learning model identified by the identifying process.

[0110] (Appendix 7) The computer further executes a process of identifying a first area including a person from the video, a second area including the product, and a relationship that identifies an interaction between the person and the product by analyzing the video; The information processing method described in Appendix 6, characterized in that the process of identifying the machine learning model identifies a machine learning model related to the product from a plurality of machine learning models based on the relationship.

[0111] (Appendix 8) The process of identifying includes referring to a storage unit in which the relationship is associated with a plurality of machine learning models in which product information has been learned, and identifying a machine learning model corresponding to the identified relationship from the plurality of machine learning models; The generating process generates content related to product information by inputting an image of the product that is the target of the identified first behavior into a specified machine learning model; The information processing method according to claim 7, characterized in that a computer executes a process of displaying content relating to the identified product information on a terminal used by the person.

[0112] (Appendix 9) We obtained footage of the inside of the store, By analyzing the captured video, the system identifies the primary behavior of people with respect to products placed in the store, generating content by inputting the video into a machine learning model; outputting the generated content to a terminal disposed within the store; analyzing the video after the content is output to the terminal to identify a second action of the person with respect to the product; Retraining the machine learning model based on the transitions from the first behavior to the second behavior, the generated content, and images of the products that are subject to the identified second behavior. An information processing device having a control unit that executes processing.

[0113] (Supplementary Note 10) The control unit further executes a process of identifying the machine learning model based on a first behavior of the identified person, The generating process generates the content by inputting the video into the machine learning model identified by the identifying process; The information processing device according to claim 9, wherein the retraining process retrains the machine learning model identified by the identifying process.

[0114] (Supplementary Note 11) The control unit further executes a process of identifying a first area including a person from the video, a second area including the product, and a relationship that identifies an interaction between the person and the product by analyzing the video; The information processing device described in Appendix 10, characterized in that the process of identifying the machine learning model identifies a machine learning model related to the product from a plurality of machine learning models based on the relationship.

[0115] (Appendix 12) The process of identifying includes referring to a storage unit in which the relationship is associated with a plurality of machine learning models in which product information has been learned, and identifying a machine learning model corresponding to the identified relationship from the plurality of machine learning models; The generating process generates content related to product information by inputting an image of the product that is the target of the identified first behavior into a specified machine learning model; The information processing device according to claim 11, wherein the control unit further executes a process of displaying content related to the identified product information on a terminal used by the person. [Explanation of symbols]

[0116] 100 Information processing device 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 HOID 142 Video Buffer 143 Model Table 144 Display device management table 145a First training data table 145b Second training data table 145c Third training data table 150 Control section 151 Acquisition Department 152 Generation part 153 Learning Processing Unit

Claims

1. Obtaining footage of the inside of the store, By analyzing the captured video, the system identifies the primary behavior of people with respect to products placed in the store, generating content by inputting the video into a machine learning model; outputting the generated content to a terminal disposed within the store; analyzing the video after the content is output to the terminal to identify a second action of the person with respect to the product; Retraining the machine learning model based on the transitions from the first behavior to the second behavior, the generated content, and images of the products that are subject to the identified second behavior. An information processing program that causes a computer to execute a process.

2. further comprising: identifying the machine learning model based on a first behavior of the identified person; The generating process generates the content by inputting the video into the machine learning model identified by the identifying process; The information processing program according to claim 1 , wherein the retraining process retrains the machine learning model identified by the identifying process.

3. and further executing a process of identifying a first area including a person from within the video, a second area including the product, and a relationship that identifies an interaction between the person and the product by analyzing the video; The information processing program according to claim 2 , characterized in that the process of identifying the machine learning model includes identifying a machine learning model related to the product from a plurality of machine learning models based on the relationship.

4. The process of identifying includes referring to a storage unit in which the relationship is associated with a plurality of machine learning models in which product information has been learned, and identifying a machine learning model corresponding to the identified relationship from the plurality of machine learning models; The generating process generates content related to product information by inputting an image of the product that is the target of the identified first behavior into a specified machine learning model; 4. The information processing program according to claim 3, further comprising a step of causing a computer to execute a process of displaying content relating to the identified product information on a terminal used by the person.

5. Obtaining footage of the inside of the store, By analyzing the captured video, the system identifies the primary behavior of people with respect to products placed in the store, generating content by inputting the video into a machine learning model; outputting the generated content to a terminal disposed within the store; analyzing the video after the content is output to the terminal to identify a second action of the person with respect to the product; Retraining the machine learning model based on the transitions from the first behavior to the second behavior, the generated content, and images of the products that are subject to the identified second behavior. An information processing method characterized in that the processing is executed by a computer.

6. Obtaining footage of the inside of the store, By analyzing the captured video, the system identifies the primary behavior of people with respect to products placed in the store, generating content by inputting the video into a machine learning model; outputting the generated content to a terminal disposed within the store; analyzing the video after the content is output to the terminal to identify a second action of the person with respect to the product; Retraining the machine learning model based on the transitions from the first behavior to the second behavior, the generated content, and images of the products that are subject to the identified second behavior. An information processing device having a control unit that executes processing.

Images