Display control program, display control method, and information processing device.

Abstract

To reduce throughput necessary for establishing a database.SOLUTION: An information processing device identifies a state of a certain user for an object among multiple persons contained in video data by analyzing the video data obtained by imaging a first area containing the person or the object. The information processing device generates a questionnaire relating to the person or the object on the basis of the state of the certain user for the object. The information processing device identifies the respective positions and directions of the multiple persons to a display device by analyzing video data obtained by imaging a second area containing the display device. The information processing device causes the display device to display, on the basis of the identified position and direction, the questionnaire relating to the certain user when the certain user is in a state maximally coming close to the display device and directed toward the display device, and when the other users are apart from the certain user and are not directed to the display device.SELECTED DRAWING: Figure 24

Description

Technical Field

[0001] The present invention relates to a display control program, a display control method, and an information processing apparatus.

Background Art

[0002] It is widely practiced to accumulate the behavior history of users, including the purchase history of goods, the usage history of facilities, the order history of restaurants, etc. in a database and utilize it for future services. For example, questionnaires are prepared on tables or the like, or questionnaires are sent to users at a later date, and the response results of the questionnaires are stored in a database.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, many processes such as collection, scrutiny, and input of questionnaire results occur, and a large amount of processing is required for constructing a database. Since it is desired to database more useful information, the number of questionnaire items tends to increase, the burden on users increases, and there are also many users who do not respond to the questionnaire.

[0005] In one aspect, an object is to provide a display control program, a display control method, and an information processing apparatus capable of reducing the amount of processing required for constructing a database.

Means for Solving the Problems

[0006] In the first proposal, the display control program is characterized by causing a computer to perform the following processes: analyze video data of a first area including a person or object to identify the state of a specific user with respect to an object among multiple people included in the video data; generate a questionnaire related to the person or object based on the state of the specific user with respect to the object; analyze video data of a second area including a display device to identify the position and orientation of each of the multiple users with respect to the display device; and, based on the identified position and orientation, cause the display device to display the questionnaire related to the specific user when the specific user is closest to the display device and facing it, and other users are far away from the display device and not facing it. [Effects of the Invention]

[0007] According to one embodiment, the amount of processing required to build the database can be reduced. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of the overall configuration of the information processing system according to Example 1. [Figure 2] Figure 2 is a diagram illustrating the reference technology. [Figure 3] Figure 3 is a diagram illustrating the information processing device according to Example 1. [Figure 4] Figure 4 is a functional block diagram showing the functional configuration of the information processing device according to Embodiment 1. [Figure 5] Figure 5 is a diagram illustrating the customer database. [Figure 6] Figure 6 is a diagram illustrating the survey database. [Figure 7] Figure 7 is a diagram illustrating the analysis results database. [Figure 8] Figure 8 is a diagram illustrating the training data. [Figure 9] Figure 9 illustrates machine learning for relational models. [Figure 10] Figure 10 is a diagram for explaining the generation of an action recognition model. [Figure 11] Figure 11 is a diagram for explaining the identification of relationships. [Figure 12] Figure 12 is a diagram for explaining the identification of relationships by HOID. [Figure 13] Figure 13 is a diagram for explaining action recognition. [Figure 14] Figure 14 is a diagram for explaining the generation and transmission of questionnaires. [Figure 15] Figure 15 is a diagram for explaining the registration of analysis results. [Figure 16] Figure 16 is a flowchart showing the processing flow according to Example 1. [Figure 17] Figure 17 is a diagram showing an example of a scene graph. [Figure 18] Figure 18 is a diagram for explaining an example of the generation of a scene graph showing the relationship between a person and an object. [Figure 19] Figure 19 is a diagram for explaining the identification of relationships by a scene graph. [Figure 20] Figure 20 is a diagram for explaining the action recognition model according to Example 3. [Figure 21] Figure 21 is a diagram for explaining the machine learning of the action recognition model according to Example 3. [Figure 22] Figure 22 is a diagram for explaining questionnaire transmission using the action recognition model according to Example 3. [Figure 23] Figure 23 is a diagram for explaining questionnaire transmission according to Example 4. [Figure 24] Figure 24 is a diagram for explaining a specific example of questionnaire transmission according to Example 4. [Figure 25] Figure 25 is a flowchart showing the processing flow according to Example 4. [Figure 26] Figure 26 is a diagram for explaining an example of a signage questionnaire display according to Example 4. [Figure 27] Figure 27 is a diagram for explaining an example of the hardware configuration of an information processing apparatus. [Figure 28] FIG. 28 is a diagram for explaining an example of the hardware configuration of signage.

Embodiment for Carrying Out the Invention

[0009] Hereinafter, embodiments of the display control program, display control method, and information processing apparatus according to the present invention will be described in detail based on the drawings. Note that the present invention is not limited by this embodiment. Also, each embodiment can be appropriately combined within a non - conflicting range.

Embodiment

[0010] <Overall Configuration> FIG. 1 is a diagram showing an example of the overall configuration of an information processing system according to Embodiment 1. As shown in FIG. 1, this information processing system includes a store 1, which is an example of a space having an area where a product, which is an example of an object, is arranged, a plurality of cameras 2 installed at different locations within the store 1, and an information processing apparatus 10 that executes analysis of video data, which are connected via a network N. Note that the network N can adopt various communication networks such as the Internet or a dedicated line, regardless of whether it is wired or wireless.

[0011] The store 1 has products for customers 5 to purchase, such as in a supermarket or convenience store, and self - checkout using, for example, electronic payment is utilized. As an example of the store 1, an unmanned store where customers 5 are registered in advance and only registered customers 5 can use it is assumed. For example, the customer 5 accesses, such as the homepage of the operator who operates the store 1, and registers name, age, contact information (such as an email address), and payment method (such as a credit card number). After registration, the customer 5 can enter the store 1 by using the user ID, password, or entry card issued, and make purchases by paying using the registered payment method.

[0012] Each of the multiple cameras 2 is an example of a surveillance camera that captures a predetermined area within store 1, and transmits the captured video data to the information processing device 10. In the following explanation, the video data may be referred to as "video data." The video data also contains multiple frames in chronological order. Each frame is assigned a frame number in ascending order of chronological order. A single frame is image data of a still image captured by camera 2 at a certain point in time.

[0013] The information processing device 10 is an example of a computer device that has a customer database that stores information about customers 5 who have been permitted to enter store 1, receives video data from multiple cameras 2, and collects various data to improve services for customers 5. The customer database registers information such as name, age, contact information (e.g., email address), and payment method (e.g., credit card number).

[0014] (Explanation of reference technology) As a measure to improve service for customer 5, a survey is conducted for customer 5. Figure 2 illustrates the reference technology. As shown in Figure 2, when customer 5 purchases a product at store 1, or when customer 5 enters store 1 but intends to purchase a product, store employee 6 hands customer 5 a survey form upon their departure. Customer 5 fills out the survey form and sends it back by mail or other means. Afterwards, store employee 7 compiles the survey forms sent in by each customer 5 and creates a database. Based on the information thus compiled into the database, considerations are made regarding the timing of staff interaction, product arrangement, and product expansion.

[0015] As described above, the reference technology involves many processes such as collecting, scrutinizing, and inputting survey results, requiring a significant amount of processing power to build the database. Furthermore, the desire to database more useful information tends to lead to an increase in the number of survey questions, increasing the burden on users and resulting in many users not responding to the survey.

[0016] (Description of Example 1) Therefore, the information processing device 10 according to Example 1 recognizes the relationships between people, objects, the environment, and behaviors, as well as the attributes of people, from video footage of the store 1, and digitizes the situation (context) of the sales floor, reducing the processing of creating a database of analyzable information. Specifically, the information processing device 10 inputs video data captured from areas within the store 1 where products are placed into a machine learning model, thereby identifying the relationship between a specific user (customer 5) and a product in the customer's behavior towards the product included in the video data. Subsequently, the information processing device 10 obtains the customer's psychological evaluation of the product whose relationship has been identified. After that, the information processing device 10 registers the results related to the identified relationship and the customer's psychological evaluation in a database showing the product analysis results stored in the memory unit.

[0017] Figure 3 is a diagram illustrating the information processing device 10 according to Example 1. As shown in Figure 3, the information processing device 10 acquires video data captured inside the store 1, inputs each frame of the video data into a trained machine learning model, and identifies the relationship between the customer 5 and the product. For example, the information processing device 10 identifies whether or not the product was purchased, the time, the location, and the customer's actions toward the product (e.g., grasping).

[0018] Next, the information processing device 10 identifies items that could not be identified from the video based on the relationship between customer 5 and the product as psychological evaluations, generates a questionnaire regarding those psychological evaluations, and sends it to customer 5's terminal or other device. For example, the information processing device 10 sends a questionnaire to customer 5 who did not purchase the product, asking "Why didn't you purchase the product?"

[0019] Subsequently, upon receiving responses to the questionnaire, the information processing device 10 associates the identified results from the video with the questionnaire results and stores them in a database. For example, the information processing device 10 stores the "age, gender, and whether or not the product was purchased" identified from the video in association with the questionnaire result "reason for not purchasing the product."

[0020] In this way, the information processing device 10 can recognize customer behavior in real time from in-store video and other sources, and automatically send questionnaires to targeted customers at the optimal timing. Therefore, the information processing device 10 can acquire only effective questionnaire results, thereby reducing the processing load required for database construction.

[0021] <Functional Configuration> Figure 4 is a functional block diagram showing the functional configuration of the information processing device 10 according to Embodiment 1. As shown in Figure 4, the information processing device 10 has a communication unit 11, a storage unit 12, and a control unit 20.

[0022] The communication unit 11 is a processing unit that controls communication between other devices, such as a communication interface. For example, the communication unit 11 receives video data from each camera 2 and outputs the processing results of the information processing device 10 to a pre-specified device.

[0023] The memory unit 12 is a processing unit that stores various data and programs executed by the control unit 20, and is implemented by, for example, memory or a hard disk. This memory unit 12 stores customer DB 13, questionnaire DB 14, video data DB 15, training data DB 16, relationship model 17, behavior recognition model 18, and analysis results DB 19.

[0024] Customer DB13 is a database that stores information about customer 5. The information stored here is that of customer (user) 5 who visits store 1 and wishes to purchase products, and is collected and registered when the user registers before visiting the store.

[0025] Figure 5 is a diagram illustrating the customer database 13. As shown in Figure 5, the customer database 13 stores information such as "customer ID, name, age, gender, family structure, notification recipient, number of visits, and card information." The "customer ID" is an identifier that identifies customer 5. The "name, age, gender, family structure, and card information" are the information entered by customer 5 during user registration, and the "number of visits" is the number of visits counted when entering the store.

[0026] The survey database 14 is a database that stores the surveys to be sent to customer 5. Figure 6 is a diagram illustrating the survey database 14. As shown in Figure 6, the survey to be sent can include multiple question items, each corresponding to a question (Q) and an answer choice.

[0027] In the example in Figure 6, Question 1 (Q1) is a question item that inquires about the customer's age and gender, with answer options such as "Female / Male, 20s / 30s / 40s / 50s / 60s / 70s and over." Question 3 (Q3) is a question item that inquires about the type of product purchased, with answer options such as "Food / Daily necessities / Other."

[0028] Furthermore, each question can be associated with the 5W1H (When, Where, Who, What, Why, How) that indicates the intent of the question. For example, Q1, "Please tell us your age and gender," can be associated with "Who," and Q6, "Please tell us why you are dissatisfied with the service," can be associated with "Why."

[0029] The video data DB15 is a database that stores video data captured by each of the multiple cameras 2 installed in store 1. For example, the video data DB15 stores video data for each camera 2, or for each time period in which the video was captured.

[0030] The training data DB16 is a database that stores various training data used to generate various machine learning models described in the embodiment, including the relationship model 17 and the action recognition model 18. The training data stored here can include supervised training data with correct answer information attached, and unsupervised training data without correct answer information attached.

[0031] Relationship Model 17 is an example of a machine learning model that identifies the relationship between a person and an object in the actions of a specific user towards an object contained in video data. Specifically, Relationship Model 17 is a machine learning-generated Human Object Interaction Detection (HOID) model that identifies the relationship between people or between a person and an object.

[0032] For example, when identifying the relationship between people, a HOID model is used as the relationship model 17, which, in response to the input of frames in the video data, identifies and outputs a first class representing the first person and first region information indicating the region in which the first person appears, a second class representing the second person and second region information indicating the region in which the second person appears, and the relationship between the first class and the second class.

[0033] Furthermore, when identifying the relationship between people and objects, a HOID model is used as the relationship model 17, which identifies and outputs a first class representing people and first region information indicating the region in which people appear, a second class representing objects and second region information indicating the region in which objects appear, and the relationship between the first class and the second class.

[0034] It should be noted that the relationships shown here are not limited to simple relationships such as "holding," but also include complex relationships such as "holding product A in the right hand," "returning product B to the shelf," and "putting the product in the shopping cart." Furthermore, the relationship model 17 may be created by using the two HOID models described above separately, or a single HOID model may be used that identifies both the relationship between people and the relationship between people and objects. Also, while the relationship model 17 is generated by the control unit 20 described later, a pre-generated model may be used.

[0035] Action recognition model 18 is an example of a machine learning model that performs skeletal information and action recognition of a person from video data. Specifically, action recognition model 18 outputs 2D skeletal information and action recognition results in response to image data input. For example, action recognition model 18 is an example of a deep learning machine that estimates the 2D joint positions (skeletal coordinates) of the head, wrists, hips, ankles, etc., from 2D image data of a person, and performs recognition of basic movements and user-defined rules.

[0036] This behavior recognition model 18 can recognize a person's basic movements and obtain the position of the ankles, the direction of the face, and the orientation of the body. Basic movements include, for example, walking, running, and stopping. User-defined rules include the transitions of skeletal information corresponding to each action up to picking up a product. The behavior recognition model 18 is generated by the control unit 20, which will be described later, but pre-generated data may also be used.

[0037] The analysis results DB19 is a database that stores information about the analysis results collected by the information processing device 10. Figure 7 is a diagram illustrating the analysis results DB19. As shown in Figure 7, the analysis results DB19 stores information such as "ID, name, user information, product, purchase status, and survey results."

[0038] "ID" is an identifier that identifies the analysis results. "Name" is the name of customer 5, which is identified using customer DB 13 when entering the store or purchasing a product. "User information" is customer 5's age, gender, family structure, etc., which is identified using customer DB 13. "Product" is information about the products purchased by customer 5, which is identified using customer DB 13 when purchasing a product. "Purchase status" is information indicating whether or not a product was purchased during the visit, which is identified using customer DB 13 when purchasing a product. "Questionnaire results" are the answers to the questionnaire sent by the control unit 20, which will be described later.

[0039] In the example in Figure 7, it is shown that "Hanako Tokachi" is a "woman in her 30s," purchased "cosmetics and food," and answered "dissatisfied with the service (the staff were unfriendly)" in the survey. The information stored here is used to detect the situation in the sales area and to determine the appropriate response. For example, if information such as "customers present, families, young men and women, woman taking the lead" is registered, the staff will be instructed to recommend family-friendly products that are popular with women.

[0040] Returning to Figure 4, the control unit 20 is the processing unit that oversees the entire information processing device 10, and is implemented by, for example, a processor. This control unit 20 has a pre-processing unit 30 and an operation processing unit 40. The pre-processing unit 30 and the operation processing unit 40 are implemented by electronic circuits of the processor or processes executed by the processor.

[0041] <Pre-processing step 30> The pre-processing unit 30 is a processing unit that generates models, rules, etc., using the training data stored in the memory unit 12, prior to the operation of the operation processing unit 40 for behavior prediction and questionnaire tabulation.

[0042] (Generating relational models) The preprocessing unit 30 is a processing unit that generates a relationship model 17 using the training data stored in the training data DB 16. Here, as an example, we will explain how to generate a HOID model using a neural network as the relationship model 17. Note that this is merely an example of generating a HOID model that identifies the relationship between people and objects, but a HOID model that identifies the relationship between people can be generated in the same way.

[0043] First, let's explain the training data used for machine learning the HOID model. Figure 8 is a diagram illustrating the training data. As shown in Figure 8, each training data set consists of input image data (explanatory variables) and correct information set for that image data (target variable).

[0044] The ground truth information includes the class of the person being detected (first class), the class of the object being purchased or manipulated by the person (second class), a relationship class indicating the interaction between the person and the object, and a Bbox (Bounding Box: object region information) indicating the region of each class. In other words, the ground truth information includes information about the object being held by the person. Note that the interaction between a person and an object is just one example of a relationship between a person and an object. Furthermore, when used to identify the relationship between two people, the second class is a class indicating the other person, the region information of the other person is used as the region information of the second class, and the relationship between the two people is used as the relationship class.

[0045] Next, we will explain the machine learning of the HOID model using the training data. Figure 9 is a diagram illustrating the machine learning of the relationship model 17. As shown in Figure 9, the preprocessing unit 30 inputs the training data into the HOID model and obtains the output result of the HOID model. This output result includes the person class detected by the HOID model, the object class, and the relationship (interaction) between the person and the object. The preprocessing unit 30 then calculates the error information between the correct information of the training data and the output result of the HOID model, and performs machine learning by updating the parameters of the HOID model through backpropagation to reduce the error.

[0046] (Generation of the behavior recognition model 18) The preprocessing unit 30 is a processing unit that generates an action recognition model 18 using training data. Specifically, the preprocessing unit 30 generates the action recognition model 18 through supervised learning using training data with correct information (labels).

[0047] Figure 10 illustrates the generation of the behavior recognition model 18. As shown in Figure 10, the preprocessing unit 30 inputs image data of basic actions, to which basic actions have been labeled, into the behavior recognition model 18, and performs machine learning on the behavior recognition model 18 to minimize the error between the output of the behavior recognition model 18 and the labels. For example, the behavior recognition model 18 is a neural network. The preprocessing unit 30 modifies the parameters of the neural network by performing machine learning on the behavior recognition model 18. The behavior recognition model 18 inputs explanatory variables, which are image data (for example, image data of a person performing a basic action), into the neural network. The behavior recognition model 18 then generates a machine learning model in which the parameters of the neural network have been modified to minimize the error between the output of the neural network and the ground truth data (target variable), which is the label of the basic action.

[0048] Furthermore, as training data, image data with labels such as "walking," "running," "stopping," "standing," "standing in front of a shelf," "picking up an item," "turning head to the right," "turning head to the left," "looking up," and "tilting head down" can be used. Note that the generation of the behavior recognition model 18 is merely an example, and other methods can be used. Additionally, the behavior recognition disclosed in Japanese Patent Publication No. 2020-71665 and Japanese Patent Publication No. 2020-77343 can also be used as the behavior recognition model 18.

[0049] <Operations Processing Unit 40> Returning to Figure 4, the operation processing unit 40 has an acquisition unit 41, a relationship identification unit 42, an action recognition unit 43, an evaluation acquisition unit 44, and a registration unit 45. It is a processing unit that sends questionnaires to people appearing in video data using each model prepared in advance by the pre-processing unit 30.

[0050] The acquisition unit 41 is a processing unit that acquires video data from each camera 2 and stores it in the video data DB 21. For example, the acquisition unit 41 may acquire data from each camera 2 as needed, or it may acquire data periodically.

[0051] Furthermore, the acquisition unit 41 acquires customer information when customer 5 enters the store and outputs it to each processing unit of the operation processing unit 40. For example, the acquisition unit 41 acquires the "customer ID" of the customer who entered the store by having the user enter a user card, fingerprint authentication, ID and password, etc. Then, the acquisition unit 41 refers to the customer DB 13 and acquires the name, age, etc. that are associated with the "customer ID".

[0052] (Identifying the relationship) The relationship identification unit 42 is a processing unit that uses the relationship model 17 to perform relationship identification processing to identify the relationships between people or between people and objects in the video data. Specifically, the relationship identification unit 42 inputs each frame included in the video data into the relationship model 17 and identifies the relationships according to the output result of the relationship model 17. The relationship identification unit 42 then outputs the identified relationships to the evaluation acquisition unit 44, the registration unit 45, etc.

[0053] Figure 11 illustrates the identification of relationships. As shown in Figure 11, the relationship identification unit 42 inputs frame 1 into a machine learning-based relationship model 17 to identify the class of the first person, the class of the second person, and the relationships between people. Alternatively, the relationship identification unit 42 inputs a frame into a machine learning-based relationship model 17 to identify the class of a person, the class of an object, and the relationships between people and objects. In this way, the relationship identification unit 42 uses the relationship model 17 to identify the relationships between people or between people and objects for each frame.

[0054] Figure 12 illustrates the identification of relationships using HOID. As shown in Figure 12, the relationship identification unit 42 inputs each frame (image data) contained in the video data into HOID (relationship model 17) and obtains the output result of HOID. Specifically, the relationship identification unit 42 obtains the Bbox of the person, the class name of the person, the Bbox of the object, the class name of the object, the probability value of the interaction between the person and the object, and the class name of the interaction between the person and the object.

[0055] As a result, for example, the relationship identification unit 42 identifies "person (customer)" and "product (item)" as classes of people, and identifies the relationship between "person (customer)" and "product (item)" as "customer holds product". The relationship identification unit 42 then performs the same relationship identification process for each subsequent frame, such as frame 2 and frame 3, to identify relationships such as "holds product A" and "hands over product A" for each frame. The relationship identification unit 42 can also obtain information on whether or not a product has been purchased from self-checkout machines or information at the time of leaving the store.

[0056] Furthermore, the relationship identification unit 42 can also identify information related to the time, location, and relationship of an action taken by a customer towards an object contained in the video data. For example, the relationship identification unit 42 can identify the time of the frame in the video data in which the relationship has been identified, the location of the camera 2 that captured the video data, and so on.

[0057] (behavior recognition) The behavior recognition unit 43 is a processing unit that uses the behavior recognition model 18 to recognize a person's actions and gestures from video data. Specifically, the behavior recognition unit 43 inputs each frame in the video data to the behavior recognition model 18, and uses the skeletal information and basic movements of each part of the person obtained from the behavior recognition model 18 to identify the person's actions and gestures, and outputs them to the evaluation acquisition unit 44, the registration unit 45, etc.

[0058] Figure 13 is a diagram illustrating action recognition. As shown in Figure 13, the action recognition unit 43 inputs frame 1, which is image data, to the action recognition model 18. The action recognition model 18 generates skeletal information for each body part in response to the input of frame 1, and outputs the movement of each body part according to the skeletal information of each body part. For example, by using the action recognition model 18, the action recognition unit 43 can obtain movement information for each body part, such as "face: facing forward, arms: raising, legs: walking, ...".

[0059] Furthermore, the action recognition unit 43 performs recognition processing using the action recognition model 18 for each frame following frame 2 and frame 3, identifying the motion information of each part of the person shown in the frame for each frame. The action recognition unit 43 can also refer to the correspondence between representative gestures and changes in behavior that are stored in a pre-associated manner, and use the changes in the action recognition results (i.e., motion information of each part) to identify more specific actions and gestures.

[0060] For example, the behavior recognition unit 43 can recognize a gesture as "dissatisfied" if it detects a pre-specified "dissatisfied behavior," such as when the face moves left or right within 5 frames, or when the product is returned to its original position 15 frames or more after being held. Furthermore, the behavior recognition unit 43 can recognize a gesture as "satisfied" if it detects a pre-specified "satisfied behavior," such as when the product is placed in the basket within 3 frames of being held.

[0061] The evaluation acquisition unit 44 is a processing unit that acquires the customer's (5) psychological evaluation of products whose relationship has been identified by the relationship identification unit 42. Specifically, the evaluation acquisition unit 44 can also use "gestures" recognized by the behavior recognition unit 43 as the psychological evaluation.

[0062] Furthermore, the evaluation unit 44 can also send a questionnaire regarding psychological indicators of the customer 2's product to a terminal associated with customer 5, and obtain the results of the questionnaire received from the terminal as a psychological evaluation of the customer.

[0063] To give a specific example, the evaluation acquisition unit 44 generates a partial questionnaire that queries the multiple items included in the questionnaire stored in the questionnaire DB 14 that were not identified from the customer 2's actions toward the product. The evaluation acquisition unit 44 can then send the partial questionnaire to the customer's terminal and acquire the questionnaire response results received from the terminal as the customer's psychological evaluation.

[0064] Figure 14 is a diagram illustrating the generation and transmission of questionnaires. As shown in Figure 14, the evaluation acquisition unit 44 uses the customer information (in their 30s, female, number of visits (10th time)) acquired by the acquisition unit 22 to automatically input "in their 30s, female" into the "age, gender" field of questionnaire Q1, and "more than two times" into the "Is this your first visit?" field of questionnaire Q2.

[0065] Furthermore, the evaluation unit 44 uses the customer-product relationship "Product A, Not Purchased" identified by the relationship identification unit 42 to exclude questionnaire Q3, which asks whether the product was purchased, and questionnaire Q4, which asks about satisfaction with the purchased product, from the questionnaire. The evaluation unit 44 uses the behavior and gesture "dissatisfied" identified by the behavior recognition unit 43 to automatically input "dissatisfied" into questionnaire Q5, "Were you satisfied with the service?".

[0066] Furthermore, the evaluation acquisition unit 44 uses the relationship between the customer and the product, "Product A, not purchased," identified by the relationship identification unit 42, and the behavior and gestures, "dissatisfied," identified by the behavior recognition unit 43, to determine "why the product was not purchased and why the customer was dissatisfied." In other words, the evaluation acquisition unit 44 determines that "why" corresponds to the customer's psychological evaluation. As a result, the evaluation acquisition unit 44 selects Q6, "Please tell us why you are dissatisfied with the service," which corresponds to "why," from among the items included in the questionnaire, as a partial questionnaire 61 and sends it to the "notification recipient" stored in the customer DB 13.

[0067] Then, when the evaluation unit 44 receives the response "The staff were unfriendly" to the partial questionnaire 61, it determines the customer's psychological evaluation to be "The staff were unfriendly." The evaluation unit 44 can also use management data that associates at least one of the 5W1H with each combination of relationship identification results and behavior recognition results to determine which questionnaire items to select as partial questionnaires. In addition, since the "why" questionnaire generally provides the most desired information, the evaluation unit 44 can also send only the questionnaire items corresponding to "why" as partial questionnaires.

[0068] The registration unit 45 is a processing unit that registers information related to the relationship between customer 2 and the product, identified by the relationship identification unit 23, and the customer 2's psychological evaluation, acquired by the evaluation acquisition unit 44, in the analysis results DB 19. Specifically, the registration unit 45 registers the identified time, place, and relationship-related information in the analysis results DB 19, associating it with the results of the partial questionnaire.

[0069] Figure 15 illustrates the registration of analysis results. As shown in Figure 15, the registration unit 45 acquires the questionnaire items "female, in her 30s, visited more than twice, dissatisfied with the service" which were automatically entered by the evaluation acquisition unit 44, and also acquires the result of the partial questionnaire 61 "staff were unfriendly". The registration unit 45 then registers the acquired "female, in her 30s, visited more than twice, dissatisfied with the service, staff were unfriendly" into the analysis results DB 19.

[0070] Furthermore, the registration unit 45 can also register various information in the analysis results DB 19, such as the time of the frame in the video data whose relationship has been identified by the relationship identification unit 42, and the location of the camera 2 that captured the video data. For example, the registration unit 45 can register information such as the time "13:00", the location "product shelf YY", and relationship information such as "held product A in hand" or "stopped at product shelf YY" in the analysis results DB 19. In addition, the registration unit 45 can also register only customer information and the results of partial questionnaires in the analysis results DB 19. In other words, the registration unit 45 can register any analysis items requested by the user.

[0071] <Processing flow> Figure 16 is a flowchart illustrating the processing flow according to Example 1. While this example describes the process from a single customer entering to leaving the store, the operation processing unit 40 is not required to track a single customer; it can perform the above processing using the video data captured by each camera 2. In this case, the operation processing unit 40 can distinguish each customer by recognizing each person in the video data at the time of entry and assigning identifiers. Furthermore, pre-processing is assumed to be completed.

[0072] As shown in Figure 16, when the operation processing unit 40 of the information processing device 10 detects that customer 2 has entered the store (S101: Yes), it identifies the person who entered the store and obtains customer information (S102).

[0073] Next, the operation processing unit 40 acquires video data (S103:Yes), uses the video data and relationship model 17 to identify the relationship between the customer and the product (S104), and uses the video data and behavior recognition model 18 to identify the customer's behavior and gestures towards the product (S105).

[0074] Subsequently, steps S103 onwards are repeated until departure from the store is detected (S106: No). If departure from the store is detected (S106: Yes), the operation processing unit 40 determines the content of the questionnaire using the identified relationships, behaviors, and gestures (S107).

[0075] Then, the operation processing unit 40 sends a questionnaire (partial questionnaire 61) to inquire about the determined questionnaire content (S108), and upon receiving the questionnaire results (S109: Yes), generates analysis results (S110), and registers the analysis results in the analysis results DB 19 (S111).

[0076] <Effects> As described above, the information processing device 10 can automatically input most of the survey items from the video data and send only the survey items that cannot be identified from the video data. Therefore, the information processing device 10 can reduce the burden on customers, increase the number of customers who respond to the survey, enable the collection of more useful information, and reduce the processing load required to build the database.

[0077] Furthermore, since the information processing device 10 can send questionnaires with pinpoint accuracy, it can reduce respondents' reluctance to participate in questionnaires and improve the response rate. [Examples]

[0078] By the way, in Example 1, we explained an example using a model for HOID to identify the relationship between customers and products, but it is not limited to this, and a scene graph, which is an example of graph data that shows the relationships between each object included in video data, can also be used.

[0079] Therefore, in Example 2, we will explain an example in which the relationship identification unit 42 of the operation processing unit 40 identifies the relationship between the customer and the product using a scene graph. A scene graph is graph data that describes the relationships between each object (such as a person or a product) included in each image data within the video data.

[0080] Figure 17 shows an example of a scene graph. As shown in Figure 17, a scene graph is a directed graph in which objects in the image data are represented as nodes, each node has attributes (e.g., object type), and relationships between nodes are represented as directed edges. In the example in Figure 17, the relationship from the node "Person" with the attribute "Shopkeeper" to the node "Person" with the attribute "Customer" is shown to be "Talk". That is, it is defined that the relationship is "Shopkeeper talks to customer". Also, the relationship from the node "Person" with the attribute "Customer" to the node "Product" with the attribute "Large" is shown to be "Stand". That is, it is defined that the relationship is "Customer stands in front of the shelf of large products".

[0081] The relationships shown here are merely examples. For example, they include not only simple relationships such as "holding," but also complex relationships such as "holding product A in the right hand." It is also possible to store separate scene graphs corresponding to relationships between people and scenes corresponding to relationships between people and objects, or to store a single scene graph that includes all of these relationships. Furthermore, while the scene graph is generated by the control unit 20 described later, pre-generated data may also be used.

[0082] Next, we will explain the generation of a scene graph. Figure 18 illustrates an example of scene graph generation showing the relationship between people and objects. As shown in Figure 18, the preprocessing unit 30 inputs image data into a trained recognition model, and as the output of the recognition model, it obtains the label "person (male)", the label "drink (green)", and the relationship "has". In other words, the preprocessing unit 30 obtains that "the man is holding a green drink". As a result, the preprocessing unit 30 generates a scene graph that associates the node "person" having the attribute "male" with the node "drink" having the relationship "has". Note that the generation of the scene graph is just one example, and other methods can be used, and it can also be generated manually by an administrator or other person.

[0083] Next, we will explain how to identify relationships using a scene graph. The relationship identification unit 42 performs relationship identification processing to identify the relationships between people or between people and objects in the video data, according to the scene graph. Specifically, the relationship identification unit 42 identifies the type of person or object in each frame included in the video data, and uses the identified information to search the scene graph and identify the relationships. The relationship identification unit 42 then outputs the identified relationships to each processing unit.

[0084] Figure 19 illustrates the identification of relationships using a scene graph. As shown in Figure 19, the relationship identification unit 42 identifies the type of person, the type of object, the number of people, etc., in frame 1 by inputting frame 1 into a machine learning model that has been trained on, or by performing known image analysis on frame 1. For example, the relationship identification unit 42 identifies "person (customer)" as the type of person and "product (product A)" as the type of object. Then, according to the scene graph, the relationship identification unit 42 identifies the relationship "person (customer) has product (product A)" between the node "person" with the attribute "customer" and the node "product A" with the attribute "food". The relationship identification unit 42 identifies relationships for each frame by performing the above relationship identification process for each subsequent frame, such as frame 2 and frame 3.

[0085] As described above, the information processing device 10 according to Example 2 can easily determine relationships appropriate for each store by using a scene graph generated for each store, for example, without having to retrain a machine learning model to suit each store. Therefore, the information processing device 10 according to Example 2 can be easily implemented in this embodiment. [Examples]

[0086] Incidentally, in addition to machine learning models that recognize a person's actions and gestures from video data, machine learning models that perform binary class classification can also be used as the behavior recognition model 18 mentioned above. That is, as the behavior recognition model 18, a model that detects "hesitant" actions corresponding to the actions or gestures of the target of the questionnaire can be used.

[0087] Figure 20 illustrates the behavior recognition model 18 according to Example 3. As shown in Figure 20, the behavior recognition model 18 determines one of two values, Class 1 "hesitated about purchasing the product" or Class 2 "not hesitated about purchasing the product," based on the input image data. The output of the behavior recognition model 18 includes the confidence level (e.g., probability value) for each class.

[0088] Next, the training of the behavior recognition model 18 according to Example 3 will be described. Figure 21 is a diagram illustrating the machine learning of the behavior recognition model 18 according to Example 3. As shown in Figure 21, the preprocessing unit 30 inputs training data into the behavior recognition model 18, which has "image data" showing a person selecting a product as an explanatory variable and "hesitated" or "not hesitated" as the correct answer label, which is the objective variable, and obtains the output result of the behavior recognition model 18. Subsequently, the preprocessing unit 30 updates the parameters of the behavior recognition model 18 so that the error between the output result of the behavior recognition model 18 and the correct answer label is reduced. In this way, the preprocessing unit 30 trains the behavior recognition model 18 and generates the behavior recognition model 18.

[0089] Next, we will describe sending questionnaires using the trained behavior recognition model 18. Figure 22 is a diagram illustrating the sending of questionnaires using the behavior recognition model 18 according to Example 3. As shown in Figure 22, the operation processing unit 40 inputs each frame in the video data captured by the camera 2 to the behavior recognition model 18 and obtains the output result of the behavior recognition model 18.

[0090] Then, the operation processing unit 40 receives Class 1 "Lost" as an output result of the behavior recognition model 18, and if the difference between the confidence level of Class 1 "Lost" and the confidence level of Class 2 "Did not lose" is greater than or equal to a threshold, and the output result is highly reliable, it suppresses the submission of the questionnaire.

[0091] On the other hand, the operation processing unit 40 obtains Class 1 "Lost" as the output result of the behavior recognition model 18, and if the difference between the confidence level of Class 1 "Lost" and the confidence level of Class 2 "Did not lose" is less than a threshold, and the output result is low confidence, it executes the submission of the questionnaire. In addition, if the operation processing unit 40 obtains Class 2 "Not lost" as the output result of the behavior recognition model 18, it executes the submission of the questionnaire regardless of the difference in confidence levels.

[0092] In other words, the operation processing unit 40 controls the sending of the questionnaire according to the confidence level when a Class 1 "I'm confused" response is identified.

[0093] Furthermore, the operation processing unit 40 can also generate retraining data using the survey results. For example, suppose the operation processing unit 40 inputs image data AA into the behavior recognition model 18, and the output result obtained is class 1 "lost" and low confidence, so it sends a survey and receives the response "did not get lost". In this case, the operation processing unit 40 can generate training data for retraining, with "image data AA" as the explanatory variable and "did not get lost" as the objective variable. The pre-processing unit 30 can improve the recognition accuracy of the behavior recognition model 18 by retraining the behavior recognition model 18 using this retraining data.

[0094] The questionnaire sent here may also be the partial questionnaire 61 described above. For example, if the recognition result is "Class 1 'Hesitated' and high confidence," which is an example of the first condition, the operation processing unit 40 registers the analysis result using the automatic acquisition method described in Example 1. On the other hand, if the recognition result is "Class 1 'Hesitated' and low confidence," which is an example of the second condition, or "Class 2 'Not Hesitated'," the operation processing unit 40 registers the analysis result using the automatic acquisition method and the response results of the partial questionnaire described in Example 1.

[0095] Furthermore, the questionnaire sent may be the entire questionnaire described in Example 1, or it may be other pre-prepared question information. In other words, the operation processing unit 40 can also send the questionnaire 60 only when a highly reliable class 1 "confused" is detected by the behavior recognition model 18 of Example 3, without performing the relationship identification process or the behavior and gesture identification process of Example 1.

[0096] Furthermore, in addition to binary classification, a multi-class classification behavior recognition model 18 can also be used. For example, the behavior recognition model 18 performs multi-class classification such as Class 1 "Very confused", Class 2 "Confused", Class 3 "Not confused", and Class 4 "Neither". In this case, if the difference between the class with the highest confidence level and the class with the second highest confidence level is greater than or equal to a threshold, the behavior recognition model 18 registers the analysis results using the automated acquisition method described in Example 1. On the other hand, if the difference between the class with the highest confidence level and the class with the second highest confidence level is less than a threshold, the behavior recognition model 18 registers the analysis results using the automated acquisition method and the results of the partial questionnaire described in Example 1.

[0097] Thus, the information processing device 10 according to Example 3 can control the transmission of questionnaires according to the reliability of the recognition results of the behavior recognition model 18. Therefore, it can obtain user evaluations through questionnaires not only when the customer's psychological evaluation is poor, but also when the customer's psychological evaluation is only slightly poor. As a result, the information processing device 10 can collect accurate analysis results. [Examples]

[0098] Incidentally, there are times when it is necessary to conduct a survey targeting specific individuals, such as elderly people or dissatisfied customers. In the case of automated surveys that do not involve human intervention, it is possible to conduct surveys targeting individuals if the survey can be sent to personal devices such as smartphones. However, in retail stores and other similar establishments, it may not be possible to send surveys to personal devices because the personal information of customers is unknown. Even in such cases, the information processing device 10 can send the survey to any location, not just the customer's device.

[0099] Figure 23 illustrates the transmission of questionnaires according to Embodiment 4. As shown in Figure 23, the operation processing unit 40 of the information processing device 10 can transmit the questionnaire 60 or partial questionnaire 61 to the display of the self-checkout register 70 or the signage 80 of the store 2.

[0100] However, when attempting to conduct an individualized survey using equipment in store 2, such as touch-enabled digital signage, there is a possibility that customers other than the target audience may answer the survey, making it impossible to conduct a survey with specific criteria.

[0101] Therefore, the information processing device 10 uses the positional relationship between the survey response signage, the survey participant, and surrounding people other than the participant, as well as information on each person's posture, to display the survey response screen on the signage only when the participant is in a situation where they can answer the survey, thereby prompting them to answer the survey.

[0102] For example, the information processing device 10 analyzes video footage of a first area including customers or products to identify the state of a customer regarding a product among multiple people in the video. Based on the customer's state regarding the product, the information processing device 10 generates a questionnaire related to the customer or product. Then, the information processing device 10 analyzes video footage of a second area including the signage to identify the position and orientation of each of the multiple customers relative to the signage. Subsequently, based on the identified position and orientation, the information processing device 10 displays the questionnaire for a specific customer on the signage when that particular customer is closest to the signage and facing it, while other customers are farther away from the signage and not facing it.

[0103] Figure 24 illustrates a specific example of questionnaire transmission according to Example 4. As shown in Figure 24, the operation processing unit 40 of the information processing device 10 inputs each image data (each frame) in the video data into the behavior recognition model 18 to identify the position and orientation of the person in each image data. Here, the operation processing unit 40 identifies customers holding products, customers making payments, and customers staying in front of product shelves for a certain period of time or longer as questionnaire subjects (specific customers) based on the processing results of the relationship identification unit 42.

[0104] Then, as shown in Figure 24(a), the operation processing unit 40 determines, based on the position and orientation of the person in each image data, that the person being surveyed is facing the signage 80 and in a position to operate it, and that the person not being surveyed is not facing the signage 80 and is not in a position to operate it, and displays the survey on the signage 80.

[0105] On the other hand, as shown in Figure 24(b), the operation processing unit 40 does not display the survey on the signage 80 if it determines, based on the position and orientation of the person in each image data, that the person being surveyed is not facing the signage 80, and that a person not being surveyed is facing the signage 80 and in a position where they can operate it.

[0106] Furthermore, as shown in Figure 24(c), the operation processing unit 40, based on the position and orientation of the people in each image data, determines that the survey respondents are facing the signage 80 but are not in a position to operate it, and that non-survey respondents are not facing the signage 80, and displays a message prompting the survey respondents to move closer to the signage 80.

[0107] Figure 25 is a flowchart showing the processing flow according to Example 4. As shown in Figure 25, the operation processing unit 40 acquires on-site video data (S201) and performs analysis of the video data (S202). For example, the operation processing unit 40 performs tasks such as identifying relationships, identifying the position and orientation of people, and identifying actions and gestures.

[0108] Next, the operation processing unit 40 performs a determination of the survey target and the survey display conditions (S203). For example, the operation processing unit 40 reads the predetermined survey content and target conditions and uses the analysis results to determine whether or not the display conditions are met.

[0109] If the operation processing unit 40 determines that it is not necessary to display the questionnaire (S204: No), it repeats steps S201 onwards. On the other hand, if the operation processing unit 40 determines that it is necessary to display the questionnaire (S204: Yes), it displays the questionnaire on a display device such as the signage 80 and accepts responses (S205).

[0110] Subsequently, when the operation processing unit 40 receives a response to the questionnaire (S206:Yes), it records the questionnaire (S207) and hides the questionnaire (S209). On the other hand, if the operation processing unit 40 does not accept a response to the questionnaire (S206:No), it displays the questionnaire on a display device such as the signage 80 and accepts responses (S205) until a timeout is reached (S208:No). If the operation processing unit 40 does not accept a response to the questionnaire (S206:No) and a timeout is reached (S208:Yes), it hides the questionnaire (S209).

[0111] Furthermore, the operation processing unit 40 can also display dummy questionnaires that are not used for analysis, especially in the case of large-screen signage 80. Figure 26 is a diagram illustrating an example of a questionnaire display on signage according to Embodiment 4.

[0112] As shown in Figure 26, the operation processing unit 40 identifies the positions and orientations of survey participants and non-survey participants based on the position and orientation of the people in each image data. The operation processing unit 40 then displays the survey 62 in the area of ​​the signage 80 facing the survey participants and displays a dummy survey 63 in the area of ​​the signage 80 facing the non-survey participants.

[0113] Subsequently, the operations processing unit 40 registers the results of the questionnaire 62 as analysis results and discards the results of the dummy questionnaire 63. It is also useful to manage the results of the dummy questionnaire 63 as information about the accompanying person.

[0114] In this way, the information processing device 10 uses video from surveillance cameras or the like to determine the position and posture of the signage 80 for answering the questionnaire, the person answering the questionnaire, and the people around them. The information processing device 10 displays the questionnaire screen on the signage 80 only when the person closest to the signage 80 is the person answering the questionnaire, the person is facing the signage 80, and no one else is facing the signage 80. As a result, the information processing device 10 can prevent situations where a person who is not the person answering the questionnaire answers it, thus reducing the quality of the responses. [Examples]

[0115] Now, although embodiments of the present invention have been described, the present invention may be implemented in various other forms besides those described above.

[0116] <Numerical values, etc.> The numerical examples, number of cameras, label names, rule examples, action examples, and state examples used in the above embodiment are merely examples and can be changed as needed. Furthermore, the processing flow explained in each flowchart can also be modified as appropriate within a consistent scope. While the above embodiment uses a store as an example, it is not limited to this and can be applied to warehouses, factories, classrooms, train cars, airplane cabins, and other similar locations.

[0117] <System> Unless otherwise specified, the processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed at will.

[0118] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown. That is, all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0119] Furthermore, each processing function performed by each device can be implemented, in whole or in part, by a CPU and a program executed for analysis by that CPU, or by wired logic hardware.

[0120] <Hardware of the information processing device 10> Figure 27 illustrates an example of the hardware configuration of the information processing device 10. As shown in Figure 27, the information processing device 10 includes a communication device 10a, an HDD (Hard Disk Drive) 10b, memory 10c, and a processor 10d. Furthermore, each of the components shown in Figure 27 is interconnected by a bus or the like.

[0121] The communication device 10a is a network interface card or the like, and communicates with other devices. The HDD 10b stores programs and databases that operate the functions shown in Figure 4.

[0122] The processor 10d operates a process that performs the functions described in Figure 4 by reading a program that performs the same processing as each processing unit shown in Figure 4 from the HDD 10b or the like and loading it into memory 10c. For example, this process performs the same functions as each processing unit of the information processing device 10. Specifically, the processor 10d reads a program that has the same functions as the pre-processing unit 30 and the operation processing unit 40 from the HDD 10b or the like. Then, the processor 10d executes a process that performs the same processing as the pre-processing unit 30 and the operation processing unit 40.

[0123] Thus, the information processing device 10 operates as an information processing device that executes an information processing method by reading and executing a program. Furthermore, the information processing device 10 can also achieve the same functionality as the embodiment described above by reading the program from the recording medium using a media reader and executing the read program. Note that the program referred to in this other embodiment is not limited to being executed by the information processing device 10. For example, the above embodiment may also be applied to cases where another computer or server executes the program, or where these computers or servers collaborate to execute the program.

[0124] This program may be distributed via a network such as the Internet. Alternatively, this program may be recorded on a computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO (Magneto-Optical disk), or DVD (Digital Versatile Disc), and executed by being read from the recording medium by a computer.

[0125] <Signage 80 Hardware> Figure 28 illustrates an example of the hardware configuration of the signage 80. As shown in Figure 28, the signage 80 includes a communication device 80a, a touch panel 80b, an HDD 80c, memory 80d, and a processor 80e. Furthermore, each component shown in Figure 28 is interconnected by a bus or the like.

[0126] The communication device 80a is a network interface card or similar, and communicates with other devices. The touch panel 80b displays a questionnaire and accepts questionnaire responses. The HDD 80c stores various programs and databases.

[0127] The processor 80e runs the processes that execute each task by reading a program that performs the same tasks as described in Example 4 from the HDD 80c or the like and loading it into memory 80d. For example, this process performs functions similar to receiving a questionnaire, displaying the questionnaire, and accepting responses to the questionnaire.

[0128] Thus, the signage 80 operates as an information processing device that executes a display method by reading and executing a program. Furthermore, the signage 80 can also achieve the same functionality as the embodiment described above by reading the program from a recording medium using a media reader and executing the read program. Note that the program referred to in this other embodiment is not limited to being executed by the signage 80. For example, the above embodiment may also be applied to cases where another computer or server executes the program, or where these devices collaborate to execute the program.

[0129] This program may be distributed via a network such as the Internet. Alternatively, this program may be recorded on a computer-readable recording medium such as a hard disk, flexible disk, CD-ROM, MO, or DVD, and executed by reading it from the recording medium by a computer. [Explanation of symbols]

[0130] 10 Information Processing Devices 11 Communications Department 12 Storage section 13 Customer DB 14. Questionnaire Database 15 Video Data Database 16 Training Data Database 17. Relationship Model 18. Behavior Recognition Models 19 Analysis result DB 20 Control Unit 30 Pre-processing 40 Operation Processing Unit 41 Acquisition Department 42 Relationship Identification Section 43 Behavior Recognition Department 44 Evaluation Acquisition Department 45 Registration Department 80 Digital Signage

Claims

1. On the computer, By analyzing video data of a first area containing a person or object, the state of a specific user relative to an object among multiple people included in the video data can be identified. Based on the state of the object of the specific user, a questionnaire related to the person or the object is generated. By analyzing video data captured from a second area including the display device, the position and orientation of each of the multiple users relative to the display device can be identified. Based on the identified position and orientation, the display device is made to display a questionnaire related to the specific user when the specific user is closest to the display device and facing it, and other users are far away from the specific user and not facing it. A display control program characterized by causing a process to be executed.

2. By inputting video data of an area where an object is placed into a machine learning model, the relationship between a person and an object in the actions of a specific user towards the object contained in the video data is identified. Obtain the person's psychological evaluation of the object whose relationship has been identified. The results related to the identified relationship and the psychological evaluation of the person are registered in a database that shows the analysis results of the object stored in the memory unit. The display control program according to claim 1, characterized in that it causes the computer to perform the processing.

3. The process to be identified is, From the actions of the specific user towards an object included in the video data, information related to the time, place, and relationship in which the action was performed is identified. The aforementioned acquisition process is, A questionnaire regarding psychological indicators related to the object of the aforementioned specific user is sent to a terminal associated with the aforementioned specific user. The results of the questionnaire received from the terminal are obtained as a psychological evaluation of the person. The aforementioned registration process is: The results of the questionnaire obtained are registered in a database that shows the analysis results of the object, in correspondence with the results related to the relationship. The display control program according to claim 2.

4. The aforementioned machine learning model, This is a Human-Object Interaction Detection (HOID) model generated by machine learning to identify a first class representing a person and first region information indicating the region in which the person appears, a second class representing an object and second region information indicating the region in which the object appears, and the relationship between the first class and the second class. The process to be identified is, The aforementioned video data is input to the HOID model, As output of the HOID model, the first class and first region information, the second class and second region information, and the relationship between the first class and the second class are obtained for the people and objects appearing in the video data. Based on the results obtained, identify the relationship between the person and the object. The display control program according to claim 2 or 3, characterized by the above.

5. The process to be identified is, Acquire video data containing objects including people and objects. Using graph data showing the relationships between each object, the relationship between people and objects in the acquired video data is identified. The display control program according to claim 2 or 3, characterized by the above.

6. The process to display the above is: Even when both the specified user and the other user are facing the display device, the first questionnaire for the specified user is displayed in the area facing the display device for the specified user, and the second questionnaire for the other user is displayed in the area facing the display device for the other user. The display control program according to feature 1.

7. Computers By analyzing video data of a first area containing a person or object, the state of a specific user relative to an object among multiple people included in the video data can be identified. Based on the state of the object of the specific user, a questionnaire related to the person or the object is generated. By analyzing video data captured from a second area including the display device, the position and orientation of each of the multiple users relative to the display device can be identified. Based on the identified position and orientation, the display device is made to display a questionnaire related to the specific user when the specific user is closest to the display device and facing it, and other users are far away from the specific user and not facing it. A display control method characterized by executing a process.

8. By analyzing video data of a first area containing a person or object, the state of a specific user relative to an object among multiple people included in the video data can be identified. Based on the state of the object of the specific user, a questionnaire related to the person or the object is generated. By analyzing video data captured from a second area including the display device, the position and orientation of each of the multiple users relative to the display device can be identified. Based on the identified position and orientation, the display device is made to display a questionnaire related to the specific user when the specific user is closest to the display device and facing it, and other users are far away from the specific user and not facing it. An information processing device characterized by having a control unit.

Images