Information processing device, information processing method, and information processing program

Abstract

To encourage usage of a digital signage to display characters that can be interactive with users.SOLUTION: An information processing device according to the present application comprises: an acquisition unit for acquiring a user video in which a user positioned in front of a digital signage is captured; a determination unit for estimating a state of the user based on analysis information related to analysis results of the user video, and determining whether or not to have a character displayed on the digital signage speak to the user in accordance with the estimated state of the user; a generation unit for, when the determination unit determines to have the character speak to the user, generating a content of dialogue texts corresponding to the state of the user; and an output control unit for controlling the dialogue texts to be output by voice.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there are known technologies for providing products and services that match a user's preferences through dialogue with the user. For example, there is known a technology for estimating a user's age based on the user's speech or a video of the user's appearance, and determining the audio or video to be output based on the estimated user's age. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-26700 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional technology merely estimates the user's age based on the voice spoken by the user or video footage of the user's appearance, and then determines the voice or video to be output based on the estimated user's age, so it is not necessarily effective in encouraging the use of digital signage that displays characters that can interact with users.

[0005] The present application aims to promote the use of digital signage that displays characters that can interact with users. [Means for solving the problem]

[0006] The information processing device of the present application comprises an acquisition unit that acquires a user video captured of a user positioned in front of a digital signage; a decision unit that estimates the user's situation based on analysis information regarding the analysis results of the user video and decides whether or not to have a character displayed on the digital signage speak to the user depending on the estimated situation of the user; a generation unit that generates a conversational message with content appropriate to the user's situation when the decision unit determines that the character will speak to the user; and an output control unit that controls the conversational message to be output by voice. [Effects of the Invention]

[0007] According to one aspect of the embodiment, it is possible to encourage the use of digital signage that displays characters that can interact with users. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of a digital signage according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating a character displayed on the digital signage according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of information processing according to the embodiment. [Figure 6] FIG. 6 is a flowchart showing a processing procedure performed by the information processing device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of information processing according to the modified example. [Figure 8] FIG. 8 is a diagram for explaining an example of information processing according to the modified example. [Figure 9]FIG. 9 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an information processing device, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to these embodiments. Furthermore, the same components in the following embodiments will be denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0010] (Embodiment) 1. Introduction Digital signage has been increasingly used to provide various kinds of information. For example, in recent years, digital signage has been introduced that displays characters and allows users to interact with them. By utilizing such digital signage, for example, at events, it is possible to introduce seminars and other events that suit users' preferences by interacting with them, and guide them.

[0011] Specifically, digital signage that allows users to interact with characters is equipped with a generative AI. The generative AI is, for example, a text generation model (a type of machine learning model) that is trained using data published on the Internet. For example, the generative AI may be a GPT (Generative Pre-trained Transformer) model that generates answers to questions. Such a model generates answers to questions from data published on the Internet, for example.

[0012] For example, let's consider the case where digital signage is installed at an event venue. In addition to data published on the Internet, digital signage also connects with external APIs (Application Programming Interfaces) and the like to obtain information about the event venue (for example, an image of a map of the event venue). Digital signage accepts voice input from users positioned in front of it. For example, digital signage accepts voice input of a conversation from a user such as "Where is XX's exhibition booth?" When the digital signage accepts voice input, it transcribes the conversation and generates a sentence such as "Where is XX's exhibition booth?" The digital signage also inputs the sentence "Where is XX's exhibition booth?" into a generation AI and generates a dialogue (hereinafter sometimes referred to as a conversation sentence) such as "This way! Here is where XXX is..." The digital signage also outputs a conversation generated by the AI, such as "This way! Here is XXX...", and displays an image of a map of the event venue on the digital signage screen.

[0013] However, few users willingly speak to digital signage that displays characters that can converse with the user. Specifically, users may have a mental barrier (also known as resistance) to speaking to a character themselves. In response to this, the information processing device according to this embodiment estimates the user's situation based on analysis information regarding the analysis results of a user video captured of the user positioned in front of the digital signage, and determines whether or not to have a character displayed on the digital signage speak to the user based on the estimated user's situation. If the information processing device determines that the character will speak to the user, it generates a conversational message appropriate to the user's situation and controls the generated conversational message to be output by voice.

[0014] This allows the information processing device to have the character displayed on the digital signage speak to the user, even if the user does not speak to the character themselves. This allows the information processing device to lower the mental barrier for the user when using digital signage that displays a character that can interact with the user. Therefore, the information processing device can encourage the use of digital signage that displays a character that can interact with the user.

[0015] [2. Information Processing System Configuration] Fig. 1 is a diagram showing an example of the configuration of an information processing system according to an embodiment. As shown in Fig. 1, the information processing system 1 includes a digital signage 100 and an information processing device 200. The digital signage 100 and the information processing device 200 are connected to each other via a predetermined communication network (network N) so as to be able to communicate with each other via a wired or wireless connection.

[0016] [3. Digital signage configuration] 2 is a diagram showing an example of the configuration of a digital signage according to an embodiment. The digital signage 100 according to the embodiment includes a communication unit 110, a storage unit 120, a control unit 130, an image capturing unit 140, an audio input unit 150, a display unit 160, and an audio output unit 170.

[0017] (Communication unit 110) The communication unit 110 is realized by a NIC (Network Interface Card) etc. Specifically, the communication unit 110 is connected to a network N (see FIG. 1) by wire or wirelessly, and transmits and receives information to and from the information processing device 200, for example.

[0018] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0019] (control unit 130) The control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs stored in a storage device inside the digital signage 100 using RAM as a work area. The control unit 130 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0020] (Photography section 140) The image capturing unit 140 is an image sensor (camera) that captures an image of a subject. For example, the image capturing unit 140 is realized by a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a CCD (Charge-Coupled Device) image sensor, or the like. Specifically, the image capturing unit 140 captures an image of a user positioned in front of the digital signage. The image capturing unit 140 also generates a user video image of the user positioned in front of the digital signage.

[0021] (Voice input unit 150) The audio input unit 150 converts an audio signal, which is a signal of a voice uttered by a user, into a digital signal and outputs the converted digital signal, which is an audio digital signal, as audio information to the control unit 130. The audio input unit 150 includes, for example, a microphone and an AD (Analog to Digital) converter that converts an audio signal, which is an electrical analog signal output from the microphone, into a digital signal. Specifically, the audio input unit 150 converts an audio signal, which is a signal of a voice uttered by a user positioned in front of the digital signage, into a digital signal and outputs the converted digital signal, which is an audio digital signal, to the control unit 130 as audio information.

[0022] (Display section 160) The display unit 160 is a display device that displays various types of information. The display unit 160 is, for example, a liquid crystal display, an organic EL (Electro Luminescence) display, or the like. The display unit 160 displays various types of information and provides it to the user. In the following description, the display unit 160 may also be referred to as a screen. Specifically, the display unit 160 displays images (moving images) of characters that can interact with the user under the control of the control unit 130.

[0023] (Audio output unit 170) The audio output unit 170 includes, for example, a DA (Digital to Analog) converter that converts a digital audio signal, which is audio information output from the control unit 130, into an analog audio signal, and a speaker that converts the analog audio signal output from the DA converter into sound and outputs it. Specifically, the audio output unit 170 outputs the voice of a character that can interact with the user under the control of the control unit 130.

[0024] FIG. 3 is a diagram illustrating a character displayed on a digital signage according to an embodiment. In FIG. 3, the digital signage 100 has a vertically elongated shape in which the vertical length is longer than the horizontal length. The display unit 160 of the digital signage 100 displays the upper chest portion of the character 10, which can interact with a user, large on the entire screen. The imaging unit 140 of the digital signage 100 is provided above the digital signage 100. The audio input unit 150 of the digital signage 100 is provided above the digital signage 100. The audio output unit 170 of the digital signage 100 is provided below the digital signage 100.

[0025] 3 illustrates a case where the image capturing unit 140, the audio input unit 150, and the audio output unit 170 are provided in the main body of the digital signage 100. However, the image capturing unit 140, the audio input unit 150, and the audio output unit 170 may be provided at a location away from the main body of the digital signage 100. For example, the image capturing unit 140, the audio input unit 150, or the audio output unit 170 may be provided at a location within a predetermined range from the main body of the digital signage 100. In this case, the image capturing unit 140, the audio input unit 150, or the audio output unit 170 may be connected to the main body of the digital signage 100 by wire or wirelessly, and may transmit and receive information to and from the digital signage 100.

[0026] 4. Configuration of Information Processing Device 4 is a diagram showing an example of the configuration of an information processing device according to the embodiment. The information processing device 200 according to the embodiment includes a communication unit 210, a storage unit 220, and a control unit 230.

[0027] (Communication unit 210) The communication unit 210 is realized by a NIC etc. Specifically, the communication unit 210 is connected to the network N (see FIG. 1) by wire or wirelessly, and transmits and receives information to and from the digital signage 100, for example.

[0028] (Storage unit 220) The storage unit 220 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 220 stores the information processing program according to the embodiment.

[0029] (control unit 230) The control unit 230 is a controller, and is realized, for example, by a CPU, an MPU, or the like, executing various programs (for example, the information processing program according to the embodiment) stored in a storage device inside the information processing device 200 using a RAM as a work area. The control unit 230 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC or an FPGA.

[0030] The control unit 230 has an acquisition unit 231, a determination unit 232, a generation unit 233, and an output control unit 234 as functional units, and may realize or execute the information processing actions described below. Note that the internal configuration of the control unit 230 is not limited to the configuration shown in FIG. 4, and may be any other configuration that performs the information processing described below. Furthermore, each functional unit indicates a function of the control unit 230, and does not necessarily have to be physically distinct.

[0031] (Acquisition part 231) The acquisition unit 231 acquires a user video captured of a user positioned in front of the digital signage 100. Specifically, the acquisition unit 231 acquires the user video from the digital signage 100. For example, the acquisition unit 231 acquires the user video captured by the imaging unit 140 of the digital signage 100 in real time. For example, the acquisition unit 231 acquires the user video at predetermined time intervals (for example, a time obtained by dividing one second by the frame rate of the user video).

[0032] (Decision unit 232) The determination unit 232 estimates the user's situation based on analysis information related to the analysis result of the user video, and determines whether to have a character displayed on the digital signage 100 address the user based on the estimated user's situation. Specifically, the determination unit 232 generates analysis information related to the analysis result of the user video from the user video. Here, the analysis information may include attribute information indicating the attributes of the user captured in the user video and time information indicating the time the user's face is facing the digital signage 100. For example, the attribute information may be information indicating the user's gender and age. The attribute information may also include information indicating that the user is using a cane, information indicating that the user is with a child, or information indicating that the user is in a wheelchair. The time information may also be information indicating the time the user is looking at the digital signage 100. More specifically, the determination unit 232 acquires a first machine learning model trained to output analysis information when the user video is input. Next, the determining unit 232 inputs the user video into the first machine learning model and acquires the analysis information output from the first machine learning model.

[0033] Next, the determination unit 232 estimates the user's situation based on the generated analysis information. For example, when the analysis information is input, the determination unit 232 acquires a second machine learning model trained to output situation information indicating the user's situation. For example, when the analysis information is input, the determination unit 232 acquires a second machine learning model trained to output situation information indicating a situation in which the user is feeling anxious, a situation in which the user is looking for something, a situation in which the user is looking at the digital signage 100, a situation in which the user has visited to make a visit reservation, or a situation in which the user is in a crowded place (such as an event venue). Next, the determination unit 232 inputs the generated analysis information to the second machine learning model and obtains the situation information output from the second machine learning model as an estimation result.

[0034] The determination unit 232 may estimate the user's facial expression based on the user video. For example, the determination unit 232 acquires an expression estimation model trained to estimate the user's facial expression from the user video. The determination unit 232 may input the user video into an expression estimation system. If the determination unit 232 estimates that the user's facial expression is an anxious expression, a crying expression, or an angry expression, the determination unit 232 may estimate that the user is feeling anxious. The determination unit 232 may also determine, based on the user video or the analysis information, whether the number of times the user's facial orientation has changed within a predetermined time is equal to or greater than a predetermined number. For example, if the determination unit 232 determines that the number of times the user's facial orientation has changed within a predetermined time is equal to or greater than a predetermined number, the determination unit 232 may estimate that the user is searching for something. The determination unit 232 may also determine, based on the user video or the analysis information, whether the time the user's face is facing the digital signage 100 is equal to or greater than a predetermined time. For example, when determining that the time period during which the user's face is facing the digital signage 100 is equal to or longer than a predetermined time period, the determining unit 232 may infer that the user is looking at the digital signage 100.

[0035] Next, the determination unit 232 determines whether or not a character (hereinafter, may be simply referred to as a "character") displayed on the digital signage 100 should speak to the user, depending on the estimated user's situation. For example, the determination unit 232 determines whether or not it is a situation where it is appropriate for the character to speak to the user, depending on the estimated user's situation. For example, the determination unit 232 acquires a first list, which is a list of information in which situation information indicating the user's situation is associated with information indicative of whether it is appropriate for the character to speak to the user. For example, the first list includes information in which situation information indicating a situation where the user is feeling anxious, a situation where the user is looking for something, a situation where the user is looking at the digital signage 100, a situation where the user has visited to make a reservation for a visit, or a situation where the user is in a crowded place (such as an event venue) is associated with information indicative of whether it is appropriate for the character to speak to the user. Next, the determination unit 232 determines whether or not it is appropriate for the character to speak to the user, referring to the acquired first list. For example, when the determination unit 232 refers to the first list and determines that the situation information indicating the estimated situation of the user is associated with information indicating that it is OK for the character to talk to the user, the determination unit 232 determines that the character will talk to the user. On the other hand, when the determination unit 232 refers to the first list and determines that the situation information indicating the estimated situation of the user is associated with information indicating that it is not OK for the character to talk to the user, the determination unit 232 determines that the character will not talk to the user.

[0036] Instead of acquiring the first list, the determination unit 232 may acquire a third machine learning model trained to output information indicating whether it is appropriate for a character to talk to the user when situation information indicating the user's situation is input. For example, when situation information indicating a situation in which the user is feeling anxious, a situation in which the user is looking for something, a situation in which the user is looking at the digital signage 100, a situation in which the user has visited to make a visit reservation, or a situation in which the user is in a crowded place (such as an event venue) is input, the determination unit 232 acquires a third machine learning model trained to output information indicating that it is not appropriate for a character to talk to the user. Next, the determination unit 232 inputs the situation information indicating the estimated user's situation into the third machine learning model and acquires information output from the third machine learning model indicating whether it is appropriate for a character to talk to the user. For example, when the determination unit 232 acquires information indicating that it is appropriate for a character to talk to the user, it determines that the character will talk to the user. On the other hand, when the determination unit 232 acquires information indicating that it is not a situation where it is appropriate for the character to speak to the user, the determination unit 232 determines that the character will not speak to the user.

[0037] Furthermore, when the determination unit 232 determines that the character will speak to the user, it generates a prompt instructing the character to generate a conversational message that corresponds to the user's situation. Specifically, the determination unit 232 acquires a second list, which is a list of situation template information in which situation information indicating the user's situation and prompt templates instructing the character to generate a conversational message that corresponds to the user's situation are in one-to-one correspondence. For example, the second list includes situation template information in which situation information indicating a situation in which the user is feeling anxious, a situation in which the user is searching for something, and a situation in which the user is looking at the digital signage 100 are in one-to-one correspondence with prompt template information such as, "You are a support center staff member. Please create a conversational message that you will use to speak to the user in order to appropriately support the user in the situation you are in." The second list also includes situation template information in which situation information indicating a situation in which the user has visited the company to make a visit appointment and prompt template information such as, "You are a receptionist who accepts visitors. Please create a conversational message that you will use to speak to the user in order to guide the user to the location related to the visit appointment are in one-to-one correspondence." The second list also includes situation template information in which situation information indicating the situation in which the user is at the event venue is in one-to-one correspondence with template information for a prompt having the content "You are an usher at the event venue. Please create a conversation that you would speak to the user to guide them to a relatively empty restroom." Next, the determination unit 232 generates a prompt that instructs the system to generate a conversation whose content is appropriate for the user's situation, based on the situation information indicating the estimated situation of the user and the acquired second list.

[0038] (Generation unit 233) When the determination unit 232 determines that the character will speak to the user, the generation unit 233 generates a conversational sentence with content that suits the user's situation. Specifically, the generation unit 233 generates a conversational sentence with content that suits the user's situation by inputting the prompt generated by the determination unit 232 into a generation AI. For example, the generation unit 233 acquires a generation AI that is a sentence generation model. Next, the generation unit 233 inputs the prompt generated by the determination unit 232 into the sentence generation model and acquires a conversational sentence output from the sentence generation model.

[0039] (output control unit 234) The output control unit 234 controls the digital signage 100 to output the conversational text generated by the generation unit 233 by voice. Specifically, the output control unit 234 transmits the conversational text generated by the generation unit 233 to the digital signage 100. The digital signage 100 receives the conversational text from the information processing device 200.

[0040] When the control unit 130 of the digital signage 100 receives a conversation from the information processing device 200, it may determine whether the received conversation includes information guiding a location. When the control unit 130 determines that the received conversation includes information guiding a location, it displays an image of a map of the corresponding location on the display unit 160. Furthermore, when the control unit 130 receives a conversation from the information processing device 200, it outputs the received conversation by voice from the audio output unit 170.

[0041] FIG. 5 is a diagram illustrating an example of information processing according to an embodiment. In FIG. 5, the determination unit 232 determines to speak to the user U1 when it estimates that the user U1 is feeling anxious. For example, the determination unit 232 refers to the first list and determines that situation information indicating the situation in which the user is feeling anxious is associated with information indicating that it is OK for the character to speak to the user. When the determination unit 232 determines that the situation information indicating the situation in which the user is feeling anxious is associated with information indicating that it is OK for the character to speak to the user, it determines that the character will speak to the user U1.

[0042] Also, in FIG. 5 , when the determination unit 232 determines that the character 10 will speak to the user U1, it generates a prompt that instructs the user to generate a conversational sentence with content corresponding to a situation in which the user is feeling anxious. For example, the determination unit 232 refers to the second list to acquire template information for a prompt corresponding to a situation in which the user is feeling anxious. For example, the determination unit 232 acquires template information for a prompt with content such as, "You are a support center staff member. Please create a conversational sentence with content that you will speak to the user in order to appropriately support the user in the situation of XX." Next, based on the acquired template information, the determination unit 232 replaces the part of the template information that says "XX situation" with the sentence "a situation in which you are feeling anxious," and generates a prompt with content such as, "You are a support center staff member. Please create a conversational sentence with content that you will speak to the user in order to appropriately support the user in the situation of anxiety." The generation unit 233 inputs the prompt generated by the determination unit 232, "You are a support center staff member. Please create a conversational sentence that you would speak to a user in order to appropriately support the user who is in a situation where he or she is feeling anxious," into the sentence generation model, and acquires a conversational sentence output from the sentence generation model. For example, the generation unit 233 acquires conversational sentence 2 output from the sentence generation model, "Is there something I can help you with? Shall I call a store clerk?"

[0043] [5. Processing Procedure] Fig. 6 is a flowchart showing a processing procedure by the information processing device according to the embodiment. In Fig. 6, the acquisition unit 231 of the information processing device 200 acquires a user video captured of a user positioned in front of the digital signage 100 (step S101). Furthermore, the determination unit 232 of the information processing device 200 generates analysis information related to the analysis result of the user video acquired by the acquisition unit 231 (step S102). Next, the determination unit 232 estimates the user's situation based on the generated analysis information, and determines whether or not to have a character displayed on the digital signage 100 speak to the user depending on the estimated user's situation (step S103).

[0044] If the determination unit 232 determines that the character displayed on the digital signage 100 will not speak to the user (step S103; No), the information processing device 200 ends the process. On the other hand, if the determination unit 232 determines that the character displayed on the digital signage 100 will speak to the user (step S103; Yes), the generation unit 233 of the information processing device 200 generates a conversation sentence according to the user's situation (step S104). Furthermore, the output control unit 234 of the information processing device 200 controls the conversation sentence generated by the generation unit 233 to be output by voice (step S105).

[0045] [6. Modifications] In the above-described embodiment, the information processing device 200 generates conversational text with content that matches the user's situation, but the present invention is not limited to this.

[0046] Fig. 7 is a diagram for explaining an example of information processing according to a modified example. In Fig. 7, when the determination unit 232 estimates that the situation is that the user U2 is looking at the digital signage 100, the determination unit 232 determines to speak to the user. For example, the determination unit 232 refers to the first list and determines that situation information indicating the situation in which the user is looking at the digital signage 100 is associated with information indicating that it is OK for the character to speak to the user. When the determination unit 232 determines that the situation information indicating the situation in which the user is looking at the digital signage 100 is associated with information indicating that it is OK for the character to speak to the user, the determination unit 232 determines that the character will speak to the user U2.

[0047] Furthermore, when the determination unit 232 determines that the character 10 should speak to the user U2, it generates a prompt instructing the user to generate a conversational sentence with content appropriate to the situation in which the user is viewing the digital signage 100. For example, the determination unit 232 refers to the second list to acquire template information for a prompt corresponding to the situation in which the user is viewing the digital signage 100. For example, the determination unit 232 acquires template information for a prompt with content such as, "You are a support center staff member. Please create a conversational sentence with content that you will speak to the user in order to appropriately support the user in the situation in which you are viewing the digital signage 100." Next, based on the acquired template information, the determination unit 232 replaces the part of the template information that says "the situation in which you are viewing the digital signage 100" with the sentence "the situation in which you are viewing the digital signage 100," and generates a prompt with content such as, "You are a support center staff member. Please create a conversational sentence with content that you will speak to the user in order to appropriately support the user in the situation in which you are viewing the digital signage 100."

[0048] 7, when the determination unit 232 determines that the character 10 should speak to the user U2, it generates a prompt instructing the determination unit 232 to generate a conversational message with content appropriate to the attributes of the user U2. For example, the determination unit 232 generates, as analysis information, information indicating that the user U2 is a person with a child. For example, based on the user video, the determination unit 232 may infer that the user U2 and the user U3 are parent and child because the user video includes the user U2, who is presumed to be an adult, and the user U3, who is presumed to be a child, and the users U2 and U3 are moving together. Specifically, when the determination unit 232 generates a prompt instructing the determination unit 232 to generate a conversational message with content appropriate to the user's situation, it generates a new prompt by adding the user's attribute information to the generated prompt. In FIG. 7, the determination unit 232 generates a new prompt by adding the sentence "In addition, the user is a person with a child" to the prompt with the content "You are a support center staff member. Please create a conversational message to speak to the user in order to appropriately support the user who is viewing the digital signage 100." The generation unit 233 also inputs the new prompt generated by the determination unit 232 into the generation AI, thereby generating a conversational sentence with content corresponding to the attributes of user U2. Specifically, the generation unit 233 inputs the prompt generated by the determination unit 232, "You are a support center staff member. Please create a conversational sentence to speak to the user in order to appropriately support the user who is viewing the digital signage 100. The user is with a child," into the sentence generation model, and acquires the conversational sentence output from the sentence generation model. For example, the generation unit 233 acquires conversational sentence 3 output from the sentence generation model with content "An event for children is being held. The event will be held here." The output control unit 234 also controls the speech output of conversational sentence 3 generated by the generation unit 233 and the display of a map image 31 indicating the location of the event.

[0049] FIG. 8 is a diagram illustrating an example of information processing according to a modified example. In FIG. 8, the acquisition unit 231 acquires reservation information regarding a reservation for a visit by user U4 in advance. For example, the reservation information includes information regarding the facial image of user U4, the name of user U4, the time of visit, and the place of visit. When the determination unit 232 estimates, based on the reservation information, that user U4 has visited to make a reservation for a visit, it determines to speak to the user. For example, the determination unit 232 determines whether or not there is a user in the user video whose face matches the facial image of user U4 included in the reservation information. When the determination unit 232 determines that there is a user in the user video whose face matches the facial image of user U4 included in the reservation information, it determines that user U4 is a visitor. In addition, the determination unit 232 determines whether or not a predetermined time (e.g., 10 minutes) has passed since the time of visit. When the determination unit 232 determines that the time is more than a predetermined time (e.g., 10 minutes) before the visit time, it presumes that the user U4 has visited to make a visit reservation. Furthermore, when the determination unit 232 determines to speak to the user U4, it refers to the second list to acquire template information of a prompt corresponding to the situation in which the user has visited to make a visit reservation. For example, the determination unit 232 refers to the second list to acquire template information of a prompt with the content, "You are a receptionist who accepts visitors to the company. Please create a conversational message that you will use to speak to the user U4 in order to guide the user to the visit location related to the visit reservation." Based on the acquired template information and the reservation information, the determination unit 232 generates a prompt that instructs the user to generate a conversational message that will guide the user U4 to the visit location related to the visit reservation. For example, the determination unit 232 generates a prompt with the content, "You are a receptionist who accepts visitors to the company. Please create a conversational message that you will use to speak to the user U4 in order to guide the user U4 to the visit location related to the visit reservation."

[0050] Also, in FIG. 8, the generation unit 233 generates a conversational sentence with content guiding user U4 to the visit location related to the visit reservation. Specifically, the generation unit 233 generates a conversational sentence with content guiding user U4 to the visit location related to the visit reservation by inputting the prompt generated by the determination unit 232 into the generation AI. Specifically, the generation unit 233 inputs the prompt generated by the determination unit 232, "You are a receptionist who accepts visitors to the company. Please create a conversational sentence with content that you will speak to user U4 to guide user U4 to the visit location related to the visit reservation," into the sentence generation model, and acquires the conversational sentence output from the sentence generation model. Note that the sentence generation model is assumed to have been trained based on reservation information. For example, the generation unit 233 acquires conversational sentence 4 with content "Mr. B, right? The meeting location will be here," output from the sentence generation model. Furthermore, the output control unit 234 controls the device so that the conversation sentence 4 generated by the generation unit 233 is output by voice, and also controls the device so that an image 41 of a map showing the location of the meeting is displayed.

[0051] Furthermore, when the determination unit 232 estimates that the user is in a situation where he or she is looking for something, it determines to speak to the user. For example, the determination unit 232 refers to the first list and determines that situation information indicating that the user is looking for something is associated with information indicating that it is OK for the character to speak to the user. When the determination unit 232 refers to the first list and determines that situation information indicating that the user is looking for something is associated with information indicating that it is OK for the character to speak to the user, it determines that it is OK for the character to speak to the user.

[0052] Furthermore, when the determination unit 232 determines that the character will speak to the user, it generates a prompt instructing the generation of a conversational sentence corresponding to a situation in which the user is looking for something. For example, the determination unit 232 refers to the second list to acquire template information for a prompt corresponding to a situation in which the user is looking for something. For example, the determination unit 232 acquires template information for a prompt with the content, "You are a support center staff member. Please create a conversational sentence with content to speak to the user in order to appropriately support the user in the situation in which you are looking for something." Next, based on the acquired template information, the determination unit 232 replaces the part "the situation in which you are looking for something" in the template information with the sentence "the situation in which you are looking for something," thereby generating a prompt with the content, "You are a support center staff member. Please create a conversational sentence with content to speak to the user in order to appropriately support the user in the situation in which you are looking for something." The generation unit 233 inputs the prompt with the content, "You are a support center staff member. Please create a conversational sentence with content to speak to the user in order to appropriately support the user in the situation in which you are looking for something," generated by the determination unit 232, into a sentence generation model and acquires a conversational sentence output from the sentence generation model.

[0053] Next, a case will be described in which the digital signage 100 is installed at an event venue (an example of a crowded location). The acquisition unit 231 acquires location video captured at a location other than the location in front of the digital signage 100. For example, the acquisition unit 231 may acquire, via the communication unit 210, video captured by a security camera installed within a predetermined range from the event venue. Alternatively, the acquisition unit 231 may acquire video captured at a security camera located within a predetermined range from the digital signage 100. The determination unit 232 estimates the congestion status at a location other than the location in front of the digital signage 100 based on other analysis information related to the analysis result of the location video, and generates a prompt instructing the user to generate a conversation based on the estimated congestion status. For example, the determination unit 232 may estimate the congestion status at each of a plurality of different locations based on analysis information related to location video captured at each of the plurality of different locations. Alternatively, the determination unit 232 may estimate a relatively less crowded location from among the plurality of different locations based on the congestion status at each of the plurality of different locations. The determination unit 232 may also generate a prompt instructing the user to generate a conversational sentence to guide the user to a relatively uncrowded location. For example, the determination unit 232 generates a prompt instructing the user to generate a conversational sentence to guide the user to a location other than the front of the digital signage 100 based on other analysis information related to the analysis result of the location image. For example, the determination unit 232 identifies a relatively uncrowded restroom among the restrooms installed at the event venue based on the location image. The determination unit 232 also acquires template information for a prompt with the content, "You are an event venue attendant. Please create a conversational sentence to speak to the user to guide the user to a relatively uncrowded restroom." Next, based on the information indicating the location of a relatively uncrowded restroom among the restrooms installed at the event venue and the acquired template information, the determination unit 232 generates a prompt with the content, "You are an event venue attendant. Please create a conversational sentence to speak to the user to guide the user to a relatively uncrowded restroom. The relatively uncrowded restroom is located at □□."The generation unit 233 inputs the prompt generated by the determination unit 232 into the generation AI, thereby generating a conversational sentence that guides the user to a location different from the front of the digital signage 100. For example, the generation unit 233 generates a conversational sentence that guides the user to a relatively empty restroom among the restrooms installed at the event venue.

[0054] Furthermore, the determination unit 232 determines whether the content of the conversational text generated by the generation unit 233 is reliable. For example, the determination unit 232 determines whether the conversational text generated by the generation unit 233 contains a keyword predetermined for each user's situation. If the determination unit 232 determines that the predetermined keyword is contained, the determination unit 232 determines that the content of the conversational text generated by the generation unit 233 is reliable. On the other hand, if the determination unit 232 determines that the predetermined keyword is not contained, the determination unit 232 determines that the content of the conversational text generated by the generation unit 233 is not reliable. If the determination unit 232 determines that the content of the conversational text is not reliable, the determination unit 232 generates a prompt instructing the generation AI to generate a conversational text containing the predetermined keyword. The generation unit 233 generates a conversational text containing the predetermined keyword by inputting the prompt instructing the generation AI to generate a conversational text containing the predetermined keyword.

[0055] Furthermore, the determination unit 232 determines whether the conversational text generated by the generation unit 233 is appropriate using a known grammar proofreading tool. The determination unit 232 inputs the conversational text generated by the generation unit 233 into a known grammar proofreading tool and obtains the check results output from the grammar proofreading tool. If the check results output from the grammar proofreading tool do not include a check result indicating a problem, the determination unit 232 determines that the content of the conversational text generated by the generation unit 233 is reliable. On the other hand, if the check results output from the grammar proofreading tool include a check result indicating a problem, the determination unit 232 determines that the content of the conversational text generated by the generation unit 233 is unreliable. If the determination unit 232 determines that the content of the conversational text is unreliable, it generates a prompt instructing the user to correct the conversational text based on the check result by the grammar proofreading tool indicating a problem. The generation unit 233 generates a corrected conversational text that reflects the check result by the grammar proofreading tool indicating a problem by inputting the prompt to the generation AI instructing the user to correct the conversational text based on the check result by the grammar proofreading tool indicating a problem.

[0056] Furthermore, the determination unit 232 generates a prompt that instructs the generation AI to check for itself whether or not the conversational sentence generated by the generation unit 233 is appropriate. The generation unit 233 determines whether or not the conversational sentence generated by the generation unit 233 is appropriate by inputting the prompt that instructs the generation AI to check for itself whether or not the conversational sentence generated by the generation unit 233 is appropriate to the generation AI.

[0057] [7. Effects] As described above, the information processing device 200 according to the embodiment includes an acquisition unit 231, a determination unit 232, a generation unit 233, and an output control unit 234. The acquisition unit 231 acquires a user video captured of a user positioned in front of the digital signage 100. The determination unit 232 estimates the user's situation based on analysis information related to the analysis result of the user's video, and determines whether or not to have a character displayed on the digital signage 100 speak to the user depending on the estimated user's situation. When the determination unit 232 determines that a character will speak to the user, the generation unit 233 generates a conversational message with content appropriate to the user's situation. The output control unit 234 controls the conversational message to be output by voice.

[0058] As a result, the information processing device 200 can make the character displayed on the digital signage 100 speak to the user, even if the user does not speak to the character himself. As a result, the information processing device 200 can lower the mental barrier for the user when using the digital signage 100 that displays a character that can converse with the user. Therefore, the information processing device 200 can encourage the use of the digital signage 100 that displays a character that can converse with the user. Furthermore, since the information processing device 200 can encourage the use of the digital signage 100 that displays a character that can converse with the user, it can contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation."

[0059] Furthermore, when the determination unit 232 determines that the character should speak to the user, it generates a prompt instructing the generation of a conversational sentence whose content corresponds to the user's situation. The generation unit 233 inputs the prompt generated by the determination unit 232 to the generation AI, thereby generating a conversational sentence whose content corresponds to the user's situation.

[0060] This allows the information processing device 200 to generate appropriate conversational text according to the user's situation, making it easier for the user to respond to the conversation when spoken to by the character. Therefore, the information processing device 200 can encourage the use of the digital signage 100 that displays characters that can converse with the user.

[0061] Furthermore, when the determination unit 232 determines that the character will speak to the user, it generates a prompt instructing the generation of a conversational sentence having content according to the attributes of the user. The generation unit 233 generates a conversational sentence having content according to the attributes of the user by inputting the prompt generated by the determination unit 232 to the generation AI.

[0062] This allows the information processing device 200 to generate appropriate conversation sentences according to the attributes of the user, making it easier for the user spoken to by the character to respond to the conversation. Therefore, the information processing device 200 can encourage the use of the digital signage 100 that displays characters that can converse with the user.

[0063] Furthermore, when the decision unit 232 estimates that the user is in a situation where he or she feels anxious, the decision unit 232 decides to talk to the user.

[0064] Here, a situation in which the user feels anxious can be said to be a situation in which the user needs some kind of support. Furthermore, a situation in which the user needs some kind of support can be said to be an appropriate timing for the character displayed on digital signage 100 to speak to the user. Therefore, information processing device 200 can cause the character displayed on digital signage 100 to speak to the user at an appropriate timing.

[0065] Furthermore, when the determining unit 232 estimates that the user is searching for something, it determines to speak to the user.

[0066] Here, a situation in which the user is searching for something can be said to be a situation in which the user needs some kind of support. Furthermore, a situation in which the user needs some kind of support can be said to be an appropriate timing for the character displayed on digital signage 100 to speak to the user. Therefore, information processing device 200 can cause the character displayed on digital signage 100 to speak to the user at an appropriate timing.

[0067] Furthermore, when it is estimated that the user is looking at the digital signage 100, the determining unit 232 determines to speak to the user.

[0068] Here, a situation in which a user is looking at digital signage 100 is a situation in which the user is likely to be interested in a character displayed on digital signage 100, and therefore it can be said to be an appropriate timing for the character displayed on digital signage 100 to speak to the user. Therefore, information processing device 200 can cause the character displayed on digital signage 100 to speak to the user at an appropriate timing for the character displayed on digital signage 100 to speak to the user.

[0069] The acquisition unit 231 acquires reservation information related to a reservation for a visit of the user in advance. The determination unit 232 determines to speak to the user when it estimates based on the reservation information that the user has visited to make a reservation for a visit. The generation unit 233 generates a conversational sentence with content to guide the user to the visit location related to the reservation for the visit.

[0070] Here, a situation in which a user has visited to make a reservation for a visit can be, for example, a situation in which the user needs detailed information about the place to be visited. Also, for example, a situation in which the user needs detailed information about the place to be visited can be, for example, an appropriate timing for the character displayed on the digital signage 100 to speak to the user. Therefore, the information processing device 200 can cause the character displayed on the digital signage 100 to speak to the user at an appropriate timing for the character displayed on the digital signage 100 to speak to the user. Also, for example, in a situation in which the user needs detailed information about the place to be visited, the information processing device 200 can make it possible to guide the user to the place to be visited related to the reservation for the visit.

[0071] Furthermore, the acquisition unit 231 acquires location video captured at a location different from the location in front of the digital signage 100. The determination unit 232 generates a prompt that instructs the generation of a conversational sentence with content that guides the user to a location different from the location in front of the digital signage 100, based on other analysis information related to the analysis result of the location video. The generation unit 233 inputs the prompt generated by the determination unit 232 into a generation AI, thereby generating a conversational sentence with content that guides the user to a location different from the location in front of the digital signage 100.

[0072] This enables the information processing device 200 to guide users to the location of a relatively empty toilet from among multiple toilets installed at an event venue or other location where congestion of users is expected, for example, at an event venue.

[0073] Furthermore, the determination unit 232 determines whether the content of the conversational text generated by the generation unit 233 is reliable or not.

[0074] This allows the information processing device 200 to generate conversational text with reliable content, thereby increasing the reliability of the dialogue between the user and the character. Therefore, the information processing device 200 can encourage the use of the digital signage 100 that displays characters that can converse with the user.

[0075] [8. Hardware Configuration] The information processing device 200 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in Fig. 9. Fig. 9 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0076] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0077] The HDD 1400 stores programs executed by the CPU 1100, data used by these programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends the data to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0078] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0079] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1200. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0080] For example, when the computer 1000 functions as the information processing device 200 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 230. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0081] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that include the embodiments described in the Disclosure of the Invention section and that have undergone various modifications and improvements based on the knowledge of those skilled in the art.

[0082] [9. Other] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0083] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0084] 1 shows a case where the digital signage 100 and the information processing device 200 are separate devices, but the digital signage 100 and the information processing device 200 may be integrated. When the digital signage 100 and the information processing device 200 are integrated into one device, the information processing device 200 has the functions of the digital signage 100.

[0085] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content. [Explanation of symbols]

[0086] 1. Information Processing Systems 100 Digital Signage 200 Information processing device 210 Communications Department 220 Storage section 230 Control Unit 231 Acquisition Department 232 Decision Section 233 Generation part 234 Output control section

Claims

1. an acquisition unit that acquires a user video image of a user to be processed that is positioned in front of the digital signage; a decision unit that estimates the status of the user to be processed based on analysis information relating to the analysis result of the user video, the analysis information including time information indicating the time the face of the user is facing the direction of the digital signage, and determines whether or not to have a character displayed on the digital signage speak to the user to be processed according to the estimated status of the user to be processed; a generation unit that terminates processing when the determination unit determines that the character will not speak to the user, and that generates conversational text with content appropriate to the situation of the user to be processed when the determination unit determines that the character will speak to the user to be processed; an output control unit that controls the conversation sentence to be output by voice; Equipped with The determination unit acquires situation template information in which situation information indicating a user's situation is in one-to-one correspondence with a prompt template that instructs the user to generate a conversational sentence according to the user's situation, and generates a prompt that instructs the user to generate a conversational sentence having content that is according to the user's situation based on the information indicating the estimated situation of the user to be processed and the acquired situation template information; The generation unit generating a conversational sentence having a content according to the situation of the user to be processed based on the prompt generated by the determination unit; Information processing device.

2. The determination unit When it is determined that the character should speak to the user to be processed, a prompt is generated to instruct the character to generate the conversational text according to the situation of the user to be processed; The generation unit The prompt generated by the determination unit is input to a generation AI to generate a conversational sentence having a content corresponding to the situation of the user to be processed. The information processing device according to claim 1 .

3. The determination unit When it is determined that the character should speak to the user to be processed, a prompt is generated to instruct the user to generate the conversation sentence having content corresponding to the attributes of the user to be processed; The generation unit The prompt generated by the determination unit is input to the generation AI to generate the conversational sentence having content according to the attributes of the user to be processed. The information processing device according to claim 2 .

4. The determination unit If it is estimated that the user to be processed is in a situation where he or she feels anxious, it is decided to talk to the user to be processed. The information processing device according to claim 1 .

5. The determination unit When it is estimated that the user to be processed is searching for something, it is determined to talk to the user to be processed. The information processing device according to claim 1 .

6. The determination unit When it is estimated that the user to be processed is looking at the digital signage, it is determined to speak to the user to be processed. The information processing device according to claim 1 .

7. The acquisition unit Reservation information regarding the reservation of the visit of the user to be processed is acquired in advance; The determination unit If it is estimated based on the reservation information that the user to be processed has visited to make a reservation for the visit, it is determined to talk to the user to be processed; The generation unit generating the conversational text with content to guide the target user to a visit location related to the visit reservation; The information processing device according to claim 1 .

8. The acquisition unit Acquire location video of a location different from the location in front of the digital signage; The determination unit generating the prompt to instruct the system to generate the conversational sentence that guides the target user to a location other than the front of the digital signage based on other analysis information related to the analysis result of the location image; The generation unit The prompt generated by the determination unit is input to the generation AI to generate the conversation sentence having content that guides the target user to a location different from the front of the digital signage. The information processing device according to claim 2 .

9. The determination unit determining whether the content of the conversational text generated by the generation unit is reliable; The information processing device according to claim 1 .

10. An information processing method realized by a program executed by an information processing device, an acquisition step of acquiring a user video of a user to be processed who is positioned in front of the digital signage; a decision process for estimating the status of the user to be processed based on analysis information relating to the analysis results of the user video, the analysis information including time information indicating the time the face of the user is facing the direction of the digital signage, and determining whether or not to have a character displayed on the digital signage speak to the user to be processed depending on the estimated status of the user to be processed; a generation step of terminating the processing when it is determined in the determination step that the character will not speak to the user, and generating a conversational sentence with content appropriate to the situation of the user to be processed when it is determined in the determination step that the character will speak to the user to be processed; an output control step of controlling the conversation to be output by voice; Including, The determining step acquires situation template information in which situation information indicating a user's situation is in one-to-one correspondence with a prompt template that instructs the user to generate a conversational sentence according to the user's situation, and generates a prompt that instructs the user to generate a conversational sentence having content that is according to the user's situation based on the information indicating the estimated situation of the user to be processed and the acquired situation template information; The generating step includes: generating a conversational sentence having a content appropriate to the situation of the user to be processed based on the prompt generated in the determining step; Information processing methods.

11. an acquisition step of acquiring a user video of a user to be processed who is positioned in front of the digital signage; a decision procedure for estimating the status of the user to be processed based on analysis information relating to the analysis results of the user video, the analysis information including time information indicating the time the user's face is facing the direction of the digital signage, and determining whether or not to have a character displayed on the digital signage speak to the user to be processed depending on the estimated status of the user to be processed; a generation step of terminating the process when it is determined by the decision step that the character will not speak to the user, and generating a conversational sentence with a content appropriate to the situation of the user to be processed when it is determined by the decision step that the character will speak to the user to be processed; an output control procedure for controlling the conversation sentence to be output by voice; on the computer, The determination procedure includes: acquires situation template information in which situation information indicating a user's situation is in one-to-one correspondence with a prompt template that instructs the user to generate a conversational sentence according to the user's situation, and generates a prompt that instructs the user to generate a conversational sentence having content that is according to the user's situation based on the information indicating the estimated situation of the user to be processed and the acquired situation template information; The generating procedure includes: generating a conversational sentence having a content appropriate to the situation of the user to be processed based on the prompt generated by the determination procedure; Information processing program.

Images