Control system, control method, and recording medium

Abstract

Provided is a control system comprising an output unit, an acquisition unit, and a determination unit. The output unit outputs, to a display device of a user of a service in a virtual reality space, an avatar and actions of the avatar of a provider that provides the service to the user. The acquisition unit acquires an action history of the user who is using the service. The determination unit determines the avatar and / or the actions of the avatar of the provider of the service in the virtual reality space on the basis of the action history of the user. The output unit switches the avatar and / or the actions of the avatar of the provider of the service to an avatar and / or actions of the determined provider, and outputs the same.

Description

Control system, control method, and recording medium

[0001] The present disclosure relates to control systems and the like.

[0002] It is known that services such as customer service and lectures can be provided in a virtual reality space. It is also known that a user's avatar can be set in the virtual reality space. For example, Patent Literature 1 describes creating an avatar based on the user's characteristics such as height.

[0003] Furthermore, for example, Patent Document 2 describes a system in which a customer selects a virtual store clerk to converse with before starting shopping, and the selected virtual store clerk serves the customer while shopping.

[0004] JP 2022-117770 A JP 2020-160641 A

[0005] Generally, when a user uses various services, the user's behavior may change depending on the provider. In such cases, it may be difficult for the user to use the service in a virtual reality space.

[0006] An example of an objective of the present disclosure is to provide a control system or the like that enables service users to easily use services in virtual reality spaces.

[0007] A control system in one aspect of the present disclosure includes an output means for outputting an avatar and the avatar's behavior of a provider of a service to a user of the service in a virtual reality space to the user's display device, an acquisition means for acquiring a behavioral history of the user while using the service, and a determination means for determining at least one of the avatar and the avatar's behavior of the provider of the service in the virtual reality space based on the behavioral history of the user, and the output means switches at least one of the avatar and behavior of the provider of the service to at least one of the determined avatar and behavior of the provider and outputs the avatar and behavior of the provider.

[0008] A control method in one aspect of the present disclosure includes a computer executing a process in which an avatar and the avatar's behavior of a provider of a service that provides the service to a user of the service in a virtual reality space are output to the user's display device, a behavioral history of the user while using the service is acquired, and based on the user's behavioral history, a determination is made of at least one of the avatar and the avatar's behavior of the provider of the service in the virtual reality space, and in the output process, at least one of the avatar and the behavior of the provider of the service is switched to at least one of the determined avatar and the behavior of the provider and output.

[0009] A program in one aspect of the present disclosure causes a computer to execute a process to output an avatar and the avatar's behavior of a provider of a service to a user of the service in a virtual reality space to the user's display device, obtain a behavioral history of the user while using the service, and determine at least one of the avatar and the avatar's behavior of the provider of the service in the virtual reality space based on the user's behavioral history, and in the output process, switch at least one of the avatar and behavior of the provider of the service to at least one of the determined avatar and behavior of the provider and output them.

[0010] Each program may be stored in a non-transitory computer-readable recording medium.

[0011] According to the present disclosure, it is possible to make it easier for service users to use services in virtual reality spaces.

[0012] 1 is a first block diagram showing a configuration example of a control system according to the present disclosure. FIG. 2 is a flowchart showing an operation example of the control system. FIG. 3 is an explanatory diagram showing an example of connection between the control system and another device. FIG. 4 is an explanatory diagram showing example 1 of a display device. FIG. 5 is an explanatory diagram showing example 2 of a display device. FIG. 6 is a second block diagram showing a configuration example of a control system according to the present disclosure. FIG. 7 is an explanatory diagram showing an example of determining a new avatar using a learning model. FIG. 8 is an explanatory diagram (part 1) showing an example of a screen for accepting instructions. FIG. 9 is an explanatory diagram (part 2) showing an example of accepting instructions. FIG. 10 is an explanatory diagram showing an example of output of the determined number of patients. FIG. 11 is a flowchart showing an operation example of a control system that switches at least one of a user's avatar and behavior. FIG. 12 is a flowchart showing an operation example of a control system that switches at least one of a provider's avatar and behavior. FIG. 13 is an explanatory diagram showing an example of the hardware configuration of a computer.

[0013] Hereinafter, with reference to the drawings, embodiments of a control system, a control method, a program, and a non-transitory recording medium for recording a program according to the present disclosure will be described in detail. The disclosed technology is not limited to these embodiments.

[0014] First Embodiment First, in the first embodiment, an example of basic functions of a control system will be described. Fig. 1 is a first block diagram showing an example configuration of a control system according to the present disclosure. A control system 10 includes an output unit 101, an acquisition unit 102, and a determination unit 103.

[0015] First, the output unit 101 outputs an avatar of a service provider providing a service to a service user in a virtual reality (VR) space and the avatar's behavior to the user's display device. The service may be, but is not limited to, customer service, practice, lessons, lectures, counseling, etc. The customer service may be, but is not limited to, customer service during shopping when a product is sold.

[0016] The avatar here includes at least one of face, clothing, gender, hairstyle, accessories, and mood. The mood of the avatar may be facial expression, age-related mood, etc. The behavior of the avatar includes at least one of the avatar's gestures and hand movements, the avatar's speaking speed, the avatar's voice, the avatar's voice volume, and the avatar's athletic movements. The athletic movements may be movements during sports practice, etc.

[0017] For example, if the service is customer service, the user is the customer and the provider is the store clerk. For example, if the service is practice, lessons, etc., the user is the student and the provider is the teacher. For example, if the service is counseling, the user is the person being consulted and the provider is the counselor. For example, if the service is counseling by a doctor, the user is the patient and the counselor is the doctor.

[0018] The acquisition unit 102 acquires the behavioral history of the user while using the service. The behavioral history of the user while using the service may include voice information of the user in conversation with the provider. The voice information may be the degree to which the user listens during the conversation with the provider. Furthermore, for example, the voice information may be the volume of the voice. Furthermore, the behavioral history of the user while using the service may include information on images that the user actually saw in the virtual reality space. Furthermore, the behavioral history may include biometric information of the user. The biometric information may include, for example, heart rate information, body temperature information, blood pressure information, sweat information, and information about the eyes.

[0019] Here, the user's behavior history is acquired via a device such as a wearable device attached to the actual user, an imaging device capable of capturing an image of the user, or a sound collecting device capable of collecting sound, etc. For example, biological information can be acquired via a wearable device, an imaging device, etc. For example, body temperature information, eye information, sweat information, etc. may be identified from an image captured by an imaging device. Heart rate information, body temperature information, blood pressure information, and sweat information may be acquired by a wearable device. Sound information can be acquired via a sound collecting device, etc. Note that if the display device is a head-mounted display (HMD), the behavior history may be acquired via the HMD worn by the user. For example, if the display device is an HMD, the acquisition unit 102 can acquire information about the user's eyes, etc. as the behavior history. Here, the information about the eyes may include gaze information indicating the user's line of sight, information about whether the user's eyes are open or closed, information about the user's blinking, etc.

[0020] The determination unit 103 determines at least one of an avatar of a service provider and an action of the avatar in the VR space based on the action history of the user using the service. Here, the service provider for which the determination unit 103 determines the avatar or action may be a provider different from the provider currently providing the service to the user, or may be the provider currently providing the service to the user.

[0021] The method by which the determining unit 103 determines the avatar of the service provider and the behavior of the avatar may use a table prepared in advance or a trained model prepared in advance.

[0022] First, an example will be described in which a table is used. For example, a table in which user behavioral histories and provider avatars are associated in advance may be prepared. The determining unit 103 may determine, from the table, an avatar associated with the user behavioral history acquired by the acquiring unit 102 as the provider avatar. Also, for example, a table in which user behavioral histories and behaviors of the provider avatar are associated in advance may be prepared. The determining unit 103 may determine, from the table, the behavior of the avatar associated with the user behavioral history acquired by the acquiring unit 102 as the provider avatar.

[0023] Next, an example of using a trained model will be described. The model is, for example, a model in which the relationship between a user's behavior history and a provider's avatar and the avatar's behavior is learned based on the past behavioral histories of multiple users and the provider's avatar and the avatar's behavior when each of the multiple users uses a service. The model may also be, for example, a model in which the relationship between a user's behavior history and a provider's avatar and changes in the avatar's behavior is learned based on the past behavioral histories of multiple users and the provider's avatar and the avatar's behavior when each of the multiple users uses a service. For example, the determination unit 103 inputs the user's behavioral history acquired by the acquisition unit 102 into the pre-trained model to determine at least one of the service provider's avatar and the behavior.

[0024] Next, more specifically, for example, a guideline by which the determining unit 103 determines at least one of the avatar of the service provider in the VR space and the behavior of the avatar based on the behavior history will be described.

[0025] First, an example will be described using the user's listening level obtained from audio information. For example, when the user's listening level obtained from audio information is high, it is highly likely that the user is highly interested in the conversation with the provider. On the other hand, when the user's listening level is low, it is highly likely that the user is not highly interested in the conversation with the provider. Therefore, for example, the determination unit 103 may determine whether to change the provider's avatar and the avatar's behavior based on the user's listening level. Note that the determination of whether the listening level is high or low may use a threshold such as a predetermined level or a predetermined range, and is not particularly limited. For example, when the user's listening level is high, the determination unit 103 may determine not to change the provider's avatar and the avatar's behavior. For example, when the user's listening level is low, the determination unit 103 may determine at least one of the provider's avatar and the avatar's behavior. For example, when the listening level is low, the determination unit 103 may determine the avatar to be a different provider. This makes it possible to determine, for example, the change in the listening level when the avatar is changed. For example, if the service is customer service, the determination unit 103 may determine the behavior of the provider's avatar to change the content of the product explanation or the recommended product when the user's level of attentive listening is low.For example, if the service is counseling, the determination unit 103 may determine the behavior of the provider's avatar to change the questions asked when the user's level of attentive listening is low.

[0026] Further, an example of words spoken by a user will be described. For example, the determination unit 103 may determine the provider's avatar and the avatar's behavior based on the user's voice information, depending on whether the user frequently utters words registered in a negative word list. For example, if the user frequently utters words registered in a negative word list, the determination unit 103 may determine the provider's avatar's behavior so that the provider's avatar utters more words registered in a positive word list. The negative word list is, for example, a list of words with negative meanings. The positive word list is, for example, a list of words with positive meanings. For example, if the user frequently utters words registered in the negative word list, the determination unit 103 may determine the provider's avatar's behavior so that the provider's avatar utters more words that empathize with the user. Taking the example of using a trained model as described above, the determination unit 103 inputs words registered in the negative word list spoken by the user into the trained model, and thereby acquires the provider's avatar and the avatar's behavior from the trained model. For example, the model is a model that learns the relationship between words uttered by users and the provider's avatar and the avatar's behavior based on words registered in a negative word list that have been uttered by each of multiple users in the past and the provider's avatar and the avatar's behavior when each of multiple users uses the service.The model may also be a model that learns the relationship between words uttered by users and changes in the provider's avatar and the avatar's behavior.

[0027] Furthermore, for example, the determination unit 103 may determine the provider's avatar and the avatar's behavior based on the user's voice information, depending on whether the user frequently utters words registered in the positive word list.

[0028] Furthermore, for example, if the user repeatedly speaks the same noun keyword, it is possible that the user is highly interested in the conversation with the provider. Therefore, for example, the determination unit 103 may determine whether or not to change the provider's avatar and the avatar's behavior based on whether the user has uttered the same word a predetermined number of times or more. For example, the determination unit 103 may determine not to change the provider's avatar and the avatar's behavior if the user has uttered the same word a predetermined number of times or more. Alternatively, for example, the determination unit 103 may determine to change the provider's avatar and the avatar's behavior if the user has not uttered the same word a predetermined number of times or more.

[0029] Furthermore, for example, the determination unit 103 may estimate the degree of positivity or negativity of the user toward the conversation with the provider based on the user's voice information when the user repeatedly speaks keywords other than nouns. Then, the determination unit 103 may determine whether or not to change the provider's avatar and the avatar's behavior based on at least one of the positivity and negativity. Taking the positivity degree as an example, the determination unit 103 may determine not to change the provider's avatar and the avatar's behavior when the positivity degree is high. Alternatively, for example, the determination unit 103 may determine to change the provider's avatar and the avatar's behavior when the positivity degree is low.

[0030] Furthermore, if a user asks multiple questions, the user may be highly interested in the conversation with the provider. Specifically, for example, if the service is customer service and the user asks multiple questions about a product being explained by the provider, the user may be highly interested in the product. Therefore, the determination unit 103 may determine whether to change the provider's avatar and the avatar's behavior based on the number of questions the user has asked obtained from the user's voice information. For example, the determination unit 103 may determine not to change the provider's avatar and the avatar's behavior if the user has asked a predetermined number of questions or more. In particular, for example, the determination unit 103 may determine not to change the provider's avatar and the avatar's behavior if the user has asked a predetermined number of questions about a product or more.

[0031] For example, the determining unit 103 may determine the provider's avatar and the avatar's behavior based on the user's tone of voice identified by the user's voice information. For example, the user's tone of voice can be used to estimate emotions, a level of tension, a level of fatigue, and the like.

[0032] An overbearing attitude or an intimidating attitude can also be estimated as a type of user emotion. Here, a case where the user has an overbearing attitude will be taken as an example. For example, if the user has an overbearing attitude, the determination unit 103 may determine an avatar of the provider whose age is higher than the age of the currently displayed avatar. For example, if the user has an overbearing attitude, the determination unit 103 may determine an avatar of the provider whose age is higher than at least one of the user's age and the age of the user's avatar.

[0033] Next, an example of the user's level of nervousness estimated from the user's tone of voice will be described. For example, a high level of nervousness may indicate that the user finds it difficult to converse with the provider. Therefore, when the user's level of nervousness is high, the determination unit 103 determines an avatar and an avatar behavior that will reduce the user's level of nervousness. Specifically, for example, when the user's level of nervousness is high, the determination unit 103 may change the gender of the provider's avatar from the gender of the avatar of the provider currently providing the service. Furthermore, for example, when the user's level of nervousness is high, the determination unit 103 may lower the age of the provider's avatar from the age of the avatar of the provider currently providing the service. For example, when the user's level of nervousness is high, the determination unit 103 may change the age of the provider's avatar to be closer to the user's age. The user's age may be the user's age registered in advance, or may be an age estimated from the user's avatar or behavioral history.

[0034] Next, an example of the user's fatigue level estimated from the user's tone of voice will be described. For example, when the user's fatigue level is high, the user may want to use the service for a short period of time. Therefore, the determination unit 103 may determine the behavior of the provider's avatar so that, when the user's fatigue level is high, the provider's utterances, such as explanations, become shorter. Alternatively, when the user's fatigue level is low, the determination unit 103 may determine the behavior of the provider's avatar so that the provider's utterances, such as explanations, become longer. If the service is customer service, the provider's explanation may be, for example, an explanation of a product.

[0035] Furthermore, for example, the determination unit 103 may determine the behavior of the provider's avatar based on the user's speech information so that the provider's avatar will ask more questions to the user if the user is silent. For example, the determination unit 103 may determine the behavior of the provider's avatar so that the provider's avatar will speak more if the user is speaking for a long time. For example, whether the user is speaking for a long or short time or whether the user is silent may be determined by comparing the user's speech time with a threshold. For example, whether the user is speaking for a long or short time or whether the user is silent may be determined by relative comparison between the provider's speech time and the user's speech time.

[0036] The above has been described with reference to examples of the degree of listening, emotions, tension, fatigue, interest, etc. that can be estimated from speech information included in the behavioral history. The degree of listening, emotions, tension, fatigue, interest, etc. can also be estimated from biological information such as at least one of heart rate information, body temperature information, blood pressure information, sweat information, and eye information included in the behavioral history.

[0037] For example, the determination unit 103 may estimate the emotion, level of tension, and level of fatigue based on at least one of the user's blood pressure and blood pressure change represented by the blood pressure information. For example, if blood pressure suddenly rises, it is estimated that the user is irritated or has a high level of tension. For example, if there is no change in blood pressure, it is estimated that the user's level of tension is low. For example, if the level of sweating represented by the sweating information is high, it is estimated that the user is irritated or has a high level of tension. For example, if the level of sweating is low, it is estimated that the user's level of tension is low.

[0038] Furthermore, for example, the determination unit 103 may estimate emotions, a level of tension, or a level of fatigue based on information about the eyes. More specifically, for example, the determination unit 103 may estimate, for example, interest in a conversation with a provider based on information about the eyes. If the user looks in a direction unrelated to the provider, it is possible that the user is not interested in a conversation with the provider. Furthermore, for example, if the service is customer service, the user's interest in a product may be estimated based on the user's gaze. Furthermore, for example, the determination unit 103 may estimate, based on information about the eyes, that the user has a high level of tension toward the provider if the user does not look at the provider. Whether or not the user is looking at the provider may be estimated based on the direction of the user's gaze and the duration for which the gaze is directed. Alternatively, whether or not the user is looking at the provider may be estimated based on whether the user's eyes are open.

[0039] Note that examples in which the determination unit 103 determines at least one of the avatar and the avatar's behavior, or determines whether or not to change it, based on estimated emotions, levels of tension, fatigue, and interest obtained from biometric information may be the same as examples in which audio information is used, and therefore detailed explanations will be omitted.

[0040] The output unit 101 switches at least one of the avatar and behavior of the service provider to at least one of the avatar and behavior of the determined provider and outputs the switched avatar and behavior. Note that if the service provider for which the determination unit 103 determines at least one of the avatar and behavior is different from the provider currently providing the service to the user, the output unit 101 may output the avatar of the determined provider additionally while keeping the avatar of the current service provider displayed.

[0041] 2 is a flowchart showing an example of the operation of the control system 10. The output unit 101 outputs the provider's avatar and the avatar's behavior (step S101). The acquisition unit 102 acquires the behavior history of the user using the service (step S102).

[0042] The determination unit 103 determines at least one of the avatar of the service provider and the avatar's behavior based on the behavior history (step S103). The output unit 101 switches to and outputs at least one of the determined avatar of the service provider and the behavior (step S104). Note that after step S104, the control system 10 may return to step S102 and repeat steps S102 to S104, or the control system 10 may end the process after step S104.

[0043] As described above, in the first embodiment, the control system 10 outputs the avatar and the avatar's behavior to the service user in the VR space, and determines at least one of the avatar and the avatar's behavior in the VR space of the service provider based on the behavior history of the user while using the service. Then, the control system 10 switches at least one of the avatar and the behavior of the service provider to at least one of the determined avatar and the behavior of the provider and outputs the avatar and the behavior of the service provider.

[0044] To make it easier for service users to use the service. For example, to make it easier for users to take actions that are in line with the intent of the service provided by the service provider. Ultimately, to make it easier for the provider to provide the service.

[0045] (Embodiment 2) Next, embodiment 2 will be described in detail with reference to the drawings. In embodiment 2, an example of determining a provider's avatar and the avatar's behavior will be described in more detail, using examples of product sales in a store and counseling by a doctor as services. In embodiment 2, an example of switching the output will be described in more detail. In addition, in embodiment 2, an example of the timing at which the output is switched will be described. Below, to the extent that the description of embodiment 2 is not unclear, explanations of content that overlaps with the above description will be omitted.

[0046] 3 is an explanatory diagram showing an example of a connection between a control system and other devices. For example, the control system 20 is connected to a display device 21 via a communication network. The display device 21 is, for example, a user's display device or a provider's display device. The number of display devices 21 may be determined depending on the number of actual users and providers, and is not particularly limited.

[0047] FIG. 4 is an explanatory diagram showing a first example of the display device 21. In FIG. 4, the display device 21 is an HMD. The HMD, which is the display device 21, may have various functions for acquiring behavioral history and the like. For example, the HMD may have functions for measuring the user's nodding, recording audio, tracking the user's eye movements such as gaze and blinking, and measuring biological information such as body temperature. Although the user has been used as an example in the description, the same applies to the provider.

[0048] FIG. 5 is an explanatory diagram showing a second example of the display device 21. In FIG. 5, the display device 21 may be a dome-shaped display. When a dome-shaped display is used as the display device 21, the control system 20 may display an image of an actual user captured in real time. For this reason, when displaying an avatar, the control system 20 performs preprocessing to composite an avatar image onto an image of the actual user. The control system 20 then displays the avatar so that it follows the movements of the actual user. When the actual user raises their hand, the control system 20 generates and displays an avatar image as if the avatar is raising their hand. The user's movements refer to the movements of the user's body, such as the face, head, and hands. When switching avatars for the same user, the control system 20 simply switches the avatar image to be composited onto the image of the actual user. Note that while the user has been used as an example, the same applies to the provider.

[0049] 6 is a second block diagram showing a configuration example of a control system 20 according to the present disclosure. The control system 20 includes an output unit 201, an acquisition unit 202, a determination unit 203, a reception unit 204, and a learning unit 205.

[0050] The output unit 201 may have, as a basic function, the function of the output unit 101 shown in Fig. 1. The acquisition unit 202 may have, as a basic function, the function of the acquisition unit 102 shown in Fig. 1. The determination unit 203 may have, as a basic function, the function of the determination unit 103 shown in Fig. 1.

[0051] Here, the user's avatar and the provider's avatar may be determined both at the start of using the service and during the use of the service.

[0052] <At the Start of Service Usage> First, the acquisition unit 202 acquires the user's behavioral history. The user's behavioral history here may be a behavioral history from past service usage. The user's behavioral history from past service usage may include voice information of the user in conversations with the provider, information on images the user actually saw in the virtual reality space, and biometric information of the user. Furthermore, the user's behavioral history at the start of service usage may include, for example, past purchase history and information representing purchasing tendencies. The purchase history may include, for example, information on products purchased by the user, purchase frequency, purchase amount, purchase date and time, etc. In addition to the user's behavioral history, the acquisition unit 202 may also acquire attribute information representing the user's attributes. The user's attributes are not limited to the user's age, gender, occupation, address, etc. Furthermore, the acquisition unit 202 may acquire the user's past service usage history. For example, the service usage history may include information such as past avatars, number of uses, usage frequency, provider identification information identifying providers who have previously provided services to the user, etc., and the service usage history is not particularly limited.

[0053] Here, various information about the user, such as the user's behavior history and attribute information, may be stored in advance as a user database in association with user identification information that identifies the user. Then, for example, upon receiving the user identification information, the acquisition unit 202 acquires the various information about the user associated with the user identification information from the user database.

[0054] Furthermore, the acquisition unit 202 acquires the provider's behavioral history when providing a service in the past. The provider's behavioral history may include the provider's voice information, the provider's biometric information, and the provider's image information. The acquisition unit 202 is not limited to the provider's behavioral history, but may also acquire attribute information representing the provider's attributes and the provider's provision history. The attribute information is not particularly limited to the provider's age, gender, work experience, etc. The provision history may include information such as past avatars and user identification information identifying users to whom the service was provided in the past, and the service provision history is not particularly limited. The provision history may also be referred to as, for example, a business history.

[0055] Here, various information about the provider, such as the behavioral history and attribute information of the provider, may be stored in advance in a provider database in association with provider identification information that identifies the provider. Then, for example, upon receiving the provider identification information, the acquiring unit 202 acquires the various information about the provider associated with the provider identification information from the provider database.

[0056] <Determining at least one of the provider avatar and behavior at the start of the service> The determination unit 203 determines the provider avatar and the behavior of the provider avatar at the start of the service based on the user's behavior history. The method for determining the provider avatar may be, for example, the same as the method for determining the provider avatar during the provision of the service described in embodiment 1. For example, the determination unit 203 may use a table or a model to determine the provider avatar and the behavior of the provider avatar. For example, when using a model, the learning unit 205 generates the model in advance as a preparation. The learning unit 205 generates a model that learns the relationship between the user's behavior history and the provider avatar and the behavior of the avatar based on the past behavior history of multiple users and the provider avatar and the behavior of the avatar when each of the multiple users uses the service. The determination unit 203 may then input the user's behavior history into the learned model and acquire the provider avatar and the behavior of the avatar from the model.

[0057] Furthermore, the determination unit 203 may determine at least one of the provider avatar and the behavior of the provider avatar at the start of the service based on the user's behavior history and the user's attribute information. More specifically, for example, the determination unit 203 may determine at least one of an avatar close to the user's age and the behavior of an avatar close to the user's age. More specifically, for example, the determination unit 203 may determine at least one of an avatar of the same gender as the user and the behavior of an avatar of the same gender as the user. More specifically, for example, the determination unit 203 may determine the behavior of the avatar inferred from the user's occupation. For example, the determination unit 203 may determine the speech style of the provider avatar according to the personality inferred from the user's occupation. Note that a table or a model may be used as a method for determining the provider avatar and the behavior of the avatar based on the user's behavior history and the user's attribute information. The model may be generated in advance by the learning unit 205.

[0058] Here, when there are multiple users, the determination unit 203 determines, for each of the multiple users, at least one of the provider avatars and the avatar's behavior that are visible to each of the multiple users, based on the user's behavior history. Taking user A and user B as an example, the determination unit 203 determines at least one of the provider avatars and the avatar's behavior that are visible to user A, based on user A's behavior history. Then, the determination unit 203 determines at least one of the provider avatars and the avatar's behavior that are visible to user B, based on user B's behavior history. The provider avatars and the avatar's behavior that are visible to user A and the provider avatars and the avatar's behavior that are visible to user B may be the same or different. Furthermore, for example, the determination unit 203 may determine the behavior of each user so that the abilities of user A and user B are balanced.

[0059] Then, the output unit 201 outputs the avatar of the service provider and the avatar's behavior to the display device 21 of the service user in the VR space.

[0060] Although the example in which the determining unit 203 determines at least one of the provider's avatar and behavior using a past behavior history has been described, for example, the output unit 201 may output a predetermined avatar and the avatar's behavior. Note that, for example, when a past behavior history or the like cannot be obtained, the output unit 201 may output a predetermined avatar and the avatar's behavior to the display device 21 of the service user.

[0061] <Determining at least one of the user's avatar and behavior at the start of service> Furthermore, the determination unit 203 may determine the user's avatar and the behavior of the user's avatar to be displayed on the provider's display device 21 based on the user's behavior history. For example, the determination unit 203 may determine at least one of the user's avatar and the avatar's behavior according to the user's behavior history. For example, the user's avatar according to the user's behavior history may be, for example, a user's avatar that represents the user's personality estimated from the user's behavior history. The behavior of the user's avatar according to the user's behavior history may be, for example, a user's avatar that represents the user's personality estimated from the user's behavior history. This allows the provider to grasp the user's personality at a glance.

[0062] Furthermore, the determination unit 203 may determine the user's avatar and the behavior of the user's avatar to be displayed on the provider's display device 21, for example, based on the user's behavior history and the user's attribute information. For example, the user's avatar according to the user's attribute information may be, for example, a user's avatar that represents the user's attributes. For example, the determination unit 203 may determine a user's avatar that is close to the user's age.

[0063] Furthermore, the determination unit 203 may determine the avatar of the user to be displayed on the provider's display device 21 based on the user's past service usage history. For example, the determination unit 203 may display the same avatar of the user that was previously displayed on the provider's display device 21. This makes it easier for the provider to identify the user.

[0064] Then, the output unit 201 outputs the avatar of the service provider and the avatar's behavior to the display device 21 of the service provider in the VR space.

[0065] Although the example in which the determining unit 203 determines at least one of the provider's avatar and behavior using a past behavior history has been described, for example, the output unit 201 may output a predetermined avatar and the avatar's behavior. Note that, for example, when a past behavior history or the like cannot be obtained, the output unit 201 may output a predetermined avatar and the avatar's behavior to the display device 21 of the service user.

[0066] <Example of processing at the start of service use when there are multiple users> Here, when there are multiple users, the determination unit 203 determines at least one of the avatars and actions that each of the multiple users will be able to see based on the behavioral history of each of the multiple users.

[0067] Taking user A and user B as an example, when user A and user B interact, the determination unit 203 determines at least one of the avatar of user B that user A can see in the VR space and the behavior of the avatar, based on the behavior history of user A. For example, the determination unit 203 determines at least one of the avatar of user B that user A can see in the VR space and the behavior of the avatar, based on the behavior history of user B. Then, the determination unit 203 determines at least one of the avatar of user A that user B can see in the VR space and the behavior of the avatar, based on the behavior history of user B. For example, the determination unit 203 determines at least one of the avatar of user A that user B can see in the VR space and the behavior of the avatar, based on the personality and the status of user B. This makes it easier for users A and B to talk to each other, and facilitates interaction.

[0068] Then, the output unit 201 outputs at least one of the determined avatar and behavior of the other user to the display device 21 of the user.

[0069] <During Service Use> Next, an example will be described in which at least one of the provider's avatar and the avatar's behavior is determined during service use. Note that detailed description of the content described in the first embodiment will be omitted.

[0070] Next, as described in the first embodiment, the acquiring unit 202 acquires the behavioral history of the user who is using the service. The behavioral history of the user may include voice information of the user, biometric information of the user, image information of the user, and the like.

[0071] The acquiring unit 202 also acquires the behavioral history of the provider while the service is being provided. The behavioral history of the service provider may include voice information, biometric information, and image information of the provider. The behavioral history of the provider while the service is being provided is acquired via a device such as a wearable device attached to the actual provider, an imaging device capable of capturing an image of the provider, or a sound collecting device capable of collecting sound, etc. For example, biometric information can be acquired via a wearable device, an imaging device, etc. Voice information can be acquired via a sound collecting device, etc. If the display device is an HMD (Head Mounted Display), the behavioral history may be acquired via the HMD worn by the provider. For example, if the display device is an HMD, the acquiring unit 102 can acquire gaze information, etc. of the provider as the behavioral history.

[0072] Then, as described in the first embodiment, for example, the determination unit 203 determines at least one of an avatar of a service provider and the behavior of the avatar in the VR space based on the behavior history of the user who is using the service. As described in the first embodiment, the service provider whose avatar or behavior the determination unit 203 determines may be a provider different from the provider currently providing the service to the user, or may be the provider currently providing the service to the user. The method by which the determination unit 203 determines at least one of an avatar of a service provider and the behavior of the avatar is as described in the first embodiment. For example, as described in the first embodiment, when the determination unit 203 determines the avatar of the provider, a table or a model may be used.

[0073] For example, when a model is used, the learning unit 205 generates the model in advance as a preliminary preparation. Specifically, for example, the learning unit 205 generates a model that learns the relationship between the user's behavior history and the provider's avatar and the avatar's behavior based on the past behavior history of multiple users and the provider's avatar and the avatar's behavior when each of the multiple users uses a service. Furthermore, for example, the learning unit 205 may generate a model that learns the relationship between the user's behavior history and the avatar and the avatar's behavior changes based on the past behavior history of multiple customers and the store clerk's avatar and the avatar's behavior changes while providing a service to the user.

[0074] For example, the determination unit 203 may input the current customer's behavior history and the current store clerk's avatar and avatar's behavior into the model learned by the learning unit 205 to determine a new avatar and avatar's behavior.

[0075] 7 is an explanatory diagram showing an example of determining a new avatar using a learning model. In FIG. 7, the example shows a case where the service is customer service, the user is a customer, and the provider is a store clerk. For example, the output unit 201 displays a store clerk avatar in a VR space on the customer's display device 21.

[0076] For example, the acquisition unit 202 acquires the customer's behavioral history during the provision of the service. Then, for example, the determination unit 203 inputs the customer's behavioral history into the learned model to determine at least one of a store clerk avatar and the avatar's behavior. In FIG. 7 , the determination unit 203 determines an avatar that is the same as the currently displayed store clerk but has different clothing and hairstyle from the currently displayed store clerk avatar. For example, the output unit 201 switches at least one of the avatar and behavior of the store clerk providing the service to at least one of the determined avatar and behavior of the store clerk and outputs the switched avatar and behavior.

[0077] Here, the timing at which the output unit 201 switches is not particularly limited. For example, the output unit 201 may switch at the timing of a new conversation, or may switch gradually. Furthermore, for example, the output unit 201 may switch in response to an instruction from a service provider or an instruction from a service user.

[0078] First, an example of switching in response to an instruction from a service provider will be described. For example, the reception unit 204 receives an instruction to switch avatars from the provider. The reception unit 204 receives the switching instruction through an operation by the provider via an input device. The type of input device is not particularly limited, and may be a controller, a pen tablet, or the like. For example, upon receiving the switching instruction, the output unit 201 switches at least one of the provider's avatar and behavior to at least one of the determined provider's avatar and behavior, and outputs the switched avatar and behavior.

[0079] Next, an example of switching in response to an instruction from a service user will be described. For example, the reception unit 204 receives an instruction to switch avatars from a user. The reception unit 204 receives the switching instruction through an operation by a provider via an input device. For example, upon receiving a switching instruction from a user, the output unit 201 switches at least one of the provider's avatar and behavior to at least one of the determined provider's avatar and behavior, and outputs the switched avatar and behavior.

[0080] To receive the instruction, the output unit 201 may output the currently displayed provider's avatar and avatar's behavior and the determined avatar and avatar's behavior to the device of the instructor who issues the switching instruction. As described above, the instructor may be the provider or the user. The instructor's device may be, for example, the instructor's display device 21 or another terminal device, and is not particularly limited.

[0081] FIG. 8 is an explanatory diagram (part 1) showing an example screen for receiving instructions. In FIG. 8 , the output unit 201 displays the currently displayed provider avatar and an avatar with the provider avatar and clothing changed side by side on the screen. In FIG. 8 , for example, the screen further includes a change button. When the change button is pressed by an instruction user, the receiving unit 204 receives a switch instruction. Then, when the receiving unit 204 receives the switch instruction, the output unit 201 switches at least one of the provider avatar and behavior to at least one of the determined provider avatar and behavior, and outputs the result.

[0082] 8, for example, the screen may include a "Do Not Change" button. When the "Do Not Change" button is pressed by the instruction user, the receiving unit 204 receives the instruction not to switch. Then, the output unit 201 outputs the current avatar and behavior of the provider.

[0083] FIG. 9 is an explanatory diagram (part 2) showing an example of receiving an instruction. In FIG. 9, the output unit 201 displays a currently displayed provider avatar and an avatar of a different gender from the provider avatar side by side on the screen. In FIG. 9, for example, the screen further includes a change button. When the change button is pressed by an instruction user, the receiving unit 204 receives a switch instruction. Then, when the receiving unit 204 receives the switch instruction, the output unit 201 switches at least one of the provider avatar and action to at least one of the determined provider avatar and action, and outputs the switched avatar and action.

[0084] 9, for example, the screen may include a "Do Not Change" button. When the presenter presses the "Do Not Change" button, the accepting unit 204 accepts an instruction not to switch. Then, the output unit 201 outputs the current provider's avatar and behavior as they are.

[0085] Furthermore, when receiving an instruction not to switch, the determining unit 203 may newly determine at least one of another avatar and the action of the avatar.

[0086] For example, if the provider of a service for which the determination unit 203 determines at least one of an avatar and an action is different from the provider currently providing the service to the user, the output unit 201 may output the avatar of the additionally determined provider while continuing to output the avatar of the current service provider, or may output the avatar of the newly determined provider without outputting the avatar of the current service provider. That is, the output unit 201 may output at least one of the newly determined avatar and an action so as to add a provider, or may output at least one of the newly determined avatar and an action so as to replace the provider. Such output switching may also be performed in response to an instruction from the operator.

[0087] Furthermore, for example, the determination unit 203 may determine at least one of the provider's avatar and the behavior based on the behavior history of the provider and the behavior history of the user. Specifically, for example, the determination unit 203 may determine at least one of the provider's avatar and the behavior so that the provider's state matches the user's state. For example, if the provider is not nervous but the user is nervous, it is assumed that the provider is easy to talk to but the user is difficult to talk to.

[0088] Therefore, the determination unit 203 may determine at least one of the provider's avatar and behavior that will reduce the user's level of tension. As an example of changing the avatar to reduce the level of tension, the determination unit 203 may determine to change the age and gender of the provider's avatar from the age and gender of the provider's avatar currently displayed on the user's display device 21. As an example of changing the avatar's behavior to reduce the level of tension, the determination unit 203 may determine to change the voice quality of the provider's avatar to a gentler or softer voice quality than the voice quality of the provider's avatar currently displayed on the user's display device 21. This makes it easier for the user and the provider to converse with each other.

[0089] The output unit 201 switches at least one of the provider's avatar and behavior to at least one of the determined provider's avatar and behavior and outputs it.

[0090] <Changing At Least Either the User's Avatar or Behavior> Furthermore, for example, the determination unit 203 may determine at least one of the user's avatar and the user's avatar's behavior to be displayed on the provider's display device 21 based on the user's behavior history. Similar to the example at the start of use, for example, the determination unit 203 may determine at least one of the user's avatar and the avatar's behavior according to the user's current behavior history. For example, the user's avatar according to the user's behavior history may be, for example, a user's avatar representing the user's personality estimated from the user's behavior history. The user's avatar's behavior according to the user's behavior history may be, for example, a user's avatar representing the user's personality or state estimated from the user's behavior history. This allows the provider to grasp the user's personality or state at a glance. For example, if it is estimated that the user is currently nervous based on the pulse rate included in the biometric information, the determination unit 203 determines an avatar that indicates nervousness.

[0091] The output unit 201 switches at least one of the currently displayed user avatar and behavior to at least one of the determined user avatar and behavior, and outputs them to the provider's display device 21. Here, the timing of the switching by the output unit 201 is not particularly limited. For example, the output unit 201 may switch at the timing of a new conversation, or may switch gradually. Furthermore, for example, the output unit 201 may switch in response to an instruction from the service provider or an instruction from the service user. An example in which the receiving unit 204 receives a switching instruction may be similar to the example in which the provider's avatar and behavior are switched, and therefore a detailed description thereof will be omitted.

[0092] The output unit 201 may also output the state of the user together with the avatar. For example, if the state of the user is a level of tension, the output unit 201 may output the level of tension together with the avatar.

[0093] <Determining at least one of the user's avatar and behavior> For example, if the provider is nervous during a conversation with the user, the provider's avatar may be changed to make it easier for the provider to converse. Therefore, for example, the determination unit 203 determines at least one of the user's avatar and behavior to be displayed on the provider's display device 21 based on the provider's behavior history. For example, the determination unit 203 may determine at least one of the user's avatar and behavior according to the provider's state estimated from the provider's behavior history. For example, the determination unit 203 may use a table or a model to determine the user's avatar and behavior, similar to the determination unit 203's method of determining the provider's avatar and behavior.

[0094] The output unit 201 switches at least one of the user's avatar and behavior currently displayed on the provider's display device 21 to at least one of the determined user's avatar and behavior, and outputs the switched avatar and behavior. Here, the timing of the switching by the output unit 201 is not particularly limited. For example, the output unit 201 may switch at the timing of a new conversation, or may switch gradually. Furthermore, for example, the output unit 201 may switch in response to an instruction from the service provider or an instruction from the service user. Note that an example in which the receiving unit 204 receives a switching instruction may be similar to the example in which the provider's avatar and behavior are switched, and therefore a detailed description thereof will be omitted.

[0095] In the examples described above, the behavior of the provider's avatar in the VR space may be that of a bot.

[0096] Furthermore, for example, the output unit 201 may output a plurality of behaviors of the provider's avatar, which is a bot, in the VR space, and the receiving unit 204 may receive a selection of an action from the plurality of behaviors of the provider's avatar, which is a bot, through an operation by an actual instructor.

[0097] <Determining Sound Effects> The determination unit 203 may further determine a sound effect. Specifically, the determination unit 203 may determine a sound effect based on the user's behavior history. For example, the determination unit 203 may determine a sound effect according to the user's state estimated from the user's biometric information. For example, the determination unit 203 may determine a sound effect according to the user's personality estimated from the user's behavior history. The sound effect may be music, etc. Alternatively, if the service is customer service, the sound effect may be, for example, the voice of another shopper.

[0098] Then, the output unit 201 outputs the determined sound effect from the audio output device provided in the display device 21 .

[0099] <Example of when the service is customer service> When the service is customer service, the user is a customer and the provider is a store clerk. When the service is customer service, it is desirable for the store clerk's avatar and behavior to increase the customer's desire to purchase.

[0100] For example, the user's past behavioral history may further include the user's purchase history, and the model may be a model that learns the relationship between the user's behavioral history and the provider's avatar and the avatar's behavior based on the past behavioral history of multiple users and the provider's avatar and the avatar's behavior when each of the multiple users uses the service.The model may be a model that learns the relationship between the user's behavioral history and the provider's avatar and changes in the avatar's behavior.For example, it is sufficient to generate a model that obtains the provider's avatar and the avatar's behavior that increases purchasing motivation.

[0101] For example, examples of changes to the clerk avatar may include the clerk's hairstyle, clothing, or mood. For example, if a customer prefers a luxurious atmosphere, the determination unit 203 determines the clerk's hairstyle, clothing, and mood according to the preference. Note that the customer's preferences may be estimated from the customer's speech or state.

[0102] For example, some customers may be motivated to purchase by watching other customers shopping. For example, the customer's past behavioral history may include a past purchase history. The customer's past behavioral history may include information representing the customer's purchasing scenes before and after purchasing a product in the past purchase history. The information representing the purchasing scenes may be the results of video analysis or memo information entered by a store clerk. The acquisition unit 202 may acquire the user's past behavioral history. The determination unit 203 may determine a purchasing scene by another user based on the purchasing scene of a past product purchase in the purchase history included in the behavioral history. For example, if a user purchases a product after seeing another user purchase the same product in a past purchase, the determination unit 203 may determine a purchasing scene in which the other user purchased the product currently recommended to the customer by the store clerk. The other user may be a customer with attributes similar to the customer's. The output unit 201 then outputs the determined purchasing scene to the customer's display device 21. For example, the output unit 201 may display the determined scene as a background.

[0103] Here, an example of displaying a scene has been described using customer service as an example, but the present invention may be applied to other examples of services.

[0104] Furthermore, for example, in customer service work, a store clerk may perform accounting tasks such as operating a cash register. In such cases, when there are many other users, it is desirable to shorten the time spent serving each user. Therefore, the determination unit 203 may determine the behavior of the service provider based on the number of users of the service. For example, the determination unit 203 may determine the behavior of the provider so that, when there are many users of the service, the utterances explaining the service are shortened. In other words, the determination unit 203 may determine the behavior of the provider so that, when there are many users of the service, the time spent serving each user is shortened. Note that the number of users may be determined based on a threshold or the number of store clerks. Furthermore, the behavior of the provider may be determined based on the environment in which the service is currently being provided, in addition to the number of users of the service.

[0105] <Example of a case where the service is medical counseling> When the service is medical counseling, the user is a patient and the provider is a doctor. In such a case, it is desirable for the doctor's avatar and behavior to make it easy for the patient to talk to the doctor.

[0106] In the case of medical counseling, the actual patient may be alone, or the VR space may be set up to resemble a group counseling environment. Alternatively, the avatars of actual patients may be mixed with avatars of bot patients. Also, the avatars of actual doctors may be mixed with avatars of bot doctors.

[0107] Therefore, the determination unit 203 may determine the number of other patients that the patient can see based on the behavioral history of the patient using the service. For example, the determination unit 203 may determine the number of other patients that the patient can see based on the patient's personality and condition estimated from the patient's behavioral history. The determination unit 203 may use a table or a model to determine the number of other patients. The model may be learned in advance by, for example, the learning unit 205. For example, for a patient who becomes more comfortable talking to other patients by listening to what other patients are saying, increasing the number of patients can make conversation easier. The output unit 201 then outputs the avatars and behaviors of the determined number of other patients. The output unit 201 outputs the avatars and behaviors of the determined number of other patients to the user's display device 21. Furthermore, the output unit 201 outputs the avatars and behaviors of the determined number of other patients to the provider's display device 21.

[0108] FIG. 10 is an explanatory diagram showing an example of outputting the determined number of patients. For example, in reality, there are patient X, patient Y, and a doctor. In the VR space, the avatars of patient X, patient Y, and the doctor are displayed. If the determination unit 203 determines that there are four patients, the output unit 201 outputs the avatars of the other two patients to the display devices 21 of the respective patients because there are actually two patients. Then, the output unit 201 can output the behavior of the avatars of the two patients who are not actually present using a bot.

[0109] Although the example is given for when the service is being used, the determination unit 203 may determine the number of other patients that the patient can see based on the patient's past behavior history before starting to use the service. Then, the output unit 201 may output the avatars and behaviors of the determined number of other patients.

[0110] Here, an example in which other virtual users are displayed has been described using medical counseling as an example, but the present invention may be applied to other examples of services.

[0111] 11 is a flowchart showing an example of operation of the control system 20 for switching at least one of a user's avatar and behavior. The acquisition unit 202 acquires the user's past behavior history (step S201). The determination unit 203 determines the provider's avatar and the avatar's behavior based on the user's past behavior history (step S202). The output unit 201 outputs the provider's avatar and the avatar's behavior to the user's display device 21 (step S203).

[0112] When the use of the service is started, the acquisition unit 202 acquires the behavior history of the user who is using the service (step S204). Note that step S204 may be performed, for example, after a predetermined time has elapsed since the start of the service, or may be performed at a timing based on the content of the conversation.

[0113] The determination unit 203 determines at least one of the provider's avatar and the avatar's behavior based on the user's behavior history (step S205). The reception unit 204 determines whether a switching instruction has been received (step S206). The person issuing the switching instruction may be the provider or the user. If a switching instruction has been received (step S206: Yes), the output unit 201 switches to and outputs at least one of the determined provider's avatar and behavior (step S207), and the control system 20 ends the processing. If a switching instruction has not been received (step S206: No), the control system 20 ends the processing. Note that the control system 20 returns to step S204 after step S207 or step S206: No, and may repeat this process during service use.

[0114] 12 is a flowchart showing an example of the operation of the control system 20 for switching at least one of the provider's avatar and behavior. In FIG. 12, an example is described in which the determination unit 203 determines at least one of the user's avatar and the avatar's behavior based on the user's behavior history.

[0115] The acquisition unit 202 acquires the user's past behavior history (step S211). The determination unit 203 determines the user's avatar and the avatar's behavior based on the user's past behavior history (step S212). The output unit 201 outputs the provider's avatar and the avatar's behavior to the provider's display device 21 (step S213).

[0116] When the use of the service is started, the acquisition unit 202 acquires the behavior history of the user who is using the service (step S214). Note that step S214 may be performed, for example, after a predetermined time has elapsed since the start of the service, or may be performed at a timing based on the content of the conversation.

[0117] The determination unit 203 determines at least one of the user's avatar and the avatar's behavior based on the user's behavior history (step S215). The reception unit 204 determines whether a switching instruction has been received (step S216). If a switching instruction has been received (step S216: Yes), the output unit 201 switches to at least one of the determined provider's avatar and behavior and outputs them (step S217), and the control system 20 ends the processing. If a switching instruction has not been received (step S216: No), the control system 20 ends the processing. Note that after step S217 or step S216: No, the control system 20 returns to step S214, and this may be repeated during service use.

[0118] The operation example shown in FIG. 11 and the operation example shown in FIG. 12 may be performed in one flow.

[0119] As described above, in the second embodiment, when there are multiple users, the control system 20 determines at least one of the avatars and actions that each of the multiple users will see based on the behavioral history of each of the multiple users while using the service, and outputs at least one of the determined user avatars and actions for each of the multiple users. This allows the avatars that the users see to be avatars that are appropriate for the state of the users while using the service. For example, this can make it easier for users to converse and reduce their level of tension. This therefore makes it easier for users of the service to use the service.

[0120] Furthermore, when there are multiple users, the control system 20 determines at least one of the avatars and the avatar's behavior of the service provider that each of the multiple users will see based on the behavioral history of each of the multiple users, and switches to and outputs at least one of the determined provider's avatars and behaviors for each of the multiple users. This allows at least one of the provider's avatars and behaviors to be output for each user in the VR space, even when there are multiple users. This makes it easier for each user to use the service, even when multiple users use the service.

[0121] Furthermore, before the service starts, the control system 20 determines an avatar of the service provider based on the user's past behavior history and outputs the determined avatar of the provider. As a result, before the service starts, an avatar of the provider according to the user's past behavior history is displayed on the user's display device 21. This makes it easier for the user of the service to use the service.

[0122] Furthermore, the control system 20 outputs the user's avatar and behavior to the provider's display device 21. Then, the control system 20 determines at least one of the user's avatar and behavior based on the user's behavior history, and switches at least one of the user's avatar and behavior to the determined at least one of the user's avatar and behavior and outputs them. As a result, during the provision of the service, at least one of the user's avatar and behavior appropriate to the user's behavior history is displayed on the provider's display device 21. This makes it easier for the provider to provide the service.

[0123] Furthermore, the control system 20 outputs the user's avatar and behavior in the VR space to the provider's display device 21. The control system 20 determines at least one of the user's avatar and behavior based on the provider's behavior history while providing the service. Then, the control system 20 switches at least one of the user's avatar and behavior to the determined at least one of the user's avatar and behavior, and outputs it to the provider's display device 21. As a result, while the service is being provided, at least one of the user's avatar and behavior that is appropriate for the provider's behavior history is displayed on the provider's display device 21. This makes it easier for the provider to provide the service.

[0124] Furthermore, the control system 20 determines the provider's avatar and behavior based on the provider's behavior history and the user's behavior history while the service is being provided, and switches at least one of the provider's avatar and behavior to the determined provider's avatar and behavior and outputs them. As a result, at least one of the provider's avatar and behavior according to the provider's behavior history and the user's behavior history is displayed on the user's display device 21 while the service is being provided. This makes it easier for the provider to provide the service.

[0125] As a method for determining the provider's avatar and behavior, the control system 20 may determine at least one of an avatar and a behavior from a predetermined plurality of avatars and a predetermined plurality of behaviors. The control system 20 may input the acquired behavioral history of users into a learned model to determine at least one of the service provider's avatar and a behavior. When a learned model is used, the control system 20 generates a model that learns the relationship between the user's behavioral history and the provider's avatar and the avatar's behavior based on the past behavioral history of the multiple users and the provider's avatar and the avatar's behavior when each of the multiple users uses the service.

[0126] The control system 20 determines at least one of an avatar and an avatar behavior of a provider different from the provider currently providing the service to the user. This makes it possible to newly assign to the user another provider that makes it easier for the user to use the service while the service is being provided. For example, the control system 20 determines at least one of an avatar and an avatar behavior of a provider currently providing the service to the user. This makes it easier for the user to use the service without changing the provider currently providing the service to the user.

[0127] Furthermore, the control system 20 may perform various switching operations in response to instructions from the provider. Alternatively, the control system 20 may perform various switching operations in response to instructions from the user. In this way, by allowing the provider or user to switch operations at the timing they desire, it is possible to reduce the sense of discomfort felt when switching operations.

[0128] Furthermore, the control system 20 may further determine a sound effect and output the determined sound effect, thereby providing a more realistic service.

[0129] The user may be a customer, and the provider may be a store clerk. For example, some customers may be motivated to purchase by seeing other customers' purchasing scenes. Therefore, for example, the control system 20 may determine the purchasing scenes of other users based on the purchasing scenes of past product purchases in the purchase history, and output the determined purchasing scenes. This may increase the user's motivation to purchase.

[0130] The user is a patient, and the provider is a doctor. The control system 20 determines the number of other patients that the user can see based on the behavioral history, and outputs avatars of the determined number of other patients. For example, if a patient finds it easier to talk to him or her by listening to what other patients are saying, increasing the number of patients can make it easier to have a conversation. Also, if a patient finds it difficult to talk to him or her when there are many other patients, the number of patients that is determined may be zero. In this way, a service can be provided in a VR space that makes it easier for users to have conversations.

[0131] This concludes the description of each embodiment. Furthermore, each embodiment may be combined. In the embodiments, the control systems 10 and 20 may be configured to include some of the functional units and information.

[0132] This concludes the description of the application examples. Furthermore, the embodiments are not limited to the above examples and can be modified in various ways. Furthermore, the configuration of the control systems 10 and 20 in the embodiments is not particularly limited. For example, the functional units of the control systems 10 and 20 may be implemented by a single device, such as a control device. Alternatively, for example, each functional unit or DB (database) of the control systems 10 and 20 in the embodiments may be implemented by a different device. For example, each functional unit of the control system 20 may be implemented by multiple servers, and the control system 20 may be implemented. For example, the control system 20 may be implemented by a database server including each DB and a server having each functional unit. Furthermore, in the embodiments, each piece of information may include part of the information described above. Furthermore, each piece of information may include information other than the information described above.

[0133] The process of generating information to be displayed on the display device 21 may be performed by the output units 101 and 201. This process may also be performed by the display device 21. That is, the display device 21 generates information for a screen to be displayed on the display device 21 based on data received from the control systems 10 and 20, and displays the screen.

[0134] (Example of Computer Hardware Configuration) Next, an example of a hardware configuration in which each device, such as the control systems 10 and 20 and the display device 21 described in the embodiment, is realized by a computer will be described. Fig. 13 is an explanatory diagram showing an example of a computer hardware configuration. For example, some or all of each device can be realized using any combination of a computer 80 and a program as shown in Fig. 13.

[0135] The computer 80 includes, for example, a processor 801, a ROM (Read Only Memory) 802, a RAM (Random Access Memory) 803, and a storage device 804. The computer 80 also includes a communication interface 805 and an input / output interface 806. The components are connected to each other, for example, via a bus 807. The number of each component is not particularly limited, and there may be one or more of each component.

[0136] The processor 801 controls the entire computer 80. The processor 801 may be, for example, a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, or a combination thereof, and is not particularly limited.

[0137] The computer 80 also includes a ROM 802, a RAM 803, and a storage device 804. Examples of the storage device 804 include a semiconductor memory such as a flash memory, a hard disk drive (HDD), and a solid state drive (SSD). For example, the storage device 804 stores an operating system (OS) program, application programs, and programs related to the embodiments. Alternatively, the ROM 802 stores application programs and programs related to the embodiments. The RAM 803 is used as a work area for the processor 801.

[0138] The processor 801 also loads programs stored in the storage device 804, ROM 802, etc. The processor 801 then executes each process coded in the program. The processor 801 may also download various programs via the communication network NT. The processor 801 also functions as a part or all of the computer 80. The processor 801 may then execute the processes or instructions in the illustrated flowchart based on the program.

[0139] The communication interface 805 is connected to a communication network NT such as a LAN or WAN (Wide Area Network) via a wireless or wired communication line. The communication network NT may be composed of multiple communication networks NT. As a result, the computer 80 is connected to external devices and external computers 80 via the communication networks NT. The communication interface 805 serves as an interface between the communication network NT and the inside of the computer 80. The communication interface 805 also controls the input and output of data from external devices and external computers 80.

[0140] Furthermore, the input / output interface 806 is connected to at least one of an input device, an output device, and an input / output device. The connection method may be wireless or wired. Examples of the input device include a keyboard, a mouse, and a microphone. Examples of the output device include a display device, a lighting device, and an audio output device that outputs audio. Examples of the input / output device include a touch panel display. Note that the input device, output device, and input / output device may be built into the computer 80 or may be external.

[0141] The hardware configuration of the computer 80 is an example. The computer 80 may have some of the components shown in FIG. 13 . The computer 80 may have components other than those shown in FIG. 13 . For example, the computer 80 may have a drive device or the like. The processor 801 may then read programs and data stored on a recording medium attached to the drive device or the like into the RAM 803. Examples of non-transitory tangible recording media include optical disks, flexible disks, magneto-optical disks, and USB (Universal Serial Bus) memories. As described above, the computer 80 may have input devices such as a keyboard and a mouse. The computer 80 may have an output device such as a display. The computer 80 may also have an input device, an output device, and an input / output device.

[0142] The computer 80 may also include various sensors (not shown). The types of sensors are not particularly limited. The computer 80 may also include an imaging device capable of capturing images or videos.

[0143] This concludes the description of the hardware configuration of each device. There are various variations in the method of realizing each device. For example, each device may be realized by any combination of a different computer and program for each component. Furthermore, multiple components of each device may be realized by any combination of a single computer and program.

[0144] Furthermore, some or all of the components of each device may be realized by circuits for specific applications. Furthermore, some or all of the components of each device may be realized by general-purpose circuits such as FPGAs (Field Programmable Gate Arrays). Furthermore, some or all of the components of each device may be realized by a combination of circuits for specific applications and general-purpose circuits. These circuits may be a single integrated circuit. Alternatively, these circuits may be divided into multiple integrated circuits. The multiple integrated circuits may be connected via a bus or the like.

[0145] Furthermore, when some or all of the components of each device are realized by a plurality of computers, circuits, etc., the plurality of computers, circuits, etc. may be centrally located or distributed.

[0146] The control methods described in the embodiments may be implemented by a computer such as the control systems 10 and 20 .

[0147] Each program, such as the formulation program and optimization program described in the embodiments, is recorded on a computer-readable recording medium such as an HDD, SSD, flexible disk, optical disk, magneto-optical disk, or USB memory. Each program is executed by being read from the recording medium by a computer. Each program may also be distributed via a communication network NT.

[0148] The functions of each of the components of control system 10 and control system 20 described above may be realized by dedicated hardware, such as a computer. Alternatively, each component may be realized by software. Alternatively, each component may be realized by a combination of hardware and software.

[0149] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. The configuration and details of each of the present disclosures may include embodiments to which various modifications that would be apparent to those skilled in the art are applied within the scope of the present disclosure. The present disclosure may include embodiments in which the details described herein are appropriately combined or substituted as necessary. For example, details described using a particular embodiment may also be applied to other embodiments to the extent that no contradiction occurs. For example, although multiple operations are described in sequence in the form of a flowchart, the order of description does not limit the order in which the multiple operations are performed. Therefore, when implementing the embodiments, the order of the multiple operations may be changed as long as it does not interfere with the content.

[0150] Some or all of the above-described embodiments can be described as follows: However, some or all of the above-described embodiments are not limited to the following.

[0151] (Supplementary Note 1) A control system comprising: output means for outputting an avatar and an action of the avatar of a provider of a service to a user of the service in a virtual reality space to a display device of the user; acquisition means for acquiring an action history of the user while using the service; and determination means for determining at least one of an avatar and an action of the avatar of the service provider in the virtual reality space based on the action history of the user, wherein the output means switches at least one of the avatar and action of the service provider to at least one of the determined avatar and action of the provider, and outputs the switched avatar and action. (Supplementary Note 2) The control system according to Supplementary Note 1, wherein, if there are multiple users, the determination means determines at least one of an avatar and an action that are visible to each of the multiple users based on the action history of each of the multiple users, and the output means outputs at least one of the determined avatar and action of the user for each of the multiple users. (Supplementary Note 3) The control system according to Supplementary Note 1 or Supplementary Note 2, wherein, when there are a plurality of users, the determination means determines at least one of an avatar of the service provider and an action of the avatar that is visible to each of the plurality of users based on the action history of each of the plurality of users, and the output means switches to and outputs at least one of the avatar of the service provider and an action of the avatar determined for each of the plurality of users. (Supplementary Note 4) The control system according to any of Supplementary Note 1 to Supplementary Note 3, wherein the action history is the degree of attentiveness of the user in a conversation with the provider. (Supplementary Note 5) The control system according to any of Supplementary Note 1 to Supplementary Note 4, wherein the action history is biometric information of the user. (Supplementary Note 6) The control system according to Supplementary Note 5, wherein the determination means determines at least one of an avatar of the service provider and an action of the avatar in the virtual reality space based on at least one of an emotion of the user, a level of tension of the user, and a level of fatigue of the user that are estimated from the biometric information of the user.(Supplementary Note 7) The control system according to any one of Supplementary Notes 1 to 6, wherein the acquisition means acquires a past behavioral history of the user, the determination means determines an avatar of the provider of the service based on the past behavioral history of the user before starting the service, and the output means outputs the determined avatar of the provider. (Supplementary Note 8) The control system according to any one of Supplementary Notes 1 to 7, wherein the output means outputs an avatar and behavior of the user to a display device of the provider in the virtual reality space, the determination means determines at least one of the avatar and behavior of the user based on the behavioral history of the user, and the output means switches at least one of the avatar and behavior of the user to the determined at least one of the avatar and behavior of the user and outputs them. (Supplementary Note 9) The control system according to any one of Supplementary Notes 1 to 8, wherein the output means outputs an avatar and an action of the user in the virtual reality space to a display device of the provider, the acquisition means acquires an action history of the provider while providing the service, the determination means further determines at least one of the avatar and the action of the user based on the action history of the provider, and the output means switches at least one of the avatar and the action of the user to at least one of the determined avatar and the action of the user and outputs the same. (Supplementary Note 10) The control system according to any one of Supplementary Notes 1 to 9, wherein the acquisition means further acquires an action history of the provider while providing the service, the determination means determines the avatar and the action of the provider based on the action history of the provider and the action history of the user, and the output means switches at least one of the avatar and the action of the provider to at least one of the determined avatar and the action of the provider and outputs the same. (Supplementary Note 11) The control system according to any one of Supplementary Note 1 to Supplementary Note 10, wherein the determining means determines at least one of an avatar and an action from a plurality of avatars and a plurality of actions.(Supplementary Note 12) The control system according to any one of Supplementary Notes 1 to 11, wherein the determination means inputs the acquired behavioral history of the users into a model that has learned the relationship between the behavioral history of the users and the provider's avatar and the behavior of the avatar based on the past behavioral history of the users and the provider's avatar and the behavior of the avatar when each of the multiple users uses the service, to determine at least one of the avatar and the behavior of the provider of the service. (Supplementary Note 13) The control system according to Supplementary Note 12, comprising: learning means that generates the model that has learned the relationship between the behavioral history of the users and the provider's avatar and the behavior of the avatar based on the past behavioral history of the users and the provider's avatar and the behavior of the avatar when each of the multiple users uses the service, wherein the determination means inputs the acquired behavioral history of the users into the generated model to determine at least one of the avatar and the behavior of the provider of the service. (Supplementary Note 14) The control system according to any one of Supplements 1 to 13, wherein the determination means determines at least one of an avatar and an avatar behavior of a provider different from the provider currently providing the service to the user. (Supplementary Note 15) The control system according to any one of Supplements 1 to 13, wherein the determination means determines at least one of an avatar and an avatar behavior of the same provider as the provider currently providing the service to the user. (Supplementary Note 16) The control system according to any one of Supplements 1 to 15, further comprising: a receiving means for receiving an avatar switching instruction from the provider, wherein the output means, upon receiving the switching instruction, switches at least one of the avatar and behavior of the provider to at least one of the avatar and behavior of the determined provider and outputs the switched avatar and behavior. (Supplementary Note 17) A control system as described in any of Supplementary Notes 1 to 15, further comprising: a receiving means for receiving an instruction to switch avatars from the user; and when the output means receives the switching instruction, the output means switches at least one of the avatar and action of the provider to at least one of the determined avatar and action of the provider and outputs the switched avatar and action.(Supplementary Note 18) The control system according to any one of Supplementary Notes 1 to 17, wherein the determination means further determines a sound effect, and the output means outputs the determined sound effect. (Supplementary Note 19) The control system according to any one of Supplementary Notes 1 to 18, wherein the user is a customer, and the provider is a store clerk. (Supplementary Note 20) The control system according to Supplementary Note 19, wherein the behavioral history includes a purchase history and a purchase scene of the user in the purchase history, the determination means determines a purchase scene by another user based on a purchase scene of a past product purchase in the purchase history, and the output means outputs the determined purchase scene. (Supplementary Note 21) The control system according to any one of Supplementary Notes 1 to 18, wherein the user is a patient, and the provider is a doctor. (Supplementary Note 22) The control system according to Supplementary Note 21, wherein the determination means determines the number of other patients visible to the user based on the behavioral history, and the output means outputs avatars of the determined number of other patients. (Supplementary Note 23) A control method in which a computer executes a process of outputting an avatar and the avatar's behavior of a provider of a service to a user of the service in a virtual reality space on the user's display device, acquiring a behavioral history of the user while using the service, and determining at least one of the avatar and the avatar's behavior of the provider of the service in the virtual reality space based on the behavioral history of the user, and in the output process, switching at least one of the avatar and behavior of the provider of the service to at least one of the determined avatar and behavior of the provider and outputting them.(Supplementary Note 24) A non-transitory computer-readable recording medium having recorded thereon a program that causes a computer to execute a process of outputting an avatar and the avatar's behavior of a provider of a service to a user of the service in a virtual reality space to the user's display device, acquiring a behavioral history of the user while using the service, and determining at least one of the avatar and the avatar's behavior of the provider of the service in the virtual reality space based on the user's behavioral history, and in the output process, switching at least one of the avatar and behavior of the provider of the service to at least one of the determined avatar and behavior of the provider and outputting them. (Appendix 25) A program that causes a computer to execute a process of outputting an avatar and the avatar's behavior of a provider of a service to a user of the service in a virtual reality space on the user's display device, acquiring a behavioral history of the user while using the service, and determining at least one of the avatar and the avatar's behavior of the provider of the service in the virtual reality space based on the user's behavioral history, and in the output process, switching at least one of the avatar and behavior of the provider of the service to at least one of the determined avatar and behavior of the provider and outputting them.

[0152] 10, 20 Control system 21 Display device 80 Computer 101, 201 Output unit 102, 202 Acquisition unit 103, 203 Determination unit 204 Reception unit 205 Learning unit 801 Processor 802 ROM 803 RAM 804 Storage device 805 Communication interface 806 Input / output interface 807 Bus NT Communication network

Claims

1. A control system comprising: an output means for outputting an avatar and the avatar's actions of a provider of a service to a user of the service in a virtual reality space on the user's display device; an acquisition means for acquiring a behavioral history of the user while using the service; and a determination means for determining at least one of the avatars of the provider of the service and the actions of the avatar in the virtual reality space based on the behavioral history of the user, wherein the output means switches at least one of the avatars and actions of the service provider to the determined at least one of the avatars and actions of the provider and outputs the switched at least one of the avatars and actions of the provider.

2. The control system described in claim 1, wherein the determination means, when there are multiple users, determines at least one of the avatars and actions that each of the multiple users will be able to see based on the behavioral history of each of the multiple users, and the output means outputs at least one of the user's avatars and actions determined for each of the multiple users.

3. A control system as described in claim 1 or claim 2, wherein the determination means, when there are multiple users, determines at least one of the provider's avatar and the avatar's behavior visible to each of the multiple users based on the behavioral history of each of the multiple users, and the output means switches to and outputs at least one of the provider's avatar and the behavior determined for each of the multiple users.

4. A control system according to any one of claims 1 to 3, wherein the behavioral history is the degree of attentive listening of the user in a conversation with the provider.

5. A control system according to any one of claims 1 to 4, wherein the behavioral history is biometric information of the user.

6. The control system described in claim 5, wherein the determination means determines at least one of the avatar of the service provider and the avatar's behavior in the virtual reality space based on at least one of the user's emotions, the user's level of tension, and the user's level of fatigue estimated from the user's biometric information.

7. A control system as described in any one of claims 1 to 6, wherein the acquisition means acquires a past behavioral history of the user, the determination means determines an avatar of the provider of the service based on the past behavioral history of the user before starting the service, and the output means outputs the determined avatar of the provider.

8. A control system as described in any one of claims 1 to 7, wherein the output means outputs the user's avatar and actions in the virtual reality space to the provider's display device, the determination means determines at least one of the user's avatar and actions based on the user's behavior history, and the output means switches at least one of the user's avatar and actions to the determined at least one of the user's avatar and actions and outputs them.

9. A control system as described in any one of claims 1 to 8, wherein the output means outputs the user's avatar and actions in the virtual reality space to the provider's display device, the acquisition means acquires an action history of the provider while providing the service, the determination means further determines at least one of the user's avatar and actions based on the provider's action history, and the output means switches at least one of the user's avatar and actions to the determined at least one of the user's avatar and actions and outputs them.

10. A control system as described in any one of claims 1 to 9, wherein the acquisition means further acquires a behavioral history of the provider providing the service, the determination means determines an avatar and behavior of the provider based on the behavioral history of the provider and the behavioral history of the user, and the output means switches at least one of the avatar and behavior of the provider to at least one of the determined avatar and behavior of the provider and outputs the switched at least one of the avatar and behavior of the provider.

11. A control system according to any one of claims 1 to 10, wherein the determining means determines at least one of an avatar and an action from a plurality of avatars and a plurality of actions.

12. A control system as described in any one of claims 1 to 11, wherein the determination means inputs the acquired behavioral history of the users into a model that learns the relationship between the behavioral history of the users and the provider's avatar and the behavior of the avatar based on the past behavioral history of the users and the provider's avatar and the behavior of the avatar when each of the multiple users uses the service, and determines at least one of the avatar and behavior of the provider of the service.

13. The control system of claim 12, further comprising: a learning means for generating the model that learns the relationship between the behavioral history of the users and the provider's avatar and the avatar's behavior based on the past behavioral history of the users and the provider's avatar and the avatar's behavior when each of the users uses the service; wherein the determination means inputs the obtained behavioral history of the users into the generated model and determines at least one of the avatar and behavior of the provider of the service.

14. A control system according to any one of claims 1 to 13, wherein the decision means decides at least one of an avatar of a provider other than the provider currently providing the service to the user and an action of the avatar.

15. A control system according to any one of claims 1 to 13, wherein the determination means determines at least one of an avatar of the provider that is the same as the provider currently providing the service to the user and an action of the avatar.

16. A control system as described in any one of claims 1 to 15, further comprising: a receiving means for receiving an instruction to switch avatars from the provider; and wherein, upon receiving the switching instruction, the output means switches at least one of the provider's avatar and action to at least one of a determined provider's avatar and action and outputs the switched avatar and action.

17. A control system as described in any one of claims 1 to 15, further comprising: a receiving means for receiving an instruction to switch avatars from the user; and when the output means receives the switching instruction, it switches at least one of the avatar and action of the provider to at least one of the determined avatar and action of the provider and outputs the switched avatar and action.

18. A control system according to any one of claims 1 to 17, wherein the determining means further determines a sound effect, and the output means outputs the determined sound effect.

19. A control system according to any one of claims 1 to 18, wherein the user is a customer, and the provider is a store clerk.

20. The control system described in claim 19, wherein the behavioral history includes a purchase history and purchasing scenes of the user in the purchase history, the determination means determines purchasing scenes by other users based on purchasing scenes at the time of past product purchases in the purchase history, and the output means outputs the determined purchasing scenes.

21. A control system according to any one of claims 1 to 18, wherein the user is a patient, and the provider is a doctor.

22. The control system according to claim 21, wherein the determination means determines the number of other patients visible to the user based on the behavioral history, and the output means outputs avatars of the determined number of other patients.

23. A control method in which a computer executes a process of outputting an avatar and the avatar's behavior of a provider of a service to a user of the service in a virtual reality space on the user's display device, acquiring a behavioral history of the user while using the service, and determining at least one of the avatar and the avatar's behavior of the provider of the service in the virtual reality space based on the behavioral history of the user, and in the output process, switching at least one of the avatar and behavior of the provider of the service to the determined at least one of the avatar and behavior of the provider and outputting the same.

24. A non-transitory computer-readable recording medium storing a program that causes a computer to execute a process of outputting an avatar and the avatar's actions of a provider of a service to a user of the service in a virtual reality space to the user's display device, acquiring a behavioral history of the user while using the service, and determining at least one of the avatar and the avatar's actions of the provider of the service in the virtual reality space based on the user's behavioral history, and in the output process, switching at least one of the avatar and actions of the provider of the service to the determined at least one of the avatar and actions of the provider and outputting them.

Images