Information processing device, information processing method, and information processing program

Abstract

To provide an information processing device, an information processing method, and an information processing program capable of improving convenience of a user.SOLUTION: An information processing device according to the present application includes: a reception part; and a reproduction processing part. The reception part receives setting of an individual identification degree indicating the degree to which a user is identified. The reproduction processing part corrects an action of the user based on the individual identification degree received by the reception part and reproduces the action in a virtual space.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] In recent years, technologies are known that present an avatar that represents a user in a virtual space and enable communication with other users' avatars within the virtual space. For example, technologies have been developed to reproduce the user's speech as the avatar's speech on the virtual space or to reproduce the user's body movements as the avatar's movements on the virtual space.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above prior art, user actions such as the user's speech and movements can be reproduced as actions within the virtual space, but there are users who do not want to be identified by their actions within the virtual space, and there is room for improvement in this regard.

[0005] The present application has been made in view of the above, and an object thereof is to provide an information processing apparatus, an information processing method, and an information processing program that can improve user convenience.

Means for Solving the Problems

[0006] The information processing apparatus according to the present application includes a reception unit and a reproduction processing unit. The reception unit receives a setting of the degree of individual identification indicating the degree to which the user is identified. The reproduction processing unit corrects the user's actions based on the degree of individual identification received by the reception unit and reproduces them in the virtual space. [Effects of the Invention]

[0007] According to one embodiment, the effect is to improve user convenience. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a diagram illustrating the information processing method performed by the information processing device according to the embodiment. [Figure 2] Figure 2 shows an example of the configuration of an information processing system according to the embodiment. [Figure 3] Figure 3 shows an example of the configuration of an information processing device according to the embodiment. [Figure 4] Figure 4 shows an example of a user information table stored in the user information storage unit according to the embodiment. [Figure 5] Figure 5 shows an example of a character information table stored in the character information storage unit according to this embodiment. [Figure 6] Figure 6 is a flowchart showing an example of information processing by the processing unit of the information processing device according to the embodiment. [Figure 7] Figure 7 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing device according to the embodiment. [Modes for carrying out the invention]

[0009] The following describes in detail, with reference to the drawings, the embodiments for implementing the information processing apparatus, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing apparatus, information processing method, and information processing program according to the present application. Furthermore, each embodiment can be appropriately combined as long as the processing content is not inconsistent. Also, the same parts are denoted by the same reference numerals in each of the following embodiments, and redundant descriptions are omitted.

[0010] [1. An example of information processing] First, with reference to Figure 1, the information processing method performed by the information processing device according to the embodiment will be described. Figure 1 is a diagram illustrating the information processing method performed by the information processing device according to the embodiment.

[0011] The information processing device 1 shown in Figure 1 provides an online service that enables communication between users in a virtual space. Such an online service is also called a metaverse service. In the following, the online service provided by the information processing device 1 may be referred to as a metaverse service.

[0012] The example shown in Figure 1 illustrates the provision of a metaverse service to users U1 and U2 of terminal devices 21 and 22. In the following, when referring to multiple terminal devices, including terminal devices 21 and 22, without individually distinguishing them, they may be referred to as "terminal device 2," and when referring to multiple users, including users U1 and U2, without individually distinguishing them, they may be referred to as "user U." Note that in the example shown in Figure 1, for the sake of clarity, the number of terminal devices 2 and users U is set to 2, but the number of terminal devices 2 and users U is 3 or more.

[0013] Each user U can configure the appearance of their avatar in the virtual space by specifying a desired character from among several characters, or by specifying a degree of personal identification that indicates the degree to which user U can be identified. The character specified by user U will be displayed as user U's avatar in the virtual space according to the degree of personal identification.

[0014] Terminal device 21 transmits user U1's configuration information to information processing device 1 (step S11), and terminal device 22 transmits user U2's configuration information to information processing device 1 (step S12). Information processing device 1 receives the configuration information transmitted from terminal devices 21 and 22 (step S2).

[0015] The setting information of each user U includes a user ID (Identifier) which is an identifier for identifying user U, personal identification degree information which is information indicating the degree of identification of user U, and the like.

[0016] The personal identification degree information included in the setting information of user U1 includes information indicating the degree of identification of user U1, and the personal identification degree information included in the setting information of user U2 includes information indicating the degree of identification of user U2.

[0017] The character designation information included in the setting information of user U1 includes information on the character designated by user U1, and the character designation information included in the setting information of user U2 includes information on the character designated by user U2.

[0018] Subsequently, the terminal device 21 transmits user information which is the information of user U1 to the information processing device 1 (step S31), and the terminal device 22 transmits user information which is the information of user U2 to the information processing device .The information processing device 1 receives the user information of users U1 and U2 transmitted from the terminal devices 21 and 22 (step S4).

[0019] The user information of user U is information indicating the actions of user U, and the actions of user U include the speech of user U, the movements (actions) of user U, etc. Also, the movements of user U include the gestures of user U, the expressions of user U, etc.

[0020] Each terminal device 2 is equipped with a microphone, a camera, a display unit, a speaker, etc., and the user information includes real-time information including the voice information of user U collected in real time by the microphone and the imaging video information which is the information of the video of user U captured in real time by the camera.

[0021] The information processing device 1 generates virtual space information for each user U based on the setting information of users U1 and U2 received in step S2 and the user information of users U1 and U2 received in step S4 (step S5). In the example shown in Figure 1, the information processing device 1 generates virtual space information for user U1 and virtual space information for user U2.

[0022] In step S5, the information processing device 1 generates an avatar for each user U based on the user information and setting information of each user U. Then, the information processing device 1 generates virtual space information for each user U based on the generated avatar and the location of each user U in the virtual space.

[0023] The information processing device 1 corrects the actions of user U, as indicated by user U's user information, based on the degree of personal identification indicated by user U's settings information, and places the resulting avatar at user U's location in the virtual space as user U's avatar, thereby reproducing user U's actions in the virtual space with the degree of personal identification corrected.

[0024] User U's actions include User U's physical movements and User U's speech. User U's physical movements include User U's facial movements (expressions or changes in facial expressions) and User U's gestures (for example, body movements other than the face).

[0025] The information processing device 1 performs image processing on the body image, which is an image of user U's body included in the captured video information of user U, according to the degree of personal identification, in order to generate video information of user U's avatar in the virtual space information. Image processing according to the degree of personal identification is performed, for example, by processing user U's face or facial expressions according to the degree of personal identification, or by processing user U's gestures according to the degree of personal identification. The degree of processing of user U's facial expressions and gestures increases as the degree of personal identification decreases.

[0026] The processing of user U's face is performed, for example, by replacing the 3D shapes of user U's facial features with 3D facial shapes obtained by combining the 3D shapes of user U's facial features with the specific 3D shapes of those facial features at a ratio corresponding to the degree of individual identification for each facial feature.

[0027] When the composite ratio of the 3D shapes of user U's facial features and the specific 3D shapes of those facial features is R1:R2, the ratio R corresponding to the degree of personal identification can be expressed, for example, as R = R2 / (R1+R2), and the lower the degree of personal identification, the larger the value. In other words, the information processing device 1 performs a composite with a greater weight on the specific 3D shapes of the facial features as the degree of personal identification decreases.

[0028] Furthermore, the processing of user U's facial expressions is carried out, for example, by identifying the type of facial expression of user U and emphasizing the expression to a degree corresponding to the degree of personal identification, or by combining a specific facial expression of the identified type with user U's facial expression at a ratio corresponding to the degree of personal identification. The emphasis of facial expressions is carried out, for example, by widening the opening of the mouth or exaggerating the movement of the eyebrows.

[0029] Furthermore, the processing of user U's gestures can be done, for example, by identifying the type of gesture used by user U and emphasizing the gesture to a degree corresponding to the degree of personal identification, or by combining a specific gesture of the identified type with user U's gesture at a ratio corresponding to the degree of personal identification. Gesture emphasis can be achieved, for example, by making arm movements larger or head movements more exaggerated.

[0030] For example, the information processing device 1 performs speech processing on the utterances of user U, represented by user U's voice information, according to the degree of personal identification, to generate voice information for user U's avatar in the virtual space information. For example, the information processing device 1 converts user U's voice information into text information using speech recognition processing, and also identifies user U's speaking style (volume, speed, intonation (emphasis), intervals between utterances (pauses), etc.) and voice quality using speech recognition processing.

[0031] The information processing device 1 identifies user U's dialect based on one or more of the character information converted by speech recognition processing and the speech patterns obtained by speech recognition processing. The information processing device 1 generates speech information in which user U's utterances are corrected to the specific dialect with a correction degree corresponding to the degree of personal identification.

[0032] For example, suppose user U's dialect is Osaka dialect, and a specific dialect is Tokyo dialect. In this case, when the information processing device 1 converts the text information converted from user U's voice information into voice information using speech synthesis processing, it converts the Osaka dialect to Tokyo dialect with a correction degree according to the degree of personal identification.

[0033] For example, the information processing device 1 can perform a first speech synthesis process using Tokyo dialect, a second speech synthesis process that is strongly Tokyo dialect but also contains Osaka dialect, a third speech synthesis process that is strongly Osaka dialect but also contains Tokyo dialect, a fourth speech synthesis process using Osaka dialect, and so on. The device selects and executes a speech synthesis process from among these that is appropriate for identifying an individual.

[0034] In this case, the information processing device 1 selects the first speech synthesis process if the degree of personal identification is less than the first threshold, and selects the second speech synthesis process if the degree of personal identification is greater than or equal to the first threshold but less than the second threshold. Furthermore, the information processing device 1 selects the third speech synthesis process if the degree of personal identification is greater than or equal to the second threshold but less than the third threshold, and selects the fourth speech synthesis process if the degree of personal identification is greater than or equal to the third threshold. Note that the types of speech synthesis processes corresponding to the degree of personal identification are not limited to the examples described above, and may be five or more, or three or fewer.

[0035] Furthermore, the information processing device 1 can perform the above-described speech synthesis processing using speech synthesis parameters that have been corrected with a correction degree corresponding to the degree of personal identification of user U, specifically at least one of user U's speaking style and voice quality. Speech synthesis parameters include, for example, volume, speed, intonation (emphasis), utterance interval (pause), and voice quality.

[0036] The information processing device 1 determines the values ​​of various speech synthesis parameters corresponding to user U's speaking style and voice quality, and corrects the determined values ​​of the speech synthesis parameters by adding a correction value according to the degree of personal identification. Then, the information processing device 1 can generate speech information in which user U's speaking style and voice quality have been corrected to a degree of personal identification by performing a speech synthesis process using the various speech synthesis parameters corrected with the correction value according to the degree of personal identification.

[0037] Furthermore, the information processing device 1 can also reproduce in the virtual space the characteristics of user U's actions, corrected to the characteristics of a specific character's actions to a degree corresponding to the degree of personal identification. In this case, the specific character is a predetermined standard character, but if user U has specified a character, then the specific character is the character specified by user U.

[0038] The configuration information transmitted from terminal device 2 to information processing device 1 may include character specification information, which is information about the character specified by user U. This allows user U to specify a particular character using terminal device 2.

[0039] Specifically, the information processing device 1 corrects User U's actions, as indicated by User U's user information, by increasing the degree of correction that brings User U's actions closer to the character specified by User U as the degree of personal identification decreases, thereby reproducing User U's avatar in a virtual space.

[0040] Character characteristics are defined by character information, which includes, for example, a specific 3D body model, motion information, and a specific speech model. The 3D body model includes a head model, which includes the face and neck, and models of other parts of the body, such as the legs, ankles, waist, arms, and wrists. Motion information defines the movement of the 3D body model. For example, motion information includes motion information for each type of facial expression and motion information for each type of gesture.

[0041] The information processing device 1, for example, detects the positions of each feature point of user U from the captured image information of user U included in the user information, and generates a 3D body model of user U from these detection results. The information processing device 1 also identifies the movement of the 3D body model from the captured image information of user U, and identifies the type of facial expression and gesture of user U.

[0042] The information processing device 1 then synthesizes the movements of the user U's 3D body model and the character's 3D body model at a ratio corresponding to the degree of personal identification, and generates a corrected 3D body model of user U.

[0043] For example, the synthesis of the movements of User U's 3D body model and the movements of the character's 3D body model at a ratio corresponding to the degree of personal identification includes the synthesis of facial expressions and gestures. For example, the information processing device 1 synthesizes User U's facial expressions with the facial expressions of the 3D body model defined by motion information corresponding to the type of facial expression of User U at a ratio corresponding to the degree of personal identification. The information processing device 1 also synthesizes User U's gestures with the gestures of the 3D body model defined by motion information corresponding to the type of gesture of User U at a ratio corresponding to the degree of personal identification.

[0044] In this way, the information processing device 1 determines the user U's actions from the captured image information of the user U, and reproduces the determined user U's actions in a virtual space using a corrected 3D model of the user U's body. As a result, the information processing device 1 can correct the user U's actions to match the actions of a specific character to a degree corresponding to the degree of personal identification, and reproduce them in a virtual space.

[0045] Furthermore, the information processing device 1 can correct user U's speech to match the speech of a specific character with a degree of correction corresponding to the degree of personal identification, and reproduce it in a virtual space. The specific speech model has the function of performing speech synthesis processing using speech synthesis parameters corresponding to the specific character.

[0046] For example, the information processing device 1 determines, based on the voice information of user U, the values ​​of various speech synthesis parameters corresponding to user U's speaking style and voice quality as the values ​​of user U's speech synthesis parameters.

[0047] The information processing device 1 then adds the values ​​of the speech synthesis parameters of user U and the speech synthesis parameters of a specific character at a ratio corresponding to the degree of personal identification, thereby generating speech synthesis parameters corrected with a correction value corresponding to the degree of personal identification. The ratio corresponding to the degree of personal identification increases as the degree of personal identification decreases, the greater the weight of the values ​​of the speech synthesis parameters of the specific character.

[0048] In this way, the information processing device 1 can generate speech information in which the speech style and voice quality of user U have been corrected to a degree corresponding to the degree of individual identification by performing speech synthesis processing using various speech synthesis parameters that have been corrected with correction values ​​corresponding to the degree of individual identification.

[0049] Next, the information processing device 1 transmits the virtual space information for user U1 generated in step S5 to terminal device 21, and transmits the virtual space information for user U2 generated in step S5 to terminal device 22 (steps S61, S62).

[0050] Terminal device 21 receives virtual space information for user U1 from information processing device 1 and displays an image of the virtual space based on the received virtual space information for user U1. Terminal device 22 also receives virtual space information for user U2 from information processing device 1 and displays an image of the virtual space based on the received virtual space information for user U2.

[0051] By reducing the degree of personal identification for user U1, user U1's actions are reproduced in the virtual space with a high degree of correction, thus reducing the possibility that the avatar in the virtual space will be identified as user U1's avatar.

[0052] Similarly, by reducing the degree of personal identification, user U2's behavior is reproduced in the virtual space with a high degree of correction, thus reducing the possibility that the avatar in the virtual space will be identified as user U2's avatar.

[0053] In this way, the information processing device 1 accepts a setting for the degree to which user U can be identified, corrects user U's actions based on the degree of identification, and reproduces them in the virtual space. As a result, the information processing device 1 can improve user U's convenience by fulfilling user U's request to not want to be identified by their actions in the virtual space using the degree of identification.

[0054] The configuration of the information processing system, including the information processing device 1 and terminal device 2 that perform such processing, will be described in detail below.

[0055] [2. Configuration of the Information Processing System] Figure 2 shows an example of the configuration of an information processing system according to the embodiment. As shown in Figure 2, the information processing system 100 according to the embodiment includes an information processing device 1 that provides a metaverse service and multiple terminal devices 2-2 owned by multiple users U that use the metaverse service. n It includes multiple terminal devices 2, where n is, for example, an integer greater than or equal to 2.

[0056] The information processing device 1 and the multiple terminal devices 2 are connected to each other via a network N, either by wire or wireless connection, enabling communication between them. Note that the information processing system 100 shown in Figure 2 may include multiple information processing devices 1. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

[0057] Each terminal device 2 is equipped with a microphone, camera, display unit, and speaker. Each terminal device 2 transmits real-time information to the information processing device 1 via the network N, including voice information of user U collected in real time by the microphone, and captured video information of user U captured in real time by the camera.

[0058] Furthermore, each terminal device 2 receives conference information transmitted from the information processing device 1 via the network N, displays the video information contained in the conference information on an unillustrated display unit, and outputs the audio information contained in the conference information from a speaker.

[0059] Each terminal device 2 may be, for example, a desktop PC (Personal Computer), a notebook PC, a tablet device, a smartphone, a mobile phone, or a PDA (Personal Digital Assistant). Each terminal device 2 is operated by user U. Note that each terminal device 2 is not limited to the examples described above, and may also be, for example, a wearable device.

[0060] Furthermore, terminal device 2 can connect to network N via wireless communication networks such as LTE (Long Term Evolution), 4G (4th Generation), and 5G (5th Generation), or via short-range wireless communication such as Bluetooth (registered trademark) and Wi-Fi, and communicate with information processing device 1.

[0061] [3. Configuration of Information Processing Device 1] The following describes an example of the functional configuration of the information processing device 1. Figure 3 is a diagram showing an example of the configuration of the information processing device 1 according to the embodiment. As shown in Figure 3, the information processing device 1 has a communication unit 10, a storage unit 11, and a processing unit 12.

[0062] [3.1. Communications Section 10] The communication unit 10 is implemented, for example, by a NIC (Network Interface Card). The communication unit 10 is connected to the network N by wire or wireless connection and transmits and receives information with various other devices. For example, the communication unit 10 transmits and receives information with each of the multiple terminal devices 2 via the network N.

[0063] [3.2. Storage section 11] The memory unit 11 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs. The memory unit 11 also includes a user information storage unit 20 and a character information storage unit 21.

[0064] [3.2.1. User Information Storage Unit 20] The user information storage unit 20 stores various information about user U. Figure 4 is a diagram showing an example of a user information table stored in the user information storage unit 20 according to this embodiment.

[0065] In the example shown in Figure 4, the user information table stored in the user information storage unit 20 includes information on items such as "User ID," "Attribute Information," "History Information," and "Setting Information."

[0066] The "User ID" is an identifier that identifies User U. The "Attribute Information" is attribute information about User U's attributes associated with the "User ID". User U's attributes include, for example, demographic attributes and psychographic attributes. Demographic attributes are demographic attributes such as age, gender, affiliation, occupation, place of residence, annual income, and family structure. Psychographic attributes are psychological attributes such as lifestyle, values, and interests.

[0067] "History information" refers to information such as the usage history of the metaverse service of user U, associated with the "User ID," and includes information on the history of the original user U's actions reproduced in the virtual space, as well as the history of the actions of user U's avatar reproduced in the virtual space with corrections according to the degree of personal identification. In addition, history information may also include the usage history of web services (for example, online shopping usage history, web content search history, web content browsing history, etc.).

[0068] "Configuration information" refers to the configuration information of user U associated with the "User ID," and includes, for example, personal identification information indicating the degree of personal identification specified by user U, and specified character information which is information about the character specified by user U.

[0069] The information stored in the user information storage unit 20 is not limited to the above and may include various types of information depending on the purpose. For example, the information stored in the user information storage unit 20 may include information indicating the context history of user U.

[0070] [3.2.2. Character Information Storage Unit 21] The character information storage unit 21 stores various information about multiple characters that can be specified by user U. Figure 5 is a diagram showing an example of a character information table stored in the character information storage unit 21 according to this embodiment.

[0071] In the example shown in Figure 5, the character information table stored in the character information storage unit 21 includes information on items such as "character ID," "3D body model," and "speech synthesis parameters."

[0072] "Character ID" is an identifier that identifies a character. "Body 3D Model" is information about the character's body 3D model associated with the "Character ID". The body 3D model includes the head model, which includes the face and neck, and the models of other parts of the body, which include the legs, ankles, waist, arms, wrists, etc.

[0073] "Speech synthesis parameters" are information about a set of parameters used to form a specific speech model, representing the speech information of a character associated with a "character ID". A specific speech model performs speech synthesis processing using a set of parameters that includes multiple speech synthesis parameters specific to that character.

[0074] The information stored in the character information storage unit 21 is not limited to the above, and may include various types of information depending on the purpose.

[0075] [3.3. Processing Unit 12] The processing unit 12 is a controller, and is realized by a processor such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs (an example of an information processing program) stored in the memory device inside the information processing device 1, using RAM as the working area. Alternatively, the processing unit 12 is a controller and can be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0076] As shown in Figure 3, the processing unit 12 includes an acquisition unit 30, a reception unit 31, a determination unit 32, a reproduction processing unit 33, and a provision unit 34, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the processing unit 12 is not limited to the configuration shown in Figure 3, and other configurations are also acceptable as long as they perform the information processing described later.

[0077] [3.3.1. Acquisition part 30] The acquisition unit 30 acquires various types of information. The acquisition unit 30 acquires various types of information from the storage unit 11. The acquisition unit 30 acquires various types of information from the user information storage unit 20 and the character information storage unit 21, etc.

[0078] The acquisition unit 30 receives various information from an external information processing device via the communication unit 10. The acquisition unit 30 also receives various information from the terminal device 2. For example, the acquisition unit 30 acquires user U information from the terminal device 2, stores the acquired user U information in the user information storage unit 20, and updates the user information table.

[0079] Furthermore, the acquisition unit 30 acquires information about characters from an external device, stores the acquired information about characters in the character information storage unit 21, and updates the character information table.

[0080] [3.3.2. Reception Department 31] The reception unit 31 receives various requests. The reception unit 31 receives various requests from the terminal device 2. The reception unit 31 receives information indicating various requests from the terminal device 2 via the communication unit 10.

[0081] For example, the reception unit 31 receives user information from the terminal device 2. The user information includes real-time information such as voice information of user U collected in real time by a microphone, and captured video information of user U captured in real time by a camera.

[0082] The reception unit 31 also receives configuration information and user information from the terminal device 2. The configuration information includes the user ID of user U and personal identification information, which indicates the degree to which user U can be identified. The configuration information may also include character designation information, which is information about the character specified by user U.

[0083] The reception unit 31 can also receive personal identification information and character designation information for each other user U to whom user U's actions in the virtual space are presented. For example, suppose user U1, U2, and U3 are users using the metaverse service. In this case, user U1's configuration information includes, for example, personal identification information and character designation information for each of users U2 and U3 to whom user U1's actions in the virtual space are presented.

[0084] Furthermore, user U2's configuration information includes, for example, personal identification information and character designation information for each user U1 and U3, to whom user U2's actions in the virtual space will be displayed. Similarly, user U3's configuration information includes, for example, personal identification information and character designation information for each user U2 and U3, to whom user U3's actions in the virtual space will be displayed.

[0085] Furthermore, the user U's configuration information may include, for example, different personal identification information and character designation information for each type of other user U to whom user U's actions in the virtual space are presented. The types of other user U are classified based on the attributes and behavioral history of other user U, such as, for example, a male in his 20s, a female in her 30s, etc.

[0086] Furthermore, the user U configuration information may include personal identification information and character designation information that are common to all other user Us whose actions in the virtual space are presented to user U.

[0087] [3.3.3. Decision Section 32] If user U does not specify a character, the determination unit 32 determines a specific character to associate with user U based on user U's action history.

[0088] User U's behavioral history is either the behavioral history of the original User U reproduced in the virtual space, or the behavioral history of User U's avatar reproduced in the virtual space with corrections according to the degree of personal identification. However, in lieu of or in addition to such history, it may also include the usage history of web services.

[0089] For example, the determination unit 32 determines that a character different from the character specified by another user U, whose actions are similar to those of user U, is a specific character that can be associated with user U.

[0090] Furthermore, the determination unit 32 has character type information to which a character is associated with each behavior pattern. The determination unit 32 can also determine that the character associated with user U's behavior pattern is a specific character associated with user U from among multiple behavior patterns.

[0091] Behavioral patterns are classified into multiple types based on factors such as the number of utterances and actions per unit of time. The determination unit 32 determines the user U's behavioral pattern from among the multiple behavioral patterns based on the user U's number of utterances and actions per unit of time.

[0092] Furthermore, the determination unit 32 can also determine a specific character to be associated with user U based on information about user U's attributes. For example, the determination unit 32 has character type information to which characters are associated for each attribute (e.g., combinations of gender, age, and occupation). The determination unit 32 can also determine a character that is associated with user U's attributes from among the attribute types as the specific character to be associated with user U.

[0093] Furthermore, the determination unit 32 can determine a specific character to associate with user U based on the degree of personal identification specified by user U. For example, the higher the degree of personal identification specified by user U, the more characters that have been specified by other users U can be determined as the specific character to associate with user U.

[0094] The determination unit 32 can determine a specific character to associate with user U based on user U's behavioral patterns, user U's attributes, and two or more of the personal identification levels specified by user U.

[0095] For example, the determination unit 32 may have character type information to which a character is associated for each combination of behavior patterns and attributes. In this case, the determination unit 32 can also determine that the character associated with the combination of behavior patterns and attributes of user U is a specific character associated with user U.

[0096] Furthermore, the decision unit 32 may have character type information to which multiple characters are associated for each behavior pattern. In this case, the decision unit 32 selects multiple characters from among the multiple behavior patterns that are associated with user U's behavior pattern. Then, from among the selected multiple characters, the decision unit 32 determines a specific character to be associated with user U based on the degree of personal identification specified by user U.

[0097] The determination unit 32 then determines, for example, among multiple characters, the character that has been designated by other users U to be the specific character associated with user U, based on the degree of personal identification specified by user U.

[0098] Furthermore, the decision unit 32 can also determine a specific character for each other user U or each type of user U to which user U's actions in the virtual space are presented. In addition, the decision unit 32 can also determine a specific character based on the usage history of web services. For example, the decision unit 32 can determine a specific character to be the character of the category belonging to the most recent search query.

[0099] [3.3.4. Reproduction Processing Unit 33] The reproduction processing unit 33 generates virtual space information for each user U based on the user U setting information received by the reception unit 31 and the user information of user U received by the reception unit 31.

[0100] The reproduction processing unit 33 generates an avatar for each user U based on the user information and settings information of each user U. Then, the reproduction processing unit 33 generates virtual space information for each user U based on the generated avatar and the position of each user U in the virtual space.

[0101] The reproduction processing unit 33 corrects the actions of user U, as indicated by user U's user information, based on the degree of personal identification indicated by user U's settings information, and places the resulting avatar at user U's position in the virtual space as user U's avatar, thereby reproducing user U's actions in the virtual space with the degree of personal identification corrected.

[0102] User U's actions include User U's physical movements and User U's speech. User U's physical movements include User U's facial movements (expressions or changes in facial expressions) and User U's gestures (for example, body movements other than the face).

[0103] The reproduction processing unit 33 performs image processing on the body image, which is an image of user U's body included in the captured video information of user U, according to the degree of personal identification, in order to generate video information of user U's avatar in the virtual space information. Image processing according to the degree of personal identification is performed, for example, by processing user U's face or facial expressions according to the degree of personal identification, or by processing user U's gestures according to the degree of personal identification. The degree of processing of user U's facial expressions and gestures increases as the degree of personal identification decreases.

[0104] The processing of user U's face is performed, for example, by replacing the 3D shapes of user U's facial features with 3D facial shapes obtained by combining the 3D shapes of user U's facial features with the specific 3D shapes of those facial features at a ratio corresponding to the degree of individual identification for each facial feature.

[0105] When the composite ratio of the 3D shapes of user U's facial features and the specific 3D shapes of the facial features is R1:R2, the ratio R corresponding to the degree of personal identification can be expressed as, for example, R = R2 / (R1+R2), and the lower the degree of personal identification, the larger the value. In other words, the reproduction processing unit 33 performs a composite with a greater weight on the specific 3D shapes of the facial features as the degree of personal identification decreases.

[0106] Furthermore, the processing of user U's facial expressions is carried out, for example, by identifying the type of facial expression of user U and emphasizing the expression to a degree corresponding to the degree of personal identification, or by combining a specific facial expression of the identified type with user U's facial expression at a ratio corresponding to the degree of personal identification. The emphasis of facial expressions is carried out, for example, by widening the opening of the mouth or exaggerating the movement of the eyebrows.

[0107] Furthermore, the processing of user U's gestures can be done, for example, by identifying the type of gesture used by user U and emphasizing the gesture to a degree corresponding to the degree of personal identification, or by combining a specific gesture of the identified type with user U's gesture at a ratio corresponding to the degree of personal identification. Gesture emphasis can be achieved, for example, by making arm movements larger or head movements more exaggerated.

[0108] The reproduction processing unit 33 performs voice processing on the utterances of user U, represented by user U's voice information, according to the degree of personal identification, to generate voice information for user U's avatar in the virtual space information. For example, the reproduction processing unit 33 converts user U's voice information into text information using speech recognition processing, and also uses speech recognition processing to identify user U's speaking style (volume, speed, intonation (emphasis), intervals between utterances (pauses), etc.) and voice quality.

[0109] The reproduction processing unit 33 identifies user U's dialect based on one or more of the character information converted by speech recognition processing and the speech patterns obtained by speech recognition processing. The reproduction processing unit 33 generates speech information in which user U's utterances are corrected to the specific dialect with a correction degree corresponding to the degree of personal identification.

[0110] For example, suppose user U's dialect is Osaka dialect, and a specific dialect is Tokyo dialect. In this case, when the reproduction processing unit 33 converts the text information converted from user U's voice information back into voice information using speech synthesis processing, it converts the Osaka dialect to Tokyo dialect with a correction degree according to the degree of personal identification.

[0111] For example, the reproduction processing unit 33 can perform a first speech synthesis process using Tokyo dialect, a second speech synthesis process that is strongly Tokyo dialect but also contains Osaka dialect, a third speech synthesis process that is strongly Osaka dialect but also contains Tokyo dialect, a fourth speech synthesis process using Osaka dialect, and so on. The unit then selects and executes a speech synthesis process from among these that matches the degree of personal identification.

[0112] In this case, the reproduction processing unit 33 selects the first speech synthesis process if the degree of personal identification is less than the first threshold, and selects the second speech synthesis process if the degree of personal identification is greater than or equal to the first threshold and less than the second threshold. Furthermore, the reproduction processing unit 33 selects the third speech synthesis process if the degree of personal identification is greater than or equal to the second threshold and less than the third threshold, and selects the fourth speech synthesis process if the degree of personal identification is greater than or equal to the third threshold. Note that the types of speech synthesis processes according to the degree of personal identification are not limited to the example described above, and may be five or more, or three or fewer.

[0113] Furthermore, the reproduction processing unit 33 can perform the above-described speech synthesis processing using speech synthesis parameters that have been corrected with a correction degree corresponding to the degree of personal identification of user U, specifically at least one of user U's speaking style and voice quality. Speech synthesis parameters include, for example, volume, speed, intonation (emphasis), utterance interval (pause), and voice quality.

[0114] The reproduction processing unit 33 determines the values ​​of various speech synthesis parameters corresponding to user U's speaking style and voice quality, and corrects the determined values ​​of the speech synthesis parameters by adding a correction value according to the degree of personal identification. Then, the reproduction processing unit 33 can generate speech information in which user U's speaking style and voice quality have been corrected to a degree of personal identification by performing a speech synthesis process using the various speech synthesis parameters corrected with the correction value according to the degree of personal identification.

[0115] Furthermore, the reproduction processing unit 33 can also reproduce in the virtual space the characteristics of user U's actions by correcting them to match the characteristics of a specific character's actions to a degree corresponding to the degree of personal identification. In this case, the specific character is a predetermined standard character, but if user U has specified a character, then the specific character is the character specified by user U.

[0116] The reproduction processing unit 33 reproduces User U's avatar in the virtual space by correcting User U's actions, which are indicated by User U's user information, by increasing the degree of correction as the degree of personal identification decreases, thereby bringing User U's actions closer to the character specified by User U.

[0117] Character characteristics are defined by character information, which includes, for example, a specific 3D body model, motion information, and a specific speech model. The 3D body model includes a head model, which includes the face and neck, and models of other parts of the body, such as the legs, ankles, waist, arms, and wrists. Motion information defines the movement of the 3D body model. For example, motion information includes motion information for each type of facial expression and motion information for each type of gesture.

[0118] The reproduction processing unit 33, for example, detects the positions of each feature point of user U from the captured image information of user U included in the user information, and generates a 3D body model of user U from these detection results. The reproduction processing unit 33 also identifies the movement of the 3D body model from the captured image information of user U, and identifies the type of facial expression and gesture of user U.

[0119] The reproduction processing unit 33 then synthesizes the movements of the user U's 3D body model and the character's 3D body model at a ratio corresponding to the degree of personal identification, and generates a corrected 3D body model of user U.

[0120] For example, the synthesis of the movements of the user U's 3D body model and the movements of the character's 3D body model at a ratio corresponding to the degree of personal identification includes the synthesis of facial expressions and gestures. For example, the reproduction processing unit 33 synthesizes the facial expressions of user U and the facial expressions of the 3D body model defined by motion information corresponding to the type of facial expression of user U at a ratio corresponding to the degree of personal identification. The reproduction processing unit 33 also synthesizes the gestures of user U and the gestures of the 3D body model defined by motion information corresponding to the type of gesture of user U at a ratio corresponding to the degree of personal identification.

[0121] In this way, the reproduction processing unit 33 determines the user U's actions from the captured image information of the user U, and reproduces the determined user U's actions in the virtual space using a corrected 3D model of the user U's body. As a result, the reproduction processing unit 33 can correct the user U's actions to match the actions of a specific character to a degree corresponding to the degree of personal identification, and reproduce them in the virtual space.

[0122] Furthermore, the reproduction processing unit 33 can correct user U's speech to match the speech of a specific character with a correction degree corresponding to the degree of personal identification, and reproduce it in the virtual space. The specific speech model performs speech synthesis processing using speech synthesis parameters corresponding to the specific character.

[0123] For example, the reproduction processing unit 33 determines, based on the voice information of user U, the values ​​of various speech synthesis parameters corresponding to user U's speaking style and voice quality as the values ​​of user U's speech synthesis parameters.

[0124] The reproduction processing unit 33 then adds the values ​​of the speech synthesis parameters of user U and the speech synthesis parameters of a specific character at a ratio corresponding to the degree of personal identification, thereby generating speech synthesis parameters corrected with a correction value corresponding to the degree of personal identification. The ratio corresponding to the degree of personal identification increases as the degree of personal identification decreases.

[0125] In this way, the reproduction processing unit 33 can generate speech information in which the speaking style and voice quality of user U have been corrected to a degree corresponding to the degree of individual identification by performing speech synthesis processing using various speech synthesis parameters that have been corrected with correction values ​​according to the degree of individual identification.

[0126] [3.3.5.Providing Department 34] The provisioning unit 34 transmits the virtual space information for each user U, generated by the reproduction processing unit 33, to the corresponding terminal device 2 among the multiple terminal devices 2 via the communication unit 10 and the network N.

[0127] For example, suppose that user U using the metaverse service is users U1, U2, and U3 of terminal devices 21, 22, and 23. In this case, the service provider 34 transmits virtual space information for user U1 to terminal device 21, virtual space information for user U2 to terminal device 22, and virtual space information for user U3 to terminal device 23.

[0128] The terminal device 21 receives virtual space information for user U1 from the provisioning unit 34 and displays an image of the virtual space based on the received virtual space information for user U1. The virtual space information for user U1 includes, for example, avatar information corrected according to the personal identification level set by user U2 for user U1, and avatar information corrected according to the personal identification level set by user U3 for user U1.

[0129] Furthermore, the terminal device 22 receives virtual space information for user U2 from the provision unit 34 and displays an image of the virtual space based on the received virtual space information for user U2. The virtual space information for user U2 includes, for example, avatar information corrected according to the personal identification level set by user U1 for user U2, and avatar information corrected according to the personal identification level set by user U3 for user U2.

[0130] Furthermore, the terminal device 23 receives virtual space information for user U3 from the provision unit 34 and displays an image of the virtual space based on the received virtual space information for user U3. The virtual space information for user U3 includes, for example, avatar information corrected according to the personal identification level set by user U1 for user U3, and avatar information corrected according to the personal identification level set by user U2 for user U3.

[0131] [4. Processing Procedure] Next, the procedure for information processing by the processing unit 12 of the information processing device 1 according to the embodiment will be described. Figure 6 is a flowchart showing an example of information processing by the processing unit 12 of the information processing device 1 according to the embodiment.

[0132] As shown in Figure 6, the processing unit 12 of the information processing device 1 determines whether or not it has received setting information from the terminal device 2 (step S10). If the processing unit 12 determines that it has received setting information (step S10: Yes), it stores the received setting information (step S11).

[0133] If the processing in step S11 is completed, or if it determines that it has not received the configuration information (step S10: No), the processing unit 12 determines whether or not it has received user information from the terminal device 2 (step S12). If the processing unit 12 determines that it has received user information (step S12: Yes), it generates user-specific virtual space information based on the user U's actions and degree of personal identification (step S13). Then, the processing unit 12 transmits the virtual space information generated in step S13 (step S14).

[0134] If the processing in step S14 is completed, or if it determines that user information has not been received (step S12: No), the processing unit 12 determines whether or not it is time to terminate the operation (step S14). For example, the processing unit 12 determines that it is time to terminate the operation if the power to the information processing device 1 is turned off, or if it determines that a termination operation has been performed by operating on an unillustrated control unit of the information processing device 1.

[0135] If the processing unit 12 determines that it is not yet time to terminate the operation (step S15: No), it proceeds to step S10. If it determines that it is time to terminate the operation (step S15: Yes), it terminates the process shown in Figure 6.

[0136] [5. Variations] In the example described above, the replay processing unit 33 detected the user U's actions using the user U's captured video information. However, the replay processing unit 33 is not limited to this example, and can also detect the user U's actions based on, for example, sensor information detected by a sensor attached to the user U, instead of or in addition to the user U's captured video information.

[0137] Furthermore, while the above example described a case where the virtual space is a three-dimensional virtual space, the virtual space may also be a two-dimensional space. In this case, the character information would include, for example, information about a two-dimensional body model instead of a three-dimensional body model.

[0138] [6. Hardware Configuration] The information processing device 1 according to the above embodiment is implemented by a computer 80 having a configuration such as that shown in Figure 7. Figure 7 is a hardware configuration diagram showing an example of a computer 80 that implements the functions of the information processing device 1 according to the embodiment. The computer 80 has a CPU 81, RAM 82, ROM (Read Only Memory) 83, HDD (Hard Disk Drive) 84, communication interface (I / F) 85, input / output interface (I / F) 86, and media interface (I / F) 87.

[0139] The CPU 81 operates based on programs stored in the ROM 83 or HDD 84, and controls various parts of the system. The ROM 83 stores boot programs executed by the CPU 81 when the computer 80 starts up, as well as programs that depend on the computer 80's hardware.

[0140] HDD84 stores programs executed by CPU81 and data used by such programs. The communication interface85 receives data from other devices via network N (see Figure 2) and sends it to CPU81, and transmits the data generated by CPU81 to other devices via network N.

[0141] The CPU 81 controls output devices such as displays and printers, and input devices such as keyboards and mice, via the input / output interface 86. The CPU 81 acquires data from input devices via the input / output interface 86. The CPU 81 also outputs data it has generated to output devices via the input / output interface 86.

[0142] The media interface 87 reads a program or data stored in the recording medium 88 and provides it to the CPU 81 via the RAM 82. The CPU 81 loads the program from the recording medium 88 onto the RAM 82 via the media interface 87 and executes the loaded program. The recording medium 88 can be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0143] For example, when computer 80 functions as an information processing device 1 according to the embodiment, the CPU 81 of computer 80 realizes the functions of processing unit 12 by executing a program loaded on RAM 82. Also, data from storage unit 11 is stored in HDD 84. The CPU 81 of computer 80 reads and executes these programs from recording medium 88, but as another example, these programs may be obtained from other devices via network N.

[0144] [7. Other] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0145] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0146] For example, the information processing device 1 described above may be implemented using multiple server computers, and the configuration can be flexibly changed, such as by calling external platforms via APIs or network computing depending on the function.

[0147] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent.

[0148] [8. Effects] As described above, the information processing device 1 according to this embodiment comprises a reception unit 31 and a reproduction processing unit 33. The reception unit 31 receives a setting of the degree of personal identification, which indicates the degree to which user U can be identified. The reproduction processing unit 33 corrects user U's actions based on the degree of personal identification received by the reception unit 31 and reproduces them in the virtual space. In this way, the information processing device 1 can improve the convenience of user U by fulfilling user U's request to not be identified by their actions in the virtual space using the degree of personal identification.

[0149] Furthermore, the reproduction processing unit 33 corrects and reproduces the user U's speech in the virtual space. This allows the information processing device 1 to improve the user U's convenience.

[0150] Furthermore, the reproduction processing unit 33 corrects user U's speech to a specific dialect and reproduces it in the virtual space. This allows the information processing device 1 to improve user U's convenience.

[0151] Furthermore, the reproduction processing unit 33 corrects and reproduces the user U's way of speaking in the virtual space. This allows the information processing device 1 to improve the user U's convenience.

[0152] Furthermore, the reproduction processing unit 33 corrects and reproduces the user U's gestures in the virtual space. This allows the information processing device 1 to improve the user U's convenience.

[0153] Furthermore, the reproduction processing unit 33 corrects and reproduces the user U's facial expressions in the virtual space. This allows the information processing device 1 to improve the user U's convenience.

[0154] Furthermore, the reproduction processing unit 33 reproduces user U in the virtual space, correcting it to the characteristics of a specific character to a degree corresponding to the degree of personal identification. This allows the information processing device 1 to improve the convenience for user U.

[0155] Furthermore, the information processing device 1 includes a determination unit 32 that determines a specific character based on the user U's behavioral history. This allows the information processing device 1 to improve user U's convenience.

[0156] Furthermore, the reception unit 31 accepts the setting of the degree of personal identification for each other user U to whom user U's actions in the virtual space will be presented. This allows the information processing device 1 to improve the convenience for user U.

[0157] Although embodiments of the present application have been described in detail based on the drawings, these are illustrative examples, and the present invention can be implemented in various other forms, including those described in the disclosure section of the invention, based on the knowledge of those skilled in the art.

[0158] Furthermore, the terms "section, module, unit" mentioned above can be replaced with "means" or "circuit," etc. For example, the acquisition unit can be replaced with acquisition means or acquisition circuit. [Explanation of symbols]

[0159] 1. Information Processing Device 2,21~2 n Terminal device 10 Communications Department 11 Storage section 12 Processing Units 20 User information storage unit 21 Character Information Storage Unit 30 Acquisition Department 31 Reception Department 32 Decision Section 33 Reproduction Processing Unit 34 Providing Department 100 Information Processing Systems

Claims

1. A reception desk that accepts settings for the degree to which a user can be identified, The system includes a reproduction processing unit that corrects the user's behavior based on the degree of personal identification received by the reception unit and reproduces it in a virtual space, The reproduction processing unit is, By combining the characteristics of the user's behavior and the characteristics of a specific character's behavior at a ratio corresponding to the degree of personal identification, the user's behavior is corrected and reproduced as the movements of the user's avatar in the virtual space. An information processing device characterized by the following:

2. The reproduction processing unit is, The user's speech is corrected and reproduced in the virtual space. The information processing apparatus according to feature 1.

3. The reproduction processing unit is, The user's speech is corrected to a specific dialect and reproduced in the virtual space. The information processing apparatus according to feature 2.

4. The reproduction processing unit is, The user's way of speaking is corrected and reproduced in the virtual space. The information processing apparatus according to claim 2 or 3, characterized by the above.

5. The reproduction processing unit is, The user's gestures are corrected and reproduced in the virtual space. The information processing apparatus according to any one of claims 1 to 3.

6. The reproduction processing unit is, The user's facial expression is corrected and reproduced in the virtual space. The information processing apparatus according to any one of claims 1 to 3.

7. The reproduction processing unit is The system generates corrected speech synthesis parameters by adding the values ​​of the speech synthesis parameters determined based on the user's voice information and the values ​​of the speech synthesis parameters of a specific character, in a ratio where the weight of the specific character's speech synthesis parameters increases as the degree of personal identification decreases. The system then uses these corrected speech synthesis parameters to perform speech synthesis processing and generate the avatar's voice. The information processing apparatus according to any one of claims 1 to 3.

8. The system includes a determination unit that determines the specific character based on the user's behavior history. The information processing apparatus according to feature 7.

9. The aforementioned reception unit is The system accepts the setting of the degree of personal identification for each other user to whom the aforementioned actions in the virtual space are presented. The information processing apparatus according to any one of claims 1 to 3.

10. A method of information processing performed by a computer, A registration process that accepts the setting of the degree to which a user can be identified, The reproduction process includes correcting the user's behavior in a virtual space based on the degree of personal identification received in the reception process, The reproduction process described above is: By combining the characteristics of the user's behavior and the characteristics of a specific character's behavior at a ratio corresponding to the degree of personal identification, the user's behavior is corrected and reproduced as the movements of the user's avatar in the virtual space. An information processing method characterized by the following:

11. A procedure for accepting user preferences that indicate the degree to which a user can be identified, The computer is instructed to execute a reproduction process procedure that corrects the user's actions in a virtual space based on the degree of personal identification received through the aforementioned reception procedure, The reproduction procedure described above is: By combining the characteristics of the user's behavior and the characteristics of a specific character's behavior at a ratio corresponding to the degree of personal identification, the user's behavior is corrected and reproduced as the movements of the user's avatar in the virtual space. An information processing program characterized by the following features.

Images