Information processing system, program, and information processing method

Abstract

[Problem] To enhance the quality of experience for users using a virtual space. [Solution] This information processing system includes a control unit that executes: control for determining whether first control information that reflects input information for controlling a character in a virtual space satisfies a conversion condition; and, if it is determined that the conversion condition is satisfied, control for generating second control information for controlling the character, obtained by converting at least a part of the first control information on the basis of conversion information corresponding to the conversion condition.

Description

Information processing system, program, and information processing method

[0001] The present disclosure relates to an information processing system, a program, and an information processing method.

[0002] In recent years, characters in virtual spaces have been controlled based on various input information. For example, Patent Document 1 below discloses a technology for controlling the behavior of a pet avatar (VR pet), which is a character in a virtual space, using information about a pet robot in real space as input information.

[0003] In the technology disclosed in Patent Document 1, for example, the behavior of a VR pet is determined according to attribute information of the pet robot and input information such as a call from the user who is the owner of the pet robot.

[0004] International Publication No. 2020 / 095368

[0005] On the other hand, it is also possible to control a character in a virtual space by reflecting the motion or sound indicated by the input information. However, there is a possibility that the situation in which the user inputting the input information is placed may not be appropriate for obtaining the input information, or that the motion or sound indicated by the input information may not be appropriate for the user to whom the input information is output in the virtual space.

[0006] Therefore, the present disclosure proposes new and improved technology that can improve the quality of experience of users who use virtual spaces.

[0007] According to the present disclosure, an information processing system is provided that includes a control unit that executes: a control to determine whether first control information that reflects input information and controls a character in a virtual space satisfies a conversion condition; and a control to generate second control information for controlling the character, which converts at least a portion of the first control information based on conversion information corresponding to the conversion condition, if it is determined that the conversion condition is satisfied.

[0008] In addition, according to another aspect of the present disclosure to solve the above problem, a program is provided that causes a computer to function as a control unit that executes control to determine whether first control information that reflects input information and controls a character in a virtual space satisfies a conversion condition, and control to generate second control information for controlling the character by converting at least a portion of the first control information based on conversion information corresponding to the conversion condition when it is determined that the conversion condition is satisfied.

[0009] In addition, according to another aspect of the present disclosure to solve the above problem, there is provided an information processing method executed by a computer, which includes determining whether first control information reflecting input information that controls a character in a virtual space satisfies a conversion condition, and if it is determined that the conversion condition is satisfied, generating second control information for controlling the character by converting at least a portion of the first control information based on conversion information that corresponds to the conversion condition.

[0010] 1 is a diagram for describing an overall configuration of an information processing system 1 according to an embodiment of the present disclosure. FIG. 1 is a block diagram illustrating an example of a configuration of a user terminal 10 according to this embodiment. FIG. 2 is a block diagram illustrating an example of a configuration of a server 20 according to this embodiment. FIG. 3 is a diagram illustrating an example of an expression conversion table set for each user. FIG. 4 is a diagram illustrating an example of a primary expression table whose type is motion. FIG. 5 is a diagram illustrating an example of a primary expression table whose type is audio. FIG. 6 is a diagram illustrating an example of a secondary expression table. FIG. 7 is a diagram illustrating an example of an editing screen for the expression conversion table displayed on the user terminal 10. FIG. 8 is a diagram illustrating an example of an editing screen for the expression conversion table displayed on the user terminal 10. FIG. 9 is a diagram illustrating an example of an image of a virtual space displayed on the user terminal 10. FIG. 10 is a diagram representing an example of a history table including an editing history and a reference history of the expression conversion table. FIG. 11 is a flowchart illustrating an example of a flow of operation processing of the server 20 according to an embodiment of the present disclosure. FIG. 12 is a diagram illustrating a first NN. FIG. 13 is a diagram illustrating a second NN. FIG. 14 is a diagram illustrating an example of converting a primary expression into a secondary expression according to a user's preference by combining a first NN 1000 and a second NN 2000. 1 is a diagram showing an example of an editing screen of an expression conversion table displayed on a user terminal 10, which accepts a user's selection of a preferred video. FIG. 2 is a block diagram showing an example of an information processing device 90.

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] The explanation will be given in the following order: 1. Overview 2. Configuration example 3. Operation processing example 4. Modification 5. Hardware configuration example 6. Supplementary information

[0013] <<1. Overview>> First, the overall configuration of an information processing system according to an embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a diagram for describing the overall configuration of an information processing system 1 according to an embodiment of the present disclosure.

[0014] 1, an information processing system 1 according to an embodiment of the present disclosure includes user terminals 10 (user terminals 10A, 10B, 10C, etc.) and a server 20. As shown in FIG. 1, each user terminal 10 is configured to be able to communicate with the server 20 via a network 30. Hereinafter, when there is no need to particularly distinguish between the user terminals 10A to 10C, they will be collectively referred to as user terminals 10.

[0015] In this embodiment, an example will be mainly described in which a user using each user terminal 10 operates an avatar corresponding to each user in a virtual space provided by the server 20. The avatar in the virtual space is an example of a character in the virtual space according to this embodiment.

[0016] A virtual space is a space that is constructed based on a program and is visible to each user, and may be, for example, a space developed by social virtual reality (VR) or a space provided in a metaverse game. Note that the virtual space may be a space that is superimposed on a physical space (real space).

[0017] The avatar may be, for example, a 2D or 3D character. The character may be a humanoid character modeled after the user, or may be a character such as an animal or a fictional creature. In the following description, an example in which the avatar is controlled based on input information input by the user of the user terminal 10 will be mainly described. However, the avatar may also be controlled based on other input information, for example, based on information automatically generated by the server 20.

[0018] (User terminal 10) The user terminal 10 is an information processing terminal used by a user. The user terminal 10 is realized by, for example, a smartphone, a tablet terminal, a personal computer, an HMD (Head Mounted Display) for AR (Augmented Reality), VR (Virtual Reality), or XR (Extended Reality) that covers the entire field of view, a glasses-type device, a projector, or a console game machine.

[0019] The user terminal 10 acquires input information input by the user to operate the avatar. The user terminal 10 may acquire, for example, user motion data as input information. The motion data may be, for example, coordinates and rotation angles of each skeletal point of the user in physical space acquired by a wearable device worn by the user. In this case, the motion of the avatar may be controlled in real time based on the input information. The user terminal 10 may, for example, connect to a wearable device worn by the user and acquire the motion data as input information.

[0020] The user terminal 10 may also acquire user voice data as input information. The user voice data may be acquired from a microphone or the like connected to the user terminal 10.

[0021] The user terminal 10 transmits the acquired input information to the server 20 .

[0022] Furthermore, the user terminal 10 outputs the virtual space constructed by the server 20 to the user. An avatar operated by the user of the user terminal 10 is placed in the virtual space constructed by the server 20. For example, the user terminal 10 may receive from the server 20 an image of the virtual space generated by the server 20 and display it to the user. The image of the virtual space may also include audio, and in this case, the user terminal 10 may receive the audio from the server 20 and output it to the user.

[0023] (Server 20) The server 20 constructs and manages a virtual space in which avatars operated by users of each user terminal 10 are placed, and provides virtual space information to one or more user terminals 10 that are connected to the server via the network 30.

[0024] The server 20 receives input information input by each user from each user terminal 10. The server 20 may generate first control information based on the received input information. The first control information is information for controlling an avatar that reflects the input information.

[0025] For example, when motion data is input information, the first control information may be coordinates and rotation angles of each skeleton point of an avatar in a virtual space, which are obtained by transforming each skeleton point of the user, as represented by the motion data. By controlling the avatar using such first control information, the motion of the user, who is the operating subject, is reproduced as the motion of the avatar in the virtual space.

[0026] The first control information may be generated by the user terminal 10. For example, the user terminal 10 may generate digital voice data as the first control information from analog voice data acquired as input information. The server 20 controls the voice data received from the user terminal 10 and generated as the first control information to be reflected as voice uttered by an avatar, thereby reproducing the voice of each user as the voice of an avatar in the virtual space.

[0027] (Summary of Issues) When an avatar is controlled by the above-described first control information, the situation of the user who inputs the input information is also important. In other words, if the situation is not appropriate for generating the first control information expected by the user, the avatar may not be operated satisfactorily.

[0028] For example, it is conceivable that even if a user wants to operate an avatar to raise both arms vertically, the user is unable to raise both arms vertically due to physical constraints in the physical space (e.g., the user is injured), and input information indicating that both arms should be extended parallel to the shoulders is input. In this case, according to the first control information generated from the input information, the avatar is controlled to extend both arms parallel to the shoulders.

[0029] Furthermore, even if a user wants to have an avatar participate in a dance party held in a virtual space and have the avatar perform vigorously swaying movements, it is conceivable that the user is unable to perform vigorously swaying movements due to environmental constraints in the physical space (e.g., the user is in a public place), and so input information representing small swaying movements is input. In this case, the avatar is controlled to perform small swaying movements according to the first control information generated from the input information.

[0030] As in the above example, if the situation in which a user inputs input information is not suitable for generating the first control information that the user expects, the behavior of the avatar reproduced by the first control information will deviate from the behavior that the user expects, which may result in a decrease in the quality of the user's experience in the virtual space.

[0031] Furthermore, depending on the content of the input information, a user viewing an avatar controlled by the above-described first control information may feel uncomfortable.

[0032] For example, it is assumed that the motion indicated by the input information is a gesture that is generally considered aggressive (e.g., a gesture of raising the middle finger), and in this case, based on the first control information generated from the input information, the avatar is controlled to perform an aggressive gesture that is inappropriate in a virtual reality platform.

[0033] Furthermore, it is possible that input information indicating a gesture that is acceptable in one region may be inappropriate in another region. For example, a peace sign that is well-received in Japan may be considered offensive in Greece. In this case, a Greek user viewing an avatar that is controlled to make a peace sign by the first control information generated from the input information may be offended.

[0034] As in the example described above, if the input information causes the avatar's behavior to be inappropriate for the user to whom the virtual space is output, it may degrade the quality of the virtual space experience for the user viewing the avatar.

[0035] Therefore, the server 20 according to this embodiment performs a process of appropriately converting the first control information. That is, the server 20 first determines whether the first control information satisfies a preset conversion condition. If the server 20 determines that the conversion condition is satisfied, the server 20 generates second control information for controlling an avatar by converting at least a portion of the first control information based on conversion information corresponding to the conversion condition. Controlling the avatar using the second control information makes it possible to reflect appropriate motions, sounds, etc. in the avatar, thereby improving the quality of the user's experience in the virtual space.

[0036] The first control information represents an expression in a virtual space that reflects input information on an avatar, and therefore, hereinafter, the expression reflected on the avatar by controlling the avatar with the first control information is also referred to as a "primary expression." Furthermore, the second control information represents an expression in a virtual space that is a conversion of the primary expression, and therefore, hereinafter, the expression reflected on the avatar by controlling the avatar with the second control information is also referred to as a "secondary expression." Furthermore, hereinafter, the first control information may be simply referred to as a "primary expression," and the second control information may be simply referred to as a "secondary expression."

[0037] <<2. Configuration Example>> <2-1. Configuration Example of User Terminal 10> Fig. 2 is a block diagram showing an example of the configuration of the user terminal 10 according to this embodiment. As shown in Fig. 2, the user terminal 10 has a communication unit 110, an input unit 120, a display unit 130, an audio output unit 140, a storage unit 150, and a control unit 160.

[0038] (Communication Unit 110) The communication unit 110 communicates with the server 20 via wired or wireless communication to exchange information. The communication unit 110 can communicate using, for example, a wired / wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), infrared communication, or a mobile communication network (4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system)), etc.

[0039] For example, the communication unit 110 transmits input information input to the input unit 120 to the server 20 .

[0040] (Input Unit 120) The input unit 120 accepts operation instructions from a user. For example, the input unit 120 acquires operations on an avatar corresponding to the user as input information. The input unit 120 may acquire input information by connecting to devices such as a keyboard, a mouse, a game controller, an RGB camera, an infrared camera, a microphone, an inertial sensor, or a wearable device equipped with any of these. Note that each device may be configured integrally with the user terminal 10. For example, if the user terminal 10 is configured as an HMD, the microphone, the inertial sensor, etc. may be configured integrally with the user terminal 10.

[0041] The input information may be, for example, the type of button and the button press state, the coordinates of each skeleton point of the user, motion data such as rotation angle, acceleration, and angular velocity, RGB images, infrared images, and audio. The input unit 120 may be connected to multiple devices to acquire various types of input information.

[0042] The input information acquired by the input unit 120 may be transmitted in real time to the server 20 by the communication unit 110. For example, in the case where a service such as an online game in which a virtual space is generated in real time by the server 20 and output to the user is provided to the user, the input information is transmitted in real time to the server 20. However, depending on the service provided to the user, the input information acquired by the input unit 120 may be controlled so that it is stored in the storage unit 150 and then transmitted to the server 20 by the communication unit 110. For example, in the case where an animated video representing the state of the virtual space is generated by the server 20, the input information may be stored in the storage unit 150 and then transmitted to the server 20 by the communication unit 110.

[0043] (Display Unit 130) The display unit 130 displays an image under the control of the control unit 160. For example, the display unit 130 may be a display panel such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display.

[0044] The display unit 130 outputs, for example, an image of the virtual space received by the communication unit 110 to the user.

[0045] (Audio Output Unit 140) The audio output unit 140 outputs audio under the control of the control unit 160. For example, when the user terminal 10 is an HMD, the audio output unit 140 may be configured as headphones worn on the user's head, or may be realized by earphones or a bone conduction speaker.

[0046] The audio output unit 140 outputs, for example, the audio of the virtual space received by the communication unit 110 to the user.

[0047] (Storage unit 150) The storage unit 150 is realized by a ROM that stores programs and calculation parameters used in the processing of the control unit 160, and a RAM that temporarily stores parameters that change as appropriate. The storage unit 150 stores, for example, a client application that executes various processes related to the virtual space.

[0048] (Control Unit 160) The control unit 160 functions as an arithmetic processing unit and a control device, and controls the overall operation of the user terminal 10 in accordance with various programs. The control unit 160 is realized by electronic circuits such as a CPU (Central Processing Unit) or a microprocessor. The control unit 160 may also include a ROM (Read Only Memory) that stores programs to be used, arithmetic parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0049] For example, the control unit 160 controls the communication unit 110 to transmit input information input to the input unit 120 to the server 20. The control unit 160 also controls the display unit 130 to display an image of the virtual space received by the communication unit 110 from the server 20. The control unit 160 also controls the audio output unit 140 to output audio corresponding to the image of the virtual space received by the communication unit 110 from the user terminal 10.

[0050] 3 is a block diagram showing an example of the configuration of the server 20 according to this embodiment. As shown in Fig. 3, the server 20 includes a communication unit 205, a primary representation generation unit 210, a secondary representation generation determination unit 215, a secondary representation generation unit 220, a representation conversion information storage unit 225, a representation conversion information setting unit 230, a virtual space reflection unit 235, a virtual space construction information storage unit 240, and a virtual space construction unit 245.

[0051] (Communication Unit 205) The communication unit 205 communicates with and exchanges various information with the user terminal 10. The communication unit 205 communicates with the user terminal 10 using, for example, a wired / wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (4G (fourth generation mobile communication method) or 5G (fifth generation mobile communication method)), or the like.

[0052] The communication unit 205 functions as, for example, an acquisition unit that acquires input information corresponding to each avatar from the user terminal 10. The communication unit 205 also transmits an image of the virtual space constructed by the virtual space construction unit 245 to the user terminal 10.

[0053] (Primary Representation Generation Unit 210) The primary representation generation unit 210 generates a primary representation that reflects the input information. If the input information is continuous information, the primary representation generation unit 210 may divide the input information and generate a primary representation from each piece of divided input information.

[0054] If the input information is motion data, the primary representation may be control information for reflecting the motion indicated by the motion data on the avatar. More specifically, the primary representation may be the coordinates and rotation angles in virtual space of each skeleton point set on the avatar.

[0055] For example, when the primary representation generation unit 210 receives, as input information via the communication unit 205, the coordinates and rotation angle of the user's right wrist and right elbow in real space, it references the information on the right wrist and right elbow defined for the avatar stored in the virtual space construction information storage unit 240. The primary representation generation unit 210 then compares the length from the user's right wrist to the right elbow obtained based on the input information with the length from the right wrist to the right elbow of a skeleton point defined for the avatar. Based on the comparison result, the primary representation generation unit 210 may then convert the coordinates and rotation angle of the user's right wrist and right elbow in real space obtained as input information into the position coordinates and rotation angle of the avatar's skeleton point in virtual space.

[0056] Furthermore, if the input information is motion data or an RGB image, the primary representation may be control information for reflecting on the avatar the motion indicated by the motion data or the motion performed by the user depicted in the RGB image.

[0057] Furthermore, if the input information is the type of button and the pressed state of the button, the primary expression may be control information for reflecting a motion or sound corresponding to the button on the avatar.

[0058] Furthermore, if the input information is voice data, the primary representation may be control information for reflecting the voice represented by the voice data in the avatar. In other words, the primary representation may be the same data as the voice data that is the input information.

[0059] Furthermore, the primary representation is not limited to control information for reflecting motion or sound on the avatar, but may also be the coordinates of grid points set on the avatar's face to configure the avatar's facial expression, or texture data for expressing the avatar's color tone.

[0060] The primary representation generation unit 210 outputs the generated primary representation to the secondary representation generation determination unit 215. The primary representation generation unit 210 may output the primary representation to the secondary representation generation determination unit 215 in real time. For example, when a service such as an online game is provided to a user in which a virtual space is generated in real time by a virtual space construction unit 245 (described later) and output to the user, the primary representation generation unit 210 outputs the primary representation to the secondary representation generation determination unit 215 in real time. However, depending on the service provided to the user, the primary representation generation unit 210 may store the primary representation in, for example, a virtual space construction information storage unit 240 (described later). For example, when an animated video representing the state of a virtual space is generated by the server 20, the primary representation may be stored in the virtual space construction information storage unit 240 and then output to the secondary representation generation determination unit 215. In this way, the secondary representation generation determination unit 215 determines whether to generate a secondary representation at an appropriate time depending on the situation at the time of animation generation, for example, the situation regarding the determination of countries to which the animation video will be distributed.

[0061] (Secondary Representation Generation Determination Unit 215) The secondary representation generation determination unit 215 determines whether or not the primary representation obtained from the primary representation generation unit 210 satisfies a conversion condition. The conversion condition is a condition that is set for determining whether or not to convert the primary representation into a secondary representation.

[0062] The conversion conditions are included in an expression conversion table stored in the expression conversion information storage unit 225 (described later). The expression conversion table includes the conversion conditions and conversion information corresponding to the conversion conditions. The expression conversion table will be described in detail later.

[0063] If the secondary representation generation determination unit 215 determines that the primary representation satisfies the conversion conditions, it outputs the primary representation and conversion information corresponding to the conversion conditions to the representation conversion information storage unit 225. Furthermore, if the secondary representation generation determination unit 215 determines that the primary representation does not satisfy the conversion conditions, it outputs the primary representation to the virtual space reflection unit 235.

[0064] The conversion condition may be, for example, that the primary expression contains a predetermined expression. The predetermined expression may be a predetermined motion or a predetermined sound. The predetermined expression may be a motion or sound that is inappropriate for the service, such as an insulting or violent motion or sound.

[0065] The conversion condition may also be a condition related to the environment of the virtual space in which the avatar reflecting the primary representation is placed. The condition related to the environment of the virtual space may be, for example, that a predetermined event is taking place in the virtual space.

[0066] Furthermore, when the primary expression is information for controlling a voice corresponding to an avatar, the conversion condition may be a condition regarding the relationship between the speech speed represented by the voice and a reference value. For example, the conversion condition may be that the speech speed is 10 words per second or more.

[0067] The secondary representation generation determination unit 215 may have a function of analyzing the primary representation when determining whether the primary representation satisfies the conversion condition. For example, if the conversion condition is that the primary representation includes a predetermined expression, the secondary representation generation determination unit 215 may analyze whether the primary representation includes the predetermined expression. For example, the secondary representation generation determination unit 215 may analyze whether the primary representation includes a predetermined motion by analyzing skeleton points obtained as the primary representation using an analytical model.

[0068] Furthermore, the secondary representation generation determination unit 215 may determine whether each primary representation satisfies each of a plurality of conversion conditions. After making a determination for each of the plurality of conversion conditions, the secondary representation generation determination unit 215 outputs conversion information corresponding to the conversion condition that is satisfied by the primary representation, out of the plurality of conversion conditions, to the representation conversion information storage unit 225.

[0069] Furthermore, priorities may be set for the multiple conversion conditions. When the primary expression satisfies multiple conversion conditions, the secondary expression generation determination unit 215 may output conversion information corresponding to a conversion condition with a higher priority to the expression conversion information storage unit 225. This makes it possible to output only appropriate conversion information to the expression conversion information storage unit 225 when the primary expression satisfies multiple conversion conditions and the conversion information corresponding to the multiple conversion conditions is contradictory.

[0070] Furthermore, the secondary representation generation determination unit 215 may determine whether each primary representation satisfies only a predetermined number of conversion conditions, in descending order of priority, among the multiple conversion conditions. This reduces the time required for processing by the secondary representation generation determination unit 215 when a large number of conversion conditions are set. By reducing the time required for such processing, it is possible to reduce the delay time between when the user terminal 10 acquires input information and when the image of the virtual space is output to the user terminal 10 in real time by the server 20. This improves the quality of the user experience.

[0071] The conversion conditions may also include conditions regarding the avatar in which the primary representation is reflected. For example, the conversion conditions may include an ID for identifying the avatar in which the primary representation is reflected.

[0072] The conversion conditions may also include conditions regarding the attributes of the user to whom the virtual space is output. The user attributes may be, for example, the user's place of residence, place of origin, nationality, or age.

[0073] Note that one conversion condition may include multiple conditions described above as conversion conditions. For example, the conversion condition may be that the primary expression includes a predetermined expression and that the attribute of the user to whom the virtual space is output is a predetermined attribute. More specifically, the conversion condition may be that the primary expression includes having the avatar make a peace sign and that the residence, birthplace, or nationality of the user to whom the virtual space is output is Greece.

[0074] The conversion condition may also include a condition that the user has given permission to convert the primary representation into the secondary representation. In this case, the conversion condition includes multiple conditions, including a condition that the user has given permission to convert the primary representation into the secondary representation.

[0075] The secondary representation generation determination unit 215 may perform control so as to request permission from the user to convert the primary representation into a secondary representation when the primary representation satisfies any of the multiple conditions included in the conversion conditions except for the user's permission to convert the primary representation into a secondary representation. For example, permission for conversion may be requested from the user who input the input information from which the primary representation was generated, i.e., the user who operates the avatar in which the primary representation or the secondary representation is reflected. Permission for conversion may also be requested from the user to whom the virtual space is output.

[0076] The secondary representation generation determination unit 215 may output information to the virtual space reflection unit 235 to generate an image of the virtual space including a display for accepting a request for permission to convert. When the user of the user terminal 10 performs a permission selection operation, information indicating permission is transmitted from the user terminal 10 to the server 20. Then, when the secondary representation generation determination unit 215 acquires the information indicating permission, it determines that the primary representation satisfies the conversion conditions.

[0077] On the other hand, if the secondary expression generation judgment unit 215 does not obtain information indicating permission (for example, if it receives information indicating non-permission or if a predetermined time has passed since sending information indicating permission), it judges that the primary expression does not satisfy the conversion conditions.

[0078] (Secondary representation generation unit 220) When the secondary representation generation determination unit 215 determines that the primary representation satisfies the conversion conditions, the secondary representation generation unit 220 converts at least a portion of the primary representation based on the conversion information corresponding to the conversion conditions output by the secondary representation generation determination unit 215.

[0079] The conversion information is included in an expression conversion table stored in the expression conversion information storage unit 225. Details of the conversion information will be described later with reference to an example of an expression conversion table.

[0080] (Expression Conversion Information Storage Unit 225) The expression conversion information storage unit 225 stores an expression conversion table including conversion conditions and conversion information corresponding to the conversion conditions. The expression conversion table may store multiple combinations of conversion conditions and conversion information. The expression conversion information storage unit 225 may also store an expression conversion table set for each user.

[0081] Next, examples of conversion conditions and conversion information will be described with reference to an example of an expression conversion table shown in Fig. 4. Fig. 4 is a diagram showing an example of an expression conversion table set for each user. The expression conversion table stores combinations of primary expressions and secondary expressions, including conversion conditions and conversion information.

[0082] 4, the expression conversion table includes a conversion ID, a priority, a target primary expression, a target secondary expression, and a remark. The conversion ID is an identification number for identifying a combination of a primary expression and a secondary expression. The priority is a numerical value representing the priority of the determination by the secondary expression generation determination unit 215. The remark is management data that is not directly used in the processing by the secondary expression generation determination unit 215 and the secondary expression generation unit 220, such as the intention behind the combination of a primary expression and a secondary expression.

[0083] The target primary expression is information indicating the conversion condition in the example shown in Fig. 4. The secondary expression generation determination unit 215 determines whether the primary expression acquired from the communication unit 205 corresponds to the target primary expression, thereby determining whether the primary expression satisfies the determination condition.

[0084] The target primary expression includes a type, an expression ID, a determination basis, an execution subject, and a conversion permission. The type is the type of the expression, and may be, for example, motion or sound as shown in FIG.

[0085] The expression ID is an identification number for identifying an expression. The expression conversion information storage unit 225 stores each expression ID and a primary expression table indicating the content of the expression represented by each expression ID. An example of the primary expression table will now be described with reference to FIGS. 5 and 6.

[0086] 5 is a diagram showing an example of a primary expression table whose type is motion. The primary expression table shown in Fig. 5 shows expression IDs whose type is motion, the content of expressions corresponding to the expression IDs, and the last update dates and times.

[0087] For example, the expression ID "0002" indicates that the skeleton point pattern identified by the analytical model AA is a "piece."

[0088] Furthermore, the expression ID may indicate that no specific expression is specified, such as expression ID "0001."

[0089] The last update date and time indicates the date and time when the expression ID and its contents were registered or updated in the primary expression table. Note that it is assumed that the expression ID and its contents are registered and updated by the system administrator of the virtual space.

[0090] 6 is a diagram showing an example of a primary expression table for which the type is audio. The primary expression table shown in Fig. 6 shows expression IDs for which the type is audio, the content of expressions corresponding to the expression IDs, and the last update dates and times of the expressions.

[0091] For example, the expression ID "0002" indicates that the emotion label of the voice identified by the analysis model EE is "happy."

[0092] The secondary representation generation determination unit 215 determines whether the primary representation includes a predetermined representation by determining whether the primary representation includes a representation represented by a representation ID. For example, when determining whether the primary representation satisfies the target primary representation with conversion ID "0002" shown in Figure 4, the secondary representation generation determination unit 215 analyzes the primary representation using analysis model AA. Then, when the analysis result obtained indicates that the skeleton point pattern is "piece" in at least a part of the primary representation, the secondary representation generation determination unit 215 determines that the primary representation is a representation represented by a representation ID.

[0093] The details of the expression ID included in the target primary expression have been described above. Next, the determination basis included in the target primary expression will be described with reference to FIG. 4 again.

[0094] The determination basis indicates the conditions under which a primary expression corresponds to the target primary expression, and may include conditions related to the environment of the virtual space in which the avatar reflecting the primary expression is placed, conditions related to the relationship between the speech speed and the reference value, conditions related to the attributes of the user, etc.

[0095] The basis for the determination may also include conditions regarding the avatar included in the virtual space output to the user for which the expression conversion table is set. For example, the determination basis for conversion ID "0004" "when executed by user within field of view" indicates that the avatar corresponding to the primary expression is an avatar included in the virtual space output to the user for which the expression conversion table is set (i.e., included in the field of view of the user).

[0096] The execution subject is a condition regarding the operator of the avatar corresponding to the primary representation. If the execution subject is "the user himself / herself," it is determined as part of the conversion conditions that the operator of the avatar corresponding to the primary representation is the user for whom the representation conversion table is set.

[0097] Furthermore, if the executing entity is "another user," it is determined as part of the conversion conditions that the operating entity of the avatar corresponding to the primary representation placed in the virtual space output to the user for whom the representation conversion table is set is a user different from the user. Note that the executing entity may include a user ID or the like for identifying a specific user.

[0098] Further, other conditions may be set as conditions for the operator of the avatar corresponding to the primary representation. For example, the other conditions may be that the operator of the avatar corresponding to the primary representation, which is placed in the virtual space output to the user for whom the representation conversion table is set, is a user who is registered as a friend of the user, or a user who is not registered as a friend.

[0099] The conversion permission indicates whether the conversion condition includes permission from the user to convert the primary representation into a secondary representation. For example, if conversion permission is "required," the conversion condition includes permission from the user to convert the primary representation into a secondary representation. If the secondary representation generation determination unit 215 determines that the primary representation satisfies the representation ID, determination basis, and execution subject, and conversion permission is "required," it may control the communication unit 205 to request permission from the user to convert the primary representation into a secondary representation.

[0100] The target primary expression included in the expression conversion table has been described above. Next, the target secondary expression will be described. The target secondary expression is information that indicates the secondary expression to which the primary expression is converted when the primary expression corresponds to the target primary expression. When the primary expression corresponds to the target primary expression, the secondary expression generation determination unit 215 outputs the primary expression, as well as the type and expression ID of the target primary expression included in the expression conversion table, and the target secondary expression to the secondary expression generation unit 220. The type and expression ID of the target primary expression included in the expression conversion table, and the target secondary expression are examples of conversion information.

[0101] The type is the type of transformation, and may be, for example, suppression, modification, or emphasis, as shown in Fig. 4. Suppression means suppressing at least a part of the primary representation to generate the secondary representation. Modification means changing at least a part of the primary representation to generate the secondary representation. Emphasis means emphasizing at least a part of the primary representation to generate the secondary representation.

[0102] The expression ID is an identification number for identifying the expression to be converted. The expression conversion information storage unit 225 stores each expression ID and a secondary expression table indicating the content of the expression represented by each expression ID. An example of the secondary expression table will now be described with reference to FIG. 7.

[0103] 7 is a diagram showing an example of a secondary expression table, which includes a type, an expression ID, and the content and details of the expression corresponding to the expression ID.

[0104] The content and details represent a method for converting a primary expression. The secondary expression generation unit 220 generates a secondary expression by converting the primary expression, among the primary expressions, the expression indicated by the type of the target primary expression and the expression ID, in accordance with the content and details of the secondary expression table.

[0105] Although the example shown here is one in which the content and details in the secondary representation table are human-interpretable natural language information, the content and details in the secondary representation table may be described in other ways. For example, expressions that cannot be described in natural language may be defined using images or skeleton points, etc.

[0106] The above has described the details of the expression conversion table. As described above, the secondary expression generation determination unit 215 and the secondary expression generation unit 220 use the expression conversion table to perform the process of determining whether or not a primary expression satisfies the conversion condition, and the process of converting a primary expression into a secondary expression, respectively.

[0107] The secondary expression generation determination unit 215 performs a process of determining the primary expression corresponding to the user's avatar and the avatar to be placed in the virtual space provided to the user for each user for whom an expression conversion table has been set.

[0108] As mentioned in the explanation of the secondary representation generation determination unit 215, the secondary representation generation determination unit 215 determines whether or not a primary representation corresponds to each target primary representation based on priority. The representation conversion table stores multiple combinations of target primary representations and target secondary representations. Therefore, it is possible that multiple combinations of target primary representations and target secondary representations may have contradictory content.

[0109] For example, a target primary representation may be a predetermined motion indicating an expression to be avoided in a certain area, and a target secondary representation may be registered indicating that the predetermined motion should be converted into another motion. However, in order to more emphasize the predetermined motion according to the user's preferences, a target secondary representation may be registered that uses the predetermined motion as a target primary representation and further emphasizes the target primary representation to perform the predetermined motion. In this case, multiple combinations of target primary representations and target secondary representations are inconsistent. Therefore, the secondary representation generation determination unit 215 can output an appropriate target secondary representation to the secondary representation generation unit 220 even when a contradictory combination exists by determining that the primary representation corresponds to a target primary representation in a combination set with a higher priority among multiple combinations.

[0110] The secondary representation generation unit 220 generates a secondary representation according to the determination result by the secondary representation generation determination unit 215. For example, the secondary representation generation unit 220 converts a predetermined motion (pose) into a different motion (pose) or disables it according to the determination result by the secondary representation generation determination unit 215. More specifically, the secondary representation generation determination unit 215 references the conversion ID "0001" in the expression conversion table shown in FIG. 4 to determine whether a peace sign motion is included in the motions serving as primary expressions for controlling the avatar of the user for whom the expression conversion table is set. Furthermore, the secondary representation generation determination unit 215 determines whether a user of Greek nationality is included in the virtual space in which the avatar of the user for whom the expression conversion table is set exists. Based on these determination results, the secondary representation generation unit 220 disables the peace sign motion in the primary representation and generates a secondary representation.

[0111] Note that the representation to be used when the representation is disabled may be stored in advance in the representation conversion information storage unit 225. For example, when a motion is disabled, the secondary representation generation unit 220 may cause the avatar to perform a motion of spreading its palms. In order to change the peace sign motion to a gesture of spreading its palms, the secondary representation generation determination unit 215 converts the coordinates and rotation angles of the skeletal points of the fingers of the hand that make up the peace sign, among the information on the skeletal points of the avatar that is making the peace sign, into the coordinates and rotation angles of the skeletal points when the palm is spread. Note that the secondary representation generation determination unit 215 maintains the coordinates and rotation angles of the skeletal points other than those of the hand that makes up the peace sign as they are represented by the primary representation.

[0112] The secondary expression generation unit 220 may have a function of further analyzing the expression generated in the process of converting the primary expression into the secondary expression. For example, if the target secondary expression is "doing a good luck pose," it may analyze whether the secondary expression generated using a model or the like is a good luck pose using another model. If the secondary expression generation unit 220 obtains a result as an analysis result that the generated secondary expression is not a good luck pose, it may generate a new secondary expression.

[0113] The above describes examples of conversion conditions and conversion information with reference to an example of an expression conversion table, but the conversion conditions and conversion information are not limited to the above examples and may be set flexibly as appropriate depending on the specifications of the virtual space system, etc.

[0114] For example, a conversion condition may be set such that the wearable device, which is the input means for inputting input information from which the primary representation is converted, is in a state where it is unable to measure the coordinates and rotation angles of some body parts due to factors such as a dead battery or malfunction. In this case, a secondary representation may be generated according to conversion information that calculates the primary representations of the coordinates and rotation angles of the avatar's body parts corresponding to the some body parts using a specific algorithm. This allows for the generation of natural representations even when the wearable device is in a state where it is unable to measure the coordinates and rotation angles of some body parts due to factors such as a dead battery or malfunction, thereby improving the quality of the user's experience in the virtual space.

[0115] Furthermore, the conversion conditions may include a dance party event being held in the virtual space and a mismatch between the rhythm of the music applied to the virtual space as background music for the dance and the rhythm of the dance movements represented by the primary representation. In this case, a secondary representation may be generated in accordance with conversion information that converts the rhythm of the dance movements represented by the primary representation to match the rhythm of the music applied to the virtual space as background music for the dance. This allows for the generation of a representation appropriate for the dance party event, even if the user is unable to input input information that matches the rhythm of the music due to skill constraints, thereby improving the quality of the user's experience in the virtual space.

[0116] (Expression Conversion Information Setting Unit 230) The expression conversion information setting unit 230 performs settings related to the expression conversion table stored in the expression conversion information storage unit 225. For example, the expression conversion information setting unit 230 functions as a reception unit that receives edits such as new registrations, updates, and deletions of information included in the expression conversion table set for each user. Furthermore, the expression conversion information setting unit 230 sets conversion conditions for each user by registering the received information in the expression conversion table for each user. The timing for editing the expression conversion table is not particularly limited, and may be when the virtual space system is in use or when it is not in use.

[0117] The expression conversion information setting unit 230 may accept editing of the expression conversion table by a system administrator of the virtual space, or may accept editing of the expression conversion table by a user for whom the expression conversion table is set. Furthermore, the range of editing permitted by the system administrator of the virtual space may differ from the range of editing permitted by a user. For example, new registration of information included in the expression conversion table may be executable by both the system administrator of the virtual space and the user. Furthermore, updating and deleting information included in the expression conversion table may be executable only by the system administrator of the virtual space.

[0118] In addition, the expression conversion table may contain a mixture of combinations of target primary expressions and target secondary expressions registered by the system administrator of the virtual space and combinations of target primary expressions and target secondary expressions registered by the user.

[0119] The expression conversion information setting unit 230 may register a combination of a target primary expression and a target secondary expression in common to the expression conversion table for all users in response to an instruction from a system administrator of the virtual space. The expression conversion information setting unit 230 may also register a combination of a target primary expression and a target secondary expression in common to users with a specific attribute in response to an instruction from the system administrator of the virtual space. The instruction from the system administrator of the virtual space may be input, for example, from an information processing terminal used by the system administrator of the virtual space.

[0120] The expression conversion information setting unit 230 may control the communication unit 205 to transmit information for displaying an editing screen for the expression conversion table on the user terminal 10. The user can edit the information displayed on the editing screen using, for example, a keyboard and mouse connected to the user terminal 10. The information displayed on the editing screen may be edited by another device, such as a controller, connected to the user terminal 10, or may be edited based on detection results of the user's line of sight or gestures. For example, if the user terminal 10 is an HMD for XR, the information displayed on the editing screen superimposed on real space may be edited based on detection results from a sensor mounted on the HMD that detects the user's line of sight and gestures.

[0121] Here, an example of an editing screen for an expression conversion table displayed on the user terminal 10 will be described with reference to Fig. 8 and Fig. 9. Fig. 8 and Fig. 9 are diagrams showing an example of an editing screen for an expression conversion table displayed on the user terminal 10. The display screen D1 shown in Fig. 8 is an editing screen for the expression conversion table T1. The user can edit the expression conversion table T1 by selecting information to be edited from the expression conversion table T1.

[0122] For example, Fig. 8 shows an example in which each piece of information with conversion ID "0015" is selected as information to be edited. Fig. 9 shows an example in which the type of the target primary expression with conversion ID "0015" has been edited. The user can edit the type by selecting a type from the type selection options TC.

[0123] In this way, by freely editing the expression conversion table set for the user, the user can set the expression of his / her avatar to change according to his / her situation, his / her preference, his / her cultural background, etc., thereby improving the quality of the user's experience. Furthermore, by freely editing the expression conversion table set for the user, the user can also change the expressions of other users included in the virtual space output for the user according to his / her preference, thereby improving the quality of the user's experience.

[0124] Although an example has been described in which users can freely edit the expression conversion table set for themselves, some of the information contained in the expression conversion table may be set so that it cannot be edited by users. For example, if there is a combination of target primary expressions and target secondary expressions set commonly for all users by a system administrator, this combination may be set so that it cannot be edited by users. For example, by setting the combination of target primary expressions and target secondary expressions for disabling insulting expressions so that it cannot be edited by users, a virtual space that matches the intentions of the system administrator is provided to users.

[0125] The expression conversion information setting unit 230 may automatically edit the expression conversion table. For example, the expression conversion information setting unit 230 may delete information that has not been referenced by the secondary expression generation unit 220 for six months or more. The expression conversion information setting unit 230 may also add new information to the expression conversion table. For example, the expression conversion information setting unit 230 may add information based on the analysis results of each user's reaction to an avatar in which an expression is reflected. As a more specific example, assume that the expression conversion information setting unit 230 has analyzed an avatar in which a specific expression is reflected and obtained analysis results showing an expression indicating displeasure as a reaction a predetermined number of times or a predetermined percentage of the number of times the specific expression is reflected. In this case, the expression conversion information setting unit 230 may newly add a combination of a target primary expression and a target secondary expression to the expression conversion table to disable the specific expression.

[0126] Furthermore, the expression conversion information setting unit 230 may automatically set the priority of a combination of a target primary expression and a target secondary expression. For example, the expression conversion information setting unit 230 may perform control so as to lower the priority of a combination of a target primary expression and a target secondary expression that is referenced less frequently by the secondary expression generation unit 220.

[0127] Furthermore, the expression conversion information setting unit 230 may edit the priority based on an evaluation of an avatar that reflects an expression placed in a virtual space, input by a user. More specifically, when an avatar that reflects an expression is highly evaluated, the expression conversion information setting unit 230 may control to increase the priority set for a combination of a target primary expression and a target secondary expression that includes the expression as a target secondary expression. Furthermore, when an avatar that reflects an expression is highly evaluated, the expression conversion information setting unit 230 may control to decrease the priority set for a combination of a target primary expression and a target secondary expression that includes the expression as a target secondary expression.

[0128] FIG. 10 is a diagram showing an example of a virtual space image displayed on the user terminal 10. As shown in FIG. 10, the display screen D2 includes an avatar A, a GOOD button B1, a BAD button B2, and a GUI display button B3. If the user rates the avatar A, which reflects expressions such as motion or audio, highly, the user presses the GOOD button B1. If the user rates the avatar A, which reflects expressions such as motion or audio, poorly, the user presses the BAD button B2. Furthermore, when the GUI display button B3 is pressed, the display screen D2 may transition to the display screen D1, which is an editing screen for the expression conversion table shown in FIG. 8. Note that, although an example in which the user manually inputs an evaluation of the avatar A has been described here, the evaluation of the avatar A may also be automatically detected based on the user's facial expression, etc.

[0129] The above has described the setting of the expression conversion table by the expression conversion information setting unit 230, but the contents included in the expression conversion table are not limited to the examples described so far. For example, the priority set for a combination of a target primary expression and a target secondary expression may include two types of priority: a priority set by the user and a priority set automatically by the expression conversion information setting unit 230. When two types of priority are set for a combination of a target primary expression and a target secondary expression, the secondary expression generation determination unit 215 may determine the priority using the priority set by the user when making a determination about the target primary expression.

[0130] The editing history of the expression conversion table by the expression conversion information setting unit 230 and the reference history of the target secondary expressions included in the expression conversion table by the secondary expression generating unit 220 may be stored in the expression conversion information storage unit 225 .

[0131] 11 is a diagram showing an example of a history table including the edit history and reference history of the expression conversion table. The expression conversion table includes the last update date and time, last reference date and time, and last updater for each conversion ID. The last update date and time and last updater may be updated when the expression conversion information setting unit 230 updates the combination of the target primary expression and the target secondary expression corresponding to each expression ID. Furthermore, the reference history of the target secondary expression by the secondary expression generation unit 220 may be updated when the secondary expression generation determination unit 215 outputs the target secondary expression to the secondary expression generation unit 220.

[0132] (Virtual space reflection unit 235) The virtual space reflection unit 235 reflects the primary representation output from the secondary representation generation unit 220 or the secondary representation output from the representation conversion information storage unit 225 in the avatar. The virtual space reflection unit 235 reflects the primary representation or secondary representation in the avatar using appearance information, etc., of each avatar stored in the virtual space construction information storage unit 240. The virtual space reflection unit 235 updates the virtual space construction information for constructing a virtual space, stored in the virtual space construction information storage unit 240, so that the information includes avatar information reflecting the primary representation or secondary representation.

[0133] (Virtual space construction information storage unit 240) The virtual space construction information storage unit 240 stores virtual space construction information. Virtual space construction information refers to a collection of various information necessary to construct a virtual space. For example, the virtual space construction information storage unit 240 stores, as virtual space construction information, the coordinates and rotation angles of the skeletal points of an avatar, texture data for expressing the appearance of the face, etc., material data for expressing the gloss of the avatar, and lighting data for the environment in the virtual space.

[0134] (Virtual space construction unit 245) The virtual space construction unit 245 constructs a virtual space by using the virtual space construction information stored in the virtual space construction information storage unit 240. More specifically, the virtual space construction unit 245 may generate an image of the virtual space to be output to each server 20. Note that the virtual space construction unit 245 may execute a process of updating the virtual space information stored in the virtual space construction information storage unit 240 depending on the situation.

[0135] (Control Unit) The server 20 may have a control unit (not shown). The control unit functions as an arithmetic processing unit and a control device, and controls the overall operation of the server 20 in accordance with various programs. The control unit is realized by electronic circuits such as a CPU or a microprocessor. The control unit may also include a ROM for storing the programs to be used, arithmetic parameters, etc., and a RAM for temporarily storing parameters that change as appropriate.

[0136] The functions of the above-mentioned primary expression generation unit 210, secondary expression generation determination unit 215, secondary expression generation unit 220, expression conversion information setting unit 230, virtual space reflection unit 235 and virtual space construction unit 245 can be realized by a control unit provided in the server 20.

[0137] <<3. Operational Processing Example>> Next, an example of the flow of operational processing of the server 20 according to an embodiment of the present disclosure will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the flow of operational processing of the server 20 according to an embodiment of the present disclosure.

[0138] First, the communication unit 205 acquires input information from the user terminal 10 (S101). Next, the primary expression generation unit 210 generates a primary expression that reflects the input information (S102). Next, the secondary expression generation determination unit 215 determines whether the primary expression satisfies the conversion condition (S103). For example, the secondary expression generation determination unit 215 may determine whether the primary expression corresponds to a target primary expression in the expression conversion table stored in the expression conversion information storage unit 225.

[0139] Next, if the primary representation satisfies the conversion condition (S103 / YES), the secondary representation generation unit 220 converts the primary representation into a secondary representation using conversion information corresponding to the conversion condition (S104). For example, the secondary representation generation determination unit 215 converts the target primary representation corresponding to the primary representation into a target secondary representation, thereby converting the primary representation into a secondary representation. Then, the process proceeds to S105.

[0140] On the other hand, if the primary expression does not satisfy the conversion condition (S103 / NO), the process proceeds to S105.

[0141] Then, the virtual space reflecting unit 235 reflects the acquired primary or secondary representation in the avatar, and the virtual space constructing unit 245 constructs a virtual space including the avatar (S105).

[0142] <<4. Modifications>> One embodiment according to the present disclosure has been described above. Various modifications can be applied to this embodiment.

[0143] For example, although the above describes an example in which the conversion information is included in the conversion information table, the conversion information may be output by a neural network (NN) model. For example, a first NN obtained by machine learning using a plurality of pairs of examples of primary expressions and examples of secondary expressions as training data may be stored in the expression conversion information storage unit 225.

[0144] 13 is a diagram illustrating the first NN. For training of the first NN, a technique called contrastive learning may be used, in which feature quantities between input pairs are trained to be close to each other, and feature quantities between pairs other than the input pair are trained to be far from each other.

[0145] 13, when pairs P1 and P2 are input to the first NN 1000, learning is performed so that the feature amounts of primary representation E1 and secondary representation E2 included in pair P1 are close to each other, and the feature amounts of primary representation E3 and secondary representation E4 included in pair P2 are close to each other. Learning is also performed so that the feature amounts of secondary representation E2 and secondary representation E4 are far from each other. Note that while only two pairs are shown here, a large number of pairs are input to the first NN 1000 during learning, and the weights of the first NN 1000 are updated sequentially.

[0146] The primary and secondary representations included in a pair may be, for example, posture data (skeletal data), facial expression data, audio data, sensor data, still images, and video, or may be data other than the examples mentioned above.

[0147] The first NN outputs conversion information when a primary representation is input. In this case, the conversion information may be information indicating features of a secondary representation to be converted. The representation conversion information storage unit 225 may further store a second NN for converting features of the secondary representation into a secondary representation to be reflected in the avatar. The second NN may be an Auto Encoder model NN.

[0148] 14 is a diagram illustrating the second NN. As shown in FIG. 14, the second NN 2000 is trained so as to be able to restore the features F2 and F3 output by the first NN 1000 to the secondary representations E2 and E3 input to the first NN 1000.

[0149] The secondary representation generation unit 220 can convert a primary representation into a secondary representation by combining the first NN 1000 and the second NN 2000. Fig. 15 is a diagram for explaining an example of converting a primary representation into a secondary representation by combining the first NN 1000 and the second NN 2000.

[0150] 15 , for example, the representation conversion information storage unit 225 inputs a primary representation E1 to a first NN 1000. In this case, the first NN 1000 outputs a feature F2 indicating a pose A to which the pose indicated by the primary representation E1 is converted from a feature map C in which the features of each pose are mapped. The representation conversion information storage unit 225 inputs the feature F2 output from the first NN 1000 to a second NN 2000. The second NN 2000 then outputs a secondary representation E2 based on the feature F2. In this way, the representation conversion information storage unit 225 can combine the first NN 1000 and the second NN 2000 to convert the primary representation E1 into a secondary representation E2.

[0151] Note that Figure 15 shows an example in which the first NN 1000 outputs the central feature F2 of the cluster of pose A in the feature map C, but the first NN 1000 may output a feature other than the central feature F2 of the cluster of pose A, for example, depending on the preference of the user to whom the virtual space is to be output.

[0152] FIG. 16 is a diagram illustrating an example of converting a primary representation into a secondary representation according to a user's preferences by combining the first NN 1000 and the second NN 2000. As shown in FIG. 16 , the first NN 1000 may learn a cluster R (R1, R2, etc.) of the user's preferences. Then, for example, when the primary representation E1 is input, the first NN 1000 may output a central feature F5 of cluster R1 in the feature map C, among the clusters of pose A to which the pose indicated by the primary representation E1 is converted. This allows the primary representation E1 to be converted into a secondary representation E5 that is different from the secondary representation E2 shown in FIG. 15 and is personalized and tailored to the user's preferences.

[0153] Note that user preferences may be registered by the user selecting a favorite video, as shown in FIG. 17 . FIG. 17 is a diagram showing an example of an editing screen for the expression conversion table displayed on the user terminal 10, which accepts the user's selection of a favorite video. As shown in FIG. 17 , on the display screen D3, which is the editing screen for the expression conversion table T2, the user's selection of a favorite video is accepted instead of selecting the ID of the target secondary expression. For example, the user can select a favorite video by selecting it from videos M1 and M2. Note that user preferences are not limited to the selection of a favorite video, and may be learned, for example, based on the frequency of conversion processes previously executed by the expression conversion information storage unit 225.

[0154] <<5. Hardware Configuration Example>> The embodiments of the present disclosure have been described above. The above-described information processing is realized by cooperation between software and hardware. Below, a hardware configuration example that can be applied to the user terminal 10 and the server 20 will be described.

[0155] Fig. 18 is a block diagram showing an example of an information processing device 90. Note that the hardware configuration example of the information processing device 90 described below is merely an example of the hardware configuration of the user terminal 10 and the server 20. Therefore, the user terminal 10 and the server 20 do not necessarily have to have the entire hardware configuration shown in Fig. 18 . Furthermore, the user terminal 10 and the server 20 may not have part of the hardware configuration shown in Fig. 18 .

[0156] 18 , the information processing device 90 includes a CPU 901, a ROM (Read Only Memory) 903, and a RAM 905. The information processing device 90 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. Instead of or in addition to the CPU 901, the information processing device 90 may include a processing circuit such as a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an ASIC (Application Specific Integrated Circuit).

[0157] The CPU 901 functions as an arithmetic processing device and control device, and controls all or part of the operations within the information processing device 90 in accordance with various programs recorded in the ROM 903, RAM 905, storage device 919, or removable recording medium 927. The ROM 903 stores programs and calculation parameters used by the CPU 901. The RAM 905 temporarily stores programs used in the execution of the CPU 901 and / or parameters that change as appropriate during the execution. The CPU 901, ROM 903, and RAM 905 are interconnected by a host bus 907, which is composed of an internal bus such as a CPU bus. Furthermore, the host bus 907 is connected to an external bus 911, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 909.

[0158] By the CPU 901 working in cooperation with the ROM 903, RAM 905, and software, the functions of, for example, the primary expression generation unit 210, the secondary expression generation determination unit 215, the secondary expression generation unit 220, the expression conversion information setting unit 230, the virtual space reflection unit 235, and the virtual space construction unit 245 can be realized.

[0159] The input device 915 is a device operated by a user, such as a button. The input device 915 may include a mouse, a keyboard, a touch panel, a switch, a lever, or the like. The input device 915 may also include a microphone that detects the user's voice. The input device 915 may be, for example, a remote control device that uses infrared or other radio waves, or an externally connected device 929 such as a mobile phone that supports operation of the information processing device 90. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs the signal to the CPU 901. The user operates the input device 915 to input various data and instruct processing operations to the information processing device 90.

[0160] The input device 915 may also include an imaging device and a sensor. The imaging device is a device that captures real space and generates a captured image using an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and various components such as a lens for controlling the formation of a subject image on the imaging element. The imaging device may capture still images or may capture moving images.

[0161] The sensors are various types of sensors, such as a distance measurement sensor, an acceleration sensor, a gyro sensor, a geomagnetic sensor, a vibration sensor, a light sensor, a sound sensor, etc. The sensors acquire information about the state of the information processing device 90 itself, such as the attitude of the housing of the information processing device 90, or information about the surrounding environment of the information processing device 90, such as the brightness or noise around the information processing device 90. The sensors may also include a Global Positioning System (GPS) sensor that receives GPS signals to measure the latitude, longitude, and altitude of the device.

[0162] The output device 917 is configured with a device capable of visually or audibly notifying the user of acquired information. The output device 917 may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker or headphones. The output device 917 may also include a PDP (Plasma Display Panel), a projector, a hologram, a printer, or the like. The output device 917 outputs the results obtained by the processing of the information processing device 90 as video such as text or images, or as sound such as voice or audio. The output device 917 may also include a lighting device that brightens the surrounding area.

[0163] The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 90. The storage device 919 is configured, for example, by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. This storage device 919 stores programs or various data executed by the CPU 901, as well as various data acquired from the outside.

[0164] The drive 921 is a reader / writer for a removable recording medium 927 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is built into or externally attached to the information processing device 90. The drive 921 reads information recorded on the attached removable recording medium 927 and outputs the information to the RAM 905. The drive 921 also writes information to the attached removable recording medium 927.

[0165] The connection port 923 is a port for directly connecting a device to the information processing device 90. The connection port 923 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface) port, or the like. The connection port 923 may also be an RS-232C port, an optical audio terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) port, or the like. By connecting an external device 929 to the connection port 923, various types of data can be exchanged between the information processing device 90 and the external device 929.

[0166] The communication device 925 is a communication interface configured with, for example, a communication device for connecting to a local network or a communication network with a wireless communication base station. The communication device 925 may be, for example, a communication card for a wired or wireless LAN, Bluetooth, Wi-Fi, or WUSB (Wireless USB). The communication device 925 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. The communication device 925 transmits and receives signals, for example, between the Internet and other communication devices using a predetermined protocol such as TCP / IP. The local network or communication network with the base station connected to the communication device 925 is a wired or wireless network, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0167] <<6. Supplementary Information>> Although preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0168] For example, in the above embodiment, an example has been described in which the user terminal 10 constructs a virtual space, but the virtual space may also be constructed by each server 20. For example, the control unit 160 of the user terminal 10 may construct a virtual space based on virtual space construction information received by the communication unit 110 from the server 20. More specifically, the control unit 160 may construct a virtual space by placing virtual objects such as avatars of each user in the virtual space based on the virtual space construction information received from the server 20. In this case, the control unit 160 reflects a primary or secondary representation in the avatar.

[0169] Furthermore, the virtual space is not limited to a space in which an avatar operated by a user of the user terminal 10 is placed. For example, the virtual space may be a space in a game or animation in which a character controlled based on input information input through operations by a creator is placed. In this case, the input information may be the coordinates and skeletal points of each skeletal point of the character input through operations by the creator. For example, an example of input information may be the coordinates and skeletal points of each skeletal point of a character in a game who nods in response to the speech of the other person. In this case, the input information is used as is as the primary representation.

[0170] It is also possible to create a computer program for causing the hardware, such as the CPU, ROM, and RAM, built into the user terminal 10 or the server 20 to perform the functions of the user terminal 10 or the server 20. A computer-readable storage medium storing the computer program is also provided.

[0171] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0172] The following configurations also fall within the technical scope of the present disclosure. (1) An information processing system comprising a control unit that executes the following: determining whether first control information that controls a character in a virtual space and reflects input information satisfies a conversion condition; and, if it is determined that the conversion condition is satisfied, generating second control information for controlling the character by converting at least a portion of the first control information based on conversion information corresponding to the conversion condition. (2) The information processing system described in (1), further comprising an acquisition unit that acquires, as the input information, information input by users who respectively operate the multiple characters in the virtual space. (3) The information processing system described in (2), wherein the control unit sets the conversion condition for each of the multiple users. (4) The information processing system described in (3), further comprising a reception unit that receives settings of the conversion condition from the user. (5) The information processing system described in (4), wherein the conversion condition received by the reception unit is a condition for the first control information that controls a character corresponding to the user who inputs the setting. (6) The information processing system according to (4), wherein the conversion conditions accepted by the accepting unit are conditions for the first control information that controls a character corresponding to a user different from the user who inputs the settings in a virtual space that is output to the user. (7) The information processing system according to any of (1) to (6), wherein the control unit determines whether the first control information satisfies each of the plurality of conversion conditions, and generates the second control information based on conversion information corresponding to the conversion condition satisfied by the first control information among the plurality of conversion conditions. (8) The information processing system according to (7), wherein priorities are set for the plurality of conversion conditions, and when the first control information satisfies the plurality of conversion conditions, the control unit generates the second control information based on conversion information corresponding to a conversion condition with a higher priority. (9) The information processing system according to (8), wherein the priorities are set based on an evaluation of a character reflecting the control information, input by a user at the output destination of the virtual space.(10) The information processing system according to any one of (1) to (9), wherein the conversion information is information output by inputting the first control information into a neural network trained using multiple pairs of examples of the first control information and examples of the second control information as training data. (11) The information processing system according to (10), wherein the conversion information is output based on the neural network in accordance with pre-trained preferences of a user to whom the virtual space is to be output. (12) The information processing system according to (2), wherein the first control information is information for controlling a motion corresponding to the character, and the acquisition unit acquires information indicating the user's motion as the input information. (13) The information processing system according to (12), wherein the conversion information is information indicating conversion of a pose reflected in the character represented by the first control information into a pose different from the pose. (14) The information processing system according to (2), wherein the first control information is information for controlling a sound corresponding to the character. (15) The information processing system according to any one of (1) to (14), wherein the conversion conditions include conditions related to attributes of a user to which the virtual space is output. (16) The information processing system according to any one of (1) to (15), wherein the conversion conditions include conditions related to an environment of the virtual space. (17) A program that causes a computer to function as a control unit that executes: control to determine whether first control information that reflects input information and that controls a character in a virtual space satisfies the conversion conditions; and control to generate second control information for controlling the character, where the first control information is converted based on conversion information corresponding to the conversion conditions, when it is determined that the conversion conditions are satisfied. (18) An information processing method executed by a computer, comprising: determining whether first control information that reflects input information and that controls a character in a virtual space satisfies the conversion conditions; and, when it is determined that the conversion conditions are satisfied, generating second control information for controlling the character, where the first control information is converted based on conversion information corresponding to the conversion conditions.

[0173] REFERENCE SIGNS LIST 1 Information processing system 10 User terminal 110 Communication unit 120 Input unit 130 Display unit 140 Audio output unit 150 Storage unit 160 Control unit 20 Server 205 Input unit 210 Primary expression generation unit 215 Secondary expression generation determination unit 220 Secondary expression generation unit 225 Expression conversion information storage unit 230 Expression conversion information setting unit 235 Virtual space reflection unit 240 Virtual space construction information storage unit 245 Virtual space construction unit 250 Output unit 30 Network

Claims

1. An information processing system comprising a control unit that executes the following controls: determining whether first control information that reflects input information and controls a character in a virtual space satisfies a conversion condition; and, if it is determined that the conversion condition is satisfied, generating second control information for controlling the character by converting at least a portion of the first control information based on conversion information corresponding to the conversion condition.

2. The information processing system according to claim 1, further comprising an acquisition unit that acquires information input by users who are the operators of each of the multiple characters in the virtual space as the input information.

3. The information processing system according to claim 2, wherein the control unit sets the conversion conditions for each of the plurality of users.

4. The information processing system according to claim 3, further comprising a receiving unit that receives the setting of the conversion condition from the user.

5. An information processing system according to claim 4, wherein the conversion conditions accepted by the accepting unit are conditions for the first control information that controls a character corresponding to the user who inputs the settings.

6. An information processing system as described in claim 4, wherein the conversion conditions accepted by the accepting unit are conditions for the first control information that controls a character corresponding to a user different from the user who inputs the settings in a virtual space that is output to the user.

7. The information processing system of claim 1, wherein the control unit determines whether the first control information satisfies each of the plurality of conversion conditions, and generates the second control information based on conversion information corresponding to the conversion condition that the first control information satisfies among the plurality of conversion conditions.

8. The information processing system of claim 7, wherein priorities are set for the plurality of conversion conditions, and when the first control information satisfies a plurality of the conversion conditions, the control unit generates the second control information based on conversion information corresponding to a conversion condition with a higher priority.

9. An information processing system according to claim 8, wherein the priority is set based on an evaluation of a character that reflects control information input by a user of the output destination of the virtual space.

10. The information processing system described in claim 1, wherein the conversion information is information output by inputting the first control information into a neural network that has been trained using multiple pairs of examples of the first control information and examples of the second control information as training data.

11. An information processing system according to claim 10, wherein the conversion information is output based on the neural network in accordance with the preferences of a user to whom the virtual space is to be output, the preferences having been learned in advance.

12. An information processing system as described in claim 2, wherein the first control information is information for controlling a motion corresponding to the character, and the acquisition unit acquires information indicating the user's motion as the input information.

13. An information processing system according to claim 12, wherein the conversion information is information indicating that the pose reflected on the character represented by the first control information is converted into a pose different from the pose.

14. The information processing system according to claim 2, wherein the first control information is information for controlling a sound corresponding to the character.

15. An information processing system according to claim 1, wherein the conversion conditions include conditions relating to attributes of a user to whom the virtual space is to be output.

16. The information processing system according to claim 1, wherein the conversion conditions include conditions relating to the environment of the virtual space.

17. A program that causes a computer to function as a control unit that executes the following controls: determining whether first control information that reflects input information and controls a character in a virtual space satisfies a conversion condition; and, if it is determined that the conversion condition is satisfied, generating second control information for controlling the character by converting at least a portion of the first control information based on conversion information corresponding to the conversion condition.

18. An information processing method executed by a computer, comprising: determining whether first control information that reflects input information and controls a character in a virtual space satisfies a conversion condition; and, if it is determined that the conversion condition is satisfied, generating second control information for controlling the character by converting at least a portion of the first control information based on conversion information corresponding to the conversion condition.

Images