Information processing device, information processing program, information processing method and information processing system

Abstract

To improve control of an avatar.SOLUTION: An information processing system 10 includes an operator terminal 12 and a user terminal 16. An action and utterance of an avatar displayed on a user terminal are controlled, according to an action and voice of an operator of an operator terminal, when a virtual object is not disposed. The operator and a user converse through the avatar. When the virtual object is created (or arranged) in a virtual space according to an operation of the operator, in an object work mode, movement of hands and arms of the avatar is created by an inverse kinematics algorithm, so that the avatar instructs the virtual object by the hands. Then, the avatar replaces the movement of the hands and arms among the action according to the action of the operator, with created movement of the hands and arms and operates.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing program, an information processing method, and an information processing system, and more particularly to an information processing device, an information processing program, an information processing method, and an information processing system that serve customers online using an avatar, for example. [Background technology]

[0002] An example of this type of information processing device is disclosed in Patent Document 1. In the communication system disclosed in Patent Document 1, when the telexistence mode is set, the operator terminal converts the facial expressions and gestures of the operator into coordinate data and transmits it to the response terminal together with the response voice data. The response terminal generates an avatar based on the coordinate data sent from the operator terminal, thereby generating character response information in which the facial expressions and gestures of the operator are reflected in the facial expressions and behavior of the avatar, and displays it to the user. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-56940 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above Patent Document 1, when the telexistence mode is set, the response terminal only displays an avatar image in which the operator's facial expression and gestures are reflected in the avatar's facial expression and gestures toward the user. In other words, since the avatar image is not displayed on the operator terminal, it is very difficult to indicate the material information or a part of the information by the movement of the avatar when explaining the material information displayed on the response terminal. Even if the avatar image is displayed on the operator terminal, the operator needs to make gestures and move the avatar while looking at the avatar image and the material information, so it is difficult to move an avatar image that appears reversed from left to right. Therefore, there is room for improvement in the control of the avatar.

[0005] Therefore, a primary object of the present invention is to provide a novel information processing device, an information processing program, an information processing method, and an information processing system.

[0006] Another object of the present invention is to provide an information processing device, an information processing method, and an information processing system that are capable of improving the control of an avatar. [Means for solving the problem]

[0007] The first invention is an information processing device used by an operator who interacts with users using an avatar, the information processing device comprising: a camera, avatar motion data generation means for generating avatar motion data corresponding to the operator in a three-dimensional virtual space based on an image of the operator captured by the camera, avatar control means for controlling the motion of the avatar in accordance with the motion data generated by the avatar motion data generation means, image generation means for generating display image data corresponding to an image of the three-dimensional virtual space captured by a virtual camera, transmission means for transmitting at least the display image data to a user terminal used by the user, placement means for placing an object in the three-dimensional virtual space in accordance with a placement operation by the operator, arm motion data generation means for generating arm motion data for the motion of the avatar's arm that points to the object based on the size and position of the object placed by the placement means, and modification means for modifying the arm motion data included in the motion data generated by the avatar motion data generation means to the arm motion data generated by the arm motion data generation means.

[0008] The second invention is dependent on the first invention and further includes a microphone, a speaker, a receiving means for receiving user voice data corresponding to the user's voice, and an output means for outputting the user voice data received by the receiving means to the speaker, and the transmitting means further transmits operator voice data corresponding to the operator's voice detected by the microphone to the user terminal.

[0009] A third invention is according to the first or second invention, and further includes a display means for displaying a UI screen for selecting and arranging UI object images, and the arrangement means arranges, in the three-dimensional virtual space, an object corresponding to the UI object image selected and arranged on the UI screen.

[0010] A fourth invention is according to the third invention, and further comprises an avatar arranged at a predetermined position in the three-dimensional virtual space, a UI screen including an image of the avatar, and an object position determination means for determining an object position, which is a three-dimensional position of an object in the three-dimensional virtual space, based on a two-dimensional positional relationship of the UI object image relative to the image of the avatar on the UI screen and a depth direction position determined so that the distance between the avatar and the object in the three-dimensional virtual space is a predetermined distance, and a hand position determination means for determining a three-dimensional position of the avatar's hand pointing to the object based on the object position, and the arm movement data generation means generates arm movement data for changing from the current posture of the avatar's arm to the posture of the arm pointing to the object, according to an inverse kinematics algorithm using parameters related to the three-dimensional positions of the avatar's hand and head, the size of the object, and the manner of pointing the object.

[0011] A fifth invention is an information processing program executed by a computer equipped with a camera and used by an operator who interacts with users using an avatar, the information processing program causing a processor of the computer to execute an avatar motion data generation step of generating avatar motion data corresponding to an operator in a three-dimensional virtual space based on an image of the operator captured by the camera, an avatar control step of controlling the motion of the avatar in accordance with the motion data generated in the avatar motion data generation step, an image generation step of generating display image data corresponding to an image captured of the three-dimensional virtual space by a virtual camera, a transmission step of transmitting at least the display image data to a user terminal used by the user, a placement step of placing an object in the three-dimensional virtual space in accordance with a placement operation by the operator, an arm motion data generation step of generating arm motion data for the motion of the avatar's arm that points to the object based on the size and position of the object placed in the placement step, and a modification step of modifying the arm motion data included in the motion data generated in the avatar motion data generation step to the arm motion data generated in the arm motion data generation step.

[0012] A sixth invention is an information processing method for a computer equipped with a camera and used by an operator who interacts with users using an avatar, in which the computer executes the following steps: (a) generating avatar movement data corresponding to the operator in a three-dimensional virtual space based on an image of the operator captured by the camera; (b) controlling the movement of the avatar in accordance with the movement data generated in step (a); (c) generating display image data corresponding to an image of the three-dimensional virtual space captured by a virtual camera; (d) transmitting at least the display image data to a user terminal used by the user; (e) placing an object in the three-dimensional virtual space in accordance with a placement operation by the operator; (f) generating arm movement data regarding the movement of the avatar's arm that points to the object based on the size and position of the object placed in step (e); and (g) changing the arm movement data included in the movement data generated in step (a) to the arm movement data generated in step (f).

[0013] A seventh invention is an information processing system having an operator terminal used by an operator who interacts with users using an avatar and a user terminal used by the user connected so as to be able to communicate with each other, the operator terminal comprising: a camera, an avatar motion data generation means for generating motion data of an avatar corresponding to the operator in a three-dimensional virtual space based on an image of the operator captured by the camera, an avatar control means for controlling the motion of the avatar in accordance with the motion data generated by the avatar motion data generation means, an image generation means for generating display image data corresponding to an image of the three-dimensional virtual space captured by the virtual camera, a transmission means for transmitting at least the display image data to a user terminal used by the user, a placement means for placing an object in the three-dimensional virtual space in accordance with a placement operation by the operator or the user, arm motion data generation means for generating arm motion data regarding the motion of the arm of the avatar that points to the object based on the size and position of the object placed by the placement means, and a modification means for modifying the arm motion data included in the motion data generated by the avatar motion data generation means to the arm motion data generated by the arm motion data generation means. Effect of the Invention

[0014] According to this invention, the arm motion of the avatar that points to an object is automatically generated based on the position of the object placed in the virtual space, so that the virtual object can be pointed to more easily and appropriately than when the avatar's motion is based only on the operator's motion, which improves the control of the avatar.

[0015] The above objects, other objects, features and advantages of the present invention will become more apparent from the following detailed description of the embodiments with reference to the drawings. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram showing an information processing system according to an embodiment of the present invention. [Diagram 2]FIG. 2 is a block diagram showing an example of an electrical configuration of the operator terminal shown in FIG. [Diagram 3] FIG. 3 is a block diagram showing an example of an electrical configuration of the user terminal shown in FIG. [Figure 4] FIG. 4 is a diagram showing a first example of a user-side interactive screen displayed on the display device of the user terminal shown in FIG. [Diagram 5] FIG. 5 is a diagram showing an example of an operator-side dialogue screen displayed on the display device of the operator terminal shown in FIG. [Figure 6] FIG. 6 is a diagram showing an example of an outline of a virtual space for generating data for an avatar screen. [Figure 7] FIG. 7 is a diagram showing a first example of an operator UI screen displayed on the display device of the operator terminal shown in FIG. [Figure 8] FIG. 8 is a diagram showing a second example of the operator UI screen displayed on the display device of the operator terminal shown in FIG. [Figure 9] FIG. 9 is a diagram showing a third example of the operator UI screen displayed on the display device of the operator terminal shown in FIG. [Figure 10] FIG. 10 is a diagram showing a fourth example of the operator UI screen displayed on the display device of the operator terminal shown in FIG. [Figure 11] FIG. 11 is a diagram showing an example of the positional relationship between the avatar, virtual object, virtual camera, and projection surface in the virtual space when the operator UI screen shown in FIG. 10 is displayed, viewed from diagonally behind and to the right of the virtual camera. [Figure 12] FIG. 12 is a diagram of the virtual space shown in FIG. 11 as viewed from directly above. [Figure 13] FIG. 13(A) is a diagram illustrating an example of a method for determining a hand that will point to a virtual object, and FIG. 13(B) is a diagram illustrating an example of a method for determining a position to which an avatar's hand will be moved. [Figure 14] FIG. 14 is a diagram showing a second example of a user's interactive screen displayed on the display device of the user terminal shown in FIG. [Figure 15]FIG. 15 is a diagram showing a third example of a user-side interactive screen displayed on the display device of the user terminal shown in FIG. [Figure 16] FIG. 16 is a diagram showing a fourth example of the user's interactive screen displayed on the display device of the user terminal shown in FIG. [Figure 17] FIG. 17 is a diagram showing an example of a memory map of the RAM of the operator terminal shown in FIG. [Figure 18] FIG. 18 is a diagram showing an example of specific contents of the data storage area shown in FIG. [Figure 19] FIG. 19 is a diagram showing an example of a memory map of the RAM of the user terminal shown in FIG. [Figure 20] FIG. 20 is a flowchart showing a first part of an example of information processing by the CPU of the operator terminal shown in FIG. [Figure 21] FIG. 21 is a flowchart showing a second part of an example of information processing by the CPU of the operator terminal shown in FIG. 2, and follows FIG. [Figure 22] FIG. 22 is a flowchart showing a third part of an example of information processing by the CPU of the operator terminal shown in FIG. 2, and follows FIG. [Figure 23] FIG. 23 is a flowchart showing a fourth part of an example of information processing by the CPU of the operator terminal shown in FIG. 2, and follows FIG. [Figure 24] FIG. 24 is a flowchart showing a fifth part of an example of information processing by the CPU of the operator terminal shown in FIG. 2, and follows FIG. [Diagram 25] FIG. 25 is a flowchart showing a first part of an example of an object operation and avatar control process by the CPU of the operator terminal shown in FIG. [Figure 26] FIG. 26 is a flowchart showing a second part of an example of the object operation and avatar control processing by the CPU of the operator terminal shown in FIG. 2, and follows FIG. [Figure 27] FIG. 27 is a flowchart showing a third part of an example of the object operation and avatar control processing by the CPU of the operator terminal shown in FIG. 2, and follows FIG. [Figure 28]FIG. 28 is a flowchart showing an example of information processing by the CPU of the user terminal shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] 1, an information processing system 10 of this embodiment includes an operator terminal 12, which is communicably connected to a user terminal, ie, a user terminal 16 and a server 18 via a network 14.

[0018] As an example, by using Web Real-Time Communication (WebRTC) technology, video and / or audio can be transmitted and received in real time between the operator terminal 12 and the user terminal 16 via a web browser. For example, the operator terminal 12 and the user terminal 16 perform P2P (Peer to Peer) communication. In this case, the server 18 functions as a signaling server for acquiring information about the communication partner by WebRTC.

[0019] Although not shown in the figure, WebRTC can be used in combination with multiple servers. Specifically, multiple servers for using WebRTC include a signaling server, a STUN server, and a TURN server. Although not described in detail, the signaling server is a server for acquiring information about a communication partner by WebRTC. In addition, the STUN server and the TURN server are servers for crossing so-called NAT (Network Address Translation) when the communication partner exists in a different network.

[0020] In this embodiment, one operator terminal 12 and one user terminal 16 are shown, but in reality, a plurality of operator terminals 12 and a plurality of user terminals 16 are connected to the network 14, and one operator terminal 12 and one user terminal 16 set to conduct a video call or Web conference are communicatively connected. However, the operator terminal 12 is a terminal used by an operator who responds to the user of the user terminal 16.

[0021] In this embodiment, a screen (hereinafter referred to as "user's interactive screen") 100 (see FIG. 4) including an image of an avatar (hereinafter referred to as "avatar image") 110, which is an avatar of the operator, is displayed on the display device 60 of the user terminal 16, and the user and the operator interact with each other through the avatar. That is, the operator responds to the user using a CG (Computer Graphics) avatar corresponding to the operator. Meanwhile, the user uses an interaction service using an avatar provided by the operator terminal 12 and the server 18.

[0022] In one example, the avatar may be a human-like character, but in other examples, the avatar may be an animal or robot-like character, an animated character, a game character, etc.

[0023] The operator terminal 12 is an information processing device, and is, for example, a general-purpose notebook PC or desktop PC, but other general-purpose terminals such as a smartphone or tablet PC can also be used.

[0024] The network 14 is composed of an IP network including the Internet, and an access network for accessing the IP network. The access network may be a public telephone network, a mobile phone network, a wired LAN, a wireless LAN, or a CATV (Cable Television) network.

[0025] The user terminal 16 is an information processing device other than the operator terminal 12, and may be a general-purpose notebook PC, desktop PC, tablet terminal, or smartphone.

[0026] The server 18 is an information processing device other than the operator terminal 12 and the user terminal 16, and may be a general-purpose server. Therefore, although not shown in the figure, the server 18 has a processor such as a CPU, memory (RAM, ROM, HDD, etc.), and a communication function.

[0027] Fig. 2 is a block diagram showing the electrical configuration of the operator terminal 12 shown in Fig. 1. As shown in Fig. 2, the operator terminal 12 includes a CPU 20, which is connected to a RAM 22, a communication interface (hereinafter referred to as "communication I / F") 24, and an input / output interface (hereinafter referred to as "input / output I / F") 26 via an internal bus.

[0028] The CPU 20 is responsible for overall control of the operator terminal 12. However, instead of the CPU 20, a system-on-a-chip (SoC) including multiple functions such as a CPU function and a graphics processing unit (GPU) function may be provided.

[0029] The RAM 22 is a main storage device and is used as a work area or a buffer area for the CPU 20. Although not shown in the figure, the operator terminal 12 is provided with a HDD and a ROM as auxiliary storage devices. However, instead of or in addition to the HDD, a non-volatile memory such as an SSD may be used.

[0030] The communication I / F 24 is a wired interface for transmitting and receiving control signals and data to and from external computers such as the user terminal 16 and the server 18 via the network 14 under the control of the CPU 20. However, the communication I / F 24 may also be a wireless interface such as a wireless LAN or Bluetooth (registered trademark).

[0031] An input device 28, a display device 30, a microphone 32, a speaker 34, and a camera 36 are connected to the input / output I / F 26. The input device 28 is a keyboard and a computer mouse. In addition, a touch panel may be provided. The touch panel is a general-purpose touch panel, and any type of touch panel, such as a capacitive type, an electromagnetic induction type, a resistive film type, or an infrared type, may be used.

[0032] The display device 30 is, for example, a liquid crystal display device. The microphone 32 is a general-purpose sound-collecting microphone. The speaker 34 is a general-purpose stereo speaker. The camera 36 is a general-purpose CCD camera (also called a Web camera).

[0033] However, when a smartphone or a tablet PC is used as the operator terminal 12, the input device 28 is a touch panel and hardware buttons.

[0034] Furthermore, the input / output I / F 26 outputs operation data (or operation information) input from the input device 28 to the CPU 20, and also outputs image data generated by the CPU 20 to the display device 30, causing the display device 30 to display a screen corresponding to the image data. However, image data received from an external computer (e.g., the user terminal 16) may also be output by the CPU 20. The image data received from the user terminal 16 is data of an image including a user (hereinafter referred to as "user image data").

[0035] Furthermore, the input / output I / F 26 outputs data of the operator's voice detected by the microphone 32 (hereinafter referred to as "operator voice data") to the CPU 20, converts the voice data output by the CPU 20 into an analog voice signal and outputs it from the speaker 34, and outputs data of a video including the operator captured (detected) by the camera 36 (hereinafter referred to as "operator video data") to the CPU 20. In this embodiment, the video including the operator is a moving image.

[0036] However, the voice data output by the CPU 20 is data of the user's voice received from the user terminal 16 (hereinafter referred to as "user voice data").

[0037] The electrical configuration of the operator terminal 12 shown in FIG. 2 is merely an example, and is not necessarily limited to this example.

[0038] In addition, if the operator terminal 12 is a smartphone, it is equipped with a calling circuit for making calls via a mobile phone network, or a mobile phone network and a public telephone network. However, in this embodiment, such calls are not made, so they are not shown in the figure.

[0039] Fig. 3 is a block diagram showing an example of the electrical configuration of the user terminal 16 shown in Fig. 1. As shown in Fig. 3, the user terminal 16 includes a CPU 50, which is connected to a RAM 52, a communication I / F 54, and an input / output I / F 56 via an internal bus.

[0040] The CPU 50 is responsible for overall control of the user terminal 16. However, instead of the CPU 50, an SoC including multiple functions such as a CPU function and a GPU function may be provided.

[0041] The RAM 52 is a main storage device and is used as a work area or a buffer area for the CPU 50. Although not shown in the figure, the user terminal 16 is provided with an HDD and a ROM as auxiliary storage devices. However, instead of or in addition to the HDD, a non-volatile memory such as an SSD may be used.

[0042] The communication I / F 54 is a wired interface for transmitting and receiving control signals and data to and from external computers such as the operator terminal 12 and the server 18 via the network 14 under the control of the CPU 50. However, a wireless interface such as wireless LAN or Bluetooth (registered trademark) can also be used as the communication I / F 54.

[0043] An input device 58, a display device 60, a microphone 62, a speaker 64, and a camera 66 are connected to the input / output I / F 56. The input device 58 is a keyboard and a computer mouse. In addition, a general-purpose touch panel may be provided.

[0044] The display device 60 is, for example, a liquid crystal display device. The microphone 62 is a general-purpose sound-collecting microphone. The speaker 64 is a general-purpose stereo speaker. The camera 66 is a general-purpose CCD camera.

[0045] Furthermore, the input / output I / F 56 outputs operation data (or operation information) input from the input device 58 to the CPU 50, and outputs image data generated by the CPU 50 to the display device 60, causing a screen or image corresponding to the image data to be displayed on the display device 60. However, there are also cases where image data (avatar screen data in this embodiment) received from an external computer (for example, the operator terminal 12) is output by the CPU 50.

[0046] Furthermore, the input / output I / F 56 converts the user's voice detected by the microphone 62 into digital voice data, i.e., user voice data, and outputs it to the CPU 50, converts the voice data output by the CPU 50 into an analog voice signal and outputs it from the speaker 64, and outputs data of a video including the user photographed (detected) by the camera 66, i.e., user video data, to the CPU 50. However, in this embodiment, the voice data output by the CPU 50 is operator voice data received from the operator terminal 12.

[0047] It should be noted that the electrical configuration of the user terminal 16 shown in FIG. 4 is merely an example and is not necessarily limited to this example.

[0048] In addition, if the user terminal 16 is a smartphone, it is equipped with a calling circuit for making calls via a mobile phone network, or a mobile phone network and a public telephone network, but in this embodiment, such calls are not made, so they are not shown in the figure.

[0049] The operation of the information processing system 10 of this embodiment will be described below. In such an information processing system 10, the operator terminal 12 and the user terminal 16, each of which has acquired connection information (such as IP addresses) of the communication partners from the server 18, conduct a video call or a Web conference, and the operator using the operator terminal 12 and the user using the user terminal 16 interact (or converse).

[0050] As an example, when a user accesses a desired EC (Electronic Commerce) site with user terminal 16, the user may request a dialogue with an avatar (or an operator), or the operator terminal 12 and user terminal 16 may access a specific URL that has been notified in advance by server 18 or the like, causing the operator terminal 12 and user terminal 16 to conduct (or start) a video call or Web conference.

[0051] 4 shows an example of an interactive screen 100 (hereinafter referred to as "user's interactive screen") displayed on the display device 60 of the user terminal 16 when a video call or a Web conference (or an online conference) is held. As shown in FIG. 4, the user's interactive screen 100 includes an image 102 corresponding to an avatar screen, and a button image 104 is provided.

[0052] Image 102 is an image of an avatar screen corresponding to avatar screen data generated by operator terminal 12 and transmitted to user terminal 16, and includes avatar image 110. Button image 104 is an icon for ending the dialogue.

[0053] In this embodiment, the movement of the avatar on the user-side interactive screen 100 displayed on the display device 60 and the voice (utterance) of the avatar output from the speaker 64 are controlled based on the movement and voice of the operator of the operator terminal 12. However, when a virtual object (hereinafter referred to as a "virtual object") is generated (or placed) by the operator, the movement of pointing to the virtual object with a part of the avatar's body (in this embodiment, the left hand and / or right hand) is controlled based on the position and size of the virtual object.

[0054] During the dialogue, avatar screen data generated by the operator terminal 12 is transmitted from the operator terminal 12 to the user terminal 16. In addition, voice data corresponding to the voice detected by the microphone 32 of the operator terminal 12, i.e., operator voice data, is transmitted from the operator terminal 12 to the user terminal 16.

[0055] In this embodiment, the movements include gestures, facial orientation, and facial expressions. The movements of the operator are detected from an image of the operator captured by the camera 36 of the operator terminal 12. The voice of the operator is detected by the microphone 32.

[0056] In the operator terminal 12, motion data for controlling the motion of the avatar based on the motion of the operator is generated, and a screen including an avatar image performing a motion according to the motion data, i.e., data of the avatar screen (hereinafter referred to as "avatar screen data") is generated. The voice data for controlling the speech of the avatar is voice data corresponding to the voice of the operator. However, the voice of the operator may be converted into a voice of a preset avatar.

[0057] The motion data is information (data) on the position and orientation of each part of the face for controlling the avatar's gestures, face orientation, and facial expressions. Specifically, the motion data is information on the three-dimensional positions (coordinate data) of each part of the face, namely the eyebrows, eyes (upper eyelids, lower eyelids), nose, and mouth (upper lip, lower lip), the three-dimensional positions (coordinate data) of the face orientation, hands (including wrists), arms (elbows), shoulders, knees, and ankles, and the orientations (rotations) of the wrists, elbows, hips, and ankles.

[0058] However, if the avatar is not displayed below the thighs, information about the position and orientation of the knees and ankles does not need to be included in the motion data.

[0059] In addition, the motion data for controlling the motion of the avatar based on the motion of the operator can be generated using an image processing library such as MediaPipe Holistic, for example. In another example, the method disclosed in Japanese Patent Application Laid-Open No. 2021-56940 in the background art can also be used.

[0060] In this embodiment, the operation data generated in a normal operation mode (hereinafter referred to as "normal mode") which is different from an operation mode (hereinafter referred to as "object work mode") in which virtual objects are generated and edited (moved, deleted, etc.) as described below, is referred to as "first manual operation data."

[0061] As described above, when a virtual object is generated (or placed), the left hand and / or right hand of the avatar are operated to point the virtual object, and the operations of the left hand and / or right hand that point the virtual object are controlled based on the operation of the operator. The generation of the operation data of the left hand and / or right hand that points to the virtual object will be described later.

[0062] During the conversation, an image including the user is captured by the camera 66, and image data corresponding to the image captured by the camera 66, i.e., user image data, is transmitted from the user terminal 16 to the operator terminal 12. The user's voice is detected by the microphone 62, and voice data corresponding to the voice detected by the microphone 62, i.e., user voice data, is transmitted from the user terminal 16 to the operator terminal 12.

[0063] The user video data is output to the display device 30 of the operator terminal 12, and the user voice data is output from the speaker 34 of the operator terminal 12. Therefore, the operator can see the user's state and hear the user's voice. In other words, the operator can grasp the content of the user's utterance and the user's state.

[0064] Fig. 5 shows an example of operator-side interactive screen 150 displayed on display device 30 of operator terminal 12. As shown in Fig. 5, operator-side interactive screen 150 includes display area 152 and display area 154 slightly above the center of the screen. Display area 152 is an area for displaying an image including a video of a user corresponding to user video data, and display area 154 is an area for displaying an image corresponding to avatar screen data, i.e., an avatar screen.

[0065] Furthermore, on the operator-side interactive screen 150, three button images 160, 162, and 164 are provided side by side below the display area 152 and the display area 154. Furthermore, below the button image 164, a button image 166 is provided. The button image 160 is an icon for generating and editing a virtual object. The button image 162 is an icon for controlling the position, direction, and zoom of a virtual camera. The button image 164 is an icon for controlling the position and direction of an avatar. The button image 166 is an icon for ending the interaction.

[0066] The operator interacts with the user while viewing the user's video and avatar screen displayed on the operator's interactive screen 150. The operator also generates a virtual object and displays an image of the virtual object on the avatar screen. The operator also controls the direction and / or zoom of the virtual camera, and controls the position and / or direction of the avatar. Thus, the operator changes the display content of the virtual space including the avatar, i.e., the avatar screen. Thus, the display content of the user's interactive screen 100 is also changed. The operator also speaks using the generated virtual object or regarding the generated virtual object.

[0067] 6 is a diagram showing an outline of a virtual space for generating data for an avatar screen. In this embodiment, the virtual space is a three-dimensional virtual space, and world coordinates are set in the three-dimensional virtual space. In this embodiment, the X axis of the world coordinates is the horizontal axis, the Y axis is the height axis, and the Z axis is the depth axis.

[0068] 6, an avatar and a virtual camera are provided in the virtual space, and a virtual screen (or a projection surface) is provided between the avatar and the virtual camera. The avatar has an appearance imitating a human being with a specific body type in the virtual space.

[0069] The avatar is placed at a predetermined position in the three-dimensional virtual space, and in the initial state, the avatar is set to be parallel to the Z axis of the world coordinate system and facing in the negative direction of the Z axis. The avatar is also placed on the XZ plane. For example, the avatar is positioned at the feet.

[0070] In the initial state, the virtual camera (viewpoint) is set to be located in the three-dimensional virtual space at a position that is a predetermined distance D1 away from a predetermined position of the avatar in the negative direction of the Z axis, and facing in the positive direction of the Z axis of the world coordinate system. The virtual camera's gaze point is set to the position of the avatar's chest, and in the initial state, the virtual camera's position in the height direction is set to the height of the avatar's chest. However, since the virtual camera's gaze point is fixed to the position of the avatar's chest, when the avatar moves, it moves in accordance with the movement of the avatar.

[0071] Furthermore, the virtual camera changes its orientation and zooms in or out according to the operator's operation. However, since the gaze point is fixed, when the orientation of the virtual camera is changed, it moves on a sphere with a radius of a predetermined distance D2 centered on the position of the avatar's chest, and its position is also changed.

[0072] Image data corresponding to an image captured by a virtual camera is generated as avatar screen data. The process of drawing an image captured by a virtual camera is a well-known technique, so a detailed description will be omitted. Briefly, the CPU 20 generates or places a character or object in a three-dimensional (or world coordinate system) virtual space, generates a certain scene, and converts the scene generated in the virtual space into a coordinate system captured by the virtual camera (i.e., camera coordinate system). For example, an image seen from the virtual camera is perspectively projected onto a virtual screen. Next, the CPU 20 performs clipping and hidden surface removal processing. Next, the CPU 20 performs shading to express the brightness (shadow) of the object surface. Furthermore, the CPU 20 performs shadowing (i.e., casting shadows) to express the shadows generated by the object. Then, the CPU 20 performs texture mapping. In this way, the CPU 20 generates (draws) a two-dimensional image of a certain scene generated in a three-dimensional virtual space, and outputs two-dimensional image data corresponding to the generated two-dimensional image to the display device 30.

[0073] 7 is a diagram showing an example of an operator UI (User Interface) screen 200 for generating and editing a virtual object. When the operator turns on a button image 160 on the operator-side interactive screen 150, the operation mode is shifted from a normal mode to an object work mode, and the operator UI screen 200 is displayed on the display device 30. At this time, the operator UI screen 200 may be displayed alongside the operator-side interactive screen 150, may be displayed in front of the operator-side interactive screen 150, or may be displayed in place of the operator-side interactive screen 150 by closing the operator-side interactive screen 150.

[0074] 7, operator UI screen 200 includes avatar screen image 202. Image 202 includes avatar image 210. Although not shown, image 202 also includes a background image. In addition, operator UI screen 200 is provided with button image 220, button image 222, and button image 224, and further includes image 230 for scale comparison.

[0075] Button image 220 and button image 222 are arranged vertically side by side on the left edge of operator UI screen 200. Furthermore, button image 224 is arranged on the lower right of operator UI screen 200. Furthermore, image 230 for scale comparison is arranged on the right edge of operator UI screen 200, above button image 224. Button image 220, button image 222, button image 224 and image 230 for scale comparison are displayed in front of image 202 with a predetermined size and a predetermined shape.

[0076] Button image 220 is an icon for generating a virtual object (see FIG. 10). Button image 222 is an icon for editing a virtual object. Button image 224 is an icon for ending the object work mode and transitioning the operation mode to the normal mode. Scale comparison image 230 is an image for determining the size of a virtual object in a virtual space when generating a virtual object. In this embodiment, scale comparison image 230 is an image of an avatar's face (or head). This is just one example, and other images such as an image of the palm of an avatar may be used as long as the scales of the vertical and horizontal widths can be compared.

[0077] Note that image 230 for scale comparison is an index for determining the size at which image 212 of the virtual object should be displayed on the avatar screen. Even if the size of the virtual object is once determined, it can be changed by editing processing. Therefore, although an index object itself equivalent to image 230 for scale comparison is provided, image 230 for scale comparison does not have to be displayed.

[0078] When the button image 220 is turned on, the object generation process starts, and an object selection window 240 for selecting a virtual object is displayed in front of the operator UI screen 200, as shown in Fig. 8. In the object selection window 240, images of objects for the UI (hereinafter referred to as "UI object images") 250 corresponding to the multiple virtual objects are displayed.

[0079] When the operator selects one UI object image 250 in the object selection window 240, the selected UI object image 250 is displayed in a default size next to the scale comparison image 230, as shown in FIG.

[0080] When the selected UI object image 250 is displayed next to the scale comparison image 230, the operator can change the size of the UI object image 250 and input text into a predetermined display area of ​​the UI object image 250. General-purpose drawing software technology can be used as a method for changing the size of the UI object image 250 and inputting text into a predetermined display area of ​​the UI object image 250. Since the general-purpose drawing software technology is well known, a description thereof will be omitted.

[0081] It should be noted that the operator does not need to change the size of the UI object image 250, and does not need to input characters into a predetermined display area of ​​the UI object image 250.

[0082] The size of the virtual object corresponding to the UI object image 250 is determined based on the aspect ratio of the UI object image 250 to the image 230 for scale comparison.

[0083] Specifically, the width of the virtual object is determined so that the ratio of the width of the UI object image 250 to the width (longest part) of the scale comparison image 230 matches the ratio of the width of the virtual object to the width (longest part) of the avatar's face in the virtual space.

[0084] In addition, the vertical width of the virtual object is determined so that the ratio of the vertical width of the UI object image 250 to the vertical width (longest part) of the scale comparison image 230 matches the ratio of the vertical width of the virtual object to the vertical width (longest part) of the avatar's face in the virtual space.

[0085] However, if the virtual object is a three-dimensional object (a solid object), the depth width of the virtual object is determined based on the ratio of the width and height. If the ratio of the width and height is the same, the ratio of the depth width of the virtual object to the width (longest part) of the avatar's head in the depth direction is also set to the same ratio of the width and height. If the ratio of the width and height is different, the ratio of the depth width of the object to the width (longest part) of the avatar's head in the depth direction is set to the ratio of the width or the height, or the average ratio of these. However, the depth width of the virtual object may be left unchanged as the default.

[0086] When the operator places the UI object image 250 at a desired position, a virtual object corresponding to the UI object image 250 is generated (or placed) in the virtual space. As an example, the operator drags and drops the UI object image 250 to a desired position on the operator UI screen 200. This is an example, and the operator may simply tap the desired position.

[0087] 10 shows an example of the operator UI screen 200 on which an image 212 of a virtual object corresponding to the UI object image 250 is displayed instead of the UI object image 250. In other words, the UI object image 250 is erased, and the image 212 of the virtual object is displayed.

[0088] In this embodiment, geometric objects are shown as virtual objects and UI objects, but this is merely an example and does not need to be limited to these. For example, if products are sold on an EC site, the virtual objects and UI objects may be objects that imitate the products sold on the EC site.

[0089] In a three-dimensional virtual space, the position of a virtual object on a plane parallel to the XY plane (in this embodiment, the position of the center) is determined based on the two-dimensional position of the UI object image 250 corresponding to the virtual object on the operator UI screen 200. In this embodiment, the length in the upward or downward direction and the length in the leftward or rightward direction of the center position of the UI object image 250 on the operator UI screen 200 when the center position of the avatar's head is used as a reference are calculated, and the position on the plane parallel to the XY plane of the center position of the virtual object when the center position of the avatar's head is used as a reference in the virtual space (i.e., the X coordinate and the Y coordinate) is determined so as to have the calculated length in the upward or downward direction and the length in the leftward or rightward direction.

[0090] The Z coordinate of the center position of the virtual object is determined according to the size of the torso and the length of the arms of the avatar. FIG. 11 and FIG. 12 are diagrams for explaining an example of a method for determining the Z coordinate of the center position of the virtual object. FIG. 11 and FIG. 12 show a straight line X=A passing through the X coordinate (A) of the center position of the virtual object determined as described above, and a straight line Y=B passing through the Y coordinate (B). The Z coordinate is the coordinate of the position in the Z axis direction of the intersection of the straight lines X=A and Y=B. Also, as shown in FIG. 11 and FIG. 12, in the initial state, a virtual camera is placed in front of the avatar.

[0091] 11 and 12, an object for determining the Z coordinate of the center position of a virtual object in a virtual space (hereinafter, referred to as a "Z coordinate determining object") is set in an avatar. As an example, the Z coordinate determining object is a capsule-shaped object.

[0092] In this embodiment, the Z coordinate determination object is arranged so that its longitudinal direction is parallel to the center line of the avatar, and the center of a circle that is a cross section in the transverse direction (a cross section parallel to the XZ plane) overlaps with the center line of the avatar. However, the center line of the avatar is a straight line that passes through the center of the avatar and extends in the Y-axis direction.

[0093] In order to indicate a virtual object with the hand of the avatar as if supporting the virtual object from below slightly above the palm of the avatar (see FIG. 10, etc.), the center of the Z coordinate determination object is set on the center line of the avatar at chest height, as shown in FIG. 11. The longitudinal length of the Z coordinate determination object is set to a length from slightly above the head of the avatar to slightly below the waist of the avatar. Furthermore, in order to prevent the virtual object from entering the inside of the avatar's torso, the short-side length of the Z coordinate determination object, i.e., the diameter R of the circle of the cross section parallel to the XZ plane, is determined by the size of the avatar's torso (shoulder width in this embodiment) as shown in FIG. 12.

[0094] In this embodiment, the Z coordinate of the virtual object is determined so that the distance between the center of the virtual object and the Z coordinate determination object is a predetermined distance D2. The predetermined distance D2 is determined based on the length of the arm of the virtual object. As an example, the predetermined distance D2 is determined by a value (e / i) obtained by dividing the length e of the arm of the virtual object by a constant i, where the constant i is a number greater than 1.

[0095] In this embodiment, the predetermined distance D2 is determined by dividing the arm length e by a constant i, but if the degree of bending of the avatar's arm is to be changed depending on whether the two-dimensional distance between the avatar and the virtual object is large or small, the predetermined distance D2 can also be determined by dividing by a variable (a number greater than 1) that changes depending on the X coordinate and / or Y coordinate of the virtual object.

[0096] It should be noted that the Z coordinate determination object is a computational object that is set in order to determine the Z coordinate of a virtual object, and therefore does not need to be actually generated within the virtual space.

[0097] Also, the shape of the Z coordinate determination object is just an example, and it may be cylindrical or semi-cylindrical. However, a semi-cylindrical Z coordinate determination object is provided forward from the center of the avatar, and the arc-shaped side surface protrudes in front of the avatar.

[0098] Once the Z coordinate of the virtual object has been determined, that is, once the three-dimensional position of the virtual object has been determined, then the hand that points to the virtual object and the position (point) to which the hand will be moved are determined.

[0099] FIG. 13(A) is a diagram illustrating an example of a method for determining a hand that will point to a virtual object, and FIG. 13(B) is a diagram illustrating an example of a method for determining a position to which an avatar's hand will be moved.

[0100] As shown in Fig. 13(A), a two-hand range, a left hand range, and a right hand range are set for an avatar. The two-hand range is determined by the shoulder width of the avatar, the left hand range is the range on the left side of the two-hand range as seen from the avatar, and the right hand range is the range on the right side of the two-hand range as seen from the avatar.

[0101] When the center position of the virtual object is within the two-hand range, it is determined that the avatar points to the virtual object with both hands. When the center position of the virtual object is within the left-hand range, it is determined that the avatar points to the virtual object with the left hand. When the center position of the virtual object is within the right-hand range, it is determined that the avatar points to the virtual object with the right hand.

[0102] 13(B), the position of the hand pointing to a virtual object is a position separated from the virtual object by a predetermined distance d, and is determined in consideration of the size of the virtual object (here, the vertical length h) because the virtual object is pointed to from below. The predetermined distance d is determined in advance by a developer or the like.

[0103] Therefore, when an avatar points to a virtual object with the left or right hand, a position (point p) that is moved downward from the center of the virtual object by half the vertical length h of the virtual object (h / 2) plus a predetermined distance d is determined to be the position of the hand pointing to the virtual object.

[0104] Furthermore, when the avatar points to a virtual object with both hands, the position of the right hand pointing to the virtual object (point m) and the position of the left hand pointing to the virtual object (point n) are determined with the point p as the center. As an example, the positions of the right hand and the left hand are set in advance to have a distance a between them. The distance a is set to be slightly larger than the combined length of the palm of the avatar's right hand and the palm of the avatar's left hand.

[0105] When the hand pointing to the virtual object and the position of the hand pointing to the virtual object are determined, motion data on the movement of the hand pointing to the virtual object is generated. In this embodiment, using the position of the avatar's head and the position of the avatar's hand pointing to the virtual object, motion data on the movement of the hand and arm from the position (initial posture) of the current hand (left hand, right hand, or both hands) to the position (target posture) of the hand pointing to the virtual object (left hand, right hand, or both hands) (hereinafter referred to as "arm motion data") is generated according to an inverse kinematics algorithm. The arm motion data is data in which the three-dimensional positions of the avatar's wrist and elbow and the orientation (rotation) of the wrist and elbow are arranged in chronological order for each frame. A frame is a unit time for updating the screen, and can be set to 1 / 30 seconds, 1 / 60 seconds, or 1 / 120 seconds.

[0106] However, the left and / or right hand of the avatar that points to a virtual object corresponds to the end effector.In addition, when the avatar points to a virtual object, the parameters required for executing the inverse kinematics algorithm, such as which part of the virtual object to point, the length of the avatar's arm (size of the palm, length of the forearm, length of the upper arm), the angle between the forearm and upper arm, i.e., the constraint on the elbow angle, and the orientation of the avatar's hand (palm), are determined in advance.

[0107] Since the inverse kinematics algorithm is well known, a description thereof will be omitted.

[0108] When the arm motion data is generated, the arm motion data among the motion data generated from the operator video data is replaced with the arm motion data generated according to the inverse kinematics algorithm. When the avatar motion data (hereinafter referred to as "second manual motion data") in which the arm motion data generated according to the inverse kinematics algorithm is replaced is generated, a series of avatar images are generated in which the avatar performs a motion to point to a virtual object.

[0109] In the series of avatar screens, after a virtual object is displayed, the avatar is operated to point to the virtual object. However, this is just an example, and the virtual object may be displayed after the avatar moves his / her hand to a position to point to the virtual object. In addition, the avatar may not only simply move his / her hand to a position to point to the virtual object, but may also execute a predetermined action before displaying the virtual object. For example, the predetermined action may be a finger snap, changing the palm of the hand from a fist to an open hand, drawing a circle with the hand to surround the placement position of the virtual object, pointing with a finger to the placement position of the virtual object, etc.

[0110] Although detailed description will be omitted, in this embodiment, a limit is set on the number of virtual objects that the operator can generate in the virtual space. When the number of generated virtual objects reaches the maximum number, the operator is notified that a virtual object must be deleted before generation. The notification is performed by displaying a text message and / or by outputting a voice. Also, in this embodiment, the maximum number of virtual objects that can be generated in the virtual space is two, but the maximum number may be one, or three or more. However, there may be no limit on the number of virtual objects that can be placed in the virtual space.

[0111] In this embodiment, even if there is only one virtual object placed in the virtual space, if the avatar is pointing at this virtual object with both hands, a second virtual object cannot be generated. In this case, the operator is notified that the virtual object will be deleted or moved. The notification is performed by displaying a text message and / or by outputting a voice message.

[0112] Furthermore, in this embodiment, the avatar determines the hand with which to point a virtual object depending on the position where the virtual object is placed, but this is not necessarily limited to this. After placing (dropping) a virtual object, the operator may determine to point with the left hand, right hand, or both hands.

[0113] The series of avatar images described above are generated by moving the avatar in accordance with the second manual operation data, placing virtual objects in the virtual space at appropriate times, and capturing images of the virtual objects using a virtual camera. Avatar image data corresponding to the generated avatar images is transmitted to user terminal 16.

[0114] Therefore, on the display device 60 of the user terminal 16, a user-side interactive screen 100 as shown in Fig. 14, Fig. 15, and Fig. 16 is displayed. In the user-side interactive screen 100 shown in Fig. 14, an avatar screen image 102 is displayed in which the avatar image 110 points to an image 112 of one virtual object with the right hand. In the user-side interactive screen 100 shown in Fig. 15, an avatar screen image 102 is displayed in which the avatar image 110 points to an image 112 of one virtual object with the left hand. In the user-side interactive screen 100 shown in Fig. 16, an avatar screen image 102 is displayed in which the avatar image 110 points to an image 112 of one virtual object with both hands.

[0115] Although not shown in the drawings, an avatar screen image 102 is displayed on the user's interactive screen 100, in which an avatar image 110 points to an image 112 of one virtual object with the right hand, and the avatar image 110 points to an image 112 of another virtual object with the left hand. However, in this embodiment, since virtual objects are generated one by one in the work mode, the avatar is operated to point to an image 112 of one virtual object with the left or right hand, and then to point to an image 112 of the other virtual object with the right or left hand. After two virtual objects are generated, the avatar can also be operated to point to an image 112 of one virtual object with the left hand, and to point to an image 112 of the other virtual object with the right hand.

[0116] The method of pointing a virtual object is just an example, and the virtual object may be pointed to with a fingertip from the top, left or right end of the virtual object, or the virtual object may be pointed to by holding it between both hands. In this case, the only difference from the embodiment is the method of determining the position to point to the virtual object and the method of pointing with the avatar's hands.

[0117] In this embodiment, since the operator performs tasks such as generating a virtual object after the operation mode is shifted from the normal mode to the object work mode until the second manual operation data is generated, the operation of the avatar is controlled according to the automatic operation data 304g (see FIG. 18) generated in advance. However, the automatic operation data 304g is operation data for an operation that a human performs unconsciously during a conversation. Examples of an operation that a human performs unconsciously include a nodding operation, an operation of moving the face from side to side several times as if looking around and returning to the original posture, and an operation of moving the right hand and / or the left hand regularly or irregularly in front of the torso. Also, if a virtual object is already pointed to by the left hand or right hand, arm operation data for the left hand or right hand pointing the virtual object is generated, and the arm operation data in the automatic operation data 304g is changed to the generated arm operation data. However, since the left hand or right hand of the avatar pointing the virtual object is in a target posture, arm operation data is generated so as to maintain that posture.

[0118] Furthermore, as described above, in normal mode, the avatar is operated in accordance with the first manual movement data, but when the generated virtual object 1 or 2 is pointed to with the left hand and / or right hand, the left hand and / or right hand of the avatar pointing to virtual object 1 or 2 is in a target posture, and arm movement data for the left hand and / or right hand is generated to maintain that posture.

[0119] Fig. 17 shows an example of a memory map 300 of the RAM 22 built into the operator terminal 12. As shown in Fig. 17, the RAM 22 includes a program storage area 302 and a data storage area 304. The program storage area 302 stores an information processing program executed by the operator terminal 12 of this embodiment.

[0120] The information processing programs executed on the operator terminal 12 include a main processing program 302a, an operation detection program 302b, a communication program 302c, an image generation program 302d, an image output program 302e, a shooting program 302f, a motion data generation program 302g, an avatar control program 302h, an object control program 302i, a voice detection program 302j, and a voice output program 302k.

[0121] The main processing program 302a is a program for executing the main routine of information processing in the operator terminal 12 of this embodiment.

[0122] The operation detection program 302b is a program for detecting operation data 304a input from the input device 28 in accordance with an operation by an operator, and storing the data in the data storage area 304.

[0123] The communication program 302c is a program for communicating with external devices, in this embodiment, the user terminal 16 and the server 18.

[0124] Image generation program 302d is a program for generating, using image generation data 304b, image data corresponding to screens (such as operator's interactive screen 150 and operator UI screen 200) to be displayed on display device 30. However, when generating image data corresponding to operator's interactive screen 150 as shown in Fig. 5, user video data 304e is also used. Image generation program 302d is also a program for generating avatar screen data 304j using image generation data 304b.

[0125] The image output program 302e is a program for outputting image data generated according to the image generation program 302d to the display device 30. Therefore, a screen corresponding to the image data is displayed on the display device 30.

[0126] The photographing program 302f is a program for causing the camera 36 to execute photographing processing and for storing the operator video data 304c input from the camera 36 in the data storage area 304.

[0127] The motion data generation program 302g is a program for detecting the motion of the operator from the operator video data 304c, and generating first manual motion data 304h and second manual motion data 304i for controlling the motion of the avatar.

[0128] The avatar control program 302h is a program for controlling the movement of the avatar in accordance with automatic movement data 304g, which will be described later, and first manual movement data 304h or second manual movement data 304i generated in accordance with the movement data generating program 302g.

[0129] The object control program 302i is a program for generating and editing a virtual object in accordance with an operation by an operator.

[0130] The voice detection program 302j is a program for detecting the voice of the operator inputted from the microphone 32 and storing in the data storage area 304 the operator voice data 304d corresponding to the detected voice.

[0131] The voice output program 302k is a program for outputting the user voice data 304f received from the user terminal 16 to the speaker . Therefore, a voice corresponding to the user voice data 304f is output from the speaker .

[0132] Although not shown in the figure, the program storage area 302 also stores the operating system and middleware of the operator terminal 12, a browser, and other application programs other than the information processing program of the present application.

[0133] Fig. 18 shows an example of the specific contents of data storage area 304 of RAM 22 shown in Fig. 17. Data storage area 304 stores operation data 304a, image generation data 304b, operator video data 304c, operator voice data 304d, user video data 304e, user voice data 304f, automatic action data 304g, first manual action data 304h, second manual action data 304i, avatar screen data 304j, UI object data 304k, avatar data 304m, virtual camera data 304n, virtual object data 304p, and the like.

[0134] The operation data 304a is data of an operation by an operator detected according to the operation detection program 302b. The operation data 304a is deleted from the data storage area 304 after being used for processing by the CPU 50.

[0135] Image generation data 304b is data used to generate a screen to be displayed on display device 30 of operator terminal 12. Image generation data 304b also includes data for generating avatar screen data 304j (such as data for generating images of avatars and various virtual objects).

[0136] The operator video data 304c is data of video captured by the camera 66. This operator video data 304c is basically data of video including the operator, but if the operator leaves his / her seat and goes outside the range of the camera 66, the video may not include the operator. Once the operator video data 304c is used for processing by the CPU 50, it is deleted from the data storage area 304.

[0137] The operator voice data 304d is data of the voice of the operator detected by the microphone 62. The operator voice data 304d is erased from the data storage area 304 after being used for processing by the CPU 50 or transmitted to the operator terminal 12.

[0138] The user video data 304e is user video data received from the user terminal 16. Once the user video data 304e has been used for processing by the CPU 20, that is, once it has been used to generate image data for the interactive screen 150 on the operator side, it is deleted from the data storage area 304.

[0139] The user voice data 304f is user voice data received from the user terminal 16. When the user voice data 304f is output to the speaker 34 by the CPU 20, it is deleted from the data storage area 304.

[0140] The automatic motion data 304g is data for controlling the motion of the avatar, and in this embodiment, is information (data) on the position and orientation of each part for controlling the avatar's gestures, face orientation, and facial expression, and is used when generating or editing a virtual object in the object work mode. Specifically, as described above, the automatic motion data 304g is information on the three-dimensional positions (coordinate data) of each part of the face, namely, the eyebrows, eyes (upper eyelids, lower eyelids), nose, and mouth (upper lip, lower lip), the three-dimensional positions (coordinate data) of the face orientation, hands (including wrists), arms (elbows), shoulders, knees, and ankles, and the orientations (rotations) of the wrists, elbows, waists, and ankles. The automatic motion data 304g is generated in advance to automatically move the avatar.

[0141] The first manual motion data 304h is also data for controlling the motion of the avatar, and is generated in the normal mode. After the first manual motion data 304h is used for processing by the CPU 20, it is erased from the data storage area 304.

[0142] The second manual operation data 304i is also data for controlling the operation of the avatar, and is generated in response to the placement of a virtual object in the virtual space in the object work mode. The second manual operation data 304i is erased from the data storage area 304 after being used for processing by the CPU 20.

[0143] Avatar screen data 304j is data on an avatar screen generated in accordance with image generating program 302d. Avatar screen data 304j is used to generate operator UI screen 200 and is transmitted to user terminal 16.

[0144] The UI object data 304k is image data of a UI object corresponding to a virtual object, and is used when generating a virtual object. In this embodiment, a plurality of virtual objects are prepared, and therefore image data for a plurality of UI objects is prepared.

[0145] Avatar data 304m is data regarding the current position and current orientation of the avatar in virtual space.

[0146] Virtual camera data 304n is data about the current position, current orientation and zoom of the virtual camera in virtual space.

[0147] The virtual object data 304p is data on the type, current position, current orientation, and size of a virtual object arranged in a virtual space. However, when a plurality of virtual objects are arranged in a virtual space, the virtual object data 304p for each virtual object is stored. In addition, the type and default size of a virtual object corresponding to each of the plurality of UI object images 250 are stored in a non-volatile memory such as a HDD, and when storing the virtual object data 304p, the type and default size of a virtual object corresponding to the UI object image 250 dragged and dropped on the operator UI screen 200 are read from the non-volatile memory.

[0148] The data storage area 304 also includes an object task flag 304q, a two-hand flag 304r, and an automatic playback flag 304s.

[0149] The object work flag 304q is a flag indicating whether the object work mode is set or not. When the object work mode is set as the operation mode of the operator terminal 12, the object work flag 304q is turned on, and when the object work mode is not set, that is, when the normal mode is set, the object work flag 304q is turned off.

[0150] The two hands flag 304r is a flag that indicates whether the avatar is pointing at a virtual object with both hands. If the avatar is pointing at a virtual object with both hands, the two hands flag 304r is turned on, and if the avatar is not pointing at a virtual object with both hands, the two hands flag 304r is turned off.

[0151] The automatic playback flag 304s is a flag that indicates whether the avatar is operated according to the automatic operation data 304g. If the avatar is operated according to the automatic operation data 304g, the automatic playback flag 304s is turned on, and if the avatar is not operated according to the automatic operation data 304g, the automatic playback flag 304s is turned off.

[0152] Although not shown in the figure, the data memory area 304 stores other data necessary for executing the information processing of the operator terminal 12, such as parameters necessary for executing the inverse kinematics algorithm, and is provided with timers (counters) and other flags necessary for executing the information processing of the operator terminal 12.

[0153] Fig. 19 shows an example of a memory map 400 of the RAM 52 built into the user terminal 16. As shown in Fig. 19, the RAM 52 includes a program storage area 402 and a data storage area 404. The program storage area 402 stores an information processing program executed by the user terminal 16 of this embodiment.

[0154] The information processing programs executed on the user terminal 16 include a main processing program 402a, an operation detection program 402b, a communication program 402c, an image generation program 402d, an image output program 402e, a photography program 402f, a voice detection program 402g, and a voice output program 402h.

[0155] The main processing program 402a is a program for executing the main routine of information processing of the user terminal 16 of this embodiment.

[0156] The operation detection program 402b is a program for detecting operation data 404a input from the input device 58 in accordance with a user's operation, and storing the data in the data storage area 404.

[0157] The communication program 402c is a program for communicating (transmitting and receiving data, etc.) with external devices, in this embodiment, the operator terminal 12 and the server 18.

[0158] Image generation program 402d is a program for using image generation data 404b to generate image data corresponding to all or a part of a screen (such as user's interactive screen 100) to be displayed on display device 60. However, when generating image data for user's interactive screen 100, avatar screen data 404e is also used.

[0159] The image output program 402e is a program for outputting image data generated according to the image generation program 402d to the display device 30. Therefore, a screen corresponding to the image data is displayed on the display device 30.

[0160] The shooting program 402f is a program for causing the camera 66 to execute shooting processing and for storing user video data 404c input from the camera 66 in the data storage area 404.

[0161] The voice detection program 402g is a program for detecting the user's voice input from the microphone 62 and storing in the data storage area 404 the user voice data 404d corresponding to the detected voice.

[0162] The voice output program 402h is a program for outputting the operator voice data 404f received from the operator terminal 12 to the speaker 64. Therefore, the speaker 64 outputs a voice corresponding to the operator voice data 404f.

[0163] Although not shown, the program storage area 402 stores the operating system and middleware of the user terminal 16, as well as a browser and other application programs other than the control program of this application.

[0164] Furthermore, data storage area 404 stores operation data 404a, image generation data 404b, user video data 404c, user voice data 404d, avatar screen data 404e, operator voice data 404f, and the like.

[0165] The operation data 404a is data of a user's operation detected according to the operation detection program 402b. The operation data 404a is deleted from the data storage area 404 after being used for processing by the CPU 50.

[0166] Image generation data 404 b is data for generating a screen to be displayed on display device 60 of user terminal 16 .

[0167] The user video data 404c is data of video captured by the camera 66. This user video data 404c is basically data of video including the user, but if the user goes out of the range of the camera 66, the video may not include the user. When the user video data 404c is transmitted to the operator terminal 12 by the CPU 50, it is erased from the data storage area 404.

[0168] The user voice data 404d is data of the user's voice detected by the microphone 62. When the user voice data 404d is transmitted to the operator terminal 12 by the CPU 50, it is deleted from the data storage area 404.

[0169] Avatar screen data 404e is avatar screen data received from operator terminal 12. Avatar screen data 404e is deleted from data storage area 404 after being used by CPU 50 to generate image data for interactive screen 100 on the user side.

[0170] The operator voice data 404f is operator voice data received from the operator terminal 12. When the operator voice data 404f is output to the speaker 64 by the CPU 50, it is deleted from the data storage area 404.

[0171] Although not shown in the figure, the data memory area 404 stores other data necessary for executing information processing of the user terminal 16, and is provided with timers (counters) and flags necessary for executing information processing of the user terminal 16.

[0172] Fig. 20 to Fig. 24 are flow charts showing an example of information processing by the CPU 20 of the operator terminal 12 shown in Fig. 2. Fig. 25 to Fig. 27 are flow charts showing an example of object operation and avatar control processing by the CPU 20 of the operator terminal 12 shown in Fig. 2.

[0173] Although not shown in the figures, the CPU 20, in parallel with the information processing of the operator terminal 12, executes a process of detecting operation data to store the operation data, a process of having the camera 36 execute a process of photographing to store operator video data, a process of detecting voice to store operator voice data, and a process of receiving and storing data transmitted from the user terminal 16. Below, the information processing of the CPU 20 of the operator terminal 12, and the object work and avatar control processing will be explained, but overlapping contents will be explained briefly.

[0174] 20, when the CPU 20 starts information processing of the operator terminal 12, in step S1, a connection state is established with the user terminal 16. That is, the operator enters the virtual conference room.

[0175] In the next step S3, interactive screen data for the operator is generated, and in step S5, interactive screen 150 for the operator is displayed on display device 30. Here, interactive screen 150 for the operator as shown in Fig. 5 is displayed, but since this is the initial screen, no user image is displayed in display area 152, and no avatar image is displayed in display area 154.

[0176] In the next step S7, it is determined whether the operation mode is the object work mode, that is, whether the object work flag 304q is on.

[0177] If "YES" in step S7, that is, if the operation mode is the object work mode, proceed to step S51 shown in Fig. 23. If "NO" in step S7, that is, if the operation mode is the normal mode, first manual operation data 304h is generated from the operator video data in step S9.

[0178] In the next step S11, it is determined whether or not a virtual object has been placed (or generated) in the virtual space. Here, the CPU 20 determines whether or not virtual object data 304p has been stored.

[0179] If step S11 is "NO", that is, if a virtual object is not placed in the virtual space, the process proceeds to step S17. On the other hand, if step S11 is "YES", that is, if a virtual object is placed in the virtual space, then in step S13, motion data of the arm pointing to the virtual object is generated, and in step S15, the arm motion data of the first manual motion data 304h is changed to the arm motion data generated in step S13, and the process proceeds to step S17. However, in normal mode, since the avatar's hand is in a target posture, in step S13, arm motion data is generated to maintain that posture.

[0180] In step S17, the avatar is controlled according to first manual action data 304h. That is, the avatar's actions including facial expressions are controlled. In the following step S19, avatar image data 304j is generated, and in step S21 shown in FIG. 21, it is determined whether or not voice input is present.

[0181] If "YES" in step S21, that is, if there is voice input, in step S23, the avatar image data 304j and the operator voice data 304d generated in step S19 are transmitted to the user terminal 16, and the process proceeds to step S27.

[0182] On the other hand, if "NO" in step S21, that is, if there is no voice input, in step S25, the avatar image data 304j generated in step S19 is transmitted to user terminal 16, and the process proceeds to step S27.

[0183] In step S27, it is determined whether or not data has been received from the user terminal 16. If "NO" in step S27, that is, if data has not been received from the user terminal 16, the process proceeds to step S33.

[0184] On the other hand, if "YES" in the step S27, that is, if data is received from the user terminal 16, it is determined in a step S29 whether or not the received data includes user voice data 304f.

[0185] If "NO" in step S29, that is, if the received data does not contain user voice data 304f, proceed to step S33. On the other hand, if "YES" in step S29, that is, if the received data contains user voice data 304f, in step S31, the user voice data 304f is output to the speaker 34, and proceed to step S33.

[0186] In step S33, interactive screen data on the operator side is generated (updated). Here, CPU 20 generates interactive screen data on the operator side by updating the user image in display area 152 and updating the avatar screen in display area 154. In the next step S35, interactive screen 150 on the operator side is displayed (updated).

[0187] 22, in the next step S37, it is determined whether or not there is a transition to the object work mode. Here, the CPU 20 determines whether or not the button image 160 has been turned on in the interactive screen 150 on the operator side.

[0188] If "NO" in step S37, that is, if there is no transition to the object work mode, the process proceeds to step S47. On the other hand, if "YES" in step S37, that is, if there is a transition to the object work mode, the object work flag 304q is turned on in step S39, the automatic playback flag 304s is turned on in step S41, the operator UI screen data is generated in step S43, the operator UI screen 200 is displayed on the display device 30 in step S45, and the process proceeds to step S47.

[0189] In step S47, it is determined whether the dialogue has ended. Here, the CPU 20 determines whether the operator has instructed to end the dialogue or conference. If the answer is "YES" in step S47, that is, if the dialogue has ended, the information processing of the operator terminal 12 is terminated.

[0190] On the other hand, if "NO" in step S47, that is, if the dialogue is not ended, other processes are executed in step S49, and the process returns to step S7 shown in Fig. 20. In step S49, if button image 162 is turned on in the dialogue screen 150 on the operator's side, the position, orientation, and zoom of the virtual camera are controlled according to the operation of the operator. Also, in step S49, if button image 164 is turned on in the dialogue screen 150 on the operator's side, the position and orientation of the avatar are controlled according to the operation of the operator. At this time, if a virtual object is also placed, the position of the object is also moved in parallel with the position of the avatar, and is changed to the same orientation and direction as the orientation of the avatar.

[0191] As described above, if the operation mode is the object work mode, the result is “YES” in step S7, and in step S51 shown in FIG. 23, the object work and avatar control processing described below (see FIGS. 25-27) is performed, in step S53, operator UI screen data is generated (updated), and in step S55, the operator UI screen 200 is displayed (updated) on the display device 30.

[0192] In the next step S57, it is determined whether or not the avatar's movement is being automatically played back. Here, the CPU 20 determines whether or not the autoplay flag 304s is on. If "YES" in step S57, that is, if the avatar's movement is being automatically played back, it is determined in step S59 whether or not a virtual object is placed in the virtual space.

[0193] If "NO" in step S59, the avatar is controlled according to automatic motion data 304g in step S65, and the process proceeds to step S73. On the other hand, if "YES" in step S59, motion data of an arm that points to a virtual object is generated in step S61, and the arm motion data of automatic motion data 304g is changed to the arm motion data generated in step S61 in step S63, and the process proceeds to step S65.

[0194] Also, if the result is "NO" in step S57, that is, if the avatar's movement is not being automatically reproduced, in step S67, the avatar is controlled in accordance with the second manual movement data 304i, and in step S69, it is determined whether the movement in accordance with the second manual movement data 304i has ended.

[0195] If "NO" in step S69, that is, if the operation according to the second manual operation data 304i has not ended, proceed to step S73. On the other hand, if "YES" in step S69, that is, if the operation according to the second manual operation data 304i has ended, the object work flag 304q is turned off in step S71, and proceed to step S73.

[0196] In step S73, avatar screen data 304j is generated, and in the next step S75, it is determined whether or not there is voice input, as shown in Fig. 24. If "YES" in step S75, that is, if there is voice input, in step S77, avatar screen data 304j generated in step S73 and operator voice data 304d are transmitted to user terminal 16, and the process proceeds to step S81. On the other hand, if "NO" in step S75, that is, if there is no voice input, in step S79, avatar screen data 304j generated in step S73 is transmitted to user terminal 16, and the process proceeds to step S81.

[0197] In step S81, it is determined whether or not data has been received from the user terminal 16. If "NO" in step S81, that is, if data has not been received from the user terminal 16, the process proceeds to step S47 shown in Fig. 22. On the other hand, if "YES" in step S81, that is, if data has been received from the user terminal 16, it is determined in step S83 whether or not the received data includes user voice data 304f.

[0198] If "NO" in step S83, that is, if the received data does not contain user voice data 304f, proceed to step S47. On the other hand, if "YES" in step S83, that is, if the received data contains user voice data 304f, in step S85, the user voice data 304f is output to the speaker 34, and proceed to step S47.

[0199] 25, when starting the object task and avatar control process, the CPU 20j determines in step S101 whether or not a virtual object is being generated. If “YES” in step S101, that is, if a virtual object is being generated, the process proceeds to step S121.

[0200] On the other hand, if "NO" in step S101, that is, if a virtual object is not being generated, it is determined in step S103 whether or not a virtual object is to be edited. Here, the CPU 20 determines whether or not the button image 222 is turned on.

[0201] If "YES" in step S103, that is, if a virtual object is to be edited, in step S105, a UI object is placed in place of the virtual object, and the process proceeds to step S151 shown in Fig. 27. Here, in the operator UI screen 200, the image 204 of the virtual object is erased, and a UI object image 250 corresponding to the virtual object is displayed in the position where the image 204 of the virtual object was displayed.

[0202] On the other hand, if "NO" in step S103, that is, if the operation is not editing of a virtual object, then in step S107, it is determined whether or not a virtual object is to be generated. Here, the CPU 20 determines whether or not the button image 220 is turned on.

[0203] If "NO" in step S107, that is, if a virtual object is not generated, other processing is executed in step S109, and the process proceeds to step S151. In step S109, a camera control processing or an object control processing is executed according to the operation of the operator.

[0204] On the other hand, if "YES" in step S107, that is, if a virtual object is to be generated, in step S111, it is determined whether or not two virtual objects have been generated. If "YES" in step S111, that is, if two virtual objects have been generated, in step S113, a notification of the deletion of the virtual object is made, and the process proceeds to step S151. In step S113, CPU 20 notifies the user by displaying text and / or outputting audio that a new virtual object should be generated after deleting the virtual object. The notification method is the same as in step S117 described later.

[0205] On the other hand, if "NO" in step S111, that is, if two virtual objects have not been generated, in step S115, it is determined whether the two hands flag 304r is on.

[0206] If "YES" in step S115, that is, if the two hands flag 304r is on, then in step S117, a notification of movement or deletion of the virtual object is made, and the process proceeds to step S151. On the other hand, if "NO" in step S115, that is, if the two hands flag 304r is off, then in step S119, the object selection window 240 is displayed, and the process proceeds to step S151.

[0207] As described above, if a virtual object is being generated, "YES" is determined in step S101, and in step S121 shown in FIG. 26, it is determined whether a UI object is selected in the object selection window 240.

[0208] If "YES" in step S121, that is, if a UI object is selected in the object selection window 240, in step S123, a UI object image 250 corresponding to the selection operation is displayed next to the scale comparison image 230, and the process proceeds to step S151.

[0209] On the other hand, if "NO" in step S121, that is, if it is not a selection of a UI object in the object selection window 240, it is determined in step S125 whether or not it is character input. If "YES" in step S127, that is, if it is character input, then in step S127, characters to be written on the virtual object are input according to a character input operation, and the process proceeds to step S151.

[0210] On the other hand, if "NO" in step S125, that is, if no character input is performed, then in step S129, it is determined whether or not scaling is performed. If "YES" in step S129, that is, if scaling is performed, then in step S131, the UI object is enlarged or reduced in accordance with the scaling operation, and the process proceeds to step S151. When the process of step S131 is performed, the vertical and / or horizontal length of the UI object image 250 is enlarged or reduced.

[0211] On the other hand, if "NO" in step S129, that is, if it is not enlargement / reduction, in step S133, it is determined whether or not a UI object is being moved. Here, the CPU 20 determines whether or not the UI object image 250 is being dragged.

[0212] If "YES" in step S133, that is, if the UI object is to be moved, in step S135, the UI object is moved in response to the drag operation, and the process proceeds to step S151. When the process of step S135 is executed, the UI object image 250 is moved.

[0213] On the other hand, if "NO" in step S133, that is, if the UI object is not moved, it is determined whether or not the position of the UI object is determined in step S137 shown in Fig. 27. Here, the CPU 20 determines whether or not the UI object image 250 is dropped.

[0214] If "NO" in step S137, that is, if the UI object image 250 has not been dropped, the process proceeds to step S151. On the other hand, if "YES" in step S137, that is, if the UI object image 250 has been dropped, the position and size of the virtual object in the virtual space are determined in step S139. Here, as described above, the CPU 20 determines the three-dimensional position of the virtual object corresponding to the two-dimensional position of the UI object image 250. Also, as described above, the CPU 20 determines the size of the virtual object in the virtual space based on the ratio of the size of the UI object image 250 corresponding to the scale comparison image 230.

[0215] In the next step S141, the positions of the avatar's hands are calculated according to the position and size of the virtual object. As described above, CPU 20 calculates the position of the avatar's left hand when the virtual object is on the avatar's left side, calculates the position of the avatar's right hand when the virtual object is on the avatar's right side, and calculates the positions of the avatar's both hands when the virtual object is in front of the avatar's torso.

[0216] Next, in step S143, second manual motion data of the avatar is generated. Here, CPU 20 calculates the motion of the hand whose position was calculated in step S139 according to an inverse kinematics algorithm, and calculates the other motion data based on the operator video data.

[0217] In the following step S145, the UI object image 250 is erased, and in step S147, a virtual object corresponding to the UI object image 250 is generated in the virtual space. However, the virtual object is generated (or placed) in the virtual space at the three-dimensional position determined in step S139 and with the size determined in step S139.

[0218] In the next step S149, the automatic playback flag 304s is turned off, and the process proceeds to step S151. After the process of step S149 is executed, the avatar is controlled in accordance with the second manual operation data in step S55, and therefore, in step S61, avatar screen data 304j is generated in which the avatar is operated to point to a virtual object.

[0219] In step S151, it is determined whether the object work mode has ended. Here, the CPU 20 determines whether the operator has instructed to end the object work mode. If "NO" in step S151, that is, if the object work mode has not ended, the process returns to the information processing of the operator terminal 12. On the other hand, if "YES" in step S151, that is, if the object work mode has ended, in step S153, the object work flag 304q is turned off and the process returns to the information processing of the operator terminal 12.

[0220] Fig. 28 is a flow diagram showing an example of information processing of the CPU 50 of the user terminal 16 shown in Fig. 3. Although not shown, the CPU 580 performs operations in parallel with the information processing of the user terminal 16, such as performing an operation data detection process to store the operation data, performing an image capture process with the camera 66 to store user video data, performing an audio detection process to store user audio data, and performing a process to receive and store data transmitted from the operator terminal 12. The information processing of the CPU 50 of the user terminal 16 will be described below, but overlapping contents will be briefly described.

[0221] 28, the CPU 50 starts information processing of the user terminal 16, and in step S201, establishes a connection state with the operator terminal 12. That is, the user enters the virtual conference room.

[0222] In the next step S203, it is determined whether or not there is voice input. If "YES" in step S203, that is, if there is voice input, in step S205, the user video data 404c and the user voice data 404d are transmitted to the operator terminal 12, and the process proceeds to step S209. On the other hand, if "NO" in step S203, that is, if there is no voice input, in step S207, the user video data 404c is transmitted to the operator terminal 12, and the process proceeds to step S209.

[0223] In step S209, it is determined whether or not data has been received from the operator terminal 12. If "NO" in step S209, that is, if data has not been received from the operator terminal 12, the process proceeds to step S217. On the other hand, if "YES" in step S209, that is, if data has been received from the operator terminal 12, it is determined in step S211 whether or not operator voice data 404f is present.

[0224] If "NO" in step S211, that is, if there is no operator voice data 404f, proceed to step S215. On the other hand, if "YES" in step S211, that is, if there is operator voice data 404f, the operator voice data 404f is output to the speaker 64 in step S213, and proceed to step S215.

[0225] In step S215, interactive screen data for the user is generated, and in step S217, interactive screen 100 for the user is displayed (updated) on display device 60, and the process proceeds to step S219. In step S215, avatar image 110 is lip-synchronized with the voice of the operator.

[0226] In step S219, it is determined whether the dialogue has ended. Here, the CPU 50 determines whether the user has instructed to end the dialogue (or the conference). If "YES" in step S219, that is, if the dialogue has ended, the information processing of the user terminal 16 is terminated. On the other hand, if "NO" in step S219, that is, if the dialogue has not ended, the process returns to step S203.

[0227] According to this embodiment, a virtual object corresponding to a UI object selected and placed on the operator UI screen is placed in the virtual space, and motion data of the arm of the avatar pointing to this virtual object is automatically generated based on the position and size of this virtual object, so that the virtual object can be pointed to more easily and appropriately than when the avatar's motion is based only on the operator's motion, which means that the control of the avatar can be improved.

[0228] In this embodiment, in the normal mode, the avatar's movement is controlled using the first manual movement data generated from the video data of the operator, but this is not limited to this. The avatar's movement may be controlled based on the automatic movement data of the embodiment, which corresponds to a series of movements to be performed by the avatar.

[0229] In this embodiment, the avatar screen data is generated in the operator terminal and transmitted to the user terminal, but the avatar screen data may be generated in the user terminal. In such a case, the operator terminal transmits avatar motion data (automatic motion data, first manual motion data, or second manual motion data), avatar data, virtual camera data, and virtual object data to the user terminal instead of the avatar screen data. The image generation data stored in the user terminal also includes image data for the avatar and various virtual objects.

[0230] Furthermore, in this embodiment, the operator generates and edits the virtual object, but the user may generate and edit the virtual object. In such a case, the UI screen as shown in the embodiment is displayed on the user terminal. In this case, the type of UI object image arranged by the user or the type of virtual object corresponding to the UI object image, and data on the position where the UI object image is arranged are transmitted from the user terminal to the operator terminal. In other words, the user instructs the operator terminal to generate and edit the virtual object.

[0231] Furthermore, in this embodiment, a two-dimensional or three-dimensional virtual object prepared in advance is placed in the virtual space, and an image of the virtual object is displayed on the user's interactive screen, but this is not necessary. The loaded image may be displayed on the user's interactive screen. In such a case, the loaded image is pasted as a texture on the surface of a predetermined two-dimensional or three-dimensional virtual object.

[0232] It should be noted that the order of steps in the flow charts shown in the above embodiments can be changed if the same results are obtained.

[0233] Furthermore, the various screens and specific numerical values ​​given in the above embodiment are merely examples and can be changed as necessary. [Explanation of symbols]

[0234] 10. Information processing systems 12 …Operator terminal 14 … Network 16...User terminal 18 ... Server 20, 50...CPU 22, 52...RAM 24, 54 ... Communication I / F 26, 56 ... Input / Output I / F 28, 58 ... Input device 30, 60...display device 32, 62...Mike 34, 64 ... Speaker 36, 66... ​​Camera

Claims

1. An information processing device used by an operator who responds to a user using an avatar, camera, an avatar motion data generating means for generating motion data of the avatar corresponding to the operator in a three-dimensional virtual space based on an image of the operator photographed by the camera; avatar control means for controlling the movement of the avatar in accordance with the movement data generated by the avatar movement data generation means; an image generating means for generating display image data corresponding to an image captured by a virtual camera of a three-dimensional virtual space; A transmitting means for transmitting at least the display image data to a user terminal used by the user; a placement means for placing an object in the three-dimensional virtual space in response to a placement operation by the operator; a hand / arm motion data generating means for generating hand / arm motion data regarding a motion of the hand / arm of the avatar for pointing to the object based on a size and a position of the object arranged by the arrangement means; and an information processing apparatus comprising: a changer for changing hand and arm motion data included in the motion data generated by the avatar motion data generation means to the hand and arm motion data generated by the hand and arm motion data generation means;

2. microphone, Speaker, a receiving means for receiving user voice data corresponding to the user's voice; and an output unit for outputting the user voice data received by the receiving unit to the speaker; The information processing apparatus according to claim 1 , wherein the transmitting means further transmits operator voice data corresponding to the operator's voice detected by the microphone to the user terminal.

3. The display device further includes a display unit for displaying a UI screen for selecting and arranging UI object images, 3 . The information processing apparatus according to claim 1 , wherein the arranging means arranges, in the three-dimensional virtual space, the object corresponding to the UI object image selected and arranged on the UI screen.

4. the avatar is placed at a predetermined position in the three-dimensional virtual space; the UI screen includes an image of the avatar; an object position determination means for determining an object position, which is a three-dimensional position of the object in the three-dimensional virtual space, based on a two-dimensional positional relationship of the UI object image with respect to the image of the avatar on the UI screen and a position in a depth direction determined so that a distance between the avatar and the object in the three-dimensional virtual space is a predetermined distance; and a hand position determining means for determining a three-dimensional position of a hand of the avatar pointing to the object based on the object position; 4. The information processing device according to claim 3, wherein the arm movement data generation means generates arm movement data for changing from a current posture of the avatar's arm to a posture of the arm pointing to the object, in accordance with an inverse kinematics algorithm using parameters relating to the three-dimensional positions of the avatar's hands and head, the size of the object, and the manner in which the object is pointed to.

5. An information processing program executed by a computer equipped with a camera and used by an operator who responds to users using an avatar, a processor of the computer, an avatar motion data generating step of generating motion data of the avatar corresponding to the operator in a three-dimensional virtual space based on an image of the operator captured by the camera; an avatar control step of controlling a movement of the avatar in accordance with the movement data generated in the avatar movement data generation step; an image generating step of generating display image data corresponding to an image captured by a virtual camera of the three-dimensional virtual space; a transmitting step of transmitting at least the display image data to a user terminal used by the user; a placement step of placing an object in the three-dimensional virtual space in response to a placement operation by the operator; a hand / arm motion data generating step of generating hand / arm motion data regarding a motion of the hand / arm of the avatar pointing to the object based on a size and a position of the object arranged in the arrangement step; and an information processing program that causes a user to execute a changing step of changing hand and arm movement data included in the movement data generated in the avatar movement data generating step to the hand and arm movement data generated in the hand and arm movement data generating step.

6. An information processing method for a computer used by an operator who is equipped with a camera and who responds to users using an avatar, comprising the steps of: The computer includes: (a) generating motion data of the avatar corresponding to the operator in a three-dimensional virtual space based on an image of the operator captured by the camera; (b) controlling a movement of the avatar in accordance with the movement data generated in step (a); (c) generating display image data corresponding to an image captured by a virtual camera of the three-dimensional virtual space; (d) a transmission step of transmitting at least the display image data to a user terminal used by the user; (e) placing an object in the three-dimensional virtual space in response to a placement operation by the operator; (f) generating hand and arm movement data regarding the movement of the hand and arm of the avatar pointing to the object based on the size and position of the object arranged in the step (e); and (g) executing a step of changing the hand / arm movement data included in the movement data generated in said step (a) to the hand / arm movement data generated in said step (f).

7. An information processing system including an operator terminal used by an operator who responds to a user using an avatar and a user terminal used by the user, the information processing system being capable of communicating with each other, The operator terminal includes: camera, an avatar motion data generating means for generating motion data of the avatar corresponding to the operator in a three-dimensional virtual space based on an image of the operator photographed by the camera; avatar control means for controlling the movement of the avatar in accordance with the movement data generated by the avatar movement data generation means; an image generating means for generating display image data corresponding to an image captured by a virtual camera of a three-dimensional virtual space; A transmitting means for transmitting at least the display image data to a user terminal used by the user; an arrangement means for arranging an object in the three-dimensional virtual space in response to an arrangement operation by the operator or the user; a hand / arm motion data generating means for generating hand / arm motion data regarding a motion of the hand / arm of the avatar for pointing to the object based on a size and a position of the object arranged by the arrangement means; and an information processing system comprising: a changer for changing hand and arm movement data included in the movement data generated by the avatar movement data generator to the hand and arm movement data generated by the hand and arm movement data generator;

Images