Information processing device, control method for information processing device, and program

Abstract

It provides the intended visuals and sounds for users in a virtual space. [Solution] The information processing device includes placement means for placing an avatar, viewpoint, listening point, and sound point relating to the user in a virtual space; video generation means for generating a virtual viewpoint video corresponding to the placed viewpoint based on 3D data of the virtual space; sound generation means for generating virtual listening point sound data corresponding to the placed listening point based on acquired sound data; and playback means for playing back the generated virtual viewpoint video and virtual listening point sound data. The placement means is capable of placing at least one of the viewpoint, listening point, and sound point independently of the avatar.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a program.

Background Art

[0002] Using viewing terminals such as HMD (Head Mount Display), PC (Personal Computer), smartphone, and tablet, access to a virtual space and viewing of images in the virtual space are performed. Also, an avatar that is the user's alter ego is placed in the virtual space, and communication is enabled by operating the avatar according to input from the user. Also, conversation between users is realized by exchanging the users' voices. Patent Document 1 discloses a technique for providing a sound field closer to reality for each user by mixing voices according to the position and posture of the user's alter ego in the virtual space.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, the relative positions of each user used for audio mixing correspond to the avatar or viewpoint position. That is, just as in real space, in virtual space, the listening point (ears) and sound production point (mouth) move in accordance with the avatar or viewpoint (eyes) position. However, for example, when watching a game in virtual space, there are times when one wants to cheer together with other users watching from the stands, while also wanting to watch the exciting plays from a position closer to the players on the field. In this case, if the avatar moves near the players, it will not only become visible to other users and be a nuisance, but it will also move away from the stands, making it impossible to cheer together. This is just one example, but it is conceivable that the viewpoint, listening point, sound production point, and the position and posture of the avatar desired by content providers and users are not necessarily the same. If the avatar and the viewpoint, listening point, and sound production point are in the same position or posture, it may not be possible to provide the video and sound intended by the user in virtual space. [Means for solving the problem]

[0005] The information processing device according to the present invention comprises: placement means for placing an avatar, viewpoint, listening point, and sound point relating to a user in a virtual space; first image generation means for generating a virtual viewpoint image corresponding to the placed viewpoint based on 3D data of the virtual space; sound generation means for generating virtual listening point sound data corresponding to the placed listening point based on acquired sound data; and playback means for playing back the generated virtual viewpoint image and virtual listening point sound data, wherein the placement means is capable of placing at least one of the viewpoint, listening point, and sound point independently of the avatar. [Effects of the Invention]

[0006] This makes it possible to provide users with the desired visuals and sounds in a virtual space. [Brief explanation of the drawing]

[0007] [Figure 1] This figure shows an example of the hardware configuration of an information processing device. [Figure 2]This figure shows an example of the functional configuration of the information processing device in Embodiment 1. [Figure 3] This figure shows an example of the configuration of the avatar control unit in Embodiment 1. [Figure 4] This is a flowchart illustrating an example of processing in the information processing device in Embodiment 1. [Figure 5] This figure shows an example of a virtual viewpoint video viewing screen in Embodiment 1. [Figure 6] This figure shows an example of the user point placement screen in Embodiment 1. [Figure 7] This figure shows an example of the user point placement screen in Embodiment 1. [Figure 8] This figure shows an example of the configuration of the avatar control unit in Embodiment 2. [Figure 9] This is a flowchart illustrating an example of processing in the information processing device in Embodiment 2. [Figure 10] This figure shows an example of the configuration of the avatar control unit in Embodiment 3. [Figure 11] This figure shows an example of the configuration of the avatar control unit in Embodiment 4. [Figure 12] This is a flowchart illustrating an example of processing in the information processing device according to Embodiment 4. [Figure 13] This figure shows an example of the functional configuration of the information processing device in Embodiment 5. [Figure 14] This is a flowchart illustrating an example of processing in the information processing device in Embodiment 5. [Figure 15] This figure shows an example of the configuration of the avatar control unit in Embodiment 6. [Figure 16] This is a flowchart illustrating an example of processing in the information processing device in Embodiment 6. [Figure 17] This figure shows an example of the configuration of the avatar control unit in Embodiment 7. [Figure 18] This is a flowchart illustrating an example of processing in the information processing device according to Embodiment 7. [Modes for carrying out the invention]

[0008] Hereinafter, embodiments of the present invention will be described based on the drawings. Note that the embodiments described below do not limit the present invention. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Also, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted. In the following description, the four points of the avatar, viewpoint, listening point, and pronunciation point regarding the user in the virtual space are collectively referred to as the user point.

[0009] <Embodiment 1> FIG. 1 is a block diagram showing a hardware configuration example of an information processing apparatus 100 in the present embodiment. The information processing apparatus 100 includes a CPU 111, a ROM 112, a RAM 113, an auxiliary storage device 114, a display unit 115, an operation unit 116, a communication I / F 117, a sound data output unit 118, a sound data input unit 119, and a bus 120. The CPU 111, the ROM 112, the RAM 113, the auxiliary storage device 114, the display unit 115, the operation unit 116, the communication I / F 117, the sound data output unit 118, and the sound data input unit 119 are connected to be communicable with each other via the bus 120.

[0010] The CPU (Central Processing Unit) 111 realizes each function of the information processing apparatus 100 by controlling the entire information processing apparatus using computer programs and data stored in the ROM 112 and the RAM 113. For example, the CPU 111 reads a computer program stored in the ROM 112 or the like, expands it in the RAM 113, and executes it, thereby realizing processing according to each flowchart described later. Note that the information processing apparatus 100 may have one or more dedicated hardware different from the CPU 111, and at least a part of the processing by the CPU 111 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor).

[0011] The ROM (Read Only Memory) 112 stores programs, data, etc. that do not require modification. The RAM (Random Access Memory) 113 temporarily stores programs and data supplied from the auxiliary storage device 114, and data, etc. supplied from the outside via the communication I / F 117. The auxiliary storage device 114 is composed of, for example, a hard disk drive (HDD), a solid state drive (SSD), etc., and stores various data such as image data and sound data.

[0012] The display unit 115 is composed of, for example, a liquid crystal display, an LED, etc., and displays the generated virtual viewpoint video, a GUI (Graphical User Interface) for the user to operate the information processing device 100, etc. The operation unit 116 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and receives operations by the user and inputs various instructions to the CPU 111. The CPU 111 also operates as a display control unit that controls the display unit 115 and an operation control unit that controls the operation unit 116. Furthermore, the CPU 111 is also responsible for controlling the ROM 112 and the RAM 113, data transfer control from the auxiliary storage device 114, and control of the bus 120 and the communication I / F 117. In addition, the CPU 111 also operates as a data transmission control unit to the sound data output unit 118 and a data reception control unit from the sound data input unit 119.

[0013] The communication I / F 117 is used for communication with external devices. For example, when the information processing device 100 is connected to an external device by wire, a communication cable is connected to the communication I / F 117. Also, for example, when the information processing device 100 has a function of wireless communication with an external device, the communication I / F 117 includes an antenna. The bus 120 connects each part of the information processing device 100 and transmits information. The sound data output unit 118 is composed of, for example, a speaker, etc., and reproduces the sound data stored in the information processing device 100 and the sound data input via the communication I / F 117. The sound data input unit 119 is composed of, for example, a microphone, etc., and acquires sound data related to the voice of the user using the information processing device 100 and the ambient environmental sound.

[0014] In this embodiment, the display unit 115, the operation unit 116, the sound data output unit 118, and the sound data input unit 119 are assumed to be located inside the information processing device 100. However, at least a portion of the display unit 115, the operation unit 116, the sound data output unit 118, and the sound data input unit 119 may exist as separate devices outside the information processing device 100.

[0015] Figure 2 is a block diagram showing an example of the functional configuration of the information processing device 100 in Embodiment 1. The information processing device 100 includes a 3D data acquisition unit 211, a content sound data acquisition unit 212, a storage unit 213, an avatar control unit 214, a virtual viewpoint image generation unit 215, an overhead view image generation unit 216, an audio generation unit 217, and an image control unit 218. The information processing device 100 also includes an image display unit 219, an audio data output unit 220, an audio data input unit 221, and an operation unit 222. Each part constituting the information processing device 100 may function using, for example, a single virtual viewpoint viewing device such as an HMD, but some functions may be implemented on an external server.

[0016] The 3D data acquisition unit 211 acquires 3D data of the virtual space, which is the content viewed by the user. The 3D data includes, for example, background data such as terrain and structures, and foreground data such as performers and athletes. In this embodiment, the 3D data is assumed to be a mesh model including texture information. However, the 3D data is not limited to a mesh model including texture information; any information that can display 3D shapes in the virtual space is acceptable. Furthermore, 3D data created and saved in advance using external applications or software may be used. In addition, 3D data created by performing shooting and shape estimation processing using multiple imaging devices may be used, and the shooting and shape estimation processing may be performed in real time.

[0017] The content sound data acquisition unit 212 acquires sound data of the content that the user is viewing. The sound data includes, for example, the voices of performers or athletes, music, sound effects, etc., and three-dimensional coordinate information in virtual space is associated with at least one sound source. Similar to the 3D data mentioned above, pre-recorded and created sound data may be used, or sound data captured in real time may be used.

[0018] The memory unit 213 stores 3D data of the virtual space acquired by the 3D data acquisition unit 211 and sound data acquired by the content sound data acquisition unit 212. It is also possible to store user point information of other users in the memory unit 213. The user point information to be stored includes the three-dimensional coordinates of the user point, posture information (including orientation), and at least one of the input sound data. By storing user point information in the memory unit 213, other users can see the avatar and hear the user's voice when viewing the virtual viewpoint video.

[0019] The avatar control unit 214 receives user operations input via the operation unit 222 and sound data input via the sound data input unit 221, and transmits information about each user point to the virtual viewpoint image generation unit 215, the overhead view image generation unit 216, and the sound generation unit 217. Details of the avatar control unit 214 will be described later. The avatar control unit 214 is an example of a placement method.

[0020] The virtual viewpoint video generation unit 215 generates video data of a viewpoint (virtual viewpoint position) set in the virtual space. The virtual viewpoint video generation unit 215 is an example of the first video generation means. For example, the virtual viewpoint video generation unit 215 acquires 3D data of the virtual space and 3D model data of an avatar that represents the user from the storage unit 213. The virtual viewpoint video generation unit 215 also acquires at least the three-dimensional coordinates and orientation information of the viewpoint and the three-dimensional coordinates and orientation information of the avatar transmitted from the avatar control unit 214. Then, based on the acquired viewpoint and avatar information, the virtual viewpoint video generation unit 215 generates video data of the virtual viewpoint position (virtual viewpoint video) corresponding to the viewpoint set in the virtual space. If the avatar is placed within the field of view, it is possible to see one's own avatar, but the user may choose to hide their own avatar.

[0021] The overhead viewpoint video generation unit 216 generates video data of an overhead viewpoint position, which is an overhead view (bird's-eye view) of the virtual space. The overhead viewpoint video generation unit 216 is an example of a second video generation means. For example, the overhead viewpoint video generation unit 216 acquires 3D data of the virtual space and 3D model data of the avatar from the storage unit 213. The overhead viewpoint video generation unit 216 also acquires at least the three-dimensional coordinates and pose information of the viewpoint and the three-dimensional coordinates of the avatar transmitted from the avatar control unit 214. Then, based on the acquired viewpoint and avatar information, the overhead viewpoint video generation unit 216 generates video data of an overhead viewpoint position (overhead viewpoint video) that provides an overhead view of the virtual space. The processing in the overhead viewpoint video generation unit 216 is the same as that of the virtual viewpoint video generation unit 215, but the overhead viewpoint video generation unit 216 generates video for displaying the user point placement screen 600, which will be described later. To generate the image used for user point placement, the 3D data of the virtual space does not need to include a foreground. Furthermore, if the shape of the virtual space is verifiable, the 3D data of the virtual space does not need to include texture information.

[0022] The sound generation unit 217 generates sound data for listening points (virtual listening point locations) set in the virtual space. The sound generation unit 217 is an example of a sound generation means. For example, the sound generation unit 217 acquires sound data of the virtual space, including sound source placement information, from the storage unit 213. The sound generation unit 217 also acquires at least the three-dimensional coordinates and sound data of the sound source point, and the three-dimensional coordinates and posture information of the listening point, transmitted from the avatar control unit 214. Then, based on the acquired information on the sound source point and listening point, the sound generation unit 217 generates sound data for the virtual listening point location (virtual listening point sound data) corresponding to the listening point set in the virtual space. The sound generation unit 217 may further acquire additional information, such as the sound absorption characteristics of structures in the virtual space, from the storage unit 213 and perform sound generation processing to reproduce a more three-dimensional sound.

[0023] The video control unit 218 switches the virtual space video to be displayed and overlays the user interface (UI) for operation based on the input from the operation unit 222, and outputs video data to the video display unit 219. The virtual space videos to be switched are the video generated by the virtual viewpoint video generation unit 215 (virtual viewpoint video) and the video generated by the overhead viewpoint video generation unit 216 (overhead viewpoint video).

[0024] The video display unit 219 displays the video output from the video control unit 218. The sound data output unit 220 receives sound data generated by the sound generation unit 217, plays back the sound data, and outputs it. The video display unit 219 and the sound data output unit 220 are examples of playback means. The sound data input unit 221 inputs the voice spoken by the user while viewing the virtual viewpoint video, or sound effects input through external devices, to the avatar control unit 214. If multiple sound data are input, the avatar control unit 214 may manage them as individual sound sources. Alternatively, the sound data input unit 221 may mix multiple sound data and treat them as a single sound data. The operation unit 222 inputs the user's operations to the avatar control unit 214 and the video control unit 218.

[0025] Figure 3 is a block diagram showing an example configuration of the avatar control unit 214 in Embodiment 1. The avatar control unit 214 includes a user point management unit 311, an avatar information management unit 312, a viewpoint information management unit 313, a listening point information management unit 314, and a sound point information management unit 315.

[0026] The user point management unit 311 manages information and the number of placements for each user point based on user operations input by the operation unit 222. User point information includes at least link information, three-dimensional coordinates, and posture information (including orientation) between user points (avatar, viewpoint, listening point, and sound point), and the data format is not restricted. The link information between user points allows the user to select and move multiple points at once by sharing three-dimensional coordinates and posture information for two or more points. Initially, the viewpoint, listening point, and sound point are all in the same position and posture as the avatar. That is, in the initial state, the three-dimensional coordinates and posture information of the avatar, viewpoint, listening point, and sound point are the same. Furthermore, in this embodiment, if multiple user points exist at the same coordinates, the user point placement screen 600, described later, will only display the model of the specific user point designated by the user point management unit 311. This prevents a decrease in visibility due to the overlapping display of models on the screen.

[0027] Furthermore, the user point information managed by the user point management unit 311 can be newly set or changed by the user via the operation unit 222. The user point management unit 311 reflects the input from the operation unit 222 to the information management units 312-315 for each user point according to the link status of the user points. As mentioned above, unless set in advance, the initial values ​​of the three-dimensional coordinates and posture information of each user point are the same as those of the avatar. This makes it possible to operate in the same way as conventional virtual viewpoint viewing when using the initial values. Limitations on the number and placement of each user point can be set for the memory unit 213 or the user point management unit 311 to prevent individual users from placing a large number of user points. In addition, time information may be added to the information of each user point managed by the user point management unit 311. By individually setting time information for each user point, for example, it becomes possible to have a real-time conversation with other users while replaying the video of the viewpoint position. The user may input all the information for each user point via the operation unit 222, or some or all of the information may be automatically input. Additionally, default values ​​may be set for the information of each user point.

[0028] The avatar information management unit 312 manages user point information (avatar information) input from the user point management unit 311 and avatar 3D model data input from the storage unit 213. The avatar model data may be generated by acquiring 2D images and giving a certain thickness to the 2D images, rather than directly acquiring 3D data. Alternatively, the avatar model data may be created by creating a 3D model of the avatar from 2D images using machine learning or the like. In other words, the avatar 3D model data is not limited to 2D / 3D or data format, as long as the avatar can be visualized in the virtual space. When multiple avatars are placed in the virtual space, the avatar information management unit 312 manages the avatar information individually. The avatar information management unit 312 also outputs the managed avatar information to the virtual viewpoint video generation unit 215 and the overhead view video generation unit 216. The avatar information includes at least three-dimensional coordinates and pose information.

[0029] The viewpoint information management unit 313 manages user point information (viewpoint information) input from the user point management unit 311 and 3D model data of viewpoints input from the storage unit 213. The viewpoint model data can be 2D / 3D or in any data format, as long as it can visualize the viewpoint in the virtual space, similar to the avatar model data. When multiple viewpoints are placed in the virtual space, the viewpoint information management unit 313 manages the viewpoint information for each viewpoint individually. The viewpoint information management unit 313 also outputs the managed viewpoint information to the virtual viewpoint image generation unit 215 and the overhead viewpoint image generation unit 216. The viewpoint information includes at least three-dimensional coordinates and orientation information. The viewpoint information may also include multiple types of virtual camera parameters such as focal length, field of view, depth of field, and exposure. Here, the three-dimensional coordinates and orientation information of the viewpoint when generating the overhead viewpoint image have initial settings such as x:0, y:0, z:+20m, and the x-axis is oriented in the positive direction.

[0030] The listening point information management unit 314 manages user point information (listening point information) input from the user point management unit 311 and 3D model data of listening points input from the storage unit 213. The listening point model data can be 2D / 3D or in any data format, as long as it can visualize the listening point in the virtual space, similar to the avatar model data. When multiple listening points are placed in the virtual space, the listening point information management unit 314 manages the listening point information for each point individually. The listening point information management unit 314 also outputs the managed listening point information to the sound generation unit 217. The listening point information includes at least three-dimensional coordinates and orientation information.

[0031] The sound point information management unit 315 manages user point information (sound point information) input from the user point management unit 311 and 3D model data of sound points input from the storage unit 213. The model data of sound points can be 2D / 3D or in any data format, as long as it can visualize the sound points in the virtual space, similar to the model data of avatars. When multiple sound points are placed in the virtual space, the sound point information management unit 315 manages the sound point information for each individual sound point. The sound point information management unit 315 also outputs the managed sound point information to the sound generation unit 217. The sound point information includes at least sound data and three-dimensional coordinates.

[0032] Referring to Figure 4, the operation of the information processing device 100 in Embodiment 1 will be described. Figure 4 is a flowchart illustrating an example of processing by the information processing device 100 in Embodiment 1.

[0033] In step S401, the 3D data acquisition unit 211 acquires 3D data of the virtual space, which is the content the user is viewing, and stores it in the storage unit 213. In addition, the content sound data acquisition unit 212 acquires sound data of the content the user is viewing and stores it in the storage unit 213.

[0034] In step S402, the overhead view image generation unit 216 generates an overhead view image of the virtual space. The overhead view image generation unit 216 generates an overhead view image of the virtual space based on the 3D data of the virtual space acquired in step S401 and stored in the storage unit 213, and the three-dimensional coordinate and orientation information related to the overhead view set according to the initial settings or user operation. The overhead view image generated by the overhead view image generation unit 216 is displayed on the video display unit 219 via the video control unit 218. The user can view the virtual space from above using the overhead view image displayed on the video display unit 219 and set the position and orientation of user points (avatar, viewpoint, listening point, and sound point) via the operation unit 222. The position and orientation of the avatar, viewpoint, listening point, and sound point can each be set independently. That is, in this embodiment, the viewpoint, listening point, and sound point can be placed independently of the avatar. User point information, including three-dimensional coordinates and orientation information related to the position and orientation of the user point set by the user, is input to and managed by the avatar control unit 214.

[0035] In step S403, the avatar control unit 214 transmits information about each user point set by the user in step S402 to the virtual viewpoint image generation unit 215, the overhead view image generation unit 216, and the sound generation unit 217. The avatar control unit 214 transmits avatar and viewpoint information, including three-dimensional coordinates and posture information, to the virtual viewpoint image generation unit 215 and the overhead view image generation unit 216. In addition to this information, virtual camera parameters may also be transmitted. The avatar control unit 214 also transmits listening point and sound point information, including three-dimensional coordinates and posture information, to the sound generation unit 217.

[0036] In step S404, the virtual viewpoint image generation unit 215 generates a virtual viewpoint image corresponding to a viewpoint placed in the virtual space, based on the avatar and viewpoint information transmitted from the avatar control unit 214 in step S403. The virtual viewpoint image generation unit 215 generates the virtual viewpoint image based, for example, on the three-dimensional coordinate and pose information of the avatar and viewpoint input from the avatar control unit 214 and the avatar model data. In step S404, the same information is also input to the overhead viewpoint image generation unit 216, and the overhead viewpoint image generation unit 216 generates an overhead viewpoint image in the virtual space. The virtual viewpoint image and overhead viewpoint image generated by the virtual viewpoint image generation unit 215 and the overhead viewpoint image generation unit 216, respectively, are transmitted to the image control unit 218.

[0037] In step S405, the sound generation unit 217 generates virtual listening point sound data corresponding to listening points placed in the virtual space, in accordance with the listening point and sound point information transmitted from the avatar control unit 214 in step S403. The sound generation unit 217 generates virtual listening point sound data based, for example, on the three-dimensional coordinates and posture information of the listening point and sound point input from the avatar control unit 214 and the sound data input to the sound data input unit 221. The sound data input to the sound data input unit 221 includes, for example, sound data such as the user's voice and sound effects generated in accordance with the avatar's actions. The virtual listening point sound data generated by the sound generation unit 217 is output to the sound data output unit 220.

[0038] In step S406, the video control unit 218 determines the video to be displayed according to the input from the operation unit 222, and outputs the virtual viewpoint video generated by the virtual viewpoint video generation unit 215 to the video display unit 219 for display. Also, the virtual listening point sound data generated by the sound generation unit 217 is input to the sound data output unit 220, and the sound data output unit 220 plays back and outputs the sound data. The video control unit 218 may also output the overhead viewpoint video generated by the overhead viewpoint video generation unit 216 to the video display unit 219 for display according to the input from the operation unit 222. As described above, the processing of this embodiment allows individual users accessing the virtual space to play back the virtual viewpoint video and sound data they desire.

[0039] Next, with reference to Figure 5, the GUI for the user to operate the virtual viewpoint will be described. Figure 5 is a diagram showing an example of the virtual viewpoint video viewing screen 500 displayed on the video display unit 219. The user can operate the virtual viewpoint video viewing screen 500 via the operation unit 222. In this embodiment, operation via the operation unit 222 is also assumed to include operation using a touch panel on a smartphone or tablet device, and typical operations of the virtual viewpoint video viewing screen 500 and the user point placement screen 600, which will be described later, are shown on the UI. However, depending on the form of the operation unit 222, operation is not limited to being performed on the UI. The virtual viewpoint video viewing screen 500 has, for example, a user point movement button 501, a user point direction change button 502, an up button 503, and a down button 504 as operation elements in the initial operable GUI configuration. The virtual viewpoint video viewing screen 500 also has, for example, an avatar position return button 505, a detailed settings button 506, and a placement mode activation button 507 as operation elements in the initial operable GUI configuration. In the following explanation, the coordinate system of three-dimensional coordinates is assumed to be a Cartesian coordinate system with x, y, and z axes, and the plane containing the x and y axes (the xy-plane) is defined as the horizontal direction.

[0040] The user point movement button 501 is a button that moves the user point horizontally. On the virtual viewpoint video viewing screen 500, the direction the viewpoint is facing is the reference for the direction of movement. That is, for example, if you press the up button of the user point movement button 501, the user point will move in the direction in front of the viewpoint, and if you press the down button, the user point will move in the direction behind the viewpoint. The user point direction change button 502 is a button that changes the direction of the user point. The up button 503 is a button that moves the user point in the positive z-axis direction. The down button 504 is a button that moves the user point in the negative z-axis direction.

[0041] The avatar position return button 505 is a button that returns the position and orientation of the user point being operated to the same position and orientation as the avatar. For example, as a default setting, the position and orientation of the viewpoint, listening point, and sound point may be temporarily returned to the same position and orientation as the avatar while the avatar position return button 505 is held down. Alternatively, by changing the settings, pressing the avatar position return button 505 may return only some of the viewpoint, listening point, and sound point to the same position and orientation as the avatar. Furthermore, pressing the avatar position return button 505 again may return all user points to the same position and orientation as the avatar (reset to their initial configuration).

[0042] The advanced settings button 506 functions as a menu button on the virtual viewpoint video viewing screen 500. For example, pressing this advanced settings button 506 may enable actions such as hiding the GUI or jumping to the listening point position or sound point position. Furthermore, the user may assign additional functions to the advanced settings button 506 at their discretion. The placement mode activation button 507 is a button that, when selected, transitions to the user point placement screen 600.

[0043] Next, with reference to Figure 6, a GUI for the user to change the position and orientation of user points (avatar, viewpoint, listening point, sound point) in the virtual space will be described. Figure 6 is a diagram showing an example of the user point placement screen 600 displayed on the video display unit 219. Operations on the user point placement screen 600 can be performed by the user via the operation unit 222. The user point placement screen 600 has, as an initial operable GUI configuration, for example, a user point list 603, a user point edit button 604, and a map rotation button 605 as operable elements. The user point placement screen 600 also has, as an initial operable GUI configuration, for example, a user point move button 606, a user point orientation change button 607, an up button 608, a down button 609, a viewpoint preview button 610, and an exit button 611 as operable elements.

[0044] In the user point placement screen 600, the viewpoint model 601 displays the viewpoint position in the virtual space as a 3D model. In the example shown in Figure 6, the viewpoint and the listening point are close together, and the 3D model display of the listening point is omitted. By omitting the 3D model display when the positions of each user point are close together in this way, the visibility of the user points can be improved. The 3D model to be displayed preferentially when each user point is close together may be set arbitrarily by the user, or it may be determined based on a pre-specified display priority. Here, user points that are close together within a certain distance are linked, and at least the three-dimensional coordinates of the user point information are shared. Linking of close user points is not automatic and may be set arbitrarily by the user. In addition, the avatar model 602 displays a 3D model of the avatar in the virtual space based on the three-dimensional coordinates and posture information entered by the user and the avatar's model data. In the example shown in Figure 6, the avatar and the sound point are close together and linked, and the 3D model display of the sound point is omitted. For example, by selecting a 3D model displayed on the user point placement screen 600 using the operation unit 222, the selected 3D model is highlighted and can be selected as a user point to be operated on.

[0045] User point list 603 shows a list of user points placed in the virtual space. In the example shown in Figure 6, four points—the viewpoint, listening point, pronunciation point, and avatar—are placed in the virtual space as user points. For example, in user point list 603, the link relationships between each user point are shown with lines and link symbols, and the currently selected user point is highlighted. In this example, the highlighting is shown by making the border of each user point thicker, but other methods such as changing the color or animation may also be used.

[0046] The User Point Edit button 604, when selected, allows editing of user points, such as linking / unlinking user points, creating new user points, and deleting user points. Selecting the User Point Edit button 604 changes the display of the User Point List 603, as shown in Figure 7(a), and the User Point Details Edit button 701, enclosed by a dotted line in Figure 7(a), is temporarily added. The user selects the checkbox for any user point in the User Point List 603 and selects the editing content using the User Point Details Edit button 701, and the editing content is applied to the selected user point. For example, if the user selects to create a new user point, a new point 711 is added as shown in Figure 7(b), a new user point 712 is created near the avatar, and a pop-up 713 prompting the user to select the type of user point (viewpoint, listening point, pronunciation point, avatar) is displayed. By selecting the type of user point, the newly created user point can be placed. Selecting the Apply button in the User Point Details Edit button 701 saves the settings and ends user point editing. Note that in Figures 7(a) and 7(b), some of the buttons shown in Figure 6 have been omitted for clarity.

[0047] In Figure 6, the map rotation button 605 is a button that rotates the overhead view of the virtual space (overhead view image). For example, pressing the left or right button of the map rotation button 605 rotates the overhead view of the virtual space (overhead view image) counterclockwise or clockwise around the center coordinates of the virtual space as the central axis. Also, for example, pressing the up or down button of the map rotation button 605 tilts the overhead view upwards or downwards around the center point of the virtual space as the central axis.

[0048] The user point movement button 606 is a button that moves the selected user point horizontally. By pressing the up, down, left, or right buttons of the user point movement button 606, the selected user point moves horizontally according to the button operation. The direction of movement is based on the overhead view position. For example, in the state shown in Figure 6, pressing the right button of the user point movement button 606 moves the selected user point in the positive x-axis direction. If the overhead view position is rotated 180 degrees by pressing the left or right button of the map rotation button 605, pressing the right button of the user point movement button 606 will move the selected user point in the negative x-axis direction.

[0049] The user point orientation change button 607 is a button that changes the orientation of the selected user point. Pressing the left or right button of the user point orientation change button 607 pans the selected user point, and pressing the up or down button tilts the selected user point. The system may also include a setting to partially or completely reverse the relationship between the user point orientation change button 607 and the pan / tilt direction corresponding to the button operation. The up button 608 is a button that moves the selected user point in the positive z-axis direction. The down button 609 is a button that moves the selected user point in the negative z-axis direction.

[0050] The viewpoint preview button 610 is a button that pops up a virtual viewpoint image of the selected user point within the user point placement screen 600. For example, the pop-up display of the virtual viewpoint image can be canceled by pressing the × button in the pop-up window 612 or the viewpoint preview button 610 again. The pop-up window is displayed in a position that does not hide other GUIs or the selected user point, making it easy to fine-tune the position and orientation of the user point. The window size for displaying the virtual viewpoint image in a pop-up may be automatically set to the largest size that satisfies the conditions described above, or it may be set to a window size specified by the user. The exit button 611 is a button that, when selected, exits the user point placement mode in which the position and orientation of the user point in the virtual space are set, and returns to the virtual viewpoint image viewing screen as shown in Figure 5.

[0051] According to this embodiment, avatars, viewpoints, listening points, and sound points related to a user can be independently placed in a virtual space, and information for each user point used to generate virtual viewpoint video and virtual listening point sound data can be individually set. Then, virtual viewpoint video corresponding to the placed viewpoint and virtual listening point sound data corresponding to the placed listening point can be generated, and the virtual viewpoint video and virtual listening point sound data can be played back and output. In this way, it becomes possible for individual users to select their desired virtual viewpoint video and sound environment, and it becomes possible to provide the video and sound intended by the user in the virtual space.

[0052] <Embodiment 2> Embodiment 2 will now be described. In Embodiment 2 described below, the same configuration and operation as in Embodiment 1 described above will be omitted, and the differences from Embodiment 1 will be explained.

[0053] Figure 8 is a block diagram showing an example configuration of the avatar control unit 214 in Embodiment 2. In Figure 8, components having the same function as those shown in Figure 3 are denoted by the same reference numerals, and redundant explanations are omitted. The avatar control unit 214 in Embodiment 2 includes a user point management unit 311, an avatar information management unit 312, a viewpoint information management unit 313, a listening point information management unit 314, a sound point information management unit 315, and an avatar modification unit 801.

[0054] The avatar modification unit 801 modifies the avatar's posture and applies effects based on the positional relationship between the viewpoint and the avatar, if the distance between the viewpoint and the avatar is large. The data on the avatar modified by the avatar modification unit 801 is input to the virtual viewpoint video generation unit 215, and the state of the avatar model is reflected in the virtual space. In addition, user point information from the avatar modification unit 801 and the sound point information management unit 315 is fed back to the storage unit 213 and reflected in the stored user point information, making it possible to see the modified avatar even in the virtual viewpoint video of another user.

[0055] Referring to Figure 9, the operation of the information processing device 100 in Embodiment 2 will be described. Figure 9 is a flowchart illustrating an example of processing by the information processing device 100 in Embodiment 2.

[0056] The processing in steps S901 to S902 in Figure 9 is the same as the processing in steps S401 to S402 in Embodiment 1 shown in Figure 4. In step S903, the avatar modification unit 801 obtains a threshold distance for avatar modification from the user point management unit 311. The avatar modification unit 801 also obtains the three-dimensional coordinates of the avatar and viewpoint from the avatar information management unit 312 and the viewpoint information management unit 313, and obtains the distance between two points (between the avatar and the viewpoint) based on the obtained three-dimensional coordinates.

[0057] In step S904, the avatar modification unit 801 determines whether the distance between the avatar and the viewpoint, acquired in step S903, is greater than or equal to a set threshold. If the avatar modification unit 801 determines that the distance between the avatar and the viewpoint is greater than or equal to the threshold (YES in S904), the process in step S905 is executed. On the other hand, if the avatar modification unit 801 determines that the distance between the avatar and the viewpoint is not greater than or equal to the threshold, that is, that the distance between the avatar and the viewpoint is less than the threshold (NO in S904), the process in step S906 is executed.

[0058] In step S905, the avatar modification unit 801 modifies the avatar's posture, model data, or both. The methods of avatar modification by the avatar modification unit 801 can be broadly classified into three types. The first is to change the avatar's posture. For example, the avatar may assume a posture as if looking at a handheld tablet, and the tablet screen may display the user's virtual viewpoint. This allows other users to objectively confirm where the user is viewing from. The second is to change the avatar's model data. Significantly changing the avatar's model data has the advantage of making the changes easily visible from a distance. The third is to add effects to the avatar's model data. For example, making the avatar model displayed in the virtual space semi-transparent makes it possible to visualize an inactive state where the viewpoint and avatar are far apart. Conversely, making the avatar model glow, or adding heart or star effects, can be used to appeal to performers and other users that they are viewing from the avatar's position. After the processing in step S905 is executed, the processing in step S906 is executed. The processing in steps S906 to S909 in Figure 9 is the same as the processing in steps S403 to S406 in Embodiment 1 shown in Figure 4.

[0059] The aforementioned avatar modifications may not be automatically applied based on the distance between the avatar and the viewpoint, but rather the user may choose to apply them at will. Furthermore, if multiple viewpoints and avatars have been created, the avatar modification unit 801 can either specify which avatars to modify or add combinations of viewpoints and avatars to the avatar modification unit 801. Additionally, the threshold distance used as the basis for avatar modification may be set to an initial value of 0. That is, the avatar modification by the avatar modification unit 801 may be applied even if the viewpoint and avatar are only slightly separated.

[0060] According to Embodiment 2, similar to Embodiment 1, it becomes possible for individual users to select their desired virtual viewpoint video and sound environment, and to provide the video and sound intended by the user in the virtual space. Furthermore, it becomes possible to objectively check the viewing status of each user, and it becomes easier for users to communicate with each other even when each user moves to their preferred viewpoint.

[0061] <Embodiment 3> Embodiment 3 will now be described. In Embodiment 3 described below, the same configuration and operation as in Embodiment 1 described above will be omitted, and the differences from Embodiment 1 will be explained.

[0062] Figure 10 is a block diagram showing an example configuration of the avatar control unit 214 in Embodiment 3. In Figure 10, components having the same function as those shown in Figure 3 are denoted by the same reference numerals, and redundant explanations are omitted. The avatar control unit 214 in Embodiment 3 includes a user point management unit 311, an avatar information management unit 312, a viewpoint information management unit 313, a listening point information management unit 314, a sound production point information management unit 315, and a user point placement correction unit 1001.

[0063] The user point placement correction unit 1001 acquires the three-dimensional coordinates and orientation information of the viewpoint and listening point from the viewpoint information management unit 313 and the listening point information management unit 314, and corrects the three-dimensional coordinates and orientation information of each user point based on the acquired information of the two user points. When the viewpoint and listening point can be placed individually, it becomes possible to arbitrarily select the video and sound environment, but if the viewpoint and listening point are more than a certain distance away from the sound source, lip-sync discrepancies may occur. The user point placement correction unit 1001 is configured to set an allowable value for lip-sync discrepancies (e.g., 0.2 seconds), and the user point placement correction unit 1001 acquires the distance obtained by multiplying the allowable value by the speed of sound (referred to as the allowable distance). Furthermore, the user point placement correction unit 1001 acquires the distance from the listening target (e.g., the sound source with the highest volume) to each of the viewpoint and listening point. If the difference between the distance from the viewing object to the viewpoint and the distance from the viewing object to the listening point exceeds the allowable distance, the user point placement correction unit 1001 adjusts the three-dimensional coordinates of the listening point. The user point placement correction unit 1001 may also refer to the three-dimensional coordinates of an object at the center of the direction the user is facing, rather than the three-dimensional coordinates of the sound source, in order to obtain the distance from the viewing object to the viewpoint and the listening point. Furthermore, if multiple sound sources exist in the virtual space, the user point placement correction unit 1001 may refer to the sound source at the center of the direction the user is facing as the viewing object, rather than the sound source with the highest volume.

[0064] When the user point placement correction unit 1001 adjusts the three-dimensional coordinates of the listening point, it moves the listening point in the direction of the extension of the straight line connecting the viewing object and the current listening point. By moving the listening point so that the distance from the viewing object to the listening point after the move is equal to the distance from the viewing object to the viewpoint, lip-sync discrepancies can be made less noticeable. Although the placement of the listening point is adjusted, it is also possible to adjust the viewpoint placement without changing the placement of the listening point.

[0065] According to Embodiment 3, similar to Embodiment 1, it becomes possible for individual users to select their desired virtual viewpoint video and sound environment, and to provide the video and sound intended by the user in the virtual space. Furthermore, by maintaining a constant distance from the viewing object to the viewpoint and listening point, it becomes possible to realize a viewing environment in which users are less likely to notice lip-sync discrepancies, etc.

[0066] <Embodiment 4> Embodiment 4 will now be described. In Embodiment 4 described below, the same configuration and operation as in Embodiment 1 described above will be omitted, and the differences from Embodiment 1 will be explained.

[0067] Figure 11 is a block diagram showing an example configuration of the avatar control unit 214 in Embodiment 4. In Figure 11, components having the same function as those shown in Figure 3 are denoted by the same reference numerals, and redundant explanations are omitted. The avatar control unit 214 in Embodiment 4 includes a user point management unit 311, an avatar information management unit 312, a viewpoint information management unit 313, a listening point information management unit 314, a sound point information management unit 315, and an other user viewpoint acquisition unit 1101.

[0068] The other user viewpoint acquisition unit 1101 manages the enable / disable information for modeling other users' viewpoints, which is input by the operation unit 222. The other user viewpoint acquisition unit 1101 also acquires the three-dimensional coordinates and orientation information of other users' viewpoints, as well as 3D model data used for displaying the other user's viewpoint model, from the storage unit 213. When modeling of other users is enabled, the other user viewpoint acquisition unit 1101 transmits the various data acquired from the storage unit 213 to the virtual viewpoint video generation unit 215, making it possible to view the other user's viewpoint model in the virtual viewpoint video.

[0069] The operation of the information processing device 100 in Embodiment 4 will be described with reference to Figure 12. Figure 12 is a flowchart illustrating an example of processing by the information processing device 100 in Embodiment 4.

[0070] The processes in steps S1201 to S1203 in Figure 12 are the same as the processes in steps S401 to S403 in Embodiment 1 shown in Figure 4. In step S1204, the other user viewpoint acquisition unit 1101 determines whether or not to display the model from the other user's viewpoint in the virtual viewpoint video. Whether or not to display it in the virtual viewpoint video is determined by the user's selection using the operation unit 222 to enable or disable the display of the model from the other user's viewpoint, and the other user viewpoint acquisition unit 1101 manages this setting. For example, displaying the model from the other user's viewpoint may obscure the original viewing object, so it is disabled in the initial state. If the other user viewpoint acquisition unit 1101 determines that the model from the other user's viewpoint should be displayed in the virtual viewpoint video (other user viewpoint model display is enabled) (YES in S1204), the process in step S1205 is executed. On the other hand, if the other user viewpoint acquisition unit 1101 determines that the model from the other user's viewpoint should not be displayed in the virtual viewpoint video (other user viewpoint model display is disabled) (NO in S1205), the process in step S1206 is executed.

[0071] In step S1205, the other user viewpoint acquisition unit 1101 transmits the three-dimensional coordinates and pose information of the other user's viewpoint, as well as 3D model data to be used for displaying the model of the other user's viewpoint, to the virtual viewpoint video generation unit 215 and instructs it to display the model of the other user's viewpoint. At this time, the number of other user viewpoints displayed in the virtual viewpoint video may be all participants in the virtual space, or only a certain number of participants selected randomly or under specific conditions. The number of other user viewpoints displayed may also be changed as appropriate according to the number of participants in the virtual space. In addition, by focusing on or selecting another user viewpoint, the three-dimensional coordinates and pose information of the other user's viewpoint can be displayed, and the same settings can also be applied to one's own viewpoint. After the processing in step S1205 is executed, the processing in step S1206 is executed. The processing in steps S1206 to S1208 in Figure 12 is the same as the processing in steps S404 to S406 in Embodiment 1 shown in Figure 4.

[0072] According to Embodiment 4, similar to Embodiment 1, it becomes possible for individual users to select their desired virtual viewpoint video and sound environment, and to provide the video and sound intended by the user in the virtual space. Furthermore, by allowing users to check the placement of other users' viewpoints, it becomes possible to assist in the placement of viewpoints and to experience viewing from the same viewpoint as other users.

[0073] <Embodiment 5> Embodiment 5 will now be described. In Embodiment 5 described below, the same configuration and operation as in Embodiment 1 described above will be omitted, and the differences from Embodiment 1 will be explained.

[0074] Figure 13 is a block diagram showing an example of the functional configuration of the information processing device 100 in Embodiment 4. In Figure 13, components having the same function as those shown in Figure 2 are denoted by the same reference numerals, and redundant explanations are omitted. The information processing device 100 includes a 3D data acquisition unit 211, a content sound data acquisition unit 212, a storage unit 213, an avatar control unit 214, a virtual viewpoint image generation unit 215, an overhead view image generation unit 216, a sound generation unit 217, and a video control unit 218. The information processing device 100 also includes a video display unit 219, a sound data output unit 220, a sound data input unit 221, an operation unit 222, and a contact information acquisition unit 1301.

[0075] The contact information acquisition unit 1301 manages contact information for other users that is input by the operation unit 222. The contact information acquisition unit 1301 also exchanges contact information from the user and other users with the storage unit 213. Furthermore, based on the contact information from other users, the contact information acquisition unit 1301 causes the video control unit 218 and the sound data output unit 220 to notify the user that there has been contact from another user.

[0076] The operation of the information processing device 100 in Embodiment 4 will be described with reference to Figure 14. Figure 14 is a flowchart illustrating an example of processing by the information processing device 100 in Embodiment 4. As an example, Figure 14 shows the processing flow from when User A contacts User B until User B is notified.

[0077] In step S1401, user A, who is viewing a virtual viewpoint video, sees the avatar of another user B in the virtual space and makes contact. There are several ways to make contact with another user, as follows: For example, it is possible to make a voice call, select an avatar through the control unit 222, use text chat, or make a specific gesture towards the avatar. Any method that allows communication with a specific user to be initiated is acceptable, even if it is not one of the methods mentioned above. Also, the communication relationship may be one-to-one, one-to-many, or many-to-many.

[0078] In step S1402, the contact information acquisition unit 1301 determines the method by which user A contacted user B. If the contact information acquisition unit 1301 determines that the method of contact from user A to user B is, for example, chat, avatar selection, or gesture, the process in step S1403 is executed. If the contact information acquisition unit 1301 determines that the method of contact from user A to user B is, for example, voice input, the process in step S1407 is executed. Note that multiple methods of voice input may be used. For example, calling out user B's name, targeting an avatar in the center of the field of view and calling out to it, or setting a specific word in advance and contacting an avatar that is in the field of view when that specific word is spoken.

[0079] In step S1403, the contact information acquisition unit 1301 determines whether user B, who has received the contact, has their viewpoint positioned at the same location as the avatar. If the contact information acquisition unit 1301 determines that user B has their viewpoint positioned at the same location as the avatar (YES in S1403), the process in step S1404 is executed. On the other hand, if the contact information acquisition unit 1301 determines that user B does not have their viewpoint positioned at the same location as the avatar (NO in S1403), the process in step S1405 is executed.

[0080] In step S1404, the contact information acquisition unit 1301 decides not to notify user B. At this point, since each user can see the other's avatar, communication by gesture is possible. If voice conversation is to be conducted, with mutual consent, the pronunciation points of both users should be moved to the avatar positions, and then the voice conversation should begin. In this case, the positions of existing pronunciation points may be retained, and new pronunciation points may be created at the avatar positions for voice conversation. If text chat communication is to be conducted, a chat window should be displayed as a pop-up on the user point placement screen 600 or the virtual viewpoint video viewing screen 500 to enable text chat communication. After the processing in step S1404 is executed, the processing in step S1406 is executed.

[0081] In step S1405, the contact information acquisition unit 1301 notifies user B, who has received the contact, via the video control unit 218 and the sound data output unit 220 that another user has contacted them. The notification can be, for example, by displaying text on the user point placement screen 600 or the virtual viewpoint video viewing screen 500, or by playing a specific sound effect. In addition, any method other than those mentioned above may be used, such as vibrating a part of the information processing device 100, as long as it allows the user to notice that they have been contacted. If user B confirms the contact notification and grants permission to communicate, user B's viewpoint position is changed to match the avatar's position, making it possible to see user A's avatar and gestures. At the same time, voice conversation becomes possible by moving user B's listening point, speaking point, or both to the same position as the avatar. In this case, the position of the existing user point may be retained, and a new user point may be created at the avatar's position. In step S1406, communication between users begins using text chat, gestures, etc.

[0082] In step S1407, the contact information acquisition unit 1301 determines whether user B, who has received the contact, has placed their listening point at the same location as the avatar. If the contact information acquisition unit 1301 determines that user B has placed their listening point at the same location as the avatar (YES in S1407), the process in step S1408 is executed. On the other hand, if the contact information acquisition unit 1301 determines that user B has not placed their listening point at the same location as the avatar (NO in S1407), the process in step S1409 is executed.

[0083] In step S1408, the contact information acquisition unit 1301 notifies user B, via the video control unit 218 and the sound data output unit 220, that another user has made contact. Additionally, user B's sound point is moved to the avatar's position. Furthermore, user B's viewpoint may be moved to the avatar's position, or new user points may be created at the avatar's position. After the processing in step S1408, the processing in step S1410 is executed.

[0084] In step S1409, the contact information acquisition unit 1301 notifies user B of contact from another user via the video control unit 218 and the sound data output unit 220. For example, the contact information acquisition unit 1301 notifies the user of contact in the same manner as in step S1405 described above. Alternatively, user B's listening point may not be moved, and only user A's voice input may be acquired from the storage unit 213 and mixed with the virtual listening point sound data. This makes it possible for user B to view at their desired listening point location and to converse with user A simultaneously. In step S1410, communication between users via voice conversation begins.

[0085] According to Embodiment 5, similar to Embodiment 1, it becomes possible for individual users to select their desired virtual viewpoint video and sound environment, and to provide the video and sound intended by the user in the virtual space. Furthermore, even if the viewpoint, listening point, and speaking point are located far from the avatar, it becomes possible to facilitate smooth communication between users through voice conversation, text chat, gestures, etc.

[0086] <Embodiment 6> Embodiment 6 will now be described. In Embodiment 6 described below, the same configuration and operation as in Embodiment 1 described above will be omitted, and the differences from Embodiment 1 will be described.

[0087] Figure 15 is a block diagram showing an example configuration of the avatar control unit 214 in Embodiment 6. In Figure 15, components having the same function as those shown in Figure 3 are denoted by the same reference numerals, and redundant explanations are omitted. The avatar control unit 214 in Embodiment 6 includes a user point management unit 311, an avatar information management unit 312, a viewpoint information management unit 313, a listening point information management unit 314, a sound point information management unit 315, and an attribute information management unit 1501.

[0088] The attribute information management unit 1501 manages user attribute information input by the operation unit 222 or user point control information for each attribute input from the storage unit 213. The attribute information management unit 1501 also outputs the user attribute information and user point control information it manages to the user point management unit 311. Here, user point control information includes all parameters that manage the user point, such as the three-dimensional coordinates of the user point, posture information, avatar model data, and camera parameters. User point control information may be controlled by type of user point, such as viewpoint, listening point, sound point, and avatar, or it may be controlled for each individual parameter of each user point.

[0089] The operation of the information processing device 100 in Embodiment 6 will be described with reference to Figure 16. Figure 16 is a flowchart illustrating an example of processing by the information processing device 100 in Embodiment 6.

[0090] In step S1601, user attribute information is set by the user inputting it via the operation unit 222. The set attribute information is managed by the attribute information management unit 1501. Examples of attribute information include being a supporter of a specific team or a fan of a specific performer. In this embodiment, user attribute information is input via the operation unit 222, but attribute information may also be set from the storage unit 213 by analyzing past content viewing history or external application usage history using machine learning or the like. Furthermore, the content distributor may directly specify a user and assign attribute information to them.

[0091] The processes in steps S1602 to S1603 in Figure 16 are the same as the processes in steps S401 to S402 in Embodiment 1 shown in Figure 4. In step S1604, the user point management unit 311 determines whether the attribute information of a configured user managed by the attribute information management unit 1501 matches the attribute information set by the content distributor. If the user point management unit 311 determines that the attribute information of a configured user matches the attribute information set by the content distributor (YES in S1604), the process in step S1605 is executed. On the other hand, if the user point management unit 311 determines that the attribute information of a configured user does not match the attribute information set by the content distributor (NO in S1604), the process in step S1606 is executed.

[0092] In step S1605, the user point management unit 311 performs processing such as forcibly changing, restricting, or recommending changes to user point information according to the user's attribute information. An example of processing targeting the three-dimensional coordinate information of an avatar will be explained using a baseball game in a virtual space as an example. Forcibly changing user point information makes it possible to specify a fixed seat in the virtual space for each user, preventing them from sitting in other seats. Restricting user point information makes it possible to gather users with the same attributes by restricting the area to only the right field stands of the baseball stadium, for example. Recommending changes to user point information makes it possible to highlight areas where a particular player's face is easily visible on the virtual space of the user point placement screen 600. The highlighting is performed by the overhead view image generation unit 216, which obtains user point information and attribute information from the avatar information management unit 312 and the viewpoint information management unit 313. If necessary, the listening point information management unit 314 and the sound point information management unit 315 may use user point information managed by them to highlight recommended placement locations. Furthermore, by changing the information being modified, there are other potential uses, such as dressing avatars of fans of a specific sports team in their uniforms, or allowing only core supporters to listen to player interviews. After the processing in step S1605 is executed, the processing in step S1606 is executed. The processing in steps S1206 to S1209 in Figure 16 is the same as the processing in steps S403 to S406 in Embodiment 1 shown in Figure 4.

[0093] According to Embodiment 6, similar to Embodiment 1, it becomes possible for individual users to select their desired virtual viewpoint video and sound environment, and to provide the video and sound intended by the user in the virtual space. Furthermore, according to this embodiment, by processing such as changing user point information according to the user's attribute information, it becomes possible to provide additional content tailored to the user and improve the content experience.

[0094] <Embodiment 7> Embodiment 7 will now be described. In Embodiment 7 described below, the same configuration and operation as in Embodiment 1 described above will be omitted, and the differences from Embodiment 1 will be explained.

[0095] Figure 17 is a block diagram showing an example configuration of the avatar control unit 214 in Embodiment 7. In Figure 17, components having the same function as those shown in Figure 3 are denoted by the same reference numerals, and redundant explanations are omitted. The avatar control unit 214 in Embodiment 7 includes a user point management unit 311, an avatar information management unit 312, a viewpoint information management unit 313, a listening point information management unit 314, a sound point information management unit 315, and a content information management unit 1701.

[0096] The content information management unit 1701 manages the content information input from the storage unit 213 and outputs user point control information to the user point management unit 311 according to the content status. The user point control information may be controlled separately for each type of user point, such as viewpoint, listening point, sound point, and avatar, as in Embodiment 5, or it may be controlled for each individual parameter of each user point.

[0097] The operation of the information processing device 100 in Embodiment 7 will be described with reference to Figure 18. Figure 18 is a flowchart illustrating an example of processing by the information processing device 100 in Embodiment 7.

[0098] In step S1801, the content information management unit 1701 obtains content information related to the content the user is viewing from the storage unit 213. Content information includes, for example, the length of the content, information about the teams playing in the match, information about the music being played, information about the players and performers, and the score of the match. The processing in steps S1802 to S1803 in Figure 18 is the same as the processing in steps S401 to S402 in Embodiment 1 shown in Figure 4.

[0099] In step S1804, the content information management unit 1701 determines whether a specific event has occurred within the content. If the content information management unit 1701 determines that a specific event has occurred within the content (YES in S1804), the process in step S1805 is executed. On the other hand, if the content information management unit 1701 determines that a specific event has not occurred within the content (NO in S1804), the process in step S1806 is executed. For example, if the content is a live stream, the process in step S1805 is executed when the content information management unit 1701 determines that it matches the content information that was set in advance. Also, for example, if the content is already recorded, the process in step S1805 is executed when the content information management unit 1701 determines that the specific time at which the event occurred has been played.

[0100] In step S1805, the user point management unit 311 performs processing such as forcibly changing, restricting, or recommending changes to user point information based on user point control information output from the content information management unit 1701 according to the content information. For example, it can be used to move the viewpoint to one that allows an overview of the winning team's supporters at the end of a match, or to change the color of the avatar model in accordance with the music being played. After the processing in step S1805 is executed, the processing in step S1806 is executed. The processing in steps S1806 to S1809 in Figure 18 is the same as the processing in steps S403 to S406 in Embodiment 1 shown in Figure 4.

[0101] According to Embodiment 7, similar to Embodiment 1, it becomes possible for individual users to select their desired virtual viewpoint video and sound environment, and to provide the video and sound intended by the user in the virtual space. Furthermore, according to this embodiment, by performing processing such as changing user point information according to content information, it becomes possible to improve the user experience to match the content.

[0102] Furthermore, the embodiments described above are not limited to being implemented independently; they may also be combined in appropriate ways.

[0103] (Other embodiments of the present invention) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0104] It should be noted that the embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0105] The disclosure of this embodiment includes the following configurations and methods, etc. (Composition 1) A means for arranging a user's avatar, viewpoint, listening point, and pronunciation point within a virtual space, A first video generation means that generates a virtual viewpoint image corresponding to the positioned viewpoint based on 3D data of a virtual space, Sound generation means that generates virtual listening point sound data corresponding to the arranged listening points based on acquired sound data, The system includes a playback means for playing back the generated virtual viewpoint video and the virtual listening point sound data, The arrangement means is characterized in that at least one of the viewpoint, listening point, and sound-producing point can be arranged independently of the avatar. (Configuration 2) The first image generation means generates the virtual viewpoint image based on at least the three-dimensional coordinates of the positioned viewpoint, The information processing device according to configuration 1, characterized in that the sound generation means generates virtual listening point sound data based on at least three-dimensional coordinates of the arranged listening point and the sound generation point. (Composition 3) The system includes a second video generation means that generates an overhead view video of the virtual space based on the 3D data of the virtual space, The information processing apparatus according to configuration 1 or 2, characterized in that the playback means plays back the generated virtual viewpoint image or the overhead viewpoint image. (Composition 4) The information processing device according to configuration 3, characterized in that at least one of the models representing the avatar, the viewpoint, the listening point, and the sound-producing point is superimposed on the overhead view image. (Composition 5) The information processing device according to configuration 3 or 4, characterized in that it displays an operation element for operating the arrangement of the avatar, the viewpoint, the listening point, and the sound-producing point by superimposing it on the virtual viewpoint image and the overhead viewpoint image. (Composition 6) The information processing device according to any one of configurations 1 to 5, characterized in that the arrangement means sets three-dimensional coordinates and posture information of the avatar, the viewpoint, the listening point, and the sound-producing point in a virtual space. (Composition 7) The information processing apparatus according to any one of configurations 1 to 5, characterized in that the arrangement means has a modification means for modifying at least one of the avatar's posture and model data based on the positional relationship between the viewpoint and the avatar in the virtual space. (Composition 8) The information processing apparatus according to any one of configurations 1 to 7, characterized in that the arrangement means modifies the arrangement of at least one of the viewpoint and the listening point based on the positional relationship between the viewpoint and the listening point in the virtual space. (Composition 9) The information processing apparatus according to any one of configurations 1 to 8, characterized in that the first video generation means generates the virtual viewpoint video which includes a model showing the viewpoint of another user. (Composition 10) The information processing device according to any one of configurations 1 to 9, characterized in that the playback means notifies the user that contact has been made in response to contact from another user. (Composition 11) The information processing device according to any one of configurations 1 to 10, characterized in that the arrangement means manages user attribute information and performs processing to forcibly change, restrict, or recommend changing the arrangement of at least one of the avatar, viewpoint, listening point, and sound point according to the attribute information. (Composition 12) The information processing device according to any one of configurations 1 to 11, characterized in that the arrangement means manages content information and performs processing to forcibly change, restrict, or recommend changing the arrangement of at least one of the avatar, viewpoint, listening point, and sound point in response to changes in the content information. (Method 1) A placement process involves arranging the user's avatar, viewpoint, listening point, and pronunciation point within a virtual space. A first video generation step generates a virtual viewpoint image corresponding to the positioned viewpoint based on 3D data of a virtual space, A sound generation process that generates virtual listening point sound data corresponding to the placed listening points based on acquired sound data, The process includes a playback step for playing back the generated virtual viewpoint video and the virtual listening point sound data, A control method for an information processing device, characterized in that, in the arrangement step, at least one of the viewpoint, the listening point, and the sound-producing point can be arranged independently of the avatar. (Program 1) A placement step involves arranging the user's avatar, viewpoint, listening point, and pronunciation point within a virtual space, A first video generation step of generating a virtual viewpoint image corresponding to the positioned viewpoint based on 3D data of a virtual space, A sound generation step that generates virtual listening point sound data corresponding to the placed listening points based on acquired sound data, The computer is made to perform a playback step that plays back the generated virtual viewpoint video and the virtual listening point sound data. The program, in the placement step, allows at least one of the viewpoint, listening point, and sound-producing point to be placed independently of the avatar. [Explanation of symbols]

[0106] 100: Information processing unit 211: 3D data acquisition unit 212: Content sound data acquisition unit 213: Storage unit 214: Avatar control unit 215: Virtual viewpoint image generation unit 216: Overhead view image generation unit 217: Sound generation unit 218: Image control unit 218: Image display unit 220: Sound data output unit 221: Sound data input unit 222: Operation unit

Claims

1. A means for arranging a user's avatar, viewpoint, listening point, and pronunciation point within a virtual space, A first video generation means generates a virtual viewpoint image corresponding to the positioned viewpoint based on 3D data of a virtual space, Sound generation means that generates virtual listening point sound data corresponding to the arranged listening points based on acquired sound data, The system includes a playback means for playing back the generated virtual viewpoint video and the virtual listening point sound data, The arrangement means is characterized in that at least one of the viewpoint, listening point, and sound-producing point can be arranged independently of the avatar.

2. The first image generation means generates the virtual viewpoint image based on at least the three-dimensional coordinates of the positioned viewpoint, The information processing apparatus according to claim 1, characterized in that the sound generation means generates virtual listening point sound data based on at least three-dimensional coordinates of the arranged listening point and the sound generation point.

3. The system has a second image generation means that generates an overhead view image of the virtual space based on the 3D data of the virtual space, The information processing apparatus according to claim 1, wherein the playback means plays back the generated virtual viewpoint image or the overhead viewpoint image.

4. The information processing device according to claim 3, characterized in that at least one of the models representing the avatar, the viewpoint, the listening point, and the sound-producing point is superimposed and displayed on the overhead view image.

5. The information processing apparatus according to claim 3, characterized in that it displays an operation element for operating the arrangement of the avatar, the viewpoint, the listening point, and the sound-producing point by superimposing it on the virtual viewpoint image and the overhead viewpoint image.

6. The information processing apparatus according to claim 1, characterized in that the arrangement means sets three-dimensional coordinates and posture information of the avatar, the viewpoint, the listening point, and the sound-producing point in a virtual space.

7. The information processing apparatus according to claim 1, characterized in that the arrangement means has a modification means for modifying at least one of the avatar's posture and model data based on the positional relationship between the viewpoint and the avatar in the virtual space.

8. The information processing apparatus according to claim 1, characterized in that the arrangement means modifies the arrangement of at least one of the viewpoint and the listening point based on the positional relationship between the viewpoint and the listening point in the virtual space.

9. The information processing apparatus according to claim 1, characterized in that the first video generation means generates the virtual viewpoint video which includes a model showing the viewpoint of another user.

10. The information processing device according to claim 1, characterized in that the playback means notifies the user that contact has been made in response to contact from another user.

11. The information processing apparatus according to claim 1, characterized in that the arrangement means manages user attribute information and performs processing to forcibly change, restrict, or recommend changing the arrangement of at least one of the avatar, viewpoint, listening point, and sound point according to the attribute information.

12. The information processing apparatus according to claim 1, characterized in that the arrangement means manages content information and performs processing to forcibly change, restrict, or recommend changing the arrangement of at least one of the avatar, viewpoint, listening point, and sound point in response to changes in the content information.

13. A placement process involves arranging the user's avatar, viewpoint, listening point, and pronunciation point within a virtual space. A first video generation step generates a virtual viewpoint image corresponding to the positioned viewpoint based on 3D data of a virtual space, A sound generation process that generates virtual listening point sound data corresponding to the placed listening points based on acquired sound data, The process includes a playback step for playing back the generated virtual viewpoint video and the virtual listening point sound data, A control method for an information processing device, characterized in that, in the arrangement step, at least one of the viewpoint, the listening point, and the sound-producing point can be arranged independently of the avatar.

14. A placement step involves arranging the user's avatar, viewpoint, listening point, and pronunciation point within a virtual space, A first video generation step of generating a virtual viewpoint image corresponding to the positioned viewpoint based on 3D data of a virtual space, A sound generation step that generates virtual listening point sound data corresponding to the placed listening points based on acquired sound data, The computer is made to perform a playback step that plays back the generated virtual viewpoint video and the virtual listening point sound data. The program allows at least one of the viewpoint, listening point, and sound-producing point to be positioned independently of the avatar in the aforementioned positioning step.

Images