Information processing device and method

Abstract

The present technology relates to an information processing device and method that make it possible to obtain a video with less motion blur. This information processing device comprises: a parameter generation unit that generates a rendering parameter for generating first video data by rendering processing, the rendering parameter including video generation cycle information indicating the frame rate of the first video data and exposure time information indicating the exposure time of the first video data; and a transmission unit that transmits an execution request for the rendering processing to a rendering server, the execution request including the rendering parameter. The present technology can be applied to an information processing system.

Description

Information processing device and method

[0001] The present technology relates to an information processing device and method, and more particularly to an information processing device and method that enable an image with little motion blur to be obtained.

[0002] 2. Description of the Related Art Conventionally, techniques have been proposed for providing content such as video in a three-dimensional (3D) virtual space (hereinafter also referred to as 3D content).

[0003] In a service that provides 3D content, for example, a rendering process called 3D rendering is performed on a server that distributes the 3D content, and the resulting video data is transmitted to a client via a network.

[0004] Furthermore, as a technology related to 3D content, a technology has been proposed in which 3D rendering of an image to be presented to one user is performed by multiple servers, thereby reducing the processing load on the client side (see, for example, Patent Document 1).

[0005] Japanese Patent Application Laid-Open No. 2020-21394

[0006] Incidentally, it is conceivable that the different frames of a video to be presented to one user could be divided among multiple different servers and generated using a 3D rendering process, and the video (frames) obtained by each server could then be multiplexed, i.e., the frames could be arranged in order, to generate the final multiple frames of video to be presented to the user.

[0007] In such a case, it is possible to lower the frame rate of the video generated by each server, while still obtaining video presented to the user with a frame rate higher than that of the video generated by each server.

[0008] However, in this case, as the frame rate of the video generated by one server decreases, a single frame of video contains information for a longer period of time. Furthermore, the video obtained by multiplexing the frames generated by each server will have overlapping information on the time axis. In other words, adjacent frames will contain information from the same time. Therefore, while the video presented to the user may appear to have a high frame rate, motion blur will occur within the video.

[0009] The present technology has been made in view of such circumstances, and makes it possible to obtain an image with little motion blur.

[0010] An information processing device according to a first aspect of the present technology includes: a parameter generation unit that generates rendering parameters for generating the first video data by a rendering process, the rendering parameters including video generation period information indicating a frame rate of first video data and exposure time information indicating an exposure time of the first video data; and a transmission unit that transmits a request to execute the rendering process, the request including the rendering parameters, to a rendering server.

[0011] An information processing method according to a first aspect of the present technology includes an information processing device generating rendering parameters for generating the first video data by a rendering process, the rendering parameters including video generation period information indicating a frame rate of first video data and exposure time information indicating an exposure time of the first video data, and transmitting a request to a rendering server to execute the rendering process, the request including the rendering parameters.

[0012] In a first aspect of the present technology, rendering parameters include image generation period information indicating a frame rate of first image data and exposure time information indicating an exposure time of the first image data, and the rendering parameters for generating the first image data are generated by a rendering process, and a request to execute the rendering process including the rendering parameters is transmitted to a rendering server.

[0013] An information processing device according to a second aspect of the present technology includes an image generation unit that generates the first image data by rendering processing based on rendering parameters including image generation period information indicating a frame rate of the first image data and exposure time information indicating an exposure time of the first image data.

[0014] An information processing method according to a second aspect of the present technology includes an information processing device generating the first video data by rendering processing based on rendering parameters including video generation period information indicating a frame rate of the first video data and exposure time information indicating an exposure time of the first video data.

[0015] In a second aspect of the present technology, the first video data is generated by a rendering process based on rendering parameters including video generation period information indicating a frame rate of the first video data and exposure time information indicating an exposure time of the first video data.

[0016] FIG. 1 is a diagram illustrating an example configuration of an information processing system. FIG. 2 is a diagram illustrating distribution video. FIG. 3 is a diagram illustrating 3D rendering processing. FIG. 4 is a diagram illustrating an example configuration of a rendering server. FIG. 5 is a diagram illustrating an example configuration of a distribution server. FIG. 6 is a diagram illustrating an example configuration of a client. FIG. 7 is a diagram illustrating generation of distribution video. FIG. 8 is a diagram illustrating generation of distribution video using two rendering servers. FIG. 9 is a diagram illustrating the present technology. FIG. 10 is a flowchart illustrating distribution processing. FIG. 11 is a flowchart illustrating data generation processing. FIG. 12 is a flowchart illustrating content playback processing. FIG. 13 is a diagram illustrating a modified example of the present technology. FIG. 14 is a diagram illustrating a modified example of the present technology. FIG. 15 is a diagram illustrating an example configuration of a computer.

[0017] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0018] First Embodiment Configuration Example of Information Processing System The present technology relates to a content distribution system that distributes, as 3D content, images and the like in a three-dimensional virtual space such as a metaverse in which one or more users can participate simultaneously.

[0019] The 3D content may consist of only video, or may consist of video and accompanying audio. For example, free viewpoint content may be distributed as 3D content, which allows a user, or more specifically, a user's avatar, to freely move within a three-dimensional virtual space and freely change their line of sight (face direction).

[0020] FIG. 1 is a diagram showing an example of the configuration of an embodiment of an information processing system to which the present technology is applied.

[0021] The information processing system 11 shown in FIG. 1 is a content distribution system that generates and distributes 3D content in real time.

[0022] The information processing system 11 has a plurality of rendering servers including rendering servers 21-1 to 21-N, a plurality of distribution servers including distribution servers 22-1 and 22-2, and a plurality of clients including clients 23-1 to 23-M.

[0023] In the following, when there is no need to particularly distinguish between the rendering servers such as rendering server 21-1 to rendering server 21-N, they will be simply referred to as rendering server 21, and when there is no need to particularly distinguish between the distribution servers such as distribution server 22-1 and distribution server 22-2, they will be simply referred to as distribution server 22. Furthermore, when there is no need to particularly distinguish between the clients such as client 23-1 to client 23-M, they will be simply referred to as client 23.

[0024] In response to a request from the distribution server 22, the rendering server 21 generates an image for a specified time (frame) with a specified position in a three-dimensional virtual space as the viewpoint position through a 3D rendering process (hereinafter simply referred to as the rendering process), and supplies the resulting generated image data to the distribution server 22.

[0025] In this example, a plurality of rendering servers 21 including rendering servers 21-1 to 21-N are connected to one distribution server 22. When the 3D content is made up of video and audio, the rendering server 21 also performs audio rendering processing, but in the following, explanation of the audio rendering processing, etc. will be omitted as appropriate.

[0026] The distribution server 22 instructs each rendering server 21 to generate generated video data, and also aggregates the generated video data supplied from the rendering servers 21 to generate distribution video data for presenting 3D content to the user.

[0027] The distributed video data is obtained, for example, by multiplexing a plurality of generated video data (generated videos), that is, by arranging frames (videos) at each time in order.

[0028] The distribution video data is generated for each client 23 (user) connected to the distribution server 22. In particular, the distribution video data generated for each client 23 is video data of an image whose viewpoint is the position of the user operating the client 23 in the three-dimensional virtual space.

[0029] The distribution server 22 supplies the generated distribution video data to each client 23. In this example, for example, clients 23-1 to 23-5 are connected to the distribution server 22-1, and the distribution server 22-1 transmits (distributes) the distribution video data to these clients 23.

[0030] Although an example in which a plurality of distribution servers 22 are provided in the information processing system 11 will be described here, the information processing system 11 may be provided with a single distribution server 22 .

[0031] The client 23 is an information processing device operated by a user who wants to view 3D content. For example, the client 23 is made up of various terminals such as a smartphone, a personal computer, an HMD (Head Mounted Display), or a game console.

[0032] The user can freely change the position and orientation of the user, more specifically, the avatar representing the user, in the three-dimensional virtual space by inputting operations to the client 23. The client 23 generates client attribute information according to the user's input operations and transmits it to the distribution server 22.

[0033] For example, the client attribute information includes device information about the client 23, such as the video frame rate that the client 23 can support when playing 3D content, and 6DoF (Degrees of Freedom) information, which is position and direction information that indicates the position and orientation of the user (avatar) within the three-dimensional virtual space.

[0034] The distribution server 22 determines which rendering server 21 should generate which frame of the distribution video data based on the client attribute information, etc. Also, by multiplexing the generated video data, distribution video data with a frame rate according to the client attribute information is generated.

[0035] Multiplexing here refers to a process in which, for example, a video (hereinafter also referred to as generated video) at a specific time (playback time) based on generated video data is treated as one frame, and multiple frames (generated videos) are arranged in a specific order to generate multiple frames of distribution video data.

[0036] Note that interpolation processing may be performed when generating the video data to be distributed. In the interpolation processing, for example, based on generated video data from different viewpoints or times, a video is generated in the user's line of sight (field of view) with the user's (avatar's) position in the three-dimensional virtual space as the viewpoint.

[0037] By multiplexing or the like by the distribution server 22, distribution video data for playing video (hereinafter also referred to as distribution video) with the direction of the user's face as the line of sight as seen from the user's (avatar's) position in the three-dimensional virtual space is obtained. Furthermore, the frame rate of the distribution video (distribution video data) may differ for each client 23, for example.

[0038] The generation of the video to be distributed will be further described with reference to FIGS.

[0039] For example, as shown in Figure 2, when user U11 is wearing an HMD as client 23, 6DoF information of user U11 is generated by input from user U11's movements, such as movement of user U11 or changes in facial direction, or by operation input from user U11.

[0040] Then, client attribute information including the generated 6DoF information is transmitted from the client 23 to the distribution server 22. Note that although only one user U11 is illustrated here, in reality, multiple users (clients 23) are connected to the distribution server 22, and these multiple users participate in events, etc., that take place in a three-dimensional virtual space.

[0041] The distribution server 22 instructs each of the rendering servers 21 to generate generated video data in accordance with the client attribute information received from each client 23. In this case, the distribution server 22 instructs the generation of generated video data with a predetermined position in the three-dimensional virtual space as the viewpoint position at a predetermined time (playback time) of the 3D content.

[0042] In response to an instruction from the distribution server 22, each rendering server 21 generates generated video data at a different viewpoint position and time.

[0043] The distribution server 22 generates distribution video data for each client 23 based on the generated video data generated by each rendering server 21 and the client attribute information received from the client 23, and transmits (distributes) the data to the client 23. As a result, the client 23 displays a two-dimensional video (2D image) corresponding to the position and orientation of the user U11 in the three-dimensional virtual space as the distribution video P11.

[0044] In the rendering server 21, as shown in FIG. 3, for example, prepared scene data is used to generate generated video data through 3D rendering processing.

[0045] The scene data includes three-dimensional object data, which is information about the video and audio of one or more objects placed in a three-dimensional virtual space, and scene description information, which indicates the placement position and orientation of the objects at each time.

[0046] For example, the three-dimensional object data includes video object data and audio object data for each object.

[0047] Video object data is data for displaying a video (image) of an object, and is made up of, for example, model data and texture data that indicate the three-dimensional shape of the object, etc. Audio object data is audio data for playing the sound of the object.

[0048] The scene description information is information that indicates which object is located at which position in the three-dimensional virtual space and in which direction it faces at each time (playback time) of the 3D content. In other words, the scene description information is information that describes what kind of scene each scene in the 3D content is.

[0049] The rendering server 21 generates generated video data by 3D rendering processing using such scene data.

[0050] In the 3D rendering process, as shown in the center of the figure, each object is arranged in a three-dimensional virtual space according to the scene description information, and a two-dimensional (2D) image of the three-dimensional virtual space as seen from a desired viewpoint position SP11 is generated. The viewpoint position SP11 may be the position of a predetermined user, or may be a position different from the user's position, i.e., an arbitrary position where no user is present. More specifically, the 3D rendering process also generates audio data for reproducing sounds to be heard at the desired viewpoint position SP11.

[0051] The generated image generated by the 3D rendering process may be a panoramic image (omnidirectional image) that is an image (image) in all directions (up, down, left, and right) as seen from viewpoint position SP11, i.e., a 360-degree image, or may be a 2D cropped image obtained by cropping a portion of the panoramic image. For example, when a cropped image is generated as the generated image, a region of the field of view when facing a predetermined direction is cropped from the panoramic image and used as the generated image.

[0052] Note that, depending on the viewpoint position, for example, when the viewpoint position is near the edge of the three-dimensional virtual space, it is not always necessary to generate an omnidirectional image, so generation of some areas of the omnidirectional image may be omitted depending on the viewpoint position, etc. In other words, it is possible to generate an omnidirectional image in which the image of the three-dimensional virtual space is not displayed in some areas, such as an image that is essentially 280 degrees.

[0053] The distribution server 22 generates a distribution video, which is a 2D video, by multiplexing the multiple generated videos generated by the rendering server 21. As a result, the client 23 displays a distribution video P21 with a field of view determined by the user's position and orientation in the three-dimensional virtual space.

[0054] In addition, the 3D rendering process may use not only scene data but also client attribute information and video data captured in real time.

[0055] For example, if the 3D content is live footage, it is conceivable to record a live performance of a performer in real space and use the resulting video and audio data for 3D rendering processing. In this way, photorealistic performers can be placed in a three-dimensional virtual space, and a live performance in a three-dimensional virtual space such as a metaverse space can be distributed as 3D content.

[0056] Furthermore, when client attribute information is used in the 3D rendering process, the client attribute information may include video object data and audio object data of the user (avatar).

[0057] <Configuration Example of Rendering Server> FIG. 4 is a diagram showing a configuration example of the rendering server 21. As shown in FIG.

[0058] The rendering server 21 includes a control unit 71 , a receiving unit 72 , a scene data storage unit 73 , a video generating unit 74 , and a transmitting unit 75 .

[0059] The control unit 71 controls the overall operation of the rendering server 21. The receiving unit 72 communicates with the distribution server 22 via the network.

[0060] For example, the receiving unit 72 receives scene data transmitted from the distribution server 22 and supplies it to the scene data storage unit 73, or receives rendering parameters transmitted from the distribution server 22 and supplies it to the image generating unit 74. More specifically, the distribution server 22 transmits to the rendering server 21 an execution request including the rendering parameters to generate a generated image, that is, to request the execution of a 3D rendering process.

[0061] Scene data is data for constructing a three-dimensional virtual space including objects such as avatars that are the subjects of the 3D content, and includes the above-mentioned three-dimensional object data and scene description information.

[0062] The rendering parameters are various parameters (information) used in the 3D rendering process. In other words, the rendering parameters are parameters for generating generated image data by the 3D rendering process. For example, the rendering parameters include time information, viewpoint information, image generation cycle information, and exposure time information.

[0063] The time information is information indicating the time (playback time) of the 3D content at which a generated video is to be generated, that is, information indicating the time at which a generated video is to be generated.

[0064] The viewpoint information is information indicating the viewpoint position of the 3D content from which the generated image is to be generated, i.e., information indicating the position (viewpoint position) in the three-dimensional virtual space that is the target for generating the generated image. Note that, when a clipped image obtained by clipping a portion of the panoramic image is generated as the generated image, the viewpoint information may also include information indicating the region from which the clipping will occur (viewing area), i.e., the line of sight direction.

[0065] In the following description, it is assumed that the viewpoint information includes position information indicating the position of the user (avatar) in the three-dimensional virtual space, i.e., the user's viewpoint position, and line-of-sight information indicating the user's line-of-sight direction.

[0066] The video generation cycle information indicates the rendering cycle, which is the cycle for generating the generated video (video generation cycle), i.e., the frame rate of the generated video. The exposure time information indicates the exposure time (shutter speed) of the generated video.

[0067] In the information processing system 11, even if the rendering cycle is lengthened, for example, the occurrence of motion blur in the generated image can be prevented by appropriately setting the exposure time (shutter speed) of the generated image separately from the rendering cycle, i.e., the frame rate of the generated image. The exposure time here refers to the length of the period of scene data used when generating one frame of generated image through 3D rendering processing. In other words, one frame of generated image is generated using data for the period set as the exposure time out of the data for the entire period of 3D content that constitutes the scene data.

[0068] The scene data storage unit 73 stores (holds) the scene data supplied from the receiving unit 72 and supplies the held scene data to the video generating unit 74 as appropriate.

[0069] The video generation unit 74 generates generated video data (generated video) by performing 3D rendering processing based on the rendering parameters supplied from the receiving unit 72 and the scene data supplied from the scene data storage unit 73, and supplies the generated video data to the transmitting unit 75.

[0070] The transmitting unit 75 transmits the generated video data supplied from the video generating unit 74 to the distribution server 22 via the network.

[0071] <Configuration Example of Distribution Server> FIG. 5 is a diagram showing a configuration example of the distribution server 22. As shown in FIG.

[0072] The distribution server 22 includes a control unit 101 , a receiving unit 102 , a rendering parameter generating unit 103 , a transmitting unit 104 , a scene data storage unit 105 , a receiving unit 106 , a video storage unit 107 , a video multiplexing unit 108 , and a transmitting unit 109 .

[0073] The control unit 101 controls the overall operation of the distribution server 22. The receiving unit 102 receives client attribute information transmitted from each client 23 via the network and supplies it to the rendering parameter generating unit 103.

[0074] The rendering parameter generation unit 103 determines the position and time in the three-dimensional virtual space where a generated image is generated by 3D rendering processing, the frame rate of the generated image (image generation cycle), and the exposure time of the generated image.

[0075] That is, the rendering parameter generation unit 103 determines the position and time at which the 3D rendering process is to be performed based on the client attribute information of each client 23 (user) supplied from the receiving unit 102. The rendering parameter generation unit 103 also determines which rendering server 21 is to generate a generated image at which position and time, i.e., which generated image is to be generated by which rendering server 21, as well as the frame rate and exposure time of the generated image.

[0076] The rendering parameter generation unit 103 generates rendering parameters for each rendering server 21 based on the determined position, time, etc. for performing the 3D rendering process and the client attribute information, and supplies the generated rendering parameters to the transmission unit 104. More specifically, an execution request including the rendering parameters is generated and supplied to the transmission unit 104.

[0077] The transmitting unit 104 transmits the rendering parameters (execution request) supplied from the rendering parameter generating unit 103 and the scene data supplied from the scene data storage unit 105 to the rendering server 21 via the network.

[0078] The scene data storage unit 105 stores (holds) scene data of 3D content in advance, and supplies the held scene data to the transmission unit 104 as appropriate.

[0079] The receiving unit 106 receives the generated video data transmitted from each rendering server 21 and supplies it to the video storage unit 107. The video storage unit 107 stores the generated video data supplied from the receiving unit 106 and supplies it to the video multiplexing unit 108 as appropriate.

[0080] The video multiplexing unit 108 functions as a video generation unit that generates distribution video data having a frame rate different from that of the generated video data based on generated video data that displays generated video at different times received from each of multiple rendering servers 21.

[0081] That is, the video multiplexing unit 108 refers to the rendering parameters generated by the rendering parameter generation unit 103 and the client attribute information of each client 23 as necessary, and performs multiplexing, etc. based on the generated video data stored in the video storage unit 107, thereby generating distribution video data for each client 23 and supplying it to the transmission unit 109.

[0082] The transmitting unit 109 transmits the distribution video data for each client 23 supplied from the video multiplexing unit 108 to each client 23 via the network.

[0083] <Configuration Example of Client> FIG. 6 is a diagram showing a configuration example of the client 23. As shown in FIG.

[0084] The client 23 is connected to an input unit 141 that is operated by the user and a display 142 that is a display unit that displays the distributed video.

[0085] For example, the input unit 141 may be composed of a mouse, a keyboard, a controller, a touch panel superimposed on the display 142, or the like, and supplies signals according to user operations to the client 23. Alternatively, the input unit 141 may be composed of various sensors that detect user movements, and may output the detection results of the user movements to the client 23 as user input.

[0086] The input unit 141 and the display 142 may be provided in the client 23 .

[0087] The client 23 includes a control unit 151 , a user input acquisition unit 152 , a client attribute information generation unit 153 , a transmission unit 154 , a reception unit 155 , and a video display control unit 156 .

[0088] The control unit 151 controls the overall operation of the client 23. The user input acquisition unit 152 supplies a signal corresponding to the user input supplied from the input unit 141 to the client attribute information generation unit 153. For example, the user input acquisition unit 152 acquires information (signals) relating to the movement of the user (avatar) and supplies the information (signals) to the client attribute information generation unit 153.

[0089] The client attribute information generation unit 153 identifies the overall movement of the user (avatar) and the movement of each part of the user based on the signal supplied from the user input acquisition unit 152, and generates client attribute information based on the identification results and information previously stored in the client 23. The client attribute information generation unit 153 supplies the generated client attribute information to the transmission unit 154.

[0090] For example, the client attribute information includes 6DoF information obtained from the results of identifying the user's movements, and device information previously stored in the client 23. In addition, the client attribute information may include information indicating the speed and direction of movement of the user (avatar), and the speed and direction of movement of each part of the user.

[0091] The transmitting unit 154 transmits the client attribute information supplied from the client attribute information generating unit 153 to the distribution server 22 via the network. The receiving unit 155 receives the distribution video data transmitted from the distribution server 22 and supplies it to the video display control unit 156.

[0092] The video display control unit 156 supplies the distribution video data supplied from the receiving unit 155 to the display 142, and causes the display 142 to display the distribution video.

[0093] <About the Present Technology> The present technology, that is, the information processing system 11, has the following features.

[0094] In other words, this technology has the feature that when 3D rendering processing is performed in parallel on multiple rendering servers 21 to obtain distribution video for one user, the exposure time of the generated video generated by the 3D rendering processing is specified separately (independently) from the frame rate of the generated video, i.e., the video generation period (rendering period).

[0095] In the distribution server 22, the generated images obtained by each of the multiple rendering servers 21 are aggregated and multiplexed to generate distribution video data, depending on the characteristics of the client 23, which is the destination (transmission destination) of the distribution video, particularly the frame rate of the video that the client 23 can handle.

[0096] In this way, by performing processing in parallel on multiple rendering servers 21, it becomes possible to obtain distributed video with a high frame rate that would be difficult to generate with a single rendering server 21. Moreover, by specifying the exposure time separately from the frame rate of the generated video, it is possible to provide distributed video with little motion blur to the client 23.

[0097] In particular, with this technology, the exposure time (shutter speed) of the generated images is determined so that there is no overlap of information contained in adjacent frames of the distributed images, depending on the number of rendering servers 21 (number of distributed processes), the processing power and current processing load of each rendering server 21, the characteristics of the clients 23, etc. Also, for example, it is possible to provide, from the same set of generated images, high-frame-rate distributed images with little motion blur to each of multiple clients 23 with different display frame rates. For example, if common generated images for the same 3D content can be used to generate distributed images for each user, the generation of distributed images can be made more efficient.

[0098] Note that processing such as aggregation and multiplexing of generated videos may be performed in the client 23 rather than the distribution server 22. In such a case, the client 23 directly acquires generated videos from multiple rendering servers 21, thereby achieving lower latency than acquiring generated videos or distributed videos via the distribution server 22. However, the client 23 needs to communicate with multiple rendering servers 21, which increases the processing load on the client 23, such as multiplexing.

[0099] This technology will now be described in further detail.

[0100] First, with reference to FIG. 7, a case will be considered in which distribution video data for one user is generated by one rendering server.

[0101] In this case, the user input information input by the user is transmitted to the rendering server SV12 via the distribution server SV11.

[0102] The user input information here includes, for example, 6DoF information indicating the user's position and orientation in the three-dimensional virtual space, and is information input by the user operating a key, button, or controller, for example.

[0103] The rendering server SV12 performs 3D rendering processing based on user input information, generates rendered video, i.e., distribution video data (distribution video), at a frame rate that can be generated, and transmits it to the distribution server SV11. In this case, the frame rate of the distribution video is limited by the processing capacity of the rendering server SV12. In this example, distribution video with a frame rate of 60 fps is generated.

[0104] When the distribution server SV11 receives the distribution video data transmitted from the rendering server SV12, it basically transmits (passes through) the distribution video data to the client CL11 as is.

[0105] When client CL11 receives the distribution video data transmitted from distribution server SV11, it supplies the distribution video data to display DP11 and displays the distribution video. At this time, a response delay occurs between the time the user inputs user input information and the time the distribution video corresponding to that input is displayed on display DP11. This response delay is also called C2P latency (Click to Photon latency).

[0106] Unless the video is streamed at a high frame rate, it is usually streamed at a frame rate of around 30 to 60 fps. While streamed video at a high frame rate can reduce response delays, the 3D rendering time required to generate one frame is also reduced, so there is a limit to how high the frame rate can be.

[0107] In contrast to the above example in which one rendering server SV12 is used, consider the case in which two rendering servers are used to generate video for one user, as shown in FIG.

[0108] In this example, a high frame rate video stream with a frame rate of 120 fps is generated by two rendering servers SV22 and SV23.

[0109] When the user input information is supplied to the distribution server SV21, the distribution server SV21 requests the rendering server SV22 and the rendering server SV23 to generate a distribution video to be presented to one user.

[0110] In this case, the distribution server SV21 transmits, together with the user input information, time information indicating the time (playback time) at which a frame in the distribution video data is to be generated to the rendering server SV22 and the rendering server SV23.

[0111] Here, rendering server SV22 is responsible for generating frames at times t1, t3, t5, t7, and t9, and rendering server SV23 is responsible for generating frames at times t2, t4, t6, t8, and t10. In other words, the 120p (120 fps) distribution video is split into 60p (60 fps) video and 60p video, and these 60p videos are generated by separate rendering servers.

[0112] That is, the rendering server SV22 performs 3D rendering processing based on the user input information and time information, and generates 60 fps distribution video consisting of frames at times t1, t3, t5, t7, and t9. Similarly, the rendering server SV23 performs 3D rendering processing based on the user input information and time information, and generates 60 fps distribution video consisting of frames at times t2, t4, t6, t8, and t10.

[0113] Generally, when generating video (moving images), the reciprocal of the frame rate is used as the exposure time for each frame, and 3D rendering is performed using data for that exposure time. Here, since the frame rate of the video to be distributed is 60 fps, one frame of video to be distributed is generated using data for a 1 / 60 second period out of the total period of data that makes up the scene data.

[0114] Furthermore, the rendering servers SV22 and SV23 transmit the generated distribution video data to the distribution server SV21 in a non-burst manner. In this case, each rendering server transmits one frame of distribution video data over a period of 1 / 60 seconds, which is the reciprocal of the frame rate (60 fps).

[0115] The distribution server SV21 multiplexes the distribution video received from the rendering server SV22 and the distribution video received from the rendering server SV23 to generate a high frame rate distribution video with a frame rate of 120 fps. When multiplexing (combining) the distribution video, time information is referenced as appropriate to identify the time sequence of the two distribution video data (video streams).

[0116] The distribution server SV21 transmits the generated 120 fps distribution video to the client CL11, which causes the 120 fps distribution video to be displayed on the display DP11 on the client CL11 side.

[0117] In such an example, even if one rendering server only has the processing power to generate 60 fps video in real time, by using two rendering servers it is possible to ultimately deliver 120 fps video.

[0118] However, since the response delay (C2P latency) is determined by the processing time of the rendering server, which takes the longest time for 3D rendering processing, there is a possibility that the delay will be greater than when generating 120p streaming video using a single rendering server.

[0119] In the example of Figure 8, as shown by arrow Q11, frames at each time point with an exposure time of 1 / 60 seconds are multiplexed (combined) to generate 120 fps distribution video data, which results in overlapping video content between adjacent frames.

[0120] For example, when generating the frames at time t1 and time t2, data for a 1 / 60 second period in the scene data is used, but the data for 1 / 120 seconds of that data is the same. In other words, data for the same 1 / 120 second period in the scene data is used. Therefore, the frames at time t1 and time t2 contain overlapping information about the subject for the same 1 / 120 second period.

[0121] Therefore, although the streamed video displayed on the DP11 display appears (externally) to be a high frame rate of 120 fps, in terms of the information contained in the video it is not a pure 120 fps video, and motion blur occurs within the video.

[0122] Therefore, in the present technology (information processing system 11), as shown in Fig. 9 for example, exposure time information indicating the exposure time of the generated image is set (specified) separately from time information, that is, the image generation cycle (frame rate), thereby suppressing motion blur in the image. In other words, it is possible to obtain an image with less motion blur.

[0123] In this example, a high frame rate video to be distributed, which has a frame rate of 120 fps, is generated by two rendering servers 21-1 and 21-2 in a shared manner.

[0124] When the user input information, that is, the client attribute information including the 6DoF information, is supplied to the distribution server 22, the distribution server 22 requests the rendering server 21-1 and the rendering server 21-2 to generate a generated image by 3D rendering processing.

[0125] In this case, viewpoint information (user input), time information, and exposure time information based on the client attribute information are transmitted to each rendering server 21. In particular, the time information supplied to the rendering server 21-1 here is information indicating times t1, t3, t5, t7, and t9, and the time information supplied to the rendering server 21-2 is information indicating times t2, t4, t6, t8, and t10. Furthermore, the exposure time information is information indicating an exposure time of "1 / 120 seconds." The exposure time is determined, for example, based on the frame rate of the video to be distributed.

[0126] The rendering server 21-1 performs 3D rendering processing based on viewpoint information, time information, exposure time information, etc., and generates a 60 fps generated video consisting of frames at times t1, t3, t5, t7, and t9. Similarly, the rendering server 21-2 performs 3D rendering processing based on viewpoint information, time information, exposure time information, etc., and generates a 60 fps generated video consisting of frames at times t2, t4, t6, t8, and t10.

[0127] In particular, in this case, one frame of the generated image is generated using data for a period of 1 / 120 seconds, which is the exposure time indicated by the exposure time information, out of the entire period of data that makes up the scene data. In other words, the frame is generated using an exposure time that is different from the time for one frame of the generated image (1 / 60 seconds), which is determined by the frame rate of the generated image, and more specifically, an exposure time that is shorter than the time for one frame.

[0128] Each rendering server 21 transmits the generated video data to the distribution server 22. For example, the transmission of the generated video data is non-burst transmission.

[0129] The distribution server 22 multiplexes (combines) the generated images received from each rendering server 21 based on the time information, generates a high frame rate distribution image with a frame rate of 120 fps, and transmits it to the client 23. For example, the transmission (transmission) of the distribution image data from the distribution server 22 to the client 23 is non-burst transmission.

[0130] The client 23 supplies the distributed video to the display 142 via a cable such as HDMI (High-Definition Multimedia Interface) (registered trademark) or DP (DisplayPort), and displays the distributed video. As a result, the distributed video is displayed on the display 142 at 120 fps.

[0131] In this example, as shown by arrow Q21, frames with an exposure time of 1 / 120 seconds are multiplexed (combined) to generate 120 fps video data for distribution, so there is no overlap in the video content between adjacent frames. In other words, smooth video with little motion blur can be obtained for distribution, with no gaps or overlaps on the time axis.

[0132] Furthermore, since the generation of each frame is shared among a plurality of rendering servers 21, even a rendering server 21 that cannot generate high frame rate video by itself can contribute to the generation of high frame rate video for distribution.

[0133] Note that the example described here is one in which 120p (120 fps) distribution video is divided into 60p (60 fps) video and 60p video, and these 60p videos are generated by separate rendering servers 21. However, the present invention is not limited to this example. Depending on the processing capabilities of the rendering servers 21, etc., the 120p (120 fps) distribution video may be divided into 60p (60 fps) video, 30p video, and 30p video, and these videos may be generated by three rendering servers 21.

[0134] Also, although an example in which the distribution server 22 sets the exposure time has been described here, the rendering server 21 may obtain information indicating the frame rate of the distributed video at the time of multiplexing from the distribution server 22 and set an exposure time that is different from the reciprocal of the frame rate of the generated video.

[0135] Additionally, the generated video obtained by each rendering server 21 may be used to generate video for distribution to multiple users, thereby improving the efficiency of generating video for distribution.

[0136] As a specific example, in the example of Figure 9, consider two users (avatars) who are in approximately the same position in a three-dimensional virtual space, and consider reducing the processing load of the 3D rendering process by generating the distribution video of each of the two users from the same generated video.

[0137] Here, it is assumed that the resources available for 3D rendering processing are two rendering servers 21, and that these rendering servers 21 are capable of generating 60p (60 fps) video through 3D rendering processing. It is also assumed that the characteristics of each user's display 142 are different, and that the distributed video will be viewed at 60p (60 fps) and 120p (120 fps), respectively.

[0138] In such a case, the frame rate of the distribution video presented to one user differs from the frame rate of the distribution video presented to the other user.

[0139] Therefore, it would be possible to perform 3D rendering processing to obtain 60p (60 fps) video distribution and 3D rendering processing to obtain 120p (120 fps) video distribution independently, but there are currently insufficient resources for 3D rendering processing to do this.

[0140] Furthermore, although it is possible to generate two streams of 60p (60 fps) generated video using two rendering servers 21 without specifying exposure time information, it is not possible to obtain 120p (120 fps) distributed video with no overlapping video content by multiplexing these generated videos.

[0141] Therefore, the distribution server 22 instructs each of the two rendering servers 21 to execute 3D rendering processing, that is, to generate a generated image, using rendering parameters determined based on the client attribute information of each user.

[0142] In this case, the rendering parameters are such that each frame of the generated video is generated with an exposure time of "1 / 120 seconds," a frame rate of "60 fps," and a rendering timing time difference of "1 / 120 seconds."

[0143] The distribution server 22 multiplexes the generated video (two video streams) thus obtained, generating a 60p (60 fps) distribution video and a 120p (120 fps) distribution video.

[0144] For example, 60p video can be obtained by down-converting 120p video obtained by multiplexing. In this case, if the 2D video is down-converted using a process that has a lower processing load than 3D rendering, the processing load on the entire information processing system 11 can be kept low.

[0145] By the above process, it is possible to generate distributed video with different frame rates for each user efficiently, i.e., with a low processing load. Also, each user can view the distributed video at an appropriate frame rate. As described above, the present technology is characterized in that rendering parameters for the 3D rendering process are determined based on the client attribute information of the client 23.

[0146] <Description of Distribution Processing> The operation of the information processing system 11 will be described.

[0147] When each client 23 plays back 3D content, the distribution server 22 transmits scene data of the 3D content to each rendering server 21 in advance at an appropriate timing. That is, the transmission unit 104 of the distribution server 22 reads out scene data from the scene data storage unit 105 and transmits it to the rendering server 21. Furthermore, the reception unit 72 of the rendering server 21 receives the scene data transmitted from the distribution server 22 and supplies it to the scene data storage unit 73 for storage (holding).

[0148] When the scene data is supplied to each rendering server 21, the distribution server 22 then performs distribution processing at an arbitrary timing to distribute 3D content.

[0149] The distribution process performed by the distribution server 22 will be described below with reference to the flowchart of FIG.

[0150] In step S11, the receiving unit 102 receives the client attribute information transmitted from each client 23 and supplies it to the rendering parameter generating unit 103. More specifically, the timing at which the client attribute information is transmitted differs for each client 23.

[0151] In step S12, the rendering parameter generating unit 103 determines the time of the generated video, that is, the frame generation time of the generated video, for each rendering server 21 based on the client attribute information supplied from the receiving unit 102.

[0152] For example, the rendering parameter generation unit 103 determines the frame rate of the video to be distributed to each client 23 based on the client attribute information of each client 23 that is the destination (distribution destination) of the video to be distributed, particularly the device information about the client 23 included in the client attribute information. More specifically, the frame rate of the video to be distributed is determined based on, for example, the frame rate that the display 142 can handle as device information, or information indicating the refresh rate of the video on the display 142.

[0153] The frame rate of the distributed video may be determined based on factors such as the communication status (congestion status) of the network between the distribution server 22 and the client 23, the current processing load of each rendering server 21, the movement speed of the user (avatar) in the three-dimensional virtual space, and the speed of movement of body parts.

[0154] The rendering parameter generation unit 103 determines which rendering server 21 is to be responsible for generating a generated video (frame) at which viewpoint and at which time, based on at least one of the results of determining the frame rate of the video to be distributed, information about each user in the three-dimensional virtual space, and information about each rendering server 21 acquired from the rendering server 21, etc. In other words, for each rendering server 21, a frame generation time, which is the time (frame) of the generated video to be generated, is determined.

[0155] Here, the information about the user includes, for example, the user's position in the three-dimensional virtual space, the direction of the user's face (direction of line of sight), movement (speed and direction of movement), etc. Furthermore, the information about the rendering server 21 includes, for example, the processing capacity of the rendering server 21, the processing load (processing resources) of the rendering server 21 at the current point in time (current time), the number of rendering servers 21 connected to the distribution server 22, etc.

[0156] As mentioned above, for users who are located at approximately the same position in the three-dimensional virtual space, a distribution video for each user may be generated from the same generated video, in which case a generated video common to multiple users will be generated.

[0157] In step S13, the rendering parameter generation unit 103 generates a request to execute 3D rendering processing for each rendering server 21 based on the results of determining the frame generation time and the frame rate of the distributed video in step S12, and supplies the request to the transmission unit 104.

[0158] Specifically, the rendering parameter generating unit 103 generates rendering parameters for the rendering server 21 .

[0159] For example, the rendering parameter generation unit 103 generates viewpoint information based on 6DoF information included in the client attribute information of the user (client 23) to whom the generated video is to be presented. In this case, the viewpoint information is information indicating the position indicated by the 6DoF information, i.e., the viewpoint position of the user (avatar) in the three-dimensional virtual space. Note that the viewpoint information may also include information indicating the line of sight of the user (avatar) in the three-dimensional virtual space.

[0160] Furthermore, the rendering parameter generating unit 103 generates time information and video generation cycle information based on the result of determining the frame generation time in step S12.

[0161] In this case, the time information generated is information indicating the time (frame generation time) of the frame of the generated video data handled by the rendering server 21. The video generation cycle information is information indicating the frame rate of the generated video (video generation cycle) that is determined by the frame rate of the distributed video data and the determination result of the frame generation time for the generated video.

[0162] Furthermore, the rendering parameter generation unit 103 determines the exposure time of the generated video based on the frame rate of the video to be distributed for each user (client 23), which is determined by the client attribute information of each client 23, and generates exposure time information indicating the exposure time. Specifically, for example, the reciprocal of the frame rate of the video to be distributed is set as the exposure time.

[0163] The rendering parameter generation unit 103 generates rendering parameters including the time information, viewpoint information, image generation cycle information, and exposure time information obtained in this manner, and supplies an execution request including the rendering parameters to the transmission unit 104. Note that if the frame rate of the generated image (image generation cycle) can be identified from the time information, the rendering parameters do not need to include image generation cycle information.

[0164] In step S14, the transmitting unit 104 transmits the execution requests for each rendering server 21 supplied from the rendering parameter generating unit 103 to each of those rendering servers 21. That is, the transmitting unit 104 transmits, to each of one or more rendering servers 21, a request to execute 3D rendering processing for generated video data that displays generated videos at mutually different positions or times.

[0165] When the execution request is sent, each rendering server 21 performs 3D rendering processing, and the resulting generated video data is sent to the distribution server 22 .

[0166] In step S15, the receiving unit 106 receives the generated video data transmitted from each rendering server 21 and supplies it to the video storage unit 107 for storage.

[0167] In step S16, the video multiplexing unit 108 performs necessary processing such as multiplexing, video clipping, down-conversion, and interpolation based on the generated video data stored in the video storage unit 107, thereby generating video data to be distributed for each client 23. At this time, the video multiplexing unit 108 generates the video data to be distributed by referring to the rendering parameters generated by the rendering parameter generation unit 103 and the client attribute information of each client 23 as necessary.

[0168] Specifically, for example, the video multiplexing unit 108 obtains distribution video data at a desired frame rate by chronologically arranging generated videos (frames) from the same viewpoint at different times to create distribution video data consisting of multiple frames.

[0169] In this case, for example, when the generated image is a panoramic image, the image multiplexing unit 108 cuts out the area of ​​the user's field of view determined by the line of sight indicated by the 6DoF information from the panoramic image, and uses the resulting two-dimensional cut-out image as one frame of distributed image.

[0170] Furthermore, for example, if the frame rate of the distribution video obtained by multiplexing is higher than the frame rate presented to the user, the video multiplexing unit 108 down-converts the distribution video data to convert it into distribution video data with a desired frame rate. Alternatively, for example, generated videos from other viewpoint positions or times may be generated by interpolation processing based on generated videos from different viewpoint positions or times, and the generated generated videos may be used to generate distribution video by multiplexing or the like.

[0171] The video multiplexing unit 108 supplies the distribution video data generated for each client 23 to the transmitting unit 109 .

[0172] In step S17, the transmitting unit 109 transmits the distribution video data for each client 23 supplied from the video multiplexing unit 108 to each client 23. In this way, the 3D content is distributed to each client 23.

[0173] If the 3D content also includes audio, audio data constituting the 3D content is also received from the rendering server 21. The transmitting unit 109 then transmits the audio data to the client 23 together with the distribution video data.

[0174] In step S18, the control unit 101 determines whether or not to end the process of distributing the 3D content. For example, in step S18, it is determined that the process is to end when distribution of the 3D content has been completed up to the last frame.

[0175] If it is determined in step S18 that the process is not yet finished, the process then returns to step S11, and the above-described process is repeated.

[0176] On the other hand, if it is determined in step S18 that the process is to be ended, the control unit 101 stops the operation of each unit of the distribution server 22, and the distribution process ends.

[0177] In this way, the distribution server 22 generates rendering parameters including exposure time information indicating the exposure time of the generated video, which is set separately from the frame rate of the generated video, and transmits an execution request including the rendering parameters to the rendering server 21. The distribution server 22 also multiplexes the generated video data to generate distribution video data for each client 23. In this way, it is possible to obtain distribution video with little motion blur.

[0178] <Description of Data Generation Process> When the distribution process described with reference to Fig. 10 is started in the distribution server 22, the rendering server 21 performs the data generation process shown in Fig. 11. Hereinafter, the data generation process by the rendering server 21 will be described with reference to the flowchart in Fig. 11.

[0179] In step S 41 , the receiving unit 72 receives the execution request transmitted from the distribution server 22 and supplies it to the video generating unit 74 .

[0180] In step S42 , the video generating unit 74 performs 3D rendering processing based on the rendering parameters included in the execution request supplied from the receiving unit 72 and the scene data stored in the scene data storage unit 73 .

[0181] For example, the video generation unit 74 identifies a target period to be subjected to the 3D rendering process based on the time information and exposure time information included in the rendering parameters. For example, the target period is a period whose start time is the time indicated by the time information (playback time) and whose length is indicated by the exposure time information.

[0182] The video generation unit 74 places images of each object in a three-dimensional virtual space based on information about the target period from the information about the entire period of the scene description information included in the scene data and the video object data in the three-dimensional object data included in the scene data.

[0183] Furthermore, for example, the video generation unit 74 places a video of a performer (artist) at a predetermined position in the three-dimensional virtual space based on video data of the performer (artist) at a live performance or the like, captured in real space. Alternatively, a user's avatar may be placed in the three-dimensional virtual space.

[0184] The video generation unit 74 generates a generated video based on the placement of objects and performers in the three-dimensional virtual space and the viewpoint information included in the rendering parameters, with the viewpoint position being the position indicated by the viewpoint information in the three-dimensional virtual space at the time indicated by the time information.

[0185] The image generation unit 74 generates, as generated images, panoramic images for one or more times indicated by the time information, at an image generation period (rendering period) indicated by the image generation period information included in the rendering parameters. This results in generated image data at a frame rate indicated by the image generation period. Note that the image generation unit 74 may also cut out a field of view determined by the user's line of sight indicated by the viewpoint information from the panoramic image, and use the resulting two-dimensional cut-out image as the final generated image.

[0186] The video generation unit 74 executes the above process as a 3D rendering process, and supplies the resulting generated video data to the transmission unit 75 .

[0187] In step S43 , the transmitting unit 75 transmits the generated video data supplied from the video generating unit 74 to the distribution server 22 .

[0188] In addition, if the 3D content also has audio, the control unit 71 or the like also generates audio data for the target period that constitutes the 3D content, and the transmission unit 75 transmits the audio data to the distribution server 22 together with the generated video data.

[0189] In step S44, the control unit 71 determines whether or not to end the process of generating generated video data. For example, in step S44, it is determined that the process is to be ended if an instruction to end the process is received from the distribution server 22.

[0190] If it is determined in step S44 that the process is not yet finished, the process then returns to step S41, and the above-described process is repeated.

[0191] On the other hand, if it is determined in step S44 that the process is to be ended, the control unit 71 stops the operation of each unit of the rendering server 21, and the data generation process ends.

[0192] In this way, the rendering server 21 performs 3D rendering processing based on the rendering parameters in response to a request from the distribution server 22, and transmits the resulting generated video data to the distribution server 22. By performing 3D rendering processing in accordance with the rendering parameters in this way, it is possible to obtain distributed video with less motion blur with a smaller processing load in the entire information processing system 11.

[0193] <Explanation of Content Playback Processing> When the distribution processing described with reference to Fig. 10 is started in the distribution server 22, the client 23 performs the content playback processing shown in Fig. 12. Hereinafter, the content playback processing by the client 23 will be described with reference to the flowchart in Fig. 12.

[0194] In step S71, the client attribute information generating unit 153 determines whether or not there is a user input.

[0195] For example, the user inputs the user's movement by operating the input unit 141 or by moving while wearing a sensor serving as the input unit 141, and a signal indicating the input is acquired as user input by the user input acquisition unit 152. Upon acquiring the user input, the user input acquisition unit 152 supplies a signal corresponding to the user input to the client attribute information generation unit 153.

[0196] When a signal corresponding to a user input is supplied from the user input acquisition unit 152, the client attribute information generation unit 153 determines in step S71 that a user input has been made.

[0197] If it is determined in step S71 that there is user input, that is, if the user moves and the position of the user (avatar) in the three-dimensional virtual space changes, the client attribute information generation unit 153 generates client attribute information in step S72.

[0198] That is, the client attribute information generation unit 153 generates 6DoF information by identifying the movement of the user (avatar) based on the signal supplied from the user input acquisition unit 152, and generates client attribute information including the 6DoF information and device information. The client attribute information generation unit 153 supplies the obtained client attribute information to the transmission unit 154.

[0199] In step S73, the transmitting unit 154 transmits the client attribute information supplied from the client attribute information generating unit 153 to the distribution server 22, and then the process proceeds to step S74.

[0200] If it is determined in step S71 that there is no user input, the processes of steps S72 and S73 are not performed, and the process then proceeds to step S74.

[0201] If the processing of step S73 has been performed or if it is determined in step S71 that there has been no user input, in step S74 the receiving unit 155 receives the distribution video data transmitted from the distribution server 22 and supplies it to the video display control unit 156.

[0202] In step S75, the video display control unit 156 supplies the distributed video data supplied from the receiving unit 155 to the display 142, causing the display 142 to display the distributed video. This results in the 3D content being played back. If the 3D content also has audio, the receiving unit 155 also receives audio data of the 3D content. The control unit 151 or the like then supplies the audio data to a speaker (not shown), and the audio of the 3D content is also played back by the speaker.

[0203] In step S76, the control unit 151 determines whether or not to end the process of playing back the 3D content. For example, in step S76, it is determined that the process should end if the playback of the 3D content has ended up to the last frame.

[0204] If it is determined in step S76 that the process is not yet finished, the process then returns to step S71, and the above-described process is repeated.

[0205] On the other hand, if it is determined in step S76 that the process is to be ended, the control unit 151 stops the operation of each unit of the client 23, and the content playback process ends.

[0206] In this way, the client 23 receives the distribution video data from the distribution server 22 and displays (plays) the distribution video, thereby allowing the user to view 3D content.

[0207] <Modification> For example, the transmission of generated video data between the rendering server 21 and the distribution server 22 and the transmission of distributed video data between the distribution server 22 and the client 23 can be non-burst transmission, but can also be burst transmission as shown in Fig. 13. Note that the description of the same parts in Fig. 13 as those in Fig. 9 will be omitted as appropriate.

[0208] In the example of Figure 13, as in the example of Figure 9, when user input information, i.e., client attribute information including 6DoF information, is supplied to the distribution server 22, the distribution server 22 requests the rendering server 21 to generate a generated video.

[0209] The rendering server 21-1 performs 3D rendering processing based on viewpoint information (user input), time information, exposure time information, etc., and generates a 60 fps generated video consisting of frames at times t1, t3, t5, t7, and t9. Similarly, the rendering server 21-2 performs 3D rendering processing based on viewpoint information, time information, exposure time information, etc., and generates a 60 fps generated video consisting of frames at times t2, t4, t6, t8, and t10.

[0210] Each rendering server 21 burst-transmits the generated video data to the distribution server 22. In burst transmission, the generated video data is transmitted at the maximum speed of the transmission path over which it is transmitted, rather than at a transmission rate (bit rate) corresponding to the frame rate of the generated video data.

[0211] The distribution server 22 multiplexes the generated videos received from each rendering server 21 to generate 120 fps distribution video, and burst-transmits the distribution video to the client 23. The client 23 supplies the distribution video to the display 142 by non-burst transmission via a cable such as HDMI (registered trademark) or DP, and displays the distribution video.

[0212] In this example, the transmission of generated video data and distributed video data is performed using burst transmission, which allows for effective use of the transmission path, i.e., the network bandwidth. Furthermore, burst transmission allows the transmission of each frame of generated video data and distributed video data to end earlier than in non-burst transmission. Therefore, on the receiving side of the generated video data and distributed video data, processing performed on a frame-by-frame basis for multiplexing and playback can be started earlier, thereby shortening the response delay (C2P latency).

[0213] As described above, it is also possible to generate video data with different frame rates depending on the resources of each rendering server 21, i.e., the current processing load and the processing capabilities of each rendering server 21.

[0214] FIG. 14 shows an example in which a 90 fps equivalent generated video and a 30 fps equivalent generated video are multiplexed to generate a 120 fps distribution video.

[0215] 9 , when user input information, i.e., client attribute information including 6DoF information, is supplied to the distribution server 22, the distribution server 22 requests the rendering server 21 to generate a generated video. In particular, in this example, the transmission unit 104 of the distribution server 22 transmits, to each of the multiple rendering servers 21, a request to execute generated video data having a different frame rate.

[0216] The rendering server 21-1 performs 3D rendering processing based on viewpoint information (user input), time information, exposure time information, etc., and generates a generated video equivalent to 90 fps consisting of frames at times t1, t2, t3, t5, t6, t7, t9, and t10. In contrast, the rendering server 21-2 performs 3D rendering processing based on viewpoint information, time information, exposure time information, etc., and generates a generated video equivalent to 30 fps consisting of frames at times t4 and t8.

[0217] Each rendering server 21 burst-transmits the generated video data to the distribution server 22. The distribution server 22 multiplexes the generated video received from each rendering server 21 to generate 120 fps distribution video, and burst-transmits the distribution video to the client 23. The client 23 supplies the distribution video to the display 142 by non-burst transmission via a cable such as HDMI (registered trademark) or DP, and displays the 120 fps distribution video.

[0218] In this way, by determining the frame rate of the generated video (video generation cycle) according to the processing load and processing capacity of each rendering server 21, it is possible to obtain video for distribution more efficiently.

[0219] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0220] FIG. 15 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0221] In the computer, a CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .

[0222] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0223] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0224] In a computer configured as described above, the CPU 501 loads a program recorded in the recording unit 508, for example, into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program, thereby performing the above-described series of processes.

[0225] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0226] In a computer, a program can be installed in the recording unit 508 via the input / output interface 505 by inserting a removable recording medium 511 into the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0227] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0228] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0229] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0230] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0231] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0232] Furthermore, the present technology can also be configured as follows.

[0233] (1) An information processing device comprising: a parameter generation unit that generates rendering parameters for generating the first video data by a rendering process, the rendering parameters including video generation period information indicating a frame rate of the first video data and exposure time information indicating an exposure time of the first video data; and a transmission unit that transmits an execution request for the rendering process including the rendering parameters to a rendering server. (2) The information processing device described in (1), wherein the transmission unit transmits each of the execution requests for the first video data for displaying videos at different times to each of the plurality of rendering servers, and further comprises an image generation unit that generates second video data having a frame rate different from that of the first video data, based on the first video data for videos at different times received from each of the plurality of rendering servers. (3) The information processing device described in (2), wherein the parameter generation unit determines the exposure time of the first video data based on the frame rate of the second video data. (4) The information processing device according to (3), wherein the parameter generation unit determines a frame rate of the second video data based on a frame rate or a video refresh rate that can be supported by a display unit at a destination of the second video data. (5) The information processing device according to any one of (1) to (4), wherein the transmission unit transmits, to each of the plurality of rendering servers, the execution requests for the first video data, the frame rates of which differ from each other. (6) The information processing device according to any one of (2) to (4), wherein the parameter generation unit determines the frame rate of the first video data based on at least any of the frame rate of the second video data, the processing load of the rendering server, the processing capacity of the rendering server, and the number of the rendering servers. (7) The information processing device according to any one of (1) to (6), wherein the first video data is transmitted to the information processing device in bursts. (8) The information processing device according to any one of (2) to (4), wherein the second video data is burst-transmitted to a destination of the second video data.(9) The information processing device according to any one of (1) to (8), wherein the rendering process is a 3D rendering process that generates the first video data for displaying a video with a predetermined position in the three-dimensional space as a viewpoint position, based on data related to an object placed in the three-dimensional space. (10) The information processing device according to (9), wherein the rendering parameters include viewpoint information that indicates the viewpoint position. (11) An information processing method, including: an information processing device generating rendering parameters for generating the first video data by a rendering process, the rendering parameters including video generation period information that indicates a frame rate of the first video data and exposure time information that indicates an exposure time of the first video data; and transmitting an execution request for the rendering process that includes the rendering parameters to a rendering server. (12) An information processing device comprising: a video generation unit that generates the first video data by a rendering process based on rendering parameters including video generation period information that indicates the frame rate of the first video data and exposure time information that indicates the exposure time of the first video data. (13) The information processing device according to (12), further comprising a receiving unit that receives a request to execute the rendering process, the request including the rendering parameters. (14) The information processing device according to (12) or (13), further comprising a transmitting unit that transmits the first video data to a server. (15) The information processing device according to (14), wherein the transmitting unit transmits the first video data in bursts. (16) The information processing device according to any one of (12) to (15), wherein the exposure time is determined based on a frame rate of second video data generated based on the first video data of videos captured at different times. (17) The information processing device according to (16), wherein the second video data and the first video data have different frame rates. (18) The information processing device according to any one of (12) to (17), wherein the rendering process is a 3D rendering process that generates the first video data for displaying an image with a viewpoint located at a predetermined position in the three-dimensional space, based on data related to an object arranged in the three-dimensional space.(19) The information processing device according to (18), wherein the rendering parameters include viewpoint information indicating the viewpoint position. (20) An information processing method, including: generating the first video data by rendering processing based on rendering parameters including video generation cycle information indicating a frame rate of the first video data and exposure time information indicating an exposure time of the first video data.

[0234] 11 Information processing system, 21-1 to 21-N, 21 Rendering server, 22-1, 22-2, 22 Distribution server, 23-1 to 23-M, 23 Client, 71 Control unit, 72 Receiving unit, 74 Video generation unit, 75 Transmitting unit, 101 Control unit, 103 Rendering parameter generation unit, 104 Transmitting unit, 106 Receiving unit, 108 Video multiplexing unit, 109 Transmitting unit

Claims

1. An information processing device comprising: a parameter generation unit that generates rendering parameters for generating the first video data by a rendering process, the rendering parameters including video generation cycle information indicating the frame rate of the first video data and exposure time information indicating the exposure time of the first video data; and a transmission unit that transmits a request to execute the rendering process, including the rendering parameters, to a rendering server.

2. The information processing device according to claim 1, wherein the transmitting unit transmits to each of the plurality of rendering servers each of the execution requests for the first video data that display video at different times, and further comprises a video generating unit that generates second video data having a frame rate different from that of the first video data based on the first video data of video at different times received from each of the plurality of rendering servers.

3. The information processing device according to claim 2, wherein the parameter generating section determines the exposure time of the first video data based on the frame rate of the second video data.

4. The information processing device according to claim 3, wherein the parameter generating unit determines the frame rate of the second video data based on a frame rate or video refresh rate that can be supported by a display unit at the destination of the second video data.

5. The information processing device according to claim 1, wherein the transmission unit transmits the execution requests for the first video data having different frame rates to each of the plurality of rendering servers.

6. The information processing device according to claim 2, wherein the parameter generation unit determines the frame rate of the first video data based on at least one of the frame rate of the second video data, the processing load of the rendering server, the processing capacity of the rendering server, and the number of the rendering servers.

7. The information processing device according to claim 1, wherein the first video data is transmitted to the information processing device in bursts.

8. The information processing device according to claim 2, wherein the second video data is transmitted in bursts to a destination of the second video data.

9. The information processing device according to claim 1, wherein the rendering process is a 3D rendering process that generates the first video data for displaying an image with a predetermined position in the three-dimensional space as a viewpoint based on data relating to an object placed in the three-dimensional space.

10. The information processing device according to claim 9, wherein the rendering parameters include viewpoint information indicating the viewpoint position.

11. An information processing method, comprising: an information processing device generating rendering parameters for generating the first video data by rendering processing, the rendering parameters including video generation cycle information indicating the frame rate of the first video data and exposure time information indicating the exposure time of the first video data; and transmitting a request to a rendering server to execute the rendering processing, the request including the rendering parameters.

12. An information processing device comprising: an image generation unit that generates the first image data by rendering processing based on rendering parameters including image generation cycle information indicating the frame rate of the first image data and exposure time information indicating the exposure time of the first image data.

13. The information processing device according to claim 12, further comprising a receiving unit that receives a request to execute the rendering process, the request including the rendering parameters.

14. The information processing device according to claim 12, further comprising a transmitting unit that transmits the first video data to a server.

15. The information processing device according to claim 14, wherein the transmission unit transmits the first video data in bursts.

16. The information processing device according to claim 12, wherein the exposure time is determined based on a frame rate of second video data generated based on the first video data of videos taken at different times.

17. The information processing device according to claim 16, wherein the second video data and the first video data have different frame rates.

18. The information processing device according to claim 12, wherein the rendering process is a 3D rendering process that generates the first video data for displaying an image with a predetermined position in the three-dimensional space as a viewpoint based on data relating to an object to be placed in the three-dimensional space.

19. The information processing device according to claim 18, wherein the rendering parameters include viewpoint information indicating the viewpoint position.

20. An information processing method including: an information processing device generating the first video data by rendering processing based on rendering parameters including video generation cycle information indicating the frame rate of the first video data and exposure time information indicating the exposure time of the first video data.

Images