Server device, terminal device, information processing system and information processing method

Abstract

[Problem] To provide technology capable of reducing the processing load on a server device side during cloud rendering. [Solution] A server device according to the present technology is provided with a control unit. On the basis of viewing position information relating to each terminal device in a viewing area including a plurality of segments, the control unit groups the terminal devices present in segments having the same viewing position, and transmits common video information to the grouped terminal devices by multicasting.

Description

[Technical field]

[0001] The present technology relates to a technology for a server device or the like that performs cloud rendering. [Background technology]

[0002] In recent years, with the increase in network bandwidth and the improvement of GPU performance, it has become possible to generate 3D video from images captured by multiple cameras and distribute it as free viewpoint video. This makes it possible to provide users with a viewing experience where they can enjoy the video from any viewing position and direction by distributing free viewpoint video at sports or music events, for example.

[0003] In the past, to deliver high-quality free-viewpoint video to realize a free-viewpoint viewing experience, a large amount of data was required, requiring a large network bandwidth. In addition, to render the free-viewpoint video, the user's terminal device needed a high-performance GPU, etc.

[0004] To address these issues, cloud rendering has been proposed, in which rendering is performed on the server device side. In cloud rendering, the terminal device first transmits information such as the viewing position and viewing direction to the server. The server device renders the requested video from the free viewpoint video according to the received viewing position and viewing direction, encodes this video as a 2D video stream, and transmits it to the terminal device.

[0005] With cloud rendering, a terminal device only needs to decode and display a two-dimensional video stream, so a high-quality viewing experience can be provided to the user even if the terminal device does not have a high-performance GPU or the like.

[0006] The following Patent Document 1 can be cited as a technique related to the present application. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2017-188649 A Summary of the Invention [Problem to be solved by the invention]

[0008] Cloud rendering has a problem in that the processing load on the server device increases in proportion to the number of terminal devices requesting viewing.

[0009] In view of the above circumstances, an object of the present technology is to provide a technology capable of reducing the processing load on the server device in cloud rendering. [Means for solving the problem]

[0010] A server device according to the present technology includes a control unit that groups terminal devices that are in the same segment and have the same viewing position based on viewing position information of each terminal device in a viewing area that includes a plurality of segments, and transmits common video information to each of the grouped terminal devices by multicast.

[0011] This makes it possible to reduce the processing load on the server device in cloud rendering.

[0012] A terminal device according to the present technology includes a control unit that groups terminal devices that are in the same viewing position in a viewing area including a plurality of segments based on viewing position information of each terminal device in the viewing area, receives common video information from a server device that transmits the common video information by multicast to each of the grouped terminal devices, and renders an image to be displayed based on the received common video information.

[0013] The information processing system according to the present technology includes a server device and a terminal device. The server device groups terminal devices whose viewing positions are in the same segment based on viewing position information of each terminal device within a viewing area that includes multiple segments, and transmits common video information to each of the grouped terminal devices by multicast. The terminal device receives the common image information and renders an image to be displayed based on the received common image information.

[0014] The information processing method of the present technology groups terminal devices whose viewing positions are in the same segment based on viewing position information of each terminal device within a viewing area that includes multiple segments, and transmits common video information to each of the grouped terminal devices via multicast. [Brief description of the drawings]

[0015] [Figure 1] 1 is a diagram illustrating an information processing system according to a first embodiment of the present technology. [Diagram 2] FIG. 2 is a block diagram showing the internal configuration of a terminal device. [Diagram 3] 2 is a block diagram showing an internal configuration of a management server. [Figure 4] 2 is a block diagram showing the internal configuration of a distribution server. FIG. [Diagram 5] FIG. 2 is a diagram showing an example of a viewing area and segments. [Figure 6] 11 is a diagram showing a process of transmitting viewing position information in the terminal device 10. FIG. [Figure 7] FIG. 13 is a diagram showing an example of a state in which a user changes a viewing position. [Figure 8] 13 is a flowchart showing grouping processing etc. in the management server. [Figure 9] FIG. 13 is a diagram showing the relationship between the distribution of the number of terminal devices per segment and a threshold value. [Figure 10] FIG. 11 is a diagram illustrating an example of a distribution server list. [Figure 11] 10 is a flowchart showing a process of requesting video information in the terminal device. [Figure 12] 5 is a flowchart showing a process of generating video information in the server device. [Figure 13] 13 is a flowchart showing a process of generating a lightweight three-dimensional image in the management server. [Figure 14] 10 is a flowchart showing image display processing and the like in the grouped terminal devices. [Figure 15] 13 is a flowchart showing image display processing and the like in a terminal device that is not grouped. [Figure 16] FIG. 2 illustrates how images are rendered from common video information. [Figure 17] FIG. 13 is a diagram showing a state when the viewing position is moved to a required viewing position and the viewing direction is changed to a required viewing direction. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0017] First Embodiment <Overall structure and each part structure> 1 is a diagram showing an information processing system 100 according to a first embodiment of the present technology. As shown in FIG. 1, the information processing system 100 includes a plurality of terminal devices 10 and a plurality of server devices 20.

[0018] The terminal device 10 may be a mobile terminal that can be carried by a user, a wearable terminal that can be worn by a user, or a stationary terminal that is installed and used.

[0019] Examples of mobile terminals include mobile phones (including smartphones), tablet PCs (Personal Computers), portable game consoles, portable music players, etc. Examples of wearable terminals include head mounted displays (HMDs), wristbands (watches), pendants, rings, etc. Examples of stationary terminals include desktop PCs, televisions, stationary game consoles, etc.

[0020] The information processing system 100 in this embodiment is used as a system in which the server device 20 generates the necessary video information by cloud rendering from a 3D image corresponding to the entirety of an actual event venue (e.g., a stadium) in real space, and live-streams the video information to the terminal device 10.

[0021] In addition, the information processing system 100 in this embodiment is used as a system in which the server device 20 generates the necessary video information by cloud rendering from a 3D image corresponding to the entirety of a virtual event venue in a virtual space (for example, a virtual stadium in a game), and live-streams the video information to the terminal device 10.

[0022] A user can enjoy a live event held in real space or a virtual space using the user's terminal device 10. In this case, since cloud rendering is used, the user can enjoy high-quality images even if the processing power of the terminal device 10 is low.

[0023] If the event is an event in real space, the user may be present at the actual event venue, etc., carrying or wearing the terminal device 10 (if the terminal device 10 is a mobile terminal or a wearable terminal). Alternatively, in this case, the user may be present at any location other than the event venue, such as at home (regardless of the type of terminal device 10).

[0024] Furthermore, if the event or the like is an event in a virtual space, the user may be in any location, such as at home (regardless of the type of terminal device 10).

[0025] Here, it is assumed that the server device 20 generates individual video information for each terminal device 10 according to the viewing position, viewing direction, etc., individually requested by each terminal device 10, and transmits all of this individual video information by unicast. In this case, the processing load on the server device 20 increases in proportion to the number of terminal devices 10 requesting viewing.

[0026] For this reason, in this embodiment, under specified conditions, the server device 20 performs a process of grouping terminal devices 10 that are in the same viewing position in a segment 2 based on the viewing position information of each terminal device 10 within a viewing area 1 that includes multiple segments 2, and transmitting common video information to each of the grouped terminal devices 10 by multicast.

[0027] For terminals that are not grouped, individual video information is transmitted from the server device 20 by unicast.

[0028] FIG. 5 is a diagram showing an example of a viewing area 1 and segments 2. In the example shown in FIG. 5, an area corresponding to the entire soccer stadium is set as the viewing area 1, and a state in which this viewing area 1 is divided into a plurality of segments 2 is shown. In the example shown in FIG. 5, an example is shown in which the viewing area 1 is divided into 6×6, or 36, segments 2 in the X-axis direction×Y-axis direction (horizontal direction). The number of segments is not particularly limited. Also, the viewing area 1 may be divided in the Z-axis direction (height direction) to set the segments 2.

[0029] In the description of this embodiment, "viewing area 1" refers to an area corresponding to an actual event venue in real space or a virtual event venue in virtual space, where video can be viewed (an area where a viewing position can be set). Also, "segment 2" refers to a certain area that divides viewing area 1.

[0030] In addition, in the description of this embodiment, the "viewing position" refers to the standpoint of the viewpoint within the viewing area 1 (indicated by a circle in FIG. 5). The viewing position is a position requested by the terminal device 10 and is a position within the viewing area 1 that can be arbitrarily set by the user. When the event is a bend in real space and the terminal device 10 is located at the actual event venue, this viewing position may be the position of the terminal device 10 at the actual event venue.

[0031] In addition, in the description of this embodiment, the "viewing direction" means the viewing direction from the viewing position. The viewing direction is a direction requested by the terminal device 10 and can be arbitrarily set by the user. When the terminal device 10 is located in an actual event venue, this viewing direction may be the direction (posture) in which the terminal device 10 (user) faces in the actual event venue.

[0032] In addition, if the event takes place in real space, a three-dimensional image corresponding to the entire event venue (corresponding to all viewing positions within viewing area 1) is generated by synthesizing image information from multiple cameras installed in the event venue.

[0033] On the other hand, if the event is an event in a virtual space, a three-dimensional image corresponding to the entire event venue etc. (corresponding to all viewing positions within viewing area 1) is generated in advance by the event organizer etc. and stored on the server device 20 side.

[0034] [Terminal device 10] 2 is a block diagram showing the internal configuration of the terminal device 10. As shown in FIG. 2, the terminal device 10 includes a control unit 11, a storage unit 12, a display unit 13, an operation unit 14, and a communication unit 15.

[0035] The display unit 13 is configured by, for example, a liquid crystal display, an EL (Electro-Luminescence) display, etc. The display unit 13 displays an image on the screen according to the control of the control unit 11.

[0036] The operation unit 14 is, for example, a push button type, a proximity type, etc. The operation unit 14 detects various operations such as designation of a viewing position or a viewing direction by a user, and outputs the operations to the control unit 11.

[0037] The communication unit 15 is configured to be able to communicate with each server device 20 .

[0038] The storage unit 12 includes a non-volatile memory for storing various programs and various data required for processing by the control unit 11, and a volatile memory used as a work area for the control unit 11. The various programs may be read from a portable recording medium such as an optical disk or a semiconductor memory, or may be downloaded from a server device 20 on a network.

[0039] The control unit 11 executes various calculations based on various programs stored in the storage unit 12, and controls each unit of the terminal device 10 in an integrated manner.

[0040] The control unit 11 is realized by hardware or a combination of hardware and software. The hardware is configured as a part or the whole of the control unit 1, and examples of this hardware include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a VPU (Vision Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a combination of two or more of these. This also applies to the control units 21 and 31 of the server device 20.

[0041] When the terminal device 10 is a wearable terminal such as an HMD or a mobile terminal such as a smartphone, the terminal device 10 may have various sensors for performing the self-location estimation process. Examples of the various sensors for performing the self-location estimation process include an imaging unit (camera, etc.), an inertial sensor (acceleration sensor, angular velocity sensor, etc.), and a GPS (Global Positioning System).

[0042] In this case, the terminal device 10 (control unit) estimates its own position and orientation using, for example, SLAM (Simultaneous Localization And Mapping) based on image information from the imaging unit, inertial information (acceleration information, angular velocity information, etc.) from the inertial sensor, position information from the GPS, etc.

[0043] For example, when the terminal device 10 (user) is located at an actual event venue in real space, the estimated self-position may be used as viewing position information. Also, when the terminal device 10 (user) is located at an actual event venue in real space, the estimated self-posture may be used as viewing direction information.

[0044] In this embodiment, the control unit 11 of the terminal device 10 typically performs, broadly speaking, "processing for transmitting viewing position information," "processing for requesting common video information," "processing for requesting individual video information," "processing for displaying images based on common video information," "processing for displaying images based on individual video information," and "processing for displaying images based on lightweight 3D video."

[0045] In this embodiment, the "lightweight three-dimensional video" refers to video information generated by reducing the amount of information of a three-dimensional video corresponding to the entire event venue or the like in real or virtual space (corresponding to all viewing positions within the viewing area 1). This lightweight three-dimensional video is typically used in the terminal device 10 when there is a significant change in the viewing position, such as beyond the segment 2.

[0046] [Server device 20] Next, the server device 20 will be described. In this embodiment, two types of server devices 20 are prepared as the server device 20. The first type is a management server 20a, and the second type is a distribution server 20b. The number of management servers 20a is typically one, and the number of distribution servers 20b is typically multiple.

[0047] In the description of this specification, when there is no particular distinction between the two types of server devices 20, they are simply referred to as server devices 20, and when there is a distinction between the two types of server devices 20, they are referred to as a management server 20a and a distribution server 20b. In this embodiment, the whole including the management server 20a and the distribution server 20b can be regarded as one server device 20.

[0048] "Management Server 20a" First, the management server 20a will be described. Fig. 3 is a block diagram showing the internal configuration of the management server 20a. As shown in Fig. 3, the management server 20a includes a control unit 21, a storage unit 22, and a communication unit 23.

[0049] The communication unit 23 is configured to be able to communicate with each terminal device 10 and other server devices 20 .

[0050] The storage unit 22 includes a non-volatile memory that stores various programs and various data necessary for the processing of the control unit 21, and a volatile memory that is used as a work area for the control unit 21. The various programs may be read from a portable recording medium such as an optical disk or a semiconductor memory, or may be downloaded from another server device on the network.

[0051] The control unit 21 executes various calculations based on various programs stored in the storage unit 22, and performs overall control of each part of the management server 20a.

[0052] In this embodiment, the control unit 21 of the management server 20a typically performs, broadly speaking, "grouping processing," "rendering resource allocation processing," "distribution server list generation processing," "common video information generation processing," "multicast processing of common video information," "individual video information generation processing," "unicast processing of individual video information," "lightweight 3D video generation processing," "lightweight 3D video multicast processing," etc.

[0053] In the description of this embodiment, the term "rendering resource" refers to a unit having a processing capability capable of rendering common video information in multicast or individual video information in unicast. In one server device 20, there may be one or more rendering resources.

[0054] In addition, in this embodiment, the "distribution server list" is a list indicating which of the multiple server devices 20 the terminal device 10 should request video information from depending on its viewing position (see FIG. 10).

[0055] "Distribution server 20b" Next, the distribution server 20b will be described. Fig. 4 is a block diagram showing the internal configuration of the distribution server 20b. As shown in Fig. 4, the distribution server 20b includes a control unit 31, a storage unit 32, and a communication unit 33.

[0056] The distribution server 20b basically has the same configuration as the management server 20a, but the processing of the control unit 31 is different.

[0057] In this embodiment, the control unit 31 of the distribution server 20b typically performs, broadly speaking, "generation processing of common video information," "multicast processing of common video information," "generation processing of individual video information," "unicast processing of individual video information," etc.

[0058] Here, the difference between the management server 20a and the distribution server 20b is that the management server 20a executes "grouping processing," "rendering resource allocation processing," "distribution server list generation processing," "lightweight 3D video generation processing," and "lightweight 3D video multicast processing," while the distribution server 20b does not execute these processes. In other words, the distribution server 20b basically executes processing related to live distribution of common video information or individual video information in response to requests from the terminal devices 10, and does not execute other processes.

[0059] In this embodiment, the management server 20a also functions as the distribution server 20b, but the management server 20a does not necessarily have to have the function of the distribution server 20b.

[0060] <Operation description> Next, each process in the terminal device 10 and the server device 20 will be described.

[0061] [Terminal device 10: Viewing position information transmission process] First, a description will be given of the "viewing position information transmission process" in the terminal device 10. FIG.

[0062] The control unit 11 of the terminal device 10 determines whether or not a viewing position has been specified (changed) by the user within the viewing area 1 (step 101). If the viewing position has not been specified (changed) (NO in step 101), the control unit 11 of the terminal device 10 returns to step 101 and determines again whether or not a viewing position has been specified (changed).

[0063] On the other hand, if the viewing position is specified (changed) by the user, the control unit 11 of the terminal device 10 transmits the viewing position information to the management server 20a (step 102). Then, the control unit 11 of the terminal device 10 returns to step 101 and determines whether the viewing position has been specified (changed).

[0064] Here, as a method of specifying the viewing position, for example, a map corresponding to the entire event venue in real space or virtual space is displayed on the display unit 13 of the terminal device 10 by a GUI (Graphical User Interface), and an arbitrary viewing position is specified by the user. Also, for example, when the user is actually in the event venue, the user's own position estimated by the terminal device 10 may be used as information on the viewing position.

[0065] In addition, after the user has specified the viewing position, the viewing position may be changed. The change in the viewing position may be a major change that exceeds segment 2, or a small change that does not exceed segment 2.

[0066] An example of a state in which a user is changing a viewing position is shown in Fig. 7. In the example shown in Fig. 7, a state in which a user is changing a viewing position by sliding a finger on the screen of a smartphone (terminal device 10) is shown (small change in viewing position).

[0067] [Management server 20a: grouping process, etc.] Next, the "grouping process," "rendering resource allocation process," "distribution server list generation process," and other processes performed by the management server 20a will be described.

[0068] 8 is a flowchart showing grouping processing in the management server 20a. First, the control unit 21 of the management server 20a receives information on viewing positions from all terminal devices 10 requesting viewing (step 201). Next, the control unit 21 of the management server 20a creates a distribution of the number of terminal devices for each segment 2 based on the information on the viewing positions of the terminal devices 10 (step 202).

[0069] Next, control unit 21 of management server 20a determines whether the total number of terminal devices requesting viewing is greater than the total number of rendering resources on the server device 20 (management server 20a, distribution server 20b) side (step 203).

[0070] If the number of terminal devices is greater than the number of rendering resources, the control unit 21 of the management server 20a sets a threshold value for determining the segment 2 for grouping the terminal devices 10 (step 204).

[0071] In other words, if the number of terminal devices is greater than the number of rendering resources, it is not possible to unicast individual video information to all terminal devices 10, so it is necessary to determine segment 2 for grouping, and a threshold value is set for this purpose.

[0072] In this embodiment, the control unit 21 of the management server 20a variably controls this threshold based on the distribution of the number of terminal devices for each segment 2 and the number of rendering resources.

[0073] Fig. 9 is a diagram showing the distribution of the number of terminal devices for each segment 2 and the relationship with the threshold value. In Fig. 9, the number of segment 2 is shown on the left side, and the number of terminal devices whose viewing positions are located in that segment 2 is shown on the right side. Also, in Fig. 9, segments 2 containing the greatest number of terminal devices are arranged from the top.

[0074] In the example shown in FIG. 9, it is assumed that the threshold value is set to 15 and the total number of rendering resources on the server device 20 side is 40.

[0075] 9, the total number of terminal devices in five segments 2, #4, #1, #7, #8, and #6, in which the number of included terminal devices is equal to or less than the threshold value (15), is 28 (=15+7+3+2+1). If individual video information is transmitted by unicast to these 28 terminal devices 10, 28 rendering resources are required. This is because, in the case of unicast, one rendering resource is required for each terminal device 10.

[0076] 9, for three segments 2 #5, #2, and #3 that contain terminal devices exceeding the threshold, if common video information is to be transmitted by multicast to the terminal devices 10 grouped for each segment 2, three rendering resources are required. This is because, in the case of multicast, one rendering resource is required for one segment 2 (one group of terminal devices 10).

[0077] Therefore, when the threshold is set to 15 (i.e., between #3 and #4), the total number of required rendering resources is 31 (28 + 3). This value of 31 is an appropriate value that does not exceed the number of rendering resources (here, 40).

[0078] Here, if the threshold is set to 33 (i.e., between #2 and #3), 63 (61+2) rendering resources are required, exceeding the total number of rendering resources (here, 40). If the threshold is set to 7 (i.e., between #4 and #1), the required rendering resources are 17 (13+4), which does not exceed the total number of rendering resources (here, 40), but the transmission of individual video information by unicast becomes unnecessarily small.

[0079] Therefore, in this example, it is appropriate to set the threshold to 15, and such a threshold value is calculated by the control unit 21 of the management server 20a.

[0080] The threshold value becomes smaller as the number of terminal devices requesting viewing increases (less unicast distribution), and the threshold value becomes larger as the number of rendering resources increases (more unicast distribution).

[0081] In the description of this embodiment, the threshold is variably controlled, but the threshold may be fixed.

[0082] Returning to Fig. 8, once the threshold is set, the control unit 21 of the management server 20a then groups, by segment 2, the terminal devices 10 whose viewing positions are in segments 2 in which the number of terminal devices exceeds the threshold (step 205). For example, in the example shown in Fig. 9, 152 terminal devices 10 whose viewing positions are in segment 2 #5 are grouped, and 52 terminal devices 10 whose viewing positions are in segment 2 #2 are grouped. In addition, 33 terminal devices 10 whose viewing positions are in segment 2 #3 are grouped.

[0083] Next, the control unit 21 of the management server 20a performs an allocation as to which rendering resource (server device 20) will be responsible for generating common video information for which group (segment 2), as well as which rendering resource (server device 20) will be responsible for generating individual video information for which terminal device 10 (step 206).

[0084] Next, a rendering resource (server device 20) that generates common video information for the group is written into the distribution server list (step 207).

[0085] Fig. 10 is a diagram showing an example of the distribution server list. As shown in Fig. 10, the distribution server list includes a server ID of a server device 20 (rendering resource) in charge of generating the common video information, segment range information indicating the range of the corresponding segment 2, and a URL (Uniform Resource Locator) of the common video information.

[0086] After writing the necessary information to the distribution server list, the control unit 21 of the management server 20a then transmits the distribution server list by multicast to all terminal devices 10 that have requested viewing (step 209). Then, the control unit 21 of the management server 20a returns to step 201.

[0087] Here, in step 203, if the number of all terminal devices requesting viewing is equal to or less than the total number of rendering resources on the server device 20 side (NO in step 203), the control unit 21 of the management server 20a proceeds to step 208. In other words, if individual video information can be transmitted to all terminal devices 10 by unicast, the control unit 21 of the management server 20a proceeds to step 208.

[0088] In step 208, the control unit 21 of the management server 20a executes allocation as to which rendering resource (server device 20) will be in charge of generating individual video information for which terminal device 10.

[0089] After step 208, the control unit 21 of the management server 20a transmits the distribution server list by multicast to all the terminal devices 10 (step 209). In this case, an empty distribution server list with nothing written therein is transmitted by multicast. Thereafter, the control unit 21 of the management server 20a returns to step 201.

[0090] [Terminal device 10: video information request processing, etc.] Next, the "common video information request process" and the "individual video information request process" in the terminal device 10 will be described.

[0091] Fig. 11 is a flow chart showing a process of requesting video information in terminal device 10. As shown in Fig. 11, control unit 11 of terminal device 10 receives the distribution server list transmitted by multicast (step 301).

[0092] Next, the control unit 11 of the terminal device 10 determines whether its own viewing position is included in any of the segment ranges shown in the distribution server list (step 302).

[0093] If the viewing position of the terminal device 10 is included within any of the segment ranges (YES in step 302), the control unit 11 of the terminal device 10 sends a request to obtain common video information to the server device 20 based on the corresponding server ID and video information URL (step 303).

[0094] On the other hand, if the viewing position is not included in any of the segment ranges (NO in step 302), the control unit 11 of the terminal device 10 transmits an acquisition request for individual video information to the server device 20 (step 304). The acquisition request for individual video information includes information on the viewing position and information on the viewing direction.

[0095] After transmitting the request to acquire the common or individual video information, the control unit 11 of the terminal device 10 returns to step 301 again.

[0096] [Server device 20: video information generation processing, etc.] Next, the "generation process of common video information," "generation process of individual video information," "multicast process of common video information," "unicast process of individual video information," etc. in server device 20 (management server 20a, distribution server 20b) will be described.

[0097] Fig. 12 is a flowchart showing the generation process of video information in the server device 20. As shown in Fig. 12, the control units 21 and 31 (rendering resources) of the server device 20 (management server 20a and distribution server 20b) determine whether generation of common video information is assigned (step 401).

[0098] When the generation of common video information is assigned (YES in step 401), the control units 21 and 31 of the server device 20 receive a request to acquire the common video information (step 402). Then, the control units 21 and 31 of the server device 20 generate common video information in the corresponding segment 2 from the 3D video corresponding to the entire event venue or the like (step 403).

[0099] This common video information includes color image information and depth information.

[0100] Next, the control units 21 and 31 of the server device 20 encode the common video information (step 404), and transmit the common video information by multicast to each terminal device 10 included in the corresponding group (step 405). Then, the control units 21 and 31 of the server device 20 return to step 401.

[0101] In step 401, if the generation of common video information has not been assigned (NO in step 401), the control units 21, 31 (rendering resources) of the server device 20 (management server 20a, distribution server 20b) determine whether the generation of individual video information has been assigned (step 406).

[0102] When the generation of individual video information is assigned (YES in step 406), the control units 21 and 31 of the server device 20 receive a request to acquire the individual video information (step 407). Then, the control units 21 and 31 of the server device 20 generate individual video information for the corresponding terminal device 10 from the 3D video corresponding to the entire event venue, etc., based on the viewing position and viewing direction included in the request to acquire the individual video information (step 408).

[0103] Next, the control units 21 and 31 of the server device 20 encode the individual video information (step 409), and transmit the individual video information by unicast to the corresponding terminal device 10 (step 410). Then, the control units 21 and 31 of the server device 20 return to step 401.

[0104] [Management server 20a: Lightweight 3D image generation processing, etc.] Next, the "lightweight three-dimensional video generation process", "lightweight three-dimensional video multicast process", and the like in the management server 20a will be described.

[0105] 13 is a flowchart showing the process of generating a lightweight 3D image in the management server 20a. First, the control unit 21 of the management server 20a reduces the weight of a 3D image corresponding to the entire event venue or the like, and generates a lightweight 3D image (step 501). Then, the control unit 21 of the management server 20a transmits the lightweight 3D image to all the terminal devices 10 by multicast (step 502), and returns to step 501.

[0106] Here, the 3D image includes meshes (geometric information) and textures (image information). The control unit 21 of the management server 20a may generate a light 3D image by, for example, reducing the number of meshes or texture resolution in the 3D image.

[0107] When generating a three-dimensional light image, the control unit 21 of the management server 20a may change at least one of the number of meshes or the texture resolution for each object included in the three-dimensional light image.

[0108] For example, an object that is viewed by a large number of users based on information about the viewing position and viewing direction of each terminal device 10 may have a higher mesh count or texture resolution than an object that is viewed by a small number of users.

[0109] Also, for example, dynamic objects may have a higher mesh count and texture resolution than static objects.

[0110] Furthermore, the control unit 21 of the management server 20a may be capable of transmitting the light 3D video on an object-by-object basis for each object included in the light 3D video. In this case, the control unit 21 of the management server 20a may change the frequency of transmitting the light 3D video on an object-by-object basis for each object.

[0111] For example, an object that is viewed by a large number of users based on information on the viewing position and viewing direction in each terminal device 10 may be transmitted more frequently on an object-by-object basis than an object that is viewed by a small number of users.

[0112] Also, for example, dynamic objects may be transmitted more frequently on an object-by-object basis than static objects.

[0113] [Terminal device 10 (grouping): image display processing, etc.] Next, the "display processing of an image based on common video information" and the "display processing of an image based on lightweight three-dimensional video" in the grouped terminal devices 10 will be described.

[0114] 14 is a flowchart showing image display processing and the like in the grouped terminal devices 10. First, the terminal device 10 receives common video information transmitted by multicast to each of the terminal devices 10 included in the corresponding group (step 601).

[0115] Next, the terminal device 10 receives the light 3D video transmitted by multicast to all the terminal devices 10 (step 602). Next, the control unit 11 of the terminal device 10 starts decoding the common video information (step 603).

[0116] Next, the control unit 11 of the terminal device 10 determines whether the decoded common video information is ready (step 604).

[0117] If the decoded common video information is ready (YES in step 604), the control unit 11 of the terminal device 10 proceeds to step 605. In step 605, the control unit 11 of the terminal device 10 renders an image from the decoded common video information (corrects the image to be rendered) based on the viewing position and viewing direction. Then, the control unit 11 of the terminal device 10 displays the rendered image on the screen of the display unit 13 (step 607) and returns to step 601.

[0118] Fig. 16 is a diagram showing a state when an image is rendered from common video information. As shown on the left side of Fig. 16, the common video information is an image with a wider angle than the display angle of view of the terminal device 10. The control unit 11 of the terminal device 10 maps this common video information onto a three-dimensional model (reconstructs three-dimensionally), and projects it according to the required viewing direction (see arrow) and display angle of view to generate a final image.

[0119] Although the viewing direction may change, the control unit 11 of the terminal device 10 can generate an image for the new viewing direction using the same decoded common video information, so that an image can be displayed with low latency when the viewing direction changes.

[0120] Here, the viewing position of the common video information is provisionally set to the center position of segment 2, but the viewing position of each terminal device 10 is not necessarily located at the center position of segment 2. Also, the viewing position may move within segment 2. Therefore, in such a case, it is necessary to change (correct) not only the viewing direction but also the viewing position.

[0121] FIG. 17 is a diagram showing a state when the viewing position is moved to a required viewing position and the viewing direction is changed to a required viewing direction.

[0122] As shown on the left side of Fig. 17, the common video information includes color image information and depth information. The control unit 11 of the terminal device 10 performs 3D reconstruction for each pixel using the depth information of each pixel. Then, the control unit 11 of the terminal device 10 performs projection according to the required viewing position, viewing direction, and display angle of view to generate a final image.

[0123] In addition, the control unit 11 of the terminal device 10 can generate an image for a new viewing position and viewing direction using the same decoded common video information, so that an image can be displayed with low latency when the viewing position and viewing direction are changed.

[0124] Returning to FIG. 14, in step 604, if the decoded common video information is not ready (NO in step 604), the control unit 11 of the terminal device 10 proceeds to step 606.

[0125] Here, for example, it is assumed that the viewing position is significantly changed by the user, and the viewing position moves from the original segment 2 to a position in another segment 2. In this case, for example, the reception of individual video information by unicast may be switched to the reception of common video information. Also, in this case, for example, the reception of a common video in the original segment 2 may be switched to the reception of a common video in another segment 2.

[0126] Immediately after such a change, the decoded common video information may not be ready, and unless some measure is taken in such a case, the displayed image may not be switched smoothly.

[0127] Therefore, when the decoded common video information is not ready (when the viewing position is beyond segment 2), the control unit 11 of the terminal device 10 renders an image from the light 3D video based on the requested viewing position and viewing direction (step 606). Then, the control unit 11 of the terminal device 10 displays the rendered image on the screen of the display unit 13 and returns to step 601.

[0128] In this way, by using lightweight 3D video, when the viewing position is significantly changed and the user moves from the original segment 2 to another segment 2, the displayed image can be switched smoothly.

[0129] [Terminal device 10 (non-grouped): image display processing, etc.] Next, the "display processing of an image based on individual video information" and the "display processing of an image based on lightweight three-dimensional video" in the terminal device 10 that is not grouped will be described.

[0130] 15 is a flowchart showing image display processing and the like in an ungrouped terminal device 10. First, the terminal device 10 receives individual video information transmitted to the terminal device 10 by unicast (step 701). Note that this individual video information differs from common video information in that it is video information that already reflects the viewing position and viewing direction required by the terminal device 10.

[0131] Next, the terminal device 10 receives the light 3D video transmitted by multicast to all the terminal devices 10 (step 702). Next, the control unit 11 of the terminal device 10 starts decoding the individual video information (step 703).

[0132] Next, the control unit 11 of the terminal device 10 determines whether the decoded individual video information is ready (step 604).

[0133] If the decoded individual video information is ready (YES in step 704), the control unit 11 of the terminal device 10 displays the individual video information on the screen of the display unit 13 (step 705), and the process returns to step 701.

[0134] On the other hand, if the decoded common video information is not ready (NO in step 704 ), the control unit 11 of the terminal device 10 proceeds to step 706 .

[0135] Here, for example, it is assumed that the viewing position is significantly changed by the user, and the viewing position moves from the original segment 2 to a position within another segment 2. In this case, for example, reception of common video information by unicast may be switched to reception of individual video information. Immediately after such a switch, the decoded common video information may not be ready.

[0136] Therefore, if the decoded common video information is not available (if the viewing position is beyond segment 2), the control unit 11 of the terminal device 10 renders an image from the lightweight 3D video based on the requested viewing position and viewing direction (step 706). Then, the control unit 11 of the terminal device 10 displays the rendered image on the screen of the display unit 13 (step 707) and returns to step 701.

[0137] In this way, by using lightweight 3D video, when the viewing position is significantly changed and the user moves from the original segment 2 to another segment 2, the displayed image can be switched smoothly.

[0138] <Effect, etc.> As described above, in this embodiment, under specified conditions, the server device 20 performs processing such as grouping terminal devices 10 that are in the same viewing position in a segment 2 based on viewing position information of each terminal device 10 within a viewing area 1 that includes a plurality of segments 2, and transmitting common video information to each of the grouped terminal devices 10 by multicast.

[0139] This reduces the processing load on the server device 20 side and requires a smaller network bandwidth. In addition, for example, rendering on the server side is possible for a large number of terminal devices 10 even in applications where computing resources are limited compared to public clouds such as edge clouds in local 5G networks.

[0140] Furthermore, in this embodiment, the threshold for determining the segments 2 to be grouped is variably controlled, which makes it possible to dynamically change the threshold to an appropriate value.

[0141] Furthermore, in this embodiment, the terminal device 10 side (grouping) can immediately respond to small changes in the viewing position or changes in the viewing direction (see FIGS. 16 and 17).

[0142] In addition, in this embodiment, by using lightweight three-dimensional video, when there is a significant change in the viewing position on the terminal device 10 side that exceeds segment 2, the image can be smoothly displayed at the new viewing position.

[0143] <<Various modified examples>> Next, a specific example of how the information processing system 100 according to this embodiment is used will be described.

[0144] 1. Watching sports at a stadium in real space For example, a user can enjoy the sense of presence of a seat in the stands while freely selecting a viewing position that cannot be seen from the stands to watch live sports. The user may carry or wear the terminal device 10 and be in the stands, or may be in a place other than the stadium.

[0145] 2. Watching eSports tournaments in real space For example, a user can watch a match between top players live from a desired location on the game field. The user may be present on the game field carrying or wearing the terminal device 10, or may be present at a location other than the game field.

[0146] 3. Watching a singer's concert in virtual space For example, a user can watch a singer's concert live from a location of his / her choice in the virtual space, such as from the audience seats, on the stage where the singer is, etc. The user may be located anywhere in the real world.

[0147] 4. Watching a Vtuber concert in virtual space For example, a user can watch a Vtuber concert live from anywhere they like in the virtual space, such as from the audience seats, on the stage where the Vtuber is, etc. The user can be anywhere in the real world.

[0148] 5. Watching a doctor perform surgery in a real operating room For example, a user (e.g., a medical intern) can watch a top doctor's operation live from any position and angle of his choice, essentially from a location other than the operating room.

[0149] 6. Viewing live broadcasts from a virtual studio For example, a user can watch a live broadcast program from any position and angle in a virtual studio, regardless of where the user is in the real world.

[0150] The present technology can also be configured as follows. (1) A control unit that groups terminal devices that are in the same segment and have the same viewing position based on viewing position information of each terminal device in a viewing area that includes a plurality of segments, and transmits common video information to each of the grouped terminal devices by multicast. A server device comprising: (2) The server device according to (1), The control unit determines a segment in which the number of terminal devices exceeds a predetermined threshold as a segment to be grouped. Server device. (3) The server device according to (2), The control unit variably controls the threshold value. Server device. (4) The server device according to (3), The control unit variably controls the threshold value based on a distribution of the number of the terminal devices for each of the segments. Server device. (5) The server device according to (3) or (4), The server device includes a plurality of rendering resources; The control unit variably controls the threshold value based on the number of the rendering resources. Server device. (6) The server device according to (1), the common video information has a wider angle of view than a display angle of a display unit of each of the terminal devices, Each of the grouped terminal devices renders an image to be displayed from the common video information based on a viewing direction and a display angle required by the terminal device itself. Server device. (7) The server device according to (6), Each of the grouped terminal devices renders the image from the common video information based on the viewing position requested by the terminal device itself. (8) The server device according to (7), the common image information includes depth information of an object in the image; Each of the grouped terminal devices renders the image based on the depth information. Server device. (9) The server device according to any one of (1) to (8), The control unit transmits individual video information to each of the terminal devices that are not grouped by unicast. Server device. (10) The server device according to (9), The control unit generates lightweight 3D images by reducing the weight of the 3D images corresponding to all viewing positions within the viewing area, and transmits the lightweight 3D images to all terminal devices by multicast. Server device. (11) The server device according to (10), Each of the terminal devices renders an image to be displayed based on the lightweight three-dimensional video when the viewing position requested by the terminal device moves beyond the segment. Server device. (12) The server device according to (10) or (11), the lightweight 3D image includes a mesh for an object within the lightweight 3D image; The control unit changes the number of meshes in the mesh for each of the objects. Server device. (13) The server device according to any one of (10) to (12), the lightweight 3D image includes textures on objects within the lightweight 3D image; The control unit changes a resolution of the texture for each of the objects. Server device. (14) The server device according to any one of (10) to (13), The control unit is capable of transmitting the lightweight three-dimensional image on an object-by-object basis for each object included in the lightweight three-dimensional image, and is configured to change a frequency of transmitting the lightweight three-dimensional image on an object-by-object basis for each object. Server device. (15) A control unit that groups terminal devices that are in the same segment and have the same viewing position based on viewing position information of each terminal device in a viewing area including a plurality of segments, receives common video information from a server device that transmits the common video information by multicast to each of the grouped terminal devices, and renders an image to be displayed based on the received common video information. A terminal device comprising: (16) A server device that groups terminal devices that are in the same segment and have the same viewing position based on viewing position information of each terminal device in a viewing area including a plurality of segments, and transmits common video information to each of the grouped terminal devices by multicast; receiving the common image information and rendering an image to be displayed based on the received common image information; An information processing system comprising: (17) Grouping terminal devices whose viewing positions are in the same segment based on viewing position information of each terminal device in a viewing area including a plurality of segments; Common video information is transmitted to each of the grouped terminal devices by multicast. Information processing methods. [Explanation of symbols]

[0151] 10...Terminal device 20...Server equipment 20a…Administration server 20b…Distribution server 100...Information processing systems

Claims

1. A control unit that groups terminal devices that are in the same segment and have the same viewing position based on viewing position information of each terminal device in a viewing area including a plurality of segments, and transmits common video information to each of the grouped terminal devices by multicast. A server device comprising:

2. 2. The server device according to claim 1, The control unit determines a segment in which the number of terminal devices exceeds a predetermined threshold as a segment to be grouped. Server device.

3. 3. The server device according to claim 2, The control unit variably controls the threshold value. Server device.

4. 2. The server device according to claim 1, the common video information has a wider angle of view than a display angle of a display unit of each of the terminal devices, Each of the grouped terminal devices renders an image to be displayed from the common video information based on a viewing direction and a display angle required by the terminal device itself. Server device.

5. 5. The server device according to claim 4, Each of the grouped terminal devices renders the image from the common video information based on the viewing position requested by the terminal device itself. Server device.

6. 6. The server device according to claim 5, the common image information includes depth information of an object in the image; Each of the grouped terminal devices renders the image based on the depth information. Server device.

7. 2. The server device according to claim 1, The control unit transmits individual video information to each of the terminal devices that are not grouped by unicast. Server device.

8. 8. The server device according to claim 7, The control unit generates lightweight three-dimensional images by reducing the weight of the three-dimensional images corresponding to all viewing positions within the viewing area, and transmits the lightweight three-dimensional images to all terminal devices by multicast. Server device.

9. 9. The server device according to claim 8, Each of the terminal devices renders an image to be displayed based on the lightweight three-dimensional video when the viewing position requested by the terminal device moves beyond the segment. Server device.

10. 9. The server device according to claim 8, the lightweight 3D image includes a mesh of an object within the lightweight 3D image; The control unit changes the number of meshes in the mesh for each of the objects. Server device.

11. 9. The server device according to claim 8, the lightweight three-dimensional image includes textures on objects within the lightweight three-dimensional image; The control unit changes a resolution of the texture for each of the objects. Server device.

12. 9. The server device according to claim 8, The control unit is capable of transmitting the lightweight three-dimensional image on an object-by-object basis for each object included in the lightweight three-dimensional image, and is configured to change a frequency of transmitting the lightweight three-dimensional image on an object-by-object basis for each object. Server device.

13. a control unit that groups terminal devices that are in the same segment and have the same viewing position based on viewing position information of each terminal device in a viewing area including a plurality of segments, receives common video information from a server device that transmits the common video information by multicast to each of the grouped terminal devices, and renders an image to be displayed based on the received common video information. A terminal device comprising:

14. a server device that groups terminal devices that are in the same segment and have the same viewing position based on viewing position information of each terminal device in a viewing area including a plurality of segments, and transmits common video information to each of the grouped terminal devices by multicast; a terminal device that receives the common image information and renders an image to be displayed based on the received common image information; An information processing system comprising:

15. Based on viewing position information of each terminal device within a viewing area including a plurality of segments, terminal devices whose viewing positions are in the same segment are grouped together; Common video information is transmitted to each of the grouped terminal devices by multicast. Information processing methods.

Images