Distribution control system, distribution control device, distribution control method, and program

Abstract

A delivery control system according to an aspect of the present disclosure comprises a placement unit that places a plurality of virtual viewpoints with respect to an object represented by three-dimensional data constituting stereoscopic video contents, a view block creation unit that divides the sum of view ranges when the object is visually recognized from the plurality of virtual viewpoints, into view blocks as many as the number of the virtual viewpoints, a first block data creation unit that creates, as non-simplified block data for each of the view blocks, three-dimensional data about a part of the object included in the view block, and a delivery unit that delivers one or more non-simplified block data pieces indicating three-dimensional data about the part of the object visually recognizable by the user, to the terminal of the user according to the position of the user on a virtual space where the object is disposed.

Description

[Technical Field]

[0001] This disclosure relates to a distribution control system, a distribution control device, a distribution control method, and a program. [Background technology]

[0002] High-quality volumetric video and holograms, which are stereoscopic video content with six degrees of freedom (6DoF), are well known. In order to distribute such content over a communication network, in addition to the use of advanced data compression technology and communication network / system load balancing technology, a mechanism to control the distribution of the content itself is necessary. In particular, a mechanism that dynamically controls the distribution of content according to the field of view information of the client XR (VR / AR / MR / SR, etc.) device and the user's location information in the virtual space is important.

[0003] Volumetric video is 3D video data composed of 3D data represented by polygon meshes (hereinafter also simply called "mesh"), point clouds, voxels, etc., and textures. It can be displayed and viewed on a screen by rendering it in conjunction with a virtual environment on the client side. On the other hand, volumetric video has a large amount of data and requires a large bandwidth on the communication network for distribution, so there is a need for efficient distribution methods.

[0004] As volumetric video distribution technologies, the technologies described in Non-Patent Documents 1 and 2 are known. Non-Patent Document 1 proposes a method in which volumetric video is rendered on the server side based on the user's head movements detected by the client AR / VR device, and transmitted to the client as 2D data. Non-Patent Document 2 proposes a method to reduce the amount of data required for playback by dynamically changing the Level of Detail of the volumetric video according to the bandwidth of the communication network. Furthermore, Patent Document 1 proposes a technology that creates one-sided 3D data by reducing the amount of data for parts that cannot be seen from each virtual viewpoint placed on the object represented by the 3D data, and then distributes the one-sided 3D data according to the user's position and field of view. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] International Publication No. 2022 / 230186 [Non-patent literature]

[0006] [Non-Patent Document 1] Serhan Gul, Dimitri Podborski, Thomas Buchholz, Thomas Schierl and Cornelius Hellge, "Low-latency cloud-based volumetric video streaming using head motion prediction", NOSSDAV '20: Proceedings of the 30th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video, June 2020, Pages 27-33 [Non-Patent Document 2] Holostream / Arctrus, Internet<URL:https: / / arcturus.studio / holostream> [Overview of the project] [Problems that the invention aims to solve]

[0007] However, the method proposed in Non-Patent Document 1 above requires the server to perform rendering for each user, resulting in a heavy server load. Furthermore, if the number of users increases, the division of server resources may lead to a degradation in the video quality viewed by each user. In addition, it is necessary to send location information from the client to the server at a high frequency and with low latency. For example, keeping the Motion to Photon delay, at which VR sickness begins to occur, below 20ms places a heavy burden on both the communication network and the server.

[0008] On the other hand, the method proposed in Non-Patent Document 2 above, when the available bandwidth of the communication network is narrow, reduces the image quality and level of detail, including the 3D data within the range visible to the user in the volumetric video being viewed (i.e., the 3D data corresponding to the front for the user), resulting in a significant decrease in perceived quality.

[0009] While the method proposed in Patent Document 1 above can solve the problems of Non-Patent Documents 1 and 2 above, increasing the number of virtual viewpoints leads to a large number of single-plane 3D data sets being created, which significantly increases the amount of data required to store the single-plane 3D data on a content server or the like.

[0010] This disclosure is made in view of the above points and provides a technology that reduces the amount of data required for storing and distributing stereoscopic video content while maintaining the user's perceived quality. [Means for solving the problem]

[0011] A distribution control system according to one aspect of the present disclosure includes: an arrangement unit that arranges a plurality of virtual viewpoints centered on an object represented by stereoscopic data constituting stereoscopic video content; a view block creation unit that divides the sum of the field of view ranges when the object is viewed from each of the plurality of virtual viewpoints into a number of view blocks equal to the number of virtual viewpoints; a first block data creation unit that creates unsimplified block data of the portion of the object included in the view block with respect to the object for each view block; and a distribution unit that distributes one or more unsimplified block data representing the portion of the object visible to the user to the user's terminal, according to the user's position in the virtual space where the object is arranged. [Effects of the Invention]

[0012] A technology is provided that reduces the amount of data required for storing and distributing stereoscopic video content while maintaining the user's perceived quality. [Brief explanation of the drawing]

[0013] [Figure 1] This figure shows an example of the overall configuration of the distribution control system according to the first embodiment. [Figure 2] This figure shows an example of the block data creation process according to the first embodiment. [Figure 3] This diagram shows an example of virtual viewpoint placement. [Figure 4] This figure shows an example of a field of view range Ri corresponding to a virtual viewpoint Vi. [Figure 5] This figure shows an example of the field of view range Ri+1 corresponding to the virtual viewpoint Vi+1. [Figure 6] This figure shows an example of the state where the field of view range Ri and the field of view range Ri+1 are superimposed. [Figure 7] This is a diagram showing an example of a visual block Bi. [Figure 8] This figure shows an example of the configuration of the field of view block Bi when α=8. [Figure 9] This figure shows an example of the configuration of the field of view block Bi when α=4. [Figure 10] This is a flowchart showing an example of the distribution process according to the first embodiment. [Figure 11] This is a diagram (part 1) showing an example of a field of view block Bi within the field of view during playback. [Figure 12] This is a diagram (part 2) showing an example of a field of view block Bi within the field of view during playback. [Figure 13] This figure shows an example of the overall configuration of the distribution control system according to the second embodiment. [Figure 14] This figure shows an example of the block data creation process according to the second embodiment. [Figure 15] This is a flowchart showing an example of the distribution process according to the second embodiment. [Figure 16] This figure shows an example of a computer hardware configuration. [Modes for carrying out the invention]

[0014] The first and second embodiments of the present invention will be described below. In each of the following embodiments, a distribution control system 1 will be described that can reduce the amount of data required for storage and distribution while maintaining the user's perceived quality, with volumetric video as an example of stereoscopic video content. Here, volumetric video refers to 3D video data (animation data) composed of 3D data (also called 3D data or stereoscopic data) represented by meshes (or point clouds or voxels, etc.) and textures. For example, the 3D data of the frame at time t is d t Therefore, volumetric video is {d t |t∈[t s ,t e It is expressed as ]}. s This is the start time of the volumetric video, t e This is the end time.

[0015] Although the following explanation focuses on volumetric video, this is merely an example, and the same principles can be applied to any stereoscopic video content with six degrees of freedom, such as holograms.

[0016] [First Embodiment] First, let me describe the first embodiment.

[0017] <Example of the overall configuration of distribution control system 1> An example of the overall configuration of the distribution control system 1 according to the first embodiment will be described with reference to Figure 1. Figure 1 is a diagram showing an example of the overall configuration of the distribution control system 1 according to the first embodiment.

[0018] As shown in Figure 1, the distribution control system 1 according to the first embodiment includes a distribution control server 10, a content server 20, a client 30, and a service site server 40. The distribution control server 10 and the client 30 are connected to each other via a communication network 50 such as the Internet. Similarly, the client 30 and the service site server 40 are connected to each other via the communication network 50.

[0019] The distribution control server 10 and the content server 20 are assumed to be located within the same local network and to be connected to each other within that local network; however, they are not limited to this arrangement, and may, for example, be connected to each other via a communication network 50.

[0020] The distribution control server 10 creates multiple block data from the 3D data constituting the given volumetric video and stores these multiple block data in the content server 20. Here, block data refers to 3D data that represents the elements (constituent elements) of an object when viewed from a certain viewpoint. Block data includes 3D data that represents the constituent elements of the object without simplification (hereinafter also called unsimplified block data) and 3D data in which the constituent elements of the object are simplified to reduce the amount of data (hereinafter also called simplified block data). When viewing an object represented by 3D data constituting a volumetric video from a certain viewpoint, for example, unsimplified block data is distributed for constituent elements that are visible from that viewpoint, and simplified block data is distributed for constituent elements that are not visible. This makes it possible to reduce the amount of data distributed compared to the original 3D data while maintaining the user's perceived quality. The term "object" refers to the subject of a volumetric video, and can mean any object that can be represented using meshes and textures, such as people, animals and plants, structures, buildings, machines, celestial bodies, natural phenomena, etc.

[0021] Furthermore, in response to a viewing request from the client 30, the distribution control server 10 identifies one or more appropriate block data based on the user's viewpoint, spatial position, and field of view (direction of gaze and field of view), and distributes these block data to the client 30. The user's viewpoint and spatial position refer to the user's position within the virtual space where the object is located.

[0022] The content server 20 stores multiple block data. In addition, in response to a data request from the distribution control server 10, the content server 20 returns one or more block data corresponding to the data request to the distribution control server 10.

[0023] Client 30 is a type of terminal used by users to view volumetric video (e.g., XR (VR / AR / MR / SR, etc.) devices), which renders one or more block data distributed from the distribution control server 10 and plays the volumetric video. XR devices include not only HMDs (Head Mount Displays) but also smartphones, tablet devices, wearable devices, etc., that are equipped with application programs that function as XR devices.

[0024] When a user views a volumetric video, the following steps are typically taken: First, the user accesses the service site server 40 via client 30 and obtains a list of content (volumetric videos) that the user can view. Next, the user selects the volumetric video they wish to view from this list and obtains a link to the selected volumetric video. When client 30 accesses this link, a viewing request is sent to the distribution control server 10, and playback of the volumetric video begins when one or more block data points are returned in response to this request.

[0025] Furthermore, the client 30 periodically transmits information such as the user's viewpoint, spatial position, gaze direction, and field of view (hereinafter, information representing the user's viewpoint, spatial position, gaze direction, and field of view is also referred to as "user viewpoint information") to the distribution control server 10. As a result, one or more block data corresponding to the user's viewpoint, spatial position, field of view, etc., are returned from the distribution control server 10 and played back by the client 30.

[0026] The service site server 40 presents a list of content (volume video) that the user can view and provides the client 30 with a link to the content selected from this list.

[0027] Here, the distribution control server 10 according to the first embodiment includes a block data creation unit 101, a distribution control unit 102, and a distribution unit 103. Each of these units is realized, for example, by a process in which one or more programs installed in the distribution control server 10 cause a processor such as a CPU (Central Processing Unit) to execute.

[0028] <000> The block data creation unit 101 creates a plurality of block data from the 3D data constituting the volumetric video given to the distribution control server 10. More specifically, the block data creation unit 101 arranges α virtual viewpoints V i (1 ≤ i ≤ α) at equal intervals (or non-uniform intervals) around the object represented by the 3D data constituting the volumetric video. Then, for each virtual viewpoint V i , based on the difference between the viewing range R i at that virtual viewpoint V i and the viewing range R i+1 at the virtual viewpoint V i+1 , a range called the viewing block B i is determined. Then, the block data creation unit 101 creates, as non-simplified block data VB i (1 ≤ i ≤ α), the range of the object included in the viewing block B i . At the same time, the block data creation unit 101 performs a simplification process (for example, geometry simplification (Level Of Detail reduction process), etc.) on the mesh represented by this non-simplified block data VB i to create simplified block data VB i '. Note that α is a predetermined integer of 2 or more.

[0029] Also, the block data creation unit 101 stores the block data (that is, the non-simplified block data set {VB i |1 ≤ i ≤ α} and the simplified block data set {VB i '|1 ≤ i ≤ α}) in the content server 20.

[0030] When the distribution control unit 102 receives user viewpoint information from the client 30, it determines one or more appropriate block data to be distributed based on this user viewpoint information. More specifically, the distribution control unit 102 identifies the virtual viewpoint closest to the viewpoint or spatial position and line of sight included in the user viewpoint information, and then identifies the field of view block B included within the field of view of this virtual viewpoint. i The distribution control unit 102 then identifies the identified field of view block B. i Corresponding unsimplified block data VB i And other than the field of view block B i Simplified block data corresponding to VB i We will decide on the target audience for distribution.

[0031] The distribution unit 103 transmits the block data that has been determined to be distributed by the distribution control unit 102 to the client 30.

[0032] Furthermore, the content server 20 according to the first embodiment has a block data storage unit 201. The block data storage unit 201 is implemented by an auxiliary storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0033] The block data storage unit 201 stores each block data created by the block data creation unit 101 (unsimplified block data set {VB i |1≦i≦α} and simplified block data set {VB i Remember '|1≦i≦α}).

[0034] Note that the overall configuration of the distribution control system 1 shown in Figure 1 is just one example, and other configurations are also possible. For example, the distribution control server 10 and the content server 20 may be configured as a single server.

[0035] <Details of various processes> The following describes the block data creation process for creating multiple block data files, and the distribution process for delivering one or more appropriate block data files to client 30. Note that the block data creation process is a pre-processing step that is executed before the distribution process.

[0036] <<Block data creation process>> First, the block data creation process according to the first embodiment will be explained with reference to Figure 2. Figure 2 is a diagram showing an example of the block data creation process according to the first embodiment. In the following, it is assumed that volumetric video is provided to the distribution control server 10.

[0037] The block data creation unit 101 obtains 3D data for one frame from the 3D data constituting the given volumetric video (step S101). Hereafter, the block data creation unit 101 obtains 3D data for a certain time t ∈ [t s ,t e 3D data of the frame d t It is assumed that it has been obtained.

[0038] Note that 3D data consists of a mesh and a texture, but since no special processing is performed on the texture below, in step S101 above, it is acceptable to obtain only the mesh of the 3D data.

[0039] Next, the block data creation unit 101 generates the 3D data d acquired in step S101 above. t Step S102 places α virtual viewpoints around the object represented by . However, the line of sight direction of each virtual viewpoint is the object. For example, the block data creation unit 101 places α virtual viewpoints with the object as their line of sight at equal intervals (or unequal intervals) on the circumference of a circle of a predetermined radius centered on the object. Note that α is a predetermined integer of 2 or more.

[0040] Here, Figure 3 shows an example of the arrangement of virtual viewpoints when α=8. In the example shown in Figure 3, virtual viewpoints V1 to V8 are arranged at equal intervals on the circumference of a circle of radius R, with the object O as the center and the object O as the line of sight. Furthermore, each virtual viewpoint is defined as having an angle relative to a certain direction. In the example shown in Figure 3, θ1=0 is defined as the direction from the object O to virtual viewpoint V1, and θ i This is expressed as =(i-1)×2π / α.

[0041] Note that placing a virtual viewpoint on the circumference of a circle is just one example and is not limited to this; for example, a virtual viewpoint may be placed on the edges or vertices of a polygon centered on the object. Also, in the example shown in Figure 3, for simplicity, virtual viewpoints V1 to V8 are placed on the xy-plane of the xyz space with the center of the object O as the origin, but this is not limited to this; for example, a virtual viewpoint may be placed on a sphere (or polyhedron) centered on the said origin.

[0042] Next, the block data creation unit 101 generates the virtual viewpoint V that was positioned in step S102 above. i Based on (1≦i≦α), α unsimplified block data VB i (1≦i≦α) is created (step S103). At this time, the block data creation unit 101 creates α unsimplified block data VB by, for example, the following procedure 1-1 to 1-3. i Create.

[0043] Step 1-1: First, the block data creation unit 101 creates each virtual viewpoint V i View range R i ={θ|θ i -π / 2 < θ < θ i Calculate {+π / 2}, where this field of view range R i The definition is just one example, and the field of view R i The definition is not limited to this.

[0044] For example, virtual viewpoint V i View range R i This is shown in Figure 4. Similarly, virtual viewpoint V i+1 View range Ri+1 This is shown in Figure 5.

[0045] Step 1-2: Next, the block data creation unit 101 generates virtual viewpoint V i Each time, the virtual viewpoint V i View range R i And, virtual viewpoint V i+1 View range R i+1 Using and, the field of view R i From the field of view R i+1 The area excluding is in view block B i It is calculated as follows: That is, the block data creation unit 101 is B i =R i -R i+1 By visual block B i Calculate V. However, when i=α, V α+1 =V1, R α+1 Let =R1.

[0046] As an example, the field of view R shown in Figure 4 i From the field of view R shown in Figure 5 i+1 Figure 6 shows the stacked state. i You can obtain this.

[0047] By following steps 1-2 above, the field of view block set {B i We obtain |1≦i≦α}. As an example, consider each field of view block B when α=8. i Figure 8 shows this. Similarly, as an example, each field of view block B when α=4. i This is shown in Figure 9.

[0048] Steps 1-3: Then, the block data creation unit 101 creates the field of view block B i Each time, that field of view block B i The parts of the object included in the range represented by are extracted as constituent elements, and the 3D data representing the extracted constituent elements is converted into unsimplified block data VB. i This will create a non-simplified block data set {VB i We obtain the value |1≦i≦α}.

[0049] Alternatively, instead of steps 1-1 to 1-2 above, follow steps 2-1 to 2-2 to obtain the field of view block set {B i You may also calculate |1≦i≦α}.

[0050] Step 2-1: First, the block data creation unit 101 calculates the normal vector of the object surface represented by the 3D data.

[0051] Step 2-2: Next, the block data creation unit 101 passes through the center of the object and the field of view block B i The vector e has an angle that bisects it. i In this case, the normals and vector e calculated in step 2-1 above i The range in which the angle with is less than or equal to π / (2α) is the field of view block set B. i This is calculated as follows: The field of view block set {B i |1≦i≦α} is obtained for each of these field of view blocks B i This is all virtual viewpoint V i Field of view R in (1≦i≦α) i It can also be said that this is the sum of divided into α parts. Note that this field of view block set {B i The value |1≦i≦α} is used in the distribution process described later, and is therefore stored, for example, in auxiliary storage.

[0052] Following step S103 above, the block data creation unit 101 generates the unsimplified block data set {VB i Using |1≦i≦α}, the simplified block data set {VB i Create '|1≦i≦α}' (step S104). That is, the block data creation unit 101 creates unsimplified block data VB i Simplification is performed on the mesh represented by to create simplified block data VB i Creates '. This allows unsimplified block data VB i Simplified block data VB with reduced data size i A simplified block data set composed of ' {VB i'{1≦i≦α} is obtained. For the simplification process, existing processes such as geometry simplification may be used.

[0053] Next, for the 3D data d of the frame at time t t The non-simplified block data set obtained from is denoted as V t and the simplified block data set is denoted as V t '.

[0054] Next, the block data creation unit 101 determines whether there is a next frame in the given volumetric video (step S105).

[0055] If it is determined in step S105 above that there is a next frame, the block data creation unit 101 returns to step S101 above, acquires the 3D data of the next frame (step S101), and then executes the processes after step S102. That is, in this case, the block data creation unit 101 sets t←t + 1 and executes the processes after step S101 above.

[0056] On the other hand, if it is not determined in step S105 above that there is a next frame (that is, when the time t = t e ), the block data creation unit 101, for each time t∈[t s , t e , saves the non-simplified block data set V t and the simplified block data set V t ' in the block data storage unit 201 of the content server 20 (step S106). Since 3D data with reduced data volume while maintaining the user's perceived quality can be constructed from these block data, it is possible to suppress an increase in the data volume required for data storage even when the virtual view point number α increases, as compared with the case of storing the single-sided 3D data described in Patent Document 1.

[0057] In the above block data creation process, the number of virtual viewpoints α is common for all frames, but it may be different for each frame. Also, in the above block data creation process, steps S101 to S104 are repeatedly executed for each frame. However, for example, within a certain time range (including the case where the time range is t e -t s ), if the 3D data d t is the same between frames, the above steps S101 to S104 may be executed only for the 3D data d t of one frame included in the time range.

[0058] <<Distribution Process>> Next, the distribution process according to the first embodiment will be described while referring to FIG. 10. FIG. 10 is a flowchart showing an example of the distribution process according to the first embodiment. Note that steps S201 to S202 in FIG. 10 are executed each time user viewpoint information is received, and steps S203 to S205 are executed for each frame interval. However, hereinafter, it is assumed that at least a part of the object is included within the user's visual field range.

[0059] The distribution control unit 102 specifies a virtual viewpoint corresponding to the user viewpoint information by using the viewpoint or spatial position and the viewing direction included in the user viewpoint information received from the client 30 (step S201). For example, the distribution control unit 102 may calculate the angle of the viewing direction by using the viewpoint or spatial position and the viewing direction, and specify the virtual viewpoint having the angle closest to this angle. Note that the same reference as when the virtual viewpoints are arranged is used as the reference for the angle.

[0060] However, the above method for specifying the virtual viewpoint is an example and is not limited thereto. For example, similar to Patent Document 1, a correspondence table between virtual viewpoints and angle ranges may be created, and the virtual viewpoint may be specified from this correspondence table.

[0061] Next, the distribution control unit 102 identifies one or more view blocks within the field of view of the virtual viewpoint identified in step S201 (step S202). That is, the virtual viewpoint identified in step S201 is V i In this case, the distribution control unit 102 will set the field of view block set {B i Using |1≦i≦α}, the virtual viewpoint V i S field of view i one or more line of sight blocks B i Identify the field of view S. i R i Similarly, {θ|θ i -π / 2 < θ < θ i You could also use} +π / 2}, or R i A wider range (for example, {θ|θ i -3π / 4 < θ < θ i You can also use `+3π / 4`.

[0062] For example, S i ={θ|θ i -π / 2 < θ < θ i The field of view S when it is {+π / 2} i Visibility block B inside i and field of view S i Outside view block B i This is shown in Figure 11. In the example shown in Figure 11, the field of view blocks B1 to B4 are within the field of view S. i Within this, the field of view blocks B5~B8 have a field of view of S i It is outside.

[0063] Another example is S i ={θ|θ i -3π / 4 < θ < θ i The field of view S when it is +3π / 4} i Visibility block B inside i and field of view S i Outside view block B i This is shown in Figure 12. In the example shown in Figure 12, the field of view blocks B1 to B5 and B8 are within the field of view S. i Within this, the field of view blocks B6~B7 have a field of view of S i It is outside.

[0064] Furthermore, as will be explained later, the field of view S i Visibility block B inside i Corresponding unsimplified block data VB i And, field of view S i Outside view block B i Simplified block data corresponding to VB i 'and is delivered to client 30, so the field of view S i Making the field of view S wider can help prevent a decrease in perceived quality in response to sudden changes in the user's position. i A wider range increases the amount of data used when streaming volumetric video.

[0065] The distribution control unit 102 determines the block data to be distributed from one or more view blocks identified in step S202 (step S203). That is, the distribution control unit 102 determines the block data to be distributed from each view block B identified in step S202. i (In other words, field of view S) i Each of the field of view blocks B within i ) corresponding unsimplified block data VB i And each of the other view blocks B identified in step S202 above. i (In other words, field of view S) i Each of the outdoor field of view blocks B i ) corresponding simplified block data VB i ' and are selected as the target for distribution. In this way, among the target objects, the field of view S i Regarding the internal parts (components), use unsimplified block data VB. i It was decided that it would be included in the distribution, with a field of view of S i For the external parts (components), use simplified block data VB. i This determines that the content will be distributed. This makes it possible to reduce the amount of data used when distributing volumetric video.

[0066] Next, the distribution unit 103 obtains the block data for the frame corresponding to the time from the content server 20, from among the block data determined to be distributed in step S203 above (step S204).

[0067] The distribution unit 103 then transmits the block data acquired in step S204 to the client 30 (step S205). As a result, this block data is integrated and rendered on the client 30, and the object is displayed on the display (i.e., volumetric video is played).

[0068] Note that in the above distribution process, the field of view S i For the external parts (components), use simplified block data VB. i ' was sent, but for example, field of view S i Data may not be transmitted for external components. This will further reduce the amount of data used during volumetric video streaming.

[0069] [Second Embodiment] Next, a second embodiment will be described. In the second embodiment, the degree of simplification is multiplied when simplifying block data to create multiple sets of simplified block data with different degrees of simplification. In Patent Document 1, when the degree of simplification of single-face 3D data is multiplied, the amount of data required for data storage increases significantly, but in the second embodiment, this increase in data volume can be suppressed compared to Patent Document 1.

[0070] In the second embodiment, the differences from the first embodiment will be explained, and explanations of parts that can be the same as in the first embodiment will be omitted.

[0071] <Example of the overall configuration of distribution control system 1> An example of the overall configuration of the distribution control system 1 according to the second embodiment will be described with reference to Figure 13. Figure 13 is a diagram showing an example of the overall configuration of the distribution control system 1 according to the second embodiment.

[0072] As shown in Figure 13, in the distribution control system 1 according to the second embodiment, the client 30 transmits the buffer amount to the distribution control server 10 in addition to the user's viewpoint information. The buffer amount is the memory area used for buffering the block data received from the distribution control server 10. The buffer amount may be transmitted to the distribution control server 10 independently of the user's viewpoint information, or it may be transmitted to the distribution control server 10 together with the user's viewpoint information.

[0073] Furthermore, in the distribution control system 1 according to the second embodiment, the block data creation unit 101 of the distribution control server 10 creates multiple simplified block data sets for each granularity d (1 ≤ d ≤ D) using a predetermined granularity D (where D is an integer of 2 or more). Granularity refers to the degree of simplification.

[0074] <Details of various processes> The following describes the block data creation process for creating multiple block data files, and the distribution process for delivering one or more appropriate block data files to client 30. Note that the block data creation process is a pre-processing step that is executed before the distribution process.

[0075] <<Block data creation process>> First, the block data creation process according to the second embodiment will be explained with reference to Figure 14. Figure 14 is a diagram showing an example of the block data creation process according to the second embodiment. Note that steps S301 to S303 and S305 in Figure 14 may be the same as steps S101 to S103 and S105 in Figure 2, respectively, so steps S304 and S306 will be explained below.

[0076] Following step S303, the block data creation unit 101 generates the unsimplified block data set {VB} obtained in step S303. iUsing |1≦i≦α}, a simplified block data set is created for each granularity d (1≦d≦D) (step S304). That is, for example, block data VB at granularity d i A simplified version of this is a simplified block data VB. i ' (d) Therefore, the block data creation unit 101 generates D simplified block data sets {VB i ' (d) Create the range |1≦i≦α}(1≦d≦D). Note that the larger the granularity, the simpler the result and the less data it will contain.

[0077] Let's take a concrete example. For instance, if we set D=2, and define the granularity when d=1 as "medium quality" and the granularity when d=2 as "low quality", then {VB i ' (1) |1≦i≦α} is a set of medium-quality simplified block data, {VB i ' (2) |1≦i≦α} is a low-quality simplified block data set. On the other hand, the unsimplified block data set {VB i |1≦i≦α} represents a high-quality block data set.

[0078] Below is the 3D data d of the frame at time t. t The unsimplified block data set obtained from V t A simplified block data set of granularity d is V t ' (d) We will express it as follows.

[0079] If it is not determined in step S305 that there is no next frame, the block data creation unit 101 calculates each time t∈[t s ,t e Unsimplified block data set V obtained from ] t and the simplified block data set V t ' (d) (1≦d≦D) is stored in the block data storage unit 201 of the content server 20 (step S306). Thus, in the second embodiment, the unsimplified block data set V tAnd V is a set of multiple simplified block data, each simplified at multiple levels of granularity. t ' (d) This is saved to content server 20.

[0080] ≪Distribution Processing≫ Next, the distribution process according to the second embodiment will be described with reference to Figure 15. Figure 15 is a flowchart showing an example of the distribution process according to the second embodiment. Note that steps S401 to S402 and steps S404 to S405 in Figure 15 may be the same as steps S201 to S202 and steps S204 to S205 in Figure 10, respectively, so step S403 will be described below.

[0081] The distribution control unit 102 identifies one or more field of view blocks (i.e., field of view range S) identified in step S402. i Based on one or more view blocks within the system, the buffer amount received from client 30, and the line speed between client 30, the system determines the block data to be distributed (step S403). For example, the distribution control unit 102 can determine the block data to be distributed depending on which of the following conditions 1 to 3 is met. In the following example, D=2 and the unsimplified block data set {VB i |1≦i≦α} and a medium-quality simplified block data set {VB i ' (1) |1≦i≦α} and a low-quality simplified block data set {VB i ' (2) Assume that the condition |1≦i≦α} is obtained.

[0082] Condition 1: Buffer size is above a predetermined threshold, AND line speed is above a predetermined threshold. In this case, the distribution control unit 102 controls the field of view S i Each of the field of view blocks B within i Corresponding unsimplified block data VB i And, field of view S i Each of the outdoor field of view blocks B iSimplified block data corresponding to each in VB i ' (2) We will decide which of these will be the target audiences for distribution.

[0083] Condition 2: Buffer size is below a predetermined threshold, or line speed is below a predetermined threshold. In this case, the distribution control unit 102 controls the field of view S i Each of the field of view blocks B within i Simplified block data corresponding to each in VB i ' (1) We will decide which ones will be included in the distribution target.

[0084] Condition 3: The time t of the frame to be delivered is below a predetermined threshold (i.e., immediately after playback of volumetric video begins). In this case, the distribution control unit 102, as in the case where condition 2 is met, controls the field of view S i Each of the field of view blocks B within i Simplified block data corresponding to each in VB i ' (1) We will decide which ones will be included in the distribution target.

[0085] This allows for playback at medium quality within the user's field of view when, for example, the buffer size is small, the connection speed is slow, or immediately after volumetric video playback begins. Conversely, when the buffer size and connection speed are sufficient (not immediately after playback begins), playback at high quality within the user's field of view becomes possible. Therefore, it is possible to dynamically control the user's perceived quality according to the environment in which the client 30 is located. However, conditions 1 to 3 above are just examples, and the block data to be delivered when each condition is met is also just an example. Needless to say, various other conditions can also be set.

[0086] [Example Hardware Configuration] Finally, examples of hardware configurations for the distribution control server 10 and content server 20 according to the first and second embodiments will be described. The distribution control server 10 and content server 20 according to the first and second embodiments can be implemented, for example, by the hardware configuration of the computer 500 shown in Figure 16. The client 30 and service site server 40 may also be implemented with a similar hardware configuration.

[0087] The computer 500 shown in Figure 16 includes an input device 501, a display device 502, an external interface 503, a communication interface 504, a processor 505, and a memory device 506. Each of these hardware components is connected to the others via a bus 507 for communication.

[0088] The input device 501 is, for example, a keyboard, mouse, or touch panel. The display device 502 is, for example, a display. Note that the computer 500 does not necessarily have to have at least one of the input device 501 and the display device 502.

[0089] External I / F 503 is an interface to external devices such as recording media 503a. Examples of recording media 503a include CD (Compact Disc), DVD (Digital Versatile Disk), SD memory card (Secure Digital memory card), and USB (Universal Serial Bus) memory card.

[0090] The communication interface 504 is an interface for data communication with other devices, equipment, systems, etc. The processor 505 is, for example, a CPU or other type of arithmetic unit. The memory device 506 is, for example, a storage device such as an HDD, SSD, RAM (Random Access Memory), ROM (Read Only Memory), or flash memory.

[0091] The distribution control server 10 and content server 20 according to the first and second embodiments can realize the various processes described above by having the hardware configuration of the computer 500 shown in Figure 16. Note that the hardware configuration of the computer 500 shown in Figure 16 is just an example, and the computer 500 may have other hardware configurations. For example, the computer 500 may have multiple processors 505 or multiple memory devices 506.

[0092] [summary] As described above, the distribution control system 1 according to the first embodiment places multiple virtual viewpoints on an object represented by 3D data constituting stereoscopic video content, divides the 3D data into multiple block data (unsimplified block data) based on these virtual viewpoints, and creates multiple simplified block data by simplifying the mesh geometry, etc., represented by those block data. As a result, for example, for components of the object that the user cannot see, simplified block data is distributed to the client 30, making it possible to reduce the amount of data distributed while maintaining the user's perceived quality. Furthermore, since the client 30 can render the object by integrating and rendering the unsimplified block data and simplified block data, compared to Patent Document 1, it is possible to suppress the increase in the amount of data required when the content server 20 saves block data, even when the number of virtual viewpoints increases.

[0093] In addition, the distribution control system 1 according to the second embodiment can multiplex the degree of simplification of the simplified block data. This makes it possible to achieve flexible distribution control according to the environment in which the client 30 is located, for example. Furthermore, compared to Patent Document 1, even when the degree of simplification is multiplexed (and even when the number of multiplexing levels increases), it is possible to suppress the increase in the amount of data required when the content server 20 stores the block data.

[0094] Therefore, according to the distribution control system 1 of the first or second embodiment described above, it is possible to achieve volumetric video distribution that simultaneously suppresses server load and maintains the user's perceived quality.

[0095] The present invention is not limited to the embodiments specifically disclosed above, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims. [Explanation of Symbols]

[0096] 1. Distribution control system 10 Distribution control server 20 Content Servers 30 clients 40 Service Site Servers 50 Communication Networks 101 Block Data Creation Section 102 Distribution Control Unit 103 Distribution Department 201 Block Data Storage Unit 500 Computers 501 Input device 502 Display device 503 External I / F 503a Recording medium 504 Communication I / F 505 Processor 506 Memory Device 507 Bus

Claims

1. A configuration unit that arranges multiple virtual viewpoints centered on an object represented by 3D data constituting 3D video content, A field of view block creation unit divides the sum of the field of view ranges when viewing the object from each of the aforementioned multiple virtual viewpoints into a number of field of view blocks equal to the number of virtual viewpoints. For each field of view block, a first block data creation unit creates three-dimensional data of the portion of the object included in the field of view block as unsimplified block data, A distribution unit delivers one or more unsimplified block data representing the three-dimensional data of a portion of the object visible to the user to the user's terminal, depending on the user's position in the virtual space where the object is placed. It has, The aforementioned visibility block creation unit is A distribution control system that creates the i-th (1 ≤ i ≤ α) field of view block Bi as the range obtained by subtracting the field of view range Ri+1 from the field of view range Ri.

2. A configuration unit that arranges a plurality of virtual viewpoints centered on an object represented by 3D data constituting a 3D video content, A field of view block creation unit divides the sum of the field of view ranges when viewing the object from each of the aforementioned multiple virtual viewpoints into a number of field of view blocks equal to the number of virtual viewpoints. For each field of view block, a first block data creation unit creates three-dimensional data of the portion of the object included in the field of view block as unsimplified block data, A distribution unit delivers one or more unsimplified block data representing the three-dimensional data of a portion of the object visible to the user to the user's terminal, depending on the user's position in the virtual space where the object is placed. It has, The aforementioned visibility block creation unit is Let α be the number of virtual viewpoints, and B be the i-th (1 ≤ i ≤ α) field of view block. ｉ , passing through the center of the object, and the field of view block B ｉ The vector e has an angle that bisects the angle. ｉ In this case, the normal to the polygon mesh representing the surface of the object and the vector e ｉ The range in which the angle formed with is π / (2α) or less is the field of view block B. ｉ A distribution control system that calculates as follows.

3. It includes a second block data creation unit that creates simplified block data by performing a simplification process on the aforementioned unsimplified block data, The aforementioned distribution unit, The distribution control system according to claim 1 or 2, which distributes to the user's terminal one or more unsimplified block data representing three-dimensional data of parts of the object that are visible to the user, and one or more simplified block data representing three-dimensional data of parts of the object that are not visible to the user.

4. The second block data creation unit described above is: The distribution control system according to claim 3, which creates simplified block data by performing a simplification process at a predetermined granularity for each predetermined granularity.

5. A configuration unit that arranges multiple virtual viewpoints centered on an object represented by 3D data constituting 3D video content, A field of view block creation unit divides the sum of the field of view ranges when viewing the object from each of the aforementioned multiple virtual viewpoints into a number of field of view blocks equal to the number of virtual viewpoints. For each field of view block, a first block data creation unit creates three-dimensional data of the portion of the object included in the field of view block as unsimplified block data, A distribution unit delivers one or more unsimplified block data representing the three-dimensional data of a portion of the object visible to the user to the user's terminal, depending on the user's position in the virtual space where the object is placed. It has, The aforementioned visibility block creation unit is A distribution control device that creates the i-th (1 ≤ i ≤ α) field of view block Bi, where α is the number of virtual viewpoints, Vi is the i-th (1 ≤ i ≤ α) virtual viewpoint, and Ri is the field of view when viewing the object from the virtual viewpoint Vi. The range obtained by subtracting the field of view range Ri + 1 from the field of view range Ri + 1 is then created.

6. A configuration unit that arranges a plurality of virtual viewpoints centered on an object represented by 3D data constituting a 3D video content, A field of view block creation unit divides the sum of the field of view ranges when viewing the object from each of the aforementioned multiple virtual viewpoints into a number of field of view blocks equal to the number of virtual viewpoints. For each field of view block, a first block data creation unit creates three-dimensional data of the portion of the object included in the field of view block as unsimplified block data, A distribution unit delivers one or more unsimplified block data representing the three-dimensional data of a portion of the object visible to the user to the user's terminal, depending on the user's position in the virtual space where the object is placed. It has, The aforementioned visibility block creation unit is A distribution control device that calculates the range in which the angle between the normal to the polygon mesh representing the surface of the object and the vector e i is π / (2α) or less, where α is the number of virtual viewpoints, Bi is the i (1 ≤ i ≤ α)th field of view block, and e i is a vector that passes through the center of the object and has an angle that bisects the field of view block Bi, and defines the field of view block Bi as the field of view block Bi.

7. A configuration procedure for arranging multiple virtual viewpoints centered on an object represented by 3D data constituting a 3D video content, A procedure for creating view blocks, which divides the sum of the field of view ranges when viewing the object from each of the aforementioned multiple virtual viewpoints into a number of view blocks equal to the number of virtual viewpoints, A first block data creation procedure for each field of view, which creates 3D data of the portion of the object included in the field of view as unsimplified block data, A distribution procedure for delivering one or more unsimplified block data representing three-dimensional data of parts of the object visible to the user to the user's terminal, according to the user's position in the virtual space where the object is placed. The computer executes this, The aforementioned procedure for creating a field block is as follows: A distribution control method in which, when the number of virtual viewpoints is α, the i-th (1 ≤ i ≤ α)th virtual viewpoint is Vi, and the field of view when viewing the object from the virtual viewpoint Vi is Ri, the range obtained by subtracting the field of view Ri + 1 from the field of view Ri is created as the i-th (1 ≤ i ≤ α)th field of view block Bi.

8. A procedure for arranging multiple virtual viewpoints centered on an object represented by stereoscopic data constituting stereoscopic video content, A procedure for creating view blocks, which divides the sum of the field of view ranges when viewing the object from each of the aforementioned multiple virtual viewpoints into a number of view blocks equal to the number of virtual viewpoints, A first block data creation procedure for each field of view, which creates 3D data of the portion of the object included in the field of view as unsimplified block data, A distribution procedure for delivering one or more unsimplified block data representing three-dimensional data of parts of the object visible to the user to the user's terminal, according to the user's position in the virtual space where the object is placed. The computer executes this, The aforementioned procedure for creating a field block is as follows: A distribution control method in which, when the number of virtual viewpoints is α, the i-th (1 ≤ i ≤ α)-th field of view block is Bi, and e i is a vector that passes through the center of the object and has an angle that bisects the field of view block Bi, the range in which the angle between the normal to the polygon mesh representing the surface of the object and the vector e i is π / (2α) or less is calculated as the field of view block Bi.

9. A configuration procedure for arranging multiple virtual viewpoints centered on an object represented by 3D data constituting a 3D video content, A procedure for creating view blocks, which divides the sum of the field of view ranges when viewing the object from each of the aforementioned multiple virtual viewpoints into a number of view blocks equal to the number of virtual viewpoints, A first block data creation procedure for each field of view, which creates 3D data of the portion of the object included in the field of view as unsimplified block data, A distribution procedure for delivering one or more unsimplified block data representing three-dimensional data of parts of the object visible to the user to the user's terminal, according to the user's position in the virtual space where the object is placed. Have the computer run it, The aforementioned procedure for creating a field block is as follows: A program that creates the i-th (1 ≤ i ≤ α) field of view block Bi, where α is the number of virtual viewpoints, Vi is the i-th (1 ≤ i ≤ α) virtual viewpoint, and Ri is the field of view when viewing the object from the virtual viewpoint Vi. The field of view block Bi is the i-th (1 ≤ i ≤ α) field of view block, obtained by subtracting the field of view range Ri + 1 from the field of view range Ri.

10. A procedure for arranging multiple virtual viewpoints centered on an object represented by stereoscopic data constituting stereoscopic video content, A procedure for creating view blocks, which divides the sum of the field of view ranges when viewing the object from each of the aforementioned multiple virtual viewpoints into a number of view blocks equal to the number of virtual viewpoints, A first block data creation procedure for each field of view, which creates 3D data of the portion of the object included in the field of view as unsimplified block data, A distribution procedure for delivering one or more unsimplified block data representing three-dimensional data of parts of the object visible to the user to the user's terminal, according to the user's position in the virtual space where the object is placed. Have the computer run it, The aforementioned procedure for creating a field block is as follows: A program that calculates the range in which the angle between the normal to the polygon mesh representing the surface of the object and the vector e i is less than or equal to π / (2α), where α is the number of virtual viewpoints, Bi is the i-th (1 ≤ i ≤ α)-th field of view block, and e i is a vector that passes through the center of the object and has an angle that bisects the field of view block Bi, and defines the field of view block Bi as the range in which the angle between the normal to the polygon mesh representing the surface of the object and the vector e i is less than or equal to π / (2α).

Images