Video transmission system, terminal device, and video transmission method

Abstract

To enable distribution of suitable video according to the lines of sight of a plurality of users, this video transmission system comprises a transmission unit that transmits second point group data extracted from first point group data representing a first region in three-dimensional space, a reception unit that receives the second point group data in accordance with line-of-sight information pertaining to the lines of sight of the users, and an output unit that outputs an image displayed to the users on the basis of the second point group data.

Description

[Technical field]

[0001] The present invention relates to a video transmission system, a terminal device, and a video transmission method. [Background technology]

[0002] Conventionally, point cloud data has been used as one type of data for expressing the three-dimensional shape of an object on a computer screen, a television screen, etc. Related technologies include the inventions disclosed in the following Patent Documents 1 and 2.

[0003] Patent Document 1 relates to a technique for transmitting point cloud data via a communication network. In the point cloud data communication system disclosed in Patent Document 1, a display request including at least one of a viewpoint and a line of sight direction specified by a user as a specification for a display state of a three-dimensional shape is transmitted to a transmitting device. The transmitting device extracts point cloud data of a portion that should be prioritized when displaying a three-dimensional shape in a three-dimensional space from the point cloud data stored in a point cloud data storage unit according to at least one of the viewpoint and the line of sight direction included in the display request.

[0004] Patent Document 2 relates to a technology for improving the efficiency of data processing using point clouds. In the data processing device disclosed in Patent Document 2, a visibility determination unit determines which of multiple blocks in a shooting space are visible to a user according to the viewpoint position, posture, and line of sight of the user (or virtual camera). The visibility determination unit then identifies the multiple blocks visible to the user as blocks to be rendered. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2020-136882 [Patent Document 2] Japanese Patent Application Publication No. 2018-032112 Summary of the Invention [Problem to be solved by the invention]

[0006] In the point cloud data communication system disclosed in Patent Document 1, when a transmitting device transmits data to multiple user terminals by multicast or broadcast distribution, it is not possible to deliver images suitable for each of the multiple users because the line of sight differs depending on the user.

[0007] In addition, in the data processing device disclosed in Patent Document 2, a visibility determination unit identifies multiple blocks visible to a user as blocks to be rendered. However, as in Patent Document 1, when a transmitting device transmits data to multiple user terminals by multicast distribution or broadcast distribution, it is not possible to distribute video suitable for each of the multiple users because the line of sight differs depending on the user.

[0008] One aspect of the present invention has been made in consideration of the above problems, and has an object to provide a technique capable of delivering suitable video according to the line of sight of multiple users. [Means for solving the problem]

[0009] A video transmission system according to one embodiment of the present invention includes a transmitting means for transmitting second point cloud data extracted from first point cloud data representing a first region in three-dimensional space, a receiving means for receiving the second point cloud data in response to gaze information regarding a user's gaze, and an output means for outputting an image to be displayed to the user based on the second point cloud data.

[0010] A terminal device according to one embodiment of the present invention includes a gaze information acquisition means for acquiring gaze information indicating a user's gaze, a receiving means for receiving second point cloud data extracted from first point cloud data representing a first area in three-dimensional space in response to the gaze information, and an output means for outputting an image to be displayed to the user based on the second point cloud data received by the receiving means.

[0011] A video transmission method according to one embodiment of the present invention transmits second point cloud data extracted from first point cloud data representing a first region in three-dimensional space, receives the second point cloud data according to line-of-sight information regarding a user's line of sight, and outputs an image to be displayed to the user based on the second point cloud data. Effect of the Invention

[0012] According to one aspect of the present invention, it is possible to deliver a suitable video according to the line of sight of a plurality of users. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a functional configuration of a video transmission system according to a first exemplary embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of a first region in a three-dimensional space. [Diagram 3] 1A to 1C are diagrams for explaining an example of a method for extracting point cloud data. [Figure 4] FIG. 13 is a diagram showing a specific example of a case where 1 / 2 point cloud data is extracted from the first point cloud data. [Diagram 5] FIG. 1 is a flow diagram showing the flow of a video transmission method. [Figure 6] FIG. 11 is a block diagram showing a functional configuration of a video transmission system according to an exemplary embodiment 2 of the present invention. [Figure 7] 10 is a diagram for explaining a method for receiving point cloud data by a receiving unit. FIG. [Figure 8] FIG. 11 is a diagram for explaining a case where a plurality of users receive point cloud data. [Figure 9] FIG. 11 is a diagram for explaining an evaluation function relating to viewing quality. [Figure 10] FIG. 11 is a sequence diagram showing the flow of a video transmission method by a video transmission system according to an exemplary embodiment 2 of the present invention. [Figure 11] 11 is a flow diagram showing the flow of a video receiving method by a terminal device according to an exemplary embodiment 2 of the present invention. [Figure 12]FIG. 11 is a block diagram showing a functional configuration of a video transmission system according to an exemplary embodiment 3 of the present invention. [Figure 13] 11 is a diagram for explaining a method for determining a delivery priority in accordance with the number of gaze areas of a plurality of users. FIG. [Figure 14] FIG. 11 is a flow chart showing the flow of a video transmission method by a video transmission system according to an exemplary embodiment 3 of the present invention. [Figure 15] FIG. 11 is a block diagram showing a functional configuration of a video transmission system according to an exemplary embodiment 4 of the present invention. [Figure 16] FIG. 11 is a block diagram showing a functional configuration of a video transmission system according to an exemplary embodiment 5 of the present invention. [Figure 17] FIG. 13 is a block diagram showing a functional configuration of a video transmission system according to an exemplary embodiment 6 of the present invention. [Figure 18] FIG. 2 illustrates an example of computer hardware. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Example 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described later.

[0015] <Overview of video transmission system 100> In brief, the video transmission system 100 according to this exemplary embodiment selects and receives second point cloud data in accordance with the user's line of sight information, and outputs the received second point cloud data.

[0016] <Configuration of video transmission system 100> The configuration of a video transmission system 100 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the video transmission system 100.

[0017] As shown in Fig. 1, the video transmission system 100 includes a transmission unit 11, a reception unit 21, and an output unit 22. The transmission unit 11 is configured to realize a transmission means in this exemplary embodiment. The reception unit 21 is configured to realize a reception means in this exemplary embodiment. The output unit 22 is configured to realize an output means in this exemplary embodiment.

[0018] The transmitter 11 transmits second point cloud data extracted from first point cloud data representing a first region in a three-dimensional space.

[0019] FIG. 2 is a diagram showing an example of a first region in a three-dimensional space. For example, the three-dimensional space is photographed by a LiDAR (Light Detection and Ranging), a depth camera, or the like. Hereinafter, the three-dimensional space that is the subject of photography by the sensor will be referred to as a subject space. Note that the three-dimensional space is a concept that includes three-dimensional spaces such as real places and virtual places.

[0020] In this exemplary embodiment, the subject space is divided into a plurality of regions. As an example, the subject space is divided into 3×3×4 regions as shown in FIG. 2. One of the divided regions is called a first region. Note that the number of divisions of the subject space shown in FIG. 2 is an example, and other numbers may be used.

[0021] The point cloud data representing the space to be photographed is transmitted for each region of the space to be photographed. For example, as shown in FIG. 2, if the space to be photographed is composed of cubes, one of the cubes may be transmitted. Also, a specific person existing in the three-dimensional space may be regarded as one region. Also, each part of a person, such as a face, an arm, and a leg, may be regarded as one region. Furthermore, each object, such as a car and a tree, may be regarded as one region.

[0022] The first point cloud data represents a first region of the space to be photographed. The point cloud data is a set of data points representing a three-dimensional space, and is, for example, data generated by the above-mentioned LiDAR or depth camera, or data obtained by processing such data. The data points include information about the position and color of an object in the three-dimensional space.

[0023] In this exemplary embodiment, extracting the second point cloud data from the first point cloud data means, for example, generating the second point cloud data from the first point cloud data so that the second point cloud data includes at least one of the divided data obtained by extracting the first point cloud data. A specific method for extracting the second point cloud data from the first point cloud data will be described later. Note that the divided data means that the original data before extraction, for example, includes only a part of the first point cloud data, and does not include the entire first point cloud data.

[0024] The receiving unit 21 receives the second point cloud data according to line-of-sight information relating to the line of sight of the user.

[0025] The gaze information is information on the direction of the gaze of a user who is using a terminal device. The gaze information may include information on the movement of the user's eyeballs, the direction of the user's face, etc. The gaze information is collected using a sensor mounted on a terminal device having the output unit 22 or a sensor provided at the location where the user is.

[0026] When the terminal device is a pair of virtual reality (VR) goggles, the user's gaze information may be collected using an acceleration sensor, an angular velocity sensor, etc., that the VR goggles are provided with. In addition, the terminal device may acquire image data captured by a camera provided at the viewing location where the user is located, and collect the user's gaze information by analyzing the image data in the terminal device.

[0027] The output unit 22 outputs an image to be displayed to the user based on the second point cloud data.

[0028] The output unit 22 is a device for displaying an image on a terminal device, and is configured to include a display panel, for example. In this exemplary embodiment, the image may be a still image or a moving image (video). Moreover, outputting an image means, for example, displaying the image. The output unit 22 displays the image represented by the second point cloud data.

[0029] As an example, the output unit 22 may be configured to apply processing for improving viewing quality, such as changing the point size of the second point cloud data or mesh processing, to the second point cloud data, and then display an image represented by the point cloud data after the processing. Examples of the terminal device include VR goggles, a television, a PC (Personal Computer), a tablet terminal, a smartphone, etc., but this does not limit the present exemplary embodiment.

[0030] (Point cloud data extraction) Fig. 3 is a diagram for explaining an example of a method for extracting point cloud data. As shown in Fig. 3, in this exemplary embodiment, as an example, first point cloud data is extracted hierarchically. More specifically, in the example shown in Fig. 3, point cloud data is extracted in a binary tree manner. That is, point cloud data with a point cloud density of 1 / 2 is extracted hierarchically from the point cloud data, and the point cloud density is halved. As an extraction method, downsampling, which samples at regular intervals, a method of random sampling, etc. are used.

[0031] In the example shown in FIG. 3, point cloud data with a point cloud density of 1 / 2 is extracted from the first point cloud data, and thus divided data A, which is one of the point cloud data, and another point cloud data different from divided data A, are obtained. Then, point cloud data with a point cloud density of 1 / 2 is further extracted from the other point cloud data. As a result, divided data B, which is one of the point cloud data, and another point cloud data different from divided data B are obtained. Then, point cloud data with a point cloud density of 1 / 2 is further extracted from the other point cloud data. As a result, divided data C, which is one of the point cloud data, and another divided data D different from divided data C are obtained. Here, point cloud data with a point cloud density of 1 / 2 may be further extracted from either divided data C or divided data D. In addition, as a termination condition for such an extraction process, a condition such as reaching a pre-specified number of point clouds or reaching a specified number of extractions may be used.

[0032] Fig. 4 is a diagram showing a specific example of extracting point cloud data with a point cloud density of 1 / 2 from the first point cloud data. As shown in Fig. 4, every other point cloud indicated by "○" in the first row of the first point cloud data is extracted as divided data A. Also, every other point cloud indicated by "△" in the first row of the first point cloud data is extracted as the other divided data.

[0033] In the second row of the first point cloud data, every other point cloud indicated by a "○" located under the "△" on the first row is extracted as divided data A. Also, every other point cloud indicated by a "△" located under the "○" on the first row is extracted as the other divided data. In this way, every other point cloud data that is not adjacent to each other vertically or horizontally is extracted, and point cloud data with 1 / 2 point cloud density is created.

[0034] 4, when creating half point cloud data from the other point cloud data extracted from the first point cloud data, the split data is created in a similar manner. When creating split data C and split data D from the other point cloud data of split data B, the split data is created in a similar manner.

[0035] In FIG. 4, the "◯" and "Δ" may represent one point of data included in the first point cloud data, or may represent data of multiple points.

[0036] In the example shown in Fig. 3, divided data A to D are extracted from the first point cloud data, and a point cloud included in one divided data does not overlap with a point cloud included in another divided data. Also, a point cloud included in the first point cloud data is included in any one of divided data A to D. Therefore, it is possible to reconstruct the first point cloud data by combining divided data A to D.

[0037] In the examples of Figures 3 and 4, hierarchical extraction has been described, but this does not limit the present exemplary embodiment, and non-hierarchical extraction may be used. For example, point cloud data with a point cloud density of 1 / N may be extracted non-hierarchically from point cloud data, and N pieces of divided data with the same number of points may be generated. Here, as an example, a positive integer expressed by a power of 2 (N=2, 4, 8, 16, etc.) may be used as N, but this does not limit the present exemplary embodiment, and other values ​​of N may be used.

[0038] As described above, the method of extracting point cloud data is not limited to the method of extracting point cloud data with a point cloud density of 1 / 2 from the point cloud data. In the following description, the point cloud data extracted from the first point cloud data may be referred to as point cloud data A to D, but the following point cloud data A to D are not limited to the data extracted by the example of the extraction method in FIG.

[0039] Instead of extracting point cloud data, the point cloud density may be reduced by aggregating the points included in the point cloud data. For example, the point cloud density may be reduced by calculating the average value of the data of a certain point and the data of surrounding points, and combining the data of multiple points into data of one point.

[0040] As described above, the transmission unit 11 transmits the second point cloud data extracted from the first point cloud data. Here, extracting the second point cloud data from the first point cloud data means generating the second point cloud data from the first point cloud data such that the second point cloud data includes at least one of the divided data obtained by dividing the first point cloud data, as described above.

[0041] Therefore, the second point cloud data is, for example, data having a point cloud density different from that of the first point cloud data.

[0042] The second point cloud data may include a plurality of point cloud data extracted from the first point cloud data and having a point cloud density different from that of the first point cloud data.

[0043] For example, the transmitting unit 11 may extract the second point cloud data from the first point cloud data such that the second point cloud data includes all of the divided data A to the divided data D shown in Fig. 3. The transmitting unit 11 may also extract the second point cloud data from the first point cloud data such that the second point cloud data includes at least one of the divided data A to the divided data D shown in Fig. 3. The transmitting unit 11 can distribute the same second point cloud data to multiple users by broadcast distribution or multicast distribution.

[0044] When transmitting unit 11 compresses and distributes the second point cloud data, compression is performed on each of the divided data included in the second point cloud data. For example, when transmitting unit 11 distributes second point cloud data including all of divided data A to D, transmitting unit 11 compresses divided data A to create compressed data A. Similarly, transmitting unit 11 individually compresses divided data B to D to create compressed data B to D. Then, transmitting unit 11 distributes the second point cloud data including compressed data A to D.

[0045] As described above, the receiving unit 21 receives the second point cloud data according to the user's line of sight information. For example, when the user's line of sight is directed to the first area shown in Fig. 2, the receiving unit 21 receives the divided data A (1 / 2), the divided data B (1 / 4), and the divided data C (1 / 8) so that the point cloud density of the first area becomes high. Then, the output unit 22 may combine and display the divided data A (1 / 2), the divided data B (1 / 4), and the divided data C (1 / 8).

[0046] When the user's line of sight is directed toward the first region, as an example, the receiver 21 receives divided data A (1 / 2) and divided data C (1 / 8) for the region adjacent to the first region. Then, the output unit 22 synthesizes and displays the divided data A (1 / 2) and divided data C (1 / 8). Therefore, as an example, the point cloud density is lower than that of the first region.

[0047] Also, when the user's line of sight is directed toward the first region, as an example, the receiving unit 21 receives only the divided data C(1 / 8) for the region away from the first region. Then, the output unit 22 displays the divided data C(1 / 8). Therefore, as an example, the point cloud density is lower than that of the region adjacent to the first region.

[0048] <Effects of the video transmission system 100> As described above, according to the video transmission system 100 according to this exemplary embodiment, the receiver 21 receives the second point cloud data in response to line-of-sight information related to the user's line of sight. Therefore, the area where the user's line of sight is directed can be displayed with a high point cloud density, and the area where the user's line of sight is not directed can be displayed with a low point cloud density, thereby providing the user with a high-quality video.

[0049] In addition, since the receiving unit 21 receives point cloud data according to the user's line of sight information, the amount of data received can be reduced, and the communication load can be reduced. As described above, the point cloud density is increased in the area where the user's line of sight is directed, and the point cloud density is gradually decreased as the area moves away from the area, so that an image closer to the human field of vision can be provided.

[0050] In addition, the transmitting unit 11 distributes the second point cloud data to multiple users by broadcast distribution or multicast distribution, the receiving units 21 corresponding to each user receive the second point cloud data according to gaze information regarding the user's gaze, and the output units 22 corresponding to each user output an image based on the second point cloud data, so that it is possible to distribute high-quality video according to the gaze of each of the multiple users.

[0051] For example, when multicast distribution is performed to multiple users, the transmitting unit 11 notifies each of the multiple users of the multicast group to which the divided data is to be distributed. Depending on which divided data each of the multiple users receives, it is determined which multicast group the user belongs to.

[0052] For example, (split data A to split data D) are distributed to users belonging to multicast group (hereinafter abbreviated as MCG) 1, (split data B to split data D) are distributed to users belonging to MCG2, and (split data C, split data D) are distributed to users belonging to MCG3. Also, for example, when split data A is distributed to MCG1, split data B is distributed to MCG2, and split data C is distributed to MCG3, receiving unit 21 may receive data belonging to one or more MCGs according to the user's line of sight information.

[0053] The receiving unit 21 receives the second point cloud data corresponding to the user's line of sight information, which is divided data of the multicast group to which the receiving unit 21 belongs. For example, if the multicast group to which the receiving unit 21 belongs is MCG2, the receiving unit 21 receives the second point cloud data corresponding to the user's line of sight information from divided data B to divided data D.

[0054] <Flow of video transmission method by video transmission system 100> The flow of a video transmission method executed by the video transmission system 100 configured as above will be described with reference to Fig. 5. Fig. 5 is a flow diagram showing the flow of the video transmission method. As shown in Fig. 5, the video transmission method includes steps S1 to S3.

[0055] First, the transmitting unit 11 transmits second point cloud data extracted from the first point cloud data representing a first region in a three-dimensional space (S1). Next, the receiving unit 21 receives the second point cloud data according to line-of-sight information related to the user's line of sight (S2). Finally, the output unit 22 outputs an image to be displayed to the user based on the second point cloud data (S3).

[0056] <Effects of video transmission method> As described above, according to the video transmission method of this exemplary embodiment, in step S2, the second point cloud data is received according to the line-of-sight information related to the user's line of sight. Therefore, the area where the user's line of sight is directed can be displayed with a high point cloud density, and the area where the user's line of sight is not directed can be displayed with a low point cloud density, and a high-quality video can be provided to the user.

[0057] In step S2, the point cloud data is received according to the user's line of sight, so the amount of data received can be reduced, and the communication load can be reduced. As described above, the point cloud density is increased in the area where the user's line of sight is directed, and the point cloud density is gradually decreased as the area moves away from the area, so that an image closer to the human field of vision can be provided.

[0058] Furthermore, in step S1, the second point cloud data is distributed to a plurality of users, for example by broadcast distribution or multicast distribution, in step S2, the second point cloud data is received according to gaze information regarding the gaze of each user, and in step S3, an image is output based on the second point cloud data corresponding to each user, thereby making it possible to distribute high-quality video according to the gazes of a plurality of users.

[0059] Furthermore, the transmission unit 11 performs multicast or broadcast distribution of the second point cloud data, thereby enabling data distribution to a plurality of users without requiring additional communication resources.

[0060] The distribution method is not limited to the above-mentioned broadcast distribution and multicast distribution.

[0061] Exemplary embodiment 2 A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0062] <Configuration of Video Transmission System 100'> The configuration of a video transmission system 100′ according to this exemplary embodiment will be described with reference to Fig. 6. Fig. 6 is a block diagram showing a functional configuration of the video transmission system 100′. As shown in Fig. 6, the video transmission system 100′ includes a server device 1 and a terminal device 2.

[0063] (Server device) As shown in FIG. 6, the server device 1 includes a transmission unit 11, a space dividing unit 12, a point cloud data extraction unit 13, and a transmission information generation unit .

[0064] The transmitting unit 11 is a component that realizes the transmitting means in this exemplary embodiment. The space dividing unit 12 is a component that realizes the space dividing means in this exemplary embodiment. The point cloud data extracting unit 13 is a component that realizes the point cloud data extracting means in this exemplary embodiment. The transmission information generating unit 14 is a component that realizes the transmission information generating means in this exemplary embodiment.

[0065] The space dividing unit 12 divides an image of the target space in a three-dimensional space into a plurality of regions. For example, as shown in Fig. 2, the space dividing unit 12 divides the target space into a plurality of regions. One of the regions divided by the space dividing unit 12 is called a first region. The first region of the target space includes first point cloud data.

[0066] The point cloud data extraction unit 13 extracts second point cloud data from the point cloud data representing the first region after being divided by the space division unit 12. Here, as described in the exemplary embodiment 1, extracting the second point cloud data from the first point cloud data means, for example, generating the second point cloud data from the first point cloud data such that the second point cloud data includes at least one of the divided data obtained by dividing the first point cloud data.

[0067] Also, the point cloud data extraction unit 13 may divide the first point cloud data to extract second point cloud data including a plurality of point cloud data with different point cloud densities. As an example, the point cloud data extraction unit 13 may divide the first point cloud data into a plurality of point cloud data, for example, divided data A to divided data D as shown in FIG. 3, and extract the data as the second point cloud data. As described in the exemplary embodiment 1, it is possible to reconstruct the first point cloud data by combining the divided data A to divided data D.

[0068] The transmission unit 11 transmits the second point cloud data. Here, the transmission of the second point cloud data is performed via a network, as an example. Although the specific configuration of the network does not limit the present exemplary embodiment, as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

[0069] The transmission unit 11 may transmit fourth point cloud data extracted from third point cloud data representing a second region in a three-dimensional space different from the first region. For example, one of the regions adjacent to the first region shown in FIG. 2 is the second region, and the point cloud data extraction unit 13 divides the third point cloud data representing the second region to extract the fourth point cloud data. Here, as a specific method for extracting the fourth point cloud data from the third point cloud data, a method similar to the method for extracting the second point cloud data from the first point cloud data described above can be used.

[0070] The transmission information generating unit 14 generates transmission information related to the second point cloud data. The transmission information generating unit 14 generates transmission information including, for example, spatial information of the divided regions, the number of points included in the point cloud data for each divided data, the bit rate of the point cloud data, channel information for distributing the second point cloud data, and the like, and outputs the transmission information to the transmitting unit 11.

[0071] An example of the spatial information is the coordinates of eight corners of a rectangular parallelepiped that constitutes each divided space. The transmission information includes, for example, the number of points in divided data A to D shown in Fig. 3 and channel information for distributing the divided data (such as the IP address of a multicast group in IP multicast). Note that, as shown in Fig. 7, the divided areas themselves may be given names and used as spatial information.

[0072] The transmission information includes information about the size of the second point cloud data. The information about the size of the second point cloud data is, for example, the number of points in the second point cloud data, the point cloud density of the second point cloud data, and the data capacity of the second point cloud data. The transmission information may also include at least one of information about the position of the first region in the shooting target space, information about the correspondence between the first region and the second point cloud data, and information about the bit rate of the second point cloud data. Here, the information about the correspondence between the first region and the second point cloud data is, for example, information indicating which region of the divided regions the second point cloud data corresponds to. The transmission information may also include information about a transmission path for transmitting the second point cloud data.

[0073] The transmission unit 11 transmits the second point cloud data and the transmission information. At this time, when the second point cloud data includes a plurality of divided data, the transmission unit 11 transmits the transmission information to the terminal device 2 in association with each of the plurality of divided data.

[0074] When the server device 1 distributes the 3D video, a different transmission path may be used for each piece of divided data. When a different transmission path is used for each piece of divided data, the transmission information may include information on the transmission path for transmitting the divided data. For example, divided data with a high point cloud density is distributed using a transmission path that can transmit at high speed. Also, divided data with a low point cloud density is distributed using another transmission path that is relatively slow.

[0075] (Terminal Device) 6, the terminal device 2 includes a gaze information collecting unit 23 that acquires gaze information indicating a user's gaze, a receiving unit 21 that receives second point cloud data extracted from first point cloud data representing a first area in a three-dimensional space in accordance with the gaze information, an output unit 22 that outputs an image to be displayed to the user based on the second point cloud data received by the receiving unit 21, and an acquired data identifying unit 24 that identifies the point cloud data to be acquired. Specific processes performed by each unit included in the terminal device 2 will be described later.

[0076] The receiving unit 21 of the terminal device 2 may receive viewpoint information indicating the direction from which the user views the three-dimensional space represented by the point cloud data. For example, in Fig. 7, the user views the three-dimensional space represented by the point cloud data from the left viewpoint.

[0077] The receiving unit 21 is a component that realizes a receiving means in this exemplary embodiment. The output unit 22 is a component that realizes an output means in this exemplary embodiment. The gaze information collecting unit 23 is a component that realizes a gaze information acquiring means in this exemplary embodiment. The acquired data identifying unit 24 is a component that realizes an identifying means in this exemplary embodiment.

[0078] FIG. 7 is a diagram for explaining a method of receiving point cloud data by the receiving unit 21. The receiving unit 21 receives the second point cloud data according to the gaze information of the user collected by the gaze information collecting unit 23 such as VR goggles. For example, as shown in FIG. 7, when the gaze of the user is directed to the area b2, the receiving unit 21 receives the second point cloud data of the area b2 so that the point cloud density of the area b2 becomes high. Note that the specific collection process of the gaze information by the gaze information collecting unit 23 does not limit this exemplary embodiment, but as an example, as described in the exemplary embodiment 1, the gaze information of the user may be collected using an acceleration sensor, an angular velocity sensor, or the like provided in the VR goggles. In addition, the gaze information collecting unit 23 may acquire imaging data by a camera provided at the viewing location where the user is present, and the gaze information collecting unit 23 may analyze the imaging data to collect the gaze information of the user.

[0079] Furthermore, the receiver 21 receives second point cloud data for the areas b1, b3, b5, and b8 adjacent to the area b2 so that the area b1, b3, b5, and b8 have a lower point cloud density than the area b2.

[0080] The receiver 21 also receives second point cloud data for areas b4, b6, b7, and b9 that are away from area b2 so that the point cloud density for these areas is lower than that for areas b1, b3, b5, and b8.

[0081] The terminal device 2 may include an acquired data specifying unit 24 that specifies the image quality to be displayed to the user, and the receiving unit 21 may receive the second point cloud data according to the specified image quality. For example, when the user's gaze is directed to the first region shown in Fig. 2, the receiving unit 21 specifies and receives divided data with a high point cloud density included in the second point cloud data so that the image of the first region has high image quality. In addition, the receiving unit 21 specifies and receives divided data with a low point cloud density included in the second point cloud data so that the image of the region other than the first region to which the user's gaze is not directed has lower image quality than the image of the first region.

[0082] Furthermore, when the transmitting unit 11 of the server device 1 extracts and further transmits fourth point cloud data from third point cloud data representing a second area different from a first area in three-dimensional space, the receiving unit 21 of the terminal device 2 may receive the fourth point cloud data according to the line of sight information, and the output unit 22 may output an image to be displayed to the user based on the second point cloud data and the fourth point cloud data.

[0083] In addition, when the transmitting unit 11 of the server device 1 transmits the second point cloud data and the transmission information, the receiving unit 21 of the terminal device 2 receives the second point cloud data according to the gaze information and the received transmission information, and the output unit 22 outputs an image to be displayed to the user based on the second point cloud data and the transmission information.

[0084] FIG. 8 is a diagram for explaining a case where a plurality of users receive point cloud data. As shown in FIG. 8, when the first user's line of sight is directed to the region b2, the receiving unit 21 of the terminal device 2 of the first user receives the second point cloud data of the region b2 so that the point cloud density of the region b2 is high. In addition, the receiving unit 21 of the terminal device 2 of the first user receives the second point cloud data of the regions b1, b3, b5, etc., adjacent to the region b2 so that the point cloud density of the regions b1, b3, b5, etc. is lower than that of the region b2. In addition, the receiving unit 21 of the terminal device 2 of the first user receives the second point cloud data of the regions b4, b6, etc., distant from the region b2 so that the point cloud density of the regions b4, b6, etc. is lower than that of the regions b1, b3, b5, etc.

[0085] On the other hand, when the second user's line of sight is directed to region b8, the receiving unit 21 of the terminal device 2 of the second user receives the second point cloud data of region b8 so that the point cloud density of region b8 is high. Also, the receiving unit 21 of the terminal device 2 of the second user receives the second point cloud data of regions b5, b7, b9, etc., adjacent to region b8 so that the point cloud density of regions b5, b7, b9, etc. is lower than that of region b8. Also, the receiving unit 21 of the terminal device 2 of the second user receives the second point cloud data of regions b4, b6, etc., distant from region b8 so that the point cloud density of regions b4, b6, etc. is lower than that of regions b5, b7, b9, etc.

[0086] In this way, the terminal devices 2 of the first user and the second user can receive and output point cloud data with optimal point cloud density according to their respective line of sight information.

[0087] The acquisition data specification unit 24 specifies the second point cloud data to be acquired based on the display distance of the first area and an evaluation function related to the viewing quality specified by the point cloud density of the second point cloud data. The display distance indicates the distance from the user's viewpoint to an arbitrary area in the three-dimensional space represented by the point cloud data. For example, in the case of the display distance of the first area, it indicates the distance from the user's viewpoint to the first area in the three-dimensional space represented by the point cloud data.

[0088] Fig. 9 is a diagram for explaining an evaluation function for viewing quality. As shown in Fig. 9, viewing quality (QoE) can be expressed by the point cloud density (Density) of point cloud data and the display distance (Distance) to the area where the user's gaze is directed (hereinafter, also referred to as the gaze area). The relationship between this point cloud density and the display distance to the gaze area is expressed as a function to be used as the evaluation function for viewing quality.

[0089] The acquisition data specifying unit 24 specifies the priority of the divided data for each region so as to maximize the total value of the viewing quality determined by, for example, the point cloud density for each divided region (if the distributed point cloud data is divided, the total value of the point cloud densities of each divided data) and the display distance. The receiving unit 21 receives the divided data from the server device 1 according to the priority specified by the acquisition data specifying unit 24.

[0090] In addition, the acquisition data identification unit 24 may identify the priority of the split data for each area so as to maximize the total value of the viewing quality determined by the bit rate of the point cloud data representing each area distributed from the server device 1 and the display distance.

[0091] If the bit rate of receivable area n is b(n,i) and the distance to area n is d(n), then the evaluation function of the viewing quality of the divided areas is U(b,d). Therefore, the total viewing quality can be expressed by (Equation 1). The constraints are as shown in (Equation 2). Note that if the point cloud data is divided, i ranges from 1 to the division number. For example, if the point cloud data is divided into four, i=1 to 4.

[0092] Total viewing quality = ΣU(b(n,i),d(n)) (Equation 1) Σb(n,i)≦min(radio bandwidth, maximum reception rate) (Equation 2) The acquisition data specifying unit 24 specifies the priority of the divided data for each region so that the total viewing quality of (Equation 1) is maximized within a range that satisfies the constraint condition of (Equation 2).

[0093] When receiving point cloud data, there are restrictions on the upper limit of the number of point clouds of point cloud data that can be acquired due to the terminal capabilities of the terminal device 2 and the constantly changing network bandwidth, so the receiving unit 21 receives the point cloud data according to priority so that the number of point clouds of the point cloud data is below the upper limit.

[0094] The output unit 22 performs a display process by synthesizing the received point cloud data corresponding to each divided region so as to integrate them. As an example, the output unit 22 may be configured to apply a process for improving the viewing quality, such as changing the point size of the second point cloud data or mesh processing, to the second point cloud data, and then display an image represented by the point cloud data after the process is applied. Note that, as an example, the terminal device may be a VR goggle, a television, a PC (Personal Computer), a tablet terminal, a smartphone, etc., but this does not limit the present exemplary embodiment.

[0095] <Flow of video transmission method by video transmission system 100> The flow of a video transmission method executed by the video transmission system 100 configured as above will be described with reference to Fig. 10. Fig. 10 is a flow diagram showing the flow of the video transmission method. As shown in Fig. 10, the video transmission method includes steps S11 to S18.

[0096] First, the space dividing unit 12 of the server device 1 divides the space to be photographed into a plurality of regions (S11). Then, the point cloud data extracting unit 13 of the server device 1 divides the first point cloud data representing the first region to extract second point cloud data including a plurality of point cloud data having different point cloud densities (S12).

[0097] Next, the transmission information generation unit 14 of the server device 1 generates transmission information from the area of ​​the point cloud data extracted by the point cloud data extraction unit 13, the point cloud density of the point cloud data, etc. (S13), and the transmission unit 11 transmits the transmission information to the terminal device 2.

[0098] Meanwhile, the acquired data identifying unit 24 of the terminal device 2 acquires the user's gaze information collected by the gaze information collecting unit 23 (S14). When the receiving unit 21 receives transmission information from the server device 1, the acquired data identifying unit 24 acquires the transmission information, identifies point cloud data to be acquired from the user's gaze information and the transmission information (S15), and notifies the receiving unit 21 of the identified point cloud data.

[0099] When the extracted point cloud data for each area is transmitted from the server device 1, the receiving unit 21 of the terminal device 2 receives only the point cloud data notified from the acquired data identification unit 24 (S16) and outputs the received point cloud data to the output unit 22.

[0100] Then, the output unit 22 of the terminal device 2 synthesizes the point cloud data received by the receiving unit 21 (S17) and outputs the synthesized image (S18). Thereafter, the processes of steps S11 to S18 are repeated. Therefore, every time the user's line of sight information is changed, in steps S15 to S18, point cloud data corresponding to the changed line of sight information is received, and the point cloud data is synthesized and an image is output.

[0101] <Flow of video reception method by terminal device 2> Hereinafter, the flow of the video receiving method executed by the terminal device 2 of the video transmission system 100 will be described with reference to Fig. 11. Fig. 11 is a flow diagram showing the flow of the video receiving method by the terminal device 2 according to the second exemplary embodiment of the present invention.

[0102] First, the acquisition data identification unit 24 identifies the priority of the point cloud data to be acquired based on the target distance to the gaze area obtained from the user's gaze information collected by the gaze information collection unit 23, the transmission information received from the server device 1, and an evaluation function related to viewing quality identified by the target distance and the point cloud density (S21).

[0103] Next, the receiving unit 21 receives the point cloud data necessary to construct a three-dimensional image from the multiple point cloud data distributed from the server device 1 based on the priority of the point cloud data identified by the acquired data identification unit 24 (S22).

[0104] Finally, the output unit 22 integrates and synthesizes the received point cloud data corresponding to each area, and performs display processing (S23).

[0105] Each unit of the server device 1 may be implemented in a separate device on the cloud. For example, the transmission unit 11 and the space division unit 12 may be a single device, and the point cloud data extraction unit 13 and the transmission information generation unit 14 may be a single device. These may be implemented in one device or in separate devices. For example, when they are implemented in separate devices, information of each unit is transmitted and received via a communication network to proceed with the processing.

[0106] Similarly, each unit of the terminal device 2 may be implemented in a separate device on the cloud. For example, the receiving unit 21 and the output unit 22 may be a single device, and the line-of-sight information collecting unit 23 and the acquired data identifying unit 24 may be a single device. These may be implemented in a single device or in separate devices. For example, when implemented in separate devices, information of each unit is transmitted and received via a communication network to proceed with the processing.

[0107] <Effects of the video transmission system 100'> As described above, according to the video transmission system 100' according to this exemplary embodiment, the transmission information generating unit 14 generates transmission information including at least one of information on the position of the first region in the shooting target space, information on the correspondence between the first region and the second point cloud data, and information on the bit rate of the second point cloud data. Therefore, the output unit 22 can appropriately combine and display the second point cloud data based on the transmission information.

[0108] The acquisition data specifying unit 24 specifies the second point cloud data to be acquired based on an evaluation function related to the viewing quality specified by the display distance to the first region and the point cloud density of the second point cloud data. Therefore, the output unit 22 can display the 3D video with high viewing quality.

[0109] Furthermore, since the terminal device 2 according to this exemplary embodiment acquires the second point cloud data according to the user's line of sight, it is possible to reduce the load on the server device 1 compared to the case where the server device 1 extracts point cloud data of each area in accordance with the line of sight of each user.

[0110] Exemplary embodiment 3 A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first and second exemplary embodiments are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0111] When the server device 1 distributes point cloud data by wireless communication, for example, it is assumed that the wireless bandwidth is insufficient and all the point cloud data cannot be distributed. In this exemplary embodiment, the server device 1 acquires user line-of-sight information (gaze area) from the terminal device 2 and distributes the point cloud data based on the line-of-sight information of multiple users, thereby reducing the consumption bandwidth.

[0112] 12 is a block diagram showing a functional configuration of a video transmission system 100A according to an exemplary embodiment 3 of the present invention. The video transmission system 100A includes a server device 1A and a terminal device 2A.

[0113] Moreover, the server device 1A includes a space dividing unit 12, a point cloud data extracting unit 13, a transmission information generating unit 14, a communication unit 15, and a transmission data identifying unit 16. The communication unit 15 transmits the second point cloud data and the transmission information to the terminal device 2A, and receives the user's gaze area from the terminal device 2A. Thus, the server device 1A, in summary, further includes a transmission data identifying unit 16 in addition to the configuration of the server device 1 according to the exemplary embodiment 2, and includes a communication unit 15 instead of the transmission unit 11.

[0114] The communication unit (transmission unit) 15 transmits the second point cloud data according to the gaze information of the user. Specifically, first, the transmission data specification unit 16 determines a delivery priority for the point cloud data based on the gaze information (gaze areas) of multiple users received by the communication unit 15, and specifies the point cloud data to be transmitted in accordance with the delivery performance of the server device 1A and the constraints of the network bandwidth. Then, the communication unit 15 delivers the point cloud data according to the delivery priority determined by the transmission data specification unit 16.

[0115] For example, the transmission data identification unit 16 may distribute point cloud data having the smallest point cloud density after division for all areas (e.g., divided data C shown in FIG. 3), and as long as there is processing performance and network bandwidth available, the point cloud data may be allocated to available bandwidth in order from the point cloud data having the higher point cloud density among the multiple divided point cloud data according to the number of user's gaze areas.

[0116] Fig. 13 is a diagram for explaining a method for determining delivery priority according to the number of gaze areas of multiple users. Scores are assigned according to the distance between each of the multiple users and each divided area. As shown in Fig. 13, the smallest score is assigned to an area closest to the area gazed upon by the user on the left, and a larger score is assigned as the distance increases.

[0117] Similarly, the smallest score is assigned to the area closest to the area gazed upon by the user on the right, and a larger score is assigned as the distance increases. The transmission data specification unit 16 assigns a score by adding up the scores of all users in each divided area. The delivery priority of the area with the smallest score is increased, and the delivery priority is decreased as the score increases.

[0118] 14 is a flow diagram showing the flow of a video transmission method by the video transmission system 100A according to the exemplary embodiment 3 of the present invention. First, the space dividing unit 12 acquires point cloud data of the shooting target space (S31), and spatially divides the point cloud data (S32).

[0119] Next, the point cloud data extraction unit 13 extracts the first point cloud data representing the first region spatially divided by the space division unit 12 so that each point cloud data has a different point cloud density (S33). The communication unit 15 determines whether the server device 1A has a processing performance capable of distributing all of the extracted point cloud data and whether there is a communication bandwidth between the terminal device 2A and the communication network (S34).

[0120] If all of the extracted point cloud data can be distributed (S34, Yes), the process proceeds to step S36. If even a part of all of the extracted point cloud data cannot be distributed (S34, No), the transmission data specifying unit 16 determines the point cloud data to be distributed based on the gaze areas and distance information of multiple users (S35).

[0121] In step S36, the communication unit 15 distributes the point cloud data determined as the distribution target by the transmission data specifying unit 16 by broadcast distribution or multicast distribution, and ends the process.

[0122] (Re-distribution of recorded 3D video) The server device 1 stores the generated split data on the server device 1, and the terminal device 2 can access the server device 1 and receive the split data according to the user's line of sight, thereby enabling the user to view the video on demand.

[0123] When the server device 1 re-distributes a recorded three-dimensional video, the server device 1 may prepare second point cloud data by referring to the gaze areas of multiple users when the three-dimensional video was previously distributed, and may re-distribute the recorded three-dimensional video.

[0124] For example, when the server device 1A distributes a three-dimensional video, it records the gaze areas of multiple users in association with each scene of the three-dimensional video. Then, when the server device 1A re-distributes the same three-dimensional video, it determines the point cloud data to be distributed by referring to the gaze areas of the previous multiple users, and transmits the three-dimensional video. Therefore, it is possible to distribute divided data so that the parts that the multiple users gazed on the previous time have high image quality.

[0125] Also, the server device 1A may prepare the second point cloud data by referring to a gaze area of ​​one user and re-distribute the recorded three-dimensional video. For example, when the server device 1A distributes a three-dimensional video, it records the gaze area of ​​a certain user in association with each scene of the three-dimensional video. Then, when the server device 1A re-distributes the three-dimensional video to the same user, it determines the point cloud data to be distributed by referring to the gaze area of ​​the user and transmits the three-dimensional video. Therefore, it is possible to distribute the divided data so that the part that the user gazed at has high image quality.

[0126] <Effects of the video transmission system 100A> As described above, in the video transmission system 100A according to this exemplary embodiment, the transmission data specification unit 16 determines a delivery priority for the point cloud data and specifies the point cloud data to be transmitted based on the gaze areas of the multiple users received by the communication unit 15. Therefore, the communication unit 15 can transmit the point cloud data in accordance with the delivery performance of the server device 1A and the constraints of the network bandwidth.

[0127] Exemplary embodiment 4 A fourth exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first to third exemplary embodiments are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0128] 15 is a block diagram showing a functional configuration of a video transmission system 100B according to an exemplary embodiment 4 of the present invention. The video transmission system 100B includes a server device 1B and a terminal device 2.

[0129] Moreover, the server device 1B includes a transmission unit 11, a space division unit 12, a point cloud data extraction unit 13, a transmission information generation unit 14, and a divided area recognition unit 17. The divided area recognition unit 17 recognizes people and objects, and identifies an area for each person or object, or for each part of a person's body.

[0130] The space dividing unit 12 divides the space according to the area specified by the divided area recognizing unit 17. For example, an area is divided for each part of a person's body, and the space corresponding to the area is divided. For example, when a user is watching a soccer game, the point cloud data of a soccer player's foot is treated as one unit, and the point cloud data extracting unit 13 extracts the point cloud data to extract the second point cloud data. The subsequent processing is similar to that described in the exemplary embodiments 1 to 3.

[0131] <Effects of the video transmission system 100B> As described above, according to the video transmission system 100B according to this exemplary embodiment, the divided area recognition unit 17 identifies an area for each person, object, or part of a human body. Therefore, it is possible to provide 3D video with the same quality for each person, object, or part of a human body.

[0132] Exemplary embodiment 5 The fifth exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first to third exemplary embodiments are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0133] 16 is a block diagram showing a functional configuration of a video transmission system 100C according to an exemplary embodiment 5 of the present invention. The video transmission system 100C includes a server device 1C and a terminal device 2.

[0134] The server device 1C also includes a transmission unit 11, a space division unit 12, a point cloud data extraction unit 13, a transmission information generation unit 14, and a gaze point identification unit 18. The gaze point identification unit 18, on the distribution side, specifies an area of ​​a person, object, etc., to which the viewer should pay attention, and specifies the area (hereinafter, referred to as a gaze-specified area) to have high image quality. For example, a mark that can be detected by a sensor is attached to the object or area to which attention should be drawn. Then, the gaze point identification unit 18 analyzes the acquired video and identifies the area to which attention should be drawn by a method such as recognizing the object marked with the mark to which attention should be drawn.

[0135] For example, when the server device 1C distributes a video of a stage such as a play, a person or object to be focused on is marked with a mark that can be detected by the sensor. This allows the gaze point identification unit 18 to recognize the object to be focused on that is marked. In addition, various sensors can be used as the sensor as long as they acquire information on the space to be photographed. As an example of the sensor, a camera or the like can be used.

[0136] The transmission information generating unit 14 generates the transmission information so that the area to be focused on identified by the gaze point identifying unit 18 has high image quality. The point cloud data extracting unit 13 extracts the second point cloud data so that the area to be focused on identified by the gaze point identifying unit 18 has high image quality. The subsequent processing is the same as that described in the exemplary embodiments 1 to 3.

[0137] Also, the server device 1C may have a configuration for restricting the line of sight of the viewer. For example, the server device 1C transmits an instruction to the terminal device 2 to invalidate the line of sight information collected by the line of sight information collecting unit 23. The acquired data identifying unit 24 of the terminal device 2 identifies the point cloud data regardless of the line of sight information of the viewer. This allows a person or object to be displayed in high image quality on the terminal device 2, regardless of the viewer's line of sight.

[0138] For example, in the case of theater viewing, a configuration may be provided that limits the direction from which the user can view the distributed video, allowing the user to view the video from a direction suitable for the theater.

[0139] <Effects of the video transmission system 100C> As described above, according to the video transmission system 100C according to the present exemplary embodiment, the gaze point specifying unit 18 specifies the gaze point so that it will have high image quality. Therefore, the distribution side can specify the area of ​​a person, object, etc. that the viewer should pay attention to, and provide a 3D image of the specified area with high image quality.

[0140] Exemplary Embodiment 6 A sixth exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first to third exemplary embodiments are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0141] In this exemplary embodiment, illustrations, photographs, etc. are superimposed on the three-dimensional video transmitted to the terminal device 2A. For example, it is possible to insert illustrations, etc. into the video of a game being played by the user. Note that illustrations, photographs, etc. may be superimposed on a three-dimensional image instead of the three-dimensional video.

[0142] 17 is a block diagram showing a functional configuration of a video transmission system 100D according to a sixth exemplary embodiment of the present invention. The video transmission system 100D includes a server device 1D and a terminal device 2A. The server device 1D also includes a space dividing unit 12, a point cloud data extracting unit 13, a transmission information generating unit 14, a communication unit 15, and a composite data receiving unit 19.

[0143] The composite data receiving unit 19 receives data representing an illustration to be superimposed on the point cloud data of the three-dimensional space being transmitted from a user who is distributing a video. The communication unit 15 of the server device 1D transmits the received data representing the illustration together with the point cloud data to the terminal device 2A. The output unit 22 of the terminal device 2A superimposes the received illustration on the point cloud data and displays it.

[0144] Alternatively, the following configuration may be used: The communication unit 25 of the terminal device 2A transmits data such as an illustration designated by the user viewing the video to the server device 1D.

[0145] The communication unit 15 of the server device 1D receives the data of the illustrations and the like transmitted from the terminal device 2A. In addition, the composite data receiving unit 19 receives the data of the illustrations and the like received by the communication unit 15 and notifies the transmission information generating unit 14 of the position, color, and the like of the illustration.

[0146] The transmission information generating unit 14 generates transmission information from the illustration, the position and color of the illustration, etc., in response to a notification from the composite data receiving unit 19. The point cloud data extracting unit 13 superimposes the illustration on the point cloud data, extracts the point cloud data on which the illustration is superimposed, and extracts the second point cloud data. The subsequent processing is the same as that described in the exemplary embodiments 1 to 3.

[0147] When a user viewing a distributed video wants to display illustrations or the like, data of the illustrations or the like may not be sent to the server device 1D, but may be sent to the terminal device 2A of the viewing user.

[0148] The terminal device 2A may also have a function of superimposing illustrations or the like on the video. In this case, the terminal device 2A receives data of illustrations or the like designated by the user who is watching the video. Then, the terminal device 2A superimposes the data of illustrations or the like received from the user on the video received from the server device 1D and displays it.

[0149] <Effects of the video transmission system 100D> As described above, according to the video transmission system 100D according to this exemplary embodiment, the point cloud data extraction unit 13 superimposes an illustration on the point cloud data and extracts the second point cloud data from the point cloud data on which the illustration is superimposed. Therefore, it becomes possible for a user to insert a desired illustration or the like into a three-dimensional video.

[0150] [Software implementation example] Some or all of the functions of the server devices 1, 1A to 1D and the terminal devices 2, 2A may be realized by hardware such as an integrated circuit (IC chip), or may be realized by software.

[0151] In the latter case, the server devices 1, 1A to 1D and the terminal devices 2, 2A are realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter, referred to as computer C) is shown in FIG. 18. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the server devices 1, 1A to 1D and the terminal devices 2, 2A. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the server devices 1, 1A to 1D and the terminal devices 2, 2A.

[0152] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination of these. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these.

[0153] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may further include a communication interface for transmitting and receiving data to and from other devices. The computer C may further include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0154] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can obtain the program P via such a recording medium M. Furthermore, the program P can be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also obtain the program P via such a transmission medium.

[0155] [Additional Note 1] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiment are also included in the technical scope of the present invention.

[0156] [Additional Note 2] Some or all of the above-described embodiments can be described as follows. However, the present invention is not limited to the aspects described below.

[0157] (Appendix 1) A transmitting means for transmitting second point cloud data extracted from the first point cloud data representing a first region in a three-dimensional space; A receiving means for receiving the second point cloud data in response to line-of-sight information relating to a user's line of sight; and an output means for outputting an image to be displayed to the user based on the second point cloud data.

[0158] According to the above configuration, it is possible to deliver a suitable video according to the line of sight of a plurality of users.

[0159] (Appendix 2) A specifying means for specifying an image quality to be displayed to the user, The video transmission system according to claim 1, wherein the receiving means receives the second point cloud data according to the specified image quality.

[0160] According to the above configuration, the receiving means can receive the second point cloud data according to the image quality of the video.

[0161] (Appendix 3) The video transmission system according to claim 1 or 2, wherein the second point cloud data includes a plurality of point cloud data extracted from the first point cloud data and having a different point cloud density from the first point cloud data.

[0162] According to the above configuration, the output means synthesizes and outputs any one of the plurality of point cloud data, making it possible to display images of different qualities.

[0163] (Appendix 4) The transmitting means is Transmitting fourth point cloud data extracted from third point cloud data representing a second region in the three-dimensional space, the second region being different from the first region; The receiving means receives the fourth point cloud data in response to the line of sight information, 4. The video transmission system according to any one of claims 1 to 3, wherein the output means outputs an image to be displayed to the user based on the second point cloud data and the fourth point cloud data.

[0164] According to the above configuration, the transmitting means can transmit point cloud data having different point cloud densities for each region in the three-dimensional space.

[0165] (Appendix 5) the video transmission system includes a transmission information generating means for generating transmission information related to the second point cloud data; The transmitting means transmits the second point cloud data and the transmission information, The receiving means receives the second point cloud data in response to the line of sight information and the received transmission information, 5. The video transmission system according to any one of claims 1 to 4, wherein the output means outputs an image to be displayed to the user based on the second point cloud data and the transmission information.

[0166] According to the above configuration, the output means can output an image based on the second point cloud data in response to the transmitted information, and a three-dimensional video can be suitably displayed.

[0167] (Appendix 6) The transmission information is Information regarding the position of the first region in the three-dimensional space; Information regarding a correspondence relationship between the first region and the second point cloud data; and Information regarding the point cloud density of the second point cloud data 6. A video transmission system as described in claim 5, comprising at least one of the above.

[0168] According to the above configuration, the output means can display three-dimensional video images more suitably.

[0169] (Appendix 7) 7. The video transmission system according to any one of claims 1 to 6, wherein the transmission means transmits the second point cloud data in response to line-of-sight information of the user.

[0170] According to the above configuration, the transmitting means can easily specify the second point cloud data to be transmitted in accordance with the line-of-sight information of the user.

[0171] (Appendix 8) A gaze information acquisition means for acquiring gaze information indicating a user's gaze; a receiving means for receiving second point cloud data extracted from first point cloud data representing a first area in a three-dimensional space in response to the line of sight information; an output means for outputting an image to be displayed to the user based on the second point cloud data received by the receiving means.

[0172] According to the above configuration, the receiving means can receive a suitable image according to the user's line of sight.

[0173] (Appendix 9) A specifying means for specifying an image quality to be displayed to the user, The terminal device according to claim 8, wherein the receiving means receives the second point cloud data in accordance with the specified image quality.

[0174] According to the above configuration, the receiving means can receive the second point cloud data according to the image quality of the video.

[0175] (Appendix 10) The terminal device according to claim 8 or 9, wherein the second point cloud data includes a plurality of point cloud data extracted from the first point cloud data and having a point cloud density different from that of the first point cloud data.

[0176] According to the above configuration, the output means synthesizes and outputs any one of the plurality of point cloud data, making it possible to display images of different qualities.

[0177] (Appendix 11) the receiving means receives, in response to the line of sight information, fourth point cloud data extracted from third point cloud data representing a second area in the three-dimensional space, the second area being different from the first area; 11. The terminal device according to any one of appendices 8 to 10, wherein the output means outputs an image to be displayed to the user based on the second point cloud data and the fourth point cloud data.

[0178] According to the above configuration, the receiving means can receive point cloud data having different point cloud densities for each region in the three-dimensional space.

[0179] (Appendix 12) the receiving means receives the second point cloud data in response to the line of sight information and transmission information regarding the received second point cloud data; 12. The terminal device according to any one of appendices 8 to 11, wherein the output means outputs an image to be displayed to the user based on the second point cloud data and the transmission information.

[0180] According to the above configuration, the output means can output an image based on the second point cloud data in response to the transmitted information, and a three-dimensional video can be suitably displayed.

[0181] (Appendix 13) Transmitting second point cloud data extracted from the first point cloud data representing a first region in a three-dimensional space; receiving the second point cloud data in response to line of sight information relating to a line of sight of a user; A video transmission method, comprising: outputting an image to be displayed to the user based on the second point cloud data.

[0182] According to the above configuration, it is possible to deliver a suitable video according to the line of sight of a plurality of users.

[0183] (Appendix 14) In the step of receiving the second point cloud data, Specifying an image quality to be displayed to the user; The video transmission method according to claim 13, further comprising receiving the second point cloud data according to the specified image quality.

[0184] According to the above configuration, it is possible to receive the second point cloud data according to the image quality of the video.

[0185] (Appendix 15) The video transmission method according to claim 13 or 14, wherein the second point cloud data includes a plurality of point cloud data extracted from the first point cloud data and having a different point cloud density from the first point cloud data.

[0186] According to the above configuration, by synthesizing and outputting any one of the plurality of point cloud data, it becomes possible to display images of different qualities.

[0187] (Appendix 16) In the step of transmitting the second point cloud data, Transmitting fourth point cloud data extracted from third point cloud data representing a second region in the three-dimensional space, the second region being different from the first region; In the step of receiving the second point cloud data, receiving the fourth point cloud data in response to the line of sight information; In the step of outputting the image, The video transmission method according to any one of appendices 13 to 15, further comprising outputting an image to be displayed to the user based on the second point cloud data and the fourth point cloud data.

[0188] According to the above configuration, it is possible to transmit point cloud data having different point cloud densities for each region in a three-dimensional space.

[0189] (Appendix 17) The video transmission method includes: generating transmission information relating to the second point cloud data; In the step of transmitting the second point cloud data, Transmitting the second point cloud data and the transmission information; In the step of receiving the second point cloud data, receiving the second point cloud data in response to the line of sight information and the received transmission information; In the step of outputting the image, The video transmission method according to any one of appendices 13 to 16, further comprising: outputting an image to be displayed to the user based on the second point cloud data and the transmission information.

[0190] According to the above configuration, it is possible to output an image based on the second point cloud data in response to the transmitted information, and it is possible to suitably display a three-dimensional video.

[0191] (Appendix 18) Information regarding the position of the first region in the three-dimensional space; Information regarding a correspondence relationship between the first region and the second point cloud data; and Information regarding the point cloud density of the second point cloud data 18. The video transmission method according to claim 17, comprising at least one of the following:

[0192] According to the above configuration, it is possible to more suitably display three-dimensional video images.

[0193] (Appendix 19) In the step of transmitting the second point cloud data, 19. The video transmission method according to any one of appendices 13 to 18, further comprising collecting line-of-sight information of the user, and transmitting the second point cloud data in accordance with the line-of-sight information.

[0194] According to the above configuration, it is possible to easily specify the second point cloud data to be transmitted in accordance with the user's line of sight information.

[0195] (Appendix 20) A space dividing means for dividing an image of a target space in a three-dimensional space into a plurality of regions; a point cloud data extraction means for extracting second point cloud data from point cloud data representing a first area obtained by division by the space division means; a transmitting means for transmitting the second point cloud data.

[0196] (Appendix 21) 21. The server device according to claim 20, wherein the point cloud data extraction means extracts the second point cloud data including a plurality of point cloud data from the first point cloud data.

[0197] (Appendix 21) The video transmission system according to claim 8, wherein the acquisition data identification means identifies the point cloud data to be acquired so as to maximize the total value of the viewing quality.

[0198] (Appendix 22) 9. The video transmission system according to any one of claims 1 to 8, wherein the output means synthesizes and displays a plurality of point cloud data included in the second point cloud data.

[0199] (Appendix 23) The method includes: transmitting second point cloud data extracted from first point cloud data representing a first region in a three-dimensional space; receiving the second point cloud data in response to line-of-sight information relating to a line of sight of a user; and outputting an image based on the second point cloud data.

[0200] The video transmission system may further include a memory, and the memory may store a program for causing the processor to execute the transmitting process, the receiving process, and the outputting process. The program may be recorded in a computer-readable, non-transitory, tangible recording medium.

[0201] (Appendix 24) The method includes: acquiring line-of-sight information indicating a line of sight of a user; receiving second point cloud data extracted from first point cloud data representing a first area in a three-dimensional space in response to the line of sight information; and outputting an image based on the received second point cloud data.

[0202] The terminal device may further include a memory that stores a program for causing the processor to execute the acquiring process, the receiving process, and the output process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium.

[0203] (Appendix 25) The apparatus includes at least one processor, the processor performing a process of dividing an image of a target space in a three-dimensional space into a plurality of regions; A process of extracting second point cloud data from the point cloud data representing the first region after division; and transmitting the second point cloud data.

[0204] The server device may further include a memory that stores a program for causing the processor to execute the division process, the extraction process, and the transmission process. The program may be recorded in a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0205] 1, 1A~1D Server equipment 2,2A Terminal Equipment 11 Transmitter 12 Space division part 13 Point cloud data extraction section 14 Transmission information generation unit 15,25 Communications Department 16 Transmission data specification part 17 Divided area recognition unit 18 Gazing point identification section 19. Synthetic Data Receiving Unit 21 Receiving section 22 Output section 23 Gaze Information Collection Unit 24 Acquired data identification section 100,100A~100D Video transmission system

Claims

1. A transmitting means for transmitting second point cloud data extracted from the first point cloud data representing a first area in a three-dimensional space; A receiving means for receiving the second point cloud data in response to line-of-sight information relating to a user's line of sight; an output unit that outputs an image to be displayed to the user based on the second point cloud data; the second point cloud data includes a plurality of point cloud data extracted from the first point cloud data and having a point cloud density different from that of the first point cloud data; the receiving means selects at least one point cloud data from the plurality of point cloud data having different point cloud densities in accordance with the line of sight information, and receives the selected point cloud data as the point cloud data to be displayed in the first area; The output means outputs an image to be displayed to the user based on the point cloud data to be displayed in the first area. Video transmission system.

2. A specifying means for specifying an image quality to be displayed to the user, The video transmission system according to claim 1 , wherein the receiving means receives the second point cloud data in accordance with the specified image quality.

3. The transmitting means is Transmitting fourth point cloud data extracted from third point cloud data representing a second region in the three-dimensional space, the second region being different from the first region; the receiving means receives the fourth point cloud data in response to the line of sight information; The video transmission system according to claim 1 , wherein the output unit outputs an image to be displayed to the user based on the second point cloud data and the fourth point cloud data.

4. the video transmission system includes a transmission information generating means for generating transmission information related to the second point cloud data; The transmitting means transmits the second point cloud data and the transmission information, the receiving means receives the second point cloud data in response to the line of sight information and the received transmission information; 4. The video transmission system according to claim 1, wherein the output means outputs an image to be displayed to the user based on the second point cloud data and the transmission information.

5. The transmission information is Information regarding the position of the first region in the three-dimensional space; Information regarding a correspondence relationship between the first region and the second point cloud data; and Information regarding the point cloud density of the second point cloud data The video transmission system according to claim 4, comprising at least one of the following:

6. The video transmission system according to any one of claims 1 to 5, wherein the transmission means transmits the second point cloud data in response to line-of-sight information of the user.

7. A gaze information acquisition means for acquiring gaze information indicating a user's gaze; a receiving means for receiving second point cloud data extracted from first point cloud data representing a first area in a three-dimensional space in response to the line-of-sight information; an output unit that outputs an image to be displayed to the user based on the second point cloud data received by the receiving unit, the second point cloud data includes a plurality of point cloud data extracted from the first point cloud data and having a point cloud density different from that of the first point cloud data; the receiving means selects at least one point cloud data from the plurality of point cloud data having different point cloud densities in accordance with the line of sight information, and receives the selected point cloud data as the point cloud data to be displayed in the first area; The output means outputs an image to be displayed to the user based on the point cloud data to be displayed in the first area. Terminal device.

8. A specifying means for specifying an image quality to be displayed to the user, The terminal device according to claim 7 , wherein the receiving means receives the second point cloud data in accordance with the specified image quality.

9. Transmitting second point cloud data extracted from the first point cloud data representing a first region in a three-dimensional space; receiving the second point cloud data in response to line of sight information relating to a line of sight of a user; and outputting an image to be displayed to the user based on the second point cloud data; Including, the second point cloud data includes a plurality of point cloud data extracted from the first point cloud data and having a point cloud density different from that of the first point cloud data; In the receiving, at least one point cloud data is selected from the plurality of point cloud data having different point cloud densities according to the line of sight information, and is received as the point cloud data to be displayed in the first area; In the outputting, an image to be displayed to the user is output based on the point cloud data to be displayed in the first area. Video transmission method.

Images