Information processing device, information processing method, and program

Abstract

This information processing device acquires: information about a transmission band available for transmission of data; a three-dimensional model of a subject in a three-dimensional space; and movement information for the subject and / or a virtual viewpoint in the three-dimensional space. The information processing device then determines a transmission data generation method on the basis of the transmission band information and the movement information, and generates, on the basis of the determined transmission data generation method, transmission data for a three-dimensional model or transmission data for a two-dimensional image based on the three-dimensional model.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing technique for generating transmission data.

[0002] A technology is known in which a three-dimensional model is generated from subject images captured from multiple viewpoints by a plurality of different imaging devices (hereinafter referred to as cameras), and the three-dimensional model is used to generate an image seen from a virtual viewpoint (hereinafter referred to as a virtual viewpoint image). One example of a system configuration in which a user can view the virtual viewpoint image is one in which a server generates transmission data such as the three-dimensional model or the virtual viewpoint image and sends the transmission data to a user terminal. However, in this system configuration, the amount of transmission data may exceed the transmission bandwidth, resulting in unintended image loss.

[0003] In response to this, Patent Document 1 discloses a technology that enables efficient reduction in the amount of three-dimensional model data for reproducing virtual viewpoint images. The technology in Patent Document 1 generates model data of multiple layers with different data amounts for each subject, and transmission data is composed of model data of a layer determined for each subject based on attribute data that associates the attributes of the content with the required layer.

[0004] JP 2019-54488 A

[0005] According to the technology of Patent Document 1 mentioned above, it is possible to reduce the amount of transmission data so that it fits within the transmission band, but there is a demand for a technology that can reduce the amount of transmission data more efficiently.

[0006] Therefore, an object of the present invention is to make it possible to more efficiently reduce the amount of data to be transmitted.

[0007] The information processing device of the present invention is characterized by having a bandwidth information acquisition means for acquiring information on a transmission bandwidth available for data transmission, a model acquisition means for acquiring a three-dimensional model of a subject in three-dimensional space, a movement information acquisition means for acquiring movement information of at least one of the subject and a virtual viewpoint in the three-dimensional space, a determination means for determining a transmission data generation method based on the transmission bandwidth information and the movement information, and a data generation means for generating transmission data of the three-dimensional model or transmission data of a two-dimensional image generated from the three-dimensional model based on the transmission data generation method.

[0008] 1 is a diagram illustrating an example of the overall configuration of a system according to an embodiment. FIG. 2 is an explanatory diagram of a use case in the system according to an embodiment. FIG. 3 is a diagram illustrating an example of the functional configuration of a transmission data generation unit according to a first embodiment. FIG. 4 is a diagram illustrating an example of the hardware configuration of a transmission data generation unit. FIG. 5 is a flowchart of transmission data generation processing according to the first embodiment. FIG. 6 is a flowchart of transmission data reduction method determination processing according to the first embodiment. FIG. 7 is an explanatory diagram of a method of reducing time direction resolution according to the first embodiment. FIG. 8 is an explanatory diagram of a method of reducing spatial direction resolution according to the first embodiment. FIG. 9 is a diagram illustrating a specific example of when a data amount reduction method is determined according to the first embodiment. FIG. 10 is a diagram illustrating the functional configuration of a transmission data generation unit according to a second embodiment. FIG. 11 is a flowchart of transmission data generation processing according to the second embodiment. FIG. 12 is a flowchart of transmission data reduction method determination processing according to the second embodiment. FIG. 13 is a diagram illustrating a specific example of when a data amount reduction method is determined according to the second embodiment.

[0009] Embodiments of the present invention will be described below with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily required for the solution of the present invention. The configurations of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). In the following embodiments, identical or similar components and processing steps will be assigned the same reference numerals, and redundant descriptions will be omitted. Note that when reference numerals assigned to identical or similar components differ only in the alphabet suffixed to the numerals, they indicate different instances of components having the same function. For example, cameras 102A and 102B in FIG. 1, which will be described later, indicate different instances having the same function. Note that having the same function refers to having at least a specific function (such as an imaging function in the case of a camera). For example, cameras 102A and 102B may have different functions and performance, for example.

[0010] <First Embodiment> Fig. 1 is a diagram showing an example of the overall configuration of a system 100 to which an information processing device according to this embodiment is applied. The system 100 according to this embodiment is configured to include multiple image capture devices (cameras 102A to 102P), a three-dimensional model generation unit 103, and a transmission data generation unit 104. The cameras 102A to 102P and the three-dimensional model generation unit 103, and further the three-dimensional model generation unit 103 and the transmission data generation unit 104, are connected via a LAN (Local Area Network) (not shown). Note that the connection mode is not limited to a LAN, and may be either a wired connection or a wireless connection.

[0011] The cameras 102A-102P are arranged at different positions around a field 101, which is an example of a subject to be photographed. Within the field 101, there is a subject to be three-dimensionally modeled. In the example of FIG. 1 , taking a sport (soccer in the example) as the subject to be photographed, the subject includes players playing the sport, equipment, tools, and the like. In the example of FIG. 1 , the subjects, namely, soccer players and a ball, are schematically illustrated as three-dimensional modeling subjects 105A-105D. Hereinafter, unless otherwise required, the cameras 102A-102P will be simply referred to as cameras 102, and the three-dimensional modeling subjects 105A-105D will be simply referred to as three-dimensional modeling subjects 105. These cameras 102 transmit all or part of the images they capture to the three-dimensional model generation unit 103.

[0012] The three-dimensional model generation unit 103 is a server that performs a three-dimensional model generation process using images acquired from each camera 102. The three-dimensional model generation unit 103 performs a shape estimation process to generate a three-dimensional model necessary for generating a virtual viewpoint image using images captured by each camera 102. The shape estimation process uses, for example, a volume intersection method. In the volume intersection process, the three-dimensional model generation unit 103 first tiled a unit volume of rectangular parallelepipeds called voxels in the target space for generating the three-dimensional model. The three-dimensional model generation unit 103 then projects the foreground silhouette of a target object of interest extracted from the captured image of each camera 102 onto a group of voxels within the imaging range of each camera 102, and removes voxels that fall outside the foreground silhouette. The three-dimensional model generation unit 103 performs this process for all cameras 102 to generate a three-dimensional model. At the same time, the 3D model generation unit 103 uses image information contained in the foreground silhouette of each camera 102 as a texture image and applies it to the 3D model generated as described above, thereby coloring the 3D model. Note that in the following description of the embodiment, the 3D model will be described as a 3D point cloud, which is a collection of colored voxels. Of course, the 3D model is not limited to this, and may be, for example, a 3D model based on a 3D polygon mesh.

[0013] The transmission data generation unit 104 is an application example of the information processing device according to this embodiment, and is a server that generates transmission data to be sent to a user terminal from the three-dimensional model generated by the above-described three-dimensional model generation unit 103. The transmission data generation unit 104 of this embodiment generates transmission data from the three-dimensional model data (three-dimensional point cloud data, which is a collection of colored voxels) generated by the three-dimensional model generation unit 103 and transmits the transmission data to the user terminal. Note that the transmission data generation unit 104 may also generate transmission data from two-dimensional image data generated by rendering using virtual viewpoint information indicating the position of a virtual viewpoint and the direction of a line of sight, and transmit the transmission data to the user terminal.

[0014] In this embodiment, an example will be described in which transmission data is generated from pre-rendering three-dimensional model data (three-dimensional point cloud data, which is a collection of colored voxels) and transmitted to a user terminal. In this way, when pre-rendering three-dimensional model data is transmitted to a user terminal, a rendering process is performed in the user terminal, and a virtual viewpoint image, which is a two-dimensional image, is generated and displayed. Note that the information processing device according to this embodiment may include not only the transmission data generation unit 104 but also the above-mentioned three-dimensional model generation unit 103.

[0015] FIG. 2 illustrates an example use case 200 in which transmission data generated by the transmission data generation unit 104 is sent to user terminals 210A to 210C, which then generate and display virtual viewpoint images from the transmission data. As shown in FIG. 2, the transmission data generation unit 104 is connected to the user terminals 210A to 210C via the Internet 201. Hereinafter, unless otherwise required, the user terminals 210A to 210C will be simply referred to as user terminals 210. The user terminal 210 includes the functions of a data receiving unit 213, a display unit 214, an operation unit 211, and an information transmission unit 212. Note that the transmission path is not limited to the Internet 201 and may be any network connecting a server (transmission data generation unit 104) that generates the transmission data and a client terminal (user terminal 210) that displays the virtual viewpoint image.

[0016] The operation unit 211 acquires the position of a virtual viewpoint that the user desires to view and the line of sight direction of the virtual viewpoint, in other words, the position of a virtual camera and its shooting direction, as operation information input by the user of the user terminal 210. Examples of the operation unit 211 capable of acquiring the position and direction of the virtual viewpoint include input devices that can specify a position and direction in three-dimensional space, such as a joystick or game controller. Other examples of the operation unit 211 include a head-mounted display equipped with a sensor that can detect the direction in which the user's face is facing (line of sight direction) when worn by the user on the head.

[0017] Virtual viewpoint information including the position and direction of the virtual viewpoint input by the user via the operation unit 211 is sent to the display unit 214 or the information transmission unit 212. For example, in the case where transmission data of three-dimensional model data is sent from the transmission data generation unit 104, the virtual viewpoint information is sent to the display unit 214. In addition, in the case where transmission data of two-dimensional image data is sent from the transmission data generation unit 104, the virtual viewpoint information is sent to the information transmission unit 212.

[0018] Additionally, virtual viewpoint information is also sent to the information transmission unit 212 in the second embodiment described later.

[0019] When the information transmission unit 212 receives virtual viewpoint information from the operation unit 211 , the information transmission unit 212 transmits the virtual viewpoint information to the transmission data generation unit 104 via the Internet 201 .

[0020] When the data receiving unit 213 of the user terminal 210 receives the transmission data from the transmission data generating unit 104 , it sends the received transmission data to the display unit 214 .

[0021] The display unit 214 displays a virtual viewpoint image based on the transmission data received by the data receiving unit 213. For example, in a case where transmission data of a three-dimensional model is sent from the transmission data generating unit 104, the display unit 214 performs rendering processing based on the three-dimensional model data and virtual viewpoint information acquired by the operation unit 211 to generate and display a virtual viewpoint image (two-dimensional image). Furthermore, in a case where two-dimensional image data (virtual viewpoint image) is sent from the transmission data generating unit 104, the display unit 214 displays the virtual viewpoint image.

[0022] For example, in the case of transmitting transmission data of three-dimensional model data, the transmission data generation unit 104 generates transmission data of the three-dimensional model sent from the three-dimensional model generation unit 103 and sends it to the user terminal 210. Furthermore, in the case of transmitting transmission data of two-dimensional image data, for example, the transmission data generation unit 104 receives virtual viewpoint information sent from the user terminal 210 and performs rendering using the virtual viewpoint information and the three-dimensional model to generate two-dimensional image data. The transmission data generation unit 104 then transmits the transmission data of the two-dimensional image data to the user terminal 210.

[0023] Here, when generating transmission data, the transmission data generation unit 104 according to the first embodiment acquires transmission bandwidth information indicating the transmission bandwidth available on the transmission path and movement information of the subject in three-dimensional space. Then, based on the available transmission bandwidth information and the movement information of the subject, the transmission data generation unit 104 generates transmission data with an amount of data that fits within the available transmission bandwidth. In other words, if the amount of transmission data does not fit within the available transmission bandwidth, the transmission data generation unit 104 reduces the amount of transmission data so that it fits within the available transmission bandwidth.

[0024] Hereinafter, the process of generating transmission data in the transmission data generating unit 104 of this embodiment will be described using an example in which the transmission data is three-dimensional model data. Note that the case in which the transmission data is rendered two-dimensional image data will be described later.

[0025] The three-dimensional model data is a three-dimensional point cloud data of a three-dimensional modeled object 105, such as that shown in FIG. 1 , and is point cloud data for each object existing in a continuous area in three-dimensional space. Therefore, the transmission data generation unit 104 can acquire position information of the three-dimensional modeled object 105 in three-dimensional space (position information of the object) by calculating the geometric mean position of the continuous three-dimensional point cloud corresponding to a certain three-dimensional modeled object 105 of interest. The three-dimensional model data is also in chronological order, generated by the three-dimensional model generation unit 103 from images sequentially acquired by the camera 102 for each frame in chronological order. Therefore, the transmission data generation unit 104 can acquire position information of each three-dimensional modeled object 105 in chronological order, and by using this chronological position information, can acquire movement information indicating the movement of the three-dimensional modeled object 105. That is, the transmission data generation unit 104 can acquire movement information of objects, such as players and equipment, shown in FIG. 1 .

[0026] In this embodiment, an example is given in which movement information is obtained from three-dimensional model data, but this is not limited to this, and the three-dimensional model generation unit 103 or the transmission data generation unit 104 may obtain movement information of the subject, etc. from images captured sequentially by the camera 102.

[0027] There are various existing techniques for acquiring information about the movement (motion) of a subject or the like from an image captured by the camera 102, and any of these may be used.

[0028] Furthermore, the transmission data generation unit 104 acquires transmission bandwidth information indicating the available transmission bandwidth based on the usage status of the Internet 201 and the transmission bandwidth previously allocated to the transmission data generation unit 104 and the user terminal 210. The available transmission bandwidth information is a value indicating the amount of data that can be sent per unit time expressed in units such as Byte / second (hereinafter referred to as B / s).

[0029] As described above, the transmission data generation unit 104 of this embodiment acquires movement information of the three-dimensional modeled object 105 and transmission bandwidth information indicating the available transmission bandwidth, and determines a method for generating transmission data that fits within the available transmission bandwidth based on that information.The transmission data generation unit 104 then generates transmission data that fits within the available transmission bandwidth from the three-dimensional model data using the determined transmission data generation method.

[0030] FIG. 3 is a diagram showing an example of the functional configuration of the transmission data generation unit 104 according to this embodiment, which acquires movement information of the three-dimensional modeled object 105 and transmission bandwidth information indicating the available transmission bandwidth, and, based on that information, is able to generate transmission data that fits within the available transmission bandwidth.

[0031] The model acquisition unit 301 acquires the three-dimensional model data 306 generated by the three-dimensional model generation unit 103 and outputs it to the subject position acquisition unit 302, the generation method determination unit 304, and the data generation unit 305. The three-dimensional model data 306 acquired by the model acquisition unit 301 is stored in a storage device such as a RAM (Random Access Memory) (not shown) provided in the transmission data generation unit 104, and is used for subsequent processing. In this embodiment, the storage device sequentially stores at least past three-dimensional model data 306 for a certain period from the present (current time).

[0032] The subject position acquisition unit 302 detects the current position of the three-dimensional modeled object 105 in three-dimensional space based on the three-dimensional model data 306 received from the model acquisition unit 301. As described above, the subject position acquisition unit 302 calculates the geometric mean position of the continuous three-dimensional point group corresponding to the three-dimensional modeled object 105, thereby acquiring position information of the three-dimensional modeled object 105 in three-dimensional space, i.e., position information of the subject. The subject position acquisition unit 302 also stores the acquired position information in the storage device described above. The current position information of the subject acquired by the subject position acquisition unit 302 and past position information for a certain period of time from the present that is already stored in the storage device are then sent to the generation method determination unit 304.

[0033] The bandwidth information acquisition unit 303 acquires available transmission bandwidth information 307 and sends it to the generation method determination unit 304. As described above, the available transmission bandwidth information 307 is a value indicating the amount of data that can be sent per unit time expressed in units such as B / s, and is acquired based on the usage status of the Internet 201 shown in FIG. 2 and a pre-assigned value.

[0034] The generation method determination unit 304 acquires movement information of the subject based on the current position information acquired by the subject position acquisition unit 302 and past position information for a certain period of time stored in the storage device. Furthermore, the generation method determination unit 304 determines a transmission data generation method to be used when the subsequent data generation unit 305 generates transmission data based on the subject movement information, three-dimensional model data 306, and available transmission bandwidth information 307. Details of the process of determining the transmission data generation method in the generation method determination unit 304 will be described later. Then, the generation method determination unit 304 sends information indicating the determined transmission data generation method to the data generation unit 305.

[0035] The data generation unit 305 generates transmission data 308 from the three-dimensional model data 306 based on the transmission data generation method determined by the generation method determination unit 304, and transmits the data to the user terminal 210. When the transmission data is generated from two-dimensional image data, the data generation unit 305 generates the transmission data from the two-dimensional image data generated by performing rendering based on the three-dimensional model data 306 and virtual viewpoint information sent from the user terminal 210. Details of the transmission data generation process in the data generation unit 305 will be described later.

[0036] Next, an example of the hardware configuration of the transmission data generation unit 104 will be described with reference to Fig. 4. As shown in Fig. 4, the hardware configuration of the transmission data generation unit 104 includes a CPU 401, a ROM 402, a RAM 403, an auxiliary storage device 404, a display unit 405, an operation unit 406, a communication I / F 407, and a bus 408.

[0037] The CPU 401 controls the entire transmission data generation unit 104 using the information processing program and data according to this embodiment stored in the ROM 402 or the auxiliary storage device 404, thereby realizing each functional unit of the transmission data generation unit 104 shown in Fig. 3. Note that the transmission data generation unit 104 may have one or more dedicated hardware components different from the CPU 401, and at least a portion of the processing by the CPU 401 may be executed by the dedicated hardware components. Examples of the dedicated hardware components include an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), and a DSP (digital signal processor).

[0038] The ROM 402 stores programs that do not require modification. The RAM 403 temporarily stores programs and data supplied from the ROM 402 or the auxiliary storage device 404, and data supplied from the outside via the communication I / F 407. The auxiliary storage device 404 is configured, for example, with a hard disk drive, and stores various data such as image data and audio data. The information processing program according to this embodiment is stored in the ROM 402 or the auxiliary storage device 404, and is expanded in the RAM 403 and executed by the CPU 401. It is assumed that the aforementioned storage device provided inside the transmission data generation unit 104 is the RAM 403.

[0039] The display unit 405 is configured to have, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) or the like for the user to operate the transmission data generation unit 104. The CPU 401 also operates as a display control unit that controls the display unit 405.

[0040] The operation unit 406 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and receives operations from a user to input various instructions to the CPU 401. The CPU 401 also operates as an operation control unit that controls the operation unit 406.

[0041] The communication I / F 407 is used when the transmission data generation unit 104 communicates with an external device, etc. For example, when the transmission data generation unit 104 is connected to an external device via a wired connection, a communication cable is connected to the communication I / F 407. When the transmission data generation unit 104 has a function of wirelessly communicating with an external device, the communication I / F 407 includes an antenna.

[0042] The bus 408 connects the various components within the transmission data generating unit 104 to transmit information.

[0043] Note that, although Figure 4 shows an example in which the display unit 405 and the operation unit 406 exist inside the transmission data generation unit 104, at least one of the display unit 405 and the operation unit 406 may exist as a separate device outside the transmission data generation unit 104.

[0044] Hereinafter, the process of generating transmission data from the three-dimensional model data 306 based on the three-dimensional model data 306, subject movement information, and available transmission bandwidth information 307 in the transmission data generation unit 104 according to the first embodiment will be described with reference to Figures 5 to 8.

[0045] Fig. 5 is a flowchart showing the flow of the transmission data generation process executed in the transmission data generation unit 104 shown in Fig. 3. In the following explanation of each flowchart, the symbol S represents a processing step (process).

[0046] In S501, the model acquisition unit 301 acquires the three-dimensional model data 306 from the three-dimensional model generation unit 103 as described above. Then, the model acquisition unit 301 outputs the three-dimensional model data 306 to the subject position acquisition unit 302, the generation method determination unit 304, and the data generation unit 305. Furthermore, the model acquisition unit 301 stores the three-dimensional model data 306 in the storage device (RAM 403). After S501, the processing of the transmission data generation unit 104 proceeds to S502.

[0047] Also, in S501, the subject position acquisition unit 302 acquires position information of the subject, which is the three-dimensional modeling target 105. As described above, the subject position acquisition unit 302 detects the current position of the subject, which is the three-dimensional modeling target 105, from the three-dimensional model data 306.

[0048] Furthermore, the subject position acquisition unit 302 sequentially stores the subject position information in the storage device described above.

[0049] Next, in S502, the generation method determination unit 304 confirms the amount of transmission data before reduction. In this embodiment, the amount of transmission data before reduction is the amount of transmission data when the time direction resolution and the spatial direction resolution of certain 3D model data 306 are the default values ​​when the 3D model is generated by the 3D model generation unit 103. In this embodiment, the default value of the time direction resolution is 60 fps (frames per second), and the default value of the spatial direction resolution is 1 mm / voxel. That is, in S502, the generation method determination unit 304 confirms that the amount of transmission data before reduction is the data amount for 60 fps and 1 mm / voxel. After S502, the processing of the transmission data generation unit 104 proceeds to S504.

[0050] In S503, which is performed in parallel with or before S501, the bandwidth information acquisition unit 303 acquires available transmission bandwidth information 307. As described above, the available transmission bandwidth is measured in units of data volume per unit time, such as MB / second. In this embodiment, the unit time is the time of one frame, and the bandwidth information acquisition unit 303 converts the available transmission bandwidth into the amount of data per frame. For example, if the available transmission bandwidth is 120 MB / second and the time direction resolution is 60 fps, the amount of data per frame is 2 MBytes. The bandwidth information acquisition unit 303 then converts the available transmission bandwidth information 307 into the amount of data per frame, and sends the information indicating the amount of transmission data obtained by converting the available transmission bandwidth information 307 into the amount of data per frame to the generation method determination unit 304. After S503, the processing of the transmission data generation unit 104 proceeds to S504.

[0051] In S504, the generation method determination unit 304 determines whether or not the transmission data amount needs to be reduced based on the transmission data amount before reduction confirmed in S502 and the available transmission bandwidth information 307 acquired by the bandwidth information acquisition unit 303 in S503. The generation method determination unit 304 then branches the process depending on the determination result. For example, if the transmission data amount before reduction is equal to or less than the amount of data that can be transmitted in the available transmission bandwidth, the generation method determination unit 304 determines that reduction of the transmission data amount is unnecessary. That is, in this case, the generation method determination unit 304 does not change the temporal resolution and spatial resolution from their default values. On the other hand, if the transmission data amount before reduction exceeds the amount of data that can be transmitted in the available transmission bandwidth, the generation method determination unit 304 proceeds to S505.

[0052] In S505, the generation method determination unit 304 determines a method for reducing the amount of transmission data.

[0053] FIG. 6 is a flowchart showing the detailed flow of the transmission data reduction method determination process executed by the generation method determination unit 304 in S505 of FIG.

[0054] In S601 of FIG. 6 , the generation method determination unit 304 executes a subject movement information acquisition process. That is, the generation method determination unit 304 acquires movement information of the subject (the three-dimensional modeling target 105) based on the current position information of the subject acquired by the subject position acquisition unit 302 and position information for a certain period of time stored in the storage device. In this embodiment, the generation method determination unit 304 calculates a movement speed as the movement information of the subject. For example, the generation method determination unit 304 acquires position information for a certain subject of interest at two different times and calculates the movement speed from the time difference between the position information of the subject at those two different times. Alternatively, for example, the generation method determination unit 304 may calculate the movement speed of the subject of interest using position information for multiple consecutive past times. However, when calculating the movement speed using subject position information for multiple times, there is a possibility that the movement direction of the subject is not linear during the period during which the multiple pieces of position information were acquired. For this reason, the generation method determination unit 304 calculates the movement speed for each of two pieces of position information among the subject position information for multiple times, and calculates the average value of the movement speed for each of the two pieces of position information as the movement speed based on the subject position information for multiple times.

[0055] Next, in the process of S602, the generation method determination unit 304 determines whether the moving speed of the subject calculated in S601 is equal to or greater than a predetermined threshold, and branches the process depending on the determination result. If the moving speed of the subject is equal to or greater than the threshold, the generation method determination unit 304 proceeds to S603, and if the moving speed is less than the threshold, the process proceeds to S604.

[0056] If the process proceeds to S603, the generation method determination unit 304 determines to use a method of reducing the spatial resolution as a method of reducing the amount of transmission data.

[0057] On the other hand, if the process proceeds to S604, the generation method determination unit 304 determines to use a method of reducing the time direction resolution as a method of reducing the amount of transmission data.

[0058] After the method for reducing the amount of transmission data is determined by the process of S603 or S604, the process of the transmission data generation unit 104 proceeds to S506 in FIG.

[0059] In S506, the generation method determination unit 304 branches the process depending on whether the transmission data amount reduction method determined in S505 is a method of reducing spatial resolution or a method of reducing temporal resolution. That is, if the transmission data amount reduction method determined in S505 is a method of reducing temporal resolution, the generation method determination unit 304 proceeds to S507, and if it is a method of reducing spatial resolution, the generation method determination unit 304 proceeds to S508.

[0060] If the process proceeds to S507, the generation method determination unit 304 determines a method for reducing the time direction resolution. As will be described in detail later, if the process proceeds to S507, the generation method determination unit 304 performs a process of determining which of a plurality of different time direction resolutions to use based on the amount of transmission data before reduction and the available transmission bandwidth, as a process for determining a method for reducing the time direction resolution.

[0061] On the other hand, if the process proceeds to S508, the generation method determination unit 304 determines a method for reducing the spatial resolution. As will be described in detail later, if the process proceeds to S508, the generation method determination unit 304 performs a process of determining which of a plurality of different spatial resolutions to use as a method for reducing the spatial resolution, based on the amount of transmission data before reduction and the available transmission bandwidth.

[0062] After the process of S 507 or S 508 , the process of the transmission data generating unit 104 proceeds to S 509 , which is executed by the data generating unit 305 .

[0063] In S509 , the data generation unit 305 generates the transmission data 308 from the three-dimensional model data 306 .

[0064] For example, if it is determined in S504 that data amount reduction is not necessary and the process proceeds to S509, the data generation unit 305 generates the transmission data 308 according to the default time direction resolution and spatial direction resolution.

[0065] For example, if the process proceeds from S504 to S505, and then from S506 to S507 to S509, the data generation unit 305 generates transmission data 308 whose spatial resolution is the default value and whose temporal resolution is the temporal resolution determined in S507.

[0066] For example, if the process proceeds from S504 to S505, and then from S506 to S508 to S509, the data generation unit 305 generates transmission data 308 in which the time direction resolution is the default value and the spatial direction resolution is the time direction resolution determined in S508.

[0067] The transmission data 308 generated in the process of S509 is then output from the transmission data generating unit 104.

[0068] Fig. 7 is a diagram showing an example of the time direction resolution determined by the generation method determination unit 304. Fig. 7 illustrates three time direction resolutions, including a default time direction resolution of 60 fps: large 701, a time direction resolution of 30 fps: medium 702, and a time direction resolution of 15 fps: small 703. Note that these are just examples, and the frame rate of the time direction resolution may be further divided into multiple smaller sets.

[0069] 7 , the time intervals from time t0 to time t1, from time t1 to time t2, and from time t3 to time t4 each indicate a time interval of 60 fps. Also, subject 710 at time t0, subject 711 at time t1, subject 712 at time t2, subject 713 at time t3, and subject 714 at time t4 indicate the position and posture of each subject at each time.

[0070] In the medium temporal resolution 702 in Figure 7, the time intervals from time t0 to time t2 and from time t2 to time t4 each indicate a time interval of 30 fps. Also, subject 710 at time t0, subject 712 at time t2, and subject 714 at time t4 indicate the position and orientation of each subject at each time. That is, in the case of medium temporal resolution 702, the time interval between frames is 1 / 30 second, and the position and orientation of the subject are the position and orientation every 1 / 30 second.

[0071] In the case of the low temporal resolution 703 in Figure 7, the time interval from time t0 to time t4 indicates a time interval of 15 fps. Also, subject 710 at time t0 and subject 714 at time t4 indicate the position and orientation of each subject at each time. That is, in the case of the low temporal resolution 703, the time interval between frames is 1 / 15 seconds, and the position and orientation of the subject are the same every 1 / 15 seconds.

[0072] Fig. 8 is a diagram showing an example of the spatial resolution determined by the generation method determination unit 304. Fig. 8 illustrates three spatial resolutions, including a default value of 1 mm / voxel: large spatial resolution 801, a default value of 2 mm / voxel: medium spatial resolution 802, and a default value of 4 mm / voxel: small spatial resolution 803. Note that these are just examples, and the voxel size of the temporal resolution may be set in even smaller increments.

[0073] In the case of the high spatial resolution 801 in FIG. 8, a three-dimensional model 811 of the subject is constructed by combining voxels 812 of 1 mm / voxel, which is the default value.

[0074] In the case of medium spatial resolution 802 in FIG. 8, a three-dimensional model 813 of the subject is constructed by combining voxels 814 of 2 mm / voxel.

[0075] In the case of small spatial resolution 803 in FIG. 8, a three-dimensional model 815 of the subject is constructed by combining voxels 816 of 4 mm / voxel.

[0076] If it is determined in S504 that data amount reduction is not necessary, the generation method determination unit 304 determines the time direction resolution and the spatial direction resolution to be the default values ​​shown in Figures 7 and 8. That is, the generation method determination unit 304 determines the time direction resolution to be 60 fps (time direction resolution: large) 701, and determines the spatial direction resolution to be 1 mm / voxel (spatial direction resolution: large) 801. As a result, the data generation unit 305 generates transmission data 308 in which the time interval between frames is 1 / 60 second, the position and orientation of the subject are represented by positions and orientations every 1 / 60 second, and a three-dimensional model of the subject is configured by a combination of voxels of 1 mm / voxel.

[0077] On the other hand, when the process proceeds to S507, the generation method determination unit 304 uses the default value of 60 fps temporal resolution: large 701 as a base and determines either 30 fps temporal resolution: medium 702 or 15 fps temporal resolution: small 703. That is, the generation method determination unit 304 determines either 30 fps temporal resolution: medium 702 or 15 fps temporal resolution: small 703, which are temporal resolutions with a lower frame rate, based on the amount of transmission data before reduction and the available transmission bandwidth. Note that when the process proceeds to S507, the generation method determination unit 304 determines the spatial resolution to be 1 mm / voxel spatial resolution: large 801, which is the default value.

[0078] For example, if the temporal resolution: medium 702 is determined, the time interval between frames is 1 / 30 seconds. In this case, the data generation unit 305 reduces the amount of transmission data by selecting frames at predetermined intervals from each frame of the base 60 fps, that is, by selecting, for example, every other frame. In the example of FIG. 7 , the data generation unit 305 selects the frames at times t0, t2, and t4 and deletes the frames at times t1 and t3 to generate data of 30 fps with a temporal resolution: medium 702. As a result, the amount of transmission data when the temporal resolution: medium 702 is determined is reduced to half of the amount of transmission data before reduction, that is, the temporal resolution: high 701.

[0079] For example, if the temporal resolution: small 703 is determined, the time interval between frames is 1 / 15 seconds. In this case, the data generation unit 305 reduces the amount of transmission data by selecting frames at predetermined intervals from each frame of the base 60 fps, i.e., by selecting every third frame, for example. In the example of FIG. 7 , the data generation unit 305 selects the frames at time t0 and time t4 and deletes the frames at time t1, time t2, and time t3, thereby generating data of 15 fps with the temporal resolution: small 703. As a result, the amount of transmission data when the temporal resolution: small 703 is determined is reduced to 1 / 4 of the amount of transmission data before reduction, which is the temporal resolution: large 701.

[0080] The method of reducing the temporal resolution is not limited to simply deleting frames as described above, but may also involve interpolating data of frames that are actually used as transmission data using data of unused frames. For example, the interpolation may be performed by blurring voxel information with respect to the relative directions of voxels of unused frames.

[0081] Although the present embodiment has been described taking as an example a case where the transmission data is three-dimensional model data, a similar method of reducing the temporal resolution can also be used when the transmission data is rendered two-dimensional image data. That is, even when the transmission data is rendered two-dimensional image data, a method of deleting unused frames can be used depending on the required reduction in the amount of transmission data.

[0082] Also, if the process proceeds to S508, the generation method determination unit 304 determines either a spatial resolution of 2 mm / voxel: medium 802 or a spatial resolution of 4 mm / voxel: small 803 based on the default value of a spatial resolution of 1 mm / voxel: large 801.

[0083] That is, based on the amount of transmission data before reduction and the available transmission bandwidth, the generation method determination unit 304 determines either a spatial resolution with a large voxel size, namely, 2 mm / voxel spatial resolution: medium 802 or 4 mm / voxel spatial resolution: small 803. Note that when proceeding to S508, the generation method determination unit 304 determines the temporal resolution to be the default value, 60 fps temporal resolution: large 701.

[0084] For example, if the spatial resolution is determined to be medium 802, a three-dimensional model 813 of the subject will be constructed from voxels 814 of 2 mm / voxel. In this case, the data generation unit 305 selects, for example, one of eight 1 mm voxels included in the 2 mm voxel. Alternatively, the data generation unit 305 may determine whether or not a voxel is present by selecting one of the eight 1 mm voxels by majority vote, or by comparing the eight 1 mm voxels with a threshold value. Furthermore, the color to be applied to the voxel may be selected from a single representative point, or may be the average color of the eight 1 mm voxels.

[0085] Furthermore, for example, if spatial resolution: low 803 is determined, a three-dimensional model 815 of the subject will be constructed from 4 mm / voxel voxels 816. In this case, the data generation unit 305 also determines whether or not a voxel is present by, for example, selecting one of the 1 mm voxels included in the 4 mm voxels, or by a majority vote or threshold value of those 1 mm voxels. Note that the color to be applied to a voxel when spatial resolution: low 803 is determined may be determined by selecting a representative point or using an average value, as described above.

[0086] The reduction in transmission data volume due to spatial resolution, as described above, when calculated solely in terms of the number of voxels, is 1 / 8 for medium spatial resolution 802 and 1 / 64 for small spatial resolution 803 compared to the pre-reduction high spatial resolution 801. However, voxels that constitute the interior of a three-dimensional model generally do not have the same amount of data as those on the surface. Therefore, when reducing the transmission data volume due to spatial resolution, the reduction in transmission data volume is estimated based on changes in the surface area of ​​the three-dimensional model. As a result, the reduction in transmission data volume due to spatial resolution is approximately 1 / 4 for medium spatial resolution 802 and approximately 1 / 16 for small spatial resolution 803 compared to the pre-reduction high spatial resolution 801. Note that the estimated reduction depends on how the voxel data that constitutes the three-dimensional model is stored, as described above, and therefore must be calculated according to the data format of the applicable system.

[0087] In this embodiment, the case where the transmitted data is three-dimensional model data has been described as an example. However, if the transmitted data is rendered two-dimensional image data, the spatial resolution can be reduced by reducing the rendering resolution. This process can be achieved by determining the amount of transmitted data reduction corresponding to medium and small spatial resolutions and determining the rendering resolution. For example, the amount of transmitted data can be set to approximately half for medium spatial resolution and approximately one-quarter for small spatial resolution, and the rendering resolution can be set to achieve these data amounts.

[0088] FIG. 9 is a diagram showing specific examples of numerical values ​​used when determining a data amount reduction method in the transmission data generation unit 104 of the first embodiment described above.

[0089] 9 , case (a) 911, case (b) 912, and case (c) 913 illustrated in case 901 each show an example of a situation in which the shape and movement speed of the subject are different. For each case 901, Fig. 9 shows examples of numerical values ​​for the number of subjects of interest 902, the amount of data transmitted per frame 903, the amount of data transmitted per second 904, the available transmission bandwidth 905, the subject movement speed 906, and the movement speed threshold 907.

[0090] The process of determining the data amount reduction method for each of the cases (a) 911 to (c) 913 will be described below in accordance with the processing steps of the flowcharts of FIGS.

[0091] The example of case (a) 911 shows a case where the 3D model data acquired in S501 of FIG. 5 is data in which the transmission data volume per frame for one person of interest is 5 MB / frame. Here, because the default frame rate is 60 fps, the transmission data volume confirmation process in S502 confirms that the transmission data volume per second before reduction is 300 MB / sec. Furthermore, in the example of case (a) 911, the available transmission bandwidth acquired in S503 is, for example, 400 MB / sec. Therefore, in the data volume reduction necessity determination process in S504 based on this information, it is determined that the data volume fits within the available transmission bandwidth and therefore no reduction in the transmission data volume is necessary. The process then proceeds to S509, where transmission data is generated without reducing the transmission data volume.

[0092] The example of case (b) 912 shows a case where the 3D model data acquired in S501 of FIG. 5 is data for one person of interest, with a transmission data volume of 10 MB / frame per frame. In this case, the transmission data volume confirmation process in S502 confirms that the transmission data volume per second before reduction is 600 MB / sec. Also in this case, the available transmission bandwidth acquired in S503 is, for example, 400 MB / sec. In this case, the data volume exceeds the available transmission bandwidth, so the data volume reduction necessity determination process in S504 determines that a reduction in the transmission data volume is necessary, and the process proceeds to S505.

[0093] When the process proceeds to S505, information on the moving speed of the target subject and the moving speed threshold is acquired as the process of S601 in Fig. 6. In the case (b) 912, the moving speed of the target subject is 15 km / h and the moving speed threshold is 10 km / h, so in the determination process of S602, it is determined that the moving speed of the target subject is equal to or greater than the threshold, and the process proceeds to S603. In other words, in S603, it is determined that the spatial resolution will be reduced, and therefore in Fig. 5, the process proceeds to S508 via the process of S506.

[0094] Then, in S508, a reduction method is determined taking into consideration how much the spatial resolution needs to be reduced to fit within the available transmission bandwidth. In case (b) 912, the amount of transmission data before reduction is 600 MB / sec, and the available transmission bandwidth is 400 MB / sec.

[0095] In other words, if the amount of transmission data before reduction is halved, for example, it can be contained within the available transmission bandwidth. Therefore, in the process of determining a method for reducing the spatial resolution in S508, spatial resolution: medium 802 in Figure 8 is determined, and the amount of transmission data will be 300 MB / sec. When proceeding to S508, the default value of 60 fps temporal resolution: large 701 is used as the temporal resolution, so in S509, transmission data of 60 fps and 2 mm / voxel is generated.

[0096] In other words, the result of the example case (b) 912 means that when the subject's movement speed is above a threshold, emphasis is placed on reproducing the subject's movement, and transmission data is generated that includes a three-dimensional model that maintains the temporal resolution (frame rate).

[0097] In the example of case (c) 913, like the example of case (b) 912 described above, the process proceeds from S501 to S505. In the case of case (c) 913, in S601 of Fig. 6, the moving speed of the target subject is 5 km / h and the moving speed threshold is 10 km / h, so in S602 it is determined that the moving speed of the target subject is less than the threshold, and the process proceeds to S604. That is, in the case of case (c) 913, it is determined in S603 that the time direction resolution is to be reduced, and therefore in Fig. 5 the process proceeds to S507 via the process of S506.

[0098] Then, in S507, a reduction method is determined taking into consideration how much the time direction resolution needs to be reduced so that it fits within the available transmission bandwidth. In case (c) 913, the amount of transmission data before reduction is 600 MB / sec, and the available transmission bandwidth is 400 MB / sec.

[0099] In other words, if the amount of transmission data before reduction is halved, for example, it can be accommodated within the available transmission bandwidth. Therefore, in the process of determining the direction of reduction of the time direction resolution in S507, the time direction resolution: medium 702 in Figure 7 is determined, and the amount of transmission data will be 300 MB / sec. When proceeding to S507, the spatial direction resolution is set to the default value of 1 mm / voxel, spatial direction resolution: large 801, and therefore transmission data of 30 fps and 1 mm / voxel is generated in S509.

[0100] In other words, the result of the example case (c) 913 means that when the subject's movement speed is less than the threshold value, emphasis is placed on reproducing the subject's shape, and transmission data is generated that includes a three-dimensional model that maintains spatial resolution.

[0101] In each of the above-described cases 901, the case where the number of target subjects is one has been described as an example, but the number of target subjects may be multiple. When the number of target subjects is multiple, the moving speed of the target subject may be, for example, the moving speed of the target subject located at the center of the image, the fastest moving speed of the targets, or the average value of the moving speeds of all the target subjects.

[0102] As described above, in the transmission data generation unit 104 of this embodiment, when the amount of data exceeds the available transmission bandwidth, the amount of transmission data can be reduced in accordance with the movement information of the subject. In other words, according to this embodiment, the amount of transmission data can be efficiently reduced.

[0103] <Second Embodiment> In the first embodiment, an example of efficiently reducing the amount of transmission data using movement information of a subject was described. In the following second embodiment, an example of reducing the amount of transmission data using movement information of a virtual viewpoint in addition to movement information of a subject will be described. Note that in the second embodiment, components with the same reference numbers as those in the first embodiment operate in the same manner as those described in the first embodiment, and therefore their description will be omitted. The system configuration of the second embodiment is the same as that in Figure 1, the use case is the same as that in Figure 2, and the hardware configuration of the transmission data generation unit 104 is also the same as that in Figure 4, so illustration and description of those components will be omitted.

[0104] 10 is a diagram showing an example of a functional configuration of the transmission data generation unit 104 according to the second embodiment, which is capable of generating transmission data that fits within the available transmission bandwidth based on movement information of the 3D modeled object 105, information on the available transmission bandwidth, and movement information of the virtual viewpoint. The transmission data generation unit 104 according to the second embodiment generates transmission data 308 that fits within the available transmission bandwidth based on 3D model data 306, available transmission bandwidth information 307, and virtual viewpoint information 1003.

[0105] The transmission data generation unit 104 of the second embodiment includes a virtual viewpoint position acquisition unit 1001. The virtual viewpoint position acquisition unit 1001 receives virtual viewpoint information 1003, which is input by the user via the operation unit 211 and transmitted from the information transmission unit 212 via the Internet 201. The virtual viewpoint position acquisition unit 1001 acquires position information of the virtual viewpoint from the virtual viewpoint information 1003. This virtual viewpoint position information is sent to a generation method determination unit 1002, and the storage device sequentially stores position information of past virtual viewpoints over a certain period of time from the present (current time). The current virtual viewpoint position information acquired from the virtual viewpoint information 1003 and the past virtual viewpoint position information over a certain period of time stored in the storage device are sent to the generation method determination unit 1002.

[0106] The generation method determination unit 1002 of the second embodiment determines a transmission data generation method based on the virtual viewpoint position information, the 3D model data 306, the subject position information (position information of the 3D modeled target 105), and the available transmission bandwidth information 307. Details of the transmission data generation method determination process in the generation method determination unit 1002 of the second embodiment will be described later. Then, the generation method determination unit 1002 sends information indicating the determined transmission data generation method to the data generation unit 305.

[0107] The data generation unit 305 generates and outputs transmission data 308 from the three-dimensional model data 306 or the two-dimensional image data based on the transmission data generation method determined by the generation method determination unit 1002, as in the example of the first embodiment described above.

[0108] Below, with reference to Figures 11 to 13, we will explain the process in which the transmission data generation unit 104 according to the second embodiment generates transmission data from the three-dimensional model data 306 based on the position information of the subject, the position information of the virtual viewpoint, and the available transmission bandwidth information.

[0109] Fig. 11 is a flowchart showing the flow of the transmission data generation process executed by the transmission data generation unit 104 according to the second embodiment shown in Fig. 10. The processes from S501 to S504 are the same as those described in Fig. 5. In the case of the flowchart of Fig. 11, if it is determined in S504 that the amount of data needs to be reduced, the process proceeds to S1101.

[0110] In S1101, the virtual viewpoint position acquisition unit 1001 acquires the position information of the virtual viewpoint from the virtual viewpoint information 1003. The virtual viewpoint position acquisition unit 1001 also stores the position information of the virtual viewpoint in the storage device (RAM 403).

[0111] Next, in the process of S1102, the generation method determination unit 304 determines a method for reducing the amount of transmission data.

[0112] FIG. 12 is a flowchart showing the detailed flow of the transmission data reduction method determination process executed by the generation method determination unit 304 in S1102 of FIG.

[0113] In S1201 of FIG. 12 , the generation method determination unit 1002 acquires the movement speed as movement information of the virtual viewpoint based on the current virtual viewpoint position information acquired by the virtual viewpoint position acquisition unit 1001 and the virtual viewpoint position information for a certain period of time stored in the storage device. The movement speed of the virtual viewpoint can be calculated, for example, from position information for two consecutive times and the time difference between those two times, similar to the calculation process for the movement information of the subject. The movement information of the virtual viewpoint can also be calculated using position information for multiple virtual viewpoints obtained at multiple consecutive past times. However, when calculating the movement speed using the position information for these multiple past virtual viewpoints, there is a possibility that the movement direction of the virtual viewpoint during the period during which the multiple position information was acquired was not linear. Therefore, when calculating the movement speed using multiple past position information, the generation method determination unit 1002 calculates the movement information for each two consecutive positions and calculates the average value as the movement speed.

[0114] Next, in the process of S1202, the generation method determination unit 1002 determines whether the moving speed of the virtual viewpoint is equal to or greater than a predetermined threshold value, and branches the process depending on the determination result. If the moving speed of the virtual viewpoint is equal to or greater than the threshold value, the generation method determination unit 1002 proceeds to S603, whereas if the moving speed of the virtual viewpoint is less than the threshold value, the process proceeds to S601.

[0115] The processes from S601 to S604 are the same as those described with reference to FIG. 6, and therefore, the description thereof will be omitted.

[0116] In the second embodiment, if it is determined in S1202 that the moving speed of the virtual viewpoint is equal to or greater than the threshold value, it is determined in S603 to use a method of reducing the spatial resolution while maintaining the temporal resolution (frame rate), regardless of the moving speed of the subject. In other words, if the moving speed of the virtual viewpoint is fast, the movement of the entire virtual viewpoint image becomes fast regardless of the moving speed of the subject, and therefore it is determined to use a method of reducing the spatial resolution.

[0117] FIG. 13 is a diagram showing a specific example of numerical values ​​used when determining the data amount reduction process in the transmission data generation unit 104 of the second embodiment.

[0118] 13 , case (a) 1311, case (b) 1312, case (c) 1313, and case (d) 1314 illustrated in case 1301 each show an example of a situation in which the shape and movement speed of the subject and the movement speed of the virtual viewpoint are different. For each case 1301, Fig. 13 shows the number of subjects of interest 1302, the amount of data transmitted per frame 1303, the amount of data transmitted per second 1304, the available transmission bandwidth 1305, the subject movement speed 1308, and a threshold value 1309. In the second embodiment, a virtual viewpoint movement speed 1306 and a threshold value 1307 for the virtual viewpoint movement speed are further added.

[0119] The process of determining the data amount reduction method for each of the cases (a) 1311 to (d) 1314 will be described below in accordance with the processing steps of the flowcharts of FIGS.

[0120] In the case of case (a) 1311, the same processing results as those in the example of case (a) 911 in Fig. 9 are obtained in S501, S502, and S503 in Fig. 11. In other words, since the amount of transmission data falls within the available transmission band, it is determined in the data amount reduction necessity determination process of S504 that reduction of the amount of transmission data is unnecessary, and the process proceeds to S509, where transmission data is generated without reducing the amount of transmission data.

[0121] The example of case (b) 1312 shows a case where the 3D model data acquired in S501 requires 10 MB / frame as the transmission data volume per frame for one person of interest. In this case, the transmission data volume confirmation process in S502 confirms that the transmission data volume per second before reduction is 600 MB / sec. Also in this case, the available transmission bandwidth acquired in S503 is, for example, 400 MB / sec. In this case, the data volume exceeds the available transmission bandwidth, so the data volume reduction necessity determination process in S504 determines that a reduction in the transmission data volume is necessary, and the process proceeds to S1101.

[0122] When the process proceeds to S1101, information on the moving speed of the virtual viewpoint and the moving speed threshold is acquired as the processing of S1201 in Fig. 12. In the case (b) 1312, the moving speed of the virtual viewpoint is 20 km / h and the moving speed threshold is 16 km / h, so in the determination processing of S1202, it is determined that the moving speed of the virtual viewpoint is equal to or greater than the threshold, and the process proceeds to S603. In other words, in S603, it is determined that the spatial direction resolution is to be reduced, and therefore in Fig. 11, the process proceeds to S508 via the processing of S506.

[0123] Then, proceeding to S508, a reduction method is determined taking into consideration how much the spatial resolution needs to be reduced to fit within the available transmission bandwidth. In the example of case (b) 1312, the transmission data volume before reduction is 600 MB / sec, and the available transmission bandwidth is 400 MB / sec. In other words, if the transmission data volume before reduction is reduced by, for example, half, it can be fit within the available transmission bandwidth. Therefore, in the spatial resolution reduction method determination process of S508, the spatial resolution: medium 802 in FIG. 8 is determined, and the transmission data volume is 300 MB / sec.

[0124] When the process proceeds to S508, the default time direction resolution of 60 fps (large time direction resolution 701) is used, and therefore transmission data of 60 fps and 2 mm / voxel is generated in S509.

[0125] In other words, the result of the example of case (b) 1312 means that when the moving speed of the virtual viewpoint is equal to or greater than a threshold value, emphasis is placed on reproducing the movement of the virtual viewpoint, and transmission data is generated that includes a three-dimensional model that maintains the temporal resolution (frame rate).

[0126] In the example of case (c) 1313, the process proceeds from S501 to S505, as in the example of case (b) 1312. In case (c) 1313, the moving speed of the virtual viewpoint is 12 km / h and the threshold value for the moving speed is 16 km / h, so in S1201 of Fig. 12 it is determined that the moving speed of the virtual viewpoint is less than the threshold value, and the process proceeds to S601.

[0127] When the process proceeds to S601, the moving speed of the target subject and the threshold value for that moving speed are acquired, as described above. In the example of case (c) 1313, the moving speed of the target subject is 15 km / h and the threshold value for the subject's moving speed is 10 km / h, so in S602 it is determined that the moving speed of the target subject is equal to or greater than the threshold value, and the process proceeds to S603. In other words, it is decided to reduce the spatial resolution, and the process proceeds to S508 via S506 in FIG. 11.

[0128] When the process proceeds to S508, a reduction method is determined taking into consideration how much the spatial resolution needs to be reduced to fit within the available transmission bandwidth. In case (c) 1313, the transmission data volume before reduction is 600 MB / sec, and the available transmission bandwidth is 400 MB / sec. In other words, if the transmission data volume before reduction is reduced by, for example, half, it can fit within the available transmission bandwidth. Therefore, in the process of determining the spatial resolution reduction method in S508, the spatial resolution: medium 802 in FIG. 8 is determined, and the transmission data volume is 300 MB / sec. When the process proceeds to S508, the default value of 60 fps temporal resolution: large 701 is used as the temporal resolution, and therefore transmission data of 60 fps and 2 mm / voxel is generated in S509.

[0129] The result of the example of case (c) 1313 means that even if the moving speed of the virtual viewpoint is less than the threshold, if the moving speed of the subject is equal to or greater than the threshold, importance is attached to reproducing the movement of the subject. In other words, in case (c) 1313, transmission data is generated that includes a three-dimensional model that maintains the temporal resolution (frame rate).

[0130] In the example of case (d) 1314, the process proceeds from S501 to S505, as in the example of case (b) 1312. In case (d) 1314, the moving speed of the virtual viewpoint is 12 km / h and the threshold value for the moving speed is 16 km / h, so in S1201 of Fig. 12 it is determined that the moving speed of the virtual viewpoint is less than the threshold value, and the process proceeds to S601.

[0131] When the process proceeds to S601, the moving speed of the target subject and the threshold value for that moving speed are acquired, as described above. In the example of case (d) 1314, the moving speed of the target subject is 5 km / h and the threshold value for the subject's moving speed is 10 km / h, so in S602 it is determined that the moving speed of the target subject is less than the threshold value and the process proceeds to S604. In other words, it is decided to reduce the time direction resolution, and the process proceeds to S507 via S506 in FIG. 11.

[0132] When the process proceeds to S507, a reduction method is determined taking into consideration how much the temporal resolution needs to be reduced to fit within the available transmission bandwidth. In case (d) 1314, the transmission data volume before reduction is 600 MB / sec, and the available transmission bandwidth is 400 MB / sec. In other words, if the transmission data volume before reduction is halved, for example, it can be fit within the available transmission bandwidth. Therefore, in the process of determining the temporal resolution reduction method in S507, the temporal resolution: medium 702 in FIG. 7 is determined, and the transmission data volume is 300 MB / sec. When the process proceeds to S507, the spatial resolution is set to the default value of 1 mm / voxel, spatial resolution: large 801, and therefore transmission data of 30 fps and 1 mm / voxel is generated in S509.

[0133] The result of the example of case (d) 1314 means that when the moving speed of the virtual viewpoint and the object is less than a threshold, emphasis is placed on reproducing the shape of the object. That is, in the example of case (d) 1314, transmission data including a three-dimensional model that maintains spatial resolution is generated.

[0134] As described above, in the transmission data generation unit 104 of the second embodiment, when the amount of data exceeds the available transmission bandwidth, the amount of transmission data can be reduced based on the movement information of the subject and the virtual viewpoint. In other words, according to this embodiment, the amount of transmission data can be efficiently reduced.

[0135] It should be noted that, while the technology disclosed in the aforementioned Patent Document 1 can reduce the amount of data by reducing the spatial resolution, it cannot reduce the amount of data by reducing the temporal resolution as in the first and second embodiments. With the technology of Patent Document 1, even in a scene in which a target subject remains stationary, for example, the transmission data for the stationary subject continues to be transmitted at a high frame rate, occupying a large portion of the communication bandwidth. In contrast, according to the first and second embodiments, even when the available transmission bandwidth is constrained, an appropriate method for reducing transmission data can be adopted in accordance with the movement of the subject or the virtual viewpoint, thereby minimizing degradation of the 3D model.

[0136] <Other Embodiments> In the first embodiment described above, an example was described in which the amount of transmission data is reduced based on information about the available transmission bandwidth and information about the movement of the subject, and in the second embodiment, an example was described in which the amount of transmission data is reduced by separately handling information about the movement of the subject and information about the movement of the virtual viewpoint. In other embodiments, the amount of transmission data may be reduced based on information about the available transmission bandwidth and information about the movement of the virtual viewpoint. In this case, if the movement speed of the virtual viewpoint is equal to or greater than a threshold, it is determined that the spatial resolution is reduced, and if the movement speed of the virtual viewpoint is less than the threshold, it is determined that the temporal resolution is reduced.

[0137] Furthermore, in the first and second embodiments, only the case where it is determined to reduce only one of the temporal resolution and the spatial resolution to satisfy the available transmission bandwidth has been described, but it is also possible to combine both the reduction of the temporal resolution and the reduction of the spatial resolution. When combining both, it is possible to gradually reduce the amount of transmission data by repeating the processes from S504 to S509 in the flowchart of FIG. 5 for the first embodiment or the flowchart of FIG. 11 for the second embodiment multiple times. When repeating the processes from S504 to S509 multiple times, the threshold values ​​for the subject movement speed and the virtual viewpoint movement speed may be appropriately changed depending on the number of repetitions, and the reduction method may be determined for each repetition loop.

[0138] In the second embodiment, the moving speed of the virtual viewpoint and the moving speed of the subject are acquired independently to determine the transmission data generation method. However, it is also possible to determine the transmission data generation method based on the relative moving speed between the virtual viewpoint and the subject. For example, when the virtual viewpoint and the subject are moving in the same direction (their velocity vectors are roughly the same), the subject as seen from the virtual viewpoint appears almost stationary in the virtual viewpoint image. In this case, maintaining the spatial resolution and reducing the temporal resolution can maintain higher image quality of the virtual viewpoint image.

[0139] The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

[0140] The above-described embodiments are merely examples of specific implementations of the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments.

[0141] That is, the present invention can be implemented in various forms without departing from its technical idea or main characteristics.

[0142] The disclosure of this embodiment includes the following configuration, method, and program.

[0143] (Configuration 1) An information processing device comprising: a bandwidth information acquisition means for acquiring information on a transmission bandwidth available for data transmission; a model acquisition means for acquiring a three-dimensional model of a subject in three-dimensional space; a movement information acquisition means for acquiring movement information between at least one of the subject and a virtual viewpoint in the three-dimensional space; a determination means for determining a transmission data generation method based on the transmission bandwidth information and the movement information; and a data generation means for generating transmission data of the three-dimensional model or transmission data of a two-dimensional image generated from the three-dimensional model based on the transmission data generation method.

[0144] (Configuration 2) The information processing device according to Configuration 1, wherein the transmission data generation method includes a plurality of transmission data generation methods each generating transmission data with a different data amount, and the determining means determines one of the plurality of transmission data generation methods based on the information on the transmission bandwidth and the movement information.

[0145] (Configuration 3) The information processing device according to Configuration 2, wherein the plurality of transmission data generation methods include a default first transmission data generation method and a second transmission data generation method that generates transmission data with a reduced data volume compared to the first transmission data generation method, and the decision means determines whether to select the second transmission data generation method based on information about the transmission bandwidth and the data volume generated by the first transmission data generation method.

[0146] (Configuration 4) The information processing device according to Configuration 3, wherein the second transmission data generation method includes a third transmission data generation method for generating transmission data with reduced time resolution and a fourth transmission data generation method for generating transmission data with reduced spatial resolution, and when it is determined that the second transmission data generation method should be selected, the decision means decides on either the third transmission data generation method or the fourth transmission data generation method based on the transmission bandwidth information and the movement information.

[0147] (Configuration 5) The information processing device according to Configuration 4, wherein the movement information acquisition means acquires a movement speed as the movement information of either the subject or the virtual viewpoint, and the decision means decides to use the fourth transmission data generation method when the movement speed is equal to or greater than a predetermined threshold value.

[0148] (Configuration 6) The information processing device according to Configuration 4 or 5, wherein the movement information acquisition means acquires a movement speed as the movement information of either the subject or the virtual viewpoint, and the decision means decides to use the third transmission data generation method when the movement speed is less than a predetermined threshold value.

[0149] (Configuration 7) The information processing device according to Configuration 4, wherein the movement information acquisition means acquires movement information of the subject and movement information of the virtual viewpoint as movement information of the subject and the virtual viewpoint, and the decision means decides to use the fourth transmission data generation method when the movement speed of the virtual viewpoint is equal to or greater than a predetermined threshold value.

[0150] (Configuration 8) The information processing device described in Configuration 4 or 7, wherein the movement information acquisition means acquires movement information of the subject and movement information of the virtual viewpoint as movement information of the subject and the virtual viewpoint, and the decision means decides to use the fourth transmission data generation method when the movement speed of the virtual viewpoint is less than a predetermined threshold and the movement speed of the subject is equal to or greater than a predetermined threshold.

[0151] (Configuration 9) The information processing device described in any one of configurations 4, 7, and 8, wherein the movement information acquisition means acquires movement information of the subject and movement information of the virtual viewpoint as movement information of the subject and the virtual viewpoint, and the decision means decides to use the third transmission data generation method when the movement speed of the virtual viewpoint is less than a predetermined threshold and the movement speed of the subject is less than a predetermined threshold.

[0152] (Configuration 10) The information processing device described in Configuration 4, wherein the movement information acquisition means acquires a relative movement speed between the subject and the virtual viewpoint as movement information of the subject and the virtual viewpoint, and the decision means decides to use the fourth transmission data generation method when the relative movement speed is equal to or greater than a predetermined threshold value.

[0153] (Configuration 11) The information processing device according to configuration 4 or 10, wherein the movement information acquisition means acquires a relative movement speed between the subject and the virtual viewpoint as movement information of the subject and the virtual viewpoint, and the decision means decides to use the third transmission data generation method when the relative movement speed is less than a predetermined threshold value.

[0154] (Configuration 12) The information processing device according to any one of configurations 4 to 11, wherein the third transmission data generation method is a method for generating transmission data with a frame rate lower than that of the first transmission data generation method.

[0155] (Configuration 13) The information processing device according to any one of configurations 4 to 11, wherein the fourth transmission data generation method is a method for generating transmission data in which the voxel size of the three-dimensional model is larger than that of the first transmission data generation method.

[0156] (Configuration 14) The information processing device according to any one of configurations 4 to 11, wherein the fourth transmission data generation method is a method for generating transmission data with a lower resolution for rendering a two-dimensional image than the first transmission data generation method.

[0157] (Configuration 15) An information processing device described in any one of configurations 5 to 9, characterized in that the movement information acquisition means acquires, as the movement speed of the target subject, an average value of multiple movement speeds calculated using multiple consecutive positions of the target subject at each time, or acquires, as the movement speed of the virtual viewpoint, an average value of multiple movement speeds calculated using multiple consecutive positions of the target subject at each time.

[0158] (Configuration 16) The information processing device according to any one of configurations 5 to 9, wherein the movement information acquisition means acquires one of the movement speed of a subject at a center position among a plurality of subjects, the movement speed of a subject moving fastest, and an average value of the movement speeds of the plurality of subjects.

[0159] (Configuration 17) The information processing device according to configuration 5 or 6, characterized in that the process of acquiring the movement information by the movement information acquisition means and the process of determining the transmission data generation method by the determination means are repeated multiple times, and the threshold value is changed depending on the number of repetitions.

[0160] (Method 1) An information processing method comprising: a bandwidth information acquisition step of acquiring information on a transmission bandwidth available for data transmission; a model acquisition step of acquiring a three-dimensional model of a subject in three-dimensional space; a movement information acquisition step of acquiring movement information between at least one of the subject and a virtual viewpoint in the three-dimensional space; a determination step of determining a transmission data generation method based on the transmission bandwidth information and the movement information; and a data generation step of generating transmission data of the three-dimensional model or transmission data of a two-dimensional image generated from the three-dimensional model based on the transmission data generation method.

[0161] (Program 1) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 17.

[0162] The present invention is not limited to the above-described embodiments, and various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to apprise the public of the scope of the present invention.

[0163] This application claims priority based on Japanese Patent Application No. 2023-120608, filed on July 25, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. An information processing device comprising: a bandwidth information acquisition means for acquiring information on a transmission bandwidth available for data transmission; a model acquisition means for acquiring a three-dimensional model of a subject in three-dimensional space; a movement information acquisition means for acquiring movement information between at least one of the subject and a virtual viewpoint in the three-dimensional space; a determination means for determining a transmission data generation method based on the transmission bandwidth information and the movement information; and a data generation means for generating transmission data of the three-dimensional model or transmission data of a two-dimensional image generated from the three-dimensional model based on the transmission data generation method.

2. The information processing device according to claim 1, characterized in that the transmission data generation method includes a plurality of transmission data generation methods each generating a different amount of transmission data, and the determination means determines one of the transmission data generation methods from among the plurality of transmission data generation methods based on the transmission bandwidth information and the mobility information.

3. The information processing device described in claim 2, characterized in that the multiple transmission data generation methods include a default first transmission data generation method and a second transmission data generation method that generates transmission data with a reduced amount of data compared to the first transmission data generation method, and the decision means determines whether to select the second transmission data generation method based on information on the transmission bandwidth and the amount of data generated by the first transmission data generation method.

4. The information processing device described in claim 3, characterized in that the second transmission data generation method includes a third transmission data generation method for generating transmission data with reduced time resolution and a fourth transmission data generation method for generating transmission data with reduced spatial resolution, and when it is determined that the second transmission data generation method should be selected, the decision means decides on either the third transmission data generation method or the fourth transmission data generation method based on the transmission bandwidth information and the movement information.

5. The information processing device according to claim 4, characterized in that the movement information acquisition means acquires a movement speed as the movement information of either the subject or the virtual viewpoint, and the decision means decides to use the fourth transmission data generation method when the movement speed is equal to or greater than a predetermined threshold value.

6. The information processing device according to claim 4, characterized in that the movement information acquisition means acquires a movement speed as the movement information of either the subject or the virtual viewpoint, and the decision means decides to use the third transmission data generation method if the movement speed is less than a predetermined threshold value.

7. The information processing device according to claim 4, characterized in that the movement information acquisition means acquires movement information of the subject and movement information of the virtual viewpoint as the movement information of the subject and the virtual viewpoint, and the determination means determines to use the fourth transmission data generation method when the movement speed of the virtual viewpoint is equal to or greater than a predetermined threshold value.

8. The information processing device according to claim 4, characterized in that the movement information acquisition means acquires movement information of the subject and movement information of the virtual viewpoint as the movement information of the subject and the virtual viewpoint, and the determination means determines to use the fourth transmission data generation method when the movement speed of the virtual viewpoint is less than a predetermined threshold value and the movement speed of the subject is equal to or greater than a predetermined threshold value.

9. The information processing device according to claim 4, characterized in that the movement information acquisition means acquires movement information of the subject and movement information of the virtual viewpoint as the movement information of the subject and the virtual viewpoint, and the determination means determines to use the third transmission data generation method when the movement speed of the virtual viewpoint is less than a predetermined threshold value and the movement speed of the subject is less than a predetermined threshold value.

10. The information processing device according to claim 4, characterized in that the movement information acquisition means acquires a relative movement speed between the subject and the virtual viewpoint as movement information of the subject and the virtual viewpoint, and the determination means determines to use the fourth transmission data generation method when the relative movement speed is equal to or greater than a predetermined threshold value.

11. The information processing device according to claim 4, characterized in that the movement information acquisition means acquires a relative movement speed between the subject and the virtual viewpoint as movement information of the subject and the virtual viewpoint, and the determination means determines to use the third transmission data generation method when the relative movement speed is less than a predetermined threshold value.

12. An information processing apparatus according to any one of claims 4 to 11, characterized in that the third transmission data generation method is a method for generating transmission data having a frame rate lower than that of the first transmission data generation method.

13. An information processing device according to any one of claims 4 to 11, characterized in that the fourth transmission data generation method is a method for generating transmission data having a larger voxel size of a three-dimensional model than the first transmission data generation method.

14. An information processing device according to any one of claims 4 to 11, characterized in that the fourth transmission data generation method is a method of generating transmission data having a lower resolution for rendering a two-dimensional image than the first transmission data generation method.

15. An information processing device according to any one of claims 5 to 9, characterized in that the movement information acquisition means acquires as the movement speed of the subject of interest an average value of multiple movement speeds calculated using multiple consecutive positions of the subject of interest at each time, or acquires as the movement speed of the virtual viewpoint an average value of multiple movement speeds calculated using multiple consecutive positions of the virtual viewpoint at each time.

16. An information processing device according to any one of claims 5 to 9, characterized in that the movement information acquisition means acquires either the movement speed of a subject at a central position among a plurality of subjects, the movement speed of a subject moving the fastest, or an average value of the movement speeds of the plurality of subjects.

17. An information processing device as described in claim 5 or 6, characterized in that the process of acquiring the movement information by the movement information acquisition means and the process of determining the transmission data generation method by the determination means are repeated multiple times, and the threshold value is changed depending on the number of repetitions.

18. An information processing method comprising: a bandwidth information acquisition step of acquiring information on a transmission bandwidth available for data transmission; a model acquisition step of acquiring a three-dimensional model of a subject in three-dimensional space; a movement information acquisition step of acquiring movement information of at least one of the subject and a virtual viewpoint in the three-dimensional space; a determination step of determining a transmission data generation method based on the transmission bandwidth information and the movement information; and a data generation step of generating transmission data of the three-dimensional model or transmission data of a two-dimensional image generated from the three-dimensional model based on the transmission data generation method.

19. A program causing a computer to function as an information processing device having: a bandwidth information acquisition means for acquiring information on a transmission bandwidth available for data transmission; a model acquisition means for acquiring a three-dimensional model of a subject in three-dimensional space; a movement information acquisition means for acquiring movement information between at least one of the subject and a virtual viewpoint in the three-dimensional space; a determination means for determining a transmission data generation method based on the transmission bandwidth information and the movement information; and a data generation means for generating transmission data of the three-dimensional model or transmission data of a two-dimensional image generated from the three-dimensional model based on the transmission data generation method.

Images