Transmission device, decoding device, transmission system, transmission method, and transmission program
The system optimizes point cloud data transmission by selecting between RAW and GPCC files based on environmental conditions, addressing inefficiencies in existing methods and reducing both transmission time and decoding costs.
Patent Information
- Application Number
- PCT/JP2024/016892
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2024-05-02
- Publication Date
- 2024-11-14
AI Technical Summary
Current techniques for transmitting point cloud data are inefficient in reducing the reception period and cost, as they either result in high decoding costs due to high communication speed and low client performance or prolonged reception times due to low communication speed, especially when using GPCC compression.
A system that determines a combination of RAW and GPCC files based on environmental information to optimize transmission, selecting RAW files for high-bandwidth environments and GPCC files for low-bandwidth environments, thereby reducing the overall reception period and cost.
This approach speeds up the display process by selecting the appropriate file type based on environmental conditions, minimizing both transmission time and decoding costs, thus enhancing the efficiency of point cloud data reception.
Smart Images

Figure JP2024016892_14112024_PF_FP_ABST
Abstract
Description
Transmitting device, decoding device, transmitting system, transmitting method, and transmitting program
[0001] The present invention relates to a transmitting device, a decoding device, a transmitting system, a transmitting method, and a transmitting program.
[0002] Techniques for transmitting point cloud data are known (see, for example, Patent Documents 1 and 2). Such techniques include a technique in which point cloud data is divided based on space or resolution, and the resulting coded divided point cloud data is transmitted to a decoding device.
[0003] One example is a technique for dividing point cloud data into uncompressed divided point cloud data containing coordinate information such as xyz and color information such as RGB (Red-Green-Blue), transmitting the encoded uncompressed divided point cloud data to a decoding device, and then decompressing and displaying the decoded point cloud data on the decoding device. Another example is a technique for compressing the divided point cloud data using GPCC (Geometry-based Point Cloud Compression), which is a method of compressing point cloud data, saving it as compressed compressed divided point cloud data, transmitting the encoded compressed divided point cloud data to a decoding device, and then decompressing and displaying the decoded point cloud data on the decoding device.
[0004] International Publication No. 2020 / 071116 International Publication No. 2019 / 078000
[0005] However, the above-described techniques may not be able to reduce the time and cost required to receive point cloud data. For example, when uncompressed divided point cloud data divided from point cloud data based on resolution is low-resolution, the number of point cloud data (number of points) is small and the size (file size) is small. On the other hand, when this uncompressed divided point cloud data is high-resolution, the number of point clouds is large and the size is large. In the above-described situation, if the uncompressed divided point cloud data is compressed using GPCC for each resolution level and transmitted to a decoding device, and then decoded by the decoding device, the reception time until the point cloud data is received by the decoding device and costs for storage, decoding, etc. may be incurred.
[0006] For example, even if the uncompressed divided point cloud data is small-sized, low-resolution point cloud data, if it is compressed using GPCC and transmitted to a decoding device, the benefit of reducing the data volume due to GPCC compression is lost, and the decoding cost by the decoding device is high. In an environment where the transmission rate is high (communication speed is fast) and the performance of the decoding device is low, it becomes clear that the decoding cost exceeds the reception time. In an environment where the transmission rate is low (communication speed is slow), it becomes clear that the reception time exceeds the decoding cost. On the other hand, even if the uncompressed divided point cloud data is large-sized, high-resolution point cloud data, if it is not compressed using GPCC and transmitted to a decoding device, it takes a long reception time. One aspect of the present disclosure enables a reduction in the reception time and cost of point cloud data.
[0007] A transmitting device according to one aspect of the present disclosure is a transmitting device having an acquisition unit that acquires environmental information regarding the environment surrounding the transmitting device, a determination unit that determines a transmission target from point cloud data that includes at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition unit, and a transmitting unit that transmits the transmission target determined by the determination unit to a decoding device.
[0008] A decoding device according to one aspect of the present disclosure includes an acquisition unit that acquires environmental information relating to the environment surrounding the decoding device, a determination unit that determines a transmission target from point cloud data that includes at least one of first point cloud data and second point cloud data based on the environmental information, and a receiving unit that receives the transmission target determined by the determination unit from a transmitting device.
[0009] A transmission system according to one aspect of the present disclosure includes a transmission device and a decoding device, wherein the transmission device includes an acquisition unit that acquires environmental information regarding the environment surrounding the transmission device, a determination unit that determines a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition unit, and a transmission unit that transmits the transmission target determined by the determination unit to a decoding device, and the decoding device includes a receiving unit that receives the transmission target determined by the determination unit from the transmission device.
[0010] A transmission method according to one aspect of the present disclosure is a transmission method executed by a transmission device, and includes an acquisition step of acquiring environmental information regarding the environment surrounding the transmission device, a determination step of determining a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition step, and a transmission step of transmitting the transmission target determined by the determination step to a decoding device.
[0011] A transmission program according to one aspect of the present disclosure causes a computer mounted on a transmitting device to execute an acquisition process for acquiring environmental information relating to the environment surrounding the transmitting device, a determination process for determining a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition process, and a transmission process for transmitting the transmission target determined by the determination process to a decoding device.
[0012] FIG. 1 is a diagram illustrating an example of a configuration of a transmission system according to a first embodiment. FIG. 2 is a diagram illustrating point cloud data. FIG. 3 is a diagram illustrating point cloud data. FIG. 4 is a diagram illustrating a transmission system according to the first embodiment. FIG. 5 is a diagram illustrating a transmission system according to the first embodiment. FIG. 6 is a diagram illustrating a transmission system according to the first embodiment. FIG. 7 is a flowchart illustrating an example of processing by a server according to the first embodiment. FIG. 8 is a diagram illustrating a transmission system according to a modified example. FIG. 9 is a diagram illustrating an example of a configuration of a transmission system according to a second embodiment. FIG. 10 is a diagram illustrating a transmission system according to the second embodiment. FIG. 11 is a diagram illustrating a transmission system according to the second embodiment. FIG. 12 is a diagram illustrating a transmission system according to the second embodiment. FIG. 13 is a diagram illustrating an example of processing by a client according to the second embodiment. FIG. 14 is a diagram illustrating an example of a hardware configuration of a device. FIG. 15 is a diagram illustrating a transmission system according to a reference technology. FIG. 16 is a diagram illustrating a transmission system according to the reference technology.
[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same elements are denoted by the same reference numerals, and redundant description may be omitted.
[0014] The present disclosure will be described in the following order: 0. Introduction 1. First embodiment 2. Modification 3. Second embodiment 4. Example of hardware configuration 5. Example of effects
[0015] 0. Introduction As described above, a known technique for transmitting point cloud data involves dividing point cloud data based on space and resolution, and transmitting the resulting encoded divided point cloud data to a decoding device. For example, in such a technique, the decoding device decodes the transmitted encoded uncompressed divided point cloud data into uncompressed divided point cloud data and displays it on a PC (Personal Computer) or the like. As an example, when superimposing uncompressed divided point cloud data related to a specific object on a satellite image, the decoding device selects boxes (voxel data) required for display from pre-prepared boxes, thereby achieving a fast display response time until the uncompressed divided point cloud data is displayed. Before describing embodiments of the present disclosure, a reference technique will be described as an example of such a technique using FIGS. 15 to 17. FIGS. 15 to 17 are diagrams illustrating a transmission system related to the reference technique.
[0016] In the example shown in Figure 15, a server in a transmission system according to the reference technology transmits only encoded RAW files containing encoded RAW data, which is uncompressed point cloud data obtained by encoding uncompressed point cloud data such as uncompressed 3D (Dimensions) point cloud data, to a client. The transmission system according to the reference technology causes the server to perform preparation and the client to display. For example, a point cloud spatial division resolution division encoding processing unit (Point-cloud-converter) in the server generates an encoded RAW file group consisting of multiple encoded RAW files obtained by dividing an uncompressed point cloud file containing uncompressed point cloud data by space and resolution.
[0017] The client identifies a file from the group of encoded RAW files corresponding to a point of interest, which is a region of interest (ROI), over a network (communication path), and converts the file transferred from the server from 3D to 2D for display. Specifically, the client performs the following processes on a web browser. First, the client displays HTML. Next, the client loads a program. For example, the client loads and executes JavaScript (registered trademark). Next, the client acquires and displays a satellite photo as a background. Next, the client displays a transmission target to be displayed from the group of encoded RAW files. For example, the client requests the transmission target from the server, receives the transmission target from the server, decodes the received transmission target, renders the decoded transmission target, and repeats the above process until there are no more transmission targets to display.
[0018] In this case, the client receives and displays a transmission target including encoded, uncompressed 3D point cloud data transferred from the server over a network. Therefore, the network communication bandwidth and other factors have a significant impact on the client's display response, and depending on the communication bandwidth and other factors, it may not be possible to reduce the time and cost required to receive the point cloud data. For example, in environments with low communication bandwidth or transmission rates, the transmission time required becomes apparent. Furthermore, even if the encoded RAW file is a large, high-resolution file, the transmission time is particularly long if it is not compressed and transmitted to the client.
[0019] In the example shown in Figure 16, a server in a transmission system according to the reference technology transmits only encoded GPCC files containing encoded GPCC data, which is encoded compressed point cloud data obtained by encoding compressed point cloud data such as uncompressed 3D (Dimensions) point cloud data, to a client. The transmission system according to the reference technology causes the server to prepare the data and causes the client to display the data. For example, a point cloud spatial division resolution division encoding processing unit (Point-cloud-converter) in the server generates an encoded GPCC file group consisting of multiple encoded GPCC files obtained by dividing a compressed point cloud file containing compressed point cloud data by space and resolution.
[0020] The client identifies a file corresponding to the point of interest from the group of encoded GPCC files over the network, decodes and decompresses the file transferred from the server back to its pre-compressed data, and then converts it from 3D to 2D for display. Specifically, the client performs the following processes on a web browser. First, the client displays HTML. Next, the client loads a program. For example, the client loads and executes JavaScript. Next, the client acquires and displays a satellite photo as a background. Next, the client assembles decoder software such as a low-level virtual machine (LLVM). Next, the client displays a transmission target to be displayed from the group of encoded GPCC files. For example, the client requests the transmission target from the server, receives the transmission target from the server, decodes and decompresses the received transmission target back to its pre-compressed data, renders the decompressed transmission target, and repeats the above process until there are no more transmission targets to display. Note that GPCC compression reduces the number of point cloud data (number of point clouds) and deteriorates color information due to quantization of coordinates and color information according to compression parameters, so even if the client decodes and expands the data to be sent back to its pre-compression state, it is not necessarily expanded to its original size.
[0021] In this case, the client receives the transmission target, including the encoded and compressed 3D point cloud data, transferred from the server over the network, decompresses it, and then displays it. Therefore, while the network communication bandwidth and other factors have little impact on the display response, the client's performance, such as decoding ability, has a significant impact on the display response. Therefore, depending on the client's performance, it may not be possible to reduce the time and cost required to receive the point cloud data. For example, in an environment where the communication bandwidth and transmission rate are high and the client's performance is poor, the decoding cost exceeds the transmission time. Furthermore, even if the encoded GPCC file is small in size and low in resolution, there is no benefit to reducing the data volume through compression, and the decoding cost is particularly high.
[0022] Next, an example of execution of a transmission system according to the reference technology will be described using FIG. 17 . In the example shown in FIG. 17 , an execution example in which a client of the transmission system according to the reference technology displays only RAW files and an execution example in which a client displays only GPCC files are compared side by side. As shown in FIG. 17 , the display response of RAW files is good for about one second after the client starts displaying the transmission target, but as the number of point clouds increases in the latter half, the display response of GPCC files improves. Thus, in the execution example shown in FIG. 17 , in the transmission system displaying GPCC files, the cost of decoding the GPCC files by the client increases as the number of point clouds increases in the latter half.
[0023] In contrast, a transmission system according to an aspect of the present disclosure determines a combination of first point cloud data, such as a RAW file, and second point cloud data, such as a GPCC file, based on environmental information related to the environment, thereby reducing the time and cost required to receive point cloud data. For example, the transmission system according to an aspect of the present disclosure determines (selects) a transmission target so that the RAW file is displayed in the first half and the GPCC file is displayed in the second half, thereby accelerating the display speed of the transmission target and shortening response times, such as display response. Furthermore, while the optimal combination of RAW files and GPCC files changes even for the same server and client depending on environmental information related to communication bandwidth during transmission, a transmission system according to an aspect of the present disclosure can change the combination to an appropriate one depending on the environmental information. Specific techniques will be described in the following embodiments.
[0024] 1. First Embodiment Fig. 1 is a diagram showing an example of the configuration of a transmission system according to a first embodiment. The transmission system 1 includes a server (transmission device) 10 and a client (decoding device) 20. The transmission system 1 acquires environmental information using the server 10, determines a transmission target from point cloud data that includes at least one of first point cloud data and second point cloud data based on the environmental information, and transmits the transmission target to the client 20.
[0025] The environmental information refers to information relating to the environment surrounding the server 10 and the client 20. Details of the environmental information will be described later. The first point cloud data is not particularly limited, but may be realized by, for example, encoded uncompressed point cloud data obtained by encoding uncompressed point cloud data, such as encoded RAW data. The second point cloud data is not particularly limited, but may be realized by, for example, encoded compressed point cloud data obtained by encoding compressed point cloud data, such as encoded GPCC data.
[0026] At least one of the first point cloud data and the second point cloud data may include at least one of 3D point cloud data and a plurality of divided point cloud data divided based on at least one of space and resolution. The divided point cloud data is not particularly limited as long as at least one of the first point cloud data and the second point cloud data is divided. The divided point cloud data may be, for example, first point cloud data, such as uncompressed point cloud data, which has been divided and then compressed, or first point cloud data, such as the divided uncompressed point cloud data, which has been combined and then compressed.
[0027] An example of point cloud data will be described below with reference to FIGS. 2 and 3. FIGS. 2 and 3 are diagrams for explaining point cloud data. In the example shown in FIG. 2, an uncompressed point cloud file including uncompressed point cloud data as first point cloud data is divided into multiple RAW files, which are multiple uncompressed divided point cloud files, by performing spatial division for each space and resolution division (frequency decomposition) for each resolution level by the server 10. The uncompressed point cloud file is also divided into resolution levels from Level 0 to Level 30 by the server 10, and each time the resolution level increases by one, the file is divided into eight RAW files. The server 10 then performs a GPCC compression process on half of the multiple RAW files so that the ratio of RAW files to GPCC files, which are compressed divided point cloud files, is 1:1, and encodes the RAW files and the GPCC files.
[0028] In the above example, the data structure of the point cloud data is a mixture of RAW files and GPCC files for each file (on a file-by-file basis). However, for example, RAW data and GPCC data may be mixed for each box within each RAW file. Also, in the above example, the data structure of the point cloud data is a 1:1 ratio of RAW files to GPCC files. However, for example, the entire data structure may be GPCC files. Also, in the data structure of the point cloud data, the compression parameters of the RAW files and the GPCC files may be the same or different. For example, the data structure of the point cloud data may be a mixture of GPCC files with multiple compression parameters, such as GPCC 1 / GPCC 2 / ... / GPCC 2 / RAW, and RAW files with a single compression parameter. Also, in the above example, the uncompressed point cloud file is divided by the server 10 into resolution levels from Level 0 to Level 30. However, the resolution division levels do not have to be 0 to 30.
[0029] In the above example, the RAW file may be a file of several thousand pages in the ".xyz" format, which contains coordinate information such as xyz and color information such as RGB (Red-Green-Blue), has a small number of point clouds, and has a hierarchical structure realized by a Dir (Directory). The GPCC file may be a file of the ".bin (binary)" format, which has an internal octree structure in addition to the characteristics of the ".xyz" format file, and in which the coordinate information such as xyz and color information contained in the RAW file are compressed by a GPCC compression process. An example of a RAW file and a GPCC file will be described in detail below using FIG. 3 .
[0030] In the example shown in Figure 3, the RAW file and GPCC file use a data format for point cloud spatial division resolution division encoding processing. The RAW file includes a header for point cloud spatial division resolution division encoding processing, coordinate information such as xyz, and color information such as RGB. The RAW file is primarily subjected to transmission processing and display processing. In the transmission processing, the time it takes for the server 10 to transmit the RAW file to the client 20 is proportional to the file size and therefore directly proportional to the number of point clouds. In the display processing of the RAW file, the client 20 performs 3D to 2D rendering.
[0031] On the other hand, a GPCC file includes a header for point cloud spatial division resolution division encoding processing, a GPCC header, a slice header, etc. For example, in addition to transmission processing and display processing, preparation processing and decoding processing are mainly performed on a GPCC file. In transmission processing, the transmission time of the GPCC from the server 10 to the client 20 depends on the compression ratio and is therefore proportional to the file size. In preparation processing, the client 20 receives the encoded GPCC file transmitted (transferred) from the server 10, compiles the LLVM, and initializes the executable binary. In decoding processing, the client 20 performs decoding and decompression processing to decompress the data. In display processing, the client 20 performs rendering from 3D to 2D.
[0032] As shown in the diagram of the relationship between the number of point clouds and size in Figure 3, the file size of a RAW file increases in direct proportion to the increase in the number of point clouds from the initial size of the header for the point cloud spatial division resolution division encoding process. On the other hand, the file size of a GPCC increases rapidly from the initial size of the GPCC header in proportion to the increase in the number of point clouds until halfway through, and then gradually increases. As shown in the diagram of the relationship between the number of point clouds and decompression processing time in Figure 3, the decompression processing time of a RAW file increases gradually in direct proportion to the increase in the number of point clouds. On the other hand, the decompression processing time of a GPCC file increases rapidly in response to the preparation processing, then increases significantly in proportion to the number of point clouds until halfway through, and then gradually increases.
[0033] The transmission system 1 can reduce the time and cost required to receive point cloud data by changing the ratio of the RAW file and the GPCC file in accordance with environmental information related to the data structure, attributes, etc. of the RAW file and the GPCC file. Next, returning to FIG. 1, the configurations of the server 10 and the client 20 of the transmission system 1 will be described in detail.
[0034] The server 10 acquires environmental information, determines a transmission target from point cloud data including at least one of the first point cloud data and the second point cloud data based on the environmental information, and transmits the transmission target to the client 20. In the example shown in FIG. 1 , the server 10 includes a control unit 11, a communication unit (transmission unit) 12, and a storage unit 13.
[0035] The control unit 11 controls the entire server 10. For example, the control unit 11 is configured with one or more processors having internal memory storing programs defining each processing procedure and control data, and the processors execute each process using the programs and internal memory. Examples of the control unit 11 include electronic circuits such as a central processing unit (CPU), a micro processing unit (MPU), and a graphics processing unit (GPU), as well as integrated circuits such as an application specific integrated circuit (ASIC) and a field programmable gate array (FPGA). In the example shown in FIG. 1 , the control unit 11 includes a division unit 111, an encoding unit 112, an acquisition unit 113, and a determination unit 114.
[0036] The dividing unit 111 divides at least one of the first point cloud data and the second point cloud data into a plurality of divided point cloud data based on at least one of space and resolution. For example, the dividing unit 111 divides a point cloud file including first point cloud data such as uncompressed point cloud data into a plurality of RAW files including RAW data for each space and for each resolution level.
[0037] The encoding unit 112 encodes each of the multiple pieces of divided point cloud data divided by the dividing unit 111. For example, the encoding unit 112 encodes multiple pieces of uncompressed divided point cloud data into multiple pieces of encoded uncompressed divided point cloud data, and compresses and encodes multiple pieces of uncompressed divided point cloud data into multiple pieces of encoded compressed divided point cloud data. The encoding unit 122 may combine multiple pieces of uncompressed divided point cloud data and encode the combined uncompressed point cloud data, or may encode pre-compressed compressed point cloud data. As an example, the encoding unit 112 encodes a RAW file containing multiple pieces of RAW data into an encoded RAW file, and encodes a GPCC file containing multiple pieces of GPCC data into an encoded GPCC file.
[0038] The acquisition unit 113 acquires environmental information related to the environment surrounding the server 10. For example, the acquisition unit 113 acquires environmental information including at least one of communication environment information related to communication between the server 10 and the client 20, server environment information (transmission device environment information) related to the server 10, client environment information (decoding device environment information) related to the client 20, and point cloud data environment information related to the point cloud data. The acquisition unit 113 may acquire environmental information including communication environment information related to the performance of the communication path between the server 10 and the client 20. The acquisition unit 113 may acquire environmental information including communication environment information related to at least one of the communication speed and communication bandwidth of the communication path. The acquisition unit 113 may acquire environmental information including client environment information related to at least one of the performance, load, decoding period, reception status, and display status of the client 20. The acquisition unit 113 may acquire environmental information including client environment information related to the decoding speed of the client 20. The acquisition unit 113 may acquire environmental information including client environment information related to the reception status of the encoded and compressed point cloud data by the client 20.
[0039] The acquisition unit 113 may acquire environmental information including server environment information related to at least one of the performance, load, and transmission period of the server 10. The acquisition unit 113 may acquire environmental information including server environment information related to the service capacity of the server 10. The acquisition unit 113 may acquire environmental information including point cloud data environment information related to the attributes of point cloud data requested by the client 20. The acquisition unit 113 may acquire point cloud data environment information related to at least one of the accuracy, size, resolution, number, space, and ROI region of the point cloud data. Information related to the accuracy of the point cloud data refers to, for example, coordinate accuracy related to the coordinates of the point cloud data and color accuracy related to the colors of the point cloud data. The number of point cloud data (point cloud number) refers to, for example, information related to the number of point cloud data within a specified space (i.e., the density of the point cloud data) and the distribution of the point cloud data. The acquisition unit 113 may acquire environmental information including point cloud data environment information related to the point cloud data requested by the client 20 to be transmitted.
[0040] The acquiring unit 113 may further acquire the acquisition target determined by the determining unit 114 as point cloud data. As one example, the acquiring unit 113 acquires point cloud data including only first point cloud data such as encoded uncompressed point cloud data. As another example, the acquiring unit 113 acquires point cloud data including only second point cloud data such as encoded compressed point cloud data. As another example, the acquiring unit 113 acquires point cloud data including both first point cloud data such as encoded uncompressed point cloud data and second point cloud data such as encoded compressed point cloud data.
[0041] In this case, the acquisition unit 113 may acquire the acquisition target from at least one of first point cloud data and second point cloud data that are prepared by means such as generation in advance and stored in the server 10. As an example, the acquisition unit 113 acquires the acquisition target from one of first point cloud data such as encoded uncompressed point cloud data and second point cloud data such as encoded compressed point cloud data that are prepared in advance. As another example, the acquisition unit 113 acquires the acquisition target from both first point cloud data such as encoded uncompressed point cloud data and second point cloud data such as encoded compressed point cloud data that are prepared in advance. The acquisition unit 113 may acquire the acquisition target from at least one of first point cloud data such as encoded uncompressed point cloud data and second point cloud data such as encoded compressed point cloud data that are generated on demand after the acquisition target is determined. The acquisition unit 113 may acquire point cloud data including at least one of first point cloud data such as encoded uncompressed point cloud data and second point cloud data such as encoded compressed point cloud data as the transmission target, in parallel with the display of the transmission target by the client 20. That is, the acquisition unit 113 may acquire the transmission target on the fly.
[0042] The acquisition unit 113 may further acquire point cloud data. For example, the acquisition unit 113 acquires an uncompressed point cloud file including first point cloud data, such as uncompressed point cloud data that has not been divided or encoded. In the example shown in FIG. 1 , the acquisition unit 113 is included in the control unit 11, but when acquiring point cloud data that has not been divided or encoded, the acquisition unit 113 may function as an input receiving unit that is external to the control unit 11. Examples of such an acquisition unit 113 include a UI (User Interface), such as a button displayed on a display unit, such as a desktop or touch panel of a PC.
[0043] The determining unit 114 determines a transmission target from point cloud data including at least one of the first point cloud data and the second point cloud data, based on the environmental information acquired by the acquiring unit 113. An example of the determination by the determining unit 114 will be described below with reference to Figures 4 to 6. Figures 4 to 6 are diagrams for explaining the transmission system according to the first embodiment.
[0044] In the example shown in FIG. 4 , the determination unit 114 determines a transmission target that includes both first point cloud data, such as encoded uncompressed point cloud data, and second point cloud data, such as encoded compressed point cloud data, from point cloud data that includes both first point cloud data, such as encoded uncompressed point cloud data, and second point cloud data, such as encoded compressed point cloud data, generated in advance by spatially dividing and resolution-dividing a point cloud file. Specifically, the determination unit 114 determines a transmission target that includes a mixture of RAW files and GPCC files from a mixed file that includes a mixture of RAW files and GPCC files. The determination unit 114 may, or may not, determine a transmission target that includes a mixture of RAW files and GPCC files from a mixed file that includes a mixture of RAW files and GPCC files previously acquired by the acquisition unit 113. In a case where this is not the case, for example, the determination unit 114 determines the acquisition target to be acquired by the acquisition unit 113 based on the environmental information acquired by the acquisition unit 113.
[0045] An example of how the determination unit 114 determines a transmission target from a mixed file containing a mixture of RAW files and GPCC files previously acquired by the acquisition unit 113 will be described below with reference to FIG. 5 . In the example shown in FIG. 5 , the division unit 111 performs resolution division on the point cloud file from Level 0 to Level 30, and then performs spatial division. The encoding unit 112 encodes the multiple divided point cloud files divided by the division unit 111 into encoded RAW files, encoded GPCC 1 files, and encoded GPCC n files, respectively. The determination unit 114 determines a transmission target from these encoded files based on the environmental information acquired by the acquisition unit 113. Note that the resolution division levels do not have to be 0 to 30.
[0046] The determination unit 114 determines the transmission target based on information included in the environmental information acquired by the acquisition unit 113. As an example, the determination unit 114 determines the transmission target based on communication environment information included in the environmental information acquired by the acquisition unit 113. For example, the determination unit 114 determines the transmission target based on environmental information including communication environment information related to the communication bandwidth of the communication path between the server 10 and the client 20. If the communication bandwidth width is within a predetermined range, the determination unit 114 can maintain display response, and therefore determines a mixed file containing a mixture of encoded RAW files and encoded GPCC files as the transmission target. As another example, the determination unit 114 determines the transmission target based on client environment information included in the environmental information acquired by the acquisition unit 113. For example, the determination unit 114 determines the transmission target based on client environment information related to the load and decoding period of the client 20.
[0047] In this case, the determination unit 114 estimates the load related to decoding and rendering of the client 20, as well as the load related to rendering and the decoding period, based on the number of point clouds, the size of the B (Bounding) box, the encoding period of the encoding process, and the like. As an example, the determination unit 114 estimates the value obtained by dividing the sum (total value) of the following (1) to (3) by the number of remaining threads that can be simultaneously processed by the current CPU as the decoding load of the client 20. When a compiled LLVM is used as the decoding unit 221 of the client 20, the determination unit 114 makes an appropriate correction to suit the actual example, such as setting the compilation load to 0. (1) The load related to the compilation of the LLVM, etc. of the client 20 (e.g., the product of the code size of the client 20 and the coefficient A, etc.) (2) The load related to the parsing process of the encoded GPCC file (e.g., the product of the size of the encoded GPCC file and the coefficient B, etc.) (3) The load related to the decoding of the encoded GPCC file (e.g., the product of the number of point clouds of the encoded GPCC file and the coefficient C, etc.)
[0048] The determination unit 114 may calculate the load related to decoding on the client 20 based further on the software of the decoding unit 221, such as the state of the software for decoding the encoded GPCC file, etc. For example, the determination unit 114 adds a load corresponding to the startup time and initialization time of the decoding software of the decoding unit 221 to the load related to decoding on the client 20. Specifically, when the decoding software is Web Assembly, the determination unit 114 adds a load based on the compilation of the LLVM and the time corresponding to the client 20 (such as the execution time of the decoding process by the decoding unit 221 and the generation time of an image) to the load related to the client 20. Furthermore, the determination unit 114 adds a load corresponding to the time required to initialize the decoding software to the load.
[0049] The determination unit 114 may calculate the decoding load of the client 20 based further on a time-dependent load that is directly or indirectly correlated with the load of the decoding process by the decoding unit 221. For example, the determination unit 114 adds a time-dependent load that is highly correlated with the load of the decoding process, such as the time required for the decoding process estimated from the stream size or the number of point clouds, to the decoding load of the client 20. As an example, the determination unit 114 adds a load corresponding to the parsing process time based on the stream size to the decoding load of the client 20. Furthermore, the determination unit 114 adds a load corresponding to the conversion process time of the transmission target based on the number of point clouds to the decoding load of the client 20.
[0050] The determination unit 114 may estimate the load on the client 20 by adding the load of the rendering process by the decryption unit 221 of the client 20 to the load on the client 20 related to decoding calculated as described above. In this case, the determination unit 114 estimates the load on the rendering process as the load on the rendering process, which is the load related to rendering corresponding to the rendering process time by the decryption unit 221 of the client 20. The determination unit 114 can determine an appropriate transmission target by determining a transmission target such that each of the processing times, such as the decoding process time and the rendering process time, described above, is minimized.
[0051] As another example, the determination unit 114 determines a transmission target based on server environment information included in the environment information acquired by the acquisition unit 113. For example, the determination unit 114 determines a transmission target based on server environment information related to the load on the server 10. In this case, when a transmission target prepared (generated) in advance on the server 10 side is transmitted to the client 20 via the communication unit 12, the communication unit 21, and the network, the determination unit 114 estimates a transmission rate of the server 10 weighted by the most recent transmission rate of the server 10. For each candidate transmission target, the determination unit 114 estimates a transmission time based on the data size of the candidate transmission target and calculates the load on the server 10 corresponding to each estimated transfer time. The determination unit 114 determines the candidate transmission target that will minimize the load on the server 10 as the transmission target.
[0052] As another example, the determination unit 114 determines the transmission target based on point cloud data environment information included in the environment information acquired by the acquisition unit 113. For example, the determination unit 114 may determine the transmission target based on point cloud data environment information related to the size of the point cloud data included in the environment information. In this case, the determination unit 114 simply determines small-sized point cloud data as the transmission target. Specifically, the determination unit 114 estimates the size of the encoded RAW file based on a RAW file attribute that the size of the RAW file is directly proportional to the number of point clouds. The determination unit 114 also estimates the size of the encoded GPCC file based on a GPCC file attribute that the size of the GPCC file is the sum of the header value and values proportional to the number of point clouds in the Geometry attribute and the Attribute attribute. The determination unit 114 may actually compare the sizes of the encoded RAW file and the encoded GPCC file and determine the smaller-sized file as the transmission target, or may determine the transmission target by estimating the size of the above-mentioned file from the size of the B box and the number of point clouds. For example, if the size of a mixed file containing a mixture of encoded RAW files and encoded GPCC files is small, the determination unit 114 determines the mixed file as the file to be sent. This reduces the size (amount of data sent) of the file to be sent from the communication unit 12, allowing the determination unit 14 to shorten the display response. Note that if point cloud data has been generated in advance, the determination unit 114 may compare the actual file sizes to determine the file to be sent, without needing to estimate the sizes of the encoded RAW files and the encoded GPCC files.
[0053] As another example, the determination unit 114 may determine the transmission target based on point cloud data environment information related to an ROI region included in the environment information. If the point cloud data is an ROI region, the determination unit 114 determines the transmission target to be an encoded RAW file in order to maintain the accuracy of the original point cloud data. If the point cloud data is a non-ROI region, the determination unit 114 determines the transmission target to be an encoded GPCC file in order to shorten transmission time and response time. For example, if the point cloud data includes both an ROI region and a non-ROI region, the determination unit 114 determines the transmission target to be a mixed file containing a mixture of encoded RAW files and encoded GPCC files. As another example, the determination unit 114 determines the transmission target based on multiple pieces of information included in the above-mentioned environment information. For example, the determination unit 114 determines the transmission target each time by comprehensively considering point cloud data environment information related to the size of the point cloud data included in the environment information, client environment information related to the load on the client 20, and the like. The determining unit 114 determines the RAW / GPCC mixed file as the file to be transmitted, and outputs the RAW / GPCC mixed file to the communication unit 12 .
[0054] An example will be described below with reference to FIG. 6 in which the determination unit 114 determines, based on environmental information, that the acquisition target to be acquired by the acquisition unit 113 is a mixed file containing a mixture of RAW files and GPCC files, and then determines the mixed file containing a mixture of RAW files and GPCC files as a transmission target. In the example shown in FIG. 6 , similar to the example shown in FIG. 5 , the division unit 111 performs resolution division and spatial division on the point cloud file. Thereafter, before encoding is performed by the encoding unit 112, the determination unit 114 determines, based on the environmental information acquired by the acquisition unit 113, the acquisition target to be acquired by the acquisition unit 113. For example, the determination unit 114 determines the acquisition target in the same manner as the determination unit 114 determines the transmission target based on the environmental information shown in FIG. 6 . In the example shown in FIG. 6 , the determination unit 114 determines the acquisition target to be an encoded RAW file and an encoded GPCC file.
[0055] After the determination unit 114 determines the acquisition target, the encoding unit 112 encodes the RAW file and the GPCC file acquired by the acquisition unit 113 into an encoded RAW file and an encoded GPCC file. The acquisition unit 113 acquires the encoded RAW file and the encoded GPCC file, and the determination unit 114 determines these files as transmission targets and outputs the transmission targets to the communication unit 12.
[0056] Returning to the explanation of Fig. 1, the communication unit 12 communicates with the communication unit 21 of the client 20 via an electric communication line such as a local area network (LAN) or the Internet. The communication unit 12 transmits the transmission target determined by the determination unit 114 to the client 20. The communication unit 12 may be a network interface card (NIC) or the like.
[0057] The storage unit 13 stores environmental information, first point cloud data, second point cloud data, transmission targets, an OS, various programs, etc. Examples of the storage unit 13 include storage devices such as HDDs (Hard Disk Drives), SSDs (Solid State Drives), and optical disks, as well as data-rewritable semiconductor memories such as RAMs (Random Access Memory), flash memories, and NVSRAMs (Non Volatile Static Random Access Memory).
[0058] The client 20 receives a transmission target from the server 10. In the example shown in Fig. 1, the client 20 includes a communication unit (receiving unit) 21, a control unit 22, a display unit 23, and a storage unit 24.
[0059] The communication unit 21 communicates with the communication unit 12 of the server 10 via an electric communication line such as a LAN or the Internet. The communication unit 21 receives the transmission target determined by the determination unit 114 from the server 10. An example of the communication unit 32 is a NIC.
[0060] The control unit 22 controls the entire client 20. For example, the control unit 22 is configured with one or more processors having an internal memory in which programs defining each processing procedure and control data are stored, and the processor executes each process using the programs and internal memory. Examples of the control unit 22 include electronic circuits such as a CPU, MPU, and GPU, as well as integrated circuits such as an ASIC and FPGA. In the example shown in FIG. 1 , the control unit 22 has a decoding unit 221 and a display control unit 222.
[0061] The decoding unit 221 decodes the transmission target received by the communication unit 21. For example, the decoding unit 221 identifies a file corresponding to the ROI region over the network, and decodes this file, which is the transmission target transferred from the server 10, according to the file type.
[0062] The display control unit 222 causes the display unit 23 to display the transmission object decoded by the decoding unit 221. For example, the display control unit 222 causes the display unit 23 to display the transmission object decoded by the decoding unit 221 and converted from 3D to 2D.
[0063] The display unit 23 displays the transmission target decoded by the decoding unit 221. The display unit 23 may be, for example, the desktop of a terminal.
[0064] The storage unit 24 stores transmission targets, an OS, various programs, etc. Examples of the storage unit 24 include storage devices such as HDDs, SSDs, and optical disks, as well as data-rewritable semiconductor memories such as RAMs, flash memories, and NVSRAMs.
[0065] Next, an example of each process executed by the server 10 according to the first embodiment will be described with reference to Fig. 7 and Fig. 8. Fig. 7 and Fig. 8 are flow charts showing an example of a process of the server according to the first embodiment. First, an example of a process of the server 10 according to the first embodiment will be described with reference to Fig. 7.
[0066] In step S1 , the acquisition unit 113 acquires environmental information relating to the environment around the server 10 .
[0067] In step S2, the determining unit 114 determines a transmission target from point cloud data including at least one of the first point cloud data and the second point cloud data, based on the environmental information acquired by the acquiring unit 113.
[0068] In step S3 , the communication unit 12 transmits the transmission target determined by the determination unit 114 to the client 20 .
[0069] Next, an example of the determination process in step S2 of FIG. 7 will be described in detail with reference to FIG.
[0070] In step S21, the determining unit 114 determines whether the acquiring unit 113 has acquired point cloud data environment information related to the ROI region.
[0071] If the determination unit 114 determines that the point cloud data environment information regarding the ROI region has been acquired by the acquisition unit 113 (Yes in step S21), the determination unit 114 determines whether the transmission target is the ROI region in step S22. If the determination unit 114 determines that the point cloud data environment information regarding the ROI region has not been acquired by the acquisition unit 113 (No in step S21), the determination unit 114 proceeds to step S24.
[0072] If the determination unit 114 determines that the transmission target is an ROI region (Yes in step S22), it determines in step S23 that the transmission target is an encoded RAW file. If the determination unit 114 determines that the transmission target is not an ROI region (No in step S22), it proceeds to step S24.
[0073] In step S24, the determining unit 114 determines whether the acquiring unit 113 has acquired only the point cloud data environment information related to the size of the point cloud data.
[0074] If the determination unit 114 determines that only the point cloud data environment information regarding the size of the point cloud data has been acquired by the acquisition unit 113 (Yes in step S24), the determination unit 114 determines in step S25 whether the size indicated by the point cloud data environment information is an encoded RAW file<an encoded GPCC file. If the determination unit 114 determines that only the point cloud data environment information regarding the size of the point cloud data has not been acquired by the acquisition unit 113 (No in step S24), the determination unit 114 proceeds to step S27.
[0075] If the determination unit 114 determines that the size indicated by the point cloud data environment information is such that the encoded RAW file is smaller than the encoded GPCC file (Yes in step S25), the determination unit 114 determines the encoded RAW file to be transmitted in step S23. If the determination unit 114 determines that the size indicated by the point cloud data environment information is such that the RAW file is equal to or larger than the encoded GPCC file (No in step S25), the determination unit 114 determines the encoded GPCC file to be transmitted in step S26.
[0076] In step S27, the determining unit 114 determines whether or not the acquiring unit 113 has acquired only the client environment information relating to the load on the client 20.
[0077] If the determination unit 114 determines that only the client environment information related to the load on the client 20 has been acquired (Yes in step S27), the determination unit 114 determines in step S28 whether the load on the client 20 is equal to or less than a predetermined level. For example, the determination unit 114 determines whether the load on the client 20 is equal to or less than a predetermined level by, for example, calculating the load related to decoding and rendering on the client 20. Note that because the load on the client 20 is also related to the number of point clouds, the determination unit 114 does not always determine an encoded RAW file as a file to be transmitted, regardless of the load on the client 20. If the determination unit 114 determines that only the client environment information related to the load on the client 20 has not been acquired (No in step S27), the determination unit 114 proceeds to step S29.
[0078] If the determination unit 114 determines that the load on the client 20 is equal to or less than a predetermined level (Yes in step S28), it determines the encoded GPCC file to be transmitted in step S26. If the determination unit 114 determines that the load on the client 20 exceeds a predetermined level (No in step S28), it determines the encoded RAW file to be transmitted in step S23.
[0079] In step S29, the determination unit 114 determines whether the sum of the transmission time and the decoding time is within a predetermined time. In this case, for example, the determination unit 114 calculates the transmission time based on point cloud data environment information regarding the size of the point cloud data and communication environment information regarding the transmission rate of the communication path, both acquired by the acquisition unit 113. The determination unit 114 also estimates the decoding time based on point cloud data environment information regarding the attributes of the point cloud data, such as the size of the point cloud data and the type of transmission stream, both acquired by the acquisition unit 113. The determination unit 114 may correct the calculated or estimated value based on the most recent transmission time or decoding time. Furthermore, if the acquisition unit 113 directly acquires server environment information regarding the transmission time and client environment information regarding the decoding time, the determination unit 114 may make the above-mentioned determination based on the transmission time and the decoding time indicated by this environment information.
[0080] If the determination unit 114 determines that the sum of the transmission time and the decoding time is within the predetermined time (Yes in step S29), it determines the encoded GPCC file to be transmitted in step S26. If the determination unit 114 determines that the sum of the transmission time and the decoding time exceeds the predetermined time (No in step S29), it determines the encoded RAW file to be transmitted in step S23.
[0081] 2. Modification In the example shown in FIG. 4 , the acquisition unit 113 of the server 10 of the transmission system 1 acquires point cloud data in which the compression parameters of the RAW file and the GPCC file are the same as those of the GPCC file. Furthermore, the determination unit 114 determines a transmission target in which encoded RAW files and encoded GPCC files are mixed from the point cloud data in which the compression parameters of the RAW file and the GPCC file are the same as those of the GPCC file. However, the configuration of the transmission system 1 is not limited to the configuration shown in FIG. 4 and may be, for example, the configuration shown in FIG. 9 . Below, an example of the configuration of the transmission system 1 and point cloud data according to the modification will be described using FIG. 9 . FIG. 9 is a diagram for explaining a transmission system according to the modification.
[0082] In the example shown in FIG. 9 , point cloud data is generated (prepared) that includes a mixture of GPCC files with different compression parameters (GPCC files High Reso, in which point cloud data is compressed at a higher resolution than normal GPCC files; GPCC files Lossy, in which point cloud data is irreversibly compressed; and GPCC files Low Reso Lossy, in which point cloud data is irreversibly compressed at a lower resolution than normal GPCC files) and RAW files with a single compression parameter. The acquisition unit 113 acquires these point cloud data as acquisition targets based on environmental information. Furthermore, the determination unit 114 determines these point cloud data as transmission targets based on the environmental information. That is, the determination unit 114 selectively switches between encoded uncompressed point cloud data and encoded compressed point cloud data selected as transmission targets from these point cloud data based on the environmental information. Note that in the example shown in FIG. 9 , in addition to or instead of the GPCC files listed as GPCC files with different compression parameters, GPCC files Losesless, which have less reduction in the number of points and less degradation in color information compared to normal GPCC files, may be generated.
[0083] Although the above example mainly describes a case where the first point cloud data is encoded RAW data and the second point cloud data is encoded GPCC data, the first point cloud data and the second point cloud data are not limited to such configurations. Each of the first point cloud data and the second point cloud data may be uncompressed or compressed. That is, the combination of the first point cloud data and the second point cloud data is not limited to uncompressed / compressed combinations, but may also be compressed / uncompressed, compressed / compressed, and uncompressed / uncompressed combinations. For example, if the first point cloud data is compressed / encoded RAW data, the communication unit 12 of the server 10 may transmit the compressed / encoded RAW data, in which the encoded EAW data is compressed in a format such as gzip, to the client 20 via an on-the-fly method or the like.
[0084] In the above example, the compressed point cloud data is described as GPCC data. However, the compressed point cloud data is not limited to GPCC data and may be ZIP data, laszip data, draco data, etc. In the above example, the compressed point cloud data is uncompressed divided point cloud data containing coordinate information such as xyz and color information such as RGB after spatial division and resolution division, compressed as is. However, the compressed point cloud data may be compressed by changing the decomposition unit (division). In this case, for example, the compressed point cloud data is compressed by changing the decomposition unit using the division unit 111 and the encoding unit 112, which function as part of the point cloud spatial division resolution division encoding processing unit (point-cloud converter) shown in FIG. 4. In the above example, the first point cloud data and the second point cloud data may be encoded point cloud data but may be unencoded point cloud data. Furthermore, in the above example, the client 20 decodes the transmission target using the decoding unit 221. However, decoding is not necessarily required. The transmission target may be stored (stored) as encoded compressed point cloud data, etc.
[0085] 3. Second Embodiment In a transmission system according to one aspect of the present disclosure, a decoding device may acquire environmental information instead of a transmission device and determine a transmission target. A transmission system 1X according to the second embodiment will be described below with reference to Fig. 10. Fig. 10 is a diagram illustrating an example of the configuration of the transmission system according to the second embodiment.
[0086] In the example shown in FIG. 10 , the transmission system 1X has a server 10X and a client 20X instead of the server 10 and the client 20 according to the first embodiment. Except for this, the transmission system 1X is similar to the transmission system 1 according to the first embodiment. The server 10X has a control unit 11X instead of the control unit 11 according to the first embodiment. Except for this, the server 10X is similar to the server 10 according to the first embodiment. The control unit 11X does not have the acquisition unit 113 and the determination unit 114 according to the first embodiment. Except for this, the control unit 11X is similar to the control unit 11 according to the first embodiment. The client 20X has a control unit 22X and a communication unit (receiving unit) 21X instead of the control unit 22 and the communication unit 21 according to the first embodiment. Except for this, the client 20X is similar to the client 20 according to the first embodiment. The control unit 22X has an acquisition unit 223 and a determination unit 224. Except for this, the control unit 22X is similar to the control unit 22 according to the first embodiment.
[0087] The acquisition unit 223 acquires environmental information related to the environment surrounding the client 20X. For example, similar to the acquisition unit 113 in the first embodiment, the acquisition unit 223 acquires environmental information including at least one of communication environment information related to communication between the server 10 and the client 20X, server environment information related to the server 10, client environment information related to the client 20X, and point cloud data environment information related to the point cloud data. For example, the acquisition unit 223 monitors the network (communication path) to acquire communication environment information related to the transmission rate (communication path) and the network bandwidth (communication bandwidth).
[0088] The determining unit 224 determines a transmission target from point cloud data including at least one of the first point cloud data and the second point cloud data, based on the environmental information acquired by the acquiring unit 223. An example of the determination by the determining unit 224 will be described below with reference to Fig. 11 and Fig. 12. Fig. 11 and Fig. 12 are diagrams for explaining a transmission system according to the second embodiment.
[0089] 11 , the determination unit 224 appropriately determines the transmission target when displaying point cloud data that includes both first point cloud data such as encoded uncompressed point cloud data and second point cloud data such as encoded compressed point cloud data, which are generated in advance by spatially dividing and resolution-dividing a point cloud file. Specifically, when displaying a mixed file that includes a mixture of RAW files and GPCC files, the determination unit 224 appropriately selects the RAW file and the GPCC file to determine the transmission target.
[0090] As an example, the determination unit 224 determines the file to be transmitted based on communication environment information regarding the transmission rate included in the environment information acquired by the acquisition unit 223. When the transmission rate most recently acquired by the acquisition unit 223 is low, such as when the transmission rate is lower than a threshold, the determination unit 224 determines a small-sized file, such as the smallest encoded GPCC file, to be transmitted from among the group of files. When the transmission rate most recently acquired by the acquisition unit 223 is high, such as when the transmission rate is equal to or higher than a threshold, the determination unit 224 determines an encoded RAW file to be transmitted from among the group of files. In this case, in order to refer to the size list of the file group, the determination unit 224 may cause the server 10X to transmit the size list in advance via the communication unit 12 and the communication unit 21X and have the acquisition unit 223 acquire it. The determination unit 224 may determine the size list to be requested from the server 10X depending on whether image quality or size is to be prioritized, and then have this size list transmitted from the server 10X via the communication unit 12 and the communication unit 21X, and have the acquisition unit 223 acquire it.
[0091] As another example, the determination unit 224 determines a transmission target based on client environment information related to the load on the client 20X and other environment information included in the environment information acquired by the acquisition unit 223. In this case, the determination unit 224 calculates a value related to the CPU margin, such as CPU usage, from the current load average of the client 20X, and estimates the load of decoding, rendering, etc. on the client 20X when the transmission target is transmitted, based on the CPU margin and compression format, etc. The determination unit 224 determines transmission target candidates from a file group based on the other environment information, and determines a transmission target from the candidate transmission targets that has the least estimated load on the client 20X.
[0092] As another example, the determination unit 224 determines the file to be transmitted based on client environment information related to the performance of the client 20X, which is included in the environment information acquired by the acquisition unit 223. In this case, if the performance of the client 20X, such as the decoding ability, most recently acquired by the acquisition unit 223, is lower than a threshold, the determination unit 224 determines the encoded RAW file from the group of files to be transmitted. If the performance of the client 20X, such as the decoding ability, most recently acquired by the acquisition unit 223, is higher than a threshold, the determination unit 224 determines the encoded GPCC ... performance of the client 20X, such as the decoding ability, most recently acquired by the acquisition unit 223, is higher than a threshold, and if the transmission rate most recently acquired by the acquisition unit 223 is higher than a threshold, the determination unit 224 determines the encoded RAW file from the group of files to be transmitted. The decision unit 224 may change the threshold for switching the ratio between encoded uncompressed point cloud data such as encoded RAW files and encoded compressed point cloud data such as encoded GPCC files, in addition to the data to be sent, based on the performance of the client 20X, such as its decoding capability.
[0093] As another example, the determination unit 224 may determine the transmission target based on point cloud data environment information related to an ROI region included in the environment information. If the point cloud data is for an ROI region, the determination unit 224 determines the transmission target to be an encoded RAW file in order to maintain the accuracy of the original point cloud data. If the point cloud data is for a non-ROI region, the determination unit 224 determines the transmission target to be an encoded GPCC file in order to shorten transmission time and response time.
[0094] As another example, the determination unit 224 determines the transmission target based on multiple pieces of information included in the above-mentioned environmental information. For example, the determination unit 114 determines the transmission target each time by comprehensively considering point cloud data environmental information related to the size of the point cloud data included in the environmental information, client environmental information related to the load on the client 20X, and the like.
[0095] As another example, the determination unit 224 determines the transmission target in parallel with the display of the transmission target by the client 20X. In the example shown in FIG. 12 , the determination unit 224 (File Selector) determines the transmission target in parallel with the display of the transmission target on the display unit 23 by the display control unit 222 of the client 20X. For example, when the display control unit 222 starts roughly rendering the transmission target on the display unit 23, the determination unit 224 determines the files included in the transmission target based on the size of each file, such as a RAW file and a GPCC file, at the same spatial position, and issues a transmission request to the server 10X for the transmission target via the communication units 21X and 12, causing the display control unit 222 to proceed with rendering the files transmitted from the server 10X. As the display control unit 222 reads the Level value of the resolution level and the display of the transmission target on the display unit 23 progresses to a more detailed level, the determination unit 224 calculates the transmission time from the file size based on the most recent transmission rate of the server 10. Furthermore, the determination unit 224 estimates the decoding time required by the decoding unit 221 of the client 20X when the encoded GPCC file is further selected as the file to be transmitted, based on the stream size and the most recent decoding time. The determination unit 224 selects the type of file to be transmitted and determines the file to be transmitted so that the total value of the transmission time and decoding time is minimized.
[0096] The communication unit 21X receives the transmission target determined by the determination unit 224 from the server 10X to which the transmission target is to be transmitted.
[0097] Next, an example of the decision processing by the decision unit 224 of the client 20X will be described with reference to FIG. 13. FIG. 13 is a flow diagram showing an example of the processing by the client according to the second embodiment. Steps S21 to S28 in FIG. 13 are the same as steps S21 to S28 in FIG. 8. That is, the processes by the decision unit 224 other than step S29X in FIG. 13 are the same as the processes by the decision unit 114 in the first embodiment.
[0098] In step S29X, the determination unit 224 determines whether the sum of the transmission time and the decoding time is within a predetermined time. In the client 20X, the determination unit 224 may determine the transmission target in parallel with the display of the transmission target by the client 20X. For example, the determination unit 224 determines the transmission target based on environmental information acquired by the acquisition unit 223 while the display control unit 222 of the client 20X is drawing the transmission target on the display unit 23. In this case, the determination unit 224 determines the type of data included in the transmission target based on the response indicated by the environmental information (such as the amount of communication bandwidth and the margin of the drawing process by the CPU of the control unit 22X, etc., including the display control unit 222). The determination unit 224 adaptively changes the type of data included in the transmission target acquired by the acquisition unit 223 and requested to be transmitted, depending on the response, thereby determining an appropriate transmission target and achieving a balance between favorable response and drawing image quality.
[0099] In the above example, the determination unit 224 of the client 20X determines the transmission target in parallel with the display of the transmission target by the client 20X. However, if the server 10X has the determination unit 224, the determination unit 224 of the server 10X may determine the transmission target in parallel with the display of the transmission target by the client 20X.
[0100] 4. Example of Hardware Configuration The various devices described above, such as the server 10, the server 10X, the client 20, and the client 20X, may be configured to include a computer. An example will be described with reference to FIG.
[0101] 14 is a diagram showing an example of the hardware configuration of the device. The illustrated computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected via a bus 1050.
[0102] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.
[0103] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .
[0104] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records generation programs for executing the operations related to the present disclosure, which are examples of program data 1450.
[0105] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.
[0106] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.
[0107] At least some of the functions of the server 10, server 10X, client 20, client 20X, etc. described above may be realized, for example, by the CPU 1100 of the computer 1000 executing a program loaded on the RAM 1200. Furthermore, the HDD 1400 stores programs and the like according to the present disclosure. The CPU 1100 reads and executes program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.
[0108] 5. Example of Effects As described with reference to Figures 1 to 13 etc., one of the disclosed technologies is a server 10 etc. The server 10 includes an acquisition unit 113 that acquires environmental information related to the environment around the server 10, a determination unit 114 that determines a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition unit 113, and a communication unit 12 that transmits the transmission target determined by the determination unit 114 to a decoding device.
[0109] The server 10 selects one or both of the first point cloud data and the second point cloud data included in the point cloud data as data to be transmitted in accordance with the environmental information, and selectively switches between the first point cloud data and the second point cloud data selected as data to be transmitted. This allows the server 10 to determine an appropriate data to be transmitted in accordance with the environmental information. As a result, the server 10 can reduce the time and cost required to receive the point cloud data.
[0110] As explained with reference to Figures 1 to 13, the acquisition unit 113 acquires environmental information including at least one of communication environment information regarding communication between the server 10 and the client 20, server environment information regarding the server 10, client environment information regarding the client 20, and point cloud data environment information regarding the point cloud data, and the determination unit 114 may determine the transmission target based on the information included in the environmental information acquired by the acquisition unit 113.
[0111] This allows the server 10 to determine an appropriate transmission target based on the communication environment information, server environment information, client environment information, and point cloud data environment information. As an example, if the communication bandwidth indicated by the communication environment information is narrow and the performance of the client 20 indicated by the client environment information is high, the server 10 determines point cloud data containing second point cloud data, such as a highly compressed transmission stream, as the transmission target. This allows the server 10 to reduce the transmission period and the subsequent display response. As another example, if the communication speed indicated by the communication environment information is fast but the performance of the client 20 indicated by the client environment information is poor, the server 10 selects point cloud data containing first point cloud data, such as encoded uncompressed point cloud data, as the transmission target. This allows the server 10 to reduce the decoding cost of the client 20. As another example, the server 10 determines the transmission target that minimizes the response, which is the sum of the transmission period indicated by the server environment information and the decoding period indicated by the client environment information. This allows the server 10 to shorten the display response until the transmission target is displayed by the client 20.
[0112] 1 to 13 , the acquisition unit 113 may acquire environment information including communication environment information related to the performance of the communication path between the server 10 and the client 20, and the determination unit 114 may determine the transmission target based on the communication environment information included in the environment information acquired by the acquisition unit 113. This allows the server 10 to determine an appropriate transmission target according to the communication environment information related to the performance of the communication path between the server 10 and the client 20. For example, the server 10 can estimate the communication period based on this communication environment information and determine an appropriate transmission target that will shorten the communication period.
[0113] 1 to 13, the acquisition unit 113 may acquire environment information including communication environment information related to at least one of the communication speed and communication bandwidth of the communication path. This allows the server 10 to determine an appropriate transmission target according to the communication environment information related to the communication speed and communication bandwidth of the communication path. For example, the server 10 can estimate the communication period based on the communication speed and determine an appropriate transmission target that will shorten the communication period.
[0114] 1 to 13 , etc., the acquisition unit 113 may acquire environment information including client environment information related to at least one of the performance, load, decoding period, reception status, and display status of the client 20, and the determination unit 114 may determine a transmission target based on the client environment information included in the environment information acquired by the acquisition unit 113. This allows the server 10 to determine an appropriate transmission target according to the client environment information related to at least one of the performance, load, decoding period, reception status, and display status of the client 20. For example, the server 10 can estimate the decoding period based on the performance of the client 20 and determine an appropriate transmission target that shortens the decoding period.
[0115] 1 to 13, etc., the acquisition unit 113 may acquire environment information including client environment information related to the decoding speed of the client 20. This allows the server 10 to determine an appropriate transmission target according to the client environment information related to the decoding speed of the client 20. For example, the server 10 can estimate the decoding period based on the decoding speed of the client 20 and determine an appropriate transmission target that will shorten the decoding period.
[0116] 1 to 13, the acquisition unit 113 may acquire environmental information including client environmental information relating to the reception status of point cloud data by the client 20. This allows the server 10 to determine an appropriate transmission target according to the reception status of the client 20.
[0117] 1 to 13 , the acquisition unit 113 may acquire environmental information including server environmental information related to at least one of the performance, load, and transmission period of the server 10, and the determination unit 114 may determine a transmission target based on the server environmental information included in the environmental information acquired by the acquisition unit 113. This allows the server 10 to determine an appropriate transmission target according to the server environmental information related to at least one of the performance, load, and transmission period of the server 10. For example, the server 10 may determine a transmission target that reduces (alleviates) the load on the server 10 related to transmission of the transmission target, the storage cost of the server 10, and the like, based on the server environmental information related to the load on the server 10.
[0118] 1 to 13, the acquisition unit 113 may acquire environment information including server environment information related to the service capabilities of the server 10. This allows the server 10 to determine an appropriate transmission target according to the server environment information related to the service capabilities of the server 10.
[0119] 1 to 13, the acquisition unit 113 may acquire environmental information including point cloud data environmental information related to the attributes of point cloud data requested by the client 20, and the determination unit 114 may determine a transmission target based on the point cloud data environmental information included in the environmental information acquired by the acquisition unit 113. This allows the server 10 to determine an appropriate transmission target according to the point cloud data environmental information.
[0120] As described with reference to FIGS. 1 to 13, the acquisition unit 113 may acquire point cloud data environment information relating to at least one of the accuracy, size, resolution, number, space, and ROI area of the point cloud data.
[0121] This allows the server 10 to determine an appropriate transmission target based on point cloud data environment information related to at least one of the accuracy, size, resolution, number, space, and ROI region of the point cloud data. As an example, the server 10 determines high-precision point cloud data as the transmission target based on point cloud data environment information related to the accuracy of the point cloud data. This allows the server 10 to determine an appropriate transmission target based on the point cloud data environment information related to the accuracy of the point cloud data. As another example, the server 10 determines the transmission target based on point cloud data environment information related to the ROI region of the point cloud data, depending on whether the transmission target is an ROI region. In this case, the server 10 determines first point cloud data, such as encoded uncompressed point cloud data, as the transmission target if the transmission target is an ROI region, and determines second point cloud data, such as encoded compressed point cloud data, as the transmission target if the transmission target is a non-ROI region. This allows the server 10 to determine an appropriate transmission target based on point cloud data environment information related to the ROI region of the point cloud data.
[0122] As explained with reference to Figures 1 to 13, the acquisition unit 113 acquires environmental information including point cloud data environmental information related to point cloud data requested as a transmission target from the client 20, and the determination unit 114 may determine the transmission target based on the point cloud data environmental information included in the environmental information acquired by the acquisition unit 113.
[0123] This allows the server 10 to determine an appropriate transmission target in accordance with the point cloud data environment information related to the point cloud data requested by the client 20. For example, if the client 20 requests data to be transmitted almost as is based on the point cloud data environment information, the server 10 determines that the data to be transmitted is first point cloud data, such as encoded uncompressed point cloud data, or mixed data containing a mixture of first point cloud data and second point cloud data, in which the proportion of second point cloud data, such as encoded compressed point cloud data, is lower than the proportion of first point cloud data, such as encoded uncompressed point cloud data. As another example, if the client 20 requests high-precision point cloud data or point cloud data that provides a fast display response based on the point cloud data environment information, the server 10 adjusts (selects) the compression level of the compressed point cloud data. In this case, the server 10 adjusts the display response by, for example, adjusting the proportion of first point cloud data, such as encoded uncompressed point cloud data, and second point cloud data, such as encoded compressed point cloud data. The server 10 also adjusts the compression parameters of the encoded uncompressed point cloud data. This allows the server 10 to determine an appropriate transmission target in response to a request from the client 20 .
[0124] 1 to 13 , the determination unit 114 may determine an acquisition target to be acquired by the acquisition unit 113 based on the environmental information acquired by the acquisition unit 113, and the acquisition unit 113 may further acquire the acquisition target determined by the determination unit 114 as point cloud data. This allows the server 10 to prepare only the point cloud data determined as a transmission target as an acquisition target, thereby reducing the cost of the server 10.
[0125] 1 to 13, the first point cloud data and the second point cloud data may be uncompressed point cloud data or compressed point cloud data, respectively, and the server 10 can advantageously reduce the time and cost required to receive point cloud data with such a data structure.
[0126] 1 to 13, at least one of the first point cloud data and the second point cloud data may include at least one of 3D point cloud data and a plurality of divided point cloud data divided based on at least one of space and resolution. The server 10 can advantageously reduce the time and cost required to receive point cloud data with such a data structure.
[0127] 1 to 13, etc., also constitutes one of the disclosed technologies. The client 20X includes an acquisition unit 223 that acquires environmental information related to the environment surrounding the client 20X, a determination unit 224 that determines a transmission target from point cloud data including first point cloud data and second point cloud data based on the environmental information acquired by the acquisition unit 223, and a communication unit 21X that receives the transmission target determined by the determination unit 224 from a transmitting device. As described above, this client 20X can also reduce the time and cost required to receive point cloud data.
[0128] The transmission system 1 described with reference to Figures 1 to 13 is also one of the disclosed technologies. The transmission system 1 includes a server 10 and a client 20. The server 10 includes an acquisition unit 113 that acquires environmental information related to the environment surrounding the server 10, a determination unit 114 that determines a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition unit 113, and a communication unit 12 that transmits the transmission target determined by the determination unit 114 to the client 20. The client 20 includes a communication unit 21 that receives the transmission target determined by the determination unit 114 from the server 10. As described above, the transmission system 1 described above can also reduce the time and cost required to receive point cloud data.
[0129] The transmission method described with reference to Figures 1 to 13 is also one of the disclosed technologies. The transmission method is executed by the server 10 and includes an acquisition step of acquiring environmental information related to the environment surrounding the server 10, a determination step of determining a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition step, and a transmission step of transmitting the transmission target determined by the determination step to a decoding device. As described above, this communication method can also reduce the time and cost required to receive point cloud data.
[0130] The transmission program described with reference to Figures 1 to 13 is also one of the disclosed technologies. The transmission program causes a computer installed in the server 10 to execute an acquisition process for acquiring environmental information related to the environment surrounding the server 10, a determination process for determining a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition process, and a transmission process for transmitting the transmission target determined by the determination process to a decoding device. As described above, such a communication program can also reduce the time and cost required to receive point cloud data.
[0131] The effects described in this disclosure are merely examples and are not limited to the disclosed contents. Other effects may also be present.
[0132] The above describes the embodiments of the present disclosure, but the technical scope of the present disclosure is not limited to the above-described embodiments as they are, and various modifications are possible within the scope that does not deviate from the gist of the present disclosure, and different components across various modifications may be combined as appropriate.
[0133] This technology may be related to Goal 9, "Industry, innovation, infrastructure," of the Sustainable Development Goals (SDGs) adopted at the United Nations Summit in 2015.
[0134] Note that the present technology may also be configured as follows. (1) A transmitting device comprising: an acquisition unit that acquires environmental information related to an environment surrounding the transmitting device; a determination unit that determines a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition unit; and a transmitting unit that transmits the transmission target determined by the determination unit to a decoding device. (2) The transmitting device according to (1), wherein the acquisition unit acquires the environmental information including at least one of communication environment information related to communication between the transmitting device and the decoding device, transmission device environment information related to the transmitting device, decoding device environment information related to the decoding device, and point cloud data environment information related to the point cloud data, and the determination unit determines the transmission target based on information included in the environmental information acquired by the acquisition unit. (3) The transmitting device according to (2), wherein the acquisition unit acquires the environmental information including the communication environment information related to performance of a communication path between the transmitting device and the decoding device, and the determination unit determines the transmission target based on the communication environment information included in the environmental information acquired by the acquisition unit. (4) The transmitting device according to (3), wherein the acquisition unit acquires the environmental information including the communication environment information related to at least one of a communication speed and a communication bandwidth of the communication path. (5) The transmitting device according to (2), wherein the acquisition unit acquires the environmental information including the decoding device environmental information related to at least one of performance, load, decoding period, reception status, and display status of the decoding device, and the determination unit determines the transmission target based on the decoding device environmental information included in the environmental information acquired by the acquisition unit. (6) The transmitting device according to (5), wherein the acquisition unit acquires the environmental information including the decoding device environmental information related to a decoding speed of the decoding device. (7) The transmission device according to (5), wherein the acquisition unit acquires the environmental information including the decoding device environmental information relating to a reception status of the point cloud data by the decoding device.(8) The transmitting device according to (2), wherein the acquisition unit acquires the environmental information including the transmitting device environmental information related to at least one of performance, load, and transmission period of the transmitting device, and the determination unit determines the transmission target based on the transmitting device environmental information included in the environmental information acquired by the acquisition unit. (9) The transmitting device according to (8), wherein the acquisition unit acquires the environmental information including the transmitting device environmental information related to service capabilities of the transmitting device. (10) The transmitting device according to (2), wherein the acquisition unit acquires the environmental information including point cloud data environmental information related to attributes of the point cloud data requested by the decoding device, and the determination unit determines the transmission target based on the point cloud data environmental information included in the environmental information acquired by the acquisition unit. (11) The transmitting device according to (10), wherein the acquisition unit acquires the point cloud data environmental information related to at least one of accuracy, size, resolution, number, space, and ROI (Region of Interest) of the point cloud data. (12) The transmitting device according to (2), wherein the acquisition unit acquires the environmental information including the point cloud data environmental information related to the point cloud data requested as the transmission target from the decoding device, and the determination unit determines the transmission target based on the point cloud data environmental information included in the environmental information acquired by the acquisition unit. (13) The transmitting device according to any of (1) to (12), wherein the determination unit determines an acquisition target to be acquired by the acquisition unit based on the environmental information acquired by the acquisition unit, and the acquisition unit further acquires the acquisition target determined by the determination unit as the point cloud data. (14) The transmitting device according to any of (1) to (13), wherein each of the first point cloud data and the second point cloud data is uncompressed point cloud data or compressed point cloud data. (15) The transmitting device according to any one of (1) to (14), wherein at least one of the first point cloud data and the second point cloud data includes at least one of 3D (Dimensions) point cloud data and a plurality of divided point cloud data divided based on at least one of space and resolution.(16) A decoding device comprising: an acquisition unit that acquires environmental information related to an environment surrounding the decoding device; a determination unit that determines a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition unit; and a receiving unit that receives the transmission target determined by the determination unit from a transmitting device. (17) The decoding device according to (16), wherein the determination unit determines the transmission target in parallel with display of the transmission target by the decoding device. (18) A transmission system comprising: a transmitting device and a decoding device, wherein the transmitting device comprises: an acquisition unit that acquires environmental information related to an environment surrounding the transmitting device; a determination unit that determines a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition unit; and a transmitting unit that transmits the transmission target determined by the determination unit to a decoding device, and the decoding device comprises: a receiving unit that receives the transmission target determined by the determination unit from the transmitting device. (19) A transmission method executed by a transmission device, comprising: an acquisition step of acquiring environmental information related to the environment around the transmission device, a determination step of determining a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition step, and a transmission step of transmitting the transmission target determined by the determination step to a decoding device. (20) A transmission program that causes a computer mounted on the transmission device to execute: an acquisition process of acquiring environmental information related to the environment around the transmission device, a determination process of determining a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition process, and a transmission process of transmitting the transmission target determined by the determination process to a decoding device.
[0135] REFERENCE SIGNS LIST 1 Transmission system 10 Server (transmission device) 11, 22 Control unit 12, 21 Communication unit 13, 24 Storage unit 20 Client (decoding device) 23 Display unit 111 Division unit 112 Encoding unit 113 Acquisition unit 114 Determination unit 221 Decoding unit 222 Display control unit
Claims
1. A transmitting device comprising: an acquisition unit that acquires environmental information relating to the environment surrounding the transmitting device; a determination unit that determines a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition unit; and a transmitting unit that transmits the transmission target determined by the determination unit to a decoding device.
2. The transmitting device of claim 1, wherein the acquisition unit acquires the environmental information including at least one of communication environment information regarding communication between the transmitting device and the decoding device, transmitting device environmental information regarding the transmitting device, decoding device environmental information regarding the decoding device, and point cloud data environmental information regarding the point cloud data, and the determination unit determines the transmission target based on information included in the environmental information acquired by the acquisition unit.
3. The transmitting device described in claim 2, wherein the acquisition unit acquires the environmental information including the communication environment information regarding the performance of the communication path between the transmitting device and the decoding device, and the determination unit determines the transmission target based on the communication environment information included in the environmental information acquired by the acquisition unit.
4. The transmitting device according to claim 3, wherein the acquiring unit acquires the environment information including the communication environment information relating to at least one of a communication speed and a communication band of the communication path.
5. The transmitting device of claim 2, wherein the acquisition unit acquires the environmental information including the decoding device environmental information relating to at least one of the performance, load, decoding period, reception status, and display status of the decoding device, and the determination unit determines the transmission target based on the decoding device environmental information included in the environmental information acquired by the acquisition unit.
6. The transmission device according to claim 5, wherein the acquisition unit acquires the environmental information including the decoding device environmental information relating to a decoding speed of the decoding device.
7. The transmission device according to claim 5, wherein the acquisition unit acquires the environmental information including the decoding device environmental information relating to a reception status of the point cloud data by the decoding device.
8. The transmitting device of claim 2, wherein the acquisition unit acquires the environmental information including the transmitting device environmental information relating to at least one of the performance, load, and transmission period of the transmitting device, and the determination unit determines the transmission target based on the transmitting device environmental information included in the environmental information acquired by the acquisition unit.
9. The transmitting device according to claim 8, wherein the acquiring unit acquires the environmental information including the transmitting device environmental information related to a service capability of the transmitting device.
10. The transmitting device described in claim 2, wherein the acquisition unit acquires the environmental information including point cloud data environmental information related to attributes of the point cloud data requested by the decoding device, and the determination unit determines the transmission target based on the point cloud data environmental information included in the environmental information acquired by the acquisition unit.
11. The transmitting device according to claim 10, wherein the acquiring unit acquires the point cloud data environment information relating to at least one of accuracy, size, resolution, number, space and a ROI (Region of Interest) area of the point cloud data.
12. The transmitting device described in claim 2, wherein the acquisition unit acquires the environmental information including the point cloud data environmental information regarding the point cloud data requested as the transmission target from the decoding device, and the determination unit determines the transmission target based on the point cloud data environmental information included in the environmental information acquired by the acquisition unit.
13. The transmitting device of claim 1, wherein the determination unit determines an acquisition target to be acquired by the acquisition unit based on the environmental information acquired by the acquisition unit, and the acquisition unit further acquires the acquisition target determined by the determination unit as the point cloud data.
14. The transmission device according to claim 1, wherein each of the first point cloud data and the second point cloud data is uncompressed point cloud data or compressed point cloud data.
15. The transmitting device of claim 1, wherein at least one of the first point cloud data and the second point cloud data includes at least one of 3D (Dimensions) point cloud data and a plurality of divided point cloud data divided based on at least one of space and resolution.
16. A decoding device comprising: an acquisition unit that acquires environmental information relating to the environment surrounding the decoding device; a determination unit that determines a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition unit; and a receiving unit that receives the transmission target determined by the determination unit from a transmitting device.
17. The decoding device according to claim 16, wherein the determination unit determines the transmission target in parallel with display of the transmission target by the decoding device.
18. A transmission system comprising a transmitting device and a decoding device, wherein the transmitting device comprises an acquisition unit that acquires environmental information relating to the environment surrounding the transmitting device, a determination unit that determines a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition unit, and a transmitting unit that transmits the transmission target determined by the determination unit to a decoding device, and the decoding device comprises a receiving unit that receives the transmission target determined by the determination unit from the transmitting device.
19. A transmission method executed by a transmitting device, comprising: an acquisition step of acquiring environmental information relating to the environment surrounding the transmitting device; a determination step of determining a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition step; and a transmission step of transmitting the transmission target determined by the determination step to a decoding device.
20. A transmission program that causes a computer mounted on a transmitting device to execute an acquisition process for acquiring environmental information regarding the environment surrounding the transmitting device, a determination process for determining a transmission target from point cloud data including at least one of first point cloud data and second point cloud data based on the environmental information acquired by the acquisition process, and a transmission process for transmitting the transmission target determined by the determination process to a decoding device.
Citation Information
Patent Citations
Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device
WO2020251019A1
Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device
WO2021210548A1
Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device
WO2021261555A1
