Information processing device and method

Abstract

The present disclosure relates to an information processing device and method which can suppress increases in the amount of data in a point cloud. In the present invention, the data format of a point cloud attribute that expresses the three-dimensional shape of an object generated using a captured image obtained by image-capturing the object by using a multi-sensor camera constituted of three or more element cameras, and includes a first element indicating color luminance information of one point within the point cloud and a second element indicating the position of that point that respectively correspond to the element cameras of the multi-sensor camera at each image-capture position, is converted by adding, to said attribute, a third element indicating the position and posture of a reference element camera, which is one of the element cameras constituting the multi-sensor camera. The present disclosure can be applied, for example, to an information processing device, an electronic apparatus, an information processing method, and a program.

Description

Information processing device and method

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that can suppress an increase in the amount of point cloud data.

[0002] Previously, NeRF was used as a 3D data technology (see, for example, Non-Patent Document 1). However, this method placed a heavy load on the reproduction process (rendering process), and it was difficult to achieve high-quality rendered images. In contrast, there was point cloud, which represents the 3D shape of an object. When applying point cloud as 3D data and capturing images of an object from multiple directions to generate point cloud data, there was a method to enhance the color and brightness representation capabilities of the point cloud by adding light ray information from multiple directions to a single point.

[0003] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng, "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis", DOI:10.1145 / 3503250, JANUARY 2022 | VOL. 65 | NO. 1 | COMMUNICATIONS OF THE ACM

[0004] However, with this method, there is a risk that the amount of data in the point cloud will increase in proportion to the number of points. Generally, the number of points in a point cloud is enormous, and the number of points increases even more if the resolution is increased. In other words, with this method, there is a risk that the amount of data in the point cloud will increase.

[0005] The present disclosure has been made in consideration of such circumstances, and makes it possible to suppress an increase in the amount of data of point clouds.

[0006] An information processing device according to one aspect of the present technology is an information processing device that includes a data format conversion unit that converts the data format of an attribute of a point cloud that represents the three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, by adding a third element that indicates the position and orientation of a reference element camera that is one of the element cameras that constitute the multi-lens camera to the attribute that has a first element that indicates color brightness information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each imaging position, and a second element that indicates the position of the point.

[0007] An information processing method according to one aspect of the present technology is an information processing method for converting the data format of an attribute of a point cloud that represents the three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, the attribute having a first element indicating color intensity information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each imaging position, and a second element indicating the position of the point, by adding a third element indicating the position and orientation of a reference element camera that is one of the element cameras that make up the multi-lens camera.

[0008] An information processing device according to another aspect of the present technology is an information processing device including: a data format inverse conversion unit that inversely converts a data format of an attribute of a point cloud that represents a three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, the attribute having a first element that indicates color brightness information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each imaging position, a second element that indicates the position of the point, and a third element that indicates the position and attitude of a reference element camera that is one of the element cameras that constitute the multi-lens camera, by converting the third element into a fourth element that indicates a light ray of the element camera; and a display image generation unit that renders the point cloud whose data format has been inversely converted, and generates a display image that corresponds to a viewpoint.

[0009] Another aspect of the information processing method of the present technology is an information processing method for inversely converting a data format of an attribute of a point cloud that represents a three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, the attribute having a first element indicating color intensity information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each imaging position, a second element indicating the position of the point, and a third element indicating the position and attitude of a reference element camera that is one of the element cameras that constitute the multi-lens camera, by converting the third element into a fourth element indicating a light ray of the element camera, rendering the point cloud with the inversely converted data format, and generating a display image that corresponds to a viewpoint.

[0010] In an information processing device and method according to one aspect of the present technology, an attribute of a point cloud representing the three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, the attribute having a first element indicating color intensity information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each imaging position, and a second element indicating the position of that point, is converted in data format by adding a third element indicating the position and orientation of a reference element camera, which is one of the element cameras that make up the multi-lens camera.

[0011] In another aspect of the information processing device and method of the present technology, an attribute of a point cloud representing the three-dimensional shape of an object is generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, the attribute having a first element indicating color intensity information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each imaging position, a second element indicating the position of that point, and a third element indicating the position and orientation of a reference element camera which is one of the element cameras that make up the multi-lens camera, the third element is converted into a fourth element indicating the light rays of the element cameras, thereby inversely converting the data format of the attribute, and the point cloud with the inversely converted data format is rendered, and a display image corresponding to the viewpoint is generated.

[0012] FIG. 1 is a diagram for explaining a multi-lens camera. FIG. 2 is a diagram for explaining an imaging method. FIG. 3 is a diagram for explaining an example of a data structure. FIG. 4 is a diagram for explaining an example of a data structure. FIG. 5 is a block diagram showing an example of the main configuration of an imaging device. FIG. 6 is a flowchart for explaining an example of the flow of imaging processing. FIG. 7 is a block diagram showing an example of the main configuration of a display device. FIG. 8 is a flowchart for explaining an example of the flow of display processing. FIG. 9 is a block diagram showing an example of the main configuration of a computer.

[0013] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. Literature etc. supporting technical content and technical terminology 2. Point cloud 3. Data format conversion 4. First embodiment (imaging device) 5. Second embodiment (display device) 6. Supplementary notes

[0014] <1. Literature, etc. supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the content described in the embodiments, but also the content described in the following non-patent documents, etc. that were publicly known at the time of filing, and the content of other documents referenced in the following non-patent documents.

[0015] Non-patent document 1: (mentioned above)

[0016] In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.

[0017] <2. Point Cloud> <Point Cloud Data Structure> Conventional 3D data technologies include NeRF, as shown in Non-Patent Document 1. However, this method places a heavy load on the reproduction process (rendering process), and it is difficult to achieve high quality in rendered images. In contrast, there is the point cloud, which represents the 3D shape of an object. When applying a point cloud as 3D data, capturing images of an object from multiple directions to generate point cloud data, there is a method for enhancing the color and brightness representation capabilities of the point cloud by adding light ray information from multiple directions to a single point.

[0018] However, with this method, there is a risk that the amount of data in the point cloud will increase in proportion to the number of points. Generally, the number of points in a point cloud is enormous, and the number of points increases even more if the resolution is increased. In other words, with this method, there is a risk that the amount of data in the point cloud will increase.

[0019] For example, suppose an object is captured using the multi-lens camera 100 shown in FIG. 1 . The multi-lens camera 100 has seven element cameras 101, and generates a multi-viewpoint captured image by capturing images with each element camera. The multi-lens camera 100 is also referred to as multi-lens camera C_x (x is an arbitrary integer). If the multi-lens camera 100 is referred to as multi-lens camera C_1, the seven element cameras 101 are also referred to as element camera C_10, element camera C_11, element camera C_12, element camera C_13, element camera C_14, element camera C_15, element camera C_16, and element camera C_17. The element camera C_10 is also referred to as the reference element camera.

[0020] In this way, by obtaining images captured from multiple viewpoints using the camera array 100, depth information of an object can be obtained using the captured images. Note that each element camera 101 is fixedly installed in the camera array 100. In other words, the relative positions of each element camera 101 are fixed.

[0021] The number of element cameras (eyes) in the multi-lens camera 100 is arbitrary, and may be any number other than seven as long as it is two or more. Furthermore, the multi-lens camera 100 that captures an object only needs to be able to obtain depth information for that object, and may be a twin-lens camera or a combination of a TOF sensor and a camera. However, to accurately represent light ray information, a multi-lens camera 100 that can record many light ray vectors spreading from a single point on an object in a single capture and in which RGB information and depth information match at the pixel level is more suitable.

[0022] Such a camera array 100 is used to capture images of an object from multiple positions, thereby obtaining multi-viewpoint captured images. For example, as shown in FIG. 2A, the position of the camera array 100 may be fixed, and an object 122 placed on a turntable 121 may be captured while the turntable 121 is rotated. Alternatively, as shown in FIG. 2B, the camera array 100 may be moved to capture images of the object 122 from multiple positions.

[0023] Regardless of the method, the object 122 is imaged at multiple imaging positions, and multiple-viewpoint images are generated for each imaging position. That is, multiple multi-viewpoint images are generated. Then, the depth is detected using these images, and a point cloud representing the three-dimensional structure of the object 122 is generated.

[0024] <Data Structure 1> As shown in Figure 3, when an M-lens multi-lens camera is moved and an object is photographed K times from various directions, a maximum of M x K light rays can be obtained that are emitted from a single point p_n. A single light ray is composed of a total of six variables: three variables for vector information and three variables for RGB brightness information. Therefore, the information on the light rays emitted from p_n has a maximum of M x K x 6 variables. Adding the global coordinates of p_n to this gives M x K x 6 + 3 variables, which is the number of elements in the point cloud data that contain light ray vector information emitted from one point. If the total number of points that make up the photographed object is N, the total number of elements D_total in the point cloud data that contain light ray vector information can be expressed as in the following equation (1).

[0025] D_total = (M×K×6+3)×N ・・・(1)

[0026] This formula (1) can be transformed into formula (2).

[0027] D_total = ((M × K × 3) + (M × K × 3) + (3)) × N = (first element + third element + second element) × N ... (2)

[0028] The first element (M x K x 3) indicates the RGB luminance information (also referred to as color luminance information) of each element camera of the camera array 100 at each imaging position. The second element (3) indicates the position (x, y, z) of the point. The third element (M x K x 3) indicates the ray vector (a vector representing the ray of light traveling from the point to the element camera) of each element camera of the camera array 100 at each imaging position.

[0029] With such a data format, there is a risk that the amount of data in the point cloud will increase as described above.

[0030] <3. Data Format Conversion> <Data Structure 2> Therefore, the data is converted to the data format shown in FIG. 4 . That is, as shown in FIG. 4A, the ray vector information is calculated from the global position and orientation information of the multi-lens camera for each image capture, the local coordinates of each camera in the multi-lens camera device, and the global coordinate of point p_n. That is, the position of the reference camera C_0 in the multi-lens camera in the global coordinate system when the image is captured with the multi-lens camera and the Euler angle information c_k0 are collected. The local coordinates of the other cameras C_1 to C_(M-1), with the reference camera C_0 in the multi-lens camera as the origin, are made available for reference as fixed values, as shown in FIG. 4B, so that the global coordinates of all cameras can be calculated and found. Note that the optical axes of the cameras C_1 to C_(M-1) may be parallel to the optical axis of the reference camera C_0, or may be offset from the optical axis of the reference camera C_0. In this case, the amount of optical axis offset can be made available for reference as a fixed value. Since the global coordinates of all cameras in the multi-camera device are determined for each shot, all ray vectors emitted from point p_n can be determined.

[0031] By doing this, the total number of elements D_total of point cloud data having ray vector information can be expressed as in the following equation (3).

[0032] D_total=(M×K×3+K×6+3)×N+(M-1)×3 ・・・(3)

[0033] This formula (3) can be transformed into formula (4).

[0034] D_total = ((M × K × 3) + (K × 6) + (3)) × N + ((M-1) × 3) = (first element + third element + second element) × N + fourth element... (4)

[0035] In this case, the first element (M x K x 3) indicates the RGB luminance information (also referred to as color luminance information) of each element camera of the camera array 100 at each imaging position. The second element (3) indicates the position (x, y, z) of the point. The third element (K x 6) indicates the position and orientation of the reference element camera at each imaging position. The fourth element ((M-1) x 3) indicates the relative position of element cameras other than the reference element camera of the camera array 100 with respect to the reference element camera.

[0036] In other words, the data structure of the attributes of a point cloud representing the three-dimensional shape of an object generated using images captured by a multi-lens camera consisting of three or more element cameras may be a data structure that includes a first element indicating the color brightness information of one point in the point cloud corresponding to each element camera, a second element indicating the position of that point, and a third element indicating the position and orientation of a reference element camera, which is one of the element cameras that make up the multi-lens camera.

[0037] In this case, the first element may include the product of color luminance information, the number of element cameras, and the number of imaging positions. Furthermore, the third element may include the product of the position and orientation of the reference element camera and the number of imaging positions. Furthermore, the color luminance information may include RGB luminance information. Furthermore, the camera array may further include a fourth element indicating the relative position of each of the element cameras other than the reference element camera of the camera array relative to the reference element camera.

[0038] By structuring the point cloud in this way, it is assumed that N is sufficiently large, so the total number of data elements can be reduced if M is 3 or more. Therefore, it is possible to suppress an increase in the amount of data in the point cloud.

[0039] 4. First Embodiment Imaging Device The present technology can be applied to any information processing device. For example, the present technology can be applied to an imaging device that captures an image of a subject using a multi-lens camera. Fig. 5 is a block diagram showing an example of the configuration of an imaging device that is one aspect of an information processing device to which the present technology is applied.

[0040] Fig. 5 shows the main processing units, data flows, etc., but is not limited to all that is shown in Fig. 5. In other words, in the imaging device 200, there may be processing units that are not shown as blocks in Fig. 5, and there may be processing and data flows that are not shown as arrows or the like in Fig. 5.

[0041] As shown in FIG. 5, the imaging device 200 (first information processing device) includes an imaging unit 211 , a depth detection unit 212 , a point cloud generation unit 213 , a data format conversion unit 214 , a storage unit 215 , and a communication unit 216 .

[0042] The imaging unit 211 has a multi-lens camera 100, captures an image of a subject, generates a multi-viewpoint captured image, and supplies the image to the depth detection unit 212. The depth detection unit 212 acquires the captured image and uses the acquired captured image to detect the depth of the object 122, which is the subject. The depth detection unit 212 supplies the detected depth to the point cloud generation unit 213. The point cloud generation unit 213 acquires the depth and uses the depth to generate a point cloud (geometry) that represents the three-dimensional structure of the object 122. The point cloud generation unit 213 also generates attributes (color and brightness information) for each point in the point cloud. That is, the first and second elements of the above-mentioned equation (2) are generated.

[0043] The data format conversion unit 214 converts the data format by adding the third element of equation (4) to the point cloud.

[0044] In other words, the data format conversion unit 214 converts the data format of an attribute of a point cloud that represents the three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, and that has a first element that indicates the color brightness information of one point in the point cloud corresponding to each element camera of the multi-lens camera at each imaging position, and a second element that indicates the position of that point, by adding a third element that indicates the position and orientation of a reference element camera, which is one of the element cameras that make up the multi-lens camera.

[0045] In this case, the first element may include the product of color luminance information, the number of element cameras, and the number of imaging positions. Furthermore, the third element may include the product of the position and orientation of the reference element camera and the number of imaging positions. Furthermore, the color luminance information may include RGB luminance information. Furthermore, the camera array may further include a fourth element indicating the relative position of each of the element cameras other than the reference element camera of the camera array relative to the reference element camera.

[0046] By doing this, it is assumed that N is sufficiently large, so the total number of data elements can be reduced if M is 3 or more. Therefore, it is possible to suppress an increase in the amount of data in the point cloud.

[0047] The data format conversion unit 214 may supply the point cloud whose data format has been converted to the storage unit 215. The storage unit 215 may store the point cloud. The storage unit 215 may also supply the stored point cloud to the communication unit 216 as necessary. The storage unit 215 may also be a removable medium, and may be configured to be detachable from the imaging device 200.

[0048] Furthermore, the data format conversion unit 214 may supply the point cloud with the converted data format to the communication unit 216. The communication unit 216 may supply (transmit) the point cloud to an external device of the imaging device 200 (for example, another device such as the display device 300 described later). Furthermore, the communication unit 216 may read out the point cloud stored in the storage unit 215 at a predetermined timing or based on a request from a user, an application, or the like, and supply (transmit) it to an external device of the imaging device 200. Therefore, the communication unit 216 can also be said to be a transmission unit.

[0049] <Flow of Image Capture Processing> An example of the flow of image capture processing executed by the image capture device 200 will be described with reference to the flowchart in Fig. 6. When the image capture processing starts, in step S201, the image capture unit 211 captures an image of an object that is a subject, and generates a multi-view captured image.

[0050] In step S202, the depth detection unit 212 detects the depth of the object using the captured image.

[0051] In step S203, the point cloud generation unit 213 generates a point cloud using the depth and the captured image.

[0052] In step S204, the data format conversion unit 214 converts the data format of the point cloud as described above. That is, the data format conversion unit 214 converts the data format of the point cloud attribute, which is generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, and which has a first element indicating color intensity information of one point in the point cloud corresponding to each element camera of the multi-lens camera at each imaging position, and a second element indicating the position of that point, by adding a third element indicating the position and orientation of a reference element camera that is one of the element cameras constituting the multi-lens camera.

[0053] In step S205, the storage unit 215 stores the point cloud whose data format has been converted.

[0054] In step S206, the communication unit 216 transmits the point cloud whose data format has been converted. When the process of step S206 ends, the imaging process ends.

[0055] By performing each process in this way, it is assumed that N is sufficiently large, so if M is 3 or more, the total number of data elements can be reduced. Therefore, the increase in the amount of data in the point cloud can be suppressed.

[0056] 5. Second Embodiment Display Device The present technology can be applied to any information processing device. For example, the present technology can be applied to a display device that renders and displays a point cloud. Fig. 7 is a block diagram showing an example of the configuration of a display device that is one aspect of an information processing device to which the present technology is applied.

[0057] Fig. 7 shows the main processing units, data flows, etc., but does not necessarily include everything shown in Fig. 7. In other words, in the display device 300, there may be processing units that are not shown as blocks in Fig. 7, and there may be processing or data flows that are not shown as arrows or the like in Fig. 7.

[0058] As shown in FIG. 7, the display device 300 (second information processing device) has a communication unit 311 , a drive 312 , a data format reverse conversion unit 313 , an input unit 314 , a display image generation unit 315 , and a display unit 316 .

[0059] The communication unit 311 communicates with other devices (for example, the imaging device 200), acquires (receives) point clouds supplied from the other devices, and supplies them to the data format inverse conversion unit 313. Therefore, the communication unit 311 can also be said to be a receiving unit.

[0060] If the storage unit 215 is a removable medium, the drive 312 reads the point cloud from the storage unit 215 attached to the drive 312 and supplies it to the data format inverse conversion unit 313 .

[0061] The data format inverse conversion unit 313 inversely converts the data format of the acquired point cloud. That is, the data format inverse conversion unit 313 inversely converts the data format of the point cloud by converting the third element of formula (4) into the third element of formula (2). That is, for an attribute of a point cloud that represents a three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera configured with three or more element cameras, the attribute having a first element indicating color intensity information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each imaging position, a second element indicating the position of the point, and a third element indicating the position and orientation of a reference element camera that is one of the element cameras constituting the multi-lens camera, the data format inverse conversion unit 313 inversely converts the data format of the attribute by converting the third element of formula (4) into a fourth element (the third element of formula (2)) indicating the light rays of the element cameras. The data format inverse conversion unit 313 supplies the point cloud whose data format has been inversely converted to the display image generation unit 315 .

[0062] The input unit 314 receives designation of the position and direction of the line of sight, and supplies the information to the display image generation unit 315 .

[0063] The display image generation unit 315 generates a display image based on the point cloud whose data format has been inversely converted, supplied from the data format inverse conversion unit 313, and the line of sight specification, supplied from the input unit 314. For example, the display image generation unit 315 renders the point cloud whose data format has been inversely converted, and generates a display image corresponding to the viewpoint. In doing so, the display image generation unit 315 may interpolate color and brightness information of the first element to generate color and brightness information corresponding to the viewpoint. The display image generation unit 315 supplies the display image generated in this manner to the display unit 316.

[0064] The display unit 316 has a display device and displays the supplied display image.

[0065] In this way, the display image of the point cloud can be displayed correctly, in other words, an increase in the amount of data of the point cloud can be suppressed.

[0066] <Flow of Display Processing> An example of the flow of the display processing executed by the display device 300 will be described with reference to the flowchart in Fig. 8. When the display processing starts, the communication unit 311 acquires a point cloud in step S301.

[0067] In step S302, the data format inverse conversion unit 313 inversely converts the data format of the point cloud. For example, the data format inverse conversion unit 313 inversely converts the data format of an attribute of a point cloud that represents a three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera configured with three or more element cameras, the attribute having a first element indicating color intensity information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each imaging position, a second element indicating the position of the point, and a third element indicating the position and orientation of a reference element camera that is one of the element cameras that configure the multi-lens camera, by converting the third element into a fourth element indicating the light rays of the element cameras.

[0068] In step S303, the input unit 314 accepts the designation of the viewpoint.

[0069] In step S304, the display image generating unit 315 generates RGB luminance information according to the viewpoint.

[0070] In step S305, the display image generation unit 315 renders the point cloud whose data format has been inversely converted according to the viewpoint, and generates a display image corresponding to that viewpoint.

[0071] In step S306, the display unit 316 displays the display image. When the process of step S306 ends, the display process ends.

[0072] By performing each process in this manner, the display image of the point cloud can be displayed correctly, in other words, an increase in the amount of data of the point cloud can be suppressed.

[0073] 6. Supplementary Notes Computer The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, the term computer includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0074] FIG. 9 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0075] In a computer 900 shown in FIG. 9, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0076] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0077] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0078] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0079] The program executed by the computer can be applied by recording it on, for example, a removable medium 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable medium 921 into the drive 915.

[0080] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0081] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0082] <Application of the Present Technology> The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0083] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).

[0084] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0085] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0086] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.

[0087] <Others> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in a bitstream, but also in a bitstream that includes differential information of the identification information relative to certain reference information. Therefore, in this specification, "flag" and "identification information" encompass not only the information itself, but also differential information relative to the reference information.

[0088] Furthermore, various types of information (e.g., metadata) related to the coded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, the associated pieces of data may be combined into one piece of data or may be separate pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Furthermore, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer not to the entire data, but to only part of the data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0089] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0090] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0091] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0092] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0093] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0094] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0095] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0096] The present technology can also be configured as follows: (1) A data structure of attributes of a point cloud representing a three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, the data structure including: a first element indicating color intensity information of one point in the point cloud corresponding to each of the element cameras; a second element indicating a position of the point; and a third element indicating a position and orientation of a reference element camera that is one of the element cameras constituting the multi-lens camera. (2) The data structure according to (1), in which the first element includes a product of the color intensity information, the number of the element cameras, and the number of imaging positions. (3) The data structure according to (1) or (2), in which the third element includes a product of the position and orientation of the reference element camera and the number of imaging positions. (4) The data structure according to any of (1) to (3), in which the color intensity information includes RGB intensity information. (5) The data structure according to any one of (1) to (4), further including a fourth element indicating a relative position of each of the element cameras other than the reference element camera of the multi-lens camera with respect to the reference element camera.

[0097] (6) An information processing device including a data format conversion unit that converts a data format of an attribute of a point cloud representing a three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, the attribute having a first element indicating color intensity information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each imaging position, and a second element indicating the position of the point, by adding a third element indicating the position and orientation of a reference element camera that is one of the element cameras constituting the multi-lens camera. (7) The information processing device according to (6), wherein the first element includes a product of the color intensity information, the number of the element cameras, and the number of imaging positions. (8) The information processing device according to (6) or (7), wherein the third element includes a product of the position and orientation of the reference element camera and the number of imaging positions. (9) The information processing device according to any of (6) to (8), wherein the color intensity information includes RGB intensity information. (10) The information processing device according to any one of (6) to (9), wherein the data format conversion unit further adds a fourth element indicating a relative position of each of the element cameras of the multi-lens camera other than the reference element camera with respect to the reference element camera. (11) The information processing device according to any one of (6) to (10), further comprising a point cloud generation unit that generates the point cloud using the captured images, wherein the data format conversion unit is configured to convert the data format of the attributes of the generated point cloud. (12) The information processing device according to (11), further comprising a depth detection unit that detects a depth of the object using the captured images, wherein the point cloud generation unit is configured to generate the point cloud using the detected depth. (13) The information processing device according to (12), further comprising an imaging unit that has the multi-lens camera, images the object, and generates the captured images, wherein the depth detection unit is configured to detect the depth of the object using the generated captured images.(14) The information processing device according to any one of (6) to (13), further comprising a storage unit that stores the point cloud whose data format has been converted. (15) The information processing device according to any one of (6) to (14), further comprising a transmission unit that transmits the point cloud whose data format has been converted. (16) An information processing method for converting the data format of an attribute of a point cloud that represents a three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, the attribute having a first element that indicates color intensity information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each imaging position, and a second element that indicates a position of the point, by adding a third element that indicates a position and orientation of a reference element camera that is one of the element cameras that constitute the multi-lens camera.

[0098] (17) An information processing device comprising: a data format inverse conversion unit that inversely converts a data format of an attribute of a point cloud representing a three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, the attribute having a first element indicating color and brightness information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each imaging position, a second element indicating a position of the point, and a third element indicating the position and orientation of a reference element camera that is one of the element cameras constituting the multi-lens camera, by converting the third element into a fourth element indicating a light ray of the element camera; and a display image generation unit that renders the point cloud whose data format has been inversely converted, and generates a display image corresponding to a viewpoint. (18) The information processing device described in (17), wherein the display image generation unit interpolates the color and brightness information of the first element to generate the color and brightness information according to the viewpoint. (19) The information processing device described in (18), further comprising a display unit that displays the display image. (20) An information processing method for inversely converting a data format of a point cloud attribute representing a three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, the attribute having a first element indicating color intensity information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each capturing position, a second element indicating the position of the point, and a third element indicating the position and orientation of a reference element camera that is one of the element cameras constituting the multi-lens camera, by converting the third element into a fourth element indicating the light rays of the element cameras, and rendering the point cloud whose data format has been inversely converted to generate a display image corresponding to a viewpoint.

[0099] 200 Imaging device, 211 Imaging unit, 212 Depth detection unit, 213 Point cloud generation unit, 214 Data format conversion unit, 215 Storage unit, 216 Communication unit, 300 Display device, 311 Communication unit, 312 Drive, 313 Data format reverse conversion unit, 314 Input unit, 315 Display image generation unit, 316 Display unit, 900 Computer

Claims

1. An information processing device having a data format conversion unit that converts the data format of a point cloud attribute that represents the three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, the attribute having a first element that indicates the color brightness information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each capturing position, and a second element that indicates the position of the point, by adding a third element that indicates the position and orientation of a reference element camera that is one of the element cameras that make up the multi-lens camera.

2. The information processing device according to claim 1, wherein the first element includes a product of the color luminance information, the number of the element cameras, and the number of imaging positions.

3. The information processing device according to claim 1, wherein the third element includes a product of the position and orientation of the reference element camera and the number of imaging positions.

4. The information processing device according to claim 1, wherein the color luminance information includes RGB luminance information.

5. The information processing device according to claim 1, wherein the data format conversion unit further adds a fourth element indicating the relative position of each of the element cameras other than the reference element camera of the camera array relative to the reference element camera.

6. The information processing device according to claim 1, further comprising a point cloud generation unit that generates the point cloud using the captured image, wherein the data format conversion unit is configured to convert the data format of the attributes of the generated point cloud.

7. The information processing device according to claim 6, further comprising a depth detection unit that detects the depth of the object using the captured image, and the point cloud generation unit is configured to generate the point cloud using the detected depth.

8. The information processing device according to claim 7, further comprising an imaging unit having the multi-lens camera, capturing an image of the object, and generating the captured image, wherein the depth detection unit is configured to detect the depth of the object using the generated captured image.

9. The information processing device according to claim 1, further comprising a storage unit that stores the point cloud whose data format has been converted.

10. The information processing device according to claim 1, further comprising a transmitting unit for transmitting the point cloud whose data format has been converted.

11. An information processing method for converting the data format of a point cloud attribute representing the three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, the attribute having a first element indicating color intensity information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each imaging position, and a second element indicating the position of the point, by adding a third element indicating the position and orientation of a reference element camera, which is one of the element cameras that make up the multi-lens camera.

12. An information processing device comprising: a data format inverse conversion unit that inversely converts the data format of a point cloud attribute that represents the three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, the attribute having a first element that indicates color intensity information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each capturing position, a second element that indicates the position of the point, and a third element that indicates the position and orientation of a reference element camera that is one of the element cameras that make up the multi-lens camera, by converting the third element into a fourth element that indicates the light rays of the element cameras; and a display image generation unit that renders the point cloud whose data format has been inversely converted and generates a display image corresponding to the viewpoint.

13. The information processing device according to claim 12, wherein the display image generating section generates the color and luminance information according to the viewpoint by interpolating the color and luminance information of the first element.

14. The information processing device according to claim 13, further comprising a display unit for displaying the display image.

15. An information processing method for generating a display image corresponding to a viewpoint by inversely converting the data format of a point cloud attribute representing the three-dimensional shape of an object generated using captured images of the object captured by a multi-lens camera consisting of three or more element cameras, the attribute having a first element indicating color intensity information of one point in the point cloud corresponding to each of the element cameras of the multi-lens camera at each capturing position, a second element indicating the position of the point, and a third element indicating the position and orientation of a reference element camera that is one of the element cameras constituting the multi-lens camera, by converting the third element into a fourth element indicating the light rays of the element cameras, and rendering the point cloud with the inversely converted data format to generate a display image corresponding to the viewpoint.

Images