Image processing device, image processing method, and program

Abstract

It is desirable to provide a technology that can reduce the size of two-dimensional images generated from 3D object representation data while suppressing a decrease in the quality of those images. [Solution] An image processing device is provided, comprising: an image generation unit that generates a two-dimensional image based on the superposition of multiple three-dimensional object representation data; and a quantization unit that determines a quantization size for each block based on the multiple three-dimensional object representation data and quantizes the two-dimensional image based on the quantization size.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] In recent years, a technique has been known for generating data representing a three-dimensional (hereinafter also referred to as "3D") object (hereinafter also referred to as "3D object representation data") and generating a two-dimensional image of the 3D object viewed from an arbitrary viewpoint based on the generated 3D object representation data. There are various types of 3D object representation data.

[0003] For example, the 3D object representation data may be composed of a combination of a plurality of triangles. In such a case, the shape of the 3D object can be represented by a combination of a plurality of triangles. Alternatively, the 3D object representation data may be composed of a point cloud or voxels instead of triangles.

[0004] Also, Non-Patent Document 1 discloses a method called 3D Gaussian Splatting as a method for representing a 3D object. In 3D Gaussian Splatting, 3D Gaussians, which are data following a three-dimensional Gaussian distribution, are included in the 3D object representation data. In the real space, since many phenomena follow a Gaussian distribution, the 3D object existing in the real space can be accurately represented by including 3D Gaussians in the 3D object representation data.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

[0006] However, to improve the quality of two-dimensional images generated from 3D object representation data, the size of the 3D object representation data needs to be increased, which in turn increases the amount of processing required to generate two-dimensional images from the 3D object representation data. Therefore, a configuration may be adopted in which two-dimensional images are generated from 3D object representation data by a server with higher processing power, and then transmitted from that server to terminal devices with lower processing power.

[0007] In this context, from the perspective of reducing the load on the network caused by the transmission of two-dimensional images, it is desirable for the data size of the two-dimensional images transmitted to the terminal device to be as small as possible.

[0008] Therefore, it is desirable to provide a technology that can reduce the data size of a two-dimensional image while suppressing the degradation of the quality of the two-dimensional image generated from 3D object representation data. [Means for solving the problem]

[0009] To solve the above problem, according to one aspect of the present invention, an image processing device is provided, comprising: an image generation unit that generates a two-dimensional image based on the superposition of a plurality of three-dimensional object representation data; and a quantization unit that determines a quantization size for each block based on the plurality of three-dimensional object representation data and quantizes the two-dimensional image based on the quantization size.

[0010] The quantization unit may determine the quantization size for each block based on two-dimensional projection data corresponding to each of the multiple three-dimensional object representation data, which is calculated by projecting the multiple three-dimensional object representation data onto a two-dimensional screen corresponding to the viewpoint.

[0011] The quantization unit may determine the minimum size of the two-dimensional shape shown by the two-dimensional projection data for each block, and determine the quantization size for each block according to the minimum size.

[0012] The quantization unit may reduce the quantization size to a smaller value than when the minimum size is equal to or greater than the first threshold if the minimum size is smaller than the first threshold.

[0013] The quantization unit may identify, for each pixel constituting the two-dimensional screen, three-dimensional object representation data whose degree of influence on the pixel value is greater than a second threshold as high-influence data, and determine the minimum size of the two-dimensional shape shown by the two-dimensional projection data corresponding to the high-influence data for each block.

[0014] The plurality of three-dimensional object representation data include a first three-dimensional object representation data, and the quantization unit may calculate the degree of influence of the first three-dimensional object representation data on the pixel value based on the distance between the position of the center point of the first three-dimensional object representation data and the position of the pixel.

[0015] The quantization unit may calculate the influence such that the influence decreases as the distance increases.

[0016] The plurality of three-dimensional object representation data include a first three-dimensional object representation data, and the quantization unit may calculate the degree of influence of the first three-dimensional object representation data on the pixel values ​​based on the opacity of the center point of the first three-dimensional object representation data.

[0017] The quantization unit may calculate the influence such that the influence increases as the opacity increases.

[0018] The plurality of three-dimensional object representation data include a first three-dimensional object representation data and a second three-dimensional object representation data. The quantization unit may, when the position of the center point of the first three-dimensional object representation data is farther than the position of the center point of the second three-dimensional object representation data with respect to the viewpoint, calculate the transparency of the first three-dimensional object representation data based on the opacity of the center point of the second three-dimensional object representation data, and calculate the degree of influence of the first three-dimensional object representation data on the pixel values ​​based on the transparency of the first three-dimensional object representation data.

[0019] The quantization unit may calculate the influence such that the influence increases as the transmittance increases.

[0020] The two-dimensional shape is an ellipse, and the size of the two-dimensional shape may be the radius of the minor axis of the ellipse.

[0021] Furthermore, in order to solve the above problems, according to another aspect of the present invention, there is a computer-based image processing method that includes generating a two-dimensional image based on superimposing a plurality of three-dimensional object representation data, determining a quantization size for each block based on the plurality of three-dimensional object representation data, and quantizing the two-dimensional image based on the quantization size.

[0022] Furthermore, in order to solve the above problems, according to another aspect of the present invention, a program is provided that causes a computer to function as an image generation unit that generates a two-dimensional image based on the superposition of a plurality of three-dimensional object representation data, and a quantization unit that determines a quantization size for each block based on the plurality of three-dimensional object representation data and quantizes the two-dimensional image based on the quantization size. [Effects of the Invention]

[0023] As described above, according to the present invention, there is provided a technology that can reduce the data size of a two-dimensional image while suppressing a decrease in the quality of the two-dimensional image generated from 3D object representation data.

Brief Description of the Drawings

[0024] [Figure 1] It is a diagram showing a functional configuration example of an image processing system according to an embodiment of the present invention. [Figure 2] It is a diagram showing a functional configuration example of a terminal device 10 according to an embodiment of the present invention. [Figure 3] It is a diagram showing a functional configuration example of an image processing device 20 according to an embodiment of the present invention. [Figure 4] It is a diagram showing examples of a plurality of 3D ellipsoid data. [Figure 5] It is a diagram showing an example of dividing a two-dimensional screen G into macro blocks. [Figure 6] It is a diagram showing an example of two-dimensional projection data. [Figure 7] It is a flowchart showing an overall operation example of an image processing device 20 according to an embodiment of the present invention. [Figure 8] It is a flowchart showing a detailed operation example (first half) of image processing (S13). [Figure 9] It is a flowchart showing a detailed operation example (second half) of image processing (S13). [Figure 10] It is a flowchart showing a detailed operation example of minimum ellipsoid size update processing (S44). [Figure 11] It is a flowchart showing a detailed operation example of quantization processing (S14). [Figure 12] It is a diagram showing a hardware configuration of an information processing device 900 as an example of an image processing device 20 according to an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0025] Preferred embodiments of the present invention will be described in detail below with reference to the attached drawings. In this specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0026] (0. Overview of the Embodiment) An overview of embodiments of the present invention will be described.

[0027] In embodiments of the present invention, 3D object representation data is generated in advance, and a two-dimensional image is generated from the 3D object representation data. The 3D object representation data is data that represents a 3D object. Embodiments of the present invention mainly describe the case in which the 3D object representation data includes data that follows a three-dimensional Gaussian distribution. Data that follows a three-dimensional Gaussian distribution may include the position of the center point of the data and the 3D covariance matrix of the data. However, the 3D object representation data may also include data other than data that follows a three-dimensional Gaussian distribution.

[0028] Here, data following a three-dimensional Gaussian distribution has a three-dimensional ellipsoidal shape. Therefore, in the following, data following a three-dimensional Gaussian distribution will also be simply referred to as "3D ellipsoidal data." Similarly, the shape represented by 3D ellipsoidal data will also be simply referred to as "3D ellipsoid."

[0029] More specifically, embodiments of the present invention primarily assume the use of the 3D Gaussian disclosed in Non-Patent Document 1 as the 3D ellipsoid data. That is, the 3D ellipsoid data is associated with the opacity of the center point of the 3D ellipsoid and the color information of the 3D ellipsoid. Here, opacity corresponds to a value indicating how difficult it is for light to pass through. The color information is information indicating the color that depends on the direction of the viewpoint relative to the position of the 3D ellipsoid.

[0030] 3D ellipsoid data may be generated in any way. For example, 3D ellipsoid data may be generated from original images obtained by capturing a 3D object from multiple viewpoints. As an example, embodiments of the present invention mainly assume that 3D ellipsoid data is generated in advance from original images during the learning phase using the method disclosed in Non-Patent Document 1.

[0031] The embodiments of the present invention have been described above.

[0032] (1. Details of the Embodiment) Next, we will describe the details of embodiments of the present invention.

[0033] (1-1. Configuration of the image processing system) First, an example of the configuration of an image processing system according to an embodiment of the present invention will be described. Figure 1 is a diagram showing an example of the functional configuration of an image processing system according to an embodiment of the present invention. As shown in Figure 1, the image processing system 1 includes a terminal device 10, an image processing device 20, and a network 30. The terminal device 10 and the image processing device 20 are each connected to the network 30.

[0034] (Terminal device 10) Terminal device 10 is implemented by a computer and displays a two-dimensional image generated by an image processing device 20. The two-dimensional image displayed by terminal device 10 can be viewed by the user of terminal device 10. Terminal device 10 may also be capable of accepting input for viewpoint for generating the two-dimensional image. For example, terminal device 10 may be a smartphone, tablet device, or PC (Personal Computer).

[0035] (Image processing device 20) The image processing device 20 is implemented by a computer and generates a two-dimensional image from 3D ellipsoid data. Specifically, the image processing device 20 can generate a two-dimensional image corresponding to the viewpoint from the 3D ellipsoid data. The image processing device 20 then transmits the generated two-dimensional image to the terminal device 10 via the network 30. The image processing device 20 can function as a server for the terminal device 10. Therefore, it is generally assumed that the processing performance of the image processing device 20 is higher than the processing performance of the terminal device 10.

[0036] (Network 30) The network 30 connects the terminal device 10 and the image processing device 20. The network 30 can also function as a communication path between the terminal device 10 and the image processing device 20.

[0037] (1-2. Functional configuration of terminal device 10) Next, an example of the functional configuration of the terminal device 10 according to an embodiment of the present invention will be described. Figure 2 is a diagram showing an example of the functional configuration of the terminal device 10 according to an embodiment of the present invention. As shown in Figure 2, the terminal device 10 comprises an operation unit 110, a control unit 120, a storage unit 130, a communication unit 140, and a display unit 150.

[0038] (Operation unit 110) The operation unit 110 has the function of receiving operations from the user. For example, the operation unit 110 may receive input from the user for a viewpoint to generate a two-dimensional image. The viewpoint for generating a two-dimensional image may correspond to the orientation and position of a virtual camera having a 2D screen onto which 3D ellipsoid data is projected. For example, the operation unit 110 may be implemented by a mouse or a touch panel.

[0039] (Control unit 120) The control unit 120 performs various calculation processes. Data necessary for calculations performed by the arithmetic unit is appropriately stored in the storage unit 130.

[0040] For example, the control unit 120 may include an arithmetic unit such as a CPU (Central Processing Unit), and its function may be realized when a program stored in ROM (Read Only Memory) is loaded into RAM (Random Access Memory) by the arithmetic unit and executed. In this case, a computer-readable recording medium on which the program is stored may also be provided. Alternatively, these blocks may be composed of dedicated hardware or a combination of multiple hardware components.

[0041] (Storage unit 130) The memory unit 130 is a memory capable of storing programs and data for operating the control unit 120. The memory unit 130 can also temporarily store various data required during the operation of the control unit 120. For example, the memory unit 130 may be a non-volatile memory.

[0042] (Communications Section 140) The communication unit 140 is configured with a communication interface and communicates with the image processing device 20 via the network 30. For example, the communication unit 140 can receive a two-dimensional image generated by the image processing device 20 via the network 30. The communication unit 140 can also transmit the viewpoint input to the operation unit 110 to the image processing device 20 via the network 30.

[0043] (Display section 150) The display unit 150 is implemented by a display and has the function of displaying various information to the user according to the control unit 120. For example, the display unit 150 can display a two-dimensional image received by the communication unit 140.

[0044] For example, the display unit 150 may be a liquid crystal display (LCD) device or an OLED (Organic Light Emitting Diode) device. Furthermore, the display unit 150 may be a head-mounted display worn on the user's head or a stationary display.

[0045] (1-3. Functional configuration of the image processing device 20) Next, an example of the functional configuration of the image processing apparatus 20 according to an embodiment of the present invention will be described. Figure 3 is a diagram showing an example of the functional configuration of the image processing apparatus 20 according to an embodiment of the present invention. As shown in Figure 3, the image processing apparatus 20 comprises a control unit 220, a storage unit 230, and a communication unit 240.

[0046] (Control unit 220) The control unit 220 performs various calculation processes. Data necessary for calculations by the arithmetic unit is appropriately stored in the storage unit 230. As shown in Figure 3, the control unit 220 comprises a data acquisition unit 221, a viewpoint determination unit 222, an image generation unit 223, a quantization unit 224, an encoding unit 225, and a transmission control unit 226. The details of the functions of the data acquisition unit 221, viewpoint determination unit 222, image generation unit 223, quantization unit 224, encoding unit 225, and transmission control unit 226 will be described later.

[0047] For example, the control unit 220 may include an arithmetic unit such as a CPU (Central Processing Unit), and its function may be realized when a program stored in ROM (Read Only Memory) is loaded into RAM (Random Access Memory) by the arithmetic unit and executed. In this case, a computer-readable recording medium on which the program is stored may also be provided. Alternatively, these blocks may be composed of dedicated hardware or a combination of multiple hardware components.

[0048] (Storage unit 230) The memory unit 230 is a memory capable of storing programs and data for operating the control unit 220. For example, the memory unit 230 stores multiple 3D ellipsoid data. The memory unit 230 also stores the opacity of the center point of each of the multiple 3D ellipsoid data and the color information of the 3D ellipsoid, which are associated with each of the multiple 3D ellipsoid data. The memory unit 230 can also temporarily store various data necessary during the operation of the control unit 220. For example, the memory unit 230 may be a non-volatile memory.

[0049] (Communications Section 240) The communication unit 240 is configured with a communication interface and communicates with the terminal device 10 via the network 30. For example, the communication unit 240 can transmit a two-dimensional image generated by the control unit 220 to the terminal device 10 via the network 30. The communication unit 240 can also receive a viewpoint for generating the two-dimensional image, transmitted from the terminal device 10, via the network 30.

[0050] The above describes an example of the functional configuration of the image processing apparatus 20 according to an embodiment of the present invention.

[0051] (1-4. Detailed functions of the image processing device 20) Next, with reference to Figures 4 to 6 (and Figures 1 to 3 as appropriate), the detailed functions of the image processing apparatus 20 according to the embodiment of the present invention will be described.

[0052] (Data acquisition unit 221) The data acquisition unit 221 acquires multiple 3D ellipsoid data from the storage unit 230. Each of the multiple 3D ellipsoid data represents the shape of the 3D ellipsoid. The data acquisition unit 221 also acquires the opacity of the center point of the 3D ellipsoid and the color information of the 3D ellipsoid, which are associated with each of the multiple 3D ellipsoid data, from the storage unit 230. Here, the multiple 3D ellipsoid data will be explained with reference to Figure 4.

[0053] Figure 4 shows an example of multiple 3D ellipsoid data sets. Referring to Figure 4, 3D ellipsoid data sets V1 to V3 are shown as multiple 3D ellipsoid data sets. Note that the number of 3D ellipsoid data sets is not limited to three; it could be two, four or more, or any number.

[0054] The 3D ellipsoid data V1 may include the position U1 of the center point of the 3D ellipsoid data V1, as well as the 3D covariance matrix of the 3D ellipsoid data V1. Similarly, the 3D ellipsoid data V2 may include the position U2 of the center point of the 3D ellipsoid data V2, as well as the 3D covariance matrix of the 3D ellipsoid data V2. Similarly, the 3D ellipsoid data V3 may include the position U3 of the center point of the 3D ellipsoid data V3, as well as the 3D covariance matrix of the 3D ellipsoid data V3.

[0055] Furthermore, referring to Figure 4, a viewpoint E for generating a two-dimensional image is shown, and a two-dimensional screen G corresponding to viewpoint E is also shown. The two-dimensional screen G is composed of multiple pixels. In Figure 4, one of these multiple pixels is shown as pixel P. Also, the field of view A of the virtual camera having the two-dimensional screen G is shown. The ellipsoid data V1~V3 are within the range of field of view A. Therefore, the 3D ellipsoid data V1~V3 can be projected onto the two-dimensional screen G.

[0056] (Perspective determination unit 222) The viewpoint determination unit 222 determines viewpoint E. The viewpoint determination unit 222 may determine viewpoint E in any way. For example, the viewpoint determination unit 222 may determine viewpoint E according to predetermined rules. As one example, the viewpoint determination unit 222 may determine the viewpoint so that it rotates around the 3D object while facing the 3D object. As another example, the viewpoint determination unit 222 may determine the viewpoint so that it moves back and forth near the 3D object while facing the 3D object. Alternatively, the viewpoint determination unit 222 may determine the viewpoint input by the user to the terminal device 10 as viewpoint E.

[0057] (Image generation unit 223) The image generation unit 223 generates a two-dimensional image based on the superimposition of multiple 3D ellipsoid data. In the example shown in Figure 4, the image generation unit 223 calculates two-dimensional projection data corresponding to each of the 3D ellipsoid data V1 to V3 by projecting each of the 3D ellipsoid data V1 to V3 onto a two-dimensional screen G. The image generation unit 223 can then generate a two-dimensional image by adding up the two-dimensional projection data corresponding to each of the 3D ellipsoid data V1 to V3.

[0058] The two-dimensional projection data corresponding to each of the 3D ellipsoid data V1 to V3 may be summed using the color information of the 3D ellipsoid associated with each of the 3D ellipsoid data V1 to V3. That is, the image generation unit 223 may generate a two-dimensional image by summing the color information corresponding to the 3D ellipsoid projected onto each pixel that constitutes the two-dimensional screen G.

[0059] In this case, the opacity of the center point of the 3D ellipsoid associated with each of the 3D ellipsoid data V1 to V3 may be used. That is, the image generation unit 223 may reduce the contribution of the color information corresponding to the 3D ellipsoid (the 3D ellipsoid shown by 3D ellipsoid data V1) to the pixel value of the pixel as the opacity of the center point of the 3D ellipsoid (the 3D ellipsoid shown by 3D ellipsoid data V2 and 3D ellipsoid data V3) that is closer to the viewpoint E than the 3D ellipsoid in question is greater.

[0060] Similarly, the image generation unit 223 may reduce the contribution of color information corresponding to a 3D ellipsoid (the 3D ellipsoid shown by 3D ellipsoid data V2) to the pixel value of a pixel, as the opacity of the center point of a 3D ellipsoid (the 3D ellipsoid shown by 3D ellipsoid data V3) that is closer to the viewpoint E than the 3D ellipsoid in question increases. The method of superimposing multiple 3D ellipsoid data is described in detail in Non-Patent Literature 1.

[0061] (Quantization unit 224) The quantization unit 224 determines the quantization size for each block of the two-dimensional screen G based on the two-dimensional projection data corresponding to each of the multiple 3D ellipsoid data, and quantizes the two-dimensional image based on the quantization size. This makes it possible to reduce the data size of the two-dimensional image while suppressing a decrease in the quality of the two-dimensional image generated from the multiple 3D ellipsoid data.

[0062] Note that a larger quantization size may mean that quantization is performed more coarsely. Therefore, the larger the quantization size, the smaller the data size of the quantized two-dimensional image may be. Also, a block is composed of multiple pixels. For this reason, in the following explanation, a block will also be called a "macroblock". How the two-dimensional screen G is divided into multiple macroblocks is not limited.

[0063] Figure 5 shows an example of dividing a two-dimensional screen G into macroblocks. Referring to Figure 5, the two-dimensional screen G is divided into M × N macroblocks B. The size of each macroblock B is not limited. For example, each macroblock B may have a size of 16 (pixels) × 16 (pixels).

[0064] The quantization unit 224 determines the quantization size for each macroblock based on the two-dimensional projection data corresponding to each of the multiple 3D ellipsoid data. More specifically, the quantization unit 224 determines the quantization size for each macroblock based on the size of the two-dimensional shape shown by the two-dimensional projection data corresponding to each of the multiple 3D ellipsoid data. Here, an example of two-dimensional projection data will be explained with reference to Figure 6.

[0065] Figure 6 shows an example of two-dimensional projection data. Referring to Figure 6, the two-dimensional projection data v1 corresponding to the 3D ellipsoid data V1 (Figure 4) is shown. As described above, if the shape shown by the 3D ellipsoid data V1 is an ellipsoid, then the two-dimensional shape shown by the two-dimensional projection data v1 is an ellipse. The position U1 of the center point of the 3D ellipsoid data V1 is projected onto the position u1 of the center point of the two-dimensional projection data v1, and the position u1 of the center point of the two-dimensional projection data v1 is macroblock B 22 It is included in [the specified location]. Also, pixel P is shown.

[0066] Distance d1 is the distance between the position u1 of the center point of the two-dimensional projection data v1 and the two-dimensional position of pixel P. The major axis radius of the two-dimensional shape represented by the two-dimensional projection data v1 is λ1, and the minor axis radius of the two-dimensional shape represented by the two-dimensional projection data v1 is λ2. In this case, the minor axis radius λ2 may be used as the size of the two-dimensional shape represented by the two-dimensional projection data v1. This increases the likelihood that the minor axis radius λ2 corresponds to the minimum ellipsoid size t_min, which will be explained later, and increases the likelihood that the macroblock will be finely quantized.

[0067] The quantization unit 224 determines the minimum size of the two-dimensional shape represented by the two-dimensional projection data corresponding to each of the multiple 3D ellipsoid data (hereinafter also referred to as the "minimum ellipsoid size") for each macroblock. In the following explanation, the minimum ellipsoid size for each macroblock will also be denoted as "t_min". Then, the quantization unit 224 determines the quantization size for each macroblock according to the minimum ellipsoid size t_min.

[0068] More specifically, when the minimum ellipsoid size t_min is smaller than the first threshold, it is considered more likely that the macroblock contains high-frequency components than when the minimum ellipsoid size t_min is greater than or equal to the first threshold. In the following explanation, the first threshold will also be denoted as "T_thr". Macroblocks containing high-frequency components are more complex, so it is desirable to perform finer quantization to suppress the degradation of the macroblock quality. On the other hand, macroblocks that do not contain high-frequency components are simpler, so it is desirable to perform coarser quantization to reduce the data size of the macroblock.

[0069] Therefore, when the minimum ellipsoid size t_min is smaller than the threshold T_thr, it is desirable for the quantization unit 224 to reduce the quantization size compared to when the minimum ellipsoid size t_min is greater than or equal to the threshold T_thr.

[0070] Furthermore, when the quantization unit 224 determines the minimum ellipsoid size t_min, the sizes of 3D ellipsoids whose influence on the pixel value of a pixel is less than or equal to a second threshold may be excluded from the candidates for the minimum ellipsoid size t_min. In the following explanation, the influence will also be denoted as "e" and the second threshold as "E_thr".

[0071] In other words, the quantization unit 224 may identify 3D ellipsoid data whose influence on the pixel value of each pixel is greater than a second threshold as high-influence data for each pixel constituting the two-dimensional screen G. The quantization unit 224 may then determine the minimum size of the two-dimensional shape shown by the two-dimensional projection data corresponding to the high-influence data as the minimum ellipsoid size t_min for each block.

[0072] As illustrated in Figure 6, the quantization unit 224 may calculate the degree of influence e of pixel P in the 3D ellipsoid data V1 based on the distance d1 between the position u1 of the center point of the two-dimensional projection data v1 and the position of pixel P. More specifically, the quantization unit 224 may calculate the degree of influence e such that the degree of influence e decreases as the distance d1 increases. Note that the 3D ellipsoid data V1 may correspond to the first three-dimensional object representation data.

[0073] Alternatively, the quantization unit 224 may calculate the degree of influence e of the pixel P of the 3D ellipsoid data V1 on the pixel value based on the opacity of the center point of the 3D ellipsoid data V1. More specifically, the quantization unit 224 may calculate the degree of influence e such that the degree of influence e increases as the opacity of the center point of the 3D ellipsoid data V1 increases.

[0074] Furthermore, as illustrated in Figure 4, the position U1 of the center point of the 3D ellipsoid data V1 is farther from the position U2 of the center point of the 3D ellipsoid data V2 and the position U3 of the center point of the 3D ellipsoid data V3, relative to the viewpoint E. In such a case, the quantization unit 224 may calculate the transparency of the 3D ellipsoid data V1 based on the opacity of the center point of the 3D ellipsoid data V2 and the opacity of the center point of the 3D ellipsoid data V3, and then calculate the degree of influence of pixel P on the pixel value based on the transparency of the 3D ellipsoid data V1.

[0075] More specifically, the lower the opacity of the center point of the 3D ellipsoid data V2, the higher the transparency of the 3D ellipsoid data V1 may be. Similarly, the lower the opacity of the center point of the 3D ellipsoid data V3, the higher the transparency of the 3D ellipsoid data V1 may be. Furthermore, the quantization unit 224 may calculate the influence degree e such that the higher the transparency of the 3D ellipsoid data V1, the greater the influence degree e of the 3D ellipsoid data V1 on the pixel value of pixel P.

[0076] Note that 3D ellipsoid data V2 and 3D ellipsoid data V3 may each correspond to second three-dimensional object representation data. The quantization unit 224 outputs the quantized two-dimensional image as quantized data to the encoding unit 225.

[0077] (encoding unit 225) The encoding unit 225 performs encoding on the quantized data. The encoding unit 225 outputs the encoded quantized data to the transmission control unit 226 as quantized encoded data.

[0078] (Transmission control unit 226) The transmission control unit 226 controls the communication unit 240 so that the quantized encoded data is transmitted to the terminal device 10 via the network 30. The quantized encoded data is received by the terminal device 10, decoded by the terminal device 10, and the decoded two-dimensional image is displayed.

[0079] The detailed functions of the image processing apparatus 20 according to the embodiment of the present invention have been described above.

[0080] (1-5. Operation of the image processing device 20) Next, with reference to Figures 7 to 11 (and Figures 1 to 6 as appropriate), an example of the operation of the image processing apparatus 20 according to an embodiment of the present invention will be described.

[0081] Figure 7 is a flowchart illustrating an overall operation example of the image processing apparatus 20 according to an embodiment of the present invention. As shown in Figure 7, the data acquisition unit 221 acquires a plurality of 3D ellipsoid data from the storage unit 230 (S11). The data acquisition unit 221 also acquires the opacity of the center point of the 3D ellipsoid and the color information of the 3D ellipsoid, which are associated with each of the plurality of 3D ellipsoid data, from the storage unit 230.

[0082] The viewpoint determination unit 222 determines viewpoint E (S12). For example, the viewpoint determination unit 222 may determine viewpoint E according to predetermined rules. As one example, the viewpoint determination unit 222 may determine the viewpoint so that it rotates around the 3D object while facing the 3D object. As another example, the viewpoint determination unit 222 may determine the viewpoint so that it moves back and forth near the 3D object while facing the 3D object. Alternatively, the viewpoint determination unit 222 may determine the viewpoint input to the terminal device 10 by the user as viewpoint E.

[0083] Next, image processing (S13) is performed. Figure 8 is a flowchart showing a detailed example of the first half of the image processing (S13) operation. Figure 9 is a flowchart showing a detailed example of the second half of the image processing (S13) operation.

[0084] As shown in Figure 8, the image generation unit 223 generates a two-dimensional image based on the superposition of multiple 3D ellipsoid data (S21). At this time, as described above, the color information of the 3D ellipsoid associated with each of the multiple 3D ellipsoid data may be used to sum the two-dimensional projection data corresponding to each of the multiple 3D ellipsoid data. Alternatively, the opacity of the center point of the 3D ellipsoid associated with each of the multiple 3D ellipsoid data may be used.

[0085] Next, the quantization unit 224 initializes the minimum ellipsoid size t_min for each macroblock to positive infinity (S22). Then, the quantization unit 224 repeats the process for each 3D ellipsoid (S23-S31). First, the quantization unit 224 calculates the 2D covariance matrix Σ' by projecting the 3D covariance matrix of the 3D ellipsoid data onto a 2D screen G corresponding to the viewpoint. Then, the quantization unit 224 calculates the eigenvalues ​​λ1 and λ2 (λ1≧λ2) of the 2D covariance matrix Σ' (S24).

[0086] If the eigenvalue λ2 is a positive real value (YES in S25), the quantization unit 224 sets the eigenvalue λ2 to the eigenvalue λ (S26), sets the processing flag to ON (S29), and proceeds to S31. On the other hand, if the eigenvalue λ2 is not a positive real value (NO in S25), the quantization unit 224 determines whether the eigenvalue λ1 is a positive real value (S27).

[0087] If the eigenvalue λ1 is a positive real value (YES in S27), the quantization unit 224 sets the eigenvalue λ1 to the eigenvalue λ (S28), sets the processing flag to ON (S29), and proceeds to S31. On the other hand, if the eigenvalue λ1 is not a positive real value (NO in S27), the quantization unit 224 sets the processing flag to OFF (S30), and proceeds to S31.

[0088] After the quantization unit 224 has finished executing the iterations (S23-S31) for each 3D ellipsoid, it executes the iterations (S41-S47) for each macroblock, as shown in Figure 9. Furthermore, within the iterations (S41-S47), the quantization unit 224 executes the iterations (S42-S46) for each pixel. In addition, within the iterations (S42-S46), the quantization unit 224 executes the iterations (S43-S45) for each 3D ellipsoid.

[0089] Furthermore, during the iterations (S43-S45), the quantization unit 224 performs a minimum ellipsoid size update process (S44).

[0090] Figure 10 is a flowchart showing a detailed example of the operation of the minimum ellipsoid size update process (S44). As shown in Figure 10, the quantization unit 224 terminates the minimum ellipsoid size update process (S44) if the processing flag is OFF (NO in S51).

[0091] On the other hand, if the processing flag is ON (YES in S51), the quantization unit 224 calculates the degree of influence e on the pixel value of the currently selected pixel (hereinafter also called the "current pixel") in S42 to S46 of the two-dimensional projection data corresponding to the 3D ellipsoid data representing the currently selected 3D ellipsoid (hereinafter also called the "current 3D ellipsoid") in S43 to S45 (S52).

[0092] For example, the quantization unit 224 calculates the distance d when calculating the influence e. Here, the distance d is expressed as shown in equation (1) below, where p is the two-dimensional position of the current pixel, u is the two-dimensional position of the center point of the ellipse obtained by projecting the position of the center point of the current 3D ellipsoid onto the two-dimensional screen G, transpose(x) is the transpose of x, inverse(x) is the inverse matrix of x, and exp(x) is the base of the natural logarithm to the power of x.

[0093] d:=(up)×inverse(Σ')×transpose(up) ...(1)

[0094] Then, if the opacity of the center point of the current 3D ellipsoid is a and the transparency of the current 3D ellipsoid is τ, the quantization unit 224 can calculate the degree of influence e based on the distance d, the opacity a, and the transparency τ, as shown in the following equation (2).

[0095] e:=exp(-0.5×d)×a×τ ...(2)

[0096] Furthermore, the transparency of the current 3D ellipsoid may be calculated such that, among 3D ellipsoids other than the current 3D ellipsoid, the transparency of the current 3D ellipsoid increases as the opacity of the center point of a 3D ellipsoid that is closer to the viewpoint E than the current 3D ellipsoid increases.

[0097] The quantization unit 224 terminates the minimum ellipsoid size update process if the influence e is less than the threshold E_thr (NO in S53). On the other hand, if the influence e is greater than or equal to the threshold E_thr (YES in S53), the quantization unit 224 determines whether the eigenvalue λ is less than the minimum ellipsoid size t_min corresponding to the macroblock currently selected in S41 to S47 (hereinafter also referred to as the "current macroblock") (S54).

[0098] The quantization unit 224 terminates the minimum ellipsoid size update process if the eigenvalue λ is greater than or equal to the minimum ellipsoid size t_min corresponding to the current macroblock (NO in S54). On the other hand, if the eigenvalue λ is less than the minimum ellipsoid size t_min corresponding to the current macroblock (YES in S54), the quantization unit 224 updates the minimum ellipsoid size corresponding to the current macroblock with the eigenvalue λ (S55) and terminates the minimum ellipsoid size update process.

[0099] Next, the quantization process (S14) is executed. Figure 11 is a flowchart showing a detailed example of the operation of the quantization process (S14).

[0100] As shown in Figure 11, the quantization unit 224 repeats (S61-S66) for each macroblock. During the repetition (S61-S66), if the minimum ellipsoid size t_min is smaller than the threshold T_thr (YES in S62), the quantization unit 224 determines a smaller quantization size (S63) and proceeds to S65. On the other hand, if the minimum ellipsoid size t_min is greater than or equal to the threshold T_thr (NO in S62), the quantization unit 224 determines a quantization size larger than the quantization size determined in S63 (S64) and proceeds to S65.

[0101] The quantization unit 224 generates a quantized two-dimensional image by quantizing the two-dimensional image according to the determined quantization size (S65). The quantization unit 224 outputs the quantized two-dimensional image as quantization data to the encoding unit 225. After the repetition (S61~S66) is complete, the quantization process is finished.

[0102] Returning to Figure 7, the explanation continues. The encoding unit 225 performs encoding processing on the quantized data (S15). Then, the encoding unit 225 outputs the encoded quantized data as quantized encoded data to the transmission control unit 226.

[0103] The transmission control unit 226 controls the communication unit 240 so that the quantized encoded data is transmitted to the terminal device 10 via the network 30 (S16). The quantized encoded data is received by the terminal device 10, decoded by the terminal device 10, and the decoded two-dimensional image is displayed. If the control unit 220 wishes to continue operation (NO in S17), it returns to S12. On the other hand, if the control unit 220 wishes to terminate operation (YES in S17), it terminates operation.

[0104] The above describes an example of the operation of the image processing system 1 according to an embodiment of the present invention.

[0105] (1-3. Effects) As described above, an image processing device 20 is provided, according to an embodiment of the present invention, comprising: an image generation unit 223 that generates a two-dimensional image based on the superposition of a plurality of three-dimensional object representation data; and a quantization unit 224 that determines a quantization size for each block based on the plurality of three-dimensional object representation data and quantizes the two-dimensional image based on the quantization size.

[0106] With this configuration, the degradation of the quality of the two-dimensional image generated from the 3D object representation data is suppressed, and the data size of the two-dimensional image can be reduced.

[0107] The effects of the image processing system 1 according to the embodiment of the present invention have been described above.

[0108] (2. Hardware Configuration Example) Next, an example of the hardware configuration of the image processing apparatus 20 according to an embodiment of the present invention will be described.

[0109] In the following, an example of the hardware configuration of an information processing device 900 will be described as an example of the hardware configuration of an image processing device 20 according to an embodiment of the present invention. Note that the example of the hardware configuration of the information processing device 900 described below is merely one example of the hardware configuration of the image processing device 20. Therefore, the hardware configuration of the image processing device 20 may be modified by removing unnecessary components from the hardware configuration of the information processing device 900 described below, or by adding new components. The hardware configuration of the terminal device 10 can also be implemented in a similar manner to the hardware configuration of the image processing device 20.

[0110] Figure 12 shows the hardware configuration of an information processing device 900 as an example of an image processing device 20 according to an embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

[0111] The CPU 901 functions as both an arithmetic processing unit and a control unit, controlling the overall operation of the information processing unit 900 according to various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs and arithmetic parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as needed during its execution. These are interconnected by a host bus 904, which consists of a CPU bus and other components.

[0112] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. It is not always necessary to configure the host bus 904, bridge 905, and external bus 906 separately; these functions may be implemented on a single bus.

[0113] The input device 908 consists of input means for the user to input information, such as a mouse, keyboard, touch panel, buttons, microphone, switches, and levers, and an input control circuit that generates input signals based on the user's input and outputs them to the CPU 901. The user operating the information processing device 900 can input various types of data to the information processing device 900 or instruct it to perform processing operations by operating this input device 908.

[0114] The output device 909 includes, for example, display devices such as CRT (Cathode Ray Tube) display devices, liquid crystal display (LCD) devices, OLED (Organic Light Emitting Diode) devices, lamps, and audio output devices such as speakers.

[0115] The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. The storage device 910 is composed of, for example, an HDD (Hard Disk Drive). This storage device 910 drives the hard disk and stores programs executed by the CPU 901 and various data.

[0116] The communication device 911 is a communication interface composed of, for example, a communication device for connecting to a network. The communication device 911 may support either wireless or wired communication.

[0117] The hardware configuration example of the image processing apparatus 20 according to the present invention has been described above.

[0118] (3. Summary) Although preferred embodiments of the present invention have been described in detail above with reference to the attached drawings, the present invention is not limited to these examples. It is clear to any person with ordinary skill in the art to which the present invention belongs that various modifications or alterations can be conceived within the scope of the technical idea described in the claims, and these are also understood to fall within the technical scope of the present invention.

[0119] For example, in an embodiment of the present invention, an example was described in which the quantization size of a two-dimensional image is determined for each macroblock based on the minimum ellipsoid size, and the macroblocks are quantized according to the determined quantization size. Various methods are already known for changing the quantization size of a two-dimensional image for each macroblock, and various known methods can be applied to embodiments of the present invention.

[0120] Furthermore, the above primarily assumed that the specific values ​​of the thresholds E_thr and T_thr were predetermined as fixed values. However, one or both of the thresholds E_thr and T_thr may be dynamically changed according to predetermined data. For example, one or both of the thresholds E_thr and T_thr may be dynamically changed according to the field of view A (Figure 4). As an example, one or both of the thresholds E_thr and T_thr may be changed more as the field of view A increases. [Explanation of symbols]

[0121] 1. Image Processing System 10 Terminal devices 110 Operation section 120 Control Unit 130 Storage section 140 Communications Department 150 Display section 20 Image Processing Devices 220 Control Unit 221 Data Acquisition Unit 222 Viewpoint Determination Unit 223 Image generation unit 224 Quantization section 225 Encoding section 226 Transmission Control Unit 230 Storage section 240 Communications Department 30 Networks

Claims

1. An image generation unit that generates a two-dimensional image based on the superimposition of multiple three-dimensional object representation data, A quantization unit determines a quantization size for each block based on the plurality of three-dimensional object representation data, and quantizes the two-dimensional image based on the quantization size, An image processing device equipped with the following features.

2. The quantization unit determines the quantization size for each block based on the two-dimensional projection data corresponding to each of the multiple three-dimensional object representation data, which is calculated by projecting the multiple three-dimensional object representation data onto a two-dimensional screen corresponding to the viewpoint. The image processing apparatus according to claim 1.

3. The quantization unit determines the minimum size of the two-dimensional shape shown by the two-dimensional projection data for each block, and determines the quantization size for each block according to the minimum size. The image processing apparatus according to claim 2.

4. The quantization unit, when the minimum size is smaller than the first threshold, reduces the quantization size to a smaller size than when the minimum size is equal to or greater than the first threshold. The image processing apparatus according to claim 3.

5. The quantization unit identifies, for each pixel constituting the two-dimensional screen, three-dimensional object representation data whose degree of influence on the pixel value is greater than a second threshold as high-influence data, and determines the minimum size of the two-dimensional shape shown by the two-dimensional projection data corresponding to the high-influence data for each block. The image processing apparatus according to claim 3.

6. The plurality of three-dimensional object representation data include the first three-dimensional object representation data, The quantization unit calculates the degree of influence of the first three-dimensional object representation data on the pixel value based on the distance between the position of the center point of the first three-dimensional object representation data and the position of the pixel. The image processing apparatus according to claim 5.

7. The quantization unit calculates the influence such that the influence decreases as the distance increases. The image processing apparatus according to claim 6.

8. The plurality of three-dimensional object representation data include the first three-dimensional object representation data, The quantization unit calculates the degree of influence of the first three-dimensional object representation data on the pixel values ​​based on the opacity of the center point of the first three-dimensional object representation data. The image processing apparatus according to claim 5.

9. The quantization unit calculates the influence such that the influence increases as the opacity increases. The image processing apparatus according to claim 8.

10. The aforementioned plurality of three-dimensional object representation data includes a first three-dimensional object representation data and a second three-dimensional object representation data. The quantization unit calculates the transparency of the first three-dimensional object representation data based on the opacity of the center point of the second three-dimensional object representation data when the position of the center point of the first three-dimensional object representation data is farther away from the viewpoint than the position of the center point of the second three-dimensional object representation data, and calculates the degree of influence of the first three-dimensional object representation data on the pixel value based on the transparency of the first three-dimensional object representation data. The image processing apparatus according to claim 5.

11. The quantization unit calculates the influence such that the influence increases as the transmittance increases. The image processing apparatus according to claim 10.

12. The aforementioned two-dimensional shape is an ellipse, The size of the two-dimensional shape is the radius of the minor axis of the ellipse. The image processing apparatus according to claim 2.

13. This involves generating a two-dimensional image based on the superposition of multiple three-dimensional object representation data, The quantization size is determined for each block based on the plurality of three-dimensional object representation data, and the two-dimensional image is quantized based on the quantization size. A computer-based image processing method, including [a specific method].

14. Computers, An image generation unit that generates a two-dimensional image based on the superimposition of multiple three-dimensional object representation data, A quantization unit determines a quantization size for each block based on the plurality of three-dimensional object representation data, and quantizes the two-dimensional image based on the quantization size, A program that makes it function as such.

Images