Image display device and image providing system
The image display device addresses the challenge of seamless switching between different display ranges in three-dimensional virtual spaces by combining and aligning three-dimensional rendered images using parallel projection, reducing processing load and enabling efficient perspective projection rendering.
Patent Information
- Application Number
- JP2023101734
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-06-21
- Publication Date
- 2025-01-09
- Estimated Expiration
- 2043-06-21
AI Technical Summary
Existing methods for providing three-dimensional virtual spaces to multiple users face challenges in seamlessly switching between different display ranges with high processing loads due to the use of fixed-point cameras, which result in varying display ranges and increased processing demands.
An image display device that combines multiple three-dimensional rendered images using parallel projection, aligns adjacent or overlapping ranges, and generates perspective projection images based on depth and color information, reducing processing load by aligning images with depth and color values.
Enables seamless switching between different display ranges in three-dimensional spaces with reduced processing load by aligning and combining three-dimensional rendered images, allowing for efficient perspective projection rendering.
Smart Images

Figure 2025001934000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an image display device and an image providing system. [Background technology]
[0002] In providing images expressing a three-dimensional virtual space to multiple users, conventionally, a three-dimensionally expressed image (e.g., a perspective projection image) with a viewpoint of a virtual camera for each user arranged in the three-dimensional space based on three-dimensional space data expressing the three-dimensional space is provided to each user's terminal. In addition, in consideration of the increase in processing load and network traffic related to the generation and transmission of images for each user when providing images to many users, instead of setting a virtual camera for each user in the three-dimensional space, a virtual fixed camera is provided at a predetermined position, and a three-dimensional image with a viewpoint of each fixed camera is provided to each user while switching, for example, according to the movement of the user in the three-dimensional space. In addition, Patent Document 1 discloses a technology for generating a virtual viewpoint image from a two-dimensional image signal and a depth image signal. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2013-225740 A Summary of the Invention [Problem to be solved by the invention]
[0004] In order to provide an image showing a three-dimensional space having a certain size, it is necessary to provide multiple fixed cameras in the three-dimensional space. It is also desirable to seamlessly switch between images with different display ranges according to, for example, the movement of a target object and the movement of a viewpoint position in the three-dimensional space. However, since the images based on each fixed camera are three-dimensionally represented, for example, perspective projection images, it is difficult to seamlessly switch between images with different display ranges due to differences in the distance and relative position between each fixed camera and the object represented in the image. Even if seamless switching is possible, the processing load is large.
[0005] Therefore, the present invention has been made in consideration of the above problems, and has an object to easily provide an image representing a three-dimensional space that allows seamless switching between different display ranges with a small processing load. [Means for solving the problem]
[0006] In order to solve the above problem, an image display device according to one aspect of the present disclosure is an image display device that displays a three-dimensional image provided from an image providing device, and includes a combination unit that combines a plurality of three-dimensional rendering images to generate a combined image, where the plurality of three-dimensional rendering images are obtained by parallel projecting the three-dimensional space in the same projection direction from each of a plurality of viewpoints based on three-dimensional spatial data representing a given three-dimensional space, the ranges of the represented three-dimensional space being adjacent or overlapping, and including depth information associated with each pixel, and a perspective projection image generation unit that generates a perspective projection image by perspective projection rendering from a given viewpoint position based on the combined image and the depth and color values associated with each pixel in the combined image.
[0007] According to the above aspect, since the 3D rendering image is an image obtained by parallel projection of the 3D space, multiple 3D rendering images whose represented ranges are adjacent or overlapping can be simply combined by simply aligning them. Therefore, the processing load for combining the images is small. Furthermore, since each pixel in the combined image is associated with a depth and a color value based on the depth information of each 3D rendering image, a perspective projection image in which the 3D space is represented can be generated by perspective projection rendering based on a given viewpoint position. Effect of the Invention
[0008] It is possible to easily provide images representing three-dimensional space that can seamlessly switch between different display ranges with little processing load. [Brief description of the drawings]
[0009] [Figure 1] 1 is a block diagram showing a device configuration of an image providing system according to an embodiment of the present invention, and functional configurations of an image display device and an image providing device. [Diagram 2] FIG. 2 is a diagram illustrating a three-dimensional space model constructed based on three-dimensional space data representing a three-dimensional space. [Diagram 3] FIG. 1 is a diagram showing an example of a plurality of three-dimensional rendering images acquired by parallel projection of a three-dimensional space. [Figure 4] FIG. 11 is a diagram showing a process in which a combined image and a perspective projection image are generated in an image display device based on a plurality of three-dimensional rendering images generated in an image providing device. [Diagram 5] FIG. 13 is a diagram showing an example of a combined image. [Figure 6] 11A and 11B are diagrams illustrating the occurrence of occlusion regions in a perspective projection image generated based on a combined image that is a parallel projection image. [Figure 7] 11A and 11B are diagrams illustrating an example of an interpolation process for an occlusion region. [Figure 8] FIG. 13 is a diagram showing an example of a perspective projection image in which an occlusion region has been interpolated. [Figure 9]1A to 1C are diagrams illustrating examples of perspective projection images generated based on different viewpoint positions. [Figure 10] 4 is a flowchart showing the processing content of an image display method in the image providing system. [Figure 11] FIG. 4 is a diagram showing the configuration of an image display program. [Figure 12] FIG. 2 is a hardware block diagram of the image display device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] An embodiment of an image providing system and an image display device according to the present disclosure will be described with reference to the drawings. Whenever possible, the same parts are denoted by the same reference numerals, and duplicated descriptions will be omitted.
[0011] Fig. 1 is a block diagram showing the device configuration of the image providing system of this embodiment and the functional configuration of the image display device and the image providing device. The image providing system 1 is a system that provides a three-dimensional image to an image display device 10, and includes the image display device 10 and the image providing device 20 as shown in Fig. 1. The image display device 10 is configured to be able to communicate with the image providing device 20 via a communication network N.
[0012] The image display device 10 of this embodiment is a device that displays a three-dimensional image provided from an image providing device 20, and may be, for example, a terminal configured with a computer. As shown in Fig. 1, the image display device 10 functionally includes an acquisition unit 11, a combination unit 12, a perspective projection image generating unit 13, and an output unit 14. These functional units 11 to 14 may be configured in one device as exemplified in Fig. 1, or may be configured separately in a plurality of devices.
[0013] The image providing device 20 of this embodiment is a device that provides a three-dimensional image to the image display device 10, and may be configured, for example, by a server computer. As shown in Fig. 1, the image providing device 20 functionally includes a parallel projection image generating unit 21 and a transmitting unit 22. These functional units 21 to 22 may be configured in one device as exemplified in Fig. 1, or may be configured distributed across multiple devices.
[0014] Each of the functional units 21 to 22 of the image providing device 20 is configured to be able to access a storage means (storage) such as a three-dimensional space model storage unit 25. The three-dimensional space model storage unit 25 may be provided in the image providing device 20 as exemplified in Fig. 1, or may be configured in another device configured to be able to be accessed from the image providing device 20.
[0015] Prior to describing the image display device 10, each functional unit of the image providing device 20 will be described. The parallel projection image generating unit 21 generates a plurality of 3D rendering images by parallel projecting the 3D space from a plurality of viewpoints in the same projection direction based on 3D space data representing a given 3D space. The parallel projection image generating unit 21 adds depth information consisting of depths associated with each pixel to the 3D rendering image.
[0016] The three-dimensional space model storage unit 25 stores given three-dimensional space data representing a three-dimensional virtual space. The parallel projection image generating unit 21 may generate a three-dimensional space model representing the three-dimensional virtual space by rendering based on the three-dimensional space data.
[0017] Fig. 2 is a diagram showing a three-dimensional space model constructed based on three-dimensional space data representing a three-dimensional space. As shown in Fig. 2, the three-dimensional space model A shows a three-dimensional space in which three objects ob1, ob2, and ob3 are arranged in front of a background bg. That is, the parallel projection image generating unit 21 may generate the three-dimensional space model A shown in Fig. 2 based on the three-dimensional space data showing the shapes and positions of the objects ob1, ob2, and ob3 and the background bg.
[0018] The parallel projection image generating unit 21 generates a plurality of 3D rendering images by parallel projecting the 3D space represented by the 3D space model A from a plurality of viewpoints in the same projection direction. In the example shown in Fig. 2, the parallel projection image generating unit 21 generates 3D rendering images by parallel projecting the virtual cameras C1 and C2 as viewpoints in the same projection directions CR1 and CR2, respectively.
[0019] 3 is a diagram showing an example of a plurality of 3D rendering images acquired by parallel projection in a 3D space. The parallel projection image generating unit 21 generates a 3D rendering image p11 by parallel projection in a projection direction CR1 with a virtual camera C1 as a viewpoint position. The parallel projection image generating unit 21 also generates a 3D rendering image p12 by parallel projection in a projection direction CR2 with a virtual camera C2 as a viewpoint position.
[0020] Since the three-dimensional rendering images p11 and p12 are parallel projection images, the distance from the viewpoint to the object depicted in the image is not expressed. For example, as shown in Fig. 2, the distance from the virtual camera C1 to the object ob2 is greater than the distance from the virtual camera C1 to the object ob1, but in the three-dimensional rendering image p11, the object ob2 is not depicted relatively smaller than the object ob1, but is depicted at the same size. Also, the distance from the virtual camera C2 to the object ob2 is greater than the distance from the virtual camera C2 to the object ob3, but in the three-dimensional rendering image p12, the object ob2 is not depicted relatively smaller than the object ob3, but is depicted at the same size.
[0021] The parallel projection image generating unit 21 generates a 3D rendering image including depth information indicating the depth of each pixel. Specifically, the parallel projection image generating unit 21 may generate a 3D rendering image including a color image and depth information.
[0022] Fig. 4 is a diagram showing a process in which a combined image and a perspective projection image are generated in the image display device 10 based on a plurality of 3D rendering images generated in the image providing device 20. As shown in Fig. 4, the parallel projection image generating unit 21 of the image providing device 20 generates a first 3D rendering image p11 and a second 3D rendering image p12. The first 3D rendering image p11 includes a color image pc1 and depth information pz1 generated by parallel projection. The second 3D rendering image p12 includes a color image pc2 and depth information pz2 generated by parallel projection.
[0023] The color images pc1, pc2 are images in which a color value is associated with each pixel. The depth information pz1, pz2 is information indicating the depth of each pixel. The greater the distance to the object represented by the pixel, the greater the depth value. The form of the depth information pz1, pz2 generated by the parallel projection image generating unit 21 as components of the three-dimensional rendering images p11, p12 is not limited, and may be, for example, a Z-buffer, a depth image, a depth map, or the like. Note that the level of detail of a pixel to which a color value is associated in the color images pc1, pc2 may differ from the level of detail of a pixel to which a depth is associated in the depth information pz1, pz2.
[0024] Referring back to FIG. 1, the transmission unit 22 transmits a plurality of three-dimensional rendering images to the image display device 10.
[0025] Next, a description will be given of each functional unit of the image display device 10. The acquisition unit 11 acquires a plurality of 3D rendering images transmitted from the image providing device 20. Specifically, referring again to Fig. 4, the acquisition unit 11 acquires a first 3D rendering image p11 and a second 3D rendering image p12.
[0026] The combining unit 12 combines a plurality of 3D rendering images to generate a combined image. Specifically, the combining unit 12 simply combines a plurality of 3D rendering images that represent adjacent or overlapping ranges of 3D space by aligning the adjacent or overlapping portions.
[0027] Fig. 5 is a diagram showing an example of a combined image. As illustrated in Fig. 5, the combining unit 12 combines a three-dimensional rendering image p11 and a three-dimensional rendering image p12 to generate a combined image p2. In this way, since a three-dimensional rendering image is an image obtained by parallel projection of a three-dimensional space, multiple three-dimensional rendering images whose represented ranges are adjacent or overlapping can be simply combined by simply aligning them. Therefore, the processing load required for combining images is small.
[0028] Furthermore, the combining unit 12 may generate mesh data based on the three-dimensional rendering image prior to generating the combined image p2. Specifically, the combining unit 12 generates mesh data corresponding to the three-dimensional rendering image based on depth information. The mesh data is made up of a set of vertices, edges, and faces for expressing the shape of an object or the like shown in the three-dimensional rendering image as a polyhedron, and is generated, for example, by a known computer graphics technique based on the depth information.
[0029] Furthermore, the combining unit 12 sets color values associated with pixels corresponding to the vertices of each mesh in the color image to the vertices of the mesh. Specifically, the combining unit 12 may attach the color values associated with pixels corresponding to the vertices of each mesh in the color image to each mesh as a texture. Then, the combining unit 12 may generate the combined image by interpolating the colors of the portions between the vertices based on the color values set to each vertex of the mesh.
[0030] 4 again, as an example of this embodiment, the combining unit 12 generates mesh data md1 corresponding to the first three-dimensional rendering image p11 based on the depth information pz1. Furthermore, the combining unit 12 sets, to the vertices of each mesh in the mesh data md1, a color value associated with a pixel corresponding to the vertex of the mesh in the color image pc1.
[0031] The combining unit 12 also generates mesh data md2 corresponding to the second three-dimensional rendering image p12 based on the depth information pz2. Furthermore, the combining unit 12 sets, to the vertices of each mesh in the mesh data md2, a color value associated with the pixel corresponding to the vertex of the mesh in the color image pc2.
[0032] The combining unit 12 may generate a combined image p2 by combining the mesh data md1, md2 in which the color values of corresponding pixels are set to each vertex of the mesh. The combining unit 12 may also generate the combined image p2 by combining the mesh data md1, md2 generated based on the depth information pz1, pz2 of the three-dimensional rendering images p11, p12, respectively, and setting color values associated with the corresponding pixels in the color images pc1, pc2 to the vertices of each mesh in the combined mesh data.
[0033] Referring again to FIG. 1, the perspective projection image generator 13 generates a perspective projection image by perspective rendering from a given viewpoint position based on the combined image and the depth and color values associated with each pixel in the combined image.
[0034] Specifically, as explained with reference to FIG. 4, since the combined image p2 is composed of mesh data reflecting depth information, the perspective projection image generating unit 13 can generate a perspective projection image p3 from a given viewpoint position based on the depth and color values of each mesh, for example by applying existing rendering technology.
[0035] Next, a description will be given of the interpolation process of the perspective projection image by the perspective projection image generating unit 13. Fig. 6 is a diagram showing an example of the occurrence of an occlusion region in a perspective projection image generated based on a combined image which is a parallel projection image.
[0036] The combined image p2 shown in Fig. 5 is an image generated by parallel projection, and therefore does not have color value information of the areas occluded by the objects ob1, ob2, and ob3 in the background bg. In such a case, a perspective projection image p31 as shown in Fig. 6 may be generated by perspective projection rendering based on the combined image p2. In the perspective projection image p31, occlusion areas (blocked areas) oc1, oc2, and oc3 may occur in parts that are not shown in the parallel projection image but are shown in the perspective projection image.
[0037] Occlusion areas occur in perspective projection rendering based on combined image p2, which is a parallel projection image having color values and depths for each pixel, due to a misalignment between the projection direction in parallel projection and the projection direction in perspective projection rendering, and occur particularly noticeably in parts of the horizontal pixel arrangement where there is a large difference in depth between adjacent pixels.
[0038] In areas where there is a large difference in depth between adjacent pixels, when the color between the vertices is interpolated based on the color values set at the vertices of the mesh, the color values of each pixel may differ significantly because the pixels corresponding to each vertex represent different things, and appropriate color interpolation may not be possible.
[0039] The perspective projection image generating unit 13 may perform an interpolation process to interpolate color values of the occlusion region. The perspective projection image generating unit 13 performs the interpolation process of the occlusion region when the depth of a second mesh vertex adjacent to the first mesh vertex in the arrangement direction is smaller than the depth of a first mesh vertex in the horizontal arrangement of meshes and mesh vertices in the mesh data corresponding to the combined image by a difference exceeding a given first threshold, and the difference between the depth of a fourth mesh vertex located in the arrangement direction of the second mesh vertex and the depth of the first mesh vertex is equal to or smaller than a given second threshold or the depth of the fourth mesh vertex is equal to or larger than the depth of the first mesh vertex.
[0040] 7 is a diagram showing an example of the interpolation process of the occlusion region. Specifically, the perspective projection image generating unit 13 determines to perform the interpolation process of the occlusion region when the depth dp2 of the mesh vertex ms2 adjacent to the mesh vertex ms1 in the array direction dr is smaller than the depth dp1 of the mesh vertex ms1 in the horizontal array direction dr in the mesh data corresponding to the combined image p2 by a difference exceeding a given first first threshold, and the difference between the depth dp4 of the mesh vertex ms4 located in the array direction dr of the mesh vertex ms2 and the depth dp1 of the mesh vertex ms1 is equal to or smaller than a given second threshold, or the depth dp4 of the mesh vertex ms4 is equal to or larger than the depth dp1 of the mesh vertex ms1.
[0041] That is, when this determination is made, there is a high possibility that the object represented by the pixels corresponding to mesh vertices ms1 and ms4 (e.g., background bg) in the combined image p2 is occluded by the object represented by the pixels corresponding to mesh vertices ms2 and ms3 (e.g., object ob). Note that mesh vertex ms3 is a mesh vertex adjacent to mesh vertex ms4 in the opposite direction of the array direction dr. In this case, combined image p2 does not have a color value of background bg corresponding to the portion from mesh vertex ms2 to mesh vertex ms3.
[0042] Therefore, when this determination is made, in the perspective projection rendering based on the combined image p2, an occlusion area occurs in the area corresponding to the portion between mesh vertex ms1 and mesh vertex ms2, or in the area corresponding to the portion between mesh vertex ms4 and mesh vertex ms3.
[0043] When it is determined that an interpolation process for the occlusion region is to be performed, the perspective projection image generating unit 13 interpolates the color value of the portion corresponding to between the first mesh vertex and the second mesh vertex in the perspective projection image, or the color value of the portion corresponding to between the third mesh vertex and the fourth mesh vertex adjacent to the fourth mesh vertex in the opposite direction of the arrangement direction, based on the color value of the first mesh vertex and the color value of the fourth mesh vertex.
[0044] Specifically, depending on the deviation between the projection direction of the parallel projection of the combined image p2 and the projection direction of the perspective projection image p3, an occlusion region occurs in a portion in the perspective projection image p3 that corresponds to between mesh vertex ms1 and mesh vertex ms2, or between mesh vertex ms4 and mesh vertex ms3 adjacent to mesh vertex ms4 in the opposite direction of the arrangement direction dr. In such a case, the perspective projection image generating unit 13 interpolates the color value of the portion in the perspective projection image p3 that corresponds to between mesh vertices ms1 and mesh vertex ms2, or the color value of the portion that corresponds to between mesh vertex ms4 and mesh vertex ms3, based on the color value of mesh vertex ms1 and the color value of mesh vertex ms4.
[0045] In the interpolation process, the perspective projection image generating unit 13 may linearly interpolate the color value of the portion corresponding to between the first mesh vertex and the second mesh vertex in the perspective projection image, or the color value of the portion corresponding to between the third mesh vertex and the fourth mesh vertex, based on the respective positions and color values of the first mesh vertex and the fourth mesh vertex.
[0046] Specifically, assuming that the pixel values of the pixels corresponding to the mesh vertices ms1 and ms4 are (r1, g1, b1) and (r2, g2, b2), respectively, the perspective projection image generating unit 13 linearly interpolates the color values from the mesh vertices ms2 to ms3 as the color value of the background bg based on the color value (r1, g1, b1) of the mesh vertex ms1 and the color value (r2, g2, b2) of the mesh vertex ms4. Then, the perspective projection image generating unit 13 sets the linearly interpolated color value in the occlusion region occurring between the mesh vertices ms1 and ms2, or in the occlusion region occurring between the mesh vertices ms4 and ms3.
[0047] Fig. 8 is a diagram showing an example of a perspective projection image in which an occlusion region is interpolated. The perspective projection image generating unit 13 can generate a perspective projection image p32 by interpolating color values of the occlusion regions oc1, oc2, and oc3 generated in the perspective projection image p31 shown in Fig. 6 by the interpolation process described with reference to Fig. 7. In the perspective projection image p32, the color values of the occlusion regions oc1, oc2, and oc3 are appropriately interpolated by the color value of the background bg.
[0048] 9 is a diagram showing an example of a perspective projection image generated based on different viewpoint positions. The perspective projection image generating unit 13 can generate a perspective projection image by perspective projection rendering from an arbitrary viewpoint position based on the combined image p2.
[0049] For example, the perspective projection image generating unit 13 can generate a perspective projection image p41 showing objects ob1 and ob2 by perspective projection rendering based on a first viewpoint position using the combined image p2. Furthermore, the perspective projection image generating unit 13 can generate a perspective projection image p42 showing objects ob2 and ob3 by perspective projection rendering based on a second viewpoint position different from the first viewpoint position.
[0050] In this way, a perspective projection image based on any viewpoint position can be generated using combined image p2, so that the perspective projection image generation unit 13 can continuously generate perspective projection images in which the viewpoint position is switched so that the character is positioned at the center of the screen in accordance with the movement of the character in the virtual space, for example.
[0051] 1 again, the output unit 14 outputs the perspective projection image generated by the perspective projection image generating unit 13. Specifically, the output unit 14 may cause the perspective projection image to be displayed on a display included in the image display device 10.
[0052] FIG. 10 is a flowchart showing the process of the image display method in the image providing system and the image display device 10.
[0053] In step S1, the parallel projection image generating unit 21 generates a plurality of 3D rendering images by parallel projecting the 3D space from a plurality of viewpoints in the same projection direction based on the 3D space data representing a given 3D space. The 3D rendering image includes depth information consisting of depths associated with each pixel. The 3D rendering image also includes a color image in which a color value is associated with each pixel.
[0054] In step S2, the transmission unit 22 transmits a plurality of three-dimensional rendering images to the image display device .
[0055] In step S3, the acquisition unit 11 acquires a plurality of three-dimensional rendering images transmitted from the image providing device 20.
[0056] In step S4, the combining unit 12 generates mesh data corresponding to the three-dimensional rendering images based on the depth information included in each three-dimensional rendering image.
[0057] In step S5, the combining unit 12 applies color values associated with pixels corresponding to the vertices of each mesh in the color image to the vertices of each mesh as texture.
[0058] In step S6, the combining unit 12 combines the mesh data corresponding to each of the three-dimensional rendering images to generate a combined image.
[0059] In step S7, the perspective projection image generator 13 generates a perspective projection image by perspective rendering from a given viewpoint position based on the combined image and the depth and color values associated with each pixel (mesh) in the combined image.
[0060] In step S8, the perspective projection image generated by the perspective projection image generating unit 13 is output. Specifically, the output unit 14 may cause the perspective projection image to be displayed on a display included in the image display device 10.
[0061] Next, an image display program for causing a computer to function as the image display device 10 of this embodiment will be described with reference to Fig. 11. Fig. 9 is a diagram showing the configuration of the image display program. The image display program P1 is configured to include a main module m10 that comprehensively controls the image display processing in the image display device 10, an acquisition module m11, a combination module m12, a perspective projection image generation module m13, and an output module m14. Each of the modules m11 to m14 realizes each function for each of the functional units 11 to 14.
[0062] The image display program P1 may be transmitted via a transmission medium such as a communication line, or may be stored in a recording medium M1 as shown in FIG.
[0063] According to the image providing system 1, image display device 10, image display method, and image display program P1 of the present embodiment described above, since the 3D rendering image is an image obtained by parallel projection of the 3D space, multiple 3D rendering images whose represented ranges are adjacent or overlapping can be simply combined by simply aligning them. Therefore, the processing load for combining images is small. Furthermore, since each pixel in the combined image is associated with a depth and a color value based on the depth information of each 3D rendering image, a perspective projection image in which the 3D space is represented can be generated by perspective projection rendering based on a given viewpoint position.
[0064] The image display device according to the present disclosure may have the following configurations. The actions and effects of each configuration will be described as follows.
[0065] An image display device according to one aspect of the present disclosure is an image display device that displays a three-dimensional image provided from an image providing device, and includes a combination unit that combines a plurality of three-dimensional rendering images to generate a combined image, where the plurality of three-dimensional rendering images are obtained by parallel projecting the three-dimensional space in the same projection direction from each of a plurality of viewpoints based on three-dimensional spatial data representing a given three-dimensional space, the ranges of the represented three-dimensional space being adjacent or overlapping, and including depth information associated with each pixel, and a perspective projection image generation unit that generates a perspective projection image by perspective projection rendering from a given viewpoint position based on the combined image and the depth and color values associated with each pixel in the combined image.
[0066] According to the above aspect, since the 3D rendering image is an image obtained by parallel projection of the 3D space, multiple 3D rendering images whose represented ranges are adjacent or overlapping can be simply combined by simply aligning them. Therefore, the processing load for combining the images is small. Furthermore, since each pixel in the combined image is associated with a depth and a color value based on the depth information of each 3D rendering image, a perspective projection image in which the 3D space is represented can be generated by perspective projection rendering based on a given viewpoint position.
[0067] In addition, in an image display device relating to another aspect, the three-dimensional rendering image includes a color image in which a color value is associated with each pixel and depth information, and the combination unit generates mesh data corresponding to the three-dimensional rendering image based on the depth information, and sets color values associated with pixels corresponding to each mesh vertex in the color image to the mesh vertices.
[0068] According to the above aspect, since the 3D rendering image includes the depth of each pixel, it is possible to generate mesh data that 3D represents what is shown in the 3D rendering image. Since each pixel of the 3D rendering image is associated with a color value, it is possible to appropriately configure what was shown in the 3D rendering image by setting a color value to each vertex of the mesh and interpolating the colors between the vertices with the color values set to the vertices.
[0069] In an image display device according to another aspect, the combining unit may combine mesh data, or mesh data in which color values of corresponding pixels are set at the vertices of each mesh.
[0070] According to the above aspect, since the combined image contains the depth of things depicted in the image as mesh data, a perspective projection image in which three-dimensional space is appropriately represented can be generated by perspective projection rendering based on the combined image.
[0071] In addition, in an image display device according to another aspect, when the depth of a second mesh vertex adjacent to the first mesh vertex in the arrangement direction is smaller than the depth of a first mesh vertex in the horizontal arrangement of meshes and mesh vertices in the mesh data corresponding to the combined image by a difference exceeding a given first threshold, and the difference between the depth of a fourth mesh vertex located in the arrangement direction of the second mesh vertex and the depth of the first mesh vertex is less than a given second threshold or the depth of the fourth mesh vertex is greater than or equal to the depth of the first mesh vertex, the perspective projection image generating unit may interpolate the color value of a portion corresponding to between the first mesh vertex and the second mesh vertex in the perspective projection image, or the color value of a portion corresponding to between a third mesh vertex and a fourth mesh vertex adjacent to the fourth mesh vertex in the opposite direction to the arrangement direction, based on the color value of the first mesh vertex and the color value of the fourth mesh vertex.
[0072] In the above aspect, the portion between the first mesh vertex and the second mesh vertex or the portion between the third mesh vertex and the fourth mesh vertex corresponds to an occlusion region in which a background occluded by an object represented at a position corresponding to the second mesh vertex and / or the third mesh vertex appears in the perspective projection image according to the viewpoint position, and it is highly likely that the portion corresponding to the occlusion region in the perspective projection image should represent the background to which the first mesh vertex and / or the fourth mesh vertex belongs. In view of this possibility, a perspective projection image in which the portion corresponding to the occlusion region is represented with appropriate color values can be generated by interpolating the color value of the portion between the first mesh vertex and the second mesh vertex or the portion between the third mesh vertex and the fourth mesh vertex based on the color values of the first mesh vertex and the fourth mesh vertex, which are color values corresponding to the background.
[0073] In addition, in an image display device relating to another aspect, the perspective projection image generating unit may linearly interpolate the color value of the portion corresponding to between the first mesh vertex and the second mesh vertex in the perspective projection image, or the color value of the portion corresponding to between the third mesh vertex and the fourth mesh vertex, based on the respective positions and color values of the first mesh vertex and the fourth mesh vertex.
[0074] According to the above aspect, color values of a portion of a perspective projection image that corresponds to an occlusion region are easily and appropriately interpolated.
[0075] An image providing system according to one aspect of the present disclosure is an image providing system that includes an image providing device and an image display device and provides a three-dimensional image to the image display device, wherein the image providing device comprises a parallel projection image generation unit that generates three-dimensional rendering images by parallel projecting the three-dimensional space from multiple viewpoints in the same projection direction based on three-dimensional spatial data representing a given three-dimensional space, the three-dimensional rendering images including depth information associated with each pixel, and the image display device comprises a combination unit that combines multiple three-dimensional rendering images representing adjacent or overlapping ranges of the three-dimensional space to generate a combined image, and a perspective projection image generation unit that generates a perspective projection image by perspective projection rendering from a given viewpoint position based on the combined image and the depth and color values associated with each pixel in the combined image.
[0076] According to the above aspect, since the 3D rendering image is an image obtained by parallel projection of the 3D space, multiple 3D rendering images whose represented ranges are adjacent or overlapping can be simply combined by simply aligning them. Therefore, the processing load for combining the images is small. Furthermore, since each pixel in the combined image is associated with a depth and a color value based on the depth information of each 3D rendering image, a perspective projection image in which the 3D space is represented can be generated by perspective projection rendering based on a given viewpoint position.
[0077] In addition, in an image providing system relating to another aspect, the parallel projection image generation unit generates a 3D rendering image including a color image in which a color value is associated with each pixel and depth information, and the combination unit generates mesh data corresponding to the 3D rendering image based on the depth information, and sets the color values associated with corresponding pixels in the color image to the vertices of each mesh.
[0078] According to the above aspect, since the 3D rendering image includes the depth of each pixel, it is possible to generate mesh data that 3D represents what is shown in the 3D rendering image. Since each pixel of the 3D rendering image is associated with a color value, it is possible to appropriately configure what was shown in the 3D rendering image by setting a color value to each vertex of the mesh and interpolating the colors between the vertices with the color values set to the vertices.
[0079] The block diagram shown in FIG. 1 shows functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. The method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by using wires, wirelessly, etc.). The functional block may be realized by combining the one device or the multiple devices with software.
[0080] Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocation, mapping, assignment, etc. For example, a functional block (component) that performs the function of transmission is called a transmitting unit or a transmitter. In either case, as described above, there is no particular limitation on the method of realization.
[0081] For example, the image display device 10 according to an embodiment of the present invention may function as a computer. Fig. 12 is a diagram showing an example of a hardware configuration of the image display device 10 according to this embodiment. The image display device 10 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.
[0082] In the following description, the term "apparatus" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the image display device 10 may be configured to include one or more of the devices shown in the drawings, or may be configured to exclude some of the devices.
[0083] Each function of the image display device 10 is realized by loading a specific software (program) onto hardware such as a processor 1001 and a memory 1002, so that the processor 1001 performs calculations and controls communication via a communication device 1004 and the reading and / or writing of data in the memory 1002 and storage 1003.
[0084] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, each of the functional units 11 to 14 shown in FIG. 1 may be realized by the processor 1001.
[0085] Moreover, the processor 1001 reads out a program (program code), software modules, and data from the storage 1003 and / or the communication device 1004 to the memory 1002, and executes various processes according to the programs. As the programs, programs that cause a computer to execute at least a part of the operations described in the above-mentioned embodiments are used. For example, each of the functional units 11 to 14 of the image display device 10 may be realized by a control program stored in the memory 1002 and operated by the processor 1001. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may be transmitted from a network via an electric communication line.
[0086] The memory 1002 is a computer-readable recording medium, and may be composed of at least one of, for example, a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store a program (program code), software modules, etc. that are executable to implement an image display method according to one embodiment of the present invention.
[0087] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray® disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy disk, a magnetic strip, and the like. Storage 1003 may be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including memory 1002 and / or storage 1003.
[0088] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via a wired and / or wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.
[0089] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).
[0090] Furthermore, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured as a single bus, or may be configured as different buses between the devices.
[0091] Furthermore, the image display device 10 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented by at least one of these pieces of hardware.
[0092] The notification of information is not limited to the aspects / embodiments described in the present disclosure, and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI), Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB), System Information Block (SIB))), other signals, or a combination thereof. In addition, the RRC signaling may be called an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.
[0093] Each aspect / embodiment described in the present disclosure may be applied to at least one of systems using LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (new Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), and other suitable systems, and next-generation systems extended based on these. In addition, a combination of multiple systems (e.g., a combination of at least one of LTE and LTE-A and 5G, etc.) may be applied.
[0094] The order of the steps, sequences, flow charts, etc. of each aspect / embodiment described in this disclosure may be changed unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.
[0095] In the present disclosure, a specific operation performed by a base station may be performed by its upper node in some cases. In a network consisting of one or more network nodes having a base station, it is clear that various operations performed for communication with a terminal may be performed by at least one of the base station and other network nodes other than the base station (e.g., MME or S-GW, etc., but are not limited to these). Although the above example illustrates a case in which there is one other network node other than the base station, it may also be a combination of multiple other network nodes (e.g., MME and S-GW).
[0096] Information, etc. may be output from a higher layer (or a lower layer) to a lower layer (or a higher layer). Information may be input / output via multiple network nodes.
[0097] The input and output information may be stored in a specific location (e.g., memory) or may be managed in a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be transmitted to another device.
[0098] The determination may be based on a value represented by a single bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).
[0099] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).
[0100] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.
[0101] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.
[0102] Additionally, software, instructions, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies, such as coaxial cable, fiber optic cable, twisted pair, and Digital Subscriber Line (DSL), and / or wireless technologies, such as infrared, radio, and microwave, these wired and / or wireless technologies are included within the definition of transmission media.
[0103] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.
[0104] In addition, terms explained in this disclosure and / or terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.
[0105] As used in this disclosure, the terms "system" and "network" are used interchangeably.
[0106] In addition, the information, parameters, etc. described in the present disclosure may be represented by absolute values, may be represented by relative values from a predetermined value, or may be represented by other corresponding information. For example, a radio resource may be indicated by an index.
[0107] The names used for the above-mentioned parameters are not limiting in any way. Moreover, the formulas using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not limiting in any way.
[0108] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment" can be interpreted as "assuming," "expecting," "considering," etc.
[0109] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."
[0110] When designations such as "first," "second," and the like are used in this disclosure, any reference to an element is not intended to generally limit the quantity or order of those elements. These designations may be used herein as a convenient way to distinguish between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed therein or that the first element must precede the second element in some way.
[0111] The "means" in the configuration of each of the above devices may be replaced with "part," "circuit," "device," etc.
[0112] To the extent that the terms "include," "including," and variations thereof are used herein in the specification or claims, these terms are intended to be inclusive, similar to the term "comprising." Further, the term "or" as used herein is not intended to be an exclusive or.
[0113] In this disclosure, where articles have been added due to translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.
[0114] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."
[0115] The image providing system 1 and the image display device 10 of the present disclosure may have the following configuration. [1] An image display device that displays a three-dimensional image provided from an image providing device, a combining unit configured to combine a plurality of three-dimensional rendering images to generate a combined image, the plurality of three-dimensional rendering images being obtained by parallel projecting a given three-dimensional space from a plurality of viewpoints with the same projection direction based on three-dimensional spatial data representing the three-dimensional space, the ranges of the three-dimensional space being adjacent or overlapping, and including depth information associated with each pixel; a perspective projection image generator that generates a perspective projection image by perspective rendering from a given viewpoint based on the combined image and depth and color values associated with each pixel in the combined image; An image display device comprising: [2] the three-dimensional rendering image includes a color image in which each pixel is associated with a color value and the depth information; the combining unit generates mesh data corresponding to the three-dimensional rendering image based on the depth information, and sets color values associated with pixels corresponding to each vertex of the mesh in the color image to the vertices of the mesh; The image display device according to [1]. [3] The combining unit combines the mesh data or the mesh data in which color values of corresponding pixels are set at vertices of each mesh. The image display device according to [2]. [4] The perspective projection image generating unit When the depth of a first mesh vertex in a horizontal arrangement of meshes and mesh vertices in the mesh data corresponding to the combined image is smaller than the depth of the first mesh vertex in the arrangement direction by a difference exceeding a given first threshold, and the difference between the depth of a fourth mesh vertex located in the arrangement direction of the second mesh vertex and the depth of the first mesh vertex is equal to or less than a given second threshold or the depth of the fourth mesh vertex is equal to or greater than the depth of the first mesh vertex, interpolating a color value of a portion between the first mesh vertex and the second mesh vertex in the perspective projection image, or a color value of a portion between a third mesh vertex adjacent to the fourth mesh vertex in the opposite direction of the arrangement direction and the fourth mesh vertex, based on the color value of the first mesh vertex and the color value of the fourth mesh vertex. The image display device according to [2] or [3]. [5] the perspective projection image generating unit linearly interpolates a color value of a portion between the first mesh vertex and the second mesh vertex in the perspective projection image, or a color value of a portion between the third mesh vertex and the fourth mesh vertex, based on the positions and color values of the first mesh vertex and the fourth mesh vertex, respectively. [4] The image display device according to [4]. [6] An image providing system including an image providing device and an image display device, for providing a three-dimensional image to the image display device, The image providing device includes: a parallel projection image generating unit configured to generate 3D rendering images by parallel projecting the 3D space from a plurality of viewpoints in the same projection direction based on 3D space data representing a given 3D space, the 3D rendering images including depth information associated with each pixel; The image display device includes: a combiner for combining a plurality of the three-dimensional rendering images in which the ranges of the three-dimensional space represented are adjacent or overlapping to generate a combined image; a perspective projection image generator that generates a perspective projection image by perspective rendering from a given viewpoint based on the combined image and depth and color values associated with each pixel in the combined image; An image providing system comprising: [7] the parallel projection image generating unit generates the 3D rendering image including a color image in which a color value is associated with each pixel and the depth information; The combining unit generates mesh data corresponding to the three-dimensional rendering image based on the depth information, and sets color values associated with corresponding pixels in the color image to vertices of each mesh. [6] The image providing system according to the present invention. [Explanation of symbols]
[0116] 1...image providing system, 10...image display device, 11...acquisition unit, 12...combination unit, 13...perspective projection image generating unit, 14...output unit, 20...image providing device, 21...parallel projection image generating unit, 22...transmission unit, 25...three-dimensional space model memory unit, M1...recording medium, m10...main module, m11...acquisition module, m12...combination module, m13...perspective projection image generating module, m14...output module, P1...image display program.
Claims
1. An image display device that displays a three-dimensional image provided from an image providing device, a combining unit configured to combine a plurality of three-dimensional rendering images to generate a combined image, the plurality of three-dimensional rendering images being obtained by parallel projecting a given three-dimensional space from a plurality of viewpoints in the same projection direction based on three-dimensional spatial data representing the three-dimensional space, the ranges of the three-dimensional space being represented being adjacent or overlapping, and including depth information associated with each pixel; a perspective projection image generator that generates a perspective projection image by perspective rendering from a given viewpoint based on the combined image and depth and color values associated with each pixel in the combined image; An image display device comprising:
2. the three-dimensional rendering image includes a color image in which each pixel is associated with a color value and the depth information; the combining unit generates mesh data corresponding to the three-dimensional rendering image based on the depth information, and sets color values associated with pixels corresponding to each vertex of the mesh in the color image to the vertices of the mesh; The image display device according to claim 1 .
3. The combining unit combines the mesh data or the mesh data in which color values of corresponding pixels are set at vertices of each mesh. The image display device according to claim 2 .
4. The perspective projection image generating unit When the depth of a first mesh vertex in a horizontal arrangement of meshes and mesh vertices in the mesh data corresponding to the combined image is smaller than the depth of the first mesh vertex in the arrangement direction by a difference exceeding a given first threshold, and the difference between the depth of a fourth mesh vertex located in the arrangement direction of the second mesh vertex and the depth of the first mesh vertex is equal to or less than a given second threshold or the depth of the fourth mesh vertex is equal to or greater than the depth of the first mesh vertex, a color value of a portion between the first mesh vertex and the second mesh vertex in the perspective projection image, or a color value of a portion between a third mesh vertex adjacent to the fourth mesh vertex in the opposite direction of the arrangement direction and the fourth mesh vertex, is interpolated based on the color value of the first mesh vertex and the color value of the fourth mesh vertex. The image display device according to claim 2 .
5. the perspective projection image generating unit linearly interpolates a color value of a portion between the first mesh vertex and the second mesh vertex in the perspective projection image, or a color value of a portion between the third mesh vertex and the fourth mesh vertex, based on the positions and color values of the first mesh vertex and the fourth mesh vertex, respectively. The image display device according to claim 4.
6. An image providing system including an image providing device and an image display device, for providing a three-dimensional image to the image display device, The image providing device includes: a parallel projection image generating unit configured to generate three-dimensional rendering images by parallel projecting the three-dimensional space from a plurality of viewpoints in the same projection direction based on three-dimensional space data representing a given three-dimensional space, the three-dimensional rendering images including depth information associated with each pixel; The image display device includes: a combiner for combining a plurality of the three-dimensional rendering images in which the ranges of the three-dimensional space represented are adjacent or overlapping to generate a combined image; a perspective projection image generator that generates a perspective projection image by perspective rendering from a given viewpoint based on the combined image and depth and color values associated with each pixel in the combined image; An image providing system comprising:
7. the parallel projection image generator generates the three-dimensional rendering image including a color image in which each pixel is associated with a color value and the depth information; The combining unit generates mesh data corresponding to the three-dimensional rendering image based on the depth information, and sets color values associated with corresponding pixels in the color image to vertices of each mesh. The image providing system according to claim 6.
Citation Information
Patent Citations
Method and system for rendering virtual reality content based on two-dimensional ("2D") captured images of a three-dimensional ("3D") scene
JP2020522803A
Intermediate view synthesis between wide-baseline panoramas
WO2022227068A1
Image formation device, image display device, and image formation method and image formation program
JP2013225740A
