Image Processing Device

Abstract

To provide an image processing apparatus that can generate highly immersive background images in a three-dimensional virtual space using the entire celestial sphere image.SOLUTION: An image processing apparatus 3 comprises: a model forming part 33 that generates a sphere model Ms with a center C as a reference point in a three-dimensional virtual space and forms a horizontal plane model Mh with a distance Dh to the center C of the sphere model Ms being set based on ground surface distance information; and a drawing part 34 that associate coordinate values of each vertex on the sphere model Ms and coordinate values of each vertex on the horizontal plane model Mh with coordinate values of an entire celestial sphere image I, replaces the coordinate values of the entire celestial sphere image I corresponding to the coordinate values of each vertex Ph on the horizontal plane model Mh with the coordinate values of the entire celestial sphere image I set at each intersection point Ps of a direction vector from the center C of the sphere model Ms to each vertex Ph and the sphere model Ms, and maps the entire celestial sphere image I to the spherical model Ms and the horizontal plane model Mh to form a background image Imap of the three-dimensional virtual space.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device that forms a background image of a three-dimensional virtual space. [Background technology]

[0002] As a technique for easily creating a background for a virtual reality (VR) space, a method of pasting a spherical image captured by an omnidirectional camera or the like onto a spherical three-dimensional model is known. When a background image created by this method is displayed on a head mounted display (HMD), a user wearing the HMD observes a three-dimensional virtual space in which the spherical image appears to be projected around them.

[0003] For example, Patent Document 1 discloses an image processing device including: a model forming means for forming a three-dimensional mesh shape model by combining a plurality of mesh shapes corresponding to features of a celestial sphere image; and a rendering means for converting the coordinate value of each pixel into a coordinate system of the celestial sphere image based on the coordinate value of a virtual reference point set in three-dimensional space and the coordinate value of each pixel of the three-dimensional mesh shape model, and mapping the celestial sphere image on the three-dimensional mesh shape model to form a celestial sphere stereoscopic image. This image processing device can provide a celestial sphere image with a three-dimensional feel in a virtual environment. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-133310 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when a background image in which a celestial sphere image is pasted only onto a spherical three-dimensional model is projected onto an HMD, the user wearing the HMD feels as if they are floating because there is no image information at a position corresponding to the ground in the three-dimensional virtual space observed by the user. In Patent Document 1 mentioned above, a planar mesh shape model is formed corresponding to the horizontal plane (ground) in the celestial sphere image, and the celestial sphere image is mapped onto a three-dimensional mesh shape model that is a combination of the planar mesh shape model and a spherical mesh shape model, thereby reducing the floating feeling of the user.

[0006] However, the planar mesh shape model in Patent Document 1 is formed to correspond to the position of a horizontal plane in the celestial sphere image, regardless of the distance between the ground and the imaging position when the celestial sphere image is actually captured. Therefore, a user who views the image of the ground mapped onto the planar mesh shape model may feel uncomfortable about the distance to the ground in the three-dimensional virtual space, which may reduce the sense of realism in the three-dimensional virtual space and reduce the sense of immersion.

[0007] The present invention has been made in light of the above points, and has an object to provide an image processing device that can generate a highly immersive background image in a three-dimensional virtual space using a spherical image. [Means for solving the problem]

[0008] To achieve the above object, one aspect of the present invention provides an image processing device including: a storage unit that stores a celestial sphere image captured by an omnidirectional camera of a real space including the ground and a three-dimensional object; a model formation unit that forms a three-dimensional model composed of a combination of multiple meshes based on features of the celestial sphere image stored in the storage unit; and a rendering unit that converts coordinate values ​​of each vertex of the three-dimensional model formed by the model formation unit into a coordinate system of the celestial sphere image stored in the storage unit based on coordinate values ​​of a reference point set in a three-dimensional virtual space, and maps the celestial sphere image onto the three-dimensional model to form a celestial sphere stereoscopic image. In this image processing device, the storage unit is configured to store ground distance information indicating the distance from an imaging position of the celestial sphere camera to the ground. The model formation unit is configured to form a spherical model centered on the reference point in the three-dimensional virtual space, and to form a horizontal plane model corresponding to the ground, which is positioned downward as viewed from the center of the spherical model and whose distance to the center is set based on the ground distance information stored in the storage unit. The rendering unit is configured to convert, based on the coordinate value of the reference point in the three-dimensional virtual space, the coordinate value of each vertex on the spherical model formed by the model forming unit into the coordinate system of the celestial sphere image stored in the storage unit, and to replace the coordinate value of each vertex on the horizontal planar model formed by the model forming unit with a coordinate value after coordinate system conversion that is set at each intersection between the spherical model and each direction vector from the center of the spherical model toward each vertex on the horizontal planar model, and to map the celestial sphere image onto the spherical model and the horizontal planar model, thereby forming the celestial sphere stereoscopic image. [Effects of the Invention]

[0009] According to the image processing device of the present invention, the sense of discomfort felt by the user regarding the distance to the ground in the three-dimensional virtual space is suppressed, and therefore, a background image with a high sense of immersion in the three-dimensional virtual space can be generated using a spherical image. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a schematic configuration of a driving simulator system to which an image processing device according to a first embodiment of the present invention is applied. [Figure 2] 2 is a conceptual diagram showing three-dimensional models (a spherical model and a horizontal plane model) formed by a model forming unit in the first embodiment. FIG. [Figure 3] FIG. 4 is a conceptual diagram illustrating a process executed by a drawing unit according to the first embodiment for associating the coordinate values ​​of each vertex on a spherical model with the coordinate values ​​of a spherical image. [Figure 4] FIG. 10 is a conceptual diagram for explaining the process of replacing UV coordinate values ​​corresponding to the coordinate values ​​of each vertex on a horizontal planar model with UV coordinate values ​​set on a spherical model, which is executed by the rendering unit in the first embodiment. [Figure 5] 10A to 10C are conceptual diagrams showing three-dimensional models (a spherical model, a horizontal plane model, and a vertical plane model) formed by a model forming unit of an image processing device according to a second embodiment of the present invention. [Figure 6] FIG. 11 is a conceptual diagram for explaining a process executed by a rendering unit in the second embodiment to replace UV coordinate values ​​corresponding to the coordinate values ​​of each vertex on a vertical plane model with UV coordinate values ​​set on a spherical model. [Figure 7] FIG. 10 is a conceptual diagram for explaining the effect of the second embodiment. [Figure 8] FIG. 11 is a conceptual diagram illustrating a cross section of a three-dimensional model onto which a spherical image is pasted in the third embodiment. [Figure 9] FIG. 10 is a conceptual diagram showing a modified example related to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a block diagram showing a schematic configuration of a driving simulator system to which an image processing device according to a first embodiment of the present invention is applied. 1, a driving simulator system 1 includes a spherical camera 2 that captures an image of a real space, an image processing device 3 according to a first embodiment that generates a background image Imap of a three-dimensional virtual space using a spherical image I captured by the spherical camera 2, and a head-mounted display (HMD) 4 that displays the background image Imap generated by the image processing device 3. This driving simulator system 1 can be used, for example, for simulating the experience of driving a vehicle in vehicle development or sales promotion activities for automobiles and the like.

[0012] The image processing device 3 includes, as its functional blocks, an input unit 31, a storage unit 32, a model formation unit 33, a drawing unit 34, and an output unit 35. Although not shown here, the hardware configuration of the image processing device 3 includes, for example, a computer system including a processor, memory, a user input interface, and a communication interface. In other words, in the image processing device 3, the processor of the computer system reads and executes a program stored in the memory, thereby realizing the function of each block.

[0013] The input unit 31 is realized by a user input interface of a computer system, and includes, for example, a keyboard, a mouse, an operation controller, etc. The input unit 31 also includes a receiving unit that receives information from the outside via a wired or wireless connection, and functions as an external information input interface that receives information from an external computer, etc. A celestial sphere image I captured by the celestial sphere camera 2 of a real space is provided to the input unit 31, and the celestial sphere image I is stored in the storage unit 32. The celestial sphere camera 2 is also called an omnidirectional camera or a 360-degree camera.

[0014] The real space captured by the omnidirectional camera 2 includes the ground, such as the road on which vehicles travel and the sidewalks beside it, and three-dimensional objects, such as buildings, traffic lights, road signs, and forests, that are located around the road. When capturing a omnidirectional image I, a distance D1 from the imaging position of the omnidirectional camera 2 to the ground and a distance D2 from the imaging position of the omnidirectional camera 2 to the three-dimensional object are measured, and information indicating the measurement results is stored in the storage unit 32 via the input unit 31 together with the omnidirectional image I. That is, the storage unit 32 of the image processing device 3 is configured to be able to store the omnidirectional image I captured by the omnidirectional camera 2 of the real space including the ground and the three-dimensional object, as well as information indicating the distance D1 to the ground (hereinafter referred to as "ground distance information") and information indicating the distance D2 to the three-dimensional object (hereinafter referred to as "three-dimensional object distance information") that were measured when capturing the omnidirectional image I.

[0015] The model forming unit 33 forms a 3D model formed by combining a plurality of meshes based on the features of the omnidirectional image I stored in the storage unit 32. Each of the plurality of meshes forming the 3D model has three or more vertices, edges connecting the vertices, and a surface closed by the edges. The density of the plurality of meshes can be set according to the accuracy required for the 3D model, etc. The model forming unit 33 in the first embodiment forms, for example, a spherical model Ms and a horizontal plane model Mh corresponding to the ground as the 3D model formed from the plurality of meshes.

[0016] FIG. 2 is a conceptual diagram showing the spherical model Ms and the horizontal plane model Mh formed in the model forming section 33. As shown in FIG. As shown in FIG. 2, the spherical model Ms is three-dimensional model data in which a spherical surface of radius r is configured by combining a plurality of meshes (not shown) with a reference point C set in the three-dimensional virtual space as its center. The three-dimensional virtual space is a virtual space provided to a user U wearing an HMD 4. The reference point of the three-dimensional virtual space is set to correspond to the position of the eyes of the user U in the three-dimensional virtual space. By mapping the omnidirectional image I onto the spherical model Ms, whose center C is set to the reference point of the three-dimensional virtual space, and projecting the image onto the HMD 4, the user U can observe the three-dimensional virtual space from a viewpoint corresponding to the imaging position of the omnidirectional camera 2. The setting of the radius r of the spherical model Ms will be described later.

[0017] The horizontal planar model Mh is two-dimensional model data that is arranged downward when viewed from the center C (reference point of the three-dimensional virtual space) of the spherical model Ms, as shown in the shaded area in Figure 2, and that forms a plane that extends roughly horizontally by combining multiple meshes. The distance Dh from the center C of the spherical model Ms to the horizontal planar model Mh, i.e., the length of the perpendicular line dropped from the center C of the spherical model Ms to the horizontal planar model Mh, is set based on the ground distance information stored in the storage unit 32.

[0018] Specifically, the distance Dh from the center C of the spherical model Ms to the horizontal planar model Mh is set to correspond to the distance D1 from the imaging position of the omnidirectional camera 2 to the ground, which is indicated by the ground distance information. That is, as the installation height of the omnidirectional camera 2 from the ground when capturing the omnidirectional image I increases (or decreases), the distance Dh from the center C of the spherical model Ms to the horizontal planar model Mh increases (or decreases). By combining such a horizontal planar model Mh with the spherical model Ms, three-dimensional model data is generated as a whole. The three-dimensional model (spherical model Ms and horizontal planar model Mh) formed by the model forming unit 33 is transmitted to the rendering unit 34.

[0019] The rendering unit 34 performs a process of associating the coordinate values ​​of each vertex on the spherical model Ms formed by the model forming unit 33 with the coordinate values ​​of the celestial sphere image I stored in the storage unit 32, based on the coordinate values ​​of the reference point in the three-dimensional virtual space, and also performs a process of associating the coordinate values ​​of each vertex on the horizontal planar model Mh formed by the model forming unit 33 with the coordinate values ​​of the celestial sphere image I stored in the storage unit 32. Furthermore, the rendering unit 34 performs a process of replacing the coordinate values ​​of the celestial sphere image I corresponding to the coordinate values ​​of each vertex on the horizontal planar model Mh with the coordinate values ​​of the celestial sphere image I set at each intersection between the spherical model Ms and each direction vector from the center C of the spherical model Ms toward each vertex on the horizontal planar model Mh. In this way, the rendering unit 34 maps the celestial sphere image I onto the spherical model Ms and the horizontal planar model Mh to form a background image Imap of the three-dimensional virtual space, and outputs the background image Imap to the HMD 4 via the output unit 35.

[0020] FIG. 3 is a conceptual diagram illustrating the process executed by the drawing unit 34 for associating the coordinate values ​​of each vertex on the spherical model Ms with the coordinate values ​​of the omnidirectional image I. As shown on the right side of Figure 3, the vertices (not shown) of a plurality of meshes that make up the spherical model Ms exist on a spherical surface of radius r with the reference point of the three-dimensional virtual space as the center C. In an orthogonal coordinate system (hereinafter referred to as the "XYZ coordinate system") consisting of three mutually orthogonal axes, X-axis, Y-axis, and Z-axis, if the coordinate values ​​of the reference point of the three-dimensional virtual space are (a, b, c), then the coordinate values ​​(x, y, z) of each vertex on the spherical model Ms are given by the spherical equation: (xa) 2 +(yb) 2 +(zc) 2 =r 2 It can be expressed using:

[0021] On the other hand, for the coordinate system of the spherical image I, as shown on the left side of FIG. 3 , a UV coordinate system is set which is used to specify the position, direction, size, and the like of the texture (image) to be pasted when mapping the texture (image) onto a three-dimensional model. The spherical image I in such a UV coordinate system is mapped onto a three-dimensional model (spherical model Ms and horizontal planar model Mh) formed by the model forming unit 33. That is, for the image mapped onto the three-dimensional model, coordinate values ​​of the UV coordinate system which are uniquely set for coordinate values ​​of the XYZ coordinate system at each vertex on the three-dimensional model are set. In the first embodiment, the image mapped onto the spherical model Ms has UV coordinate values ​​which are set in a one-to-one relationship with the XYZ coordinate values ​​of each vertex on the spherical model Ms as described above. Furthermore, the image mapped onto the horizontal planar model Mh has UV coordinate values ​​which are set in a one-to-one relationship with the XYZ coordinate values ​​of each vertex on the horizontal planar model Mh as described later.

[0022] Specifically, a UV coordinate system, which is a two-dimensional Cartesian coordinate system with the horizontal axis designated U and the vertical axis designated V, is set for the image (spherical image I) to be mapped to the spherical model Ms (left side of FIG. 3 ). Vertices are set on the spherical image I in an evenly spaced grid pattern, and the UV coordinate values ​​of each vertex are associated with the XYZ coordinate values ​​of each vertex on the spherical model Ms. The UV coordinate values ​​are set within the range of 0 to 1 for each of the U and V axes. Setting the vertices on the spherical image I in an evenly spaced grid pattern in this way reduces distortion of the spherical image I pasted on the spherical model Ms. However, for the convenience of creating a mesh on the spherical model Ms, triangular wave-like vertices, rather than grid-like vertices, are set in the corresponding areas of the spherical image I near the poles that serve as end points (the ranges surrounded by dashed lines on the right side of FIG. 3 ). For this reason, near the poles of the spherical model Ms, there is a small area in the celestial sphere image I that cannot be pasted onto the spherical model Ms. The diagonally shaded area on the left side of FIG. 3 represents the area of ​​the celestial sphere image I that can be pasted onto the spherical model Ms.

[0023] Fig. 4 is a conceptual diagram for explaining the process of replacing UV coordinate values ​​corresponding to the coordinate values ​​of each vertex on the horizontal planar model Mh with UV coordinate values ​​set on the spherical model Ms, which is executed by the drawing unit 34. Note that Fig. 4 shows a cross section of the spherical model Ms and the horizontal planar model Mh of Fig. 2 cut along the XZ plane.

[0024] As shown in the upper part of Figure 4, the horizontal planar model Mh is located at a distance Dh downward from the center C of the spherical model Ms. Each vertex Ph (denoted by a thin white circle) of the mesh that makes up the horizontal planar model Mh is located on the horizontal planar model Mh. Each vertex Ph of the horizontal planar model Mh is assigned a coordinate value (x, y, z) in the XYZ coordinate system in the three-dimensional virtual space. As mentioned above, if the coordinate values ​​of the reference point in the three-dimensional virtual space are (a, b, c), then the coordinate value of each vertex Ph in the Z-axis direction is expressed as z = c - Dh.

[0025] As described above, the image mapped onto the horizontal planar model Mh has UV coordinate values ​​that are set in a one-to-one relationship with the XYZ coordinate values ​​of each vertex on the horizontal planar model Mh. In the first embodiment, the rendering unit 34 performs a process of replacing the UV coordinate values ​​corresponding to the XYZ coordinate values ​​of each vertex on the horizontal planar model Mh with the UV coordinate values ​​set at each intersection of the spherical model Ms and each directional vector extending from the center C of the spherical model Ms to each vertex on the horizontal planar model Mh. In this process, as shown in the middle part of FIG. 4, the rendering unit 34 first sets a straight line connecting the center C of the spherical model Ms to each vertex Ph on the horizontal planar model Mh and then extends each straight line radially outward (dotted lines in FIG. 4). These straight lines correspond to the directional vector extending from the center C of the spherical model Ms to each vertex Ph on the horizontal planar model Mh.

[0026] The rendering unit 34 then identifies each intersection Ps (thick white circle) between each line (directional vector) and the spherical model Ms. At each intersection Ps, a UV coordinate value of the omnidirectional image I mapped to the spherical model Ms is set. Therefore, the rendering unit 34 acquires the UV coordinate value set at each intersection Ps, and, as shown in the lower part of FIG. 4, replaces the UV coordinate value corresponding to the XYZ coordinate value of each vertex Ph on the horizontal planar model with the acquired UV coordinate value. Note that, among the vertices Ph on the horizontal planar model Mh, the UV coordinate value corresponding to the XYZ coordinate value of a vertex Ph located at a portion where the spherical model Ms and the horizontal planar model Mh overlap is replaced with the UV coordinate value set at the point on the spherical model Ms that coincides with the vertex Ph. Through this series of processes, an image of the omnidirectional image I corresponding to the ground is pasted onto the horizontal planar model Mh.

[0027] The background image Imap of the three-dimensional virtual space formed by mapping the omnidirectional image I onto the spherical model Ms and the horizontal plane model Mh by the rendering unit 34 as described above is transmitted to the output unit 35 ( FIG. 1 ) of the image processing device 3. The output unit 35 outputs the background image Imap from the rendering unit 34 to the HMD 4 by using a communication interface of the computer system.

[0028] The HMD 4 is a well-known display device that can project a background image Imap from the image processing device 3 as a background of the three-dimensional virtual space, thereby allowing the user U to perceive a three-dimensional virtual space. The HMD 4 may also be provided with a head tracking function that detects the movement of the user U's head and a position tracking function that detects the movement of the user U, using a sensor or the like built into the HMD 4. Information regarding the movement of the user U detected by these functions is transmitted to the rendering unit 34 via the input unit 31 of the image processing device 3, and is used for updating the background image Imap in conjunction with the movement of the user U, etc.

[0029] In the driving simulator system 1 to which the image processing device 3 according to the first embodiment as described above is applied, the omnidirectional image I captured by the omnidirectional camera 2 of the real space is mapped onto the spherical model Ms and the horizontal plane model Mh by the image processing device 3 to form a background image Imap of the three-dimensional virtual space, and the background image Imap is projected onto the HMD 4. This allows the user U wearing the HMD 4 to observe the background image Imap of the three-dimensional virtual space projected onto the HMD 4 from a viewpoint corresponding to the imaging position of the omnidirectional camera 2.

[0030] At this time, the distance Dh from the horizontal planar model Mh corresponding to the ground to the center C (reference point of the three-dimensional virtual space) of the spherical model Ms is set to correspond to the distance D1 from the imaging position of the omnidirectional camera 2 to the ground when the omnidirectional image I is captured, so it is possible to prevent the user U, who views the image of the ground mapped on the horizontal planar model Mh, from feeling uncomfortable about the distance to the ground in the three-dimensional virtual space. Therefore, according to the image processing device 3 of the first embodiment, it is possible to generate a background image that provides a high level of immersion in the three-dimensional virtual space using the omnidirectional image I captured by the omnidirectional camera 2. In particular, by associating the XYZ coordinate values ​​of each vertex of the spherical model Ms and the horizontal planar model Mh with the UV coordinate values ​​of the omnidirectional image I, it is possible to realize highly versatile mapping processing.

[0031] Next, an image processing apparatus according to a second embodiment of the present invention will be described. In the image processing device 3 according to the first embodiment described above, a case has been described in which a three-dimensional model combining a spherical model Ms and a horizontal plane model Mh corresponding to the ground is formed by the model forming unit 33. In the image processing device 3 according to the second embodiment, a case will be described in which the model forming unit 33 forms a vertical plane model Mv corresponding to a three-dimensional object in addition to the spherical model Ms and the horizontal plane model Mh.

[0032] 5 is a conceptual diagram showing three-dimensional models (spherical model Ms, horizontal plane model Mh, and vertical plane model Mv) formed by the model forming unit 33 in the second embodiment. Note that the functional blocks of the image processing device 3 according to the second embodiment and the configuration of the driving simulator system 1 to which the image processing device 3 according to the second embodiment is applied are similar to the configuration of the first embodiment shown in FIG. 1 above, and therefore description thereof will be omitted here.

[0033] As shown in FIG. 5 , the model forming unit 33 in the second embodiment forms a spherical model Ms and a horizontal planar model Mh in the same manner as in the first embodiment described above, and also forms a vertical planar model Mv corresponding to a three-dimensional object included in the omnidirectional image I. As shown in the hatched area in FIG. 5 , the vertical planar model Mv is arranged in the horizontal direction as viewed from the center C of the spherical model Ms (the reference point of the three-dimensional virtual space), and is two-dimensional model data in which a plane extending in a substantially vertical direction is configured by combining a plurality of meshes. In the example shown in FIG. 5 , the vertical planar model Mv is arranged at an interval in the X-axis direction from the center C of the spherical model Ms, and is formed substantially parallel to the YZ plane. The distance Dv from the center C of the spherical model Ms to the vertical planar model Mv is set based on three-dimensional object distance information stored in the storage unit 32.

[0034] Specifically, the distance Dv from the center C of the spherical model Ms to the vertical-plane model Mv is set to correspond to the distance D2 from the imaging position of the omnidirectional camera 2 to the three-dimensional object, which is indicated by the three-dimensional object distance information. That is, as the three-dimensional object moves farther away (or closer) from the installation position of the omnidirectional camera 2 when capturing the omnidirectional image I, the distance Dv from the center C of the spherical model Ms to the vertical-plane model Mv becomes longer (or shorter). By combining such a vertical-plane model Mv with the spherical model Ms and the horizontal-plane model Mh, three-dimensional model data is generated as a whole. The three-dimensional model (spherical model Ms, horizontal-plane model Mh, and vertical-plane model Mv) formed by the model forming unit 33 is transmitted to the rendering unit 34.

[0035] As in the first embodiment, the rendering unit 34 in the second embodiment replaces the UV coordinate values ​​corresponding to the XYZ coordinate values ​​of each vertex on the horizontal planar model Mh with the UV coordinate values ​​set at each intersection between the spherical model Ms and each directional vector extending from the center C of the spherical model Ms to each vertex on the horizontal planar model Mh, and also replaces the UV coordinate values ​​corresponding to the XYZ coordinate values ​​of each vertex on the vertical planar model Mv with the UV coordinate values ​​set at each intersection between the spherical model Ms and each directional vector extending from the center C of the spherical model Ms to each vertex on the vertical planar model Mv. The rendering unit 34 then maps the omnidirectional image I onto the spherical model Ms, the horizontal planar model Mh, and the vertical planar model Mv to form a background image Imap of the three-dimensional virtual space, and outputs the background image Imap to the HMD 4 via the output unit 35.

[0036] Fig. 6 is a conceptual diagram for explaining the process of replacing UV coordinate values ​​corresponding to the coordinate values ​​of each vertex on the vertical planar model Mv with UV coordinate values ​​set on the spherical model Ms, which is executed by the rendering unit 34 in the second embodiment. Fig. 6 shows a cross section of the spherical model Ms, horizontal planar model Mh, and vertical planar model Mv in Fig. 5 cut along the XZ plane.

[0037] As shown in the upper part of FIG. 6, the vertical planar model Mv is provided at a position a distance Dv away from the center C of the spherical model Ms in the horizontal direction (X-axis direction). Each vertex Pv (thin white circle) of the mesh that constitutes the vertical planar model Mv is arranged on the vertical planar model Mv. Coordinate values ​​(x, y, z) of the XYZ coordinate system in the three-dimensional virtual space are set for each vertex Pv of the vertical planar model Mv. As in the first embodiment described above, if the coordinate values ​​of the reference point in the three-dimensional virtual space are (a, b, c), the coordinate value of each vertex Pv in the X-axis direction is expressed as x = a - Dv.

[0038] The image mapped onto the vertical planar model Mv as described above has UV coordinate values ​​that are set in a one-to-one relationship with the XYZ coordinate values ​​of each vertex on the vertical planar model Mv. In the second embodiment, the rendering unit 34 performs a process of replacing the UV coordinate values ​​corresponding to the XYZ coordinate values ​​of each vertex on the vertical planar model Mv with the UV coordinate values ​​set at each intersection of the spherical model Ms and each directional vector extending from the center C of the spherical model Ms to each vertex on the vertical planar model Mv. In this process, as shown in the middle part of FIG. 6, the rendering unit 34 first sets straight lines connecting the center C of the spherical model Ms to each vertex Pv on the vertical planar model Mv and then extends each straight line radially outward (dotted lines in FIG. 6). These straight lines correspond to directional vectors extending from the center C of the spherical model Ms to each vertex Pv on the vertical planar model Mv.

[0039] Then, the rendering unit 34 identifies each intersection Ps (thick white circle) between each straight line (directional vector) and the spherical model Ms. At each intersection Ps, a UV coordinate value of the omnidirectional image I mapped to the spherical model Ms is set. Therefore, the rendering unit 34 acquires the UV coordinate value set at each intersection Ps, and replaces the UV coordinate value corresponding to the XYZ coordinate value of each vertex Pv on the vertical planar model with the acquired UV coordinate value, as shown in the lower part of FIG. 6 . Note that, among the vertices Pv on the vertical planar model Mv, the UV coordinate value corresponding to the XYZ coordinate value of a vertex Pv located outside the spherical model Ms is replaced with the UV coordinate value set at the intersection Ps of the spherical model Ms on the line segment connecting the vertex Pv and the center C of the spherical model Ms. Through this series of processes, an image area of ​​the omnidirectional image I corresponding to a three-dimensional object is pasted onto the vertical planar model Mv.

[0040] In the driving simulator system 1 to which the image processing device 3 according to the second embodiment as described above is applied, the omnidirectional image I captured by the omnidirectional camera 2 of the real space is mapped onto the spherical model Ms, the horizontal plane model Mh, and the vertical plane model Mv by the image processing device 3 to form a background image Imap of the three-dimensional virtual space, and the background image Imap is projected onto the HMD 4. By forming the vertical plane model Mv corresponding to the three-dimensional object and mapping the omnidirectional image I, it is possible to generate a background image of the three-dimensional virtual space that gives a less awkward feeling. Fig. 7 is a conceptual diagram for explaining the effect of the second embodiment.

[0041] As shown in the upper and middle sections of FIG. 7 , when a background image of a three-dimensional virtual space is created using a celestial sphere image I obtained by capturing an image of a real space including the ground G and a three-dimensional object O with the celestial sphere camera 2, in the first embodiment in which only a spherical model Ms and a horizontal planar model Mh are formed, the farther the position of the three-dimensional object O is from the imaging position of the celestial sphere camera 2, the more noticeable the coarseness of the mesh of the three-dimensional model onto which the image Io of the three-dimensional object is pasted, and distortion may occur in the image Io of the three-dimensional object pasted on the horizontal planar model Mh depending on the distance D2 from the three-dimensional object O to the imaging position. Such mesh coarseness and distortion of the image Io of the three-dimensional object can be reduced by preparing a vertical planar model Mv corresponding to the three-dimensional object, as shown in the lower section of FIG. 7 . In particular, if the distance Dv from the vertical planar model Mv to the center C of the spherical model Ms is set to correspond to the distance D2 from the imaging position of the celestial sphere camera 2 to the three-dimensional object O, it becomes possible to paste the image Io of the three-dimensional object onto the vertical planar model Mv without distortion. Therefore, according to the image processing device 3 of the second embodiment, it is possible to generate a background image with a higher immersive feeling in a three-dimensional virtual space using the omnidirectional image I captured by the omnidirectional camera 2.

[0042] Next, an image processing apparatus according to a third embodiment of the present invention will be described. In the image processing device 3 according to the second embodiment described above, a case has been described in which mesh coarseness and distortion of the image Io of a three-dimensional object are reduced by forming a vertical planar model Mv corresponding to the three-dimensional object. In the image processing device 3 according to the third embodiment, an example will be described in which, instead of forming a vertical planar model Mv, the radius r of the spherical model Ms is adjusted to achieve the same effect as in the second embodiment. Note that the functional blocks of the image processing device 3 according to the third embodiment and the configuration of the driving simulator system 1 to which the image processing device 3 according to the third embodiment is applied are similar to the configuration of the first embodiment shown in FIG. 1 above, and therefore will not be described here.

[0043] FIG. 8 is a conceptual diagram showing a cross section of a three-dimensional model (a spherical model Ms and a horizontal plane model Mh) on which a spherical image I is pasted in the third embodiment, cut along an XZ plane. 8, in the image processing device 3 according to the third embodiment, when at least a part of an image Io corresponding to a three-dimensional object in a background image Imap formed by the rendering unit 34 in the same manner as in the first embodiment described above is mapped to a horizontal plane model Mh (upper part of FIG. 8), the model forming unit 33 adjusts the radius of the spherical model Ms from r to r' (lower part of FIG. 8) based on the three-dimensional object distance information stored in the storage unit 32. By adjusting the radius of the spherical model Ms to r' in this way, the image Io corresponding to a three-dimensional object in the celestial sphere image I is mapped only to the spherical model.

[0044] Regarding the adjustment of the radius of the spherical model Ms as described above, let us consider, for example, the creation of a 3D virtual space background based on the viewpoint of a driver of a vehicle traveling on a typical urban road. The minimum road width for a 2.5-m-wide vehicle is 6.5 m. If we assume that the driver's position is approximately 1 m outward from the center of the road, the distance from the driver's viewpoint to the edge of the road is (6.5 / 2) - 1.0 = 2.25 m. If we further assume that there is a 1.0-m-wide sidewalk outside the road edge, and that there is a three-dimensional object such as a house, building, or forest beyond that, the distance from the driver's viewpoint to the three-dimensional object is 2.25 + 1.0 = 3.25 m.

[0045] Therefore, when considering a vehicle traveling on a typical urban road, it is preferable to set the adjusted radius r' of the spherical model Ms formed by the model formation unit 33 to 3.25 m. By adjusting the radius of the spherical model Ms based on the width of the road on which the vehicle is traveling in this way, it is possible to project onto the HMD 4 a background image Imap of the 3D virtual space that provides a natural appearance of surrounding three-dimensional objects to the driver in a driving scene on a typical urban road.

[0046] Furthermore, for example, when traveling in an open area such as a parking lot, it is preferable to set the radius r' of the spherical model Ms to the distance to the three-dimensional object closest to the omnidirectional camera 2. That is, when multiple three-dimensional objects are included in the real space captured by the omnidirectional camera 2, the distance from the capturing position of the omnidirectional camera 2 to the three-dimensional object that is closest to the omnidirectional camera 2 among the multiple three-dimensional objects is measured, and three-dimensional object shortest distance information indicating this distance is stored in the storage unit 32 of the image processing device 3. Then, when the radius of the spherical model Ms is adjusted by the model forming unit 33, the radius r' of the spherical model Ms is set based on the three-dimensional object shortest distance information stored in the storage unit 32.

[0047] By adjusting the radius of the spherical model Ms in this way, it is possible to project onto the HMD 4 a background image Imap of the 3D virtual space that provides a natural appearance of three-dimensional objects to the driver, even when driving through open areas such as parking lots in typical urban road scenes. Note that if the radius r' of the spherical model Ms after adjustment is too small, the horizontal plane model Mh corresponding to the ground will be positioned outside the spherical model Ms, which may make it unsuitable as a 3D model for creating the background of the 3D virtual space.

[0048] Although the first to third embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications and changes are possible based on the technical concept of the present invention. For example, in the above-described first to third embodiments, an example in which a UV coordinate system is set for the spherical image I has been shown, but a coordinate system other than the UV coordinate system may be set for the spherical image I.

[0049] Furthermore, in the second embodiment described above, the case where only one vertical planar model Mv corresponding to a three-dimensional object is formed has been described, but two or more vertical planar models Mv may be formed. As a specific example, in a case where three-dimensional objects are arranged only on the left and right sides of a road on which a vehicle travels, such as in a tunnel or an urban area, as shown in Fig. 9, the radius r of the spherical model Ms may be set to an appropriate size, and a vertical planar model Mv' similar to the vertical planar model Mv of the second embodiment described above may be formed on the opposite side of the center C of the spherical model Ms to the vertical planar model Mv, and the celestial sphere image I may be mapped onto the left and right vertical planar models Mv, Mv'. [Explanation of symbols]

[0050] 1. Driving simulator system 2...Spherical camera 3...Image processing device 4...Head-mounted display (HMD) 31...Input section 32...Storage section 33...Model formation section 34...Drawing section 35...Output section C: Center of the spherical model (reference point in the 3D virtual space) D1: Distance from the imaging position to the ground D2: Distance from the imaging position to the three-dimensional object Dh: Distance from the center of the spherical model to the horizontal plane model Dv: Distance from the center of the spherical model to the vertical plane model G...ground I...Spherical image Io: Image of a three-dimensional object Imap: Background image of 3D virtual space Mh...Horizontal plane model Ms...Spherical model Mv, Mv'...Vertical plane model O...Three-dimensional object Ph: The apex of the horizontal plane model Ps: Intersection point between the direction vector and the spherical model Pv: The vertex of the vertical plane model r, r'...radius of the spherical model

Claims

1. a storage unit that stores a spherical image captured by an omnidirectional camera of a real space including the ground and a three-dimensional object; a model forming unit that forms a three-dimensional model configured by combining a plurality of meshes based on features of the celestial sphere image stored in the storage unit; and a rendering unit that forms a background image of the three-dimensional virtual space by mapping the omnidirectional image stored in the storage unit onto the three-dimensional model formed by the model forming unit based on coordinate values ​​of a reference point set in the three-dimensional virtual space, the storage unit is configured to store ground distance information indicating a distance from an imaging position of the omnidirectional camera to the ground, the model forming unit is configured to form a spherical model centered on the reference point in the three-dimensional virtual space, and to form a horizontal plane model corresponding to the ground, the horizontal plane model being disposed downward as viewed from the center of the spherical model and the distance to the center being set based on the ground distance information stored in the storage unit; the rendering unit associates, based on the coordinate value of the reference point in the three-dimensional virtual space, the coordinate value of each vertex on the spherical model formed by the model forming unit with the coordinate value of the celestial sphere image stored in the storage unit, and associates the coordinate value of each vertex on the horizontal planar model formed by the model forming unit with the coordinate value of the celestial sphere image stored in the storage unit; and replaces the coordinate value of the celestial sphere image corresponding to the coordinate value of each vertex on the horizontal planar model with the coordinate value of the celestial sphere image set at each intersection of the spherical model and each direction vector from the center of the spherical model toward the vertex on the horizontal planar model.

2. a UV coordinate value in a UV coordinate system is set for each pixel of the spherical image, 2. The image processing device according to claim 1, wherein the drawing unit is configured to set UV coordinate values ​​of the celestial sphere image corresponding to coordinate values ​​of each vertex in accordance with a positional relationship between the reference point in the three-dimensional virtual space, and each vertex on the spherical model and each vertex on the horizontal planar model, extend straight lines connecting a center of the spherical model and each vertex on the horizontal planar model radially outward, acquire UV coordinate values ​​set at intersections between the straight lines and the spherical model, and replace UV coordinate values ​​of the celestial sphere image corresponding to the coordinate values ​​of each vertex on the horizontal planar model with the acquired UV coordinate values.

3. the storage unit is configured to store three-dimensional object distance information indicating a distance from an imaging position of the omnidirectional camera to the three-dimensional object, the model forming unit is configured to form a vertical-plane model corresponding to the three-dimensional object, the vertical-plane model being arranged in a horizontal direction as viewed from the center of the spherical model, and the distance to the center being set based on the three-dimensional object distance information stored in the storage unit; 2. The image processing device according to claim 1, wherein the rendering unit is configured to associate a coordinate value of each vertex of the vertical planar model formed by the model forming unit with a coordinate value of the celestial sphere image stored in the storage unit, based on a coordinate value of the reference point in the three-dimensional virtual space, and to replace a coordinate value of the celestial sphere image corresponding to the coordinate value of each vertex on the vertical planar model with a coordinate value of the celestial sphere image that is set at an intersection between the spherical model and each direction vector from a center of the spherical model toward each vertex on the vertical planar model.

4. a UV coordinate value in a UV coordinate system is set for each pixel of the spherical image, 4. The image processing device according to claim 3, wherein the rendering unit is configured to set UV coordinate values ​​of the celestial sphere image corresponding to coordinate values ​​of each vertex, in accordance with a positional relationship between the reference point in the three-dimensional virtual space and each vertex on the spherical model, each vertex on the horizontal planar model, and each vertex on the vertical planar model, extend straight lines connecting a center of the spherical model to each vertex on the horizontal planar model and each vertex on the vertical planar model radially outward, acquire UV coordinate values ​​set at intersections between the straight lines and the spherical model, and replace UV coordinate values ​​of the celestial sphere image corresponding to the coordinate values ​​of each vertex on the horizontal planar model and each vertex on the vertical planar model with the acquired UV coordinate values.

5. 4. The image processing device according to claim 3, wherein when at least a portion of an image corresponding to the three-dimensional object in the background image formed by the drawing unit is mapped to the horizontal plane model, the model forming unit adjusts a radius of the spherical model based on the three-dimensional object distance information stored in the memory unit, so that the entire image corresponding to the three-dimensional object is mapped to the spherical model.

6. 6. The image processing device according to claim 5, wherein the model forming unit is configured to adjust the radius of the spherical model based on the width of a road on which a vehicle travels.

7. the storage unit is configured to store, when the omnidirectional image includes images of a plurality of three-dimensional objects, three-dimensional object shortest distance information indicating a distance from an imaging position of the omnidirectional camera to a three-dimensional object that is closest to the omnidirectional camera among the plurality of three-dimensional objects; 6. The image processing device according to claim 5, wherein the model forming unit is configured to adjust a radius of the spherical model based on the three-dimensional object shortest distance information stored in the storage unit.

Images