Image generation device, image generation method, and program

Abstract

To effectively utilize image data acquired by an imaging apparatus.SOLUTION: An image generation device comprises a data acquisition unit which acquires a plurality of pieces of image data having a parallax, a first image generation unit which generates a plurality of three-dimensional images of a plurality of specific areas based on the plurality of pieces of image data, and a second image generation unit which generates a space swimming image corresponding to a specific movement locus by using data of the plurality of three-dimensional images.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image generating device, an image generating method, and a program. [Background technology]

[0002] In recent years, three-dimensional measurement using a stereo camera has been attracting attention. A stereo camera generates a three-dimensional image based on an image captured by one camera and an image captured by the other camera. That is, pixel blocks that are correlated with pixel blocks in an image captured by one camera are identified in an image captured by the other camera, and depth information is calculated using a photogrammetry technique from the parallax, which is the relative amount of deviation between the two images. For example, an example of a three-dimensional measurement technique is described in Patent Document 1 below. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2001-285692 A Summary of the Invention [Problem to be solved by the invention]

[0004] A stereo camera generates a 3D image from multiple different images with parallax. In the future, new and effective uses of 3D images generated from multiple images with parallax are desired.

[0005] The present invention has been made in view of the above, and has an object to make effective use of image data acquired by an imaging device. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the objective, the image generating device of the present invention comprises a data acquisition unit that acquires multiple image data having parallax, a first image generating unit that generates multiple three-dimensional images for multiple specific regions based on the multiple image data, and a second image generating unit that generates a space-floating image according to a specific movement trajectory using data of the multiple three-dimensional images.

[0007] The image generating method of the present invention includes the steps of acquiring multiple image data having parallax, generating multiple three-dimensional images for multiple specific regions based on the multiple image data, and generating a spatial swimming image corresponding to a specific movement trajectory using the data of the multiple three-dimensional images.

[0008] The program of the present invention causes a computer operating as an image generating device to execute the following steps: acquiring multiple image data having parallax; generating multiple 3D images for multiple specific areas based on the multiple image data; and generating a spatial swimming image corresponding to a specific movement trajectory using the data of the multiple 3D images. Effect of the Invention

[0009] According to the present invention, it is possible to effectively utilize image data acquired by an image capturing device. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of a display system according to this embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing a camera device. [Diagram 3] FIG. 3 is a schematic diagram for explaining the concept of the space swimming image. [Figure 4] FIG. 4 is a schematic diagram showing a method for generating a three-dimensional image. [Diagram 5] FIG. 5 is a flow chart illustrating the image generation method. [Figure 6]FIG. 6 is a block diagram showing a specific configuration of the first image generating unit. [Figure 7] FIG. 7 is an explanatory diagram showing the positional relationship between two images to which the photogrammetry principle is applied. [Figure 8] FIG. 8 is an explanatory diagram showing the positional relationship between the two images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An image generating apparatus, an image generating method, and a program according to the present invention will be described in detail below with reference to the accompanying drawings. Note that the present invention is not limited to the following embodiments.

[0012] <Display system configuration> FIG. 1 is a conceptual diagram showing an imaging system according to the present embodiment.

[0013] As shown in Fig. 1, the display system 10 acquires a plurality of image data of the surroundings from a plurality of cameras arranged at different positions. The plurality of image data are a plurality of image data having parallax. The display system 10 generates a 3D image for each specific region based on the plurality of image data having parallax. In this case, by generating one 3D image for one specific region among the surrounding regions, a plurality of 3D images are generated for the surrounding regions.

[0014] When a user instructs the display system 10 to display an image of a desired movement trajectory floating in a specific space, the display system 10 generates and displays a space swimming image according to the instructed movement trajectory using data of multiple three-dimensional images.

[0015] The display system 10 includes a camera device 11, an image generating device 12, a storage unit 13, an operation unit 14, a display control unit 15, and a display unit 16.

[0016] The camera device 11 is configured by mounting a plurality of cameras 21, 22, 23, 24, 25, 26... on the outer periphery of a regular polyhedron (Platonic solid). The regular polyhedron is, for example, a regular tetrahedron, a regular hexahedron, a regular octahedron, a regular dodecahedron, a regular icosahedron, etc. In this embodiment, the regular polyhedron will be described as a regular dodecahedron. The cameras 21, 22, 23, 24, 25, 26... are mounted on each face of a regular pentagon in the regular icosahedron. Details of the camera device 11 will be described later.

[0017] The image generating device 12 is connected to the camera device 11 (cameras 21, 22, 23, 24, 25, 26, etc.). A plurality of image data captured by the camera device 11 is input to the image generating device 12. The image generating device 12 generates a three-dimensional image based on the plurality of input image data. The image generating device 12 generates a space swimming image according to a specific movement trajectory based on the generated three-dimensional image. Details of the image generating device 12 will be described later. The image generating device 12 is composed of an arithmetic circuit such as a CPU (Central Processing Unit).

[0018] The storage unit 13 is connected to the image generating device 12. The storage unit 13 stores data of three-dimensional images generated by the image generating device 12. As will be described later, the image generating device 12 divides a space into a plurality of regions and generates a three-dimensional image for each of the plurality of regions. Therefore, the storage unit 13 stores data of the plurality of three-dimensional images generated by the image generating device 12. The storage unit 13 is, for example, an external storage device such as an HDD (Hard Disk Drive) or a memory.

[0019] The operation unit 14 is connected to the image generating device 12. The operation unit 14 can be operated by a user. The operation unit 14 can output a command signal to the image generating device 12. The operation unit 14 can output a command signal of an image to be displayed on the display unit 16 to the image generating device 12. Specifically, the user uses the operation unit 14 to input movement trajectory data for movement in a specified space. The image generating device 12 then generates a space swimming image according to the movement trajectory data input from the operation unit 14.

[0020] Furthermore, the operation unit 14 can start and end the display on the display unit 16. The operation unit 14 can switch the display content on the display unit 16. The operation unit 14 is a keyboard, a mouse, a touch panel, a button, a switch, or the like.

[0021] The display control unit 15 is connected to the image generation device 12. The display control unit 15 controls the display of the space swim image generated by the image generation device 12. In other words, the display control unit 15 displays the space swim image generated by the image generation device 12 on the display unit 16. The display control unit 15 is configured by an arithmetic circuit such as a CPU (Central Processing Unit), for example.

[0022] The display unit 16 is connected to the display control unit 15. The display unit 16 can be controlled by the display control unit 15. The display unit 16 displays a space swimming image input from the display control unit 15. The display unit 16 is, for example, a display. However, the display unit 16 is not limited to this configuration, and may be, for example, a touch panel type display in which the operation unit 14 and the display unit 16 are integrated.

[0023] <Image Generation Device> The image generating device 12 includes a data acquiring section 31 , a first image generating section 32 , and a second image generating section 33 .

[0024] The data acquisition unit 31 acquires a plurality of image data acquired by the plurality of cameras 21, 22, 23, 24, 25, 26,... that constitute the camera device 11. The plurality of cameras 21, 22, 23, 24, 25, 26,... are capable of capturing images of the entire surroundings in a specific space. Therefore, the data acquisition unit 31 acquires a plurality of image data from the plurality of cameras 21, 22, 23, 24, 25, 26,... to acquire image data of the entire surroundings in a specific space.

[0025] The first image generating unit 32 generates a three-dimensional image (three-dimensional model) based on a plurality of image data acquired by the data acquiring unit 31. In the camera device 11, among the plurality of cameras 21, 22, 23, 24, 25, 26, etc., two adjacent cameras have parallax due to different positions, and a part of the shooting field of view overlaps. For example, the two adjacent cameras 21 and 22 have parallax because the positions at which they are placed in the camera device 11 are different. The first image generating unit 32 generates a three-dimensional image (three-dimensional model) based on two image data (stereo images) having parallax captured by the cameras 21 and 22. In this case, a part of the shooting field of view of the camera 21 and a part of the shooting field of view of the camera 22 overlap. The first image generating unit 32 also generates a three-dimensional image (three-dimensional model) based on three image data (stereo images) having parallax captured by the cameras 21 and 22 as well as the camera 23 adjacent to the cameras 21 and 22.

[0026] Here, a 3D image is a three-dimensional (3-dimensional) digital data with information on length, width, and height arranged according to a specific rule. Parallax is the difference in the direction in which a target point is seen due to the difference in the positions of two observation points, or the difference in angle.

[0027] The three-dimensional image generated by the first image generating unit 32 is a three-dimensional image in a specific area. That is, the cameras 21, 22 (23) have a set field of view. Then, the image data acquired by the first image generating unit 32 from the cameras 21, 22 (23) becomes image data in the field of view of the cameras 21, 22 (23). Therefore, the three-dimensional image generated by the first image generating unit 32 is a three-dimensional image in the area of ​​the field of view of the cameras 21, 22 (23).

[0028] The first image generating unit 32 generates one 3D image of an area within the field of view of the multiple cameras based on multiple image data captured by two or more adjacent cameras among the multiple cameras 21, 22, 23, 24, 25, 26, etc. The first image generating unit 32 changes the combination of multiple adjacent cameras for a specific space, and generates one 3D image of an area within the field of view of the multiple cameras based on multiple image data captured by a different combination of multiple cameras.

[0029] The first image generating unit 32 generates one 3D image based on multiple image data for one area (field of view of multiple cameras). As a result, the first image generating unit 32 generates multiple 3D images for multiple specific areas based on multiple image data acquired from multiple cameras 21, 22, 23, 24, 25, 26, etc.

[0030] The second image generating unit 33 generates a space swimming image according to a specific movement trajectory using data of a plurality of three-dimensional images generated by the first image generating unit 32. The specific movement trajectory is input to the second image generating unit 33 by the user using the operation unit 14. Here, the movement trajectory is composed of a plurality of consecutive point data each consisting of spatial coordinates in a specific space and a movement direction in the spatial coordinates.

[0031] The three-dimensional image generated by the first image generating unit 32 is generated based on a plurality of image data captured from the positions of the camera device 11 (cameras 21, 22, 23, 24, 25, 26, etc.). On the other hand, the spatial coordinates of the point data may be any point within the field of view of the camera device 11. The moving direction of the point data may be a direction different from the shooting direction of the camera device 11. The second image generating unit 33 generates a two-dimensional image of the space according to the point data by performing perspective transformation on the three-dimensional image generated by the first image generating unit 32. The perspective transformation is a transformation from a three-dimensional normal coordinate system to a two-dimensional perspective coordinate system. In this case, it is necessary to set the viewpoint of the two-dimensional image, and the second image generating unit 33 performs perspective transformation on the three-dimensional image of the space visible from the position of the camera device 11, and sets the viewpoint to the position (point data) designated by the user.

[0032] Since the movement trajectory is made up of a plurality of consecutive point data, the second image generating section 33 generates a space wandering image by connecting two-dimensional images of the plurality of point data.

[0033] The first image generating unit 32 and the second image generating unit 33 may be the same hardware, and do not have to exist separately as hardware. As described above, the image generating device 12 is configured by an arithmetic circuit such as a CPU, and the first image generating unit 32 and the second image generating unit 33 may also be configured by an arithmetic circuit such as a CPU.

[0034] The image generating device 12 may have an image interpolation unit. The image interpolation unit performs image data interpolation for a two-dimensional image corresponding to the location data generated by the second image generating unit 33. The image interpolation unit estimates and interpolates an image of an area that the data acquiring unit 31 could not acquire when the second image generating unit 33 generates a two-dimensional image of the space. Here, the interpolation means to compensate for an occlusion part. Occlusion is related to the front-back direction as well as the up-down and left-right directions, and refers to an area with missing depth information that occurs when an object in front hides an object behind it and cannot be seen.

[0035] That is, since the position of the camera device 11 differs from the position of the spot data, the area visible from the camera device 11 differs from the area of ​​the spot data. Therefore, when the second image generating unit 33 generates a two-dimensional image, an image of the area in the spot data may be insufficient, and occlusion may occur. Therefore, the image interpolating unit adds the missing image in the area in the spot data to eliminate the occlusion.

[0036] The image interpolation unit adds an image of the missing area by, for example, extending lines or colors of an image (texture) adjacent to the missing area. The image interpolation unit also photographs the missing area by, for example, moving the camera device 11 or arranging a plurality of camera devices 11, and adds the image of the missing area from the photographed image. The image interpolation unit also adds an image of the missing area by, for example, recognizing the overall shape and color of the space and predicting the image of the missing area. The image interpolation unit also adds an image of the missing area by, for example, machine learning.

[0037] <Camera equipment> FIG. 2 is a schematic diagram showing a camera device.

[0038] The camera device 11 has a regular dodecahedron support 41, cameras 21, 22, 23, 24, 25, 26, . . ., a support rod 42, and a support base 43.

[0039] The support 41 has 12 faces 41a, 41b, 41c, 41d, 41e, 41f... forming a regular pentagon. The cameras 21, 22, 23, 24, 25, 26... are attached to the faces 41a, 41b, 41c, 41d, 41e, 41f... of the support 41. The center lines of the photographing fields of the cameras 21, 22, 23, 24, 25, 26... are set outward in the normal direction of the faces 41a, 41b, 41c, 41d, 41e, 41f... In addition, it is preferable that the cameras 21, 22, 23, 24, 25, 26... have a photographing field of view of 180 degrees around, that is, 90 degrees with respect to the normal line.

[0040] A support base 43 is connected to the lower part of the support body 41 via a support rod 42. The camera device 11 is placed in a specific space by installing the support base 43 on a specific floor surface.

[0041] <Spatial swimming image> FIG. 3 is a schematic diagram for explaining the concept of the space wandering image, and FIG. 4 is a schematic diagram showing a method of generating a three-dimensional image.

[0042] 3 and 4, the camera device 11 is placed in a specific space 51. The specific space 51 may be, for example, a room in a building or a specific location outdoors. The camera device 11 has multiple cameras 21, 22, 23, 24, 25, 26, etc. capable of capturing images of the entire periphery (360 degrees) of the space 51. At this time, the fields of view of the adjacent cameras 21, 22, 23, 24, 25, 26, etc. overlap partially.

[0043] For example, two adjacent cameras 21 and 22 acquire image data 21a and 22a having a predetermined field of view, respectively. The cameras 21 and 22 have a parallax (angle difference) α1, and their fields of view overlap. That is, the image data 21a and 22a are image data in which a part of the image data overlaps. Also, for example, when a target B is present within the field of view of the cameras 21 and 22, the image data 21a and 22a include at least image data of a shooting range (angle) β1 for the target B. Therefore, the first image generating unit 32 can generate a three-dimensional image in the shooting range β1 including the target B based on the image data 21a and 22a having parallax captured by the cameras 21 and 22. It is desirable that the first image generating unit 32 generates a three-dimensional image in the shooting range β1 including the target B using image data of all cameras capable of capturing the target B.

[0044] Similarly, the first image generation unit 32 generates a plurality of three-dimensional images in a shooting range including targets A, C, D, E, F, and G based on a plurality of image data having parallax captured by the plurality of cameras 21, 22, 23, 24, 25, 26, etc. That is, the first image generation unit 32 generates a plurality of three-dimensional images corresponding to a plurality of regions including targets A, B, C, D, E, F, and G for a specific space 51. In this case, it is preferable that the plurality of three-dimensional images corresponding to the plurality of regions generated by the first image generation unit 32 have some overlapping images.

[0045] The user inputs a specific movement trajectory to the second image generating unit 33 using the operation unit 14. The movement trajectory is, for example, a series of point data Aa, Ba, Ca, Da, Ea, Fa, Ga, each of which is composed of spatial coordinates (X, Y, Z) at targets A, B, C, D, E, F, G and a movement direction at the spatial coordinates (X, Y, Z). That is, the movement trajectory is a trajectory in which the point data Aa, Ba, Ca, Da, Ea, Fa, Ga are continuous. The second image generating unit 33 generates a space swimming image according to the movement trajectory in which the point data Aa, Ba, Ca, Da, Ea, Fa, Ga are continuous. The space swimming image is an image in which two-dimensional images at the point data Aa, Ba, Ca, Da, Ea, Fa, Ga are continuous. The movement trajectory is not limited to a trajectory in which the point data Aa, Ba, Ca, Da, Ea, Fa, and Ga are successive, and can be appropriately set by the user using the operation unit 14.

[0046] In the above description, the specific space 51 is viewed from one side, but the space 51 is a three-dimensional space.

[0047] That is, the first image generating unit 32 generates a plurality of three-dimensional images corresponding to a plurality of areas including targets A, B, C, D, E, F, and G in a specific space 51. The second image generating unit 33 generates a space swimming image according to point data Aa, Ba, Ca, Da, Ea, Fa, and Ga consisting of spatial coordinates (X, Y, Z) and a moving direction as a moving trajectory instructed by the user, based on the plurality of three-dimensional images generated by the first image generating unit 32.

[0048] In this case, the movement trajectory instructed by the user may be specified in advance, or the spatial swimming image may be automatically generated when the object specified in advance is recognized. Then, the storage unit 13 stores information linking the object with the movement trajectory in advance. Next, the first image generating unit 32 acquires the information linking the object with the movement trajectory from the storage unit 13. The first image generating unit 32 judges whether or not the object of the information linking the object with the movement trajectory is included in the multiple image data captured by the multiple cameras 21, 22, 23, 24, 25, 26, etc. In order to judge whether or not the object is included, various known techniques such as image recognition by pattern matching may be used. When the first image generating unit 32 judges that the object is included in the image data, the movement trajectory of the information linking the object with the movement trajectory is set as the movement trajectory instructed by the user. The first image generating unit 32 may determine that an object is captured when a predetermined number of image data contain the object, or may determine that an object is captured when all of the multiple image data contain the object. The spatial coordinates (X, Y, Z) may be the midpoint or center of gravity of the object from which multiple images are derived. The movement direction is not limited to a circular movement centered on the camera device 11, and may be set appropriately. If an object or a surrounding object is on the movement trajectory, the movement trajectory may be corrected to avoid the object or the surrounding object.

[0049] Also, the movement trajectory instructed by the user may be changed as necessary. The objects (target objects A, B, C, D, E, F, G) are people or objects (furniture, etc.). In this case, the position to be viewed differs for each object, so the spatial swimming image is made different. For example, when the object is determined to be an object by image recognition, it is preferable that the movement trajectory is a circular movement diagonal to the horizontal centered on the object. Also, when the object is determined to be a person by image recognition, it is preferable that the movement trajectory is a horizontal circular movement centered on the object. Specifically, the storage unit 13 stores information in advance that links the object and the movement trajectory, with the movement trajectory being different for each object.

[0050] The movement trajectory may be changed according to the spatial coordinates of the object. In this case, the angle is different between an object at a high position and an object at a low position. Therefore, by changing the height of the movement trajectory according to the spatial coordinates of the object, a natural space-floating image can be generated. In this case, the storage unit 13 stores a threshold value of the spatial coordinates in advance. The first image generating unit 32 acquires the threshold value of the spatial coordinates from the storage unit 13. When the first image generating unit 32 determines that the object is included in the image data, it determines whether or not the spatial coordinates (X, Y, Z) based on the object and the point data Aa, Ba, Ca, Da, Ea, Fa, Ga, etc., consisting of the movement direction of the movement trajectory of the information linking the object and the movement trajectory, are within the threshold value of the spatial coordinates. When the first image generating unit 32 determines that the spatial coordinates are not within the threshold value of the spatial coordinates, it generates a movement trajectory changed to be within the threshold value of the spatial coordinates, and sets it as the movement trajectory instructed by the user.

[0051] <Image generation method> FIG. 5 is a flow chart illustrating the image generation method.

[0052] 1, 3, and 5, in step S11, the data acquisition unit 31 acquires a plurality of image data acquired by the plurality of cameras 21, 22, 23, 24, 25, 26, etc. in the camera device 11. In step S12, the first image generation unit 32 generates a three-dimensional image based on the plurality of image data acquired by the data acquisition unit 31. The three-dimensional images generated by the first image generation unit 52 are, for example, a plurality of three-dimensional images corresponding to a plurality of regions each including targets A, B, C, D, E, F, and G for a specific space 51.

[0053] In step S13, the multiple 3D images generated by the first image generation unit 32 are stored in the memory unit 13. The multiple 3D images stored in the memory unit 13 are, for example, multiple 3D images corresponding to multiple areas each including targets A, B, C, D, E, F, and G for a specific space 51. In step S14, the second image generation unit 33 determines whether or not a specific movement trajectory has been input from the operation unit 14. Here, if the second image generation unit 33 determines that a specific movement trajectory has not been input from the operation unit 14 (No), it waits as is.

[0054] On the other hand, when the second image generating unit 33 determines that a specific movement trajectory has been input from the operation unit 14 (Yes), the process proceeds to step S15. In step S15, the second image generating unit 33 performs perspective transformation (viewpoint transformation) according to the movement trajectory (point data Aa, Ba, Ca, Da, Ea, Fa, Ga) on the multiple 3D images generated by the first image generating unit 32, thereby generating multiple 2D images according to the point data Aa, Ba, Ca, Da, Ea, Fa, Ga. Note that when the first image generating unit generates or corrects the movement trajectory, it may be determined that a specific movement trajectory has been input from the operation unit 14.

[0055] In step S16, the second image generation unit 33 generates a space wandering image by connecting multiple two-dimensional images corresponding to the location data Aa, Ba, Ca, Da, Ea, Fa, and Ga. Then, in step S17, the second image generation unit 33 outputs the generated space wandering image to the display control unit 15, and the display control unit 15 displays the space wandering image on the display unit 16.

[0056] If it is determined in step S14 that a different movement trajectory has been input, then a plurality of two-dimensional images corresponding to the newly input movement trajectory are generated, and a space swimming image generated by connecting the plurality of two-dimensional images is displayed.

[0057] <Example of the direction of spatial swimming image generation> FIG. 6 is a block diagram showing the specific configuration of the first image generation unit, FIG. 7 is an explanatory diagram showing the positional relationship between two images to which the photogrammetry principle is applied, and FIG. 8 is an explanatory diagram showing the positional relationship between the two images.

[0058] As shown in FIG. 6, the first image generating unit 32 has an epipolar line direction calculator 61 , an epipolar line orthogonal direction calculator 62 , a search range determiner 63 , a corresponding point detector 64 , and a distance calculator 65 .

[0059] Based on a plurality of (e.g., two) pieces of image data acquired by the data acquisition unit 31, the epipolar line direction calculator 61 calculates the direction of an epipolar line connecting corresponding pixel points of the plurality of image data for a region including the target A. The epipolar line direction calculator 61 sends the calculated direction of the epipolar line to the epipolar line orthogonal direction calculator 62.

[0060] The epipolar line orthogonal direction calculator 62 calculates an orthogonal direction perpendicular to the epipolar line. The orthogonal direction relative to the epipolar line calculated by the epipolar line orthogonal direction calculator 62 is output to the search range determiner 63.

[0061] The search range determiner 63 determines a two-dimensional search range on the screen so as to include a plurality of pixel points corresponding to each other in the direction of the epipolar line and in the direction perpendicular to the epipolar line. The search range determiner 63 outputs the determined two-dimensional search range to the corresponding point detector 64.

[0062] The corresponding point detector 64 performs a corresponding point search based on the multiple image data acquired by the data acquisition unit 31 and the determined two-dimensional search range to obtain a disparity vector. The corresponding point detector 64 sends the obtained disparity vector to the distance calculator 65.

[0063] The distance calculator 65 maps the disparity vector onto an epipolar line to obtain the epipolar line direction component of the disparity vector, and calculates the distance to the object T based on the obtained epipolar line direction component. The distance calculator 65 sends the calculated distance to the object T to the second image generator 53.

[0064] A specific description will be given below of a method for generating a three-dimensional image (three-dimensional model) by the first image generating unit 32. Here, a case where a three-dimensional image is generated from two pieces of image data will be described.

[0065] The data acquisition unit 31 acquires a pair of image data obtained by capturing an area including the target A from the cameras 21 and 22. A 3D point cloud is obtained by triangulating this pair of image data, and the relative positions and the image of the point cloud are expanded by collecting these points. In addition, the texture of the part corresponding to this image data is associated with the image data and managed in memory for later mapping.

[0066] First, two sets of image data are obtained for the target A by the cameras 21 and 22. Next, the corresponding point detector 64 searches for corresponding points of the feature points based on the two sets of image data. The corresponding point detector 64, for example, performs correspondence for each pixel and searches for the position where the difference is minimum. Here, as shown in Figs. 6 and 7, the second cameras 26 and 27, which are assumed to exist simultaneously at two viewpoints, are assumed to be disposed in a relationship of Yl=Yr so that the optical axes Ol and Or are included on the same XZ coordinate plane. Using the corresponding points searched for by the corresponding point detector 64, a disparity vector corresponding to the angle difference for each pixel is calculated.

[0067] Since the obtained disparity vector corresponds to the distance from the cameras 21, 22 in the depth direction, the distance calculator 65 calculates the distance in proportion to the magnitude of the disparity by the perspective law. If it is assumed that the cameras 21, 22 move only substantially horizontally, the cameras 21, 22 are arranged so that their optical axes Ol, Or are included in the same XZ coordinate plane, and the search for corresponding points can be performed only on the scanning lines that are the epipolar lines Epl, Epr. The distance calculator 65 generates a three-dimensional image of the object T using the two image data of the object T and the respective distances from the cameras 21, 22 to the object T.

[0068] On the other hand, when a point Ql (Xl, Yl) on the left image corresponds to a point Qr (Xr, Yr) on the right image, the disparity vector at the point Ql (Xl, Yl) is Vp (Xl-Xr, Yl-Yr). Here, since the two points Ql and Qr are on the same scanning line (epipolar line), Yl=Yr, and the disparity vector is expressed as Vp (Xl-Xr, 0). The epipolar line direction calculator 61 obtains such a disparity vector Vp for all pixel points on the image, and creates a group of disparity vectors to obtain information on the depth direction of the image. Incidentally, for a set in which the epipolar line is not horizontal, the height of one of the camera positions may be different (with a low probability). In this case, the epipolar line orthogonal direction calculator 62 searches within a rectangle in the epipolar line direction and in a direction orthogonal to the epipolar line, which is about the deviation from the horizontal, in a large search range, compared to a case where a corresponding point search is performed in a large two-dimensional area without considering corresponding point matching, so that the amount of calculation for the minimum rectangle is reduced and becomes rational. Then, as shown in FIG. 8, the search range determiner 63 shows a search range in which the epipolar line direction search range for the minimum rectangle is a~b=c~d, and the orthogonal direction search range is b~c=d~a. In this case, the search width in the epipolar line direction is ΔE, and the search width in the direction F orthogonal to the epipolar line is ΔF. The smallest non-inclined rectangle ABCD that includes the minimum inclined rectangle abcd is the area to be sought.

[0069] As shown in Fig. 6, the first image generating unit 32 obtains a parallax vector from corresponding points of the feature points of the multiple cameras 21 and 22 under the epipolar constraint condition, obtains information on the depth direction of each point, maps the texture on the surface of the three-dimensional shape, and generates a three-dimensional image. As a result, the model of the part in the image data used for calculation can reproduce the spatial image seen from the front hemisphere, that is, the three-dimensional image seen from the camera device 11. The first image generating unit 32 sends the generated three-dimensional image to the second image generating unit 33.

[0070] The second image generating unit 33 can convert and reconstruct a three-dimensional image into a two-dimensional image corresponding to the point data by using a method similar to that of the first image generating unit 32, using a transparent conversion corresponding to the point data of the input movement trajectory. The second image generating unit 33 can generate a space swimming image corresponding to the movement trajectory by connecting multiple generated two-dimensional images.

[0071] In the above description, the camera device 11 is arranged at the center of the specific space 51, but the arrangement is not limited to this. For example, the camera device 11 may be arranged around an object located in the specific space. In this case, it is preferable to arrange a plurality of camera devices 11 around the object.

[0072] [Effects of this embodiment] The image generating device of this embodiment includes a data acquisition unit 31 that acquires multiple image data having parallax, a first image generating unit 32 that generates multiple three-dimensional images for multiple specific areas based on the multiple image data, and a second image generating unit 33 that generates a spatial swimming image according to a specific movement trajectory using the data of the multiple three-dimensional images.

[0073] Therefore, multiple 3D images are generated for multiple specific regions based on multiple image data with parallax, and a space wandering image according to a specific movement trajectory is generated using the data of the multiple 3D images. Therefore, images of the space as seen from a place where the user is not present can be viewed continuously, and image data acquired by the camera device (photographing device) 11 can be effectively used.

[0074] In the image generating device of this embodiment, the movement trajectory is composed of multiple consecutive point data each consisting of a movement direction at a specific spatial coordinate, and the second image generating unit 33 generates a space swimming image by connecting two-dimensional images at the multiple point data. Therefore, the user can easily see the space swimming image as seen from a place where the user is not present.

[0075] In the image generating device of this embodiment, the first image generating unit 32 judges whether or not a pre-specified object is captured in a plurality of image data, and if so, sets the movement trajectory associated with the object as the specific movement trajectory. Therefore, the user can easily see the swimming image of the space as seen from a place where the user is not present.

[0076] The image generating device according to the present invention has been described above, but the image generating device may be embodied in various different forms other than the above-described embodiment.

[0077] Each component of the illustrated image generating device is a functional concept, and does not necessarily have to be physically configured as illustrated. In other words, the specific form of each device is not limited to that shown in the figure, and all or part of the device may be functionally or physically distributed or integrated in any unit depending on the processing load and usage status of each device.

[0078] The configuration of the image generating device is realized, for example, as software, by a program loaded into a memory. In the above embodiment, the configuration has been described as a functional block realized by cooperation of the hardware and software. In other words, the functional block can be realized in various forms by only hardware, only software, or a combination of both.

[0079] The above-mentioned components include those that a person skilled in the art can easily imagine and those that are substantially the same. Furthermore, the above-mentioned configurations can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the configurations are possible without departing from the scope of the present invention. [Explanation of symbols]

[0080] 10 Display System 11 Camera equipment 12 Image generation device 13 Storage section 14 Control section 15 Display control section 16 Display section 21,22,23,24,25 Camera 31 Data Acquisition Section 32 First image generating unit 33 Second image generating unit 41 Support 42 Support rod 43 Support stand 51 Space 61 Epipolar line direction calculator 62 Epipolar Line Orthogonal Direction Calculator 63 Search Range Decider 64 Correspondence Detector 65 Distance Calculator A,B,C,D,E,F,G Target

Claims

1. a data acquisition unit that acquires a plurality of image data having parallax; a first image generating unit that generates a plurality of three-dimensional images for each of a plurality of specific regions based on the plurality of image data; a second image generation unit that generates a space swimming image according to a specific movement trajectory using data of the plurality of three-dimensional images; Equipped with the first image generation unit determines whether or not a previously identified object is captured in the plurality of image data, and if an object is captured as the identified object, determines that the specific movement trajectory is a circular movement diagonal to the horizontal centered on the object, and if a person is captured as the identified object, determines that the specific movement trajectory is a circular movement diagonal to the horizontal centered on the person. Image generating device.

2. the movement trajectory is configured by a plurality of successive point data each of which includes a movement direction at a specific spatial coordinate; The second image generation unit generates the space swimming image by connecting two-dimensional images of the plurality of point data. The image generating device of claim 1 .

3. When the object is on the specific movement trajectory, the first image generation unit corrects the movement trajectory to avoid the object.

3. The image generating device according to claim 1 or 2.

4. acquiring a plurality of image data having parallax; generating a plurality of three-dimensional images for each of a plurality of specific regions based on the plurality of image data; generating a space wandering image according to a specific movement trajectory using data of the plurality of three-dimensional images; determining whether or not a previously identified object is captured in the plurality of image data, and if an object is captured as the identified object, determining a circular motion diagonal to the horizontal centered on the object as the specific movement trajectory, and if a person is captured as the identified object, determining a circular motion diagonal to the horizontal centered on the person as the specific movement trajectory; An image generation method comprising:

5. acquiring a plurality of image data having parallax; generating a plurality of three-dimensional images for each of a plurality of specific regions based on the plurality of image data; generating a space wandering image according to a specific movement trajectory using data of the plurality of three-dimensional images; determining whether or not a previously identified object is captured in the plurality of image data, and if an object is captured as the identified object, determining a circular motion diagonal to the horizontal centered on the object as the specific movement trajectory, and if a person is captured as the identified object, determining a circular motion diagonal to the horizontal centered on the person as the specific movement trajectory; A program that causes a computer operating as an image generating device to execute the above.

Images