Image generating device and method for controlling image generating device

Abstract

To provide an image generating device capable of reducing a sense of incongruity even when 3D data having a deficit in a part of a subject.SOLUTION: An image generating device has: 3D data acquisition means for acquiring 3D data; deficient area acquisition means for acquiring a deficient area of the 3D data; virtual camera movable range information acquisition means for acquiring a movable range of a virtual camera; and mask data generation means, wherein the mask data generation means generates mask data in a space between the virtual camera and the deficient area.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image generating method and an image generating device, and more particularly to a three-dimensional image generating device and a three-dimensional image generating method. [Background technology]

[0002] Conventionally, there are known technologies capable of acquiring distance distribution information, such as stereo cameras and light field cameras. A point cloud can be obtained by performing perspective projection transformation on the distance distribution information. Furthermore, it is possible to generate a 3D surface model having a surface by converting the point cloud into polygons.

[0003] By acquiring images (color distribution information) at the same time as distance distribution information, it is possible to generate 3D data with texture information. Unlike ordinary 2D images, 3D data has the advantage that it can be enjoyed from any viewpoint.

[0004] For example, Patent Document 1 discloses a method for generating a virtual viewpoint image in which the position of the virtual viewpoint and the position of the focus are changed based on depth information calculated according to the principle of a light field camera. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2013-165475 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, 3D data created using a stereo camera or light field camera may not fit the entire subject in the field of view, resulting in a cut-off subject. If the subject is cut off in this way, the 3D data will be missing parts, and when it is combined with a 3D background to enhance the sense of immersion, it can create an awkward feeling, for example, as if the subject is floating in the air.

[0007] Therefore, an object of the present invention is to provide an image generating device that can reduce the sense of incongruity even when 3D data is used that includes a portion of a subject that is missing. [Means for solving the problem]

[0008] In order to achieve the above object, the image generating device of the present invention has a first acquisition means for acquiring 3D data and a missing area of ​​the 3D data, a second movable range acquisition means for acquiring a movable range of a virtual camera, and a generation means for generating mask data, wherein the generation means generates the mask data based on the movable range of the virtual camera and the missing area so that the missing area is not visible within the movable range of the virtual camera.

[0009] In addition, the control method for the image generating device of the present invention includes a first acquisition step of acquiring 3D data and a missing area of ​​the 3D data, a second movable range acquisition step of acquiring a movable range of a virtual camera, and a generation step of generating mask data, wherein the generation step generates the mask data based on the movable range of the virtual camera and the missing area so that the missing area is not visible within the movable range of the virtual camera. Effect of the Invention

[0010] According to the present invention, since appropriate three-dimensional mask data can be generated for missing areas of 3D data, it is possible to provide 3D data that gives a less unnatural feeling. [Brief description of the drawings]

[0011] [Figure 1]FIG. 1 is a diagram illustrating a hardware configuration of an image generating device 100 according to an embodiment of the present invention. [Diagram 2] FIG. 4 is a diagram illustrating a hardware configuration of an image capturing apparatus 400 according to an embodiment of the present invention. [Diagram 3] FIG. 1 is a diagram for explaining an image capturing unit according to an embodiment of the present invention; [Figure 4] FIG. 1 is a diagram illustrating 3D data and a virtual camera viewpoint according to an embodiment of the present invention. [Diagram 5] 1 is a flowchart illustrating a generation process executed by the image generation device 100 according to the first embodiment. [Figure 6] FIG. 1 is a diagram showing the positional relationship between 3D data and a virtual camera in the first embodiment. [Figure 7] 1 is a flowchart illustrating a method for acquiring a missing area according to a first embodiment. [Figure 8] FIG. 2 is a diagram illustrating a missing area of ​​model data and 3D data in the first embodiment. [Figure 9] 1 is a flowchart illustrating a mask data generating method according to a first embodiment. [Figure 10] FIG. 1 is a diagram illustrating an intersection region P in the first embodiment. [Figure 11] FIG. 1 is a diagram for explaining the effects of the first embodiment. [Figure 12] FIG. 1 is a diagram illustrating a method for reducing a search range in the mask data generation method according to the first embodiment. [Figure 13] FIG. 2 is a diagram illustrating a method for reducing a search range in the mask data generation method according to the first embodiment. [Figure 14] 11 is a flowchart illustrating a generation process executed by the image generating device 100 according to a modified example. [Figure 15] 11 is a flowchart illustrating a method for acquiring a missing area according to a modified example. [Figure 16] 11 is a flowchart illustrating a mask data generating method according to a modified example. [Figure 17] FIG. 13 is a diagram illustrating a hardware configuration of an image generating device 100 according to a second embodiment. [Figure 18]11 is a flowchart illustrating a generation process executed by the image generating device 100 according to the second embodiment. [Figure 19] FIG. 13 is a diagram showing an example of 3D data having a missing portion and 3D data to be combined with the 3D data in the second embodiment; [Figure 20] FIG. 13 is a diagram illustrating a shielding region in the second embodiment. [Figure 21] FIG. 13 is a diagram showing the positional relationship between 3D data and a virtual camera in the second embodiment. [Figure 22] FIG. 13 is a diagram for explaining the effect of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, in which: Fig. 1 shows an image processing apparatus according to an embodiment of the present invention.

[0013] (First embodiment) <Configuration> Hereinafter, an image processing device for generating a virtual viewpoint image according to a first embodiment of the present invention will be described with reference to FIG. 1. The image generating device 100 shown in FIG. 1 generates and outputs an image seen from a virtual camera viewpoint (hereinafter referred to as a "camera viewpoint") designated by a user or the like based on input three-dimensional image data. The image generating device 100 of this embodiment is composed of an image processing device 200 and a user interface 300. A dedicated application for performing various operations such as displaying a camera viewpoint image and instructing a viewpoint change on the image processing device 200 is installed in the user interface 300. The image processing device 200 generates a camera viewpoint image based on three-dimensional data (hereinafter also referred to as 3D data) and displays it on the user interface 300. Furthermore, when the image processing device 200 receives an instruction to change the camera viewpoint sent from the dedicated application of the user interface 300, it generates an image according to the change in the camera viewpoint position and provides it to the user interface 300.

[0014] The image processing device 200 is, for example, a server computer, and has a processing unit 201, a data acquisition unit 202, an image generation unit 203, a missing area detection unit 205, and a mask data generation adjustment unit 206. The user interface 300 is, for example, a personal computer or the like, which is electrically connected to the image processing device 200, and has a display unit 301 and an operation unit 302.

[0015] The processing unit 201 provided in the image processing device 200 uses programs and data stored in advance to control the entire image processing device 200, and realizes each functional unit shown in FIG. 1. The processing unit 201 may have one or more dedicated hardware different from the processing unit 201, and at least a part of the processing by the processing unit 201 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor). The 3D data acquisition unit 202, the image generation unit 203, the defective area detection unit 205, and the mask data generation unit 206 are stored as programs in a ROM (not shown), and the functions are fulfilled by the processing unit 201 executing the programs. The RAM (not shown) temporarily stores data provided from the processing unit 201 and data provided from the outside via a communication I / F (not shown).

[0016] The display unit 301 provided in the user interface 300 is formed of, for example, a liquid crystal display, and displays a GUI and the like for the user to operate the image processing device. The operation unit 302 is formed of, for example, a keyboard, a mouse, a joystick, and the like, and inputs various instructions to the processing unit 201 in response to operations by the user. The operation unit 302 may be integrated with the display unit 301, for example, a liquid crystal display equipped with a touch panel.

[0017] <About 3D data generation> Next, an example of a method for generating 3D data according to this embodiment will be described with reference to the configuration diagram of the imaging device 400 in FIG. 2. The imaging device 400 is provided with an imaging unit 401 capable of capturing 3D data, and the captured 3D data can be stored in a recording medium. The imaging unit 401 can have any configuration as long as it functions as an acquisition means capable of acquiring 3D data. More preferably, it may be configured to simultaneously acquire color distribution information from the same viewpoint as the distance distribution information. Examples of configurations that realize such functions include a stereo camera equipped with two imaging systems consisting of an optical system capable of acquiring RGB images and an imaging element, and a ToF camera equipped with a ToF module capable of acquiring distance distribution information and an imaging system capable of acquiring RGB images. Below, a configuration of the imaging unit 401 capable of simultaneously acquiring distance distribution information and color distribution information by an imaging surface phase difference ranging method will be described as an example.

[0018] 3 shows the configuration of the photographing unit 401. The photographing unit 401 is composed of an imaging optical system 501 and an image sensor 502, and captures an image by collecting light from a subject located on an object plane using the imaging optical system 501 and exposing the image sensor 502, which is located at a position approximately optically conjugate with the object plane.

[0019] The image sensor 502 has a structure in which, for example, tens of millions of pixels 503, which are photoelectric conversion elements, are arranged in a grid pattern. By providing each pixel with a color filter that transmits light of a specific wavelength, such as red, green, or blue, and arranging them in, for example, a Bayer array, it becomes possible to obtain color distribution information.

[0020] One pixel 503 is composed of a microlens 504, a first photoelectric conversion unit 505, and a second photoelectric conversion unit 506, and obtains different light information according to its position by the two photoelectric conversion units 505, 506 arranged in the horizontal direction (X direction). From the received light information of all pixels, a first image configured as the luminance distribution of light received by the first photoelectric conversion unit 505 and a second image configured as the luminance distribution of light received by the second photoelectric conversion unit 506 are obtained.

[0021] The incident surface of the imaging element 502 and the light receiving surfaces of the first photoelectric conversion unit 505 and the second photoelectric conversion unit 506 are in a nearly Fourier conjugate relationship via the microlens 504. Therefore, the light receiving surface of the photoelectric conversion unit and the exit pupil of the imaging optical system 501 are in a nearly optically conjugate relationship. Since the position distribution at the exit pupil and the position distribution at the light receiving surface of the photoelectric conversion unit correspond to each other, by providing two different photoelectric conversion units, it is possible to separate and receive light beams that have passed through different pupil regions. The first image and the second image are luminance distribution information obtained by light beams that have passed through different pupil regions.

[0022] A light ray that forms an image on the imaging surface (ideally) enters the same point on the image sensor 502 regardless of the position on the pupil through which the light ray passes. However, for a defocused light ray, the position at which the light ray enters the image sensor 502 changes depending on the position on the pupil through which the light ray passes. In other words, an image shift occurs according to the amount of defocus.

[0023] The amount of image shift can be calculated, for example, by stereo matching between the first image and the second image. A patch of a small area of ​​one image is matched with the other image along the direction of the epipolar line, and the image shift amount is calculated by identifying the position with the highest correlation. The image shift amount calculated in this way and the focal length and focus position obtained from the shooting information are used to convert the position into the world coordinate system, generating 3D data.

[0024] In addition, data captured by the image capture unit 401 is grouped by object (by type of subject) by object detection 402. At this time, if the object is a person, a face organ detection unit 403 detects facial feature points and calculates face organ coordinates. This information is linked to the 3D data and stored in a recording medium.

[0025] <About 3D data> Next, the 3D data according to this embodiment will be described. Fig. 4(a) is a two-dimensional image captured by the above-mentioned image capture device, and Fig. 4(b) is a diagram showing the positional relationship between the generated 3D data of the object and the virtual camera A and virtual camera B. The positional relationship between the virtual camera A and the 3D data of the object is set to be the same as the positional relationship between the image capture device and the object in real space. Fig. 4(c) is a two-dimensional image generated from the viewpoint of the virtual camera A.

[0026] Here, we will explain the coordinate axes. As shown in Fig. 4(b), the optical axis of virtual camera A, which is located in the same position as the shooting camera, is set as the z-axis for the coordinate axes in the virtual space. Also, as shown in Fig. 4(c), the direction of gravity of the shooting data is set as the y-axis, and the axis perpendicular to the y and z-axes is set as the x-axis.

[0027] Figure 4(d) is a 2D image generated from the viewpoint of virtual camera B shown in Figure 4(b). In this way, by converting the image according to the viewpoint of the virtual camera, you can enjoy the 3D data of the object from any viewpoint.

[0028] In addition, the captured image shows only the upper half of the subject's body, with the lower half and below being cut off. In this case, the 3D data for the object is also generated with only the upper half of the body being generated, with the lower half and below being missing.

[0029] The 3D data of the object in this embodiment is not limited to only the subject photographed by the photographing device as in Fig. 4, but may be 3D data in which the subject is combined with 3D background data separately prepared as object data. Also, the subject is not limited to a person, but may be an object or a living thing.

[0030] <Overall processing flow> Next, the flow of processing in which the image generating device 100 according to this embodiment generates mask data from 3D data will be described with reference to the flowchart in Fig. 5. The processing in the flowchart in Fig. 5 is realized by the processing unit 201 receiving an instruction for viewpoint change processing from the user interface 300 and executing a predetermined processing program. The processing starts when the user performs an operation such as inputting 3D data for viewpoint change processing into the dedicated application described above, and then the user can select whether or not to perform this processing, and an instruction is issued based on the user's selection. In the following description, the symbol "S" represents a step.

[0031] In S101, 3D data is acquired. The acquired 3D data is cut out by recognizing individual objects. In addition, the data contains face recognition information so that it can be determined that the object is a person.

[0032] Next, the processing unit 201 acquires information about the movable range of the virtual camera (S102). The information about the movable range of the virtual camera is set in advance within the application and indicates the range in which the virtual camera can move. This setting can be changed by operating the user interface 300.

[0033] Next, the processing unit 201 executes the missing area detection unit 205 to detect missing areas of the object in the 3D data (S103). First, the image data of the object is labeled using Deep Learning to estimate the overall shape. Next, the estimated shape is compared with the 3D data to calculate the boundary surface where the object in the 3D data is missing, and it is determined as the missing area.

[0034] When a missing region is detected, the process proceeds to a mask data generation process (S104). In the mask data generation process, the processing unit 201 generates mask data so that the missing region is not visible from the viewpoint of the virtual camera.

[0035] When the above-mentioned processing is completed, this processing ends.

[0036] The above is an outline of the mask data generation process in the image processing device 200.

[0037] <About the movable range of the virtual camera> Next, the movable range of the virtual camera in the virtual space will be described. FIG. 6 is an example showing the movable range of the virtual camera according to this embodiment. FIG. 6(a) is a top view of the virtual camera and 3D data in the virtual space, and FIG. 6(b) is a side view of the same. As shown in FIG. 6(a), the movable range of the virtual camera in the xz plane is determined by the shortest distance r1 and the longest distance r2 between the object and the camera, and the rotation angle θ, with the center of gravity of the object in the 3D data as the center. In this embodiment, r1=1m, r2=3m, and θ=60°. Also, as shown in FIG. 6(b), the movable range of the virtual camera in the yz plane is set in the same way.

[0038] In this embodiment, the z direction is set to the optical axis direction of the shooting camera, but it may be the front direction of a virtual camera set in world coordinates in a three-dimensional space. The z direction is the vertical direction in the world coordinates, and the y direction is the horizontal direction in the world coordinates that is perpendicular to the z direction.

[0039] By installing a virtual camera in a specific spatial region and generating virtual viewpoint image data in this way, the position of the virtual camera can be restricted, making it possible to reduce discomfort when viewing an image from a virtual viewpoint. In addition, the user may freely set the movable range of the virtual camera depending on the object.

[0040] <About missing areas> Next, a method for obtaining a missing area of ​​an object in 3D data will be described with reference to the flow chart of FIG.

[0041] First, the processing unit 201 labels the input 3D data for each object (S201). For example, labeling is performed using Deep Learning, and labels categorized as semantic objects are attached. Alternatively, a method in which the user specifies an object and inputs a label may be used. In this embodiment, when the image in FIG. 4(a) is input, it is labeled as "person."

[0042] Next, the processing unit 201 estimates the overall shape of the target object based on the label (S202). For example, based on the model shape of a representative model prepared in advance, posture and shape correction is performed using Deep Learning from the image data of the target object, and the overall shape of the object is estimated. In this embodiment, the whole body shape is estimated as shown in FIG. 8(a).

[0043] Then, the processing unit 201 compares the target object of the 3D data with the overall shape estimated earlier (S203), calculates the missing part of the object, and determines the boundary between the missing part and the object as the missing area (S204). In this embodiment, the thick line part shown in Fig. 8(b) was determined as the missing area.

[0044] In this manner, in this embodiment, object detection and shape estimation are performed using Deep Learning to determine the missing area, but a configuration in which the user directly inputs and specifies the missing area via the operation unit 302 is also possible. In this case, the user interface 300 is provided with a function for accepting input of the missing area, and the missing area is determined by the user's operation.

[0045] <How to generate mask data> Next, a method for generating mask data will be described with reference to the flow chart of FIG.

[0046] First, the processing unit 201 acquires the distribution of z coordinate values ​​of the target object from the 3D data, calculates the position z' that is a predetermined distance away from the minimum value of the acquired z coordinate value distribution, and determines the Z' plane, which is the mask data generation plane, at the position z = z' (S301). In this embodiment, the predetermined value is set to 50 mm to calculate z'.

[0047] Next, the processing unit 201 draws lines connecting each coordinate in the movable range of the virtual camera acquired as described above with each coordinate in the missing area (S302), and calculates an intersection area P where this line intersects with the Z' plane (S303).

[0048] 10 is a diagram showing the relationship between the virtual camera movable range, the 3D data object, and the intersection area P on the yz plane according to this embodiment. In this way, the intersection area P is determined from the straight line connecting the virtual camera movable range and the missing area and the Z' plane.

[0049] Thereafter, the processing unit 201 generates mask data M so as to overlap the intersection area P (S304), and ends the process.

[0050] <Effects> Figures 11(a) and 11(b) respectively show 3D data resulting from generating mask data M for the scene shown in Figure 4, and 3D data obtained by further combining Figure 11(a) with a background image. In this way, by using the image generating device 100 according to the present invention, even if a user combines 3D data having a missing area with other 3D data, mask data is generated over the missing area, making it possible to generate 3D data that does not feel unnatural due to the missing area.

[0051] In the above embodiment, the intersection area P where the mask data M is superimposed is calculated by calculating the intersections for all coordinates in the movable range of the virtual camera, but to speed up the processing, the number of search coordinates for the movable range may be reduced. For example, the same effect can be obtained by setting the outer periphery of the movable range as the search range and calculating the intersection area P.

[0052] In addition, if the mask data M is sufficiently large relative to the missing area, the search range can be further reduced. For example, if the mask data M is sufficiently large relative to the missing area in the -y direction and ±x direction, the search range can be limited as follows:

[0053] Specifically, when the direction in which the missing area exists with respect to the object center is defined as the missing direction, one surface of the outer periphery in the opposite direction to the missing direction is set as a search surface within the movable range of the virtual camera, as shown in FIG. 12. Next, a straight line is created between the set search surface and the missing area, and the intersection with the Z' plane set above is calculated to obtain the upper limit area Pm. This is the maximum in the y-axis direction within the movable range of the virtual camera, so by generating mask data M of the above-mentioned size so as to be superimposed on the obtained upper limit area Pm, the missing area can be appropriately shielded for other virtual camera positions as well.

[0054] Furthermore, if the search surface in the movable range of the virtual camera is a plane, the search range can be further reduced.

[0055] Specifically, as shown in Fig. 13(a), consider the case where the coordinates where the extension of the search surface intersects with the object in the y-axis direction are greater than the coordinates of the missing area. In this case, the intersection point Pmin between the point Zmin, which is the point on the search surface close to the object, and the Z' plane and the intersection point Pmax between the point Zmax, which is the point on the search surface farthest from the object, and the Z' plane are located on the outer side. Therefore, the same results can be obtained even if the search range is limited to the point Zmin.

[0056] Conversely, as shown in Figure 13(b), if the coordinate where the extension of the search surface intersects with the object in the y-axis direction is smaller than the coordinate of the missing area, the intersection point Pmax is located outside, so the same result can be obtained even if the search range is limited to point Zmax.

[0057] (Variation 1) Although the 3D data used in the above-mentioned embodiment is a still image, the effect of the present invention can be obtained similarly even if the data is a moving image. In this case, it is not preferable that the mask data M moves for each frame. Therefore, in this embodiment, a method for generating one mask data M that can be used for all frames will be described. Below, as a first modification of the first embodiment, a process in the case where the 3D data is a moving image will be described with reference to the flowchart in FIG. 14.

[0058] In this embodiment, 3D data is treated as 3D video data consisting of a series of 3D still images. When acquiring 3D data (S401), the processing unit 201 breaks down the 3D data into frames and processes them. The virtual camera movable range is the same as in the first embodiment, so a description thereof will be omitted.

[0059] Next, the difference between the detection of a defective area and the first embodiment will be described with reference to the flowchart of FIG.

[0060] First, labeling of 3D data is performed for any one frame since it is sufficient to know what the target object is (S501). Next, the processing unit 201 estimates the overall shape of the target object based on the labels for all frames (S502). The method of shape estimation is the same as in the first embodiment, so it is omitted.

[0061] Next, the 3D data of each frame is compared with the estimated overall shape (S503). The boundary surfaces of the defect areas are calculated for all frames, and the outer periphery of the entire defect area obtained by integrating all of these is determined as the defect area (S504).

[0062] Similarly, mask data is generated by taking all frames into consideration. The differences from the first embodiment will be described with reference to the flow chart of FIG.

[0063] To determine the Z' plane, the z position distribution of all frames is calculated, and the Z' plane is set at a position a predetermined distance away from the minimum value (S601). After that, the same process as in the first embodiment is performed to generate mask data M.

[0064] By generating mask data M for missing areas for all frames in this manner, it becomes possible to mask missing areas while keeping the mask data M stationary in the moving image.

[0065] Second embodiment In the first embodiment, an example of a process of acquiring virtual camera viewpoint information and missing area information for any 3D data having a missing area and generating a mask in the space between the virtual camera viewpoint and the missing area was described. In this embodiment, synthesis of 3D data having a missing object and 3D data having only objects without missing objects is considered. Specifically, an example of a method of arranging an object having a missing area in the synthesis will be described. Note that the same reference numerals are used for configurations similar to those in the first embodiment, and explanations will be omitted.

[0066] 17 is a block diagram showing the configuration of this embodiment. The image generating device 100 of this embodiment has a position determination unit 207, which stores processing for determining the position of the 3D data acquired by the 3D data acquisition unit 202. The processing unit 201 executes this processing to determine the position of each piece of 3D data when the 3D data is synthesized.

[0067] Furthermore, the image generating device 100 has a 3D data synthesis unit 208, and the processing unit 201 synthesizes 3D data based on the synthesis position information determined by the position determination unit 207 described above.

[0068] Fig. 18 is a flowchart explaining the processing of this embodiment. The user inputs multiple pieces of 3D data to be synthesized (S701). For example, the 3D data is 3D data having object data of a missing object A shown in Fig. 19(a) and 3D data having object data of an object B and background data shown in Fig. 19(b).

[0069] When these multiple pieces of 3D data are input, the processing unit 201 obtains the movable range of the virtual camera and detects the missing area (S702, S703). Here, these operations in this embodiment are similar to those in the first embodiment, so the description will be omitted.

[0070] Next, the processing unit 201 determines the position of object A having a missing region (S704). First, the processing unit 201 calculates the occluded region occluded by object B from the coordinates of each virtual camera.

[0071] 20 is a diagram showing the movable range of the virtual camera as viewed from the x-axis direction of the world coordinate system, object B, and the occluded area. In this way, the occluded area is set as an area that is a blind spot from any virtual camera.

[0072] Once the occlusion region is calculated, the processing unit 201 determines the position of object A. First, the processing unit 201 determines the orientation of object A. For example, if object A is a person, the orientation of the object is set to a frontal orientation relative to the virtual camera based on the model estimated in S703. Alternatively, if object A was created from a shooting camera, the direction of the optical axis of the shooting camera may be set. Also, a UI operation unit that allows the user to input the orientation may be provided, and the user may input the orientation.

[0073] Next, the processing unit 201 places the object A in the orientation determined above so that the missing area is inside the occluded area. At this time, the processing unit 201 calculates the occluded volume other than the missing area so that the non-missing area does not enter the occluded area as much as possible, and determines the position where the area is the smallest.

[0074] When the placement positions are determined in this way, the processing unit 201 executes the process stored in the 3D data synthesis unit 208 to synthesize the 3D data. Fig. 21 is a diagram showing the positional relationship between the object data of object A and object data of object B synthesized by the above-mentioned process and the virtual camera.

[0075] 22 is a view of 3D data synthesized by the processing of this embodiment, viewed from a certain virtual camera position. In this way, by using the image generating device 100 according to the present invention, even if a user synthesizes 3D data having a missing area with other 3D data, the missing area is placed in the occluded area of ​​the 3D data, making it possible to synthesize 3D data without the sense of incongruity caused by the missing area.

[0076] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0077] 201 Processing section 203 Image Generation Unit 204 Coordinate conversion section 205 Defective area detection unit 206 Mask Data Generator 207 Position determination section 208 3D Data Synthesis Department 301 Display section 302 Operation section 401 Photography Department

Claims

1. A first acquisition means for acquiring 3D data and a missing area of ​​the 3D data; A second acquisition means for acquiring a movable range of the virtual camera; A generating means for generating mask data, The image generating device is characterized in that the generation means generates the mask data based on the moving range of the virtual camera and the missing area so that the missing area is not visible within the moving range of the virtual camera.

2. The 3D data is composed of background data, which is the background in a three-dimensional space, and object data, which is the subject of the image. an object detection means for detecting a type of the object data; a third acquisition means for acquiring a representative model of the type of subject detected by the object detection means, 2. The image generating device according to claim 1, wherein the first acquisition means acquires the missing area of ​​the 3D data by comparing the representative model acquired by the third acquisition means with the object data.

3. The first acquisition means includes: an estimation means for estimating an overall shape of the 3D data from the model shape of the representative model acquired by the third acquisition means and the 3D data; 3. The image generating apparatus according to claim 2, wherein the missing area is determined by comparing the overall shape estimated by said estimation means with said object data.

4. An input means for a user to input the missing area, 2. The image generating apparatus according to claim 1, wherein the first acquisition means acquires the defective area inputted by the input means.

5. The image generating device according to claim 1 , wherein the 3D data is 3D video data consisting of a series of 3D data for a plurality of still images.

6. The generating means includes: determining a generation surface for generating the mask data based on a distance distribution of the 3D data with respect to a front direction of the virtual camera; The image generating device according to claim 1, further comprising a determination means for determining an intersection area from the outer peripheral surface of the movable range of the virtual camera, a missing area of ​​the 3D data, and the generated surface, and generating the mask data in the space between the virtual camera and the missing area.

7. The generating means includes: determining a generation surface for generating the mask data based on a distance distribution of the 3D data with respect to a front direction of the virtual camera; determining a search range based on position information of a missing area of ​​the 3D data and the movable range of the virtual camera within the movable range of the virtual camera; The image generating device according to claim 1 , further comprising: a calculation unit that calculates an intersection area from the search range, a missing area of ​​the 3D data, and the generation surface; and generates the mask data in a space between the virtual camera and the missing area.

8. A first acquisition means for acquiring a plurality of 3D data, at least one of which has a missing area, and information on the missing area of ​​the 3D data; A synthesis means for synthesizing the plurality of 3D data to generate one 3D data; A second acquisition means for acquiring a movable range of the virtual camera; a first determination means for determining a synthesis position of the 3D data having the missing area when synthesizing the plurality of 3D data, The synthesis means is characterized in that it places 3D data having a missing area at a position determined by the first determination means so that the missing area is hidden by the other 3D data from the viewpoint of the virtual camera, thereby synthesizing the multiple 3D data.

9. The 3D data is composed of background data that is a background in a three-dimensional space and object data that is a subject, an object detection means for detecting a type of the object data; a third acquisition means for acquiring a representative model of the type of subject detected by the object detection means, 9. The image generating device according to claim 8, wherein the first acquisition means acquires the missing area of ​​the 3D data by comparing the representative model acquired by the third acquisition means with the object data.

10. an input means for a user to input a position of a missing area of ​​the 3D data; 9. The image generating apparatus according to claim 8, wherein the first determining means determines the position of synthesis based on a position input by the input means.

11. A first acquisition step of acquiring 3D data and a missing area of ​​the 3D data; a second movable range acquisition step of acquiring a movable range of the virtual camera; A generating step of generating mask data, A control method for an image generating device, characterized in that in the generation step, the mask data is generated based on the moving range of the virtual camera and the missing area so that the missing area is not visible within the moving range of the virtual camera.

Images