Image processing device, 3D model generation method and program

Abstract

The present technology relates to an image processing device, a 3D model generation method, and a program, which enable the number of imaging failures to be reduced in multi-view imaging for generating a 3D model. The image processing device is provided with: a 3D area calculation unit for generating a 3D area in an imaging range generated from a plurality of multi-view images; and a determination unit for determining the subject imaging state of an imaging device based on an area image obtained by projecting the 3D area to a specific viewpoint and a subject image from an imaging device corresponding to the specific viewpoint. The present technology can be applied to, for example, an image processing device for generating a 3D model, etc.

Description

Technical Field

[0001] The present technology relates to an image processing device, a 3D model generation method, and a program, and more particularly, to an image processing device, a 3D model generation method, and a program capable of reducing failed image capture in multi-view image capture for 3D model generation. Background Art

[0002] There is a technology that generates a 3D model of a subject from moving images captured from multiple viewpoints, and generates virtual viewpoint images of the 3D model corresponding to arbitrary viewing / listening positions to provide images from free viewpoints. This technology is called volumetric capture technology, etc.

[0003] In the generation of 3D models, methods such as Visual Hull are used (for example, refer to Patent Document 1), which cuts out the three-dimensional shape of the subject using multiple captured images from different directions. Since the accuracy of the 3D model is improved by increasing the number of cameras, it is preferable to use multiple cameras to capture images of the subject from different directions.

[0004] Reference List

[0005] Patent Literature

[0006] Patent document 1: WO 2018 / 150933 A Summary of the invention

[0007] Problems to be solved by the present invention

[0008] However, in the case where the number of cameras is large, there are cases where the subject is out of the image capturing range of a certain camera when the subject moves and image capturing needs to be performed again.

[0009] The present technology has been made in view of such circumstances, and aims to reduce failed image capture in multi-view image capture for 3D model generation.

[0010] Solution to the problem

[0011] According to one aspect of the present technology, an image processing device includes: a generation unit that generates a 3D area of ​​an image capture range generated based on multiple multi-view images; and a determination unit that determines a condition in which an image capture device captures a subject based on a region image obtained by projecting the 3D area onto a specific viewpoint and a subject image from an image capture device corresponding to the specific viewpoint.

[0012] A 3D model generation method according to one aspect of the present technology is a 3D model generation method for generating a 3D model of a subject, and includes: generating a 3D area of ​​an image capture range generated based on a plurality of multi-view images; determining a condition in which an image capture device captures the subject based on an area image obtained by projecting the 3D area onto a specific viewpoint and a subject image from an image capture device corresponding to the specific viewpoint; and generating a 3D model of the subject based on the plurality of multi-view images.

[0013] A program according to one aspect of the present technology causes a computer to function as: a generation unit that generates a 3D area of ​​an image capture range generated based on a plurality of multi-view images; and a determination unit that determines a condition in which an image capture device captures a subject based on an area image obtained by projecting the 3D area onto a specific viewpoint and a subject image from the image capture device corresponding to the specific viewpoint.

[0014] In one aspect of the present technology, a 3D area of ​​an image capture range generated from a plurality of multi-view images is generated, and a condition in which an image capture device captures a subject is determined based on an area image obtained by projecting the 3D area onto a specific viewpoint and a subject image from an image capture device corresponding to the specific viewpoint.

[0015] Note that the image processing device according to one aspect of the present technology can be realized by causing a computer to execute a program. The program executed by the computer can be provided by being transmitted via a transmission medium or by being recorded on a recording medium.

[0016] The image processing device may be an independent device or an internal block constituting one device. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A configuration example of an image processing system to which the present technology is applied is shown.

[0018] Figure 2 is a diagram describing a notification example of the image processing system.

[0019] Figure 3 is a block diagram showing a configuration example of a first embodiment of an image processing device.

[0020] Figure 4 is a diagram showing an example of a process of calculating a 3D area.

[0021] Figure 5 is a diagram showing an example of processing of an effective area mask image.

[0022] Figure 6 is a diagram illustrating an example of visual recognition camera information.

[0023] Figure 7 2 is a diagram showing a processing example of the outline extraction unit.

[0024] Figure 8 is a diagram describing a camera determination process by a determination unit.

[0025] Fig. 9 is a flowchart describing the first image capturing range determination process.

[0026] Fig.10 is a block diagram showing a configuration example of a second embodiment of an image processing device.

[0027] Fig.11 is a flowchart describing the second image capturing range determination process.

[0028] Fig.12 is a block diagram showing a configuration example of a third embodiment of an image processing device.

[0029] Fig.13 is a diagram describing the processing of the 3D area calculation unit according to the third embodiment.

[0030] Fig.14 2 is a diagram illustrating processing of a projection unit according to the third embodiment.

[0031] Fig.15 is a diagram showing other examples of camera arrangement.

[0032] Fig.16 is a block diagram illustrating a configuration example of an embodiment of a computer to which the present technology is applied.

[0033] Mode for carrying out the invention

[0034] Hereinafter, a mode for carrying out the present technology (hereinafter, referred to as an embodiment) will be described. Note that the description will be provided in the following order.

[0035] 1. Configuration example of image processing system

[0036] 2. First Embodiment of Image Processing Device

[0037] 3. First Image Capture Range Determination Process

[0038] 4. Second Embodiment of Image Processing Device

[0039] 5. Second Image Capturing Range Determination Process

[0040] 6. Third Embodiment of Image Processing Device

[0041] 7. Third Image Capturing Range Determination Process

[0042] 8. Other examples of camera placement

[0043] 9. Computer configuration example

[0044] <1. Configuration Example of Image Processing System>

[0045] Figure 1 A configuration example of an image processing system to which the present technology is applied is shown.

[0046] For example, Figure 1 The image processing system in CAM-1 includes N image capturing devices CAM-1 to CAM-N, an image processing apparatus 11 and a display device 12.

[0047] like Figure 1 As shown, the image capturing devices CAM-1 to CAM-N (N>1) are arranged to surround a person as a subject, capture an image of the subject, and supply the captured images obtained as a result to the image processing device 11. The captured images are sequentially supplied from the respective image capturing devices CAM to the image processing device 11 as moving images or serial number still images. The plurality of captured images supplied from the image capturing devices CAM-1 to CAM-N are a plurality of multi-view images capturing the subject from different viewpoints. Figure 1 The example in shows an example where N=8, and eight image capturing devices CAM-1 to CAM-8 are arranged to surround a subject.

[0048] The image processing device 11 generates a 3D model of an object as a subject based on a plurality of captured images provided by the image capture devices CAM-1 to CAM-N. The data of the generated 3D model of the object includes, for example, image data of the captured images of the subject obtained from the respective image capture devices CAM and 3D shape data representing the 3D shape of the subject. The 3D shape data may be, for example, a depth image corresponding to the captured images captured by the plurality of image capture devices CAM, or may be represented by a point cloud, a polygonal mesh, or the like, wherein the point cloud uses a collection of points to represent the three-dimensional position of the object, and the polygonal mesh uses connections between vertices to represent the three-dimensional position.

[0049] The display device 12 displays a predetermined image supplied from the image processing apparatus 11 .

[0050] Note that the image processing device 11 does not directly acquire captured images from the image capturing devices CAM-1 to CAM-N, but may acquire captured images once stored in a predetermined storage unit such as a data server to generate a 3D model of the object. In addition, a 3D model of the object may be generated by combining the captured images acquired from the image capturing devices CAM-1 to CAM-N with the captured images stored in the storage unit.

[0051] In the present embodiment, the moving image generated by each of the image capturing devices CAM includes a captured image (RGB image) including RGB wavelengths, but may also include a multispectral image including an infrared (IR) image.

[0052] Furthermore, each of the image capturing devices CAM can perform a plurality of image capturing operations while changing image capturing conditions such as exposure conditions, light source position, and light source color, and supply captured images obtained as a result to the image processing apparatus 11 .

[0053] In addition, each of the image capture devices CAM may include a distance measuring sensor, measure the distance to the subject, generate not only an RGB captured image serving as texture information of the subject, but also generate a depth image, and provide the depth image to the image processing device 11, in which the distance to the subject is stored as a depth value in association with each pixel of the captured image. In addition, the distance measuring sensor may be arranged independently of each image capture device CAM. In this case, the number of image capture devices CAM and the number of distance measuring sensors (distance measuring devices) may be different.

[0054] The image processing device 11 may be a server device on the cloud, and the display device 12 may be a device including a display, such as a smart phone (mobile terminal). Communication may be performed between the plurality of image capturing devices CAM and the image processing device 11 and between the image processing device 11 and the display device 12 by direct connection or by connection via a predetermined network such as a local area network (LAN) and the Internet.

[0055] In the generation of a 3D model of an object, since the accuracy of the 3D model is improved by increasing the number of image capture devices CAM, it is preferable to use a plurality of image capture devices CAM to capture images of the object from different directions. However, when the number of image capture devices CAM is large, there is a case where, when the object moves, the object is out of the image capture range of one or more image capture devices CAM, and image capture needs to be performed again.

[0056] The image processing device 11 can generate thumbnail images based on the captured images sent from the N individual image capture devices CAM during image capture, and can display the thumbnail images on the display device 12. Note that the thumbnail images displayed on the display device 12 may be all thumbnail images corresponding to the captured images of the N individual image capture devices CAM, or may be one thumbnail image (representative thumbnail image) corresponding to the captured image of one representative image capture device CAM. In addition, a plurality of thumbnail images corresponding to a predetermined plurality of image capture devices CAM may be used. The thumbnail image is an image obtained by reducing the resolution and frame rate of the captured image obtained from the image capture device CAM.

[0057] In addition, the image processing device 11 can determine whether the subject is outside the image capturing range of the predetermined image capturing device CAM due to the movement of the subject during image capturing, and can notify the user that the subject is outside the image capturing range. Here, the user is a person who is the image capturer or the subject.

[0058] Figure 2 A notification example is shown in the case where the image processing apparatus 11 determines whether a subject is outside the image capturing range of a predetermined image capturing device CAM and notifies the user that the subject is outside the image capturing range.

[0059] Figure 2 A shows an example of notifying the user that a subject is outside the image capturing range of the image capturing device CAM-5 by displaying a message (characters) of “Warning: CAM05” on the display device 12 as a dialog.

[0060] Figure 2 B shows an example in which a message of “Warning: CAM05” is superimposed on the representative thumbnail image displayed on the display device 12 to notify the user that the subject is outside the image capturing range of the image capturing device CAM-5.

[0061] Note that, in the case of displaying all thumbnail images corresponding to N image capturing devices CAM, it is only necessary to superimpose a "warning" message on the thumbnail image of the image capturing device CAM whose subject is outside the image capturing range to notify the user.

[0062] Figure 2 C shows the following example: in the outline image of the subject generated based on the captured image obtained from the predetermined image capturing device CAM, the area of ​​the subject outside the image capturing range is colored to notify the user that the colored area of ​​the subject is outside the image capturing range.

[0063] The outline image is an image representing a region of a subject (subject region) as an object to be drawn with an outline, and is, for example, a binary image in which the pixel value of each pixel is binarized to "0" or "1". For example, in the outline image, the region of the subject has a pixel value set to "1" and is represented in white, and the region other than the subject has a pixel value set to "0" and is represented in black.

[0064] exist Figure 2 In the example of C of FIG. 1 , in the subject area of ​​the outline image represented in white, the foot area is colored red, and the user is notified that the foot area is outside the image capturing range. Note that Figure 2 In the figure, the red area is shaded due to drawing limitations.

[0065] As described above, in addition to the object generation function of generating a 3D model of an object using a subject as an object, the image processing device 11 also has an image capture range determination function - used to determine the positional relationship between the subject and the image capture range, and to notify (warn) the user when the subject is outside the image capture range.

[0066] Note that in terms of notifying the user when the subject is outside the image capturing range, the user may be notified when the subject is outside the image capturing range of one image capturing device CAM, or, in the case where a large number of image capturing devices CAM are provided, the user may be notified when the subject is outside the image capturing range of a predetermined number or more of image capturing devices CAM set in advance. In this case, the number of devices or more of devices for notification may be set (specified) in the determination unit 36 ​​described later.

[0067] In addition, Figure 2In the example of , an example of giving a notification that a subject is outside a predetermined image capture range by displaying a character or an image on the display device 12 has been described, but the notification may be given in other ways. For example, a notification may be given by outputting a beep or by outputting a voice such as "the subject is outside the range of the fifth camera" and "the foot is out of range". In the case where a notification is given by a sound such as a beep and a voice, a speaker may be additionally provided, or a speaker of the display device 12 may be used. In addition, both a screen display of the display device 12 and a sound output may be provided. In addition, instead of notifying immediately, the notification information may be stored in a storage unit together with the captured image so that a notification may be given later. For example, the notification information may be embedded in the captured image or 3D shape data as tag information, or a file for storing the notification information may be generated and stored separately from the captured image and the 3D shape data. When a user views a captured image or a 3D model, a notification may be given by a character, an image, a sound, or the like based on the notification information included in the tag information or the file.

[0068] Hereinafter, the image capturing range determination function of the image processing device 11 will be described in detail. Note that the object generation function of the image processing device 11 will be briefly described as being implemented using a general method.

[0069] <2. First Embodiment of Image Processing Device>

[0070] Figure 3 : is a block diagram showing a configuration example of a first embodiment of the image processing device 11 .

[0071] The image processing device 11 includes an image input unit 31, a calibration processing unit 32, an effective area calculation unit 33, a contour extraction unit 34, a semantic processing unit 35, a determination unit 36, a notification unit 37, a 3D model calculation unit 38, and a storage unit 39. The effective area calculation unit 33 includes a 3D area calculation unit 41 and a projection unit 42. Note that the semantic processing unit 35 is provided as an option and can be omitted.

[0072] The image input unit 31 acquires captured images sent from N individual image capture devices CAM. The captured images are sequentially input as motion images. The N captured images from the N individual image capture devices CAM are provided to the calibration processing unit 32, the contour extraction unit 34, the semantic processing unit 35, the notification unit 37, and the 3D model calculation unit 38.

[0073] The calibration processing unit 32 performs a calibration process for calculating the camera parameters of each image capture device CAM based on the N captured images captured by the N image capture devices CAM. The calibration processing unit 32 performs the calibration process using, for example, the N captured images obtained by capturing a chessboard having a black and white checkered checkered pattern by means of the N image capture devices CAM. Through the calibration process, characteristic parameters that individually represent the characteristics of each of the N image capture devices CAM and external parameters that represent the relationship with another image capture device CAM are determined. The characteristic parameters that represent the characteristics of a single image capture device CAM include, for example, internal parameters such as the focal length, principal point, and viewing angle of the image capture device CAM, and optical characteristic parameters such as distortion (aberration). External parameters include relative coordinate values ​​(relative positions) of the own image capture device CAM relative to another image capture device CAM.

[0074] Note that, in the case where the positions of the N image capture devices CAM are fixed, all that is required before performing the calibration process is to capture the chessboard once before capturing the image used to generate the 3D model. In the case where the position of the image capture device CAM is moved or the magnification (viewing angle) is changed, in addition to the calibration process before starting image capture, it is also necessary to perform real-time calibration processing based on feature point detection of the subject, etc.

[0075] Note that the calibration process is not limited to the above method using the checkerboard, and other methods may be used as long as the method enables the determination of the required parameters. The various parameters determined by the calibration process are provided as camera parameters to the effective area calculation unit 33 (the 3D area calculation unit 41 and the projection unit 42).

[0076] The 3D area calculation unit 41 of the effective area calculation unit 33 calculates the 3D area included in the image capturing ranges of all the N image capturing devices CAM based on the camera parameters supplied from the calibration processing unit 32 .

[0077] Figure 4 There is shown a processing example in which the 3D area calculation unit 41 calculates a 3D area 51 which is an area included in the image capturing ranges of six respective image capturing devices CAM- 1 to CAM- 6 (where N=6).

[0078] As a method for calculating the 3D area included in the image capture range of all N image capture devices CAM, for example, a method such as a visual shell can be used, which is used to project the image capture range of each image capture device CAM according to camera parameters and cut out a three-dimensional shape. Alternatively, the 3D area can be calculated using three-dimensional geometry, etc. The data representing the calculated 3D area is provided to the projection unit 42 in a predetermined 3D data format, and the predetermined 3D data format is, for example, a point cloud that represents the three-dimensional position of an object using a set of points, voxel data that represents the three-dimensional position of an object using a set of rectangular body meshes, and a polygon mesh that represents the three-dimensional position of an object using connections between vertices.

[0079] The projection unit 42 is supplied with the 3D area included in the image capturing ranges of all the N image capturing devices CAM from the 3D area calculation unit 41 , and the camera parameters from the calibration processing unit 32 .

[0080] The projection unit 42 projects the 3D area from the 3D area calculation unit 41 onto the viewpoint of a predetermined image capture device CAM used as a target camera, and generates an effective area mask image (area image) represented as a binary image in which an area corresponding to the 3D area is set to white (pixel value "1"), and an area other than the 3D area is set to black (pixel value "0"). Since each of the N image capture devices CAM is used as a target camera, an effective area mask image is generated for each of all the image capture devices CAM, and thus as many effective area mask images as the number N of the image capture devices CAM (i.e., N images) are generated. Hereinafter, the area represented in white in the effective area mask image is referred to as a projected 3D area.

[0081] Figure 5 An example of an effective area mask image 52 is shown, in which the image capture device CAM-1 is used as the target camera, by Figure 4 The 3D area 51 corresponding to the six image capture devices CAM-1 to CAM-6 shown in FIG. 5 is projected onto the viewpoint of the target camera to obtain the effective area mask image 52.

[0082] In the projection plane (image capture range) obtained by projecting the 3D area 51 onto the viewpoint of the image capture device CAM-1, the effective area mask image 52 is represented as a binary image in which the projected 3D area 53 corresponding to the 3D area 51 is set to white (pixel value "1"), and the area other than the projected 3D area 53 is set to black (pixel value "0").

[0083] Furthermore, the projection unit 42 may generate visual recognition camera information including number information indicating how many image capture devices CAM can view the surface of the subject and camera information indicating which image capture device CAM can view the surface of the subject.

[0084] Figure 6 An example of visual recognition camera information when images are captured by five image capturing devices CAM-1 to CAM-5 is shown.

[0085] like Figure 6 As shown, the inside of the projected 3D area 53 of the effective area mask image 52 is divided according to the number of image capture devices CAM that can view the surface of the subject. This number represents number information indicating the number of image capture devices CAM that can directly visually recognize the subject. By projecting each point on the surface of the 3D area 51 onto each image capture device CAM, it can be determined whether the target image capture device CAM can directly visually recognize the surface. By clarifying which image capture devices CAM can view the surface of the subject (3D area 51), it is possible to count how many image capture devices CAM can view the surface of the subject (3D area 51) in total.

[0086] return Figure 3 The projection unit 42 provides the generated N effective area mask images and visual recognition camera information to the determination unit 36.

[0087] The contour extraction unit 34 generates a contour image obtained by extracting a region of a subject as an object to be drawn with a contour for each of the N captured images supplied from the image input unit 31. The generated contour image is supplied to the determination unit 36.

[0088] The detection method for detecting the contour of the subject in the texture image is not particularly limited, and any method may be adopted. Figure 7 As shown, the contour image 63 can be generated by pre-capturing and storing a background image 62 in which only the background excluding the subject is captured, and using a background subtraction method that takes the difference between the captured image 61 and the background image 62.

[0089] Furthermore, when a depth image is also acquired in which the distance to the subject is stored as a depth value, a contour image can be generated by separating a foreground area as a subject area from a background area based on distance information of the depth image.

[0090] Alternatively, a contour image may be generated by extracting a subject region from only one captured image using an image matting technique such as Graph Cut and chroma key.

[0091] The semantic processing unit 35 performs a semantic segmentation process for identifying semantic information of an object appearing as a subject in a captured image and adding the semantic information to each predetermined area. For example, in the case where the subject is a person, the semantic processing unit 35 can identify and classify the subject area of ​​the entire person into various parts, such as the head, hands (right hand and left hand), arms (right arm and left arm), feet (right foot and left foot), torso, etc. through semantic segmentation processing. In the case where the subject is not a person, the semantic processing unit 35 can identify the type of object, such as a ball, a racket, and a car, and add semantic information thereto. The semantic processing unit 35 provides the semantic information added to each predetermined area of ​​the identified subject to the determination unit 36. Instance semantic segmentation can be performed, in which the semantic information includes individual information. For example, by utilizing individual information, processing focusing on the number of subjects (such as one person and two people) or a specific subject can be performed.

[0092] The determination unit 36 ​​determines the conditions in which the respective image capture devices CAM capture the subject based on the N effective area mask images corresponding to the N image capture devices CAM from the projection unit 42 and the N contour images supplied from the contour extraction unit 34 .

[0093] Specifically, the determination unit 36 ​​performs other camera determination processing and own camera determination processing described below with each of the N image capture devices CAM as a target camera. The determination unit 36 ​​performs other camera determination processing for determining whether a subject is outside the image capture range of any image capture device CAM other than the target camera by using a pair of effective area mask images and a contour image corresponding to the image capture device CAM serving as the target camera. In addition, the determination unit 36 ​​performs own camera determination processing for determining whether a subject is outside the capture range of the image capture device CAM serving as the target camera by using the contour image corresponding to the image capture device CAM serving as the target camera.

[0094] In other camera determination processes, such as Figure 8 As shown in A, the determination unit 36 ​​compares, for each pixel of the object area 71, whether the object area 71 represented by white in the outline image exceeds the projection 3D area 53 represented by white in the effective area mask image 52.

[0095] In the self-camera determination process, such as Figure 8As shown in FIG. 8B , the determination unit 36 ​​compares, for each pixel of the object region 71 , whether the object region 71 represented in white in the outline image is in contact with the four sides of the outer periphery of the outline image 63 .

[0096] Note that, as described above, the semantic processing unit 35 can be provided as an option, and in the case where the semantic processing unit 35 is set, the determination unit 36 ​​can specify a part of the subject (for example, the right foot, left hand, etc.), an object (ball), etc. that exceeds the projected 3D area 53 by using semantic information from the semantic processing unit 35.

[0097] As determination information indicating the determination result, the determination unit 36 ​​provides the notification unit 37 with the result of whether the subject is outside the image capturing range of each image capturing device CAM serving as the target camera, and device specifying information indicating which image capturing device CAM is the image capturing device CAM whose image capturing range the subject is outside in the case where the subject is outside the image capturing range. Note that the determination information may be constantly provided to the notification unit 37, or may be provided to the notification unit 37 only in the case where the subject is outside the image capturing range.

[0098] In addition, when the semantic processing unit 35 is set, the determination unit 36 ​​can also provide semantic designation information as part of the determination information to the notification unit 37 when the subject is outside the image capture range, and the semantic designation information is such as the object and the part of the subject outside the image capture range.

[0099] In addition, if Figure 2 In B and C, when the notification unit 37 notifies the user that the subject is outside the image capturing range together with the captured image and the contour image, the contour image and the captured image of the target camera with the subject outside its image capturing range are also provided from the determination unit 36 ​​to the notification unit 37.

[0100] The notification unit 37 generates notification information for notifying the user that the subject is outside the image capturing range of the specific image capturing device CAM based on the determination information from the determination unit 36 ​​, and outputs the notification information to the display device 12 .

[0101] exist Figure 2 In the case of the dialog display shown in A, the notification information is display control information for displaying a message dialog including information for indicating that the image capturing device CAM is outside its image capturing range. Figure 2In the example of B or C, the notification information is display control information for displaying the following information: the image capture device CAM indicating that the subject is outside its image capture range, and the outline image or captured image. The outline image or captured image can be reduced as needed, etc., and can be output to the display device 12 as a thumbnail image.

[0102] In addition, in the case where a buzzer sound or a voice such as "the subject is outside the range of the fifth camera" is output to notify the user, the notification information is sound control information including an acoustic signal. The notification information may be both a screen display and a sound, or one of them. In the case where the semantic processing unit 35 is provided, notifications such as "the foot is outside the image capture range", "the ball is outside the image capture range" can be provided. The display device 12 notifies the user that the subject is outside the image capture range of the specific image capture device CAM based on the notification information from the notification unit 37.

[0103] The 3D model calculation unit 38 generates a 3D model of an object (with the object as an object) using captured images of the object supplied from the N respective image capturing devices CAM, and supplies data of the generated 3D model of the object (3D model data) to the storage unit 39 .

[0104] For example, the 3D model calculation unit 38 extracts the difference between the captured image including the subject and the background image provided from the image capture device CAM to generate a contour image in which the subject area is extracted. Then, the 3D model calculation unit 38 generates a 3D model of the object by using the camera parameters of the N individual image capture devices CAM and a plurality of contour images in different directions with the aid of a method such as a visual shell. The contour image and the camera parameters can be obtained from the calibration processing unit 32 and the contour extraction unit 34, or can be calculated separately. The details of the generation of the 3D model of the object are omitted herein.

[0105] The storage unit 39 stores the 3D model data generated in the 3D model calculation unit 38. The stored 3D model data is supplied to a reproduction device or the like that reproduces a 3D model of an object as needed, and is reproduced (restored) in the reproduction device.

[0106] The image processing apparatus 11 is configured as described above.

[0107] Note that the image capture range determination function and the object generation function that can be executed in the image processing device 11 may be implemented in different devices rather than in the same device. In this case, the image processing device 11 that executes the image capture range determination function includes, for example, an image input unit 31, a calibration processing unit 32, an effective area calculation unit 33, a contour extraction unit 34, a semantic processing unit 35, a determination unit 36, and a notification unit 37. The image processing device 11 that executes the object generation function includes, for example, the image input unit 31, a 3D model calculation unit 38, and a storage unit 39.

[0108] <3. First Image Capturing Range Determination Process>

[0109] Next, we will refer to Fig. 9 The flowchart in describes a first image capturing range determination process performed by the first embodiment of the image processing apparatus 11. For example, this process starts simultaneously with the start of image capturing in the N image capturing devices CAM.

[0110] First, in step S1, the image input unit 31 acquires captured images sent from N respective image capture devices CAM. The acquired N captured images are provided to the calibration processing unit 32, the contour extraction unit 34, the semantic processing unit 35, the notification unit 37 and the 3D model calculation unit 38.

[0111] In step S2, the calibration processing unit 32 performs calibration processing for calculating camera parameters of each image capture device CAM based on the N captured images captured by the N image capture devices CAM. Through this processing, characteristic parameters (internal parameters and optical characteristic parameters) representing the characteristics of each of the image capture devices CAM and external parameters representing the relative positional relationship between the image capture devices CAM are obtained.

[0112] In step S3 , the 3D area calculation unit 41 calculates a 3D area included in the image capturing ranges of all the N image capturing devices CAM based on the camera parameters supplied from the calibration processing unit 32 .

[0113] In step S4, the projection unit 42 generates an effective area mask image obtained by projecting the 3D area calculated in the 3D area calculation unit 41 onto the viewpoints of the respective image capture devices CAM. N effective area mask images are generated, which are as many images as the number of image capture devices CAM. In addition, the projection unit 42 may also generate visual recognition camera information representing number information indicating the number of image capture devices CAM that can view the image capture range of the image capture device CAM, and camera information indicating which image capture device CAM can view the image capture range. The generated N effective area mask images and N visual recognition camera information are provided to the determination unit 36.

[0114] In step S5, the contour extraction unit 34 generates N contour images from the N captured images supplied from the image input unit 31. The generated N contour images are supplied to the determination unit 36.

[0115] In step S6, the semantic processing unit 35 performs semantic segmentation processing on the N captured images supplied from the image input unit 31 to generate semantic information, and supplies the semantic information to the determination unit 36. This processing may be omitted in the case where there is no need to specify a part of an object or person outside the image capturing range of the image capturing device CAM.

[0116] In step S7, the determination unit 36 ​​performs a camera determination process for determining whether the subject is outside the image capturing range of the image capturing device CAM based on the N effective area mask images from the projection unit 42 and the N contour images from the contour extraction unit 34. More specifically, the determination unit 36 ​​performs an other camera determination process for determining whether the subject is outside the image capturing range of any image capturing device CAM other than the target camera, and an own camera determination process for determining whether the subject is outside the image capturing range of the image capturing device CAM serving as the target camera. Determination information indicating the determination results of the other camera determination process and the own camera determination process performed with each of the image capturing devices CAM as the target camera is supplied from the determination unit 36 ​​to the notification unit 37.

[0117] In step S8, based on the determination information from the determination unit 36, the notification unit 37 generates notification information for notifying the user that the subject is outside the image capturing range of the specific image capturing device CAM, and transmits the notification information to the display device 12. In the case where the subject is not outside the image capturing range of any image capturing device CAM, the notification information is not particularly required.

[0118] In step S9 , based on the notification information from the notification unit 37 , the display device 12 notifies the user that the subject is outside the image capturing range of the specific image capturing device CAM.

[0119] At this point, the first image capture range determination process ends. According to the first image capture range determination process, since the user can be notified when the subject is outside the image capture range of the multiple image capture devices CAM that capture the subject, failed image capture in multi-view image capture for 3D model generation can be reduced.

[0120] In the above-mentioned first image capture range determination processing, the determination unit 36 ​​determines whether the subject is outside the image capture range of the image capture device CAM. However, not only whether the subject is completely outside the image capture range, the determination unit 36 ​​can also determine whether the subject is about to be outside the image capture range when the subject moves slightly, and notify the user. In this case, in a case where the subject area is outside a range that is smaller than a predetermined ratio (for example, 10%) than the projected 3D area 53 obtained by projecting the 3D area 51 calculated by the 3D area calculation unit 41 onto the viewpoint of the target camera, such a situation can be regarded as a state where the subject is about to be outside the image capture range, and a notification can be given. Alternatively, in a case where a motion vector of the subject is detected and the position of the subject predicted after a predetermined period of time is outside the projected 3D area 53, such a situation can be regarded as a state where the subject is about to be outside the image capture range, and a notification can be given. Furthermore, a notification may be given when the subject exceeds the projected 3D area 53 by a certain amount, or when a certain part is detected to be outside the area based on semantic information obtained in the semantic processing unit 35 .

[0121] Whenever a new captured image captured by the N image capturing devices CAM is input, the reference is repeatedly performed. Fig. 9 The first image capture range determination process described above is described, but the process that does not change in the update of the captured image can be appropriately omitted. For example, in the case where the positions and viewing angles of the N image capture devices CAM do not change, the calibration process in step S2 and the process of generating the effective area mask image in steps S3 and S4 only need to be performed once in the first determination process. Alternatively, the process can be performed before performing image capture for generating a 3D model.

[0122] When the captured images captured by the N individual image capture devices CAM are sequentially input into the image processing apparatus 11, a 3D model generation process by means of the 3D model calculation unit 38 is also performed in parallel with the above-mentioned image capture range determination process. The image capture range determination process and the 3D model generation process may be performed independently, or may be performed in collaboration. For example, in the image capture range determination process, determination information indicating that the subject is outside the image capture range of the image capture device CAM may also be provided from the determination unit 36 ​​to the 3D model calculation unit 38, and in the case where the subject is outside the image capture range of the image capture device CAM, the generation of the 3D model may be stopped or interrupted.

[0123] <4. Second Embodiment of Image Processing Device>

[0124] Fig.10 is a block diagram showing a configuration example of a second embodiment of the image processing device 11 .

[0125] exist Fig.10 In, with Figure 3 Portions corresponding to those of the first embodiment shown in FIG. 1 are marked with the same reference numerals, and descriptions of these portions will be appropriately omitted.

[0126] While the image processing apparatus 11 according to the first embodiment described above has a configuration emphasizing determination accuracy as to whether a subject is outside the image capturing range of the image capturing device CAM, the image processing apparatus 11 according to the second embodiment has a configuration emphasizing real-time processing performance.

[0127] In the second embodiment, the image processing device 11 performs camera determination processing for determining whether a subject is outside the image capturing range of the image capturing device CAM using only captured images of M (N>M) image capturing devices CAM selected from N image capturing devices CAM.

[0128] Fig.10 The image processing apparatus 11 according to the second embodiment in FIG. 1 is different from the image processing apparatus according to the first embodiment in that a camera selection unit 81 and a reduction processing unit 82 are newly provided.

[0129] The camera selection unit 81 is supplied with the camera parameters of the respective image capture devices CAM from the calibration processing unit 32 .

[0130] The camera selection unit 81 selects M (N>M) image capturing devices CAM from among the N image capturing devices CAM. The camera selection unit 81 may manually select a device by a user's designation, or may automatically select a device without a user's designation.

[0131] In the case of manual selection, the camera selection unit 81 displays the arrangement and viewing angle information of each image capturing device CAM on the display device 12 based on the camera parameters of each image capturing device CAM, and enables the user to specify M image capturing devices CAM to be used.

[0132] On the other hand, in the case of automatic selection, the camera selection unit 81 groups several image capture device CAMs that are close to each other based on the camera parameters of the respective image capture device CAMs to classify the N image capture device CAMs into M groups. Then, the camera selection unit 81 selects one image capture device CAM from each group, thereby selecting the M image capture device CAMs to be used. Alternatively, in the case where the subject is a person, the camera selection unit 81 may set the image capture device CAM that captures the face of the person as a reference image capture device CAM, and sequentially select an image capture device CAM at a position away from the reference image capture device CAM and an image capture device CAM at a position symmetrical to the selected image capture device CAM until the number of image capture device CAMs becomes M.

[0133] The camera selection unit 81 supplies camera selection information for specifying the selected M image capturing devices CAM to the 3D area calculation unit 41 of the effective area calculation unit 33 and the reduction processing unit 82 .

[0134] The 3D area calculation unit 41 of the effective area calculation unit 33 calculates the 3D area included in the image capture range of the M image capture devices CAM using the camera parameters of the M image capture devices CAM selected in the camera selection unit 81. In the second embodiment, since the 3D area calculation process using N camera parameters according to the first embodiment is changed to the 3D area calculation process using M camera parameters, the processing load is reduced. In addition, by setting the resolution of the points or vertices representing the three-dimensional shape of the object when calculating the 3D area, the voxel size, etc. to setting values ​​that are coarser than those in the first embodiment, the processing load can be further reduced.

[0135] The projection unit 42 generates M effective area mask images obtained by projecting the 3D area calculated in the 3D area calculation unit 41 onto the viewpoints of the M respective image capture devices CAM. In addition, the projection unit 42 reduces the generated M effective area mask images according to the reduction ratio of the reduction processing performed by the reduction processing unit 82.

[0136] The reduction processing unit 82 selects M captured images corresponding to the M image capture devices CAM selected in the camera selection unit 81 from among the N captured images captured by the N image capture devices CAM provided by the image input unit 31. Then, the reduction processing unit 82 generates a low-resolution captured image (hereinafter, referred to as a low-resolution image) obtained by reducing the selected M captured images at a predetermined reduction ratio. For example, in the case where the captured image provided from the image input unit 31 has a resolution of 1920×1080 or a 4K resolution (which is four times the resolution of HD) called high definition (HD), the reduction processing unit 82 reduces the captured image to a low-resolution image of 640×480 called VGA. The method of the reduction processing is not particularly limited, and any method can be selected. For example, a nearest neighbor method, a linear interpolation method, a bicubic method, an area average method, etc. can be adopted. In addition, a method of simply thinning the number of pixels in each of the horizontal and vertical directions at a predetermined ratio such as 1 / 2 and 1 / 4 can be used.

[0137] The reduction ratio of the reduction processing unit 82 and the reduction ratio when the projection unit 42 reduces the effective area mask image are set to be the same in advance.

[0138] The contour extraction unit 34, semantic processing unit 35, and determination unit 36 ​​are similar to those of the first embodiment, except that processing is performed on M images selected from N images and the images have low resolution.

[0139] Other points of the second embodiment are similar to those of the above-described first embodiment.

[0140] <5. Second Image Capturing Range Determination Processing>

[0141] Next, we will refer to Fig.11 The flowchart in describes the second image capturing range determination processing performed by the second embodiment of the image processing apparatus 11. For example, this processing starts simultaneously with the start of image capturing in the N image capturing devices CAM.

[0142] Since the processing in step S21 and step S22 is similar to Fig. 9 The processing in step S1 and step S2 of the first image capturing range determination processing in FIG. 1 is described below, and thus a description thereof will be omitted.

[0143] In step S23, the camera selection unit 81 selects M (N>M) image capture devices CAM from the N image capture devices CAM. The camera selection unit 81 supplies camera selection information for specifying the selected M image capture devices CAM to the 3D area calculation unit 41 of the effective area calculation unit 33 and the reduction processing unit 82.

[0144] In step S24 , the 3D area included in the image capturing range of the M image capturing devices CAM selected in the camera selection unit 81 is calculated using the camera parameters of the M image capturing devices CAM.

[0145] In step S25, the projection unit 42 generates M effective area mask images obtained by projecting the 3D area calculated in the 3D area calculation unit 41 onto the viewpoints of the M respective image capture devices CAM, and reduces the effective area mask images at a predetermined reduction ratio. The M effective area mask images after the reduction process are provided to the determination unit 36. Note that, similar to the first embodiment, the projection unit 42 may generate visual recognition camera information.

[0146] In step S26, the reduction processing unit 82 selects M captured images corresponding to the M image capturing devices CAM selected in the camera selection unit 81 from among the N captured images provided by the image input unit 31, and reduces the M captured images at a predetermined reduction ratio. The M low-resolution images after reduction are provided to the contour extraction unit 34 and the semantic processing unit 35.

[0147] In step S27, the contour extraction unit 34 generates M contour images from the M low-resolution images supplied from the reduction processing unit 82. The generated M contour images are supplied to the determination unit 36.

[0148] In step S28, the semantic processing unit 35 performs semantic segmentation processing on the M low-resolution images supplied from the reduction processing unit 82 to generate semantic information, and supplies the semantic information to the determination unit 36. Note that this processing may be omitted when real-time processing performance is given priority.

[0149] In step S29, the determination unit 36 ​​performs a camera determination process for determining whether the subject is outside the image capturing range of the image capturing device CAM based on the M effective area mask images from the projection unit 42 and the M contour images from the contour extraction unit 34. This process is similar to the first image capturing range determination process except that M effective area mask images and contour images are used instead of N effective area mask images and contour images.

[0150] Since step S30 and step S31 are similar to Fig. 9Steps S8 and S9 of the first image capturing range determination process are described above, and thus description thereof will be omitted.

[0151] At this point, the second image capture range determination process ends. Also according to the second image capture range determination process, since the user can be notified when the subject is outside the image capture range of the multiple image capture devices CAM that capture the subject, failed image capture in multi-view image capture for 3D model generation can be reduced.

[0152] According to the second image capture range determination processing, since only the captured images of M image capture devices CAM out of N image capture devices CAM are used to perform camera determination processing for determining whether the subject is outside the image capture range of the image capture device CAM, the processing load can be reduced and the real-time processing performance can be improved.

[0153] Note that in the second embodiment, a processing unit that changes the resolution of a captured image to a low resolution is provided in the image processing apparatus 11, but the resolution setting of the image capture device CAM may be changed so that the captured image provided from the image capture device CAM has a low resolution. In this case, the captured image used for the 3D model generation process performed in parallel with the second image capture range determination process also has a low resolution. In the case where it is desired to set the captured image used for the 3D model generation process to a high-resolution image, the 3D model generation process only needs to be performed in a time division manner so that after the image capture range is confirmed in the second image capture range determination process, the resolution setting of the image capture device CAM is changed to a high-resolution setting.

[0154] <6. Third Embodiment of Image Processing Device>

[0155] Fig.12 : is a block diagram showing a configuration example of a third embodiment of the image processing device 11 .

[0156] exist Fig.12 In, with Figure 3 Portions corresponding to those of the first embodiment shown in FIG. 1 are marked with the same reference numerals, and descriptions of these portions will be appropriately omitted.

[0157] The image processing device 11 according to the third embodiment is common to the second embodiment in that it has a configuration that reduces arithmetic processing and reduces processing load. On the other hand, in the second embodiment, the process itself of the effective area calculation unit 33 generating the effective area mask image is not changed, and the number and resolution of captured images are changed, while the third embodiment is different from the second embodiment in that the process of the effective area calculation unit 33 generating the effective area mask image is changed.

[0158] When the general Fig.12 The configuration example of the third embodiment in Figure 3 When compared with the configuration example of the first embodiment shown in Figure 3 The 3D area calculation unit 41 and the projection unit 42 of the effective area calculation unit 33 are Fig.12 The 3D area calculation unit 91 and the projection unit 92 are replaced by the 3D area calculation unit 91 and the projection unit 92, and the other configurations are the same.

[0159] like Fig.13 As shown, the 3D area calculation unit 91 calculates the image capturing range of each of the N image capturing devices CAM by approximating the image capturing range to the shape of a quadrangular pyramid based on the camera parameters supplied from the calibration processing unit 32 . Fig.13 The range (Zmax and Zmin) in the depth direction in is set (input) as a parameter.

[0160] The 3D area calculation unit 91 calculates the coordinates of the eight vertices of the quadrangular pyramid for each of the image capturing devices CAM as the image capturing range of the image capturing device CAM, and supplies the coordinates to the projection unit 92 .

[0161] The projection unit 92 performs the following processing using each of the N image capturing devices CAM that generates the effective area mask image as a target camera.

[0162] The projection unit 92 projects the coordinates of the eight vertices of the pyramid of each of the N image capture devices CAM provided from the 3D area calculation unit 91 onto the viewpoint of the target camera to generate N pyramid projection areas obtained by projecting the simple 3D areas of the N pyramids onto the projection plane of the target camera.

[0163] Will refer to Fig.14 The processing of the projection unit 92 is described. Fig.14 In order to help understand the drawings, the following example will be described: assuming that the target camera is the image capturing device CAM-3, the image capturing ranges of the two image capturing devices CAM-1 and CAM-2 are projected onto the viewpoint of the image capturing device CAM-3.

[0164] Fig.14 A shows a state where the image capturing range of the image capturing device CAM-1 is viewed from the viewpoint of the image capturing device CAM-3 serving as the target camera. This state can be generated by projecting the image capturing range of the image capturing device CAM-1 onto the viewpoint of the image capturing device CAM-3 serving as the target camera.

[0165] Fig.14B shows a state where the image capturing range of the image capturing device CAM-2 is viewed from the viewpoint of the image capturing device CAM-3 serving as the target camera. This state can be generated by projecting the image capturing range of the image capturing device CAM-2 onto the viewpoint of the image capturing device CAM-3 serving as the target camera.

[0166] Then, the projection unit 92 generates an effective area mask image in which the area where all N quadrangular pyramid projection areas overlap is set as the projection 3D area of ​​the target camera.

[0167] Fig.14 C is a conceptual diagram showing an effective area mask image generated by overlapping the image capturing range of the image capturing device CAM- 1 with the image capturing range of the image capturing device CAM- 2 .

[0168] Fig.14 The effective area mask image in C corresponds to that in the first embodiment. Figure 5 The effective area mask image 52 is shown in FIG. Figure 5 The binary image of the effective area mask image 52 in . Fig.14 In C, the shaded area is an area where the image capturing range of the image capturing device CAM-1 overlaps with the image capturing range of the image capturing device CAM-2, and corresponds to the projected 3D area 53 of the target camera.

[0169] Also in the third embodiment, the projection unit 92 may generate visual recognition camera information. The visual recognition camera information in the third embodiment includes number information indicating the number of image capture devices CAM from which the projection plane (image capture range) of the target camera can be seen, and camera information indicating the image capture devices CAM from which the projection plane (image capture range) of the target camera can be seen.

[0170] Fig.14 D shows an example of number information indicating the number of image capturing devices CAM from which the image capturing range of the target camera can be seen.

[0171] exist Fig.14 In D, the image capturing range of the target camera is divided into four areas 101 to 104, and ... Fig.14 The number information corresponding to the projection states in A to C.

[0172] Since the area 101 is an area viewed only by the image capture device CAM-3 as the target camera, the number information is "1". Since the area 102 is an area viewed by both the image capture device CAM-3 and the image capture device CAM-2, the number information is "2". Since the area 103 is an area viewed by the image capture devices CAM-1 to CAM-3, the number information is "3". Since the area 104 is an area viewed by both the image capture device CAM-1 and the image capture device CAM-3, the number information is "2".

[0173] Fig.14 E shows an example of camera information indicating the image capturing device CAM from which the image capturing range of the target camera can be seen.

[0174] exist Fig.14 In E, the image capturing range of the image capturing device CAM-3 as the target camera is divided into four areas 101 to 104, and ... Fig.14 The projection states A to C correspond to the camera information.

[0175] Note that since the area is obviously visible from the own image capturing device CAM, only information indicating whether the area is visible from other image capturing devices CAM is recorded as camera information.

[0176] Since the area 101 is an area viewed only by the image capturing device CAM-3, there is no camera information for the area 101. For the area 102, camera information indicating that the area 102 is viewed from the image capturing device CAM-2 other than the own device is recorded. For the area 103, camera information indicating that the area 103 is viewed from the image capturing devices CAM-1 and CAM-2 other than the own device is recorded. For the area 104, camera information indicating that the area 104 is viewed from the image capturing device CAM-1 other than the own device is recorded.

[0177] Other points of the third embodiment are similar to those of the above-described first embodiment.

[0178] <7. Third Image Capturing Range Determination Processing>

[0179] In the third image capturing range determination process performed by the third embodiment of the image processing device 11, referring to Fig. 9 Steps S3 and S4 of the described first image-capturing range determination process are replaced with the following steps S3 ′ and S4 ′.

[0180] In step S3 ′, the 3D area calculation unit 91 calculates the image capturing range of each of the N image capturing devices CAM by approximating the image capturing range to a quadrangular pyramid based on the camera parameters supplied from the calibration processing unit 32 .

[0181] In step S4', the projection unit 92 performs the following processing using each of the N image capture devices CAM that generate the effective area mask image as a target camera. The projection unit 92 projects the coordinates of the eight vertices of the pyramid of each of the N image capture devices CAM provided from the 3D area calculation unit 91 onto the viewpoint of the target camera to generate N pyramid projection areas obtained by projecting the simple 3D areas of the N pyramids onto the projection plane of the target camera. Subsequently, the projection unit 92 generates an effective area mask image in which the area where all the N pyramid projection areas overlap is set as the projection 3D area of ​​the target camera.

[0182] Therefore, in the first image capturing range determination processing, the effective area calculation unit 33 calculates the image capturing ranges of N individual image capturing devices CAM, each of which is formed in the shape of a quadrangular pyramid, integrates all the N image capturing ranges to calculate the 3D area, and then projects the integrated 3D area onto the viewpoint of each image capturing device CAM to generate the effective area mask image of the target camera. That is, the effective area calculation unit 33 of the first embodiment performs processing in the order of calculating the N image capturing ranges, integrating, and projecting.

[0183] On the other hand, in the third image capturing range determination processing, the effective area calculation unit 33 calculates the image capturing ranges of N individual image capturing devices CAM, each of which is formed in the shape of a quadrangular pyramid, projects the image capturing ranges onto the viewpoints of the individual image capturing devices CAM to generate N quadrangular pyramid projection areas, and integrates the projected N quadrangular pyramid projection areas into the projected 3D area of ​​the target camera to generate an effective area mask image. That is, the effective area calculation unit 33 of the third embodiment performs processing in the order of calculating N image capturing ranges, projecting, and integrating.

[0184] It can be said that the 3D area calculation unit 41 of the first embodiment is a generation unit that generates a 3D area based on N captured images (multiple multi-view images) of N individual image capture devices CAM, wherein the image capture ranges of the N image capture devices CAM are integrated into the one 3D area.

[0185] It can be said that the 3D area calculation unit 91 of the third embodiment is a generation unit that generates N 3D areas of N individual image capturing devices CAM based on N captured images (multiple multi-view images) of the N individual image capturing devices CAM.

[0186] Also according to the third image capturing range determination processing, since the user can be notified when the subject is outside the image capturing range of the plurality of image capturing devices CAM that capture the subject, failed image capture in multi-view image capturing for 3D model generation can be reduced.

[0187] <8. Other examples of camera placement>

[0188] In each of the above embodiments, if Figure 1 As shown, it has been described that N image capturing devices CAM are arranged on the periphery of the subject to surround the subject, but the arrangement of the image capturing devices CAM is not limited thereto.

[0189] For example, Fig.15 As shown in A, the multiple image capture devices CAM can be arranged to face outward, or as Fig.15 As shown in B, a plurality of image capturing devices CAM may be arranged to face a specific direction in a focused manner, such as in a stage of a concert hall.

[0190] <9. Example of computer configuration>

[0191] The above series of processing can be performed by hardware or software. In the case of performing a series of processing by software, a program including the software is installed in a computer. Here, for example, the computer includes a microcomputer combined in dedicated hardware, a general-purpose personal computer capable of performing various functions by installing various programs, etc.

[0192] Fig.16 : is a block diagram showing a configuration example of hardware of a computer that executes the above-described series of processes with the help of a program.

[0193] In the computer, a central processing unit (CPU) 301 , a read only memory (ROM) 302 , and a random access memory (RAM) 303 are connected to each other via a bus 304 .

[0194] An input / output interface 305 is also connected to the bus 304 . An input unit 306 , an output unit 307 , a storage unit 308 , a communication unit 309 , and a drive 310 are connected to the input / output interface 305 .

[0195] The input unit 306 includes a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 307 includes a display, a speaker, an output terminal, etc. The storage unit 308 includes a hard disk, a RAM disk, a nonvolatile memory, etc. The communication unit 309 includes a network interface, etc. The drive 310 drives a removable recording medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0196] In the computer configured as described above, for example, the CPU 301 loads the program stored in the storage unit 308 into the RAM 303 via the input / output interface 305 and the bus 304, and executes the program so that the above series of processing is performed. The RAM 303 also appropriately stores data and the like necessary for the CPU 301 to perform various processing.

[0197] For example, the program can be provided by recording the program executed by the computer (CPU 301) in the removable medium 311 as a package medium, etc. Furthermore, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, and digital satellite broadcasting.

[0198] In the computer, by attaching the removable recording medium 311 to the drive 310, the program can be installed in the storage unit 308 via the input / output interface 305. In addition, the program can be received in the communication unit 309 via a wired or wireless transmission medium and installed in the storage unit 308. In addition, the program can be pre-installed in the ROM 302 or the storage unit 308.

[0199] Note that in this specification, the steps described in the flowchart may be executed in chronological order according to the described order, or may not necessarily be executed in chronological order, and may be executed in parallel or at a necessary time (eg, when a call is executed).

[0200] In this specification, a system refers to a group of multiple components (devices, modules (components), etc.), and it is not important whether all components are in the same housing. Therefore, a collection of multiple devices housed in separate housings and connected via a network and a device with multiple modules housed in one housing are both systems.

[0201] The embodiments of the present technology are not limited to the above-mentioned embodiments, and various changes may be made without departing from the scope of the present technology.

[0202] For example, a mode in which all or some of the above-described embodiments are combined may be adopted.

[0203] For example, the present technology may be configured as cloud computing in which one function is shared and jointly processed by a plurality of devices via a network.

[0204] Furthermore, each step described in the above-mentioned flowchart may be performed not only by one device but also shared and performed by a plurality of devices.

[0205] Furthermore, in the case where one step includes a plurality of processes, the plurality of processes included in the step may be executed not only by one device but also by being shared and executed by a plurality of devices.

[0206] Note that the effects described in this specification are merely illustrative and should not be limiting, and there may be effects other than the effects described in this specification.

[0207] Note that the present technology can adopt the following configurations. (1)

[0209] An image processing device, comprising:

[0210] a generating unit that generates a 3D area of ​​an image capturing range generated from a plurality of multi-view images; and

[0211] A determination unit determines a condition in which the image capture device captures a subject based on an area image obtained by projecting the 3D area onto a specific viewpoint and a subject image from the image capture device corresponding to the specific viewpoint. (2)

[0213] According to the image processing device described in (1) above,

[0214] The determination unit uses the area image and the subject image as a pair to determine whether the subject is outside the image capturing range of the plurality of image capturing devices that have captured the plurality of multi-view images. (3)

[0216] According to the image processing device described in (1) or (2) above,

[0217] The determination unit uses the area image and the subject image as a pair to determine whether the subject will be outside the image capturing range of the plurality of image capturing devices that have captured the plurality of multi-view images. (4)

[0219] According to the image processing device described in (2) or (3) above,

[0220] Wherein, in a case where the subject is outside the image capturing ranges of the plurality of image capturing devices, the determining unit further determines which image capturing device the subject is outside the image capturing range of. (5)

[0222] The image processing device according to any one of (1) to (4) above,

[0223] The determination unit uses the object image to determine whether the object is outside an image capturing range of an image capturing device corresponding to the specific viewpoint. (6)

[0225] The image processing device according to any one of (1) to (5) above, further comprising

[0226] a semantic processing unit, which identifies semantic information of the subject,

[0227] The determining unit uses semantic information of the object to perform determination. (7)

[0229] The image processing device according to any one of (1) to (6) above, further comprising

[0230] a calculation unit that calculates camera parameters of a plurality of image capture devices that have captured the plurality of multi-view images based on the plurality of multi-view images,

[0231] The generating unit generates the 3D area of ​​the image capturing range based on the calculated camera parameters of the plurality of image capturing devices. (8)

[0233] According to the image processing device described in (7) above,

[0234] The generating unit generates a 3D area, which is a 3D area included in the image capturing ranges of all the image capturing devices that have captured the plurality of multi-view images. (9)

[0236] The image processing device according to any one of (1) to (8) above, further comprising

[0237] A projection unit generates a region image obtained by projecting the 3D region onto the specific viewpoint. (10)

[0239] The image processing device according to any one of (1) to (9) above,

[0240] Wherein, the region image is a binary image. (11)

[0242] The image processing device according to any one of (1) to (10) above, further comprising

[0243] a contour extraction unit that generates a contour image obtained by performing contour extraction on a region of the subject from each of the plurality of multi-view images,

[0244] The determination unit determines a condition in which the image capture device captures the subject based on the area image and the contour image as the subject image. (12)

[0246] The image processing device according to any one of (1) to (11) above, further comprising

[0247] a selection unit that selects a plurality of image capture devices from a plurality of image capture devices corresponding to the plurality of multi-view images,

[0248] wherein the generating unit generates the 3D area of ​​the image capturing range generated based on the multi-view images corresponding to the selected plurality of image capturing devices, and

[0249] A condition in which the image capture device captures the subject is determined based on an area image obtained by projecting the 3D area onto the specific viewpoint and a subject image from an image capture device corresponding to the specific viewpoint. (13)

[0251] The image processing device according to any one of (1) to (12) above, further comprising

[0252] a reduction processing unit that reduces the plurality of multi-view images at a predetermined reduction ratio,

[0253] The determining unit determines a condition in which the image capturing device captures the object based on the reduced area image and the object image. (14)

[0255] The image processing device according to any one of (1) to (13) above,

[0256] wherein the generating unit generates a plurality of 3D regions obtained by approximating respective image capturing ranges of a plurality of image capturing devices corresponding to the plurality of multi-view images, and

[0257] The determination unit determines a condition in which the image capture device captures the subject based on an area image obtained by projecting the plurality of 3D areas onto the specific viewpoint and a subject image from the image capture device corresponding to the specific viewpoint. (15)

[0259] According to the image processing device described in (14) above,

[0260] The 3D area has a shape of a quadrangular pyramid. (16)

[0262] The image processing device according to any one of (1) to (15) above, further comprising

[0263] A notification unit notifies a user of a determination result of the determination unit. (17)

[0265] According to the image processing device described in (16) above,

[0266] wherein the notification unit refers to the determination result stored in a predetermined storage unit and notifies the user of the determination result. (18)

[0268] A 3D model generation method for generating a 3D model of a subject, comprising:

[0269] generating a 3D region of an image capture range generated from a plurality of multi-view images;

[0270] determining a condition in which the image capture device captures the subject based on an area image obtained by projecting the 3D area onto a specific viewpoint and a subject image from an image capture device corresponding to the specific viewpoint; and

[0271] A 3D model of the object is generated based on the multiple multi-view images. (19)

[0273] A program that causes a computer to:

[0274] a generating unit that generates a 3D area of ​​an image capturing range generated from a plurality of multi-view images, and

[0275] A determination unit determines a condition in which the image capture device captures a subject based on an area image obtained by projecting the 3D area onto a specific viewpoint and a subject image from the image capture device corresponding to the specific viewpoint.

[0276] Reference numerals list

[0277] CAM-1 to CAM-8 Image Capture Devices

[0278] 11 Image processing device

[0279] 12 Display Devices

[0280] 31 Image input unit

[0281] 32 Calibration Processing Unit

[0282] 33 Effective area calculation unit

[0283] 34 Contour extraction unit

[0284] 35 Semantic Processing Unit

[0285] 36 Determine unit

[0286] 37 Notification Unit

[0287] 38 3D model calculation unit

[0288] 41 3D area calculation unit

[0289] 42 projection units

[0290] 81 Camera selection unit

[0291] 82 Reduce processing unit

[0292] 91 3D area calculation unit

[0293] 92 projection units

[0294] 301 CPU

[0295] 302 ROM

[0296] 303 RAM

[0297] 306 Input Unit

[0298] 307 Output Unit

[0299] 308 Storage Unit

[0300] 309 Communication Unit

[0301] 310 Driver

Claims

1. An image processing device, comprising: a generating unit that generates a 3D area, wherein a plurality of multi-view images are obtained by capturing a subject from different viewpoints by a plurality of image capturing devices, the 3D area being an area commonly included in image capturing ranges of all the image capturing devices; and a determination unit that determines a condition in which the image capture device captures a subject based on an area image obtained by projecting the 3D area onto a viewpoint of each of the plurality of image capture devices and a subject image from each image capture device, The determination unit uses the area image and the object image associated with each of the plurality of image capture devices as a pair to determine whether the object is outside an image capture range of the plurality of image capture devices.

2. The image processing device according to claim 1, in, The determination unit determines whether the subject will be outside the image capturing range of the plurality of image capturing devices using the area image and the subject image associated with each of the plurality of image capturing devices as a pair.

3. The image processing device according to claim 1, in, In a case where the subject is outside the image capturing ranges of the plurality of image capturing devices, the determination unit further determines which image capturing device the subject is outside the image capturing range of.

4. The image processing apparatus according to claim 1, further comprising a semantic processing unit, which identifies semantic information of the subject, in, The determination unit performs determination using semantic information of the object.

5. The image processing apparatus according to claim 1, further comprising a calculation unit that calculates camera parameters of a plurality of image capture devices that have captured the plurality of multi-view images based on the plurality of multi-view images, in, The generating unit generates the 3D area based on the calculated camera parameters of the plurality of image capturing devices.

6. The image processing apparatus according to claim 1, further comprising A projection unit generates an area image obtained by projecting the 3D area onto a viewpoint of each image capturing device.

7. The image processing device according to claim 1, in, The region image is a binary image.

8. The image processing apparatus according to claim 1, further comprising a contour extraction unit that generates a contour image obtained by performing contour extraction on a region of the subject from each of the plurality of multi-view images, in, The determination unit determines a condition in which the image capturing device captures the subject based on the area image and the contour image as the subject image.

9. The image processing apparatus according to claim 1, further comprising a selection unit that selects a plurality of image capture devices from a plurality of image capture devices corresponding to the plurality of multi-view images, in, The generating unit generates a 3D area commonly included in the image capturing ranges of the selected plurality of image capturing devices, and Based on an area image obtained by projecting the 3D area onto a viewpoint of each of the selected several image capture devices and an object image from each of the selected several image capture devices, a condition in which the image capture device captures the object is determined.

10. The image processing apparatus according to claim 1, further comprising a reduction processing unit that reduces the plurality of multi-view images at a predetermined reduction ratio, in, The determination unit determines a condition in which the image capture device captures the subject based on the reduced area image and the subject image.

11. The image processing device according to claim 1, in, The generating unit generates a plurality of 3D areas obtained by approximating respective image capturing ranges of a plurality of image capturing devices corresponding to the plurality of multi-view images, and The determination unit determines a condition in which the image capture device captures the subject based on an area image obtained by projecting the plurality of 3D areas onto a viewpoint of each of the plurality of image capture devices and a subject image from each image capture device.

12. The image processing device according to claim 11, in, The 3D area has a shape of a quadrangular pyramid.

13. The image processing apparatus according to claim 1, further comprising A notification unit notifies a user of a determination result of the determination unit.

14. The image processing device according to claim 13, in, The notification unit refers to the determination result stored in a predetermined storage unit, and notifies the user of the determination result.

15. A 3D model generation method for generating a 3D model of a subject, comprising: generating a 3D region, wherein a plurality of multi-view images are obtained by capturing a subject from different viewpoints by a plurality of image capturing devices, and the 3D region is a region commonly included in image capturing ranges of all image capturing devices; determining a condition in which the image capture device captures the subject based on an area image obtained by projecting the 3D area onto a viewpoint of each of the plurality of image capture devices and a subject image from each image capture device; and generating a 3D model of the object based on the multiple multi-view images, Wherein, the area image and the object image associated with each of the plurality of image capturing devices are used as a pair to determine whether the object is outside the image capturing range of the plurality of image capturing devices.

16. A recording medium having a program recorded thereon, wherein when the program is executed by a computer, the computer is caused to function as: Generate a unit to generate a 3D region, where A plurality of multi-view images are obtained by capturing a subject from different viewpoints by a plurality of image capturing devices, the 3D area being an area commonly included in image capturing ranges of all the image capturing devices, and a determination unit that determines a condition in which the image capture device captures a subject based on an area image obtained by projecting the 3D area onto a viewpoint of each of the plurality of image capture devices and a subject image from each image capture device, The determination unit uses the area image and the object image associated with each of the plurality of image capture devices as a pair to determine whether the object is outside an image capture range of the plurality of image capture devices.

Images