Image processing device, image processing method, and program

Abstract

To solve the problem that pieces of three-dimensional data displayed in time series generate a feeling of strangeness when an anomaly is detected in an image to be used for generating the three-dimensional data.SOLUTION: An image processing apparatus is configured to: obtain a plurality of silhouette images each representing a region of an object in each of images captured by a plurality of imaging apparatuses, the number of silhouette images corresponding to the number of frames constituting a video image; specify, when an anomaly is detected in the acquired silhouette images, a frame and an imaging apparatus corresponding to a silhouette image in which the anomaly has been detected; and generates, when generating three-dimensional data in multiple frames corresponding to a period from the detection of the anomaly to satisfaction of a predetermined condition, using a silhouette image corresponding to an imaging apparatus different from the specified imaging apparatus, without using a silhouette image corresponding to the specified imaging apparatus.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to generating data based on captured images. [Background technology]

[0002] There is a method for generating three-dimensional shape data (hereinafter sometimes referred to as a three-dimensional model) representing the three-dimensional shape of an object present in an imaging space based on multiple captured images obtained by multiple imaging devices arranged around the imaging space.

[0003] Patent Document 1 describes a method of extracting a foreground region from images captured by a plurality of cameras and generating a three-dimensional model of the foreground object by a volume intersection method.

[0004] Patent Document 2 describes generating a three-dimensional model without using a foreground image corresponding to a camera that has detected an abnormality. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2017-211828 [Patent Document 2] JP 2018-63500 A Summary of the Invention [Problem to be solved by the invention]

[0006] When comparing a 3D model generated using foreground images corresponding to all cameras installed with a 3D model generated without using foreground images corresponding to some cameras, some parts of the shape may differ even for stationary objects.

[0007] For example, suppose that frames that should be output from a certain camera are intermittently missing. In this case, the method of Patent Document 2 generates a 3D model using foreground images corresponding to all cameras if frames are output normally from all cameras, but generates a 3D model without using the foreground image of a certain camera if frames are missing from that camera. As a result, the method of Patent Document 2 may generate 3D models of stationary objects with partially different shapes alternately within a short period of time. In this case, if the generated 3D models are displayed in the order in which they were generated to display them as a video, flickering will occur. This may result in an image that gives a sense of discomfort to the viewer. [Means for solving the problem]

[0008] The image processing device of the present disclosure includes an acquisition means for acquiring a plurality of images representing an area of ​​an object corresponding to a plurality of imaging devices, the plurality of images corresponding to a plurality of frames constituting a moving image; an identification means for detecting an abnormality in at least one of the plurality of images acquired by the acquisition means and identifying the frame and imaging device corresponding to the image in which the abnormality was detected; and a generation means for generating three-dimensional shape data of the object using the plurality of images, wherein when generating the three-dimensional shape data for a plurality of frames corresponding to a period from when the abnormality was detected until a predetermined condition is satisfied, the generation means does not use the images corresponding to the imaging device identified by the identification means, but generates the three-dimensional shape data using images corresponding to imaging devices different from the imaging device identified by the identification means, and the predetermined condition is when the abnormality is not detected in the images corresponding to the imaging device identified by the identification means for a number of consecutive frames corresponding to a predetermined number of frames. The predetermined number of frames is changed depending on the type of the cause of the abnormality, and the predetermined number of frames is determined depending on the length of recovery from the abnormality. It is characterized by: [Effects of the Invention]

[0009] According to the technology of the present disclosure, even if an abnormality is detected in an image used to generate three-dimensional shape data, it is possible to reduce the sense of incongruity that may occur when the three-dimensional shape data is displayed in chronological order. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing the configuration of an image generation system. [Figure 2] FIG. 1 is a diagram showing the hardware configuration of a three-dimensional model generating device. [Figure 3] FIG. 1 is a block diagram showing the functional configuration of a three-dimensional model generating device. [Figure 4] 1A and 1B are diagrams for explaining a method for generating a three-dimensional model using a volume intersection method. [Figure 5] FIG. 1 is a diagram for explaining a three-dimensional model of a foreground using voxels. [Figure 6] 10 is a flowchart illustrating a process for generating a three-dimensional model. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, the technology of the present disclosure will be described in detail based on embodiments with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the technology of the present disclosure is not limited to the configurations shown in the drawings.

[0012] In addition, terms that differ only in the alphabet following the reference number indicate different instances of devices with the same function, and the alphabet in the reference number may be omitted when referring to any of the devices with the same function.

[0013] <Embodiment 1> [System Configuration] FIG. 1 is a diagram illustrating an example of an image processing system 100 that generates a virtual viewpoint image. The virtual viewpoint image is an image that represents a view from a virtual viewpoint independent of the viewpoint of an actual imaging device. The virtual viewpoint image is generated using multiple images obtained by capturing images time-synchronized from multiple viewpoints by installing multiple imaging devices at different locations. The virtual viewpoint image allows a user to view highlight scenes of a sport such as soccer from various angles, thereby providing the user with a more realistic feeling than a normal captured image. The virtual viewpoint image may be a video or a still image. In the following embodiment, the virtual viewpoint image will be described as a video.

[0014] The image processing system 100 includes imaging devices 101a-d, foreground extraction devices 102a-d connected to the imaging devices 101a-d, respectively, a three-dimensional model generation device 200, a storage device 300, a rendering device 400, and a virtual viewpoint path generation device 500.

[0015] The imaging devices 101a-d are devices such as digital video cameras that capture images (videos). Each of the imaging devices 101a-d receives, for example, a GPS or external synchronization signal and captures images at the same time. The imaging devices 101a-d capture images of objects present in the imaging environment from multiple directions and at various angles, and output image data of the captured images to the foreground extraction devices 102a-d. While FIG. 1 shows an example in which four imaging devices 101a-d are arranged, the number of imaging devices in the image processing system 100 is not limited to four. Any number of imaging devices may be arranged. Furthermore, the imaging devices 101a-d may be devices that capture not only images but also audio and other sensor information. Furthermore, unless otherwise specified, when referring to any of the imaging devices 101a-d, the alphabet in the reference number may be omitted and the device may be referred to as the imaging device 101.

[0016] The foreground extraction devices 102a-d are image processing devices corresponding to the respective imaging devices 101a-d. Captured images obtained as a result of imaging by the corresponding imaging device 101 are input to the foreground extraction devices 102a-d. The foreground extraction devices 102a-d perform image processing on the input captured images. The image processing performed by the foreground extraction devices 102a-d includes processing to extract foreground areas that indicate the silhouettes of objects included in the input captured images. Then, silhouette images that indicate the foreground and non-foreground areas included in the captured images using binary values ​​are generated. Furthermore, texture images of the foreground objects are generated. The foreground silhouette image and foreground texture image are collectively referred to as object images or foreground data.

[0017] The object represented as the foreground in the captured image is a subject that can be viewed from any angle of the virtual viewpoint, and may be, for example, a person on a stadium field. Alternatively, the object may be an object with a predetermined image pattern, such as a ball or a goal. The object may be either a moving object or a stationary object. The type of object to be extracted as the foreground can be specified in advance by a user or the like.

[0018] One method for extracting the foreground from a captured image is to use background difference information. In this method, for example, an imaging environment in which no objects exist is captured and stored in advance as a background image. Then, an area in which the difference in pixel values ​​between the captured image and the background image is greater than a threshold is determined to be the foreground. Note that the method for extracting the foreground is not limited to the method using background difference information. Other methods for extracting the foreground may also be used, such as a method using parallax, a method using feature values, or a method using machine learning. The generated foreground data is output to the three-dimensional model generation device 200.

[0019] 1 shows an example in which four foreground extraction devices 102a-d are arranged, but the number of foreground extraction devices is not limited to four. Any number of foreground extraction devices may be arranged. Furthermore, in this embodiment, the foreground extraction device 102 and the image capture device 101 are different devices, but they may be integrated into one device, or may be realized by different devices for each function. Unless otherwise specified, when referring to any of the foreground extraction devices 102a-d, the alphabet in the reference number may be omitted and the device may be referred to as the foreground extraction device 102.

[0020] The three-dimensional model generation device 200 is an image processing device realized by a computer such as a PC, workstation, or server. The three-dimensional model generation device 200 acquires foreground data based on captured images (frames) obtained by capturing different field-of-view ranges from the foreground extraction devices 102a-d. The three-dimensional model generation device 200 then generates three-dimensional shape data of an object based on the foreground data. The three-dimensional shape data of an object represented by voxels in the three-dimensional shape data is also called a three-dimensional model. The generated three-dimensional model is output in a point cloud format, which is a collection of voxels in three-dimensional space. The format of the three-dimensional model may also be a mesh format or another different format.

[0021] The storage device 300 is a device realized by a computer such as a PC, a workstation, or a server. The storage device 300 is a device that stores point cloud format data, which is a 3D model generated by the 3D model generation device 200, and foreground data generated by the foreground extraction device 102. The storage unit 301 for storing data in the storage device 300 may be realized by any of volatile memory, non-volatile memory, a hard disk, and an SSD. Note that in this embodiment, the storage unit 301 of the storage device 300 will be described as storing data output from the 3D model generation device 200. Alternatively, the 3D model generation device 200 may output data directly to the rendering device 400. In this case, the image processing system 100 may be configured without the storage device 300.

[0022] The rendering device 400 is an image processing device realized by a computer such as a PC, a workstation, a server, etc. The rendering device 400 generates a virtual viewpoint image based on point cloud format data (three-dimensional model) stored in the storage device 300, foreground data, and a virtual viewpoint path generated by a virtual viewpoint path generating device 500 (described later).

[0023] The virtual viewpoint path includes data such as the location, angle of view, movement speed, playback speed, etc. From the data of the virtual viewpoint path, the rendering device 400 acquires point cloud format data (three-dimensional model) for generating an image corresponding to the viewpoint, and a texture image included in the foreground data, from the storage device 300. Then, the rendering device 400 generates a virtual viewpoint image by projecting the texture of the foreground data onto the three-dimensional shape indicated by the point cloud format data.

[0024] The virtual viewpoint path generating device 500 is a device realized by a computer such as a PC, a workstation, a server, etc. The virtual viewpoint path generating device 500 is a device that generates information about a virtual viewpoint (virtual viewpoint path information), which is a viewpoint from a virtual camera for generating a virtual viewpoint image, and outputs the generated virtual viewpoint information to the rendering device 400.

[0025] The 3D model generation device 200, foreground extraction device 102, storage device 300, rendering device 400, and virtual viewpoint path generation device 500 are connected via a LAN network. In this embodiment, the network topology is described as a daisy chain connection, but the network topology is not limited to a daisy chain connection. Star, bus, or mesh type connections via relay devices such as hubs may also be used. Furthermore, the connections between devices are not limited to a LAN network; they may be other wired connections, wireless LANs, public wireless networks, or the Internet. Furthermore, image transmission interfaces such as SDI (Serial Digital Interface) and DVI (Digital Visual Interface) may also be used.

[0026] [Hardware configuration] 2 is a diagram showing the hardware configuration of the 3D model generation device 200. The hardware configurations of the foreground extraction device 102, the rendering device 400, and the virtual viewpoint path generation device 500 are also similar to the configuration of the 3D model generation device 200 described below.

[0027] The three-dimensional model generating device 200 includes a CPU 211 , a ROM 212 , a RAM 213 , an auxiliary storage device 214 , a display unit 215 , an operation unit 216 , a communication I / F 217 , and a bus 218 .

[0028] The CPU 211 controls the entire three-dimensional model generation device 200 using computer programs and data stored in the ROM 212 and RAM 213, thereby realizing each function shown in Fig. 3. Note that the three-dimensional model generation device 200 may have one or more dedicated hardware pieces different from the CPU 211, and at least a part of the processing by the CPU 211 may be executed by the dedicated hardware. Examples of dedicated hardware include an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), and a DSP (digital signal processor).

[0029] The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, and data supplied from the outside via the communication I / F 217. The auxiliary storage device 214 is configured, for example, with a hard disk drive or the like, and stores various data such as image data and audio data.

[0030] The display unit 215 is configured with, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) or the like for the user to operate the three-dimensional model generation device 200. The operation unit 216 is configured with, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and receives operations by the user to input various instructions to the CPU 211. The CPU 211 operates as a display control unit that controls the display unit 215 and an operation control unit that controls the operation unit 216. In this embodiment, the display unit 215 and the operation unit 216 are described as being present inside the three-dimensional model generation device 200, but at least one of the display unit 215 and the operation unit 216 may be present as a separate device outside the three-dimensional model generation device 200.

[0031] The communication I / F 217 is used for communication with devices external to the three-dimensional model generation device 200. For example, if the three-dimensional model generation device 200 is connected to an external device via a wired connection, a communication cable is connected to the communication I / F 217. If the three-dimensional model generation device 200 has a function for wireless communication with external devices, the communication I / F 217 is equipped with an antenna. The bus 218 connects each part of the three-dimensional model generation device 200 to transmit information.

[0032] [Function Configuration] 3 is a block diagram showing the functional configuration of the three-dimensional model generating device 200, the storage device 300, and the rendering device 400. The functions of each device constituting the image processing system 100 will be described with reference to FIG.

[0033] The three-dimensional model generating device 200 includes a foreground data acquiring unit 201 , a camera specifying unit 202 , a monitoring unit 203 , a three-dimensional model generating unit 204 , a camera parameter setting unit 205 , and a frame interpolating unit 206 .

[0034] The foreground data acquisition unit 201 acquires foreground data corresponding to the imaging device 101 from each of the multiple foreground extraction devices 102. Note that, although the present embodiment will be described assuming that one 3D model generation device 200 acquires foreground data from multiple foreground extraction devices 102, the number of devices that acquires foreground data is not limited to one. When multiple 3D model generation devices 200 are arranged on a network, the multiple 3D model generation devices 200 may acquire foreground data.

[0035] The camera identifying unit 202 checks the foreground data acquired from the multiple foreground extraction devices 102 and identifies whether any foreground data corresponding to the image capture devices 101a-d has not been acquired. That is, the foreground data acquiring unit 201 normally acquires the multiple pieces of data corresponding to the multiple image capture devices 101a-b in amounts corresponding to frames, but there are cases where the foreground data corresponding to one of the image capture devices cannot be acquired. In this case, the camera identifying unit 202 identifies the image capture device and frame corresponding to the foreground data that has not been acquired, and the foreground data that has not been acquired is identified as abnormal foreground data.

[0036] Alternatively, the camera identification unit 202 may be configured to detect whether an abnormality has occurred in the acquisition of foreground data, and identify the frame corresponding to the detected abnormality and the imaging device corresponding to the detected abnormality. In this case, if the foreground data cannot be acquired, it may be detected as an abnormality.

[0037] Camera identifying unit 202 notifies monitoring unit 203, which will be described later, of the identified imaging device. Furthermore, camera identifying unit 202 transfers, from among the foreground data transmitted from multiple foreground extraction devices 102, foreground data transmitted from foreground extraction devices 102 corresponding to imaging devices not subject to monitoring by monitoring unit 203 to three-dimensional model generation unit 204. Furthermore, camera identifying unit 202 transfers the foreground data transmitted from foreground extraction devices 102 to storage device 300. When transferring the foreground data to storage device 300, camera identifying unit 202 may add additional information for determining whether the foreground data is subject to monitoring by monitoring unit 203. Alternatively, camera identifying unit 202 may not transfer data corresponding to imaging devices subject to monitoring by monitoring unit 203 to storage device 300.

[0038] The monitoring unit 203 acquires information about the image capturing device 101 identified by the camera identifying unit 202, and registers the image capturing device identified by the camera identifying unit 202 as a monitoring target. The monitoring unit 203 has a function of monitoring foreground data generated based on images captured by the monitoring target image capturing device. The method of registration and exclusion from monitoring targets will be described later.

[0039] The frame complementing unit 206 generates complementary data that is data that substitutes for foreground data corresponding to an imaging device that is registered as a monitoring target by the monitoring unit 203. The frame complementing unit 206 generates complementary data that substitutes for foreground data corresponding to the imaging device based on foreground data of a frame of the imaging device 101 that is a monitoring target, before the imaging device is registered as a monitoring target or after the imaging device is excluded from the monitoring target. In this case, taking into account that the foreground object moves, the foreground region may be expanded to generate new complementary data. The extent to which the foreground region is expanded may be set based on the amount of movement of the foreground object, etc.

[0040] Furthermore, if foreground data corresponding to the imaging device to be monitored can be acquired, frame complementing unit 206 may generate complemented data based on the foreground data. In this case, taking into consideration the possibility that the foreground data may not be normal, the foreground area may be expanded to generate complemented data.

[0041] Alternatively, instead of generating complementary data based on other foreground data, a silhouette image whose entire surface indicates the area of ​​the object may be generated as complementary data for the silhouette image corresponding to the imaging device to be monitored.

[0042] The camera parameter setting unit 205 sets information on the multiple image capture devices 101 as camera parameters and outputs them to the three-dimensional model generation unit 204. The camera parameters include the three-dimensional position of each image capture device 101 in world coordinates, its attitude (the orientation of the image capture device), focal length, and principal point (the center of the captured image). Note that the camera parameters are measured and calculated, for example, by capturing an image of a calibration pattern in advance for calibrating the position of the image capture device, and performing coordinate conversion between three-dimensional coordinates defined based on the calibration pattern and two-dimensional image capture unit coordinates.

[0043] The 3D model generation unit 204 generates a 3D model of the object using the camera parameters output from the camera parameter setting unit 205 and the foreground data output from the camera identification unit 202. If the frame interpolation unit 206 generates interpolated data, the 3D model generation unit 204 may use the interpolated data instead of the foreground data to generate the 3D model. In this embodiment, the description will be given assuming that VisualHull is used as a method for generating a 3D model. VisualHull will be described later.

[0044] The three-dimensional model generation unit 204 stores information about the generated three-dimensional model in the storage device 300 in a point cloud format. The three-dimensional model generation unit 204 also stores, from the foreground data output from the camera identification unit 202, foreground data used to generate a virtual viewpoint image in the storage device 300. The three-dimensional model generation unit 204 generates a three-dimensional model corresponding to a plurality of frames constituting a moving image obtained from the imaging device 101. Therefore, the three-dimensional model generation unit 204 periodically performs a process of generating three-dimensional data every time it acquires foreground data based on a frame, and the point cloud format (three-dimensional model) and foreground data are periodically written to the storage unit 301.

[0045] The rendering device 400 includes a rendering unit 401 , an output unit 402 , and a virtual viewpoint path acquisition unit 403 .

[0046] The virtual viewpoint path acquisition unit 403 acquires the virtual viewpoint path information output from the virtual viewpoint path generation device 500 and notifies the result to the rendering unit 401. The virtual viewpoint path information includes data such as the position, angle of view, and movement speed of the virtual viewpoint desired by the user. The virtual viewpoint path information is generated in the same format as the camera parameters.

[0047] The rendering unit 401 acquires the point cloud format and foreground data stored in the storage device 300, and the virtual viewpoint path information output from the virtual viewpoint path acquisition unit 403. Then, it performs processing to generate a virtual viewpoint image by rendering in accordance with the point cloud format (three-dimensional model), foreground data, and virtual viewpoint path information. Specifically, it colors the three-dimensional shape indicated by the acquired point cloud format based on a texture image, and outputs the state projected onto two-dimensional coordinates from a virtual viewpoint based on the virtual viewpoint path information as the rendering result.

[0048] The output unit 402 converts the rendering result generated by the rendering unit 401 into an image format and outputs it. The image format to be output is DVI, but any image format is acceptable. In addition to DVI, SDI or video files may be output as streaming data over a network.

[0049] Each of the above-described functional units is realized by the CPU of each device executing a predetermined program, but this is not limiting. Other hardware, such as a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array), may also be used to speed up calculations. Each functional unit may be realized by a combination of software and hardware, such as a dedicated IC, or some or all of the functions may be realized solely by hardware.

[0050] The storage device 300 includes a memory unit 301. The memory unit 301 has a function of storing the point cloud format generated by the three-dimensional model generation unit 204 and the foreground data acquired by the foreground data acquisition unit 201 in association with each other.

[0051] 3 will be described as being realized by the devices described above, but the devices by which the functions of the units are realized are not limited. For example, the 3D model generation device 200 may have all of the functions described above. Alternatively, the 3D model generation device 200 or the rendering device 400 may have the storage unit 301, and in that case, the 3D model generation device 200 and the rendering device 400 may be connected without the storage device 300.

[0052] [How to generate 3D shape data] Figure 4 shows the basic principles of VisualHull. Using Figure 4, we will explain an overview of VisualHull, which is used to generate 3D models. Compared to methods that match feature amounts calculated from images from each imaging device and generate a 3D model from the positional relationship between those feature amounts, VisualHull is a method that, in principle, has lower concave accuracy but allows for faster processing. Note that whether VisualHull or a method that uses feature amount matching is used, the more imaging devices are placed in the vicinity, the more accurate the 3D model will be, but the amount of image data captured by the imaging devices will increase.

[0053] 4(a) is a diagram of an image captured by an imaging device of an object, that is, a target object C. As described above, a silhouette image including a two-dimensional silhouette (foreground region) of the target object C is obtained by binarizing the captured image based on the difference in color or brightness between the captured image obtained by capturing the target object C and the background image.

[0054] Figure 4(b) shows a cone extending into three-dimensional space from the projection center (Pa) of the imaging device through each point on the contour of the two-dimensional silhouette Da. This cone is called the view volume Va of the imaging device. Figure 4(c) shows how a three-dimensional model of an object is calculated from multiple view volumes. As shown in Figure 4(c), multiple view volumes are calculated for each imaging device from a two-dimensional silhouette Da based on images captured synchronously by multiple different imaging devices at different positions. In generating a three-dimensional model using VisualHull, a three-dimensional model of the target object is generated by determining the intersection (common area) of the view volumes corresponding to the multiple imaging devices. The generated three-dimensional model is represented by a collection of voxels.

[0055] Examples of computation algorithms using VisualHull include the Volume Intersection Method (VIM) and the Space Carving Method (SCM). In this embodiment, the three-dimensional model generation unit 204 will be described as using the SCM.

[0056] FIG. 5 is a diagram illustrating an overview of the generation of a 3D model by the 3D model generation unit 204. FIG. 5(a) is a diagram showing voxels, which are tiny cubes as shown in the figure. As shown in FIG. 5(b), the generation target space for generating a 3D model is tiled with voxels. In SCM, when individual voxels in the generation target space are back-projected onto the plane of each image capture device 101, a silhouette in / out determination process is performed in which voxels that are back-projected within the foreground regions (silhouettes) of all image capture devices 101 are retained as foreground voxels and the rest are deleted. A threshold may be set so that voxels that are back-projected within the foreground regions (silhouettes) of a number of image capture devices equal to or greater than the threshold are retained. In this way, by removing voxels that do not fit within the foreground region of the silhouette image, a 3D model of the quadrangular pyramid object shown in FIG. 5(c) is generated using voxels as shown in FIG. 5(d).

[0057] When an abnormality is detected in a certain imaging device, a 3D model may be generated based on the common area of ​​the view volume corresponding to the imaging devices other than the imaging device in which the abnormality was detected. The 3D model generated in this case may have a part of a different shape compared to a 3D model generated using the common area of ​​the view volume of all imaging devices when no abnormality is detected from all imaging devices. Similarly, when a 3D model is generated using complementary data instead of a silhouette image corresponding to an imaging device in which an abnormality was detected, a 3D model may be generated in which a part of the shape of the object is different from normal.

[0058] [3D model generation process flow] Fig. 6 is a flowchart illustrating an example of processing according to this embodiment for generating a three-dimensional model of an object in a target space. The series of processing steps shown in the flowchart in Fig. 6 are performed by the CPU of the three-dimensional model generation device 200 expanding program code stored in ROM into RAM and executing it. Some or all of the functions of the steps in Fig. 6 may be realized by hardware such as an ASIC or electronic circuit. Note that the symbol "S" in the description of each process indicates a step in the flowchart.

[0059] In S601, the camera identification unit 202 acquires a list of the image capture devices 101 included in the image processing system 100. In the case where the image capture devices 101 each have a corresponding foreground extraction device 102 as in the system configuration of FIG. 1 , the list of the foreground extraction devices 102 included in the image processing system 100 may be acquired.

[0060] In S602, the foreground data acquisition unit 201 receives one frame of foreground data from multiple foreground extraction devices 102. Note that the following steps will be described for the case where a three-dimensional model corresponding to one frame to be processed, acquired in S602, is generated.

[0061] S603 to S614 are loop processes, and by repeating the loop process, foreground data to be used for generating a three-dimensional model or a virtual viewpoint image is determined from the foreground data for one frame corresponding to each image capture device that is the processing target obtained in S602. In other words, foreground data that is not used for generating a three-dimensional model or a virtual viewpoint image is determined from the foreground data for one frame corresponding to each image capture device. In the first loop process, as a rule, S603 is determined as NO and the process proceeds to S604.

[0062] In S604, an imaging device to be determined is selected from the imaging devices included in the list of imaging devices 101 acquired in S601, and the processes of S605 to S614 are then performed for the imaging device to be determined.

[0063] When the imaging device to be determined is selected in S604, the process proceeds to S605, where the camera identification unit 202 determines whether foreground data corresponding to the imaging device to be determined and based on the frame to be processed has been acquired.

[0064] For example, if the three-dimensional model generation device 200 is unable to acquire foreground data corresponding to the imaging device to be determined even though an object exists in the area captured by the imaging device to be determined, the camera identification unit 202 determines that foreground data has not been acquired. For example, if the imaging device to be determined is unable to output the frame to be processed, foreground data is not generated, and therefore it is determined that foreground data has not been acquired.

[0065] Alternatively, in S605, it may be determined whether an abnormality has been detected in the foreground data of the frame to be processed that corresponds to the imaging device to be determined. In this case, the detected abnormality may include a case where no foreground data has been acquired. In the following description, it will be described that in S605, it is determined whether an abnormality has been detected in the foreground data based on the frame to be processed of the imaging device to be determined.

[0066] If an error occurs in the imaging device 101 or the foreground extraction device 102, the imaging device 101 or the foreground extraction device 102 may be configured to transmit internal error information indicating the content of the error to the 3D model generation device 200. If the camera identification unit 202 receives internal error information from the imaging device 101 or the foreground extraction device 102, the camera identification unit 202 may detect an abnormality in the foreground data corresponding to the imaging device to be determined based on the content of the received error. Furthermore, the foreground extraction device 102 may be controlled not to transmit foreground data if it has transmitted internal error information.

[0067] Examples of errors transmitted by the foreground extraction device 102 include internal errors such as a program error, an error when extracting an object, and an error when the data size is too large to transmit. For example, if the ratio of the size of the foreground area extracted by the foreground extraction device 102 (the ratio to the entire screen) is larger than a predetermined value, the foreground extraction device 102 may be configured to transmit an error when extracting an object.

[0068] Furthermore, the camera identification unit 202 may be configured to be able to detect an abnormality that occurs after foreground data is transmitted, based on packet loss information on the transmission path. In this way, by performing a process of determining whether an abnormality has been detected in the imaging device to be determined, it is possible to identify foreground data in which an abnormality has occurred or foreground data that the 3D model generation device 200 has not been able to acquire.

[0069] If an abnormality is detected in the foreground data corresponding to the image capture device 101 to be determined (YES in S605), the process proceeds to S606. In S606, the monitoring unit 203 acquires information about the image capture device 101 to be determined from the camera identification unit 202, and determines whether the image capture device 101 to be determined is an image capture device to be monitored.

[0070] If the imaging device 101 to be determined is not a monitoring target imaging device (NO in S606), the monitoring unit 203 adds the imaging device to the monitoring target list in S607 to register it as a new monitoring target. Then, the process proceeds to S608, where the monitoring unit 203 initializes the continuous acquisition counter of the imaging device 101 to be determined to 0.

[0071] The three-dimensional model generating device 200 has continuous acquisition counters associated with the multiple image capturing devices 101a-d included in the image processing system 100. The continuous acquisition counter is a counter for storing the number of consecutive times over time when no abnormality has been detected in foreground data based on frames from a certain image capturing device. Therefore, if an abnormality is detected, the continuous acquisition counter is reset to 0.

[0072] If the imaging device 101 to be determined is registered as a monitoring target (YES in S606), the process proceeds to S608, where the monitoring unit 203 initializes the continuous acquisition counter of the imaging device 101 to be determined to 0. Then, when the process of S608 ends, the process proceeds to S609.

[0073] In S609, if foreground data of the image capturing device 101 to be determined has been acquired in S602, the camera identifying unit 202 performs settings so that the foreground data is not used to generate a three-dimensional model corresponding to the frame to be processed.

[0074] For example, the camera identifying unit 202 assigns information indicating that the foreground data based on the frame to be processed in the image capturing device 101 to be determined is foreground data that will not be used to generate a three-dimensional model. Alternatively, the camera identifying unit 202 may hold a table including information (records) in which a list of image capturing devices 101 is associated with a flag indicating whether the data will be used to generate a three-dimensional model. Then, in S609, the camera identifying unit 202 may update the table by assigning a flag indicating that the data will not be used to generate a three-dimensional model to the record of the image capturing device 101 to be determined. In this way, when an abnormality is detected in the foreground data corresponding to the image capturing device to be determined, it is possible to prevent the foreground data corresponding to the image capturing device to be determined from being used to generate a three-dimensional model.

[0075] The process of S609 is a process for preventing foreground data corresponding to the imaging device to be monitored from being used as is for generating a 3D model. Even if the imaging device is a monitoring target, if the 3D model generation device 200 can successfully acquire foreground data, the frame interpolation unit 206 may generate interpolation data based on the foreground data.

[0076] On the other hand, if no abnormality is detected in the foreground data corresponding to the imaging device to be determined (NO in S605), the process proceeds to S610, where the monitoring unit 203 determines whether the imaging device 101 to be determined is registered as a monitoring target. For example, the monitoring unit 203 determines whether the imaging device to be determined is registered in a monitoring target list.

[0077] If the imaging device 101 to be determined is registered as a monitoring target (YES in S610), in S611 the monitoring unit 203 increments the value of the continuous acquisition counter of the imaging device 101 to be determined by 1, thereby updating the value of the continuous acquisition counter.

[0078] Then, the process proceeds to S612, where the monitoring unit 203 compares the value of the continuous acquisition counter of the image capturing device 101 to be determined with a preset threshold value n.

[0079] If the value of the continuous acquisition counter of the image capture device to be determined is smaller than the threshold value (NO in S612), the process proceeds to S609. Then, the camera identification unit 202 sets the foreground data corresponding to the image capture device 101 to be determined, which was acquired in S602, so that it is not used to generate a 3D model corresponding to the frame to be processed.

[0080] If the value of the continuous acquisition counter is smaller than the value of threshold n, no abnormality is detected in the foreground data based on the current frame, but the period during which foreground data was acquired without any abnormalities is short and it is determined that the data is not stable. Therefore, even if foreground data in which no abnormality is detected is acquired, if the determination is NO in S612, the process proceeds to S609, and the foreground data based on the currently processed frame is processed so that it is not used in generating a 3D model.

[0081] For example, suppose that silhouette images corresponding to a certain imaging device are transmitted to the 3D model generation device 200, alternating between normal and abnormal images for each frame. If an abnormality in the silhouette image is detected, a 3D model is generated based on only the normal silhouette image, or based on the normal silhouette image and complementary data. In this case, the generated 3D model may have a portion of its three-dimensional shape that differs from the normal shape. As a result, even for a stationary object, a 3D model with a different shape may be generated for each frame. If this 3D model is displayed in chronological order as a video, the portions with different shapes may appear to flash or flicker, which may cause a sense of discomfort to the viewer.

[0082] For this reason, in this embodiment, even if foreground data is successfully acquired from the foreground extraction device 102 corresponding to the imaging device 101 to be monitored, the imaging device is not excluded from the monitoring targets until it is confirmed that normal foreground data has been acquired for a predetermined number of consecutive frames. In other words, the foreground data of the imaging device to be monitored is not used to generate a three-dimensional model until a frame that satisfies a predetermined condition indicating that normal foreground data has been stably received from the imaging device 101 to be monitored is acquired. For this reason, in this embodiment, a portion of the three-dimensional shape represented by the three-dimensional model does not vary over a short period of time, and it is possible to reduce the sense of discomfort felt by the user (viewer) when the generated three-dimensional model is played back in chronological order.

[0083] On the other hand, if the value of the continuous acquisition counter is equal to or greater than the threshold value (YES in S612), the process proceeds to S613. In this case, the monitoring unit 203 excludes the image capture device to be determined from the monitoring target list.

[0084] Then, the process proceeds to S614, where the camera identification unit 202 sets the foreground data corresponding to the image capture device 101 to be determined, acquired in S602, to be used in generating a three-dimensional model. If the value of the continuous acquisition counter is equal to or greater than the threshold value n, it is considered that the foreground data corresponding to the image capture device to be determined has been acquired stably, and therefore the foreground data is processed to be used in generating a three-dimensional model.

[0085] Note that, when the image capturing device to be processed is excluded from the monitoring targets in S613 because it is determined in S612 that the value of the continuous acquisition counter is greater than the threshold value, the setting of foreground data corresponding to the previous frame may be changed so as to be used for generating a 3D model. For example, the setting of foreground data in which no abnormality was detected consecutively for the number of times equal to the threshold value n before the image capturing device was excluded from the monitoring targets may be changed so that it can be used for generating a 3D model.

[0086] Furthermore, in S612, the value of the continuous acquisition counter is compared with a preset threshold n to determine whether to exclude the foreground data from monitoring, but the threshold n used in S612 may be changed depending on the type of cause of the abnormality related to the foreground data. For example, if recovery from the abnormality is expected to be quick, the threshold n may be changed to a smaller value. Alternatively, if recovery from the abnormality is expected to be slow, the threshold n may be changed to a larger value. Furthermore, past monitoring history including the cause of the abnormality in the foreground data may be managed, and if the cause is the same as a cause that occurred in the past, the threshold n may be changed to a larger value.

[0087] Alternatively, a different value may be set for each of the multiple image capture devices 101a-d as the threshold value n used in the determination in S612. For example, importance information may be added to the list of image capture devices 101 or foreground extraction devices 102 acquired in S601. Then, for an image capture device 101 (foreground extraction device 102) with high importance, the threshold value n may be set to be smaller than that for an image capture device 101 (foreground extraction device 102) with low importance. The list of image capture devices 101 may include the threshold value n for each image capture device.

[0088] Alternatively, if the cause of the detected abnormality is identified and it is determined that the detected abnormality occurred due to an irrecoverable cause, S612 may always be determined as NO so that the device cannot be excluded from monitoring. For example, if an error recovery during imaging is difficult due to a circuit abnormality within the device, a cable break, or the like, and recovery by remote operation is not possible, it is determined that the abnormality occurred due to an irrecoverable cause. In this case, since the three-dimensional model generation device 200 cannot receive the error, it determines whether the abnormality is due to an irrecoverable cause from information other than the error.

[0089] On the other hand, if it is determined in S610 that the image capture device 101 to be determined is not registered as a monitored device (NO in S610), the image capture device to be determined is not a monitored image capture device and no abnormality has been detected in the foreground data, so the process proceeds to S614. Then, the foreground data corresponding to the image capture device 101 to be determined is set to be used to generate a three-dimensional model.

[0090] After S609 or S614 is completed, the process returns to S603, and it is determined whether or not processing has been completed for all of the imaging devices 101, based on the list information of the imaging devices 101 acquired in S601. If processing has not been completed for all of the imaging devices 101 (NO in S603), the process proceeds to S604 again, an imaging device to be determined is selected from the undetermined imaging devices, and the processes of S605 to S614 are repeated. If processing has been completed for all of the imaging devices 101 (YES in S603), the process proceeds to S615.

[0091] In S615, the 3D model generation unit 204 generates a 3D model of the frame to be processed using silhouette images of the image capture devices that are set to be generated into a 3D model, among the silhouette images based on the frame to be processed acquired in S602. That is, a 3D model is generated using silhouette images corresponding to image capture devices other than the image capture device to be monitored.

[0092] As described above, the 3D model generation unit 204 may generate a 3D model by determining a common area of ​​the view volumes of the image capture devices that are set to be generated in the 3D model. Alternatively, the 3D model may be generated using the interpolated data generated by the frame interpolation unit 206 instead of the silhouette image corresponding to the image capture device 101 that is a monitoring target and is set not to be used in generating the 3D model.

[0093] When the imaging devices 101a-d capture a moving image, foreground data is generated for each frame that makes up the moving image. Therefore, if there is a next frame, the processes of S602-S615 are repeated, and in the next step S602, foreground data based on the next frame is acquired. By repeating the processing steps of this flowchart each time foreground data for one frame is acquired, a three-dimensional model corresponding to the frames that make up the moving image is generated and saved in the storage device 300.

[0094] The rendering unit 401 of the rendering device 400 selects a texture image based on the additional information set in the foreground data transmitted from the foreground extraction device 102, and colors the three-dimensional model.

[0095] For example, if a texture image corresponding to a certain imaging device is repeatedly transmitted frame by frame between normal and abnormal states, and coloring is performed using that texture image, the color difference may vary from frame to frame, causing a sense of incongruity to the viewer. For this reason, the rendering unit 401 may perform coloring using a texture image corresponding to an imaging device other than the monitoring target, rather than using a texture image corresponding to the imaging device 101 that is the monitoring target. By not using the texture image generated by the foreground extraction device 102 that corresponds to the monitoring target imaging device, color difference does not vary over a short period of time, thereby reducing the sense of incongruity of the virtual viewpoint image that occurs when the images are played back in chronological order.

[0096] As described above, according to this embodiment, acquired foreground data is not used to generate a 3D model or virtual viewpoint images until stable, normal foreground data can be acquired. Therefore, according to this embodiment, it is possible to reduce the sense of discomfort felt by the user when the 3D model or virtual viewpoint images are played back in chronological order.

[0097] Depending on the type of error that prevented the foreground data from being transmitted, the foreground extraction device 102 may stop transmitting the foreground data for a period set for each error cause, even if the foreground extraction device 102 has recovered from the error. This control can reduce the amount of data transmitted from multiple foreground extraction devices 102, thereby reducing the network load between the foreground extraction device 102 and the 3D model generation device 200. In this case, the freed bandwidth can be used to increase the amount of data transmitted from the other foreground extraction devices 102.

[0098] <Other embodiments> The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0099] 200 Three-dimensional model generation device 201 Foreground data acquisition unit 202 Camera Identification Unit 203 Monitoring Department 204 3D model generation unit

Claims

1. an acquisition means for acquiring a plurality of images representing an area of ​​an object corresponding to a plurality of imaging devices, the plurality of images corresponding to a plurality of frames constituting a moving image; an identification means for detecting an abnormality in at least one of the plurality of images acquired by the acquisition means, and identifying a frame and an imaging device corresponding to the image in which the abnormality was detected; generating means for generating three-dimensional shape data of the object using the plurality of images; The generating means When generating the three-dimensional shape data for a plurality of frames corresponding to a period from when the abnormality is detected until a predetermined condition is satisfied, the images corresponding to the imaging device identified by the identification means are not used, and the three-dimensional shape data is generated using images corresponding to an imaging device different from the imaging device identified by the identification means; the predetermined condition is that the abnormality has not been detected in the image corresponding to the imaging device identified by the identification unit for a predetermined number of consecutive frames; The predetermined number of frames is changed depending on the type of cause of the abnormality, and the predetermined number of frames is determined depending on the length of recovery from the abnormality.

1. An image processing device comprising:

2. The specifying means also detects a case where the acquiring means is unable to acquire the image corresponding to any one of the plurality of image capturing devices as the abnormality, and specifies the frame and the image capturing device from which the image could not be acquired.

2. The image processing device according to claim 1, wherein:

3. The imaging device further includes a registration unit that registers the imaging device identified by the identification unit as a monitoring target.

3. The image processing device according to claim 1, wherein the image processing device is a computer.

4. a complementing means for generating an image corresponding to the image captured by the imaging device of the monitored object, The generating means When generating the three-dimensional shape data for a plurality of frames corresponding to the period, the three-dimensional shape data is generated using the image corresponding to an imaging device different from the imaging device identified by the identification means and an image that serves as a substitute for the image corresponding to the imaging device identified by the identification means.

4. The image processing device according to claim 3.

5. The complementing means comprises: generating an image to replace the image corresponding to the imaging device of the monitoring target based on the image corresponding to the imaging device of the monitoring target when the abnormality is not detected; 5. The image processing device according to claim 4.

6. The complementing means comprises: An image to replace the image corresponding to the imaging device of the monitoring target is generated based on a frame of the imaging device when it is not registered with the monitoring target.

5. The image processing device according to claim 4.

7. The complementing means comprises: generating an image representing the area of ​​the object as a substitute for the image corresponding to the imaging device of the monitored object; 5. The image processing device according to claim 4.

8. The registration means If the abnormality is not detected in the images corresponding to the predetermined number of consecutive frames of the image capturing device to be monitored, the image capturing device to be monitored is excluded from the monitoring targets.

8. The image processing device according to claim 4, wherein the image processing device is a computer.

9. further comprising an extraction unit that extracts an area of ​​the object from frames that constitute the moving image captured by the plurality of image capturing devices, and generates the image corresponding to the image capturing device; The acquisition means does not acquire the image generated by the extraction means from a frame captured by an imaging device in which an abnormality has been detected by the identification means for a predetermined period of time.

9. The image processing device according to claim 1, wherein the image processing device is a computer.

10. a second generating means for generating a virtual viewpoint image by coloring the three-dimensional shape data; the acquiring means further acquires a plurality of texture images representing textures of the object corresponding to the plurality of imaging devices, the plurality of texture images corresponding to a plurality of frames constituting the moving image; The second generating means When generating a virtual viewpoint image for a plurality of frames corresponding to the period, the texture image corresponding to the imaging device identified by the identification means is not used, and the virtual viewpoint image is generated using a texture image corresponding to an imaging device different from the imaging device identified by the identification means.

10. The image processing device according to claim 1, wherein the image processing device is a computer.

11. The generating means Generate the three-dimensional shape data using VisualHull 11. The image processing device according to claim 1, wherein the image processing device is a computer.

12. The predetermined number of frames is set for each of the plurality of imaging devices, The predetermined number of frames set for a target imaging device for which an importance level is set among the plurality of imaging devices is set further based on the importance level set for the target imaging device.

12. The image processing device according to claim 1, wherein the image processing device is a computer.

13. an acquisition means for acquiring a plurality of images representing a region of the object corresponding to a plurality of image capture devices; an identification unit that identifies an imaging device from which an image has not been acquired by the acquisition unit, among the plurality of imaging devices; generating means for generating three-dimensional shape data of the object using the plurality of images; the generating means, for an imaging device identified by the identifying means as one from which images are not acquired, does not use the images corresponding to the imaging device identified by the identifying means until the acquiring means can acquire a number of images corresponding to a predetermined number of consecutive frames from the time the imaging device is identified by the identifying means, but generates the three-dimensional shape data using the images corresponding to an imaging device different from the imaging device identified by the identifying means; The predetermined number of frames is changed depending on the type of the abnormality that caused the image not to be acquired, and the predetermined number of frames is determined depending on the length of recovery from the abnormality.

1. An image processing device comprising:

14. 1. An image processing method executed by one or more processors, comprising: acquiring, by the one or more processors, a plurality of images representing a region of the object corresponding to a plurality of image capture devices, the plurality of images corresponding to a plurality of frames constituting a motion picture; an identifying step in which the one or more processors detect an anomaly in at least one image of the plurality of images acquired by the acquiring step, and identify a frame and an imaging device corresponding to the image in which the anomaly was detected; a generating step in which the one or more processors generate three-dimensional shape data of the object using the plurality of images; In the generating step, When generating the three-dimensional shape data for a plurality of frames corresponding to a period from when the abnormality is detected until a predetermined condition is satisfied, the one or more processors do not use the images corresponding to the imaging device identified in the identifying step, but generate the three-dimensional shape data using images corresponding to an imaging device different from the imaging device identified in the identifying step; the predetermined condition is that the abnormality is not detected in the images corresponding to the imaging device identified in the identifying step for a number of consecutive frames corresponding to a predetermined number of frames, The predetermined number of frames is changed depending on the type of cause of the abnormality, and the predetermined number of frames is determined depending on the length of recovery from the abnormality. An image processing method comprising:

15. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 13.

Images