Information processing device, information processing method, and program

Abstract

To generate an appropriate virtual viewpoint image.SOLUTION: Shape data of an object used for generating a virtual viewpoint image is generated using a plurality of captured images having different viewpoints, and it is determined whether the object related to the generated shape data corresponds to an unnecessary object. Then, a virtual viewpoint image in which the display of the object corresponding to the unnecessary object is suppressed is generated.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The technology disclosed herein relates to a technology for generating a virtual viewpoint image from images captured by a plurality of imaging devices. [Background technology]

[0002] Recently, a technology that generates a virtual viewpoint image that represents a view from a specified viewpoint (virtual viewpoint) by using the multiple viewpoint images obtained by synchronously shooting with multiple imaging devices installed at different positions and capturing the images has been attracting attention. When generating a virtual viewpoint image, a three-dimensional model of a subject (foreground object) such as a person existing in the imaging target area is calculated to generate an image of the imaging target area as seen from a virtual viewpoint. There are various targets for generating virtual viewpoint images, such as sporting events held in stadiums. Such sporting events are often filmed by staff (cameramen) of a broadcasting station for the purpose of broadcasting or distribution. In addition, a special filming system called a wire cam is sometimes used for large-scale events. A wire cam is a system that shoots images by suspending imaging devices from multiple wires stretched across the imaging target space, and can capture dynamic bird's-eye images of field sports such as soccer and rugby.

[0003] As described above, in large-scale sporting events, broadcasting station cameramen and wire cameras move around the space to be photographed. As a result, they may appear in the virtual viewpoint image, as do athletes, and interfere with viewing. It is desirable that such people and objects that are not the original foreground objects and interfere with viewing are not displayed on the virtual viewpoint image. Patent Document 1 proposes a surveillance camera system that, when an obstruction is detected by a first camera, performs viewpoint conversion processing on an image captured by a second camera and synthesizes it with an image captured by the first camera, thereby enabling a subject blocked by an obstruction to be confirmed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP2011-77981A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology of Patent Document 1 obtains an image without obstructions by performing viewpoint conversion processing on an image captured by a camera that does not detect obstructions among multiple cameras that are fixed in position. Therefore, it is difficult to obtain a virtual viewpoint image from which objects that interfere with viewing have been appropriately removed by directly applying the technology to the generation of the virtual viewpoint image.

[0006] Therefore, an object of the technology disclosed herein is to generate an appropriate virtual viewpoint image. [Means for solving the problem]

[0007] The information processing device according to the present disclosure includes a first generating means for generating shape data representing a three-dimensional shape of an object using a plurality of captured images having different viewpoints, and a determining means for determining whether or not an object related to the shape data generated by the first generating means corresponds to an object that does not need to be displayed in a virtual viewpoint image, based on at least a condition related to the shape of the object. Using the shape data and a second generating means for generating a virtual viewpoint image in which the display of an object determined by the determining means to be an object not requiring display is made more transparent than the display of an object not determined by the determining means to be an object not requiring display. Effect of the Invention

[0008] According to the technology of the present disclosure, an appropriate virtual viewpoint image can be generated. [Brief description of the drawings]

[0009] [Figure 1]FIG. 1 is a block diagram showing an example of the configuration of an image processing system that generates a virtual viewpoint image. [Diagram 2] FIG. 2 is a diagram for explaining the arrangement of cameras installed in a space to be photographed. [Diagram 3] FIG. 1 is a block diagram showing a hardware configuration of an information processing device. [Figure 4] FIG. 2 is a block diagram showing the software configuration of a server according to the first embodiment. [Diagram 5] 13A and 13B are diagrams showing an example of a UI screen. [Figure 6] FIG. 4 is a functional block diagram showing the internal configuration of a three-dimensional model analysis unit. [Figure 7] 4A to 4L are diagrams showing examples of basic shape patterns of a three-dimensional model. [Figure 8] 11 is a table showing an example of feature information of a three-dimensional model. [Figure 9] 5 is a flowchart showing the flow of a virtual visual point image generating process according to the first embodiment. [Figure 10] 4A and 4B are diagrams illustrating the effects of the first embodiment. [Figure 11] FIG. 11 is a block diagram showing the software configuration of a server according to the second embodiment. [Figure 12] 5 is a flowchart showing the flow of a virtual visual point image generating process according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. Note that the same components are described with the same reference numerals.

[0011] [Embodiment 1] In this embodiment, shape data (hereinafter referred to as "three-dimensional model") representing the three-dimensional shape of an object is analyzed, and information indicating that the three-dimensional model of a specific object does not need to be displayed in the virtual viewpoint image is added to the three-dimensional model to which the information is added, and a transparent process is performed on the three-dimensional model to obtain a virtual viewpoint image from which objects that hinder viewing have been removed. Note that the virtual viewpoint image is an image generated by an end user and / or a designated operator freely manipulating the position and attitude of a virtual camera, and is also called a free viewpoint image or an arbitrary viewpoint image. In addition, the virtual viewpoint image may be a video or a still image. In the following, an image processing system that realizes this embodiment will be described by taking the case where the virtual viewpoint image is a video as an example.

[0012] (Basic system configuration) FIG. 1 is a block diagram showing an example of the configuration of an image processing system for generating a virtual viewpoint image according to this embodiment. The image processing system 100 includes a plurality of imaging modules 110, a switching hub 115, a server 116, a database (DB) 117, a control device 118, and a display device 119. Each of the plurality of imaging modules 110 includes a camera 111 as an imaging device and a camera adapter 112, which are connected to each other by internal wiring. Each imaging module 110 transmits data through a network cable. A switching hub (hereinafter, referred to as "HUB") 115 is a device that performs routing between each network device. Each imaging module 110 is connected to the HUB 115 by a network cable 113. Similarly, the server 116, the DB 117, and the control device 118 are also connected to the HUB 115 by the network cable 113. The control device 118 and the display device 119 are connected to each other by a video cable 114. Each camera 111 performs shooting in high accuracy synchronization with each other based on a synchronization signal. 2, in this embodiment, a plurality of imaging modules 110 (only ten are shown for convenience) are installed to surround the field of a stadium, which is the space to be photographed. In addition, a wire cam 120 for photographing images for broadcasting, which is not used to generate a virtual viewpoint image, is installed suspended from a wire stretched above the field.

[0013] The server 116 is an information processing device that processes the captured images obtained by the imaging module 110, generates and analyzes three-dimensional models of objects, and colors the generated three-dimensional models (also called "texture pasting" or "texture mapping"). The server 116 also has a time server function that generates a time synchronization signal for performing time synchronization of the present system. In this embodiment, the objects for which three-dimensional models are generated are moving objects such as players and balls. In this case, the wire cam 120 that moves around the field and the cameraman who shoots images for broadcasting while moving around the field are also formally considered to be moving objects, and it is a problem in the present disclosed technology that they become targets for generating three-dimensional models. The database (hereinafter, referred to as "DB") 117 accumulates data such as captured images obtained by each imaging module 110 and generated three-dimensional models, and provides the accumulated data to the server 116 and the control device 118. The control device 118 is an information processing device that controls each imaging module 110 and the server 116. The control device 118 is also used to set a virtual camera (virtual viewpoint). The display device 119 displays a setting user interface screen (UI screen) for the user to specify a virtual viewpoint in the control device 118, a UI screen for viewing the generated virtual viewpoint image, etc. The display device 119 may be, for example, a television, a computer monitor, or a liquid crystal display unit of a tablet or smartphone, and the type of device is not important.

[0014] (Image processing system operation) Next, an outline of the operation of the image processing system 100 will be described. The captured image obtained by the imaging module 110 is subjected to predetermined image processing such as foreground / background separation, and then transmitted to the next imaging module 110. Similarly, the next imaging module 110 combines the captured image obtained by itself with the captured image received from the previous imaging module 110 and transmits the combined image to the next imaging module 110. By continuing such operations, 100 sets of captured images (including foreground images) are transmitted to the server 116 via the HUB 115.

[0015] The server 116 generates a virtual viewpoint image by generating a three-dimensional model of an object and performing rendering processing based on the captured image data from all the imaging modules 110 with different viewpoints. The server 116 also transmits a time and a synchronization signal to each imaging module 110. Each imaging module 110 that receives the time and synchronization signal captures an image using the received time and synchronization signal, and performs frame synchronization of the captured image. That is, each imaging module 110 captures an image in units of a frame in synchronization with the same time. The format of the captured image data is not particularly limited. For example, the bit depth of each pixel (8 bits, 10 bits, etc.) and the format expressing the color of each pixel (YUV444, YUV422, RGB, etc.) are not limited. The image file format is also not limited. For example, it is assumed that the format is a general PNG (Portable Network Graphics) or JPEG (Joint Photographic Experts Group).

[0016] When generating a virtual viewpoint image, first, the silhouette of each object in the foreground is extracted from a plurality of captured images from different viewpoints. Next, a three-dimensional model of the object is generated by, for example, the VisualHull method (visual volume intersection method) using an image (foreground image) showing the extracted silhouette of the object. Finally, the generated three-dimensional model of the object is colored using color information (texture information) included in the captured images of each physical camera 111. By the above-mentioned processing, a virtual viewpoint image showing the appearance from an arbitrary virtual viewpoint is obtained. Note that, when generating a virtual viewpoint image, a morphing method or a billboarding method may be used instead of the above-mentioned model-based method. The morphing method is a method of synthesizing images between nearby cameras by interpolating them according to the positional relationship of the cameras, and generating a virtual viewpoint image between the cameras. The billboarding method is a method of generating a virtual viewpoint image by extracting a two-dimensional image of a subject of a captured image and changing its orientation in a three-dimensional space by a projection calculation process according to a virtual viewpoint. Such a method of acquiring a virtual viewpoint image by synthesizing and projective transformation of a plurality of captured images by physical cameras is called image-based rendering.

[0017] (Hardware configuration of information processing device) Next, the hardware configuration of the information processing device such as the server 116 and the control device 118 will be described with reference to Fig. 3. Note that the camera adapters 112a to 112j in the imaging modules 110a to 110j basically have the same hardware configuration. The information processing device has a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a communication I / F 205, an operation unit 206, and a bus 207.

[0018] The CPU 201 controls the entire information processing device using programs and data stored in the ROM 202 and the RAM 203. Note that one or more dedicated hardware different from the CPU 201 may be provided, and at least a part of the processing by the CPU 201 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor). The ROM 202 stores programs that do not require modification. The RAM 203 temporarily stores programs and data provided from the auxiliary storage device 204, and data provided from the outside via the communication I / F 205. The auxiliary storage device 204 is composed of, for example, an HDD or SSD, and stores various data and programs such as input data such as image data and audio data, tables referenced in various processes described below, and various application programs. The communication I / F 205 is used for communication with an external device. For example, when connected to an external device by wire, a communication cable is connected to the communication I / F 205, and when having a function of wireless communication with an external device, the communication I / F 205 is equipped with an antenna. The operation unit 206 is composed of, for example, a keyboard, mouse, joystick, touch panel, etc., and inputs various instructions to the CPU 201 in response to operations by a user. The bus 207 connects the above-mentioned units to transmit data and signals.

[0019] In this embodiment, the server 116 generates both the three-dimensional model and the virtual viewpoint image, but the system configuration is not limited to this. For example, a server that generates the three-dimensional model and a server that generates the virtual viewpoint image may exist separately.

[0020] (Server software configuration) FIG. 4 is a block diagram showing the software configuration of the server 116 according to this embodiment. In the technology of the present disclosure, a three-dimensional model to be hidden is determined based on the characteristics of the three-dimensional model of an object that interferes with appreciation. Then, the three-dimensional model determined to be hidden is made transparent and displayed, thereby making the object that interferes with appreciation invisible and improving the quality of the virtual viewpoint image. As shown in FIG. 4, the server 116 has a data input unit 401, a three-dimensional model generation unit 402, a three-dimensional model analysis unit 403, a non-display setting unit 404, and a virtual viewpoint image generation unit 405. Each unit will be described below in order.

[0021] The data input unit 401 acquires data of a plurality of captured images with different viewpoints that are captured by the imaging module 110 and stored in the DB 117. The acquired data of the plurality of captured images is sent to a three-dimensional model generation unit 402 and a virtual viewpoint image generation unit 405. The data input unit 401 also acquires information consisting of various parameters required for generating a virtual viewpoint image, such as the position and orientation of the virtual viewpoint, the angle of view, and the focal length (hereinafter referred to as "virtual viewpoint information") from the control device 118. The acquired virtual viewpoint information is sent to the virtual viewpoint image generation unit 405.

[0022] The three-dimensional model generating unit 402 generates a three-dimensional model of a moving object present in each captured image based on the captured image data received from the data input unit 401. A specific generation procedure will be briefly described below. First, each captured image is subjected to foreground / background separation processing to generate a foreground image showing the silhouette of the object. Here, a background subtraction method is used as a method for foreground / background separation. In the background subtraction method, first, different frames in a time series among a captured image consisting of multiple frames are compared with each other, and a portion where the difference in pixel value is small is specified as a pixel with no motion. Then, a background image is generated using the specified pixel with no motion. By comparing the background image generated in this way with a frame of interest in the captured image, pixels in the frame with a large difference from the background are specified as pixels that become the foreground, and a foreground image is generated. The above processing is performed for each of a plurality of captured images with different viewpoints. Next, a three-dimensional model is extracted by a volume intersection method using a foreground image corresponding to each captured image. In the visual volume intersection method, the target three-dimensional space is divided into small unit cubes (voxels), the pixel position of each voxel when it appears in multiple captured images is calculated by three-dimensional calculation, and it is determined whether each voxel corresponds to a foreground pixel. If the voxel is determined to be a foreground pixel in the captured images of all the imaging modules, the voxel is identified as a voxel that constitutes an object in the target three-dimensional space. In this way, only the identified voxels are left, and the other voxels are deleted. Then, the group of voxels that finally remain becomes a three-dimensional model that represents the three-dimensional shape of the object existing in the target three-dimensional space. The generated three-dimensional model for each object is sent to the three-dimensional model analysis unit 403 together with the captured image data on which it is based.

[0023] The three-dimensional model analysis unit 403 performs an analysis process on the three-dimensional model generated by the three-dimensional model generation unit 402. In this analysis process, first, it is determined whether or not the three-dimensional model is a three-dimensional model of a specific object (hereinafter referred to as an "exclusion object") to be excluded from the virtual viewpoint image by referring to the exclusion conditions described below. Then, for the three-dimensional model determined to be an exclusion object, information indicating that it does not need to be displayed in the virtual viewpoint image (for example, a flag that assigns "1" when it does not need to be displayed and "0" otherwise; hereinafter referred to as a "non-display flag") is added. The three-dimensional model analysis unit 403 will be described in detail later.

[0024] The non-display setting unit 404 performs settings necessary for realizing display control so that excluded objects are not displayed on the virtual viewpoint image. Specifically, the operation mode (non-display mode) for controlling whether or not to perform non-display control of excluded objects is enabled / disabled, and conditions (exclusion conditions) for specifying what objects are to be excluded objects are set based on a user instruction. FIG. 5(a) shows an example of a user interface screen (UI screen) for a user to set the non-display mode. A user who wants to prevent a specific object that interferes with appreciation from being displayed on the virtual viewpoint image checks a radio button 501 for enabling the non-display mode and presses an OK button 503. Then, the screen transitions to a screen for setting the exclusion conditions shown in FIG. 5(b). In the UI screen after this transition, the user can specify detailed conditions for the excluded object (here, the color, shape, and position in three-dimensional space of the object). Now, in FIG. 5(b), a button 511 for specifying a position in three-dimensional space is selected to display a pop-up window 512, and a value "3000 (mm)" for specifying the height from the ground is input in an input field 513. In this case, objects that are 3 m or higher above the ground are set as excluded objects, and the wire cam 120 moving around above the field is determined to be an excluded object in the analysis process. In addition, as a condition for the height position information, an object that is higher than the object to be displayed may be set as an excluded object.

[0025] Here, the only condition for the position is that the height (z coordinate) in the three-dimensional space is equal to or greater than a certain value. However, for example, thresholds may be set for each of the x-axis, y-axis, and z-axis, and conditions may be set such that the coordinate value for each axis is equal to or greater than the threshold, equal to or less than the threshold, or the difference from the threshold is within a certain value. Also, a virtual viewpoint image corresponding to an arbitrary frame may be displayed in the preview area 515, and the user may select an object that interferes with the viewing from the image with a mouse or the like, and the characteristics (color, shape, position, size, etc.) of the object may be automatically set as the exclusion condition. Furthermore, when the user selects an object, the characteristics may be displayed in a list, and the user may select the characteristic to be the exclusion condition from the list. The user who has specified various conditions that define the excluded object in this way specifies a combination condition 516 that is a criterion for determining whether or not to make the object a target of the non-display control. For the combination condition, general logical operations such as a general AND condition (logical AND) or an OR condition (logical OR) may be used. For example, when an AND condition is used, "AND" may be entered in the input field 517. When an AND condition is specified, a three-dimensional model for which all the analysis results of the analysis units 602 to 604 "satisfy the condition" is determined as a three-dimensional model to be hidden in the virtual viewpoint image. Similarly, when an OR condition is used, a three-dimensional model for which any of the analysis results of the analysis units 602 to 604 "satisfy the condition" is determined as a three-dimensional model to be hidden in the virtual viewpoint image. Based on such a user instruction given via the control device 118, etc., a condition for enabling / disabling the non-display mode and for identifying an excluded object is set.

[0026] The virtual viewpoint image generating unit 405 generates a virtual viewpoint image using the captured image data and virtual viewpoint information input from the data input unit 401, and the three-dimensional model input from the three-dimensional model analyzing unit 403. Specifically, it calculates how each three-dimensional model looks in three-dimensional space at the angle of view specified by the virtual viewpoint information, and generates an image that represents the view from the virtual viewpoint by coloring it using the corresponding captured image of the real camera. At that time, if the non-display mode is enabled, the display of the three-dimensional model of the excluded object is suppressed. Specifically, instead of performing normal coloring, a process of making the three-dimensional model semi-transparent or completely transparent is performed using a method such as alpha blending that can control the transparency. If the non-display mode is disabled, normal coloring is performed on all three-dimensional models regardless of the content of the non-display flag, and an image that represents the view from the virtual viewpoint is generated. The data of the virtual viewpoint image obtained in this way is sent to the display device 119 via the control device 118, and is provided for viewing by the user.

[0027] The above five functional units may be shared among a plurality of information processing devices. For example, the output result of the three-dimensional model analysis unit 403 may be input to another information processing device having the function of the virtual viewpoint image generation unit 405.

[0028] (Details of the 3D model analysis section) Next, a detailed description will be given of the non-display determination and feature information assignment performed by the three-dimensional model analysis unit 403. Fig. 6 is a functional block diagram showing the internal configuration of the three-dimensional model analysis unit 403. As shown in the figure, the three-dimensional model analysis unit 403 has an exclusion condition storage unit 601, a color analysis unit 602, a position analysis unit 603, a shape analysis unit 604, a non-display determination unit 605, and a feature information assignment unit 606.

[0029] The exclusion condition holding unit 601 holds the above-mentioned exclusion conditions set by the non-display setting unit 404, and outputs conditions according to the analysis items to the three analysis units 602 to 604. That is, among the exclusion conditions, it outputs conditions related to position to the position analysis unit 602, conditions related to color to the color analysis unit 603, and conditions related to shape to the shape analysis unit 604. The exclusion condition holding unit 601 also holds combination conditions of the analysis results in the analysis units 602 to 604, and outputs them to the non-display determination unit 505.

[0030] The position analysis unit 602 performs a position analysis process to determine whether or not the position of the three-dimensional model in the virtual viewpoint image space provided by the three-dimensional model generation unit 402 matches the position-related condition (height from the ground in the above example) in the exclusion condition. Specifically, if the height represented by the z coordinate of the bottom end of the bounding box of the target three-dimensional model exceeds the height specified in the condition, the object of the target three-dimensional model satisfies the position-related condition. Here, the bounding box means a circumscribed rectangular parallelepiped of a group of voxels representing the three-dimensional shape of the object. The result of the position analysis process is assigned to the target three-dimensional model and sent to the non-display determination unit 605.

[0031] The color analysis unit 603 performs color analysis processing to determine whether or not the color corresponding to each voxel constituting the three-dimensional model provided from the three-dimensional model generation unit 402 matches the color-related condition in the exclusion condition. Now, assume that each pixel of the captured image has a color expressed by a color brightness value of the maximum value of "4095" for each of RGB. In this case, a condition such that the color brightness value of each of RGB is equal to or less than "100" may be set as the color-related condition. In this case, if the target three-dimensional model has a color close to black, the object satisfies the condition of the excluded object. More specifically, for each voxel constituting the target three-dimensional model, a captured image to be referred to in the color analysis processing is determined based on the distance and angle to each real camera (the line of sight direction of the real camera), and the like, and the corresponding color is assigned. Then, the assigned color is compared with the above condition to determine whether the color brightness value of each of RGB is equal to or less than a threshold value. At this time, it may be determined whether the colors of all the voxels match the above condition, or it may be determined whether the average value of the colors of all the voxels matches the above condition. The result of the color analysis process is assigned to the target three-dimensional model and sent to the non-display determination unit 605 .

[0032] The shape analysis unit 604 performs a shape analysis process to determine whether or not the shape of the three-dimensional model provided from the three-dimensional model generation unit 402 matches the shape-related condition in the exclusion condition. (a) to (l) in FIG. 7 show examples of basic shape patterns of a three-dimensional model. As a shape-related condition, for example, a condition may be set that the shape of the target three-dimensional model does not correspond to any of the shape patterns of (j) human shape, (k) ellipse, and (l) sphere, based on such basic shape patterns. In the case of this condition, if the shape of the target three-dimensional model has a shape other than a human shape, ellipse, or sphere, such as a cylindrical shape, the object corresponds to an excluded object. The result of the shape analysis process is assigned to the target three-dimensional model and sent to the non-display determination unit 605.

[0033] The non-display determination unit 605 performs a non-display determination for the three-dimensional model of each object present in the virtual viewpoint video space based on the results of each analysis process in the position analysis unit 602, color analysis unit 603, and shape analysis unit 604, and the combined conditions provided by the condition storage unit 601. Note that it is also possible to make a determination based on conditions other than the above-mentioned combined conditions, such as hiding a target three-dimensional model that "does not satisfy the conditions" depending on how threshold values ​​are set in the various conditions used in the analysis units 602 to 604. Each three-dimensional model that has completed such a determination process is output to the feature information addition unit 606 together with the determination result.

[0034] The feature information adding unit 606 adds feature information to each three-dimensional model received from the non-display determining unit 605. The feature information includes the above-mentioned non-display flag determined based on the result of each non-display determination. FIG. 8 is a table showing an example of feature information of a three-dimensional model. As described above, a three-dimensional model is composed of a plurality of voxels, and a bounding box (circumscribed rectangular parallelepiped) of the entire voxel group representing the three-dimensional shape of one object is first defined. Then, for each three-dimensional model, information corresponding to each item such as its size, shape, and color used for coloring is stored. In this case, the size information is the volume value of the bounding box and the number of voxels that actually exist. In addition, the shape information stores information on the most approximate basic shape pattern from the basic shape pattern shown in FIG. 7 described above. In addition, in the color information, a representative color indicating the most common color among the colors corresponding to each voxel and an average value of each of RGB values ​​are stored.

[0035] In the above example, the three analysis elements are the position, color, and shape of the three-dimensional model, but the analysis elements are not limited to these. For example, the size of the three-dimensional model may be an analysis element in addition to these. When determining whether a certain condition is satisfied, the comparison with the threshold value can use "above threshold", "below threshold", "greater than threshold", "smaller than threshold", "difference from the reference value within XX", "difference from the reference value more than △△", etc.

[0036] (Flow of virtual viewpoint image generation process) 9 is a flowchart showing the flow of a virtual viewpoint image generation process in the server 116 according to this embodiment. A series of processes shown in this flowchart is realized by the CPU 201 of the server 116 reading a predetermined program from the ROM 202 or the auxiliary storage device 204 and executing the program.

[0037] Hereinafter, the flow of the virtual viewpoint image generation process according to this embodiment will be described with reference to the flowchart in Fig. 9. It is assumed that the flow in Fig. 9 is started in response to a start signal for generating a virtual viewpoint image from the control device 118. The start signal is added with the above-mentioned virtual viewpoint information and mode selection information indicating whether the non-display mode is enabled or disabled. It is also assumed that at the start of execution, the captured image data on which the virtual viewpoint image is based has already been input to the data input unit 401. In the following description, the symbol "S" means a step.

[0038] In S901, the non-display setting unit 404 sets the non-display mode to enabled / disabled based on the mode selection information input from the control device 118. In this case, when the non-display mode is set to enabled, an exclusion condition for identifying a target object (exclusion object) of the non-display control is also set.

[0039] In S902, the 3D model generation unit 402 acquires an image of the frame to be processed based on the time information included in the virtual viewpoint information. Here, the time information is information on the start time and end time that specifies a predetermined period in the captured image data. In the following S903, the 3D model generation unit 402 generates a 3D model of each object present in the image of the frame to be processed. The generated 3D model for each object is sent to the 3D model analysis unit 403.

[0040] In S904, the three-dimensional model analysis unit 403 performs the above-mentioned analysis process on the three-dimensional model for each object generated in S903. That is, for each three-dimensional model, the characteristics are specified in terms of position, color, and shape, and it is determined whether the specified characteristics match the characteristics of the excluded object by referring to the exclusion conditions set in S901. Then, characteristic information including a non-display flag according to the determination result is assigned to each three-dimensional model. In this way, the three-dimensional model to which the characteristic information is assigned according to the analysis result is sent to the virtual viewpoint image generation unit 405.

[0041] In S906, processing is allocated according to the setting contents of the non-display mode set in S901. That is, if the setting of the non-display mode is invalid (OFF), the processing proceeds to S907, and if it is valid (ON), the processing proceeds to S908.

[0042] In S907, when the non-display mode is off, the virtual viewpoint image generating unit 405 generates a virtual viewpoint image in which all objects present in the frame to be processed are displayed. Specifically, regardless of the value of the non-display flag, normal coloring is performed on all input 3D models to generate a virtual viewpoint image corresponding to the frame to be processed.

[0043] On the other hand, in S908 when the non-display mode is enabled, the virtual viewpoint image generating unit 405 generates a virtual viewpoint image in which the excluded object is hidden in the frame to be processed. Specifically, the virtual viewpoint image corresponding to the frame to be processed is generated by performing normal coloring for three-dimensional models whose non-display flag value is "0" and performing transparency processing for three-dimensional models whose non-display flag value is "1".

[0044] In S909, it is determined whether to end the generation of the virtual viewpoint image. If the processing is completed for all frames specified by the time information included in the virtual viewpoint information, this processing ends. On the other hand, if there are unprocessed frames, the processing returns to S902, and the image of the next processing target frame is obtained and processing continues.

[0045] The above is the flow of the virtual viewpoint image generation process according to this embodiment. (a) and (b) of FIG. 10 are diagrams for explaining the effect of this embodiment. (a) of FIG. 10 is an example of a virtual viewpoint image generated by a conventional method, and (b) of FIG. 10 is an example of a virtual viewpoint image generated by the method of this embodiment. In the case of the virtual viewpoint image by the conventional method in (a) of FIG. 10, the wire cam 120 is treated as a foreground object and is displayed as it is. In contrast, in the case of the virtual viewpoint image by the method of this embodiment in (b) of FIG. 10, the three-dimensional model of the wire cam 120 is made transparent, so that the viewer cannot see it in the completed virtual viewpoint image.

[0046] In the above example, the virtual viewpoint image is generated by sequentially processing all frames within a predetermined period specified by time information, but the present invention is not limited to this. For example, the virtual viewpoint image may be generated using a general key frame method. The key frame method is a method in which a plurality of reference frames (called "key frames") corresponding to any virtual viewpoint are set, path information of the virtual viewpoint is obtained by interpolating between the set plurality of key frames, and a virtual viewpoint image corresponding to the predetermined period is generated. The method of this embodiment is also applicable to such a key frame method.

[0047] In addition, in the above example, for the sake of simplicity, the non-display mode is enabled / disabled before the frame-by-frame loop processing, but the present invention is not limited to this. For example, the non-display mode setting may be changed during the generation of the virtual viewpoint image.

[0048] As described above, according to this embodiment, the 3D model of an object that satisfies certain conditions based on the characteristics of the 3D model of the object present in the captured image is made transparent when it is colored. This makes the object that interferes with the viewing invisible in the virtual viewpoint image, allowing the user to view the image without worrying about the object that interferes with the viewing.

[0049] Although the wire cam has been described as an example, the object that interferes with viewing is not limited to this. For example, the object that interferes with viewing may be a small aircraft such as a drone or a flying object with a camera attached. Also, the object that interferes with viewing may be a fixed camera.

[0050] [Embodiment 2] In the first embodiment, when an object interferes with viewing, the three-dimensional model is subjected to a transparent process when it is colored, thereby suppressing the display of the object in the virtual viewpoint image. Next, in the second embodiment, a three-dimensional model is deleted in advance when the object corresponds to an excluded object that interferes with viewing, and rendering is performed using only the remaining three-dimensional model, thereby suppressing the display of the excluded object in the virtual viewpoint image. Note that the description of the contents common to the first embodiment, such as the basic configuration of the image processing system, will be omitted or simplified, and the following description will focus on the differences.

[0051] (Server software configuration) Fig. 11 is a block diagram showing the software configuration of the server 116 according to this embodiment. In this embodiment, a three-dimensional model that has been determined to be hidden is deleted, thereby generating a virtual viewpoint image in which no objects that would hinder viewing are present in the first place. As shown in Fig. 11, this embodiment also has five functional units (a data input unit 401', a three-dimensional model generation unit 402', a three-dimensional model analysis unit 403', a hidden setting unit 404', and a virtual viewpoint image generation unit 405') like the first embodiment. Each unit will be described below in order, focusing on the parts that differ from the first embodiment.

[0052] A data input unit 401' acquires captured image data from the DB 117 and virtual viewpoint information from the control device 118, and then sends them to a three-dimensional model generating unit 402'. The three-dimensional model generating unit 402' generates a three-dimensional model of the moving object based on the captured image data received from the data input unit 401'. Furthermore, the three-dimensional model generating unit 402' performs a process of adding color information to each voxel constituting the generated three-dimensional model. In adding this color information, a captured image that will be the source of the color information is determined based on the positional relationship between the voxel of interest and each real camera, such as the distance and angle between the voxel of interest and each real camera (weighting may be performed as necessary in the case of multiple captured images), and the RGB value of the corresponding color in the determined captured image is added. The three-dimensional model to which color information has been added in this way on a voxel-by-voxel basis (hereinafter referred to as a "colored three-dimensional model") is sent to the three-dimensional model analyzing unit 403'.

[0053] The three-dimensional model analysis unit 403' performs an analysis process to determine whether the colored three-dimensional model generated by the three-dimensional model generation unit 402' is a three-dimensional model of an excluded object. At this time, it is sufficient to perform the analysis process only when the non-display mode is enabled. The content of the analysis process is basically the same as that described in the first embodiment. However, the color analysis process in this embodiment differs from the first embodiment in that the three-dimensional model already has color information, and the color analysis is performed using the color information. In addition, in this embodiment in which the three-dimensional model of the excluded object is deleted, the non-display flag is not necessary, so the feature information attached to each three-dimensional model does not include the non-display flag. Then, the three-dimensional model analysis unit 403' deletes the colored three-dimensional model determined to match the exclusion condition (=combination condition) by the analysis process without sending it to the virtual viewpoint image generation unit 405'. In other words, only the colored three-dimensional models of the non-excluded objects that remain without being deleted are sent to the virtual viewpoint image generation unit 405'.

[0054] The virtual viewpoint image generating unit 405' generates a virtual viewpoint image using the virtual viewpoint information input from the data input unit 401 and the colored three-dimensional model input from the three-dimensional model analyzing unit 403'. That is, the virtual viewpoint image generating unit 405' performs coloring processing on all the input colored three-dimensional models using color information assigned to each three-dimensional model, and generates an image that represents the view from the virtual viewpoint. In the case of this embodiment, since the three-dimensional models of excluded objects that hinder the viewing are not input to the virtual viewpoint image generating unit 405' in the first place, control by a non-display flag is not performed.

[0055] As in the first embodiment, the above five functional units can be shared among a plurality of information processing devices. For example, the output result of the three-dimensional model analysis unit 403' may be input to another information processing device having the function of the virtual viewpoint image generation unit 405'. In particular, in this embodiment, the three-dimensional model of the excluded object is deleted and display control by the non-display flag is not required, so that it is easy to combine with a general-purpose information processing device (virtual viewpoint image generation device) that receives the three-dimensional model and generates a virtual viewpoint image.

[0056] (Flow of virtual viewpoint image generation process) 12 is a flowchart showing the flow of a virtual viewpoint image generation process in the server 116 according to this embodiment. A series of processes shown in this flowchart is realized by the CPU 201 of the server 116 reading a predetermined program from the ROM 202 or the auxiliary storage device 204 and executing the program.

[0057] Hereinafter, the flow of the virtual viewpoint image generation process according to this embodiment will be described with reference to the flowchart in FIG. 12. Note that, like the flow in FIG. 9 of the first embodiment, the flow in FIG. 12 is assumed to start in response to a start signal for virtual viewpoint image generation from the control device 118. The start signal is assumed to include the above-mentioned virtual viewpoint information and mode selection information indicating whether the non-display mode is enabled or disabled. Also, it is assumed that at the time of starting execution, the captured image data on which the virtual viewpoint image is based has already been input to the data input unit 401'. Below, the differences from the flow in FIG. 9 of the first embodiment will be mainly described.

[0058] In S1201, the non-display setting unit 404' sets the non-display mode to be enabled / disabled based on the mode selection information input from the control device 118. As in S901 in the first embodiment, when the non-display mode is set to be enabled, an exclusion condition for identifying a target object (exclusion object) of the non-display control is also set. In this embodiment, information on the enabled / disabled state of the non-display mode that has been set is sent to the three-dimensional model analysis unit 403', not to the virtual viewpoint image generation unit 405'.

[0059] In S1202, the three-dimensional model generation unit 402' acquires an image of the processing target frame based on the time information included in the virtual viewpoint information. In the following S1203, the three-dimensional model generation unit 402' generates a three-dimensional model with color information added for each object present in the image of the processing target frame. The generated colored three-dimensional model is sent to the three-dimensional model analysis unit 403'.

[0060] In S1204, processing is allocated based on the information on whether the non-display mode is enabled or disabled input from the non-display setting unit 401'. If the non-display mode is disabled, the process proceeds to S1208. At this time, feature information not including a non-display flag is added to all the colored 3D models input from the 3D model generation unit 402', and the models are sent to the virtual viewpoint image generation unit 405' via the 3D model analysis unit 403'.

[0061] On the other hand, if the non-display mode is active, in S1205, the 3D model analysis unit 403' performs the above-mentioned analysis process on each of the colored 3D models input from the 3D model generation unit 402'. That is, for each colored 3D model, its features are identified in terms of position, color, and shape, and it is determined whether the identified features match the features of the excluded object by referring to the exclusion conditions set in S1201.

[0062] In S1206, processing is assigned based on the result of the above analysis processing. If an object determined to be an excluded object exists, the processing proceeds to S1207, and if not, the processing proceeds to S1208.

[0063] In S1207, the 3D model analysis unit 403′ deletes the 3D models of the objects determined to be excluded objects. Then, characteristic information not including a hidden flag is added to the colored 3D models determined not to be excluded objects (i.e., not deleted), and the colored 3D models are sent to the virtual viewpoint image generation unit 405.

[0064] In S1208, the virtual viewpoint image generating unit 405' generates a virtual viewpoint image corresponding to the frame to be processed by using all the input colored 3D models. Specifically, for each input colored 3D model, normal coloring is performed using the assigned color information to generate a virtual viewpoint image that represents the view from the virtual viewpoint. In S1209, it is determined whether to end the generation of the virtual viewpoint image. If the processing is completed for all frames specified by the time information included in the virtual viewpoint information, this processing ends. On the other hand, if there are unprocessed frames, the processing returns to S1202, and the captured image of the next processing target frame is obtained and the processing continues.

[0065] The above is the flow of the virtual viewpoint image generation process according to this embodiment. In the above embodiment, the three-dimensional model of the excluded object is deleted after adding color information to each three-dimensional model, but it is not essential to add color information in advance. For example, instead of adding color information, a deletion flag may be added, and a virtual viewpoint image may be generated using only three-dimensional models whose deletion flag is off. In this case, when coloring, it is preferable to perform occlusion judgment by referring to the information of the bounding box of the three-dimensional model whose deletion flag is on. This makes it possible to prevent the color of the three-dimensional model of the excluded object that is to be deleted from being erroneously used for coloring (texture pasting) and resulting in an unnatural virtual viewpoint image.

[0066] As described above, according to this embodiment, the virtual viewpoint image generating unit 405' can generate a virtual viewpoint image without being aware of the deleted three-dimensional model. In this way, the method of the second embodiment can also appropriately remove objects that hinder appreciation from the virtual viewpoint image. In addition, in the case of the second embodiment, since unnecessary objects are deleted in advance, when an information processing device that receives a three-dimensional model and generates a virtual viewpoint image is a separate device, the separate device does not need to have a control function based on a non-display flag. In other words, a general-purpose information processing device for generating a virtual viewpoint image is sufficient, so the system configuration can be made simpler.

[0067] [Other embodiments] The technology of the present disclosure can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0068] 116 Server 401 Data Entry Department 402 3D model generation unit 403 3D Model Analysis Department 404 Hidden setting section 405 Virtual viewpoint image generation unit

Claims

1. a first generating means for generating shape data representing a three-dimensional shape of an object using a plurality of captured images having different viewpoints; a determining means for determining whether or not an object related to shape data generated by the first generating means corresponds to an object that does not need to be displayed in a virtual viewpoint image, based on at least a condition related to a shape of the object, by using the shape data; a second generating means for generating a virtual viewpoint image in which a display of an object determined by the determining means as an object not requiring display is made more transparent than a display of an object not determined by the determining means as an object not requiring display; 13. An information processing device comprising:

2. the second generating means performs a transparency process on shape data of an object determined not to need to be displayed in the virtual viewpoint image among the shape data generated by the first generating means, thereby generating the virtual viewpoint image.

2. The information processing apparatus according to claim 1,

3. The information processing apparatus according to claim 2 , wherein the transparency processing includes a process of making the image semi-transparent.

4. 4. The information processing device according to claim 2 or 3, characterized in that the determination means further includes an assignment means for assigning information indicating that an object does not need to be displayed in the virtual viewpoint image to shape data of the object that corresponds to an object that does not need to be displayed in the virtual viewpoint image.

5. The first generating means generates shape data to which color information of the object is assigned, The second generating means performs a coloring process according to whether or not the shape data generated by the first generating means corresponds to an object that does not need to be displayed in the virtual viewpoint image, using color information assigned to the shape data generated by the first generating means, to generate the virtual viewpoint image.

5. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

6. a first generating means for generating shape data representing a three-dimensional shape of an object using a plurality of captured images having different viewpoints; a determining means for determining whether or not an object related to shape data generated by the first generating means corresponds to an object that does not need to be displayed in a virtual viewpoint image, based on at least a condition related to a shape of the object, by using the shape data; a second generating means for generating a virtual viewpoint image by using shape data of objects other than the objects determined by the determining means as objects not requiring display, among the shape data of the objects generated by the first generating means; 13. An information processing device comprising:

7. a deletion means for deleting shape data of an object determined as an object that does not need to be displayed in the virtual viewpoint image; 7. The information processing apparatus according to claim 6, wherein the second generating means generates the virtual viewpoint image by using shape data of objects that have not been deleted.

8. The information processing apparatus according to claim 7 , wherein the second generating means generates the virtual viewpoint image by performing occlusion determination with reference to information of a bounding box included in shape data of the object to be deleted.

9. The information processing device according to any one of claims 1 to 8, characterized in that the determination means further makes the determination based on conditions related to any one of the position, size, and color in three-dimensional space of an object that does not need to be displayed in the virtual viewpoint image.

10. 10. The information processing apparatus according to claim 1, further comprising a setting unit for setting the conditions used for the judgment based on a user instruction.

11. The information processing device according to claim 10, characterized in that the setting means sets, based on a user instruction specifying an object included in the captured image that does not need to be displayed in the virtual viewpoint image, a characteristic of the specified object as the condition.

12. 12. The information processing apparatus according to claim 1, wherein the object that does not need to be displayed in the virtual viewpoint image is a specific moving object.

13. 13 . The information processing apparatus according to claim 12 , wherein the specific moving object is a photographing device that is suspended from a wire stretched across a target space for photographing the photographed image and photographs the image.

14. a first generation step of generating shape data representing a three-dimensional shape of an object using a plurality of captured images having different viewpoints; a determining step of determining whether or not an object related to the shape data generated in the first generating step corresponds to an object that does not need to be displayed in a virtual viewpoint image, based on at least a condition related to a shape of the object, using the shape data; a second generation step of generating a virtual viewpoint image in which display of an object determined in the determination step as an object not requiring display is made more transparent than display of an object not determined in the determination step as an object not requiring display; 13. An information processing method comprising:

15. a first generation step of generating shape data representing a three-dimensional shape of an object using a plurality of captured images having different viewpoints; a determining step of determining whether or not an object related to the shape data generated in the first generating step corresponds to an object that does not need to be displayed in a virtual viewpoint image, based on at least a condition related to a shape of the object, using the shape data; a second generating step of generating a virtual viewpoint image using shape data of objects other than the objects determined in the determining step as objects not requiring display, among the shape data of the objects generated in the first generating step; 13. An information processing method comprising:

16. A program for causing a computer to function as the information processing device according to any one of claims 1 to 13.

Images