Information processing device, information processing method, and program

Abstract

To provide a technique for generating a virtual viewpoint image that makes a user feel less uncomfortable.SOLUTION: A three-dimensional model of an object obtained by photographing the object within a shooting area from multiple positions and information indicative of a position of the object in the shooting area are acquired. A virtual viewpoint image of the object representing a scene viewed from a virtual viewpoint is generated based on the three-dimensional model of the object. Correction according to the position of the object is then made on the object in the virtual viewpoint image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and more particularly to a system that generates a virtual viewpoint image of a subject as viewed from a virtual viewpoint. [Background technology]

[0002] A technology that generates a virtual viewpoint image using a plurality of images obtained by installing a plurality of image capturing devices at different positions and performing synchronized image capturing has been attracting attention. For example, by using such a technology to generate a three-dimensional model of a foreground subject over a plurality of frames, a virtual viewpoint image for analyzing the motion and trajectory of the subject can be generated.

[0003] Patent Document 1 discloses that in generating a virtual viewpoint image, in order to solve the problem of reduced resolution if the distance between the virtual viewpoint and a three-dimensional model of the subject is too close, the transparency of the subject close to the virtual viewpoint in the virtual viewpoint image is increased. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-144638 A [Patent Document 2] Patent Publication No. 2021-33525 Summary of the Invention [Problem to be solved by the invention]

[0005] When creating a three-dimensional model of a subject, a configuration is considered in which an area for generating the three-dimensional model is determined in advance, and multiple imaging devices are installed facing this area. However, when such a configuration is adopted, the quality of the image in the virtual viewpoint image of a subject that exists in a position where it is difficult to capture with high accuracy, such as near the boundary of the area, may be reduced. In addition, by adopting a configuration that does not generate a three-dimensional model of a subject located outside this area, the amount of data of the generated three-dimensional model can be reduced, but when such a configuration is adopted, the shape of the three-dimensional model may be partially missing. As a result, for example, when a part of the face of a human subject is missing, the back of the head may be displayed in the virtual viewpoint image through the face.

[0006] An object of the present disclosure is to provide a technology for generating a virtual viewpoint image that reduces the sense of discomfort felt by the user. [Means for solving the problem]

[0007] An information processing device according to an embodiment of the present disclosure has the following configuration. an acquisition means for acquiring a three-dimensional model of a subject obtained by capturing images of the subject within an imaging area from a plurality of positions, and information indicating a position of the subject relative to the imaging area; A generating means for generating a virtual viewpoint image including the subject based on a three-dimensional model of the subject, the generating means performing a correction on the subject in the virtual viewpoint image according to a position of the subject; Have death, The generating means changes the strength of the correction depending on the distance between the subject and the boundary of the imaging area. . Effect of the Invention

[0008] According to the present disclosure, it is possible to generate an imaginary viewpoint image that reduces the sense of discomfort felt by the user. [Brief description of the drawings]

[0009] [Figure 1]FIG. 1 is a diagram showing an example of a system configuration according to an embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of a hardware configuration of an information processing apparatus according to an embodiment. [Diagram 3] 1 is a flowchart of an information processing method according to an embodiment. [Figure 4] FIG. 4 is a diagram showing the relationship between a subject, an imaging area, and a virtual viewpoint. [Diagram 5] 11 is a graph showing an example of a method for calculating a quality evaluation value and a shading processing amount. [Figure 6] FIG. 13 is a diagram showing an example of a virtual viewpoint image. [Figure 7] FIG. 13 is a diagram showing an example of a virtual viewpoint image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] An information processing device according to an embodiment of the present disclosure generates a virtual viewpoint image of a subject from a virtual viewpoint based on a three-dimensional model of the subject obtained by capturing images of the subject in an imaging area from multiple positions. Hereinafter, an image processing system including such an information processing device and multiple imaging devices will be described. This image processing system generates a virtual viewpoint image representing a view from a specified virtual viewpoint based on multiple images based on images captured by the multiple imaging devices and a specified virtual viewpoint. Note that in this specification, the image is not limited to a still image, and may be an image captured or played over a continuous time period. In addition, in such an image, the virtual viewpoint may be fixed or may move. A virtual viewpoint that observes a subject while moving in this way can also be called camerawork.

[0012] 1 shows an example of the configuration of an image processing system according to an embodiment of the present disclosure. The image processing system 1 has the following components.

[0013] The multiple imaging devices 100 capture images of an imaging area from multiple directions. The imaging area may be, for example, an indoor imaging studio or a stage where a play is performed. The multiple imaging devices 100 are installed at different positions so as to surround such an imaging area, and capture images in synchronization. The multiple imaging devices 100 do not have to be installed all around the imaging area. For example, due to restrictions on installation locations, the imaging devices may be installed only at positions away from the imaging area in a specific direction. The number of imaging devices is not particularly limited. For example, when the imaging area is a soccer stadium, about 30 imaging devices may be installed around the stadium. Imaging devices with different functions may be installed, for example, a telephoto camera and a wide-angle camera.

[0014] The imaging device information 111 is data indicating the position and imaging range of each of the multiple imaging devices 100. For example, the imaging device information 111 may include parameters indicating the three-dimensional position of each of the multiple imaging devices 100 and parameters indicating the imaging direction of the imaging device in the pan, tilt, and roll directions. The imaging device information 111 may also include parameters indicating the size of the field of view (angle of view) and resolution of the imaging device. The imaging device information 111 can be calculated in advance by performing camera calibration using a method that is not particularly limited. For example, the imaging device information 111 can be calculated by associating points in multiple images obtained by imaging with the multiple imaging devices 100 with each other and performing geometric calculations. Note that the content of the imaging device information is not limited to the above content. The imaging device information 111 may have multiple parameter sets. For example, the imaging device information 111 may have multiple parameter sets corresponding to multiple frames constituting a video obtained by imaging with the imaging device. Such imaging device information 111 can indicate the position and direction of the imaging device at each of multiple consecutive time points.

[0015] The object generating device 110 generates a three-dimensional model (three-dimensional object) of a subject based on images received from the multiple imaging devices 100, i.e., images from multiple viewpoints, and imaging device information 111. However, the information processing device 200 may have the function of the object generating device 110.

[0016] The type of the subject is not particularly limited, but may be a person such as a stage performer, who is a foreground subject. The method of generating the three-dimensional model is not particularly limited, but for example, the method described in Patent Document 1 can be used. The three-dimensional model of the foreground subject generated by the object generating device 110 is recorded as a foreground object 101 and transmitted to the information processing device 200.

[0017] A three-dimensional model is data in which information indicating a shape and a color is described. For example, a three-dimensional model may be composed of a textured mesh model or a three-dimensional point cloud in which each point is colored. A three-dimensional model may be composed of a set of data indicating the shape of a subject (for example, a mesh model or a three-dimensional point cloud) and a captured image, which are used in image-based rendering described later. Note that a three-dimensional model may be data in which information indicating a shape is described, and may not necessarily include information indicating a color.

[0018] The background object 112 is a three-dimensional model representing an environment in which the three-dimensional model of the subject is placed. The background object 112 may be a three-dimensional model of an environment different from the foreground, such as a large concert hall, a soccer stadium, or a small indoor room. The background object 112 may be design data such as CAD, or data indicating the shape and color of the background scanned using a laser scanner or the like. Furthermore, the background object 112 may be generated using a computer vision technique such as Structure from Motion using a group of images from multiple viewpoints. Such a background object 112 may be loaded into the image processing system 1 in advance.

[0019] The information processing device 200 according to an embodiment of the present disclosure includes an object acquisition unit 201, an area acquisition unit 202, and a display control unit 203.

[0020] The object acquisition unit 201 acquires a three-dimensional model of a subject obtained by capturing images of the subject in an imaging area from a plurality of positions. As described above, the object acquisition unit 201 can acquire the foreground object 101 generated by the object generation device 110. Here, the foreground object 101 can include three-dimensional models of the subject at a plurality of times corresponding to the shapes and colors of the subject at a plurality of times. The object acquisition unit 201 can further acquire a background object 112.

[0021] The area acquisition unit 202 acquires information indicating the position of the subject relative to the imaging area. In this embodiment, the area acquisition unit 202 acquires area information 102 describing geometric information of the imaging area surrounded by multiple imaging devices 100. The area information 102 can indicate the position and size of the area in three-dimensional space, and may be, for example, geometric information of a rectangular parallelepiped describing the center coordinates and the lengths of the X, Y, and Z axes. Such geometric information can be defined by a coordinate system used to indicate the position of the imaging device 100 described in the imaging device information 111.

[0022] In this embodiment, the foreground object 101 defines the position of the subject in three-dimensional space by, for example, a coordinate system used to indicate the position of the imaging device 100 described in the imaging device information 111. Therefore, the position of the subject relative to the imaging area can be known by such area information 102. Particularly, in this embodiment, the foreground object 101 defines the position of each part of the subject in three-dimensional space. Therefore, the position of each part of the subject relative to the imaging area can be known by such area information 102. The area information 102 divides the three-dimensional space into two or more parts and can indicate any shape such as a sphere or a cylinder. The area information 102 may indicate a plurality of areas. Note that such area information 102 may be included in the foreground object 101. Furthermore, the foreground object 101 may include information indicating the position of the subject relative to the imaging area in a format different from the area information 102 in addition to the three-dimensional model of the subject. Note that the information indicating the position of the subject relative to the imaging area does not need to be information that specifies the position of the subject relative to the imaging area in one location. This information may be information indicating the distance of the subject from the boundary of the imaging area.

[0023] The display control unit 203 generates a virtual viewpoint image including the subject from the virtual viewpoint based on a three-dimensional model of the subject. For example, the display control unit 203 can render a virtual viewpoint image showing the foreground object seen from the virtual viewpoint 103 based on the foreground object 101, the area information 102, and the virtual viewpoint 103. Furthermore, the display control unit 203 can render a background object seen from the virtual viewpoint 103 based on the background object 112 into the virtual viewpoint image.

[0024] Here, the display control unit 203 performs correction on the subject in the virtual viewpoint image according to the position of the subject relative to the imaging area. For example, the display control unit 203 can perform correction on the subject in the virtual viewpoint image with a strength set according to the position of the subject relative to the imaging area. In the case of this embodiment, the display control unit 203 corrects a part of the subject in the virtual viewpoint image selected according to the position of each part of the subject relative to the imaging area. In other words, the display control unit 203 performs correction with a certain degree of strength on one part of the subject, while performing correction with a strength of 0 on the other part of the subject. In addition, the display control unit 203 may perform correction with a strength on each part of the subject in the virtual viewpoint image according to the position relative to the imaging area.

[0025] The display control unit 203 can select the portion to be corrected or change the strength of the correction depending on the distance between the subject and the boundary of the imaging area. In the embodiment described below, the display control unit 203 performs a shading process on the area boundary portion of the foreground object, thereby correcting the area boundary portion of the subject. The strength of the shading process increases the closer to the area boundary portion. With this configuration, it is possible to perform a shading process on a defective portion of the three-dimensional model of the subject when the subject is located at the area boundary.

[0026] The display control unit 203 can also transmit data of the virtual viewpoint image to the display device 300. The display device 300 is a device capable of displaying the received virtual viewpoint image, and can be, for example, a liquid crystal display, a projector, or a head-mounted display.

[0027] The virtual viewpoint 103 is data indicating the position and imaging range of a virtual viewpoint. For example, the virtual viewpoint 103 can describe the three-dimensional position, direction, angle of view, and resolution of a virtual camera virtually existing at the virtual viewpoint. The virtual viewpoint 103 may describe this information in the same format as the imaging device information 111. The virtual viewpoint 103 may be received from an operation interface such as an external controller (not shown).

[0028] 2 shows an example of the hardware configuration of the information processing device 200. Note that the object generating device 110 and the display device 300 can also be realized by the same hardware configuration as the information processing device 200 described below.

[0029] The information processing device 200 includes a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a display unit 215, an operation unit 216, a communication I / F 217, and a bus 218. The CPU 211 realizes each function of the information processing device 200 shown in FIG. 1 by controlling the entire information processing device 200 using computer programs and data stored in the ROM 212 or the RAM 213. Note that the information processing device 200 may include one or more dedicated hardware pieces different from the CPU 211, in which case the dedicated hardware pieces can execute at least a part of the processing by the CPU 211. Examples of the dedicated hardware pieces include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor).

[0030] The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs or data supplied from the auxiliary storage device 214, and data supplied from the outside via the communication I / F 217. The auxiliary storage device 214 is configured with a storage such as a hard disk drive, and stores various data such as image data or audio data. The display unit 215 is configured with a display device such as a liquid crystal display or an LED, and displays a GUI (Graphical User Interface) that the user uses to operate the information processing device 200. The operation unit 216 is configured with a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 211 according to operations by the user.

[0031] The CPU 211 operates as a display control unit that controls the display unit 215 and an operation control unit that controls the operation unit 216. The communication I / F 217 is used for communication with an external device of the information processing device 200. For example, when the information processing device 200 is connected to an external device by wire, a communication cable is connected to the communication I / F 217. When the information processing device 200 performs wireless communication with an external device, the communication I / F 217 is provided with an antenna. The bus 218 connects each unit of the information processing device 200 to transmit information. In the example of FIG. 2, the display unit 215 and the operation unit 216 are present inside the information processing device 200, but at least one of the display unit 215 and the operation unit 216 may be present as a separate device outside the information processing device 200. In addition, the information processing device 200 may be configured by a plurality of information processing devices connected via a network, for example.

[0032] Next, the processing performed by the information processing device 200 will be described with reference to Fig. 3. In S310, the area acquisition unit 202 acquires the area information 102. In this example, the area information 102 indicates, in the same coordinate system as the imaging device information 111, a rectangular parallelepiped whose center is (0,0,2) relative to the origin (0,0,0), and which extends 10 m in the X- and Y-axis directions and 4 m in the Z-axis direction, as an area. The area acquisition unit 202 acquires the area information 102 from the auxiliary storage device 214, and temporarily stores it in the RAM 213.

[0033] In S320, the object acquisition unit 201 acquires the foreground object 101 and the background object 112, and temporarily stores them in the RAM 213. In this example, these objects are three-dimensional mesh polygon data recorded in a general-purpose format such as Stanford PLY or Wavefront OBJ. The object acquisition unit 201 may acquire three-dimensional point cloud data instead of a three-dimensional mesh. At the same time, in S320, the display control unit 203 acquires the virtual viewpoint 103, and temporarily stores it in the RAM 213.

[0034] 4 is an overhead view of the area indicated by the area information 102, the foreground object 101, and a virtual camera virtually placed at the virtual viewpoint 103. The foreground object 101 includes objects 1010 and 1011 representing two people. Since part of the face and torso of the person represented by the object 1011 is outside the area, the object 1011 does not show the shape of this part 10110. This part 10110 of the object 1011 has a hole, and the inside of the object 1011 is hollow.

[0035] In S330, the display control unit 203 evaluates the position of the subject relative to the imaging area. In this example, the display control unit 203 calculates a quality evaluation value of the foreground object 101 based on the distance between the area indicated in the area information 102 and the foreground object 101. As shown in FIG. 4, the shape of the three-dimensional model is missing at the boundary of the area. Also, there is a possibility that there are few cameras directed at the boundary of the area, and the quality of the image of the subject may be reduced. In this way, the display control unit 203 can estimate or evaluate the quality of the image of each part of the subject in the virtual viewpoint image according to the position of this part relative to the imaging area, more specifically, according to the distance of each part of the subject to the boundary of the area.

[0036] The quality evaluation value is calculated for each component of the foreground object 101, that is, for each vertex or three-dimensional point of the mesh. This quality evaluation value may be a real value in the range of 0 to 1.0, for example, and the shorter the shortest distance (length of perpendicular line) from the component to the boundary of the area, the smaller the value, and the longer the shortest distance, the larger the value. FIG. 5(A) is a graph showing an example of the relationship between distance and quality evaluation value. In FIG. 5(A), the horizontal axis indicates the shortest distance between the boundary of the area indicated by the area information 102 and the component of the foreground object 101, and the vertical axis indicates the quality evaluation value. In this example, the position of each component of the foreground object 101 is indicated in the same three-dimensional coordinate system as the area information 102 and the imaging device information 111. Therefore, the position of each component of the foreground object 101 corresponds to the position of the part of the subject corresponding to this component. When a component of the foreground object is in contact with the boundary of the area, that is, when the distance is 0, the quality evaluation value of this component of the foreground object 101 is 0. Furthermore, when the distance is 5 cm, the quality evaluation value is 1.0, and when the distance is longer than that, the quality evaluation value is also 1.0. Through this process, each component of object 1010 has a quality evaluation value of 1.0, and the quality evaluation value of the periphery of part 10110 of the component of object 1011 that is outside the area becomes a value close to 0.

[0037] At this time, the display control unit 203 can interpolate the missing part of the three-dimensional model of the subject. For example, the display control unit 203 can interpolate the missing part of the three-dimensional model of the subject based on the distance between the part of the subject corresponding to each component of the three-dimensional model of the subject and the boundary of the imaging area. Specifically, the display control unit 203 can generate, by interpolation, a surface of the object 1011 surrounded by contour lines having the same quality evaluation value. At this time, the same quality evaluation value as that of the outline part can be applied to the interpolated part. With this configuration, the part 10110 of the object 1011 that has no shape can be interpolated, and holes in the object 1011 can be filled.

[0038] In S340, the display control unit 203 determines the amount of shading processing based on the quality evaluation value. Fig. 5(B) is an example of a graph showing the relationship between the quality evaluation value and the amount of shading processing. In Fig. 5(B), the horizontal axis indicates the quality evaluation value, and the vertical axis indicates the amount of shading processing. In boundary areas where the quality evaluation value is 0, the amount of shading processing is 1.0. Also, in areas where the quality evaluation value is 1.0 or more, the amount of shading processing is 0, meaning that no shading processing is performed in these areas.

[0039] In S350, the display control unit 203 generates a virtual viewpoint image of the subject from a virtual viewpoint based on a three-dimensional model of the subject. In this example, the display control unit 203 renders the foreground object 101 and the background object 112 by projecting these objects onto the virtual viewpoint 103. As a projection method, model-based rendering or image-based rendering described in Patent Document 1 can be used. For example, if the foreground object 101 is a colored three-dimensional model, the color of each point can be projected onto the virtual viewpoint image. Also, if the foreground object 101 is composed of a plurality of images depicting the foreground in addition to three-dimensional shape data, the color of each point can be determined by referring to the images during rendering.

[0040] Furthermore, the display control unit 203 performs correction on the subject in the virtual viewpoint image according to the position of the subject relative to the imaging area. In this example, a shading process is performed on the components of the foreground object 101 having a quality evaluation value less than 1.0. More specifically, a shading image can be superimposed on an area of ​​the virtual viewpoint image in which the components of the foreground object 101 having a quality evaluation value less than 1.0 are reflected. With this method, the display control unit 203 can correct a part of the subject in the virtual viewpoint image selected according to the position relative to the imaging area.

[0041] An example of the mesh processing will be described below. First, a layer image for generating a mesh image is prepared. Next, the color of the mesh can be set in advance by a user input, and is set to cyan in this example. Then, a mesh pattern is drawn on the layer image. The mesh pattern may be composed of lines with a thickness of 1 pixel drawn at intervals of 1 pixel, for example. Furthermore, lines with a thickness of 2 pixels representing the contour portion are drawn on the layer image. The position of the contour portion can be determined, for example, by projecting onto the virtual viewpoint image the components of the foreground object 101 with a quality evaluation value of 0, or the components of the contour portion of the surface interpolated as described above according to the lowest quality evaluation value. In this way, the display control unit 203 can emphasize the contour of the portion on the virtual viewpoint image that corresponds to the missing portion of the three-dimensional model. With this configuration, it is easy to understand that the quality-degraded portion reflected in the virtual viewpoint image is due to a missing portion of the three-dimensional model.

[0042] The layer image thus generated and the rendering image of the object are alpha blended using the amount of shading as a coefficient. That is, the luminance value of each pixel of the virtual viewpoint image onto which the foreground object 101 and the background object 112 are projected is (R, G, B), the luminance value of each pixel of the layer image is (Lr, Lg, Lb), and the amount of shading is a. In this case, the luminance value (Ro, Go, Bo) of each pixel after shading is calculated as (Ro, Go, Bo) = a × (Lr, Lg, Lb) + (1-a) × (R, G, B).

[0043] FIG. 6 shows a virtual viewpoint image 1030 as an example of the rendering result. The object 1010 is not affected by the shading process because it is 5 cm or more away from the boundary of the area. A shapeless portion 10110 of the object 1011 is shading processed with a shading amount of 1.0. Moreover, a peripheral portion 10111 of the object 1011 is shading processed with a smaller shading amount as it is farther from the shapeless portion 10110. With this method, it is possible to correct the intensity of each portion of the subject in the virtual viewpoint image according to its position relative to the imaging area. On the other hand, no shading process is performed on the portion of the object 1011 that is 5 cm or more away from the boundary of the area. When foreground objects overlap on the virtual viewpoint image, a distance image of each object is created based on the shape, and occlusion determination is performed based on this distance image, so that rendering can be performed so that the images of the subjects overlap.

[0044] In S360, the display device 300 updates the display with the virtual viewpoint image 1030.

[0045] In S370, the object acquisition unit 201 and the display control unit 203 search inside the auxiliary storage device 214 to check whether the foreground object 101 and the virtual viewpoint 103 of the next frame are present. If they are present, the frame is updated in S380 and the process proceeds to S320. If they are not present, the process ends.

[0046] As described above, in this embodiment, the correction according to the positional relationship of the subject to the imaging area is performed on the subject in the virtual viewpoint image. Specifically, a shading process is performed on a portion of the foreground object close to the boundary of the imaging area. In this way, by performing a shading process on a portion of the virtual viewpoint image where the quality of the image of the subject deteriorates, the unnatural portion becomes less noticeable, and the sense of incongruity felt by the user can be reduced. In addition, with this configuration, the possibility that the inside of the three-dimensional model is rendered through a portion of the three-dimensional model that is missing because the subject is located at the boundary of the area is reduced, and deterioration of image quality can be reduced.

[0047] However, the method of the shading process is not limited to the above method. For example, the color of the net, as well as the thickness and interval of the lines, may be changed. Also, a layer image having a shading pattern may be prepared in advance. Also, the method of correcting the subject in the virtual viewpoint image is not limited to the shading process. For example, the display control unit 203 can perform color correction on the subject. Specifically, the display control unit 203 can perform shading process, color blending process, blurring process, or subject transparency process. When performing color blending process, for example, a layer image with one side filled in can be used instead of a layer image having a shading pattern. Furthermore, instead of performing shading process on a part in which a component of the foreground object 101 with a low quality evaluation value is reflected, blurring process using a Gaussian filter or the like may be performed on this part. In this way, the display control unit 203 can perform correction to reduce the visibility of the subject using various methods. Even with such a configuration, by changing the color of the subject in a part of the virtual viewpoint image with low quality, this part becomes less noticeable, so that the sense of incongruity given to the user can be reduced.

[0048] As another such example, a method of making a foreground object close to an area boundary transparent instead of performing a shading process will be described. In this example, in S330, the display control unit 203 calculates a quality evaluation value for each foreground object based on the distance between the area boundary and the component of the foreground object that is closest to the area boundary. In addition, in S340, the display control unit 203 determines an alpha value based on the quality evaluation value. For example, the alpha value can be set to 0 in a boundary portion where the quality evaluation value is 0. In addition, when the quality evaluation value is from 0 to 1.0, the alpha value can be increased linearly, and when the quality evaluation value is 1.0, the alpha value can be set to 1.0.

[0049] Then, in S350, the display control unit 203 performs alpha blending on the foreground object 101 and the background object 112. That is, for each pixel, the luminance value of the foreground object 101 projected onto the virtual viewpoint image is (R_fg, G_fg, B_fg), the luminance value of the background object 112 is (R_bg, G_bg, B_bg), and the alpha value is α. In this case, the luminance value (Ro, Go, Bo) of each pixel after the transparency process can be expressed as (Ro, Go, Bo) = α × (R_fg, G_fg, B_fg) + (1 - α) × (R_bg, G_bg, B_bg).

[0050] According to this method, a transparency process is performed on the image of the subject whose quality deteriorates in the virtual viewpoint image. FIG. 7 shows an example of a virtual viewpoint image 1030 rendered in this way. The object 1010 is not affected by the transparency process because it is 5 cm or more away from the boundary of the area. On the other hand, the object 1011 is in contact with the boundary of the area and the distance is 0, so that the transparency is maximum. That is, a correction is performed so that the object 1011 does not appear on the virtual viewpoint image 1030. According to this method, a transparency process is performed on the foreground object so that the closer the foreground object is to the boundary of the area, the more transparent it becomes. Even with this transparency process, an object including a low-quality part, for example a part with a missing shape, is made transparent, so that the sense of incongruity felt by the user can be reduced, and deterioration of image quality can be reduced.

[0051] In the above-described embodiment, the distance between the area boundary and the foreground object is used as information indicating the positional relationship of the subject to the imaging area. However, the information indicating the positional relationship of the subject to the imaging area is not limited to such a distance. In the following, a case will be described in which the strength of correction is changed based on an evaluation value set in advance for each position in the imaging area based on the arrangement of multiple imaging devices relative to the imaging area and the position of the subject. In the following example, a quality evaluation value is defined in advance for each point in the imaging area, and correction based on the quality evaluation value of the position of the subject in the imaging area is performed on the subject in the virtual viewpoint image.

[0052] The quality evaluation value of each point in the imaging area can be determined according to the image quality at that point or the distance to the imaging device. As a specific example, the method described in Patent Document 2 can be used. That is, coordinates serving as evaluation indices are set on lattice points in the area. Then, the quality evaluation value can be determined according to the size or position of an object present at each position in the imaging area in an image captured by one of the imaging devices. More specifically, when each coordinate is viewed from each of eight directions, for example, the evaluation value can be calculated according to the viewpoint, the coordinate, and the angle formed by the imaging device that captures the image at the angle closest to the viewpoint, and the imaging resolution of the coordinate by that imaging device. Here, the imaging device that captures the image at the angle closest to the viewpoint is the imaging device that, among multiple imaging devices, forms the smallest angle between the viewpoint, the coordinate, and the imaging device.

[0053] In this case, the evaluation value can be calculated as follows. When the above angle is θ and the resolution is δ (px / mm), the evaluation value q can be calculated as q=cos(θ)×(δ-δ_min) / (δ_max-δ_min). Here, δ_min and δ_max are the minimum and maximum values ​​of δ in all lattice points and all directions in the area, respectively, and the second term in this formula is for normalizing q so that the range of values ​​is 0 to 1.0. Such evaluation values ​​can be calculated for each lattice point and each of the eight directions. Then, the evaluation value when viewing the evaluation index closest to the foreground object or each component of the foreground object from the direction closest to the direction from the virtual viewpoint among the eight directions can be used as the quality evaluation value. The evaluation value calculated in advance in this way can be acquired from the auxiliary storage device 214 by the display control unit 203 in S310 and temporarily stored in the RAM 213.

[0054] Even with this configuration, the sense of incongruity felt by the user can be reduced by applying stronger correction processing such as shading or transparency processing to low-quality parts of the virtual viewpoint image. In particular, degradation of image quality can be reduced by determining the amount of shading processing based on the degradation of quality caused by the number, angle, or resolution of the cameras capturing images of each point within the area.

[0055] These quality evaluation methods can also be used in combination. For example, a quality evaluation value calculated based on the distance from the area boundary and the above-mentioned quality evaluation value predefined for each point in the area can be integrated, and the strength of the correction process can be determined based on the integrated quality value. Examples of the integration method include a method of calculating a statistical quantity such as an average, a minimum, or a maximum. Also, three or more types of quality evaluation values ​​can be integrated and used.

[0056] So far, we have explained a method for performing correction on a subject in a virtual viewpoint image according to the subject's position relative to the imaging area. Below, we will explain a method for performing correction on a portion of the virtual viewpoint image in which the back side of a three-dimensional model is shown. In the following example, front / back analysis is performed using the shape of the foreground object without using area information.

[0057] The information processing device 200 according to this embodiment has a shape analysis unit 221 instead of the area acquisition unit 202. The shape analysis unit 221 determines the back side of each part of the three-dimensional model of the subject. Here, the front side of the three-dimensional model refers to the surface side of the subject corresponding to the three-dimensional model, and the back side of the three-dimensional model refers to the internal side of the subject corresponding to the three-dimensional model. Specifically, when the three-dimensional shape of the subject is represented by mesh data, it is the back surface of the polygon that constitutes the mesh data and represents the surface of the subject. Such a polygon has a normal direction determined, and the polygon has a front surface and a back surface with respect to the normal direction. In this way, the shape analysis unit 221 analyzes the front and back based on the shape of the object, and transmits the analysis result to the display control unit 203. Then, the display control unit 203 determines the part of the virtual viewpoint image where the back side of the three-dimensional model is reflected, and performs correction for the part where the back side is reflected.

[0058] Such an embodiment can also be performed according to the flowchart shown in FIG. 3, and the following describes the parts that are different from the above embodiment. S310 is omitted. In S330, the shape analysis unit 221 determines the direction of the back side of each part of the three-dimensional model. For example, the shape analysis unit 221 can determine the front and back of a component of the three-dimensional model such as a face or a point. If the three-dimensional model of the foreground object contains normal information indicating the front and back, the shape analysis unit 221 can use this normal information. On the other hand, if such normal information is not included, the shape analysis unit 221 can newly calculate normal information.

[0059] The normal of a face (polygon) can be calculated as follows. That is, two vectors are determined so as to connect the vertices constituting the face in order, and the direction of the cross product of these vectors is the direction of the normal. Furthermore, a point is determined outside the foreground object, and the face whose normal faces the outside of the object can be determined based on the positive or negative inner product of the vector from that point toward a face or point on the object and the previously calculated normal vector, and this face becomes the front face. The opposite face becomes the back face. Note that the average of vectors from a focused part of an object to various other parts of the same object may be calculated, and the side of the focused part that faces the end point of the vector may be determined as the back face. The normal of a point can also be calculated as follows. When one of the vertices on the object is focused on, the average of the normal vectors of the adjacent faces is calculated, and this can be used as the normal of the vertex.

[0060] Furthermore, the shape analysis unit 221 sets the quality evaluation value of the front side of the foreground object high and the quality evaluation value of the back side lower than this. For example, the quality evaluation value of the front side can be set to 1.0 and the quality evaluation value of the back side can be set to 0.

[0061] Thereafter, in S340, the display control unit 203 determines the strength of correction such as shading or transparency processing based on the quality evaluation value. In this example, the display control unit 203 projects the foreground object to the virtual viewpoint, and can set the amount of shading processing to 1.0 for pixels onto which the back surface of the foreground object is projected, and set the amount of shading processing to 0 for pixels onto which the front surface of the foreground object is projected. The amount of shading processing for pixels onto which the back surface is projected may be set according to a user input. Furthermore, the quality evaluation value or the strength of correction may be determined according to the distance between the virtual viewpoint and the object, the area of ​​the region onto which the back surface is projected, or the like.

[0062] As described above, by performing correction based on the shape analysis of the foreground object, it is possible to reduce the discomfort felt by the user in low-quality areas, for example, areas where the back side of an object is displayed, even without area information. Of course, correction based on the shape analysis of the foreground object and correction based on the positional relationship of the subject with respect to the imaging area may be used in combination.

[0063] In the above embodiment, the rendering of foreground objects has been mainly described. On the other hand, the above-mentioned correction can also be performed in the rendering of a shadow that a foreground object casts on another foreground object or a background object. That is, the rendering of such a shadow of a foreground object is also included in the rendering of a foreground object in this specification. In this case, the correction to the image of the foreground object can also be applied to the shadow of the foreground object. That is, the display control unit 203 can draw the shadow of the subject in the virtual viewpoint image and perform correction of the strength according to the position on the shadow of the subject in the virtual viewpoint image.

[0064] As an example, a case where a transparent process is performed on a shadow will be described. In this case, a layer image in which only the shadow is rendered by ray tracing can be created after setting a light source according to a shadow rendering method in a normal computer graphics technology. Furthermore, a transparent shadow can be drawn by alpha blending a layer image with a background object rendered using an alpha value similar to that of the foreground object for a virtual viewpoint image in which a background object is rendered. With this configuration, the virtual viewpoint image can be corrected so that the shadow of the subject becomes transparent in accordance with the subject becoming transparent based on the positional relationship with respect to the area, thereby reducing the sense of incongruity felt by the user.

[0065] (Other Examples) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions. [Explanation of symbols]

[0066] 1: image processing system, 100: imaging device, 110: object generating device, 200: information processing device, 201: object acquiring section, 202: area acquiring section, 203: display control section, 221: shape analyzing section, 300: display device

Claims

1. an acquisition means for acquiring a three-dimensional model of a subject obtained by capturing images of the subject within an imaging area from a plurality of positions, and information indicating a position of the subject relative to the imaging area; A generating means for generating a virtual viewpoint image including the subject based on a three-dimensional model of the subject, the generating means performing a correction on the subject in the virtual viewpoint image according to a position of the subject; having The information processing apparatus according to claim 1, wherein the generation means changes the strength of the correction depending on the distance between the subject and a boundary of the imaging area.

2. 2. The information processing device according to claim 1, wherein the generating means interpolates missing portions of the three-dimensional model of the subject based on distances between portions of the subject corresponding to each component of the three-dimensional model of the subject and a boundary of the imaging area.

3. the acquiring means acquires information indicating a position of each part of the subject relative to the imaging area; 3 . The information processing apparatus according to claim 1 , wherein the generating means corrects a part of the subject in the virtual viewpoint image selected according to a position of each part of the subject.

4. the acquiring means acquires information indicating a position of each part of the subject relative to the imaging area; 3 . The information processing apparatus according to claim 1 , wherein the generating means corrects the intensity of each part of the subject in the virtual viewpoint image according to the position of each part of the subject.

5. An acquisition means for acquiring a three-dimensional model of a subject obtained by imaging the subject from a plurality of positions within an imaging area and information indicating a position of the subject relative to the imaging area; A generating means for generating a virtual viewpoint image including the subject based on a three-dimensional model of the subject, the generating means performing a correction on the subject in the virtual viewpoint image according to a position of the subject; having The information processing device characterized in that the generation means changes the strength of the correction based on an evaluation value that is set in advance for each position within the imaging area based on an arrangement of multiple imaging devices relative to the imaging area, and based on the position of the subject.

6. 6. The information processing device according to claim 5, wherein the evaluation value is determined according to a size or a position of the subject present at each position within the imaging area in an image captured by any one of the imaging devices.

7. The information processing device according to claim 1 , wherein the generating means renders a shadow of the subject in the virtual viewpoint image and corrects the strength of the shadow of the subject in the virtual viewpoint image according to the position of the subject.

8. The information processing apparatus according to claim 1 , wherein the correction performed by the generating means is color correction.

9. The information processing apparatus according to claim 1 , wherein the correction performed by the generating means is a screen processing, a color blending processing, a blurring processing, or a processing for making the subject transparent.

10. 10. The information processing apparatus according to claim 1, wherein the generating means enhances an outline of a portion on the virtual viewpoint image that corresponds to a missing portion of the three-dimensional model.

11. The information processing apparatus according to claim 1 , wherein the generating means performs the correction so as to reduce visibility of the subject.

12. An information processing method performed by an information processing device, obtaining a three-dimensional model of a subject obtained by imaging the subject from a plurality of positions within an imaging area and information indicating a position of the subject relative to the imaging area; generating a virtual viewpoint image including the subject based on a three-dimensional model of the subject, and performing a correction on the subject in the virtual viewpoint image according to a position of the subject; having The information processing method according to claim 1, wherein in the generating step, the strength of the correction is changed depending on the distance between the subject and a boundary of the imaging area.

13. An information processing method performed by an information processing device, comprising: obtaining a three-dimensional model of a subject obtained by imaging the subject from a plurality of positions within an imaging area and information indicating a position of the subject relative to the imaging area; generating a virtual viewpoint image including the subject based on a three-dimensional model of the subject, and performing a correction on the subject in the virtual viewpoint image according to a position of the subject; having an information processing method, characterized in that in the generating step, the strength of the correction is changed based on an evaluation value that is set in advance for each position within the imaging area based on an arrangement of multiple imaging devices relative to the imaging area, and on the position of the subject.

14. A program for causing a computer to function as the information processing device according to any one of claims 1 to 11.

Images