Information processing device, method for controlling information processing device and program

Abstract

To provide a technology for generating a virtual viewpoint image by reducing complexity for searching an image to be a noise cause and effectively using image data other than a noise area.SOLUTION: An information processing device for generating a virtual viewpoint image on the basis of an image group obtained by photographing a subject from a plurality of directions includes virtual viewpoint image acquisition means for acquiring a first virtual viewpoint image, image coordinate acquisition means for acquiring at least one image coordinates of a specific area on the first virtual viewpoint image, calculation means for calculating a parameter for generating the virtual viewpoint image on the basis of a pixel value of the least one image coordinates, and generation means for generating a second viewpoint image by using the calculated parameter.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] Recently, volumetric technology has been attracting attention, in which multiple image capturing devices are installed in different positions, capture images synchronously, and generate a virtual viewpoint image by synthesizing the multiple images based on the capture. With conventional volumetric technology, the virtual viewpoint image can have unnatural colors due to errors in the shape of the generated subject or noise in the images captured by each image capturing device.

[0003] Patent Document 1 discloses that the images used in synthesizing the virtual viewpoint image are displayed in descending order of their contribution rate. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-106617 A [Patent Document 2] JP 2018-42237 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the technology described in Patent Document 1, the user needs to search through multiple displayed images to find the image that is causing the noise, and must take measures such as disabling the image in order to eliminate the noise. This creates problems such as the complexity of the work and the waste of time caused by disabling image data other than noise.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology that reduces the complexity of searching for images that cause noise, and effectively uses image data other than noise areas to generate a virtual viewpoint image. [Means for solving the problem]

[0007] To achieve the above object, an information processing device according to one aspect of the present invention comprises: An information processing device that generates a virtual viewpoint image based on a group of images obtained by photographing a subject from a plurality of directions, A virtual viewpoint image acquisition means for acquiring a first virtual viewpoint image; an image coordinate acquisition means for acquiring at least one image coordinate of a specific area on the first virtual viewpoint image; a calculation means for calculating a parameter for generating the virtual viewpoint image based on a pixel value of the at least one image coordinate; a generating means for generating a second virtual viewpoint image by using the calculated parameters; The present invention is characterized by comprising: Effect of the Invention

[0008] According to the present invention, it is possible to reduce the complexity of searching for images that cause noise, and to generate a virtual viewpoint image by effectively using image data other than noise areas. [Brief description of the drawings]

[0009] [Figure 1] 1 shows a system configuration diagram according to a first embodiment. [Diagram 2] 1 is a schematic diagram of a space to be photographed according to a first embodiment. [Diagram 3] 1 shows a hardware configuration diagram of an information processing device according to a first embodiment. [Figure 4] 4 is a flowchart showing the procedure of a process executed by the information processing device according to the first embodiment. [Diagram 5] 3 shows a UI screen displayed on the display device according to the first embodiment. [Figure 6] 4 shows a processing result displayed on the display device according to the first embodiment. [Figure 7] 13 shows a UI screen displayed on a display device according to a modified example of the first embodiment. [Figure 8] 13 shows a processing result displayed on a display device according to a modified example of the first embodiment. [Figure 9] 1 shows a system configuration diagram according to a second embodiment. [Figure 10] 11 is a flowchart showing the procedure of a process executed by an information processing device according to the second embodiment. [Figure 11] 11 shows a UI screen displayed on a display device according to the second embodiment. [Figure 12] 11 shows a processing result displayed on a display device according to the second embodiment. [Figure 13] 13 shows a system configuration diagram according to a third embodiment. [Figure 14] 11 is a flowchart showing the procedure of a process executed by an information processing device according to a third embodiment. [Figure 15] 13A shows an input image of the learning model according to the third embodiment, and FIG. 13B shows an inference result image of the learning model according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] (Embodiment 1) In this embodiment, in a system that generates a virtual viewpoint image based on images of a subject photographed from multiple directions, an example is described in which a virtual viewpoint image is generated in which noise has been removed based on the pixel values ​​of noise in an area specified by a user.

[0012] It should be noted that the images described in this embodiment are not limited to still images, but also include video that is shot or played back over a continuous period of time.

[0013] <System configuration> Fig. 1 shows a system configuration diagram according to the first embodiment. A plurality of image capturing devices 100 capture images of an image capturing area from a plurality of directions. The image capturing area may be, for example, an indoor film studio or a stage where a play is performed. Fig. 2 shows a schematic diagram of a space in which image capturing takes place. The plurality of image capturing devices 100 are fixedly installed at different positions, for example, so as to surround the image capturing area, and capture images synchronously.

[0014] The multiple image capturing devices 100 do not have to be installed all around the image capturing area, and may be installed only in a part of the image capturing area depending on the restrictions of the installation location. The number of image capturing devices is not limited to the example shown in Figures 1 and 2. For example, if the image capturing area is a soccer stadium, multiple image capturing devices may be installed around the stadium. Image capturing devices with different functions, such as a telephoto camera and a wide-angle camera, may be installed.

[0015] 2, the imaging devices 100 are connected in a cascade connection using wired communication cables, and the ends are bundled together using a switch hub or the like and then connected to the volumetric data generation device 112. The connection method is not limited to this, and wireless transmission may be performed, or a star connection configuration in which each of the imaging devices 100 and the volumetric data generation device 112 are directly connected to each other may be used.

[0016] The volumetric data generating device 112 acquires images from the image capturing devices 100, generates image capturing device information using the images, and stores the information in the volumetric data storage unit 30. Here, the image capturing device information is a set of parameters including the three-dimensional positions of the multiple image capturing devices 100, the orientation of the image capturing devices in the pan, tilt, and roll directions, and the size of the field of view (angle of view) and resolution of the image capturing devices. The image capturing device information is calculated in advance using a known camera calibration procedure. That is, the image coordinates at which the markers are detected in multiple images acquired by simultaneously capturing the markers using the multiple image capturing devices 100 are associated with each other, and the image capturing device information is calculated using geometric calculations.

[0017] The contents of the imaging device information are not limited to the above. The imaging device information may have a plurality of parameter sets. For example, the imaging device information may have a plurality of parameter sets corresponding to a plurality of frames constituting a video of the imaging device, and may be information indicating the position and direction of the imaging device at each of a plurality of consecutive time points.

[0018] Furthermore, there is no particular limitation on the method of generating volumetric data in the volumetric data generating device 112. For example, shape data of the subject 5 may be generated by the volume intersection method described in Patent Document 1, and stored as volumetric data in the volumetric data storage unit 30. Furthermore, the volumetric data storage unit 30 may include a group of images captured by the imaging device 100.

[0019] The information processing device 200 acquires virtual viewpoint image data from the virtual viewpoint image storage unit 10, and acquires image coordinate data from the image coordinate designation device 20. Moreover, the information processing device 200 acquires volumetric data from the volumetric data storage unit 30. Then, the information processing device 200 updates the virtual viewpoint image data and displays a virtual viewpoint image on the display device 300. Here, the virtual viewpoint image data is generated in advance in the information processing device 200, as shown in step S02020 described later with reference to FIG.

[0020] The image coordinate designation device 20 is, for example, a mouse, and transmits image coordinate data of a position designated by a pointer or clicked on an operation screen described later as, for example, integer values ​​of orthogonal x and y axes with the bottom left corner of the screen as the origin. Alternatively, for example, a touch panel may be used. The display device 300 is a device such as a liquid crystal display or a tablet terminal, and displays the received virtual viewpoint image data on the screen.

[0021] <Functional configuration of information processing device> The information processing device 200 includes a virtual viewpoint image acquisition unit 210 , an image coordinate acquisition unit 220 , a volumetric data acquisition unit 230 , a parameter calculation unit 240 , and a virtual viewpoint image generation unit 250 .

[0022] The virtual viewpoint image acquisition unit 210 acquires virtual viewpoint image data, for example, in full HD (High Definition) resolution and RGB 3 channel format. The data format of the virtual viewpoint image is not limited to this, and may be 4K resolution, RGBA 4 channel format, grayscale image, etc. Furthermore, it may be a format expressed by luminance and color difference.

[0023] The image coordinate acquisition unit 220 acquires image coordinate data designated by the image coordinate designation device 20 as integer values ​​of x and y. The volumetric data acquisition unit 230 acquires volumetric data from the volumetric data storage unit 30.

[0024] Using the acquired virtual viewpoint image, image coordinates, volumetric data, and virtual viewpoint information, the parameter calculation unit 240 calculates virtual viewpoint image generation parameters to be used by the virtual viewpoint image generation unit 250. The virtual viewpoint image generation parameters will be described in detail later.

[0025] In the initial state, the virtual viewpoint is determined by a virtual camera 6 virtually placed at a position looking up at the subject 5 shown in FIG. 2 from below. This is because a camera cannot generally be installed under the floor, and noise is likely to occur. However, this initial state is not limited to this. In addition, the information on the virtual viewpoint is data such as the three-dimensional position and angle of view of the virtual viewpoint described in the same format as the imaging device information, and a data file representing the information on the virtual viewpoint is read in advance and stored in a storage unit (not shown).

[0026] In addition, when the image coordinates are not specified and obtained, the virtual viewpoint image generation parameters may be calculated by the method described in Patent Document 2. The "virtual viewpoint image generation parameters" according to the present embodiment correspond to the "weight" described in Patent Document 2.

[0027] The virtual viewpoint image generating unit 250 generates a virtual viewpoint image by the image blending technique described in Patent Document 2, using the volumetric data, the virtual viewpoint information, and the virtual viewpoint image generation parameters calculated by the parameter calculating unit 240. That is, for each pixel of the virtual viewpoint image, a group of images from each image capturing device 100 is associated based on the shape of the subject, and the color is calculated by blending based on the virtual viewpoint image generation parameters.

[0028] The virtual viewpoint image generated here is converted into data in the same data format as the virtual viewpoint image data in the virtual viewpoint image storage unit 10. Then, the virtual viewpoint image data in the virtual viewpoint image storage unit 10 is updated and transmitted to the display device 300.

[0029] <Hardware configuration of information processing device> 3 shows a hardware configuration diagram of the information processing device 200 according to this embodiment. The hardware configurations of the volumetric data generating device 112 and the display device 300 are also similar to the configuration of the information processing device 200 described below.

[0030] The information processing device 200 includes a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a display unit 215, an operation unit 216, a communication I / F 217, and a bus 218. The CPU 211 controls the entire information processing device 200 using computer programs and data stored in the ROM 212 and the RAM 213, thereby realizing each function of the information processing device 200 shown in FIG.

[0031] The information processing device 200 may have one or more dedicated hardware components different from the CPU 211, and the dedicated hardware components may execute at least a part of the processing performed by the CPU 211. Examples of the dedicated hardware components include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor).

[0032] The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, and data supplied from the outside via the communication I / F 217. The auxiliary storage device 214 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data. The display unit 215 is composed of, for example, a liquid crystal display, an LED, and displays a GUI (Graphical User Interface) for the user to operate the information processing device 200. The operation unit 216 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, and inputs various instructions to the CPU 211 in response to operations by the user. The CPU 211 operates as a display control unit that controls the display unit 215 and an operation control unit that controls the operation unit 216.

[0033] The communication I / F 217 is used for communication with an external device of the information processing device 200. For example, when the information processing device 200 is connected to an external device by wire, a communication cable is connected to the communication I / F 217. When the information processing device 200 has a function of wireless communication with an external device, the communication I / F 217 is equipped with an antenna. The bus 218 connects each unit of the information processing device 200 to transmit information. In this embodiment, the display unit 215 and the operation unit 216 are assumed to exist inside the information processing device 200, but at least one of the display unit 215 and the operation unit 216 may exist as a separate device outside the information processing device 200.

[0034] <Processing> Next, the procedure of the process performed by the information processing device 200 according to this embodiment will be described with reference to the flowchart in Fig. 4. In step S02010, the volumetric data acquisition unit 230 acquires volumetric data from the volumetric data storage unit 30.

[0035] In step S02020, the information processing device 200 reads out predetermined virtual viewpoint information from a storage device (not shown) in a state where image coordinates are not specified. Then, the information processing device 200 calculates virtual viewpoint image generation parameters based on equation (2)-1 described later to generate a virtual viewpoint image. The generated virtual viewpoint image is displayed on the display device 300. In step S02030, the virtual viewpoint image acquisition unit 210 acquires virtual viewpoint image data from the virtual viewpoint image storage unit 10.

[0036] Then, the user, while viewing the virtual viewpoint image via the display device 300, specifies the image coordinates of a specific region (noise region) using the image coordinate specification device 20. FIG. 5 shows an example of a UI screen for specifying image coordinates according to this embodiment, which is displayed on the display device 300. While referring to the virtual viewpoint image, the user clicks on the image coordinates 22 on the noise region 11 at the position of the image coordinate specification device 20, that is, the mouse pointer 21 that moves in response to the operation of the mouse. The image coordinates 22 are the image coordinates on the noise region specified by the user. Note that the image coordinates 22 may be one point (one pixel) or a pixel group including a plurality of pixels. As a result, the image coordinates are input, and the image coordinate data is transmitted from the image coordinate specification device 20.

[0037] In step S02040, the image coordinate acquisition unit 220 receives the image coordinate data transmitted from the image coordinate designation device 20. The received image coordinates 22 are presented to the user via the display device 300. The image coordinates may be not only one point, but also any area that can be input by mouse operation. Alternatively, multiple coordinates may be collectively designated as a rectangular area. In this embodiment, it is assumed that a rectangular area of ​​5×5 pixels is designated. However, the size of the rectangular area is not limited to this example.

[0038] In step S02050, in response to the user pressing the "calc" button 310 on the UI screen, the image coordinate parameter calculation unit 240 starts calculating virtual viewpoint image generation parameters based on the image coordinate data received in step S02040.

[0039] In a system for generating a virtual viewpoint image using volumetric data, a group of images captured by a plurality of image capturing devices 100 are synthesized to calculate pixel values ​​of the virtual viewpoint image. The virtual viewpoint image generation parameter according to this embodiment is a coefficient by which the pixel value of each element in the group of images is multiplied. That is, the pixel value [Ro, Go, Bo] at the pixel position [x, y] of the virtual viewpoint image is calculated using the following formula (1).

[0040]

number

[0041] where I(x,y) and I n (x, y) represents the pixel value of each element in the virtual viewpoint image and the image group at the pixel position x, y received in step S02040. cam represents the number of imaging devices 100, and a n represents the coefficients that are the virtual viewpoint image generation parameters. n is an image calculated by projecting pixels onto the shape data of the subject 5 using only the n-th image in the image group, and is calculated using the virtual viewpoint image generation parameters a n The values ​​are displayed in descending order. Other elements can be viewed by scrolling the screen.

[0042] In this step, the virtual viewpoint image generation parameter a n I,I n The degree of similarity is calculated using the following formula (2).

[0043]

number

[0044] When the image coordinates are not specified, the virtual viewpoint image generation parameters a are calculated by the function f(θ, dist) as described in Patent Document 2 using the formula (2)-1. n Here, θ is the angle between the three-dimensional vector from the virtual viewpoint toward the pixel (x, y) and the three-dimensional vector from the n-th image capturing device 100 toward the subject 5.

[0045] Moreover, dist is the distance to the image edge of the image coordinates on each image in the image group capturing a point on the three-dimensional model projected onto pixel x, y of the virtual viewpoint image. The function f is designed so that the smaller θ is, that is, the smaller the angle between the virtual viewpoint and the capturing device is, the larger the value becomes. The function f is also designed so that the smaller the distance dist is, the smaller the value becomes. Note that the design of the function f is not limited to the above. ΔE is calculated by multiplying I,In is a function that represents the color difference between the two colors, and is calculated as the Euclidean distance in the RGB space using equation (3).

[0046]

number

[0047] In addition, the design of ΔE is not limited to this, and instead, for example, a color difference calculation formula such as delta E76 may be used. If ΔE is smaller than the threshold thΔE, the virtual viewpoint image generation parameter a is determined to be an outlier having a color similar to that of the noise region using formula (2)-2. n If not, the value of the virtual viewpoint image generation parameter a is reduced (by multiplying it by 0.1). If not, the virtual viewpoint image generation parameter a is reduced (by multiplying it by 0.1) by using equation (2)-3, and the virtual viewpoint image generation parameter a is reduced (by multiplying it by 0.1). n Returns the value of .

[0048] Since the image coordinate 22 is designated as a noise region to be excluded, the virtual viewpoint image generation parameter a calculated in advance in step S02020 is calculated using the formula (2)-2. n The above process is performed for all image coordinates 22, and the virtual viewpoint image generation parameter a n Calculate and update.

[0049] In this way, when the similarity between the pixel value of a specific region (noise region) of the virtual viewpoint image and the pixel value in the image group of each image capturing device 100 that contributes to the determination of the pixel value is equal to or greater than a threshold (the difference is equal to or less than a threshold), the virtual viewpoint image generation parameter a is adjusted using equation (2)-2 so that the degree of contribution is smaller than when this is not the case. n Alternatively, the virtual viewpoint image generation parameter a is calculated so that the degree of contribution decreases as the similarity increases. n may be calculated.

[0050] Also, instead of the method of determining inliers and outliers using color difference and a threshold value, a clustering method such as k-means clustering in a color space may be used to determine inliers and outliers. In addition, a robust estimation method such as M-estimation may be used to remove outliers and calculate weights, i.e., coefficients.

[0051] In step S02060, the virtual viewpoint image generating unit 250 generates the volumetric data and the virtual viewpoint image generating parameters a n and calculates the pixel value of the virtual viewpoint image according to the formula (1). Note that, in order to reduce unnecessary processing, the pixels to be calculated may be limited to the image coordinate 22, and the pixel value may be updated.

[0052] In step S03010, the virtual viewpoint image generation unit 250 outputs the virtual viewpoint image to the display device 300, and the display device 300 updates and displays the virtual viewpoint image 101 on the UI screen. The updated and displayed UI screen is shown in Fig. 6. The value of weight(a58) corresponding to the pixel value of noise in camera:058 is reduced, and the coefficient is smaller. As a result, the value of the average pixel value color_ave within the rectangular area of ​​the image coordinates 22 is improved, and the noise 11 is reduced.

[0053] The above steps may be executed multiple times repeatedly while the user is checking them. In addition, an Undo button 311 or a Redo button 312 may be provided on the UI screen, and a function may be implemented in which, when pressed, past cache data stored in the RAM 213 is read and the virtual viewpoint image 101 is returned to the previous image.

[0054] As described above, based on the pixel values ​​of noise in an area specified by the user, virtual viewpoint image generation parameters are calculated so that the coefficients of the elements that contributed to determining that pixel value are reduced, and the virtual viewpoint image is updated, thereby reducing the effects of noise.

[0055] Therefore, it is possible to reduce the complexity of searching for images that cause noise, and to generate a virtual viewpoint image by effectively using image data other than the noise area.

[0056] [Modification of the first embodiment] In the first embodiment, an example was described in which a noise region is specified, and the effect of noise is reduced by decreasing the coefficient of an element having a pixel value similar to the pixel value. In this modified example, an example is described in which an expected pixel value for the noise region is further specified, and the coefficient of an element having a pixel value similar to the pixel value is increased to control the pixel value to approach the expected pixel value. Descriptions of configurations similar to those in the first embodiment are omitted.

[0057] FIG. 7 shows a UI screen displayed on the display device 300 in step S02040 of FIG. 4 in the case of this modified example. The user selects the instruction type (inlier or outlier) with the check box 24, and then inputs the inlier region 23 having the expected pixel value and / or the outlier region 22 that should be removed as a noise region by clicking the mouse. The input instructions are displayed in a list. Unnecessary instructions can be deleted by clicking the discard icon 25. After inputting the instructions, pressing the "calc" button 310 starts the parameter calculation process. At this time, the virtual viewpoint image generation parameter a n is calculated for the image coordinates x, y of the outlier region 22 using equation (2), and then is updated using the following equation (4).

[0058]

number

[0059] Here, I(xinl, yinl) is a representative value of pixel values ​​of the virtual viewpoint image 101 at a plurality of image coordinates (xinl, yinl) in the inlier region 23. If it is close to this value, the virtual viewpoint image generation parameter a is calculated using equation (4)-3. n Increase the value of (by 1.5), and if it is far away, use equation (4)-2 to find nThe value of is decreased (made 0.1 times). Note that the representative value may be the pixel value at the center of the inlier region 23, or the average or median of the pixel values ​​of the inlier region 23.

[0060] In this way, when the similarity between the pixel values ​​in the image group of each image capturing device 100 that contribute to the determination of the pixel values ​​of a specific region (for example, the outlier region 22 that should be removed as a noise region) and the pixel values ​​of a region different from the specific region (for example, the inlier region 23 having the expected pixel value) is equal to or greater than a threshold (the difference is equal to or less than a threshold), the virtual viewpoint image generation parameter a is adjusted so that the degree of contribution is greater, using equation (4)-3, compared to when this is not the case. n Alternatively, the virtual viewpoint image generation parameter a is calculated so that the degree of contribution increases as the similarity increases. n may be calculated.

[0061] Then, when the similarity between the pixel values ​​in the image group of each image capturing device 100 that contribute to the determination of the pixel values ​​of a specific region (for example, the outlier region 22 that should be removed as a noise region) and the pixel values ​​of a region different from the specific region (for example, the inlier region 23 having an expected pixel value) is less than a threshold (the difference is greater than a threshold), the virtual viewpoint image generation parameter a is adjusted using equation (4)-2 so that the degree of contribution is smaller than when this is not the case. n Alternatively, the virtual viewpoint image generation parameter a is calculated so that the degree of contribution decreases as the similarity decreases. n may be calculated.

[0062] 8 shows a UI screen in which the virtual viewpoint image 101 is updated and displayed in this modified example. The coefficients of the images of the cameras 19, 32, and 28 are multiplied by 1.5, and the coefficient of the camera 58 is multiplied by 0.01 (=0.1×0.1) through the formulas (2) and (4). This brings the pixel values ​​of the noise region 22 closer to the pixel values ​​of the inlier region, thereby further reducing noise. However, these values ​​are not limited to these and can be determined based on experience or the like.

[0063] As described above, by specifying an inlier region in addition to an outlier region, a clearer distinction can be made between the cameras for which the virtual viewpoint image generation parameters are increased or decreased, making it easier to reduce the effects of noise.

[0064] The volumetric data is not limited to the above, and may be composed of the following elements, for example. That is, the volumetric data may be a colored three-dimensional point cloud indicating the three-dimensional shape and color of the subject 5, or imaging device information. Instead of the three-dimensional point cloud, the volumetric data may be composed of a three-dimensional model format having both a polygon mesh and a texture image. In addition, in order to perform image-based rendering, the image group captured by the multiple imaging devices 100 and the shape data of the subject may be composed of a three-dimensional point cloud format of coordinates only. For example, a virtual viewpoint image may be generated by image-based rendering based on the pixel values ​​of the image group of the imaging devices 100 and the shape data of the subject 5. In addition, as described in Patent Document 2, the shape data may be in the form of a depth image in which the distance to the subject 5 is stored in pixel units for each imaging device 100.

[0065] (Embodiment 2) In this embodiment, an example will be described in which a region similar to the region input by the user is searched in the peripheral direction and the input is interpolated to make the user operation easier. Descriptions of the same configuration as in the first embodiment will be omitted.

[0066] <System configuration and functional configuration of information processing device> Fig. 9 shows a system configuration diagram according to this embodiment. In this embodiment, the information processing device 200 further includes a coordinate update unit 260 in addition to the components in Fig. 1. The image coordinate data acquired by the image coordinate acquisition unit 220 is sent to the image coordinate update unit 260. The image coordinate update unit 260 searches for a similar area around the acquired image coordinates, updates the image coordinate data, and then transmits the updated image coordinate data to the parameter calculation unit 240.

[0067] <Processing> Next, the procedure of the process performed by the information processing device 200 according to this embodiment will be described with reference to the flowchart of Fig. 10. In the process of Fig. 10, step S02070 is added to the process of Fig. 4.

[0068] In step S02070, the image coordinate update unit 260 calculates the color difference for the eight neighboring pixels around the image coordinate (x, y) in the same manner as in formula (3), and if the color difference is less than the threshold, that is, if the pixels are similar, it is considered to be a noise region and repeatedly searches for further neighboring pixels. The search continues until there are no more pixels similar to the neighboring pixels. Alternatively, the search range may be set to 10px up, down, left, and right and search in the surrounding directions.

[0069] 11 shows a UI screen displayed on the display device 300 according to this embodiment. As a result of searching for adjacent pixels, the image coordinates 22 are updated to include the noise region 11. If the virtual viewpoint image 101 is video data having frames in time series, the image coordinates (x, y) of frames t-1 and t+1 before and after the specified frame t specified using the seek bar 26 or adjacent pixels may be further searched in a similar manner, and the search may continue until no similar pixels are found in the time series. In this way, a search may also be performed in the time direction.

[0070] 12 shows a UI screen displayed on display device 300 in step S03010 according to this embodiment. Region 22 designated by the user in step S02040 is updated to include noise region 11. Parameters are calculated for this updated region 22 in step S02050, and virtual viewpoint image 101 is updated in step S02060, reducing the influence of noise in the entire noise region 11. Although not shown, the influence of noise is also reduced in previous and subsequent frames.

[0071] As described above, the processing range is expanded in the image space or time direction by searching the periphery of the image coordinates specified by the user and interpolating the input, and similar noise regions can be updated collectively without the user having to specify the noise coordinates one by one. This makes it possible to simplify user operations.

[0072] (Embodiment 3) In the above-mentioned embodiment, the user needs to specify an area on the virtual viewpoint image. In contrast, in this embodiment, a learning model that learns the area specified by the user for the input of the virtual viewpoint image is constructed and used to automatically specify the noise area. The description of the same configuration as in the first embodiment is omitted.

[0073] <System configuration and functional configuration of information processing device> Fig. 13 shows a system configuration diagram according to this embodiment. The system according to this embodiment includes an image coordinate learning model storage unit 40 instead of the image coordinate designation device 20. The information processing device 200 according to this embodiment further includes a learning model acquisition unit 270 and an image coordinate detection unit 280 in addition to the components in Fig. 1.

[0074] The learning model acquisition unit 270 acquires image coordinate learning model data from the image coordinate learning model storage unit 40. The image coordinate detection unit 280 acquires virtual viewpoint image data from the virtual viewpoint image storage unit 10, and acquires image coordinate learning model data from the learning model acquisition unit 270. Then, the image coordinate detection unit 280 detects image coordinates by an inference process using the learning model, and then transmits the detected image coordinate data to the image coordinate acquisition unit 220.

[0075] <Processing> Next, the procedure of the process performed by the information processing device 200 according to this embodiment will be described with reference to the flowchart of Fig. 14. In the process of Fig. 14, steps S02080 and S02090 are added to the process of Fig. 4.

[0076] In step S02080, the learning model acquisition unit 270 acquires the image coordinate learning model from the image coordinate learning model storage unit 40. The image coordinate learning model is implemented, for example, using an open source machine learning software library, and is a neural network that has been learned using the area designated by the user in the above-mentioned embodiment as teacher data. In generating the image coordinate learning model, a learning model generation unit that performs learning processing using the area designated by the user in the first embodiment and virtual viewpoint image data as input may be added separately. In this embodiment, the image coordinate learning model is acquired as a data file from an external recording device and stored in the image coordinate learning model storage unit 40.

[0077] In step S02090, the image coordinate detection unit 280 acquires the virtual viewpoint image acquired in step S2030 as an input, and outputs an inference result image using the image coordinate learning model. The inference result image is, for example, a 1-channel 8-bit grayscale image of the same image size as the virtual viewpoint image, and the value of each pixel represents the probability of being a noise region in gradations of 0 to 255.

[0078] Here, Fig. 15(a) is an input virtual viewpoint image, and Fig. 15(b) is an inference result image for that input. For the sake of explanation, the contour of the subject 5 is shown with a dashed line, but in reality the pixel values ​​of the contour are 0 and displayed black. The periphery of the noise region 12 under the chin of the subject 5 is inferred to have a high probability, so the pixel values ​​are high and displayed white. The pixel values ​​around the contour of the noise region 12 are inclined so that the probability decreases as it approaches the outside.

[0079] In the next step S02040, the image coordinate acquisition unit 220 acquires the inference result image as image coordinate data. In step S02050, parameters are calculated based on the pixel values ​​of the acquired inference result image. That is, parameters are calculated for pixels (x, y) in the inference result image whose pixel values ​​are not 0 based on the following formula (5).

[0080]

number

[0081] Here I gray is the pixel value of the inference result image, and by dividing this by 255 and subtracting the normalized value from 1, the higher the probability of it being a noise region, the smaller the virtual viewpoint image generation parameter a n Equation (5) is designed so that the degree of update is large.

[0082] Note that the method of using the pixel values ​​of the inference result image is not limited to this. For example, instead, coordinates where the pixel values ​​of the inference result image are greater than a predetermined value may be extracted and used as image coordinate data, and parameters may be calculated in a manner similar to that of embodiment 1.

[0083] As described above, by constructing and using a learning model that learns the area designated by the user in response to an input of a virtual viewpoint image, it is possible to automatically designate a noise area.

[0084] [Other variations] In the above embodiment, an example of rendering a virtual viewpoint image by a blending technique has been described, but the present invention is not limited to this. For example, instead of this, a virtual viewpoint image may be rendered by model-based rendering. For example, a virtual viewpoint image may be generated by coloring shape data acquired from the volumetric data storage unit 30 as intermediate data and rendering the colored data to a virtual viewpoint.

[0085] In this case, when coloring the shape data, it is necessary to generate a colored three-dimensional point cloud or texture image by synthesizing pixel values ​​of the image group captured by the image capture device 100. In generating the color of the three-dimensional point cloud or the texture image, the influence of noise can be reduced by the same noise region designation method and parameter calculation method as in the first embodiment. In addition, when the shape data is in a format that cannot be used as is for generating a texture image, such as a depth image format, a three-dimensional model generation unit that converts the shape data into a polygon mesh may be provided separately. The three-dimensional model generation unit generates a three-dimensional model of the subject 5 based on the pixel values ​​of the image group of the image capture device 100, the virtual viewpoint image generation parameters, and the shape data of the subject 5.

[0086] The disclosure of this specification includes the following information processing device, a control method for an information processing device, and a program.

[0087] (Item 1) An information processing device that generates a virtual viewpoint image based on a group of images obtained by photographing a subject from a plurality of directions, A virtual viewpoint image acquisition means for acquiring a first virtual viewpoint image; an image coordinate acquisition means for acquiring at least one image coordinate of a specific area on the first virtual viewpoint image; a calculation means for calculating a parameter for generating the virtual viewpoint image based on a pixel value of the at least one image coordinate; a generating means for generating a second virtual viewpoint image by using the calculated parameters; An information processing device comprising:

[0088] (Item 2) An image coordinate updating unit searches for a pixel having a pixel value similar to the pixel value of the at least one image coordinate and updates the image coordinate, 2. The information processing device according to item 1, wherein the calculation means calculates the parameters based on pixel values ​​of the image coordinates updated by the image coordinate update means.

[0089] (Item 3) 3. The information processing device according to item 2, wherein the image coordinate update means searches for pixels in a peripheral direction of the pixel of the at least one image coordinate.

[0090] (Item 4) 4. The information processing device according to item 2 or 3, wherein the image coordinate update means searches for pixels in frames before and after the first virtual viewpoint image.

[0091] (Item 5) A learning model acquisition means for acquiring a learning model that outputs image coordinates of the specific area in response to an input of the first virtual viewpoint image; and an image coordinate detection means for outputting image coordinates of the specific area based on the learning model, 2. The information processing device according to item 1, wherein the image coordinate acquisition means acquires the image coordinates output by the image coordinate detection means.

[0092] (Item 6) A learning model generating means for generating a learning model that outputs image coordinates of the specific area in response to an input of the first virtual viewpoint image, 6. The information processing device according to item 5, wherein the learning model acquisition means acquires the learning model generated by the learning model generation means.

[0093] (Item 7) The information processing device described in any one of items 1 to 6, characterized in that the calculation means calculates the parameter so that when a similarity between a pixel value of the specific region and a pixel value in the image group that contributes to the determination of the pixel value is equal to or greater than a threshold, the degree of contribution is smaller than when the similarity is less than the threshold.

[0094] (Item 8) The information processing device described in any one of items 1 to 7, characterized in that the calculation means calculates the parameter so that when a similarity between a pixel value in the image group that contributes to determining the pixel value of the specific region and a pixel value of a region other than the specific region is equal to or greater than a threshold, the degree of contribution is greater than when the similarity is less than the threshold.

[0095] (Item 9) The information processing device according to any one of items 1 to 8, wherein the generating means generates the second virtual viewpoint image by image-based rendering based on pixel values ​​of the image group and shape data of the subject.

[0096] (Item 10) a three-dimensional model generating unit that generates a three-dimensional model of the subject based on pixel values ​​of the image group, the parameters, and shape data of the subject; 9. The information processing device according to any one of items 1 to 8, wherein the generating means generates the second virtual viewpoint image by model-based rendering using the three-dimensional model.

[0097] (Item 11) 11. The information processing device according to any one of items 1 to 10, characterized in that the at least one image coordinate is acquired based on an input from a pointing device that accepts an instruction to specify an image coordinate on the first virtual viewpoint image.

[0098] (Item 12) 12. The information processing device according to item 11, wherein the image coordinates on the first virtual viewpoint image are specified as a rectangular area.

[0099] (Item 13) 13. The information processing device according to any one of items 1 to 12, wherein the specific region is a noise region that appears in the first virtual viewpoint image.

[0100] (Item 14) A method for controlling an information processing device that generates a virtual viewpoint image based on a group of images obtained by photographing a subject from a plurality of directions, comprising: acquiring a first virtual viewpoint image; acquiring at least one image coordinate of a specific area on the first virtual viewpoint image; calculating a parameter for generating the virtual viewpoint image based on a pixel value of the at least one image coordinate; generating a second virtual viewpoint image using the calculated parameters; 13. A method for controlling an information processing apparatus comprising the steps of:

[0101] (Item 15) Item 15. A program for causing a computer to execute the control method for an information processing device according to item 14.

[0102] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0103] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0104] 5: subject, 6: virtual camera, 10: virtual viewpoint image storage unit, 20: image coordinate designation device, 30: volumetric data storage unit, 40: image coordinate learning model storage unit, 100: imaging device, 112: volumetric data generation device, 200: information processing device, 210: virtual viewpoint image acquisition unit, 220: image coordinate acquisition unit, 230: volumetric data acquisition unit, 240: parameter calculation unit, 250: virtual viewpoint image generation unit, 260: image coordinate update unit, 270: learning model acquisition unit, 280: image coordinate detection unit, 300: display device

Claims

1. An information processing device that generates a virtual viewpoint image based on a group of images obtained by photographing a subject from a plurality of directions, a virtual viewpoint image acquisition means for acquiring a first virtual viewpoint image; an image coordinate acquisition means for acquiring at least one image coordinate of a specific area on the first virtual viewpoint image; a calculation means for calculating a parameter for generating the virtual viewpoint image based on a pixel value of the at least one image coordinate; a generating means for generating a second virtual viewpoint image by using the calculated parameters; An information processing device comprising:

2. An image coordinate updating unit searches for a pixel having a pixel value similar to the pixel value of the at least one image coordinate and updates the image coordinate, 2. The information processing apparatus according to claim 1, wherein said calculation means calculates said parameters based on pixel values ​​of the image coordinates updated by said image coordinate update means.

3. 3. The information processing apparatus according to claim 2, wherein said image coordinate updating means searches for pixels in a peripheral direction of the pixel of said at least one image coordinate.

4. 3. The information processing apparatus according to claim 2, wherein said image coordinate updating means searches for pixels in frames before and after said first virtual viewpoint image.

5. a learning model acquisition means for acquiring a learning model that outputs image coordinates of the specific area in response to an input of the first virtual viewpoint image; and an image coordinate detection means for outputting image coordinates of the specific area based on the learning model, 2. The information processing apparatus according to claim 1, wherein the image coordinate acquisition means acquires the image coordinates output by the image coordinate detection means.

6. A learning model generating means for generating a learning model that outputs image coordinates of the specific area in response to an input of the first virtual viewpoint image, 6. The information processing apparatus according to claim 5, wherein the learning model acquisition means acquires the learning model generated by the learning model generation means.

7. The information processing device according to claim 1, characterized in that the calculation means calculates the parameters so that when a similarity between a pixel value of the specific region and a pixel value in the image group that contributes to determining the pixel value is equal to or greater than a threshold, the degree of contribution is smaller than when the similarity is less than the threshold.

8. The information processing device according to claim 1, characterized in that the calculation means calculates the parameter so that when a similarity between a pixel value in the image group that contributes to determining the pixel value of the specific region and a pixel value of a region other than the specific region is equal to or greater than a threshold, the degree of contribution is greater than when the similarity is less than the threshold.

9. 2 . The information processing apparatus according to claim 1 , wherein the generating means generates the second virtual viewpoint image by image-based rendering based on pixel values ​​of the image group and shape data of the subject.

10. a three-dimensional model generating unit that generates a three-dimensional model of the subject based on pixel values ​​of the image group, the parameters, and shape data of the subject; 2 . The information processing apparatus according to claim 1 , wherein the generating means generates the second virtual viewpoint image by model-based rendering using the three-dimensional model.

11. 2 . The information processing apparatus according to claim 1 , wherein the at least one image coordinate is acquired based on an input from a pointing device that accepts an instruction to specify an image coordinate on the first virtual viewpoint image.

12. 12. The information processing apparatus according to claim 11, wherein the image coordinates on the first virtual viewpoint image are specified as a rectangular area.

13. The information processing apparatus according to claim 1 , wherein the specific region is a noise region that appears in the first virtual viewpoint image.

14. A method for controlling an information processing device that generates a virtual viewpoint image based on a group of images obtained by photographing a subject from a plurality of directions, comprising: acquiring a first virtual viewpoint image; acquiring at least one image coordinate of a specific area on the first virtual viewpoint image; calculating a parameter for generating the virtual viewpoint image based on a pixel value of the at least one image coordinate; generating a second virtual viewpoint image using the calculated parameters; 13. A method for controlling an information processing apparatus comprising the steps of:

15. A program for causing a computer to execute the method for controlling an information processing device according to claim 14.

Images