Image processing apparatus, image processing method, and program

Abstract

Generating a virtual reality (VR) image capable of generating a direction image in which an image corresponding to an object present at a position far from a base point can be expressed with high definition SOLUTION: An image processing apparatus 100 according to the present disclosure acquires an object model indicating a three-dimensional shape of an object existing in an imaging region, which is generated based on a plurality of captured images obtained by imaging from a plurality of positions, determines a position of a virtual object in a virtual space, acquires a texture of the virtual object, sets a position of a base point in the virtual space, and generates a VR image corresponding to the base point based on the position of the base point, the object model, the virtual object, and the texture of the virtual object.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an image processing technique for generating a virtual reality (VR) image based on a plurality of captured images obtained by capturing images from a plurality of positions. [Background technology]

[0002] There is a technology for generating an image (hereinafter referred to as a "virtual viewpoint image") corresponding to a view from an arbitrary virtual viewpoint (hereinafter referred to as a "virtual viewpoint") using a plurality of captured images obtained by synchronized imaging using a plurality of imaging devices arranged at different positions. Hereinafter, the above-mentioned plurality of captured images will be referred to as a "multi-viewpoint image." There is also a technology for generating or acquiring an image corresponding to a view in an arbitrary direction from a predetermined position (hereinafter referred to as a "base point") by cutting out a partial image area from a panoramic image or a celestial sphere image corresponding to a view of the surroundings within a range of up to 360 degrees from the base point. Hereinafter, the image will be referred to as a "directional image."

[0003] Patent Document 1 discloses a technique for generating an image such as a panoramic image or a spherical image corresponding to a base point by synthesizing a plurality of virtual viewpoint images. Specifically, the technique disclosed in Patent Document 1 uses multi-viewpoint images to generate virtual viewpoint images corresponding to each of a plurality of virtual viewpoints set at the same position but having different line-of-sight directions, and synthesizes the generated virtual viewpoint images. Patent Document 1 also discloses a technique for delivering an image corresponding to a base point generated by synthesizing the virtual viewpoint images to a user terminal, and generating a directional image corresponding to a direction specified by the user at the user terminal using the image corresponding to the base point. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-68513 Summary of the Invention [Problem to be solved by the invention]

[0005] In the directional image generated by the technology disclosed in Patent Document 1, the image resolution of an object located farther from the base point decreases, which can make it difficult to visually recognize the object with high accuracy when the directional image is displayed. For example, if the subject of the image capture is a game at a baseball stadium and the base point is located near the catcher, the resolution of the image of the scoreboard located above the back screen will be low in the directional image corresponding to the direction from the base point toward the pitcher.

[0006] The present disclosure aims to provide a technology for generating VR images that can generate directional images that can represent, with high definition, images corresponding to objects that are located far from a base point. [Means for solving the problem]

[0007] The image processing device according to the present disclosure includes a model acquisition means for acquiring an object model indicating the three-dimensional shape of an object present in an imaging area, the object model being generated based on a plurality of captured images obtained by capturing images from a plurality of positions; a determination means for determining the position of a virtual object in a virtual space; a texture acquisition means for acquiring the texture of the virtual object; a setting means for setting the position of a base point in the virtual space; and an image generation means for generating a VR image corresponding to the base point based on the position of the base point, the object model, the virtual object, and the texture of the virtual object. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to generate a VR image capable of generating a directional image capable of expressing an image corresponding to an object located far from the base point with high definition. [Brief explanation of the drawings]

[0009] [Figure 1]FIG. 1 is a diagram illustrating an example of the configuration of an image processing system according to a first embodiment. [Figure 2] 1 is a block diagram showing an example of a hardware configuration of an image processing device according to a first embodiment. [Figure 3] 1 is a block diagram showing an example of a functional configuration of an image processing device according to a first embodiment. [Figure 4] 4A to 4C are diagrams illustrating an example of a virtual object generation process in a virtual object generation unit according to the first embodiment. [Figure 5] FIG. 2 is a diagram showing an example of a VR image generated by an image generating unit according to the first embodiment. [Figure 6] 4 is a flowchart showing an example of a processing flow in the image processing device according to the first embodiment. [Figure 7] 10 is a flowchart showing an example of the flow of a virtual object generation process in a virtual object generation unit according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Preferred embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that the following embodiments do not limit the technology of the present disclosure, and not all combinations of features described in the embodiments are necessarily essential to the solution according to the present disclosure. Note that in the following embodiments, a virtual viewpoint image is an image generated based on the position and orientation of a virtual imaging device (hereinafter referred to as a "virtual camera") defined by at least the position of a virtual viewpoint and the direction of the line of sight at the virtual viewpoint. A virtual viewpoint image is also called a free viewpoint image or an arbitrary viewpoint image.

[0011] VR images are images that can be displayed in a VR format, as described below. VR images include omnidirectional images (spherical images) and panoramic images with a wider image range (effective image range) than the display range that can be displayed at one time on a display unit. VR images are not limited to still images, but also include videos. VR images have an image range (effective image range) of up to 360 degrees horizontally and vertically. VR images also include images with a wider field of view than can be captured by a typical imaging device, or a wider image range than the display range that can be displayed at one time on a display unit, even if the field of view is less than 360 degrees horizontally or vertically. VR images can be displayed in a VR format, for example, by setting the display mode of a display device (a display device capable of displaying VR images) to "VR view." A portion of a VR image with a 360-degree field of view is displayed, and the user can move the displayed range by changing the orientation of the display device left and right (horizontal rotation direction), allowing them to view a seamless omnidirectional image in the horizontal direction.

[0012] Here, VR display (VR view) is a display method (display mode) that displays VR images with a field of view that corresponds to the orientation of the display device, and allows for a change in the display range. VR display includes "single-eye VR display (single-eye VR view)," which displays a single image by mapping a VR image onto a virtual sphere (deformation that corrects distortion). VR display also includes "two-eye VR display (two-eye VR view)," which displays a VR image for the left eye and a VR image for the right eye side by side in left and right regions by mapping each onto a virtual sphere. Stereoscopic viewing is possible by performing "two-eye VR display" using a VR image for the left eye and a VR image for the right eye that have parallax from each other.

[0013] Regardless of the VR display, for example, when a user wears a display device such as an HMD (head-mounted display), an image with a field of view corresponding to the orientation of the user's face is displayed. For example, suppose that a VR image is displayed with a field of view centered at 0 degrees left and right (a specific direction, for example, north) and 90 degrees up and down (90 degrees from the zenith, i.e., horizontal) at a certain point in time. If the orientation of the display device is flipped from this state (for example, the display surface is changed from facing south to facing north), the display range changes to an image of the same VR image with a field of view centered at 180 degrees left and right (the opposite direction, for example, south) and 90 degrees up and down. In other words, when a user wears an HMD and turns their face from north to south (i.e., facing backwards), the image displayed on the HMD also changes from a north image to a south image.

[0014] [Embodiment 1] In this embodiment, a virtual object that imitates a real object is placed near a base point in a virtual three-dimensional space (hereinafter simply referred to as "virtual space") using multi-viewpoint images, and a VR image corresponding to the base point is generated. Here, a real object is a physically existing object. Also, a virtual object (hereinafter referred to as a "virtual object") is an object model corresponding to a virtual object that does not physically exist and is generated using CG (computer graphics) technology or the like.

[0015] <Image processing system configuration> Fig. 1 is a diagram illustrating an example of the configuration of an image processing system 1 according to the first embodiment. Specifically, Fig. 1(a) is a block diagram illustrating an example of the configuration of the image processing system 1. The image processing system 1 includes an image processing device 100, a plurality of sensor systems 11-1 to 11-n (n is an integer equal to or greater than 2), an input device 12, a display device 13, a distribution server 14, and one or more user terminals 15. Hereinafter, when there is no need to particularly distinguish between the sensor systems 11-1 to 11-n, the sensor systems 11-1 to 11-n will be described without distinction and simply referred to as sensor system 11.

[0016] Each sensor system 11 includes one or more imaging devices. The imaging devices are configured with digital still cameras, digital video cameras, or the like. Captured images (multi-view images) obtained by imaging using the imaging devices included in each sensor system 11 are transmitted to the image processing device 100, which acquires them. Specifically, the imaging devices included in the multiple sensor systems 11 are disposed at different positions and perform synchronized imaging. The multiple images constituting the multi-view image may be the captured images themselves obtained by imaging using the imaging devices, or may be images obtained by performing image processing such as extracting a predetermined area from the captured images. The multi-view image obtained by synchronized imaging using the imaging devices included in the multiple sensor systems 11 is transmitted to the image processing device 100 along with imaging parameters of each imaging device included in the multiple sensor systems 11.

[0017] The imaging parameters include data such as external parameters, internal parameters, and the image size of the captured image obtained by imaging. External parameters are parameters that indicate the position and orientation of the imaging device, including parameters expressed by a rotation matrix and a position vector. Internal parameters are parameters that indicate information specific to the imaging device, including parameters that indicate the focal length, the center position of the captured image, and distortion of the optical system such as a lens. The image size is expressed by the number of pixels in the horizontal and vertical directions of the captured image. FIG. 1(b) is a diagram showing an example of the arrangement of multiple sensor systems 11. The multiple sensor systems 11 are arranged to surround an imaging area 120. In the following description, the imaging area 120 is assumed to be a baseball field where baseball games are played, and multiple sensor systems 11, such as 100 units, are arranged to surround the baseball field. It goes without saying that the imaging area 120 is not limited to a baseball field, and may be, for example, an indoor court where a basketball game is played.

[0018] The image processing device 100 is configured with a personal computer, a server device, or the like, and generates VR images such as omnidirectional images (spherical images) or panoramic images using acquired multi-viewpoint images. Specifically, the image processing device 100 first generates an object model indicating the three-dimensional shape of a real object present in the imaging area 120 using the acquired multi-viewpoint images. Hereinafter, the object model generated by the image processing device 100 is assumed to include information indicating the shape of the object and information indicating the color of the shape. The object model corresponding to the real object generated by the image processing device 100 is associated with the time code used when capturing the multi-viewpoint images and stored in a database not shown in FIG. 1.

[0019] The time code is information that can uniquely identify the time when an imaging device captured an image (or a frame if the captured image is a moving image), and is information expressed in a format such as "day:hour:minute:second.frame number." Next, the image processing device 100 places a virtual object that does not exist in the imaging area 120 in a virtual space corresponding to the imaging area 120. Next, the image processing device 100 generates a VR image such as a panoramic image by combining (also referred to as "rendering") the virtual object with an object model corresponding to the generated real object. The generated VR image is output to a distribution server 14 and distributed to a user terminal 15 via the distribution server 14.

[0020] More specifically, the image processing device 100 first acquires multi-view images and imaging parameters of each imaging device. Next, the image processing device 100 generates shape information indicating the three-dimensional shape of a foreground object (hereinafter referred to as a "foreground object") and color information (also referred to as "texture information") of the three-dimensional shape based on the acquired multi-view images and imaging parameters. The shape information is also referred to as "geometry information." A foreground object is, for example, a moving object such as a human or a ball that exists within the imaging area 120. The shape information of the foreground object is generated using, for example, a visual hull method. For example, when generating shape information using the visual hull method, the shape information is obtained as a three-dimensional point cloud, which is a set of points expressed by three-dimensional coordinates. The method of generating the shape information of the foreground object using the captured images is not limited to the visual hull method. Furthermore, the method of expressing the shape information of the foreground object is not limited to a three-dimensional point cloud, and the shape information may be expressed using a polygon mesh, voxels, or the like.

[0021] The color information of the foreground object is generated by the following method. For example, the image processing device 100 sets an arbitrary point in the three-dimensional point cloud as a focus point, and when a point in the imaging area 120 corresponding to the focus point is projected onto a captured image obtained by imaging with an imaging device capable of capturing images, the color value of the pixel at the projection destination is set as the color value of the focus point. The image processing device 100 generates color information of the foreground object by performing this process sequentially using all or some of the points included in the three-dimensional point cloud as focus points. Note that the method of generating color information of the foreground object is not limited to the method described above. The rendering of the foreground object is performed by projecting each point of the three-dimensional point cloud onto the imaging plane of the virtual camera.

[0022] Next, the image processing device 100 generates color information (texture information) of a background object (hereinafter referred to as a "background object") based on the multi-viewpoint image, imaging parameters, and shape information of the background object. When the imaging subject is a game at a baseball stadium, the background object refers to objects other than foreground objects such as the fence, back screen, scoreboard, signboard, stands where spectator seats are located, and the field that make up the baseball stadium. The shape information of the background object is information that indicates the three-dimensional shape of the background object.

[0023] For example, shape information of a background object is created in advance and stored in a storage device, and the image processing device 100 acquires the shape information by reading it from the storage device and generates color information corresponding to the acquired shape information of the background object. Note that the shape information of the background object may be acquired by the image processing device 100 generating the shape information of the background object based on multi-viewpoint images, imaging parameters, etc. In the following description, it is assumed that the shape information of the background object is data represented by a polygon mesh, and the color information is data represented by a texture image.

[0024] A texture image is an image used for UV mapping to the three-dimensional shape of a background object indicated by the shape information of the background object. The texture image is generated by the following method. For example, the image processing device 100 first cuts out, as a partial image, an image region including pixels corresponding to each vertex of a plurality of polygons constituting a polygon mesh corresponding to the background object when the vertices are projected onto each captured image. Specifically, the image processing device 100 cuts out, as a partial image, an image region including pixels corresponding to each vertex when the vertices of the plurality of polygons are projected onto a captured image obtained by capturing an image of at least a portion of the background object. Next, the image processing device 100 generates a texture image by stitching together the plurality of partial images cut out from each captured image. Note that the method of generating color information of a background object is not limited to the above-described method.

[0025] The rendering of the background object is performed by virtually capturing an image of a background object model corresponding to the background object, which is obtained by UV-mapping a texture image onto a polygon mesh corresponding to the background object, using a virtual camera. The image processing device 100 places a virtual object in a virtual space. A method for placing a virtual object will be described in detail later. The position at which the virtual object is placed is determined, for example, based on a user operation on the input device 12, which will be described later.

[0026] Next, the image processing device 100 generates a VR image by rendering the virtual object and an object model corresponding to the real object so that they fit within the angle of view of a virtual camera virtually placed at a base point. Specifically, the VR image generated by the image processing device 100 is an image obtained by virtually capturing an image of the entire periphery of a base point set in a virtual space using a virtual camera. The position of the base point is determined, for example, based on a user operation on the input device 12, which will be described later. Specifically, the format of the VR image may be any format required by the distribution server 14, and examples of such formats include equirectangular projection and cube map. The VR image generated by the image processing device 100 is output to the distribution server 14.

[0027] The input device 12 accepts an operation by a user (hereinafter referred to as an "operator") of the image processing device 100 and transmits a signal corresponding to the operation to the image processing device 100. Specifically, for example, the operator operates the input device 12 to input data to specify the position of the base point of the VR image and the position where the virtual object is to be placed, and the input device 12 transmits a signal corresponding to the input to the image processing device 100. The image processing device 100 determines the position of the base point of the VR image and the position where the virtual object is to be placed based on the signal received from the input device 12, and sets the base point of the VR image and places the virtual object. The input device 12 has an input unit such as a joystick, a touch panel, a keyboard, or a mouse, or an input unit including two or more of these. The operator operates the input unit to input the position of the base point of the VR image and the position where the virtual object is to be placed. The position of the base point and the position where the virtual object is to be placed are specified, for example, by three-dimensional coordinates in a virtual space.

[0028] Hereinafter, as an example, the input device 12 will be described as including a keyboard as an input unit, and the operator will use the keyboard to input three-dimensional coordinate values ​​of the position of the base point and the position where the virtual object is to be placed. Furthermore, the description will be made assuming that the values ​​of each component (x, y, z) of a three-dimensional coordinate system preset in the virtual space are input as the three-dimensional coordinate values. Note that the method for determining the position of the base point of the VR image and the position where the virtual object is to be placed in the image processing device 100 is not limited to a method based on input by the operator via the input device 12. For example, the image processing device 100 may make the determination by reading from a storage device a file or the like in which three-dimensional coordinate values ​​corresponding to the position of the base point of the VR image and the position where the virtual object is to be placed are preset and stored.

[0029] The display device 13 is configured with a liquid crystal display or the like, and displays the VR image generated by the image processing device 100, a GUI (graphical user interface) for operation, and the like, as information necessary for the operator to operate the device. The operator views the VR image displayed on the display device 13 and inputs the position of the base point or the position where a virtual object is to be placed, etc., via the input device 12. For example, if the imaging target is a game at a baseball stadium, the operator inputs, via the input device 12, the coordinates of a position in the virtual space corresponding to the position of the pitcher or a position nearby, or the position of the catcher or a position nearby, etc. The distribution server 14 is configured with a personal computer or a server device, and acquires the VR image output from the image processing device 100 and distributes the VR image to one or more user terminals 15.

[0030] The user terminal 15 is configured with a personal computer, smartphone, tablet terminal, HMD, or the like, and generates an image corresponding to an arbitrary line of sight direction from the VR image received from the distribution server 14. The user terminal 15 then displays the generated image on a display device possessed by the user terminal 15 or a display device connected to the user terminal 15. That is, the image displayed on the display device by the user terminal 15 is a partial image area of ​​the VR image. A user of the user terminal 15 (hereinafter simply referred to as "user") can specify an arbitrary position in the VR image using a direction from a base point and display an image area in the VR image corresponding to that position as that image on the display device. That is, each user terminal 15 can specify a different arbitrary direction and display a partial image area in the VR image on its respective display device.

[0031] When the user terminal 15 is a portable or wearable device such as a smartphone, a tablet terminal, or an HMD, the user terminal 15 may detect a change in the direction of the display surface of the display device using, for example, a gyro sensor or a geomagnetic sensor included in the user terminal 15. In this case, the user terminal 15 may change the direction from the base point, i.e., the image area of ​​the VR image to be displayed on the display device, according to the amount of change in the detected direction. Note that the direction from the base point may not be specified by the above-mentioned gyro sensor or the like, but may also be specified by the user operating an input device included in the user terminal 15 or an input device connected to the user terminal.

[0032] <Hardware configuration of image processing device> The hardware configuration of the image processing device 100 will be described with reference to FIG. 2. FIG. 2 is a block diagram showing an example of the hardware configuration of the image processing device 100 according to the first embodiment. The image processing device 100 includes, as its hardware configuration, a CPU 201, a RAM 202, a ROM 203, a communication unit 204, an input / output unit 205, and a GPU 206. The CPU 201 is a processor that uses the RAM 202 as a work memory, executes programs stored in the ROM 203, and performs overall control of each component of the image processing device 100. The CPU 201 executes various programs to realize the functions of each component of the image processing device 100, as shown in FIG. 3 (described later). The RAM 202 temporarily stores computer programs read from the ROM 203 and data such as intermediate calculation results. The ROM 203 holds computer programs and data that do not require modification. In the following description, the ROM 203 is assumed to hold shape information of a background object.

[0033] The communication unit 204 is a communication interface such as Ethernet (registered trademark) or USB, and is used for data communication with external devices. The communication unit 204 transmits VR images to the distribution server 14 via Ethernet, for example. The input / output unit 205 inputs and outputs data via an input interface and an output interface. The input / output unit 205 receives, for example, signals related to the position of the base point of the VR image and the position where a virtual object is to be placed from the input device 12. The input / output unit 205 also outputs, for example, signals related to the VR image and an operation GUI to the display device 13. The GPU 206 is an arithmetic device or processor specialized for image processing. The GPU 206 performs image processing such as generating a virtual viewpoint image or a VR image from multi-viewpoint images input from multiple sensor systems 11.

[0034] <Functional configuration of image processing device> 3 is a block diagram showing an example of the functional configuration of the image processing device 100 according to the first embodiment. The image processing device 100 has, as its functional configuration, an acquisition unit 300, a model generation unit 301, a virtual object generation unit 302, a base point setting unit 303, a virtual object position determination unit 304, a virtual object placement unit 305, an image generation unit 306, and an output unit 307. Each unit included in the functional configuration of the image processing device 100 is realized by the CPU 201 executing a program stored in the ROM 203 using the RAM 202 as a work memory. The acquisition unit 300 acquires multi-view images output from a plurality of sensor systems 11 and imaging parameters of the imaging devices included in each sensor system 11 via the communication unit 204. The multi-view images and imaging parameters acquired by the acquisition unit 300 are stored in the RAM 202 and used for processing by the model generation unit 301, the virtual object generation unit 302, the image generation unit 306, etc.

[0035] The model generation unit 301 generates an object model corresponding to a foreground object and an object model corresponding to a background object (background object model). Specifically, the model generation unit 301 generates the object model corresponding to the foreground object using the multi-view image and imaging parameters acquired by the acquisition unit 300. The method for generating the object model corresponding to the foreground object has been described above, so a description thereof will be omitted.

[0036] Furthermore, the model generation unit 301 generates a background object model using shape information of the background object stored in ROM 203 and the multi-viewpoint images and imaging parameters acquired by the acquisition unit 300. Specifically, first, the model generation unit 301 generates a texture image to be UV-mapped onto a polygon mesh corresponding to the background object using the shape information, multi-viewpoint images, and imaging parameters. Next, the model generation unit 301 UV-maps the generated texture image. A background object is generated by UV-mapping onto a polygon mesh corresponding to the background object. The method for generating a texture image to be UV-mapped onto a polygon mesh has been described above, so a description thereof will be omitted. The object model corresponding to the foreground object and the background object model corresponding to the background object generated by the model generation unit 301 are stored in RAM 202 and used for processing by the virtual object generation unit 302, the image generation unit 306, etc.

[0037] The base point setting unit 303 sets a base point of the VR image. Specifically, for example, the base point setting unit 303 determines the base point of the VR image at that position in the virtual space based on a signal related to the position of the base point of the VR image output from the input device 12 or information related to that position stored in advance in the ROM 203 or the like. Information related to the position of the base point of the VR image determined by the base point setting unit 303, such as three-dimensional coordinate values ​​of that position in the virtual space, is stored in the RAM 202 as a setting value related to the position of the base point.

[0038] The virtual object position determination unit 304 determines a position in the virtual space where a virtual object is to be placed. Specifically, for example, the virtual object position determination unit 304 determines a position where a virtual object is to be placed based on a signal related to the position where the virtual object is to be placed that is output from the input device 12, or information related to the position that is pre-stored in the ROM 203 or the like. The method of determining the position where a virtual object is to be placed is not limited thereto, and for example, the virtual object position determination unit 304 may determine a position where a virtual object is to be placed based on the position of a base point of the VR image that is set by the base point setting unit 303. Specifically, for example, the virtual object position determination unit 304 determines a position near the base point of the VR image as the position where the virtual object is to be placed.

[0039] Furthermore, for example, the position at which the virtual object is placed may be determined based on the position of the base point of the VR image set by the base point setting unit 303 and the position of the object model corresponding to the foreground object generated by the model generation unit 301. Specifically, for example, the virtual object position determination unit 304 determines the position at which the virtual object is placed based on the positional relationship between the position of the base point of the VR image and the position of the object model corresponding to the foreground object. More specifically, for example, the virtual object position determination unit 304 determines, as the position at which the virtual object is placed, a position at which the object model is not occluded by the virtual object when viewed from the base point of the VR image. Information regarding the position at which the virtual object is placed determined by the virtual object position determination unit 304, such as three-dimensional coordinate values ​​of the position in the virtual space, is stored in RAM 202.

[0040] The virtual object generation unit 302 generates a virtual object. Details of the virtual object generation process by the virtual object generation unit 302 will be described later. The virtual object generated by the virtual object generation unit 302 is stored in the RAM 202. The virtual object placement unit 305 places the virtual object at the position determined by the virtual object position determination unit 304.

[0041] The virtual object generation process will be described. For example, if the imaging target is a game at a baseball stadium, the virtual object is, for example, a panel-shaped virtual object that imitates a scoreboard or the like placed above a back screen. Shape information indicating the three-dimensional shape of the virtual object is, for example, prepared in advance and stored in the ROM 203 or the like. The virtual object generation unit 302 generates the virtual object by attaching a virtual viewpoint image obtained by virtually imaging a background object model or the like using a virtual camera as a texture image to a panel-shaped polygon mesh prepared in advance. Here, the above-mentioned virtual camera is a virtual camera different from the one used to generate the VR image. The above-mentioned virtual camera captures an object model corresponding to a real object of interest, such as a scoreboard, with high resolution. A more detailed method for generating the virtual object will be described later with reference to FIG. 4.

[0042] By placing the virtual object generated in this manner at a position near the base point of the VR image, the following effects can be expected. Specifically, the image of a real object that exists far away when viewed from the position corresponding to the base point in the imaging area 120, which conventionally appears small and low-resolution and difficult to view in a VR image, can be displayed large on the VR image. As a result, the image of the real object can be displayed clearly or in a manner that is visible to the user on the display device of the user terminal 15.

[0043] In this embodiment, as described above, the virtual object generation unit 302 uses a virtual viewpoint image as a texture image to be applied to a panel-shaped polygon mesh, but the present invention is not limited to this. For example, the virtual object generation unit 302 may use, as a texture image, an image region including an image of a target object such as a scoreboard, cut out from a texture image that is UV-mapped to the shape of a real object such as a background object. Furthermore, for example, the virtual object generation unit 302 may use, as a texture image, an image region including an image of a target object such as a scoreboard cut out from captured images that constitute the multi-viewpoint image acquired by the acquisition unit 300.

[0044] Furthermore, in this embodiment, as described above, the virtual object is described as having a panel-like shape, but the shape of the virtual object is not limited to this. For example, the virtual object generation unit 302 may extract shape information indicating the three-dimensional shape of a background object of interest, such as a scoreboard located in the distance, from the shape information of the background object, and use the extracted shape information as the shape information of the virtual object. This allows the shape of the virtual object to be made identical or approximately identical to the shape of the background object of interest, such as the scoreboard.

[0045] The image generation unit 306 generates a VR image. Specifically, the image generation unit 306 generates a VR image corresponding to the appearance of object models corresponding to foreground objects and background objects and virtual objects from a base point in the virtual space. The method for generating a VR image has been described above, so a description thereof will be omitted. The VR image generated by the image generation unit 306 is stored in the RAM 202. The output unit 307 outputs the VR image generated by the image generation unit 306 to the distribution server 14 via the communication unit 204.

[0046] <Virtual object generation process> Referring to FIG. 4, the generation process of the virtual object will be described. FIG. 4 is a diagram for explaining an example of the generation process of the virtual object in the virtual object generation unit 302 according to Embodiment 1. FIG. 4 shows a schematic view when the imaging target, as an example, is a game in a baseball stadium and is viewed from a distance. In FIG. 4, the area indicated by the gray rectangle mimics the players during the game. In FIG. 4, the object 403 is a scoreboard arranged above a background screen that is a part of the baseball field 402, which is a real object of interest in this embodiment corresponding to the virtual object. That is, the virtual object has, for example, a panel-like shape mimicking the shape of the object 403.

[0047] The imaging device 401 shown in FIG. 4 represents a virtual imaging device, that is, it shows the position and orientation of a virtual camera in the virtual space and does not exist in the imaging area 120. The position 404 corresponds to the position of the reference point set in the virtual space by the reference point setting unit 303. The virtual object 405 shows the shape and position of the virtual object arranged in the virtual space by the virtual object arrangement unit 305 and does not exist in the imaging area 120. The imaging device 401, which is a virtual camera, virtually images the object model corresponding to the object 403 in the virtual space to generate a virtual viewpoint image corresponding to the image of the object 403. The virtual object generation unit 302 generates the virtual object 405 by pasting the generated virtual viewpoint image as a texture image onto the panel-like virtual object 405. The generated virtual object is arranged at a position in the virtual space corresponding to the vicinity of the position 404 where the catcher exists in the baseball field 402, which is set as the reference point.

[0048] <Example of VR image generation> 5 is a diagram showing an example of a VR image 501 generated by the image generation unit 306 according to the first embodiment. The VR image 501 includes an image 502 of an object 403 and an image 503 of a virtual object 405, and the VR image 501 is displayed on the display device 13. The image processing device 100 displays a base point position designation unit 504, which the operator uses to input three-dimensional coordinate values ​​of a base point, and a virtual object position designation unit 505, which the operator uses to input three-dimensional coordinate values ​​where a virtual object is to be placed, superimposed on the VR image 501. The operator inputs three-dimensional coordinate values ​​to the base point position designation unit 504 and the virtual object position designation unit 505 using the input device 12. The image processing device 100 receives a signal based on the input, sets a base point of the VR image, and determines a position where the virtual object is to be placed.

[0049] Because the background object model corresponding to object 403 exists at a position far away from base point position 404 in the virtual space, the resolution of image 502 of object 403 is low in the VR image. That is, when the user sets the line of sight from the base point on user terminal 15 in the direction in which the background object model corresponding to object 403 is located, the resolution of the image of object 403 is also low in the image displayed on the display device of user terminal 15. Therefore, it is difficult for the user to visually recognize the score of the game, player information, etc. included in the image.

[0050] On the other hand, a virtual object 405, to which a virtual viewpoint image obtained by virtually capturing an image of a background object model corresponding to object 403 with a virtual camera is attached as a texture image, is placed near base position 404. Therefore, in the VR image, an image 503 of virtual object 405 becomes larger and its resolution becomes higher. That is, the resolution of the image of virtual object 405 also becomes higher in the image displayed on the display device of user terminal 15, so that the user can visually recognize the score of the game, player information, etc. included in the image.

[0051] The VR image generated by the image processing device 100 is distributed to a plurality of user terminals 15 via a distribution server 14. The user of each user terminal 15 can specify an arbitrary line of sight direction to select an image area from the VR image corresponding to the specified line of sight direction and display it on the display device of the user terminal 15. Therefore, for example, when the imaging target is a game at a baseball stadium, the user of each user terminal 15 can use their own user terminal 15 to watch the baseball game and obtain information useful for watching the game that is presented in the object 403.

[0052] <Operation of image processing device> The operation of the image processing device 100 will be described with reference to Figs. 6 and 7. Fig. 6 is a flowchart showing an example of the flow of processing in the image processing device 100 according to the first embodiment. The processing of the flowchart shown in Fig. 6 is realized by the CPU 201 loading a control program stored in the ROM 203 into the RAM 202 and executing it. Note that the processing of the flowchart is repeatedly executed every time a multi-viewpoint image and imaging parameters are output from the multiple sensor systems 11. Note that hereinafter, each processing step (process) is represented by adding "S" to the beginning of the reference numeral. First, in S600, the acquisition unit 300 acquires the multi-viewpoint image and imaging parameters output from the multiple sensor systems 11. The acquired multi-viewpoint image and imaging parameters are stored in the RAM 202 and used for processing by the model generation unit 301, etc.

[0053] Next, in S601, the model generation unit 301 generates object models corresponding to real foreground objects and background objects based on the multi-viewpoint images. The generated object models are stored in the RAM 202 and used in processing by the virtual object generation unit 302, image generation unit 306, etc. Next, in S602, the virtual object generation unit 302 generates a virtual object based on the object model corresponding to the background object, etc. generated in S601. The generated virtual object is stored in the RAM 202 and used in processing by the virtual object placement unit 305, image generation unit 306, etc. Next, in S603, the base point setting unit 303 sets the position of the base point of the VR image based on a signal transmitted from the input device 12. The setting value for the position of the base point is, for example, a three-dimensional coordinate value in the virtual space indicating the position of the base point. The setting value indicating the set position of the base point of the VR image is stored in the RAM 202 and used in processing by the image generation unit 306, etc.

[0054] Next, in S604, the virtual object position determination unit 304 determines a position at which to place the virtual object generated in S602, based on a signal transmitted from the input device 12. The position at which to place the virtual object is determined, for example, as three-dimensional coordinates in the virtual space at which the virtual object is to be placed. The orientation at which the virtual object is to be placed is determined so that the normal to the surface to which the texture image is attached to the virtual object faces the direction of the base point. Information regarding the determined position at which the virtual object is to be placed is stored in RAM 202 and used for processing by the virtual object placement unit 305, etc. Next, in S605, the virtual object placement unit 305 places the virtual object generated in S602 at the position in the virtual space determined in S604.

[0055] Next, in S606, the image generation unit 306 generates a VR image including, as images, the object model generated in S601 and the virtual object generated in S602 and arranged in S605, based on the position of the base point set in S603. The generated VR image is stored in the RAM 202 and used for processing by the output unit 307, etc. Next, in S607, the output unit 307 outputs the VR image generated in S606 to the distribution server 14. After S607, the image processing device 100 ends the processing of the flowchart shown in FIG.

[0056] Although the above description has been given assuming that the processes of S600 to S605 are executed sequentially, this is not limiting. For example, as long as the process of S600 is executed before the processes of S601 and S602, and the process of S604 is executed before the process of S605, the order of the processes of S600 to S605 is arbitrary, and two or more processes of S600 to S605 may be executed in parallel.

[0057] <Virtual object generation process> The flow of the virtual object generation processing by the virtual object generation unit 302 in S602 will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of the flow of the virtual object generation processing by the virtual object generation unit 302 according to the first embodiment, and is a flowchart showing an example of the processing flow in S602. Hereinafter, as an example of the virtual object generation processing, a case will be described in which a virtual object is generated using a virtual viewpoint image obtained by virtually capturing an image of a background object model corresponding to the background object described with reference to Fig. 4 using a virtual camera. The processing of the flowchart shown in Fig. 7 is executed by the virtual object generation unit 302 after the processing of S601.

[0058] After S601, in S701, the virtual object generation unit 302 determines the position and orientation of a virtual camera capable of capturing an image of an object model corresponding to a real object of interest, such as object 403, based on a signal from the input device 12. In this embodiment, as described above, the operator uses the input device 12 to specify the position in the virtual space of the object model corresponding to the object of interest, but the method for determining the position and orientation of the virtual camera is not limited to this. For example, the virtual object generation unit 302 may determine the position and orientation of the virtual camera by reading from the ROM 203 a file or data that previously contains information such as three-dimensional coordinate values ​​related to the position.

[0059] Next, in S702, the virtual object generation unit 302 places a virtual camera at the position and orientation determined in S701 and virtually captures an object model corresponding to the object of interest with the virtual camera, thereby generating a virtual viewpoint image. Next, in S703, the virtual object generation unit 302 generates a virtual object by pasting the virtual viewpoint image generated in S702 onto a shape such as a panel represented by a polygon mesh of the virtual object. Here, the virtual object generation unit 302 may paste an image in which a predetermined image region in the virtual viewpoint image is masked onto the shape of the virtual object. The predetermined image region is, for example, an image region that includes information unrelated to the purpose of the image capture target, such as a baseball game, and that includes information that is not appropriate for placement near the base point.

[0060] The virtual object generation unit 302 may also paste an image obtained by clipping only a predetermined image region from the virtual viewpoint image onto the shape of the virtual object. The virtual object generation unit 302 may also paste an image obtained by stitching together two or more images clipped from two or more image regions from the virtual viewpoint image onto the shape of the virtual object. The virtual object generation unit 302 may also paste an image obtained by changing the image size or color of the virtual viewpoint image onto the shape of the virtual object. The virtual object generation unit 302 may also paste an image obtained by changing the transparency of the virtual viewpoint image onto the shape of the virtual object. By pasting the image onto the shape of the virtual object, even when the virtual object is placed, it becomes possible to view the image of a rear object that is blocked by the virtual object when viewed from the base point. After S703, the virtual object generation unit 302 ends the processing of the flowchart shown in FIG. 7, i.e., the processing of S602.

[0061] As described above, the image processing device 100 is configured to generate an object model corresponding to a real object using a multi-viewpoint image and generate a virtual object based on the generated object model or the multi-viewpoint image. Furthermore, the image processing device 100 is configured to place the generated virtual object near a base point and generate a VR image corresponding to the appearance from the base point in a state where the virtual object is placed. The image processing device 100 configured in this manner can generate a VR image that can express, in high definition, the image of an object that is difficult to see when cut out from a VR image, such as one that is located far from the base point. As a result, the user can clearly see the image of the object when displayed on the user terminal 15.

[0062] [Modification of the first embodiment] In the first embodiment, an example has been described in which an object model is generated in the image processing device 100, but the image processing device 100 may be configured to acquire an object model generated by an external device not shown in Fig. 1. Furthermore, in the first embodiment, an example has been described in which the image processing device 100 generates an image such as a virtual viewpoint image to be pasted as a texture image on a virtual object, but the present invention is not limited to this. For example, the image processing device 100 may be configured to acquire a texture image generated by an external device not shown in Fig. 1 and paste the texture image on a virtual object.

[0063] Furthermore, in the first embodiment, as an example, a mode has been described in which a texture image such as a virtual viewpoint image is pasted onto a virtual object in the image processing device 100, but the present invention is not limited to this. For example, the image processing device 100 may be configured to acquire a virtual object onto which a texture image has been pasted in an external device not shown in Fig. 1, and place the acquired virtual object at a determined position.

[0064] Furthermore, in the first embodiment, as an example, one texture image is pasted onto one virtual object. However, multiple texture images may be pasted onto one virtual object. Specifically, for example, multiple images obtained by cutting out multiple image regions from one virtual viewpoint image, multiple images cut out from one captured image, or a combination thereof may be pasted onto the shape of the virtual object while changing the pasting position. Furthermore, for example, multiple virtual viewpoint images obtained by multiple virtual cameras, multiple images cut out from multiple captured images constituting a multi-viewpoint image, or a combination thereof may be pasted onto the shape of the virtual object while changing the pasting position. With this configuration, when there are multiple objects of interest, it is possible to generate a VR image in which images of the multiple objects can be simultaneously displayed on the user terminal 15.

[0065] Furthermore, in the first embodiment, an aspect in which one virtual object is placed in a virtual space has been described as an example, but multiple virtual objects may be placed in a virtual space. In this case, different texture images or the same texture image may be applied to each virtual object. By applying different texture images to each virtual object, for example, when there are multiple objects of interest, each virtual object can be placed according to the positional relationship of the multiple objects. Furthermore, by applying different but identical texture images to each virtual object, for example, a VR image can be generated in which the image of the object of interest is always displayed even if the user changes the line of sight of the image displayed on the user terminal 15.

[0066] In addition, in the first embodiment, an example in which one VR image is generated for a base point has been described, but for example, multiple VR images with different resolutions or image sizes may be generated for one base point. With this configuration, the user can select a VR image to be distributed from the distribution server 14, taking into account the drawing processing capacity of the user terminal 15, the state of the communication line between the distribution server 14 and the user terminal 15, the amount of data when receiving the VR image, etc.

[0067] Furthermore, in the first embodiment, an example in which one base point is set has been described. However, multiple base points may be set, and one or more VR images may be generated for each of the multiple set base points. With this configuration, a user can select a VR image corresponding to a viewpoint position that the user wants to display on the display device of the user terminal 15 from multiple VR images corresponding to multiple base points. Specifically, when the imaging target is a game at a baseball stadium, for example, a user can display an image with the catcher's viewpoint on the display device of the user terminal 15 that the user uses. At the same time, another user can display an image with the pitcher's viewpoint on the display device of the user terminal 15 that the user uses.

[0068] Furthermore, in the first embodiment, if the position of the base point is set before generating the object model, the model generation unit 301 may generate an object model corresponding to the foreground object based on the set position of the base point. Specifically, for example, the model generation unit 301 may omit or simplify the generation of parts that are invisible when the object model is viewed from the position of the base point, so that a high-precision object model is generated only when the object model is viewed from the position of the set base point. With this configuration, the amount of calculation required to generate an object model corresponding to the foreground object can be reduced.

[0069] Furthermore, in the first embodiment, as an example, a mode has been described in which a VR image is generated in which an image obtained by capturing an image using a monocular imaging device is displayed on the display device of the user terminal 15. However, the present invention is not limited to this. Specifically, for example, the image processing device 100 may generate two VR images from which two images that can be viewed stereoscopically can be cut out on the display device of the user terminal 15. In this case, for example, the image processing device 100 sets two base points that generate an appropriate parallax, and generates VR images corresponding to each of the two set base points. Furthermore, so that an image that can be viewed stereoscopically can be cut out even if the user changes the line of sight of the image displayed on the user terminal 15, the image processing device 100 may generate multiple VR images corresponding to positions obtained by rotating the two base points around the midpoint of the two base points.

[0070] [Other embodiments] The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0071] It should be noted that within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0072] [Technical Features of the Disclosure] The present disclosure includes the following configurations, methods, and programs.

[0073] <Configuration 1> a model acquisition means for acquiring an object model representing a three-dimensional shape of an object present in an imaging area, the object model being generated based on a plurality of captured images obtained by imaging from a plurality of positions; a determining means for determining a position of a virtual object in a virtual space; a texture acquisition means for acquiring a texture of the virtual object; a setting means for setting the position of a base point in the virtual space; an image generation means for generating a VR image corresponding to the base point based on the position of the base point, the object model, the virtual object, and a texture of the virtual object; 1. An image processing device comprising:

[0074] <Configuration 2> the determining means determines a position of the virtual object in the virtual space based on the position of the base point; 2. The image processing device according to configuration 1,

[0075] <Configuration 3> the determining means determines a position near the position of the base point as the position of the virtual object in the virtual space; 3. The image processing device according to configuration 1 or 2, characterized in that:

[0076] <Configuration 4> the determining means determines a position of the virtual object in the virtual space based on the position of the base point and the position of the object model; 4. The image processing device according to any one of configurations 1 to 3, characterized in that:

[0077] <Configuration 5> the determining means determines, as the position of the virtual object in the virtual space, a position at which the object model is not occluded by the virtual object when viewed from the position of the base point in a state in which the virtual object is placed in the virtual space; 5. The image processing device according to configuration 4,

[0078] <Configuration 6> the determining means determines a position of the virtual object in the virtual space based on an input from a user via an input device; 6. The image processing device according to any one of configurations 1 to 5, characterized in that:

[0079] <Configuration 7> the setting means sets the position of the base point in the virtual space based on an input from a user via an input device; 7. The image processing device according to any one of configurations 1 to 6,

[0080] <Configuration 8> the texture acquisition means acquires the texture of the virtual object by generating it based on a virtual viewpoint image obtained by virtually capturing an image of at least a part of the object model with a virtual camera; 8. The image processing device according to any one of configurations 1 to 7, characterized in that:

[0081] <Configuration 9> the texture acquisition means acquires the texture of the virtual object by generating it based on an image obtained by cutting out an image region including at least a part of an image of the object from the plurality of captured images; 9. The image processing device according to any one of configurations 1 to 8,

[0082] <Configuration 10> The model acquisition means acquires the object model by generating it based on the plurality of captured images. 10. The image processing device according to any one of configurations 1 to 9, characterized in that:

[0083] <Method> a model acquisition step of acquiring an object model that indicates a three-dimensional shape of an object present in an imaging area, the object model being generated based on a plurality of captured images obtained by imaging from a plurality of positions; a determining step of determining a position of a virtual object in a virtual space; a texture acquisition step of acquiring a texture of the virtual object; a setting step of setting a position of a base point in the virtual space; an image generation step of generating a VR image corresponding to the base point based on the position of the base point, the object model, the virtual object, and a texture of the virtual object; An image processing method comprising:

[0084] <Program> 11. A program for causing a computer to function as the image processing device according to any one of configurations 1 to 10. [Explanation of symbols]

[0085] 100 Image processing device 301 Model Generation Unit 302 Virtual Object Generation Unit 303 Base point setting section 304 Virtual object position determination unit 305 Virtual Object Placement Unit 306 Image Generation Unit

Claims

1. a model acquisition means for acquiring an object model representing a three-dimensional shape of an object present in an imaging area, the object model being generated based on a plurality of captured images obtained by imaging from a plurality of positions; a determining means for determining a position of a virtual object in a virtual space; a texture acquisition means for acquiring a texture of the virtual object; a setting means for setting the position of a base point in the virtual space; an image generation means for generating a VR image corresponding to the base point based on the position of the base point, the object model, the virtual object, and a texture of the virtual object; 1. An image processing device comprising:

2. the determining means determines a position of the virtual object in the virtual space based on the position of the base point; 2. The image processing device according to claim 1, wherein:

3. the determining means determines a position near the position of the base point as the position of the virtual object in the virtual space; 2. The image processing device according to claim 1, wherein:

4. the determining means determines a position of the virtual object in the virtual space based on the position of the base point and the position of the object model; 2. The image processing device according to claim 1, wherein:

5. the determining means determines, as the position of the virtual object in the virtual space, a position at which the object model is not occluded by the virtual object when viewed from the position of the base point in a state in which the virtual object is placed in the virtual space; 5. The image processing device according to claim 4, wherein:

6. the determining means determines a position of the virtual object in the virtual space based on an input from a user via an input device; 2. The image processing device according to claim 1, wherein:

7. the setting means sets the position of the base point in the virtual space based on an input from a user via an input device; 2. The image processing device according to claim 1, wherein:

8. the texture acquisition means acquires the texture of the virtual object by generating it based on a virtual viewpoint image obtained by virtually capturing an image of at least a part of the object model with a virtual camera; 2. The image processing device according to claim 1, wherein:

9. the texture acquisition means acquires the texture of the virtual object by generating it based on an image obtained by cutting out an image region including at least a part of an image of the object from the plurality of captured images; 2. The image processing device according to claim 1, wherein:

10. The model acquisition means acquires the object model by generating it based on the plurality of captured images.

2. The image processing device according to claim 1, wherein:

11. a model acquisition step of acquiring an object model representing a three-dimensional shape of an object present in an imaging area, the object model being generated based on a plurality of captured images obtained by imaging from a plurality of positions; a determining step of determining a position of a virtual object in a virtual space; a texture acquisition step of acquiring a texture of the virtual object; a setting step of setting a position of a base point in the virtual space; an image generation step of generating a VR image corresponding to the base point based on the position of the base point, the object model, the virtual object, and a texture of the virtual object; An image processing method comprising:

12. A program for causing a computer to function as the image processing device according to any one of claims 1 to 10.

Images