Information processing device, information processing method, and program

Abstract

To allow for generating a virtual viewpoint image of a given scene corresponding to a modeled object.SOLUTION: An information processing device disclosed herein is configured to: acquire captured image of a generated object generated according to a scene corresponding to a specific time in a scene of a virtual viewpoint image; and generate a virtual viewpoint image of the scene corresponding to the specific time using the captured image of the generated object.SELECTED DRAWING: Figure 6

Description

Technical Field

[0004] , , , , , , , ,

[0005] , , , ,

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] There is a technique for calculating the position and orientation of a mobile terminal based on a captured image obtained by capturing a two-dimensional marker with a mobile terminal such as a smartphone, and generating a virtual viewpoint image according to the calculated position and orientation. This technique is used in various fields, for example, in techniques related to augmented reality (AR). Patent Document 1 discloses a technique for correcting the tilt of a virtual viewpoint image caused by camera shake or the like during imaging of a two-dimensional marker.

[0003] On the other hand, in recent years, using a 3D printer or the like, a figure of a person or the like is also generated based on a 3D model (shape data representing a three-dimensional shape) of an object obtained by imaging or scanning an actual person or the like.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] <QQ000028>Although it is possible to generate a figure of a person or the like based on a scene of a virtual viewpoint image, there is a desire to deepen knowledge about the generated figure, such as which scene of the original virtual viewpoint image the scene corresponding to the generated figure is. Also, it is sometimes necessary to generate a new virtual viewpoint image corresponding to another virtual viewpoint from a scene of a virtual viewpoint image. In this case as well, there is a desire to deepen knowledge about the generated virtual viewpoint image.

Means for Solving the Problems

[0006] The information processing device relating to this disclosure includes an acquisition means for acquiring information for identifying a specific scene corresponding to a three-dimensional object contained in a captured image, and an output means for outputting an instruction to output three-dimensional shape data corresponding to the specific scene, which is generated based on a plurality of captured images, based on the information. The information is a time code associated with the three-dimensional shape data. It is characterized by the following: [Effects of the Invention]

[0007] According to this disclosure, it is possible to generate a virtual viewpoint image of a predetermined scene corresponding to a modeled object. [Brief explanation of the drawing]

[0008] [Figure 1] Diagram showing an example of the configuration of an information processing system. [Figure 2] A diagram illustrating examples of information managed by a database. [Figure 3] Figures showing examples of the first and second output products. [Figure 4] Diagram showing an example configuration of an image generation device. [Figure 5] Diagram explaining a virtual camera [Figure 6] A flowchart showing the process for generating the second output. [Figure 7] Figures showing examples of the first and second output products. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. Note that the following embodiments are not intended to limit the scope of the claims of this disclosure, and not all combinations of features described in these embodiments are necessarily essential to the solutions of this disclosure. The same reference numerals are used for identical components, and their descriptions are omitted.

[0010] [Embodiment 1] This embodiment describes a method for generating a virtual viewpoint image corresponding to the same 3D model used to generate the D model, or for obtaining information related thereto, from captured images of the generated product, which is based on the same 3D model used to generate the virtual viewpoint image. A virtual viewpoint image is an image generated by an end user and / or a designated operator manipulating the position and orientation (direction) of a camera (virtual camera) corresponding to the virtual viewpoint, and is also called a free viewpoint image or arbitrary viewpoint image. The virtual viewpoint image may be a video or a still image, but this embodiment will describe the case of a video as an example. A 3D model is shape data that represents the three-dimensional shape of an object.

[0011] (System Configuration) Figure 1 shows an example configuration of an information processing system (virtual viewpoint image generation system) that generates a virtual viewpoint image or related information corresponding to a 3D model of a first output object (product) as a second output object, based on an image of the first output object (product). Figure 1(a) shows an example configuration of the information processing system 100, and Figure 1(b) shows an example of the installation of the sensor system of the information processing system. The information processing system (virtual viewpoint image generation system) 100 includes a database 103, an image generation device 104, a mobile terminal 105, and a molding device 106.

[0012] Database 103 manages event information and 3D model information, etc. Event information includes data indicating the storage location of 3D model information for each object, associated with all time codes of the events being captured. Objects may include people and things that the user wants to model, or they may include people and things that are not the target of modeling. 3D model information includes information about the 3D model of the object.

[0013] This information may be, for example, information acquired by a sensor system of the information processing system 100, or information acquired by a sensor system of a system different from the information processing system 100. The sensor system includes, for example, sensor systems 101a-101n, each having at least one imaging device, which is a camera, as shown in Figure 1(b). In the following, unless otherwise specified, the n sensor systems from sensor system 101a to sensor system 101n will not be distinguished and will be referred to as the multiple sensor system 101. The multiple sensor system 101 is installed so as to surround the region 120, which is the area to be imaged, and the cameras of the multiple sensor system 101 each image the region 120 from different directions. The virtual camera 110 images the region 120 from a different direction than the cameras of the multiple sensor system 101. Details of the virtual camera 110 will be described later.

[0014] If the imaging target is a professional sports match such as rugby or soccer, the region 120 will be the stadium field, and the n (e.g., 100) multi-sensor systems 101 will be installed around the field. In addition, the imaging target region 120 may include not only people on the field but also balls and other objects. The imaging target is not limited to a stadium field; it may also be a music concert held in an arena or a commercial shoot held in a studio, as long as the multi-sensor systems 101 can be installed. There is no limit to the number of sensor systems 101 installed. Furthermore, the multi-sensor systems 101 do not need to be installed around the entire perimeter of region 120; depending on the installation location constraints, they may be installed only around a part of region 120. In addition, the multiple cameras in the multi-sensor system 101 may include imaging devices with different functions, such as telephoto cameras and wide-angle cameras.

[0015] The multiple cameras of the multiple sensor system 101 perform image capture in synchronous motion, acquiring multiple images. Each of these images may be an captured image, or it may be an image obtained by performing image processing on the captured image, such as extracting a predetermined region.

[0016] Each of the sensor systems 101a - 101n may have a microphone (not shown) in addition to the camera. The microphones of the plurality of sensor systems 101 synchronously pick up sound. Based on the picked-up sound, an acoustic signal can be generated to be reproduced together with the display of the image in the image generation device 104. Hereinafter, for the sake of simplifying the description, the description of the sound will be omitted, but basically the image and the sound are both processed.

[0017] The plurality of images acquired by the plurality of sensor systems 101 and the time code used for imaging are combined and stored in the database 103. The time code is time information represented by an absolute value for uniquely identifying the imaging time, and is, for example, time information that can be specified in a format such as day:hour:minute:second.frame number.

[0018] An example of a table of event information and 3D model information managed by the database 103 will be described with reference to the drawings. FIG. 2 is a diagram showing an example of a table of information managed by the database 103. FIG. 2(a) shows a table of event information, and FIG. 2(b) shows a table of 3D model information. As shown in FIG. 2(a), the table 210 of event information managed by the database 103 indicates the storage destination of 3D model information for each object for all time codes of the event to be imaged. In the table 210 of event information, for example, it shows that the storage destination of the 3D model information with the time code of "16:14:24.041" and the object being object A is "DataA100". Note that the table 210 of event information is not limited to all time codes, and may indicate the storage destination of 3D model information for some time codes. The time code is an absolute value for uniquely identifying the imaging time, and can be specified, for example, in a format such as "day:hour:minute:second.frame number".

[0019] As shown in Fig. 2(b), the table 220 of 3D model information managed by the database 103 stores data for each item of "3D model", "texture", "average coordinates", "center of gravity coordinates", and "maximum / minimum coordinates". In the "3D model", for example, data related to the 3D model itself such as point clouds and meshes is stored. In the "texture", data related to the texture image applied to the 3D model is stored. In the "average coordinates", data related to the coordinates of the point obtained by averaging all the coordinates of the point cloud constituting the 3D model is stored. In the "center of gravity coordinates", data related to the coordinates of the point that becomes the center of gravity based on all the coordinates of the point cloud constituting the 3D model is stored. In the "maximum / minimum coordinates", data related to the coordinates of the maximum / minimum points among the coordinates of the point cloud constituting the 3D model is stored. Note that the items of data stored in the table 220 of 3D model information are not limited to all of "3D model", "texture", "average coordinates", "center of gravity coordinates", and "maximum / minimum coordinates". For example, only "3D model" and "texture" may be used, or other items may be added to these items. In addition, the database 103 manages information related to the 3D model (not shown). Information related to the 3D model includes information such as the date and place of the rugby match, the confrontation card, and information related to rugby players if the imaging target is a rugby match.

[0020] By using the information shown in Fig. 2, when a certain time code is specified, it is possible to obtain the 3D model and texture image for each object as the 3D model information at the specified time code. For example, in Fig. 2(a), DataN100 can be obtained as the 3D model information of object N at the time code "16:14:24.041". The 3D model information for other time codes and objects can also be obtained by specifying them in the same way. Also, a certain time code can be specified, and the 3D model information of all objects at that time code can be obtained collectively. For example, the time code "16:14:25.021" can be specified, and the 3D model information DataA141, DataB141, ···, DataN141 of all objects can be obtained.

[0021] As described above, images obtained from imaging by a multi-sensor system are called multi-view images, and 3D models representing the three-dimensional shape of an object can be generated from such multi-view images. Specifically, foreground images are obtained by extracting the foreground region corresponding to an object such as a person or a ball, and background images are obtained by extracting the background region other than the foreground region. Based on these multiple foreground images, a 3D model of the foreground can be generated for each object. These 3D models are generated by a shape estimation method such as Visual Hull and consist of point clouds. However, the data format of the 3D model representing the shape of each object is not limited to this. The 3D models generated in this way are recorded in the database 103 for each time code and each object. Note that the method of generating the 3D models is not limited to this; it is sufficient that the 3D models are recorded in the database 103.

[0022] Furthermore, 3D models of background objects such as the field and spectator stands may be recorded in database 103 as 3D model information at timecode 00:00:00.000, etc.

[0023] Returning to the explanation of Figure 1(a), the image generation device 104 acquires a 3D model from the database 103 and generates a virtual viewpoint image based on the acquired 3D model. Specifically, the image generation device 104 acquires appropriate pixel values ​​from the multi-view image for each point constituting the acquired 3D model and performs colorization processing. Then, it places the colorized 3D model in a three-dimensional virtual space, projects it onto a virtual camera, and renders it to generate a virtual viewpoint image. The image generation device 104 also generates information related to the virtual viewpoint image.

[0024] The virtual camera 110 is set up in a virtual space associated with the target region 120, as shown in Figure 1(b), for example, and can view the region 120 from a viewpoint different from any of the cameras in the multiple sensor system 101. The viewpoint of the virtual camera 110 is determined by its position and orientation. Details of the position and orientation of the virtual camera 110 will be described later.

[0025] The image generation device 104 may determine the position and orientation of the virtual camera based on information sent from the mobile terminal 105. Details of this process will be described later with reference to a diagram.

[0026] The virtual viewpoint image is an image representing the view from the virtual camera 110, and is also called a free-viewpoint image. The virtual viewpoint image and related information generated by the image generation device 104 may be displayed on the image generation device 104, or they may be returned as a response to the mobile terminal 105, which is another device that is the source of the imaging data, and displayed on the mobile terminal 105. Naturally, the virtual viewpoint image may be displayed on both the image generation device 104 and the mobile terminal 105.

[0027] The molding device 106 is, for example, a 3D printer, and it molds a first output object 107, such as a doll (3D model figure), based on a virtual viewpoint image generated by the image generation device 104. The first output object 107 is an output object generated based on a 3D model recorded in the database 103, and in this embodiment, it is an object that is imaged by the mobile terminal 105. An example of a first output object is a 3D model figure molded using a 3D model by a molding device 106 such as a 3D printer. Here, a 3D model figure, which is an example of a first output object 107, will be explained with reference to a figure.

[0028] Figure 3 illustrates the process of capturing images of a 3D model figure and generating related information about the 3D model figure based on these captured images. Figure 3(a) shows an example of a 3D model figure, Figure 3(b) shows an example of an image of the 3D model figure in Figure 3(a), and Figure 3(c) shows an example of a virtual viewpoint image corresponding to the 3D model figure in Figure 3(a). In Figure 3(a), a 3D model of a rugby match is recorded in database 103, and an example of modeling using a 3D model at a certain time code is shown. Specifically, a scene where an offload pass is made as a decisive moment leading to a score during a rugby match is selected, and an example of modeling using a 3D model associated with that time code is shown.

[0029] As shown in Figure 3(a), the 3D model figure has a base 301, a first figure body 302, a second figure body 303, and a third figure body 304. The first figure body 302, the second figure body 303, and the third figure body 304 each correspond to the individual players, which are objects, and are fixed on the base 301. In Figure 3(a), the third figure body 304 is a 3D model figure of a player holding the ball and making an offload pass.

[0030] Furthermore, the front and sides of the base 301 are affixed with two-dimensional codes 311 and 312, which hold information such as the imaging direction and orientation when imaging is performed by the mobile terminal 105. The base 301 of the first output shown in Figure 3(a) is affixed with two two-dimensional codes 311 and 312, but the number and position of two-dimensional codes affixed to the first output are not limited to this. In addition, the information contained in the two-dimensional codes may include not only the imaging direction and orientation, but also image information related to the first output, such as match information in which the 3D model was captured. Note that the form in which information is affixed to the first output is not limited to two-dimensional codes, and other forms such as watermark information may also be used.

[0031] The first output 107 is not limited to three-dimensional objects such as 3D model figures, but can be anything generated using a 3D model in the database 103. For example, it could be a printed object on a plate or a virtual viewpoint image displayed on a screen.

[0032] The image generation device 104 may acquire a 3D model from the database 103, output it to the molding device 106, and then mold the first output object 107, such as a 3D model figure.

[0033] Let's return to the explanation of Figure 1. The mobile terminal 105 captures an image of the first output object 107, such as a 3D model figure, and transmits the captured data to the image generation device 104. The captured data (captured image) may be image data or data that includes attribute information in addition to the image data. Here, an example of a captured image will be explained using Figure 3(b). Figure 3(b) shows the state in which an image obtained by capturing the 3D model figure shown in Figure 3(a) as the first output object is displayed on the display unit of the mobile terminal. The mobile terminal 105 transmits the captured data of the first output object to the image generation device, and in response, receives and displays a virtual viewpoint image and related image information generated by the image generation device 104 as a second output object. Details of these processes will be described later with reference to the figures.

[0034] In this embodiment, the explanation will primarily focus on the case where the virtual viewpoint image is a video, but a still image may also be used.

[0035] (Configuration of the image generation device) An example configuration of the image generation device 104 will be explained using a diagram. Figure 4 shows an example configuration of the image generation device 104, with Figure 4(a) showing an example of the functional configuration of the image generation device 104 and Figure 4(b) showing an example of the hardware configuration of the image generation device 104.

[0036] As shown in Figure 4(a), the image generation device 104 includes an imaging data processing unit 401, a virtual camera control unit 402, a 3D model acquisition unit 403, an image generation unit 404, and an output data control unit 405. Using the aforementioned functional units, the image generation device 104 uses imaging data obtained by imaging a first output object generated from a 3D model to generate a virtual viewpoint image generated from the 3D model and related image information as a second output object. Here, an overview of each function is explained, and the details of each process will be described later.

[0037] The imaging data processing unit 401 receives imaging data obtained when the mobile terminal 105 images the first output object. The imaging data processing unit 401 then obtains generation information for the first output object from the received imaging data. Generation information for the first output object is information used to identify the data necessary to generate the first output object. For example, this could include the identifier of the database in which the 3D model that formed the basis of the first output object is recorded, and the time code of that 3D model. The database identifier is information used to uniquely identify the corresponding database. Therefore, this information makes it possible to uniquely identify the 3D model that formed the basis of the first output object.

[0038] Furthermore, the imaging data processing unit 401 acquires information such as the position and orientation of the imaging device that captured the first output object, and the focal length (hereinafter referred to as imaging information) from the imaging data. The position and orientation of the imaging device may be acquired from the 2D codes 311, 312, etc., included in the imaging data, or by other methods. Examples of other acquisition methods are shown in Embodiment 2.

[0039] The generation information and imaging information of the first output obtained from the imaging data are not limited to these. For example, if the first output is related to a sports match, it may include related information such as the match results and information on the opposing teams.

[0040] The virtual camera control unit 402 controls the virtual camera according to the position, orientation, and focal length of the imaging device acquired via the imaging data processing unit 401. Details of the virtual camera, its position, and orientation will be described later with reference to the diagram.

[0041] The 3D model acquisition unit 403 retrieves the corresponding 3D model from the database 103 by specifying the database identifier and time code obtained via the imaging data processing unit 401. The time code may also be specified by user operation on the mobile terminal 105.

[0042] The image generation unit 404 generates a virtual viewpoint image based on the 3D model acquired by the 3D model acquisition unit 403. Specifically, the image generation unit 404 acquires appropriate pixel values ​​from the image for each point that makes up the 3D model and performs coloring. Then, the image generation unit 404 places the colored 3D model in a three-dimensional virtual space and projects it onto a virtual camera (virtual viewpoint) controlled by the virtual camera control unit 402 and renders it to generate a virtual viewpoint image.

[0043] The virtual viewpoint image generated here is created by the imaging data processing unit 401 using the generation information of the first output object (information identifying the 3D model) and imaging information (the position and orientation of the imaging device that captured the first output object) obtained from the imaging data of the first output object.

[0044] The method for generating the virtual viewpoint image is not limited to this, and various methods may be used, such as generating the virtual viewpoint image by projective transformation of captured images without using a 3D model.

[0045] The output data control unit 405 outputs the virtual viewpoint image generated by the image generation unit 404 as a second output to an external device, such as a mobile terminal 105. Alternatively, the imaging data processing unit 401 may output related information regarding the first output to the external device as a second output.

[0046] Alternatively, the output data control unit 405 may generate molding data from the 3D model, output the generated molding data to the molding device 106, and obtain the first output object from the molding device 106.

[0047] (Hardware configuration of the image generation device) Next, the hardware configuration of the image generation device 104 will be explained using Figure 4(b). As shown in Figure 4(b), the image generation device 104 includes a CPU 411, RAM 412, ROM 413, operation input unit 414, display unit 415, and communication I / F (interface) unit 416.

[0048] The CPU (Central Processing Unit) 411 processes data using programs and data stored in the RAM (Random Access Memory) 412 and ROM (Read Only Memory) 413.

[0049] The CPU 411 controls the overall operation of the image generation device 104 and executes processing to realize each function shown in Figure 4(a). The image generation device 104 may have one or more dedicated hardware components separate from the CPU 411, and at least a portion of the processing performed by the CPU 411 may be executed by the dedicated hardware. Examples of dedicated hardware include ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and DSPs (Digital Signal Processors).

[0050] ROM413 holds programs and data. RAM412 has a work area for temporarily storing programs and data read from ROM413. RAM412 also provides a work area used by CPU411 when executing various processes.

[0051] The operation input unit 414 is, for example, a touch panel, which accepts user input and acquires information entered through the accepted user operation. Examples of input information include information about the virtual camera and information about the timecode of the virtual viewpoint image to be generated. The operation input unit 414 may also be connected to an external controller to receive user input information regarding the operation. The external controller may be, for example, a three-axis controller such as a joystick or an operating device such as a mouse. However, the external controller is not limited to these.

[0052] The display unit 415 displays a virtual viewpoint image generated by a touch panel or screen. In the case of a touch panel, the operation input unit 414 and the display unit 415 are integrated into a single configuration.

[0053] The communication interface unit 416 transmits and receives information with the database 103, mobile terminal 105, and molding device 106, for example, via a LAN. The communication interface unit 416 may also transmit information to an external screen via an image output port that supports the following communication standards. Examples of image output ports that support the communication standards include HDMI (High-Definition Multimedia Interface) and SDI (Serial Digital Interface). The communication interface unit 416 also acquires 3D model information from the database 103, for example, via Ethernet. The communication interface unit 416 receives imaging data from the mobile terminal 105 and transmits second output items such as virtual viewpoint images and related image information, for example, via Ethernet or short-range communication.

[0054] (Virtual viewpoint (virtual camera) and its operation screen) The virtual camera 110 and its position and orientation will be explained using diagrams. Figure 5 is a diagram illustrating the virtual camera 110 and its position and orientation. Figure 5(a) shows the coordinate system, Figure 5(b) shows an example of a field to which the coordinate system of Figure 5(a) is applied, Figures 5(c) and 5(d) show examples of the drawing area of ​​the virtual camera, and Figure 5(e) shows an example of the movement of the virtual camera.

[0055] First, we will explain the coordinate system that represents the three-dimensional space of the object to be imaged, which serves as the basis for setting the virtual specifications. As shown in Figure 5(a), in this embodiment, a Cartesian coordinate system is used that represents the three-dimensional space with three axes: the X, Y, and Z axes. This Cartesian coordinate system is set for each object shown in Figure 5(b), namely the rugby field 591, the ball 592 on it, the players 593, etc. Furthermore, it may also be set for equipment within the rugby field, such as spectator stands and billboards surrounding the field 591. Specifically, the origin (0,0,0) is set to the center of the field 591. The X axis is set in the direction of the long side of the field 591, the Y axis in the direction of the short side of the field 591, and the Z axis in the direction perpendicular to the field. Note that the direction of each axis is not limited to these. The position and orientation of the virtual camera 110 are set using such a coordinate system.

[0056] Next, the rendering range of the virtual camera will be explained using a diagram. In the square pyramid 500 shown in Figure 5(c), vertex 501 represents the position of the virtual camera 110, and the line-of-sight vector 502, originating from vertex 501, represents the orientation of the virtual camera 110. Vector 502 is also called the optical axis vector of the virtual camera. The position of the virtual camera is expressed by the components (x, y, z) of each axis, and the orientation of the virtual camera 110 is expressed by a unit vector with scalar components of each axis. The vector 502 representing the orientation of the virtual camera 110 is assumed to pass through the center points of the front clipping plane 503 and the rear clipping plane 504. The viewing frustum of the virtual camera, which is the projection range (rendering range) of the 3D model, is the space 505 sandwiched between the front clipping plane 503 and the rear clipping plane 504.

[0057] Next, the components indicating the rendering range of the virtual camera will be explained using a diagram. Figure 5(d) is a view of the virtual viewpoint in Figure 5(c) from above (Z-axis). The rendering range is determined by the following values: the distance 511 from vertex 501 to the front clipping plane 503, the distance 512 from vertex 501 to the rear clipping plane 504, and the field of view 513 of the virtual camera 110. These values ​​may be predetermined values ​​(default values) set in advance, or they may be set values ​​that have been changed by user operation. In addition, the field of view 513 may be a value obtained by using the focal length of the virtual camera 110 as a variable. Note that the relationship between the field of view and focal length is a common technique, so its explanation will be omitted.

[0058] Next, we will explain how to change the position of the virtual camera 110 (movement of the virtual viewpoint) and how to change the orientation of the virtual camera 110 (rotation). The virtual viewpoint can be moved and rotated in a space represented by three-dimensional coordinates. Figure 5(e) is a diagram illustrating the movement and rotation of the virtual camera. In Figure 5(e), the dashed arrow 506 represents the movement of the virtual camera (virtual viewpoint), and the dashed arrow 507 represents the rotation of the moved virtual camera (virtual viewpoint). The movement of the virtual camera is expressed by the components of each axis (x, y, z), and the rotation of the virtual camera is expressed by yaw (rotation around the Z axis), pitch (rotation around the X axis), and roll (rotation around the Y axis). In this way, the virtual camera can be freely moved and rotated in the three-dimensional space of the subject (field), so it can be generated as a virtual viewpoint image in which any area of ​​the subject becomes the drawing range.

[0059] In this embodiment, as described above, the position, orientation, and focal length of the imaging device are acquired as imaging information from the imaging data of the first output object 107 (3D model figure, etc.) captured by the mobile terminal 105, etc. Then, control such as moving and rotating the virtual camera is performed according to the acquired values ​​of the position, orientation, and focal length of the imaging device.

[0060] (Generation process for the second output) Next, the process of generating the second output in the image generation device 104 will be explained using diagrams. Figure 6 is a flowchart showing the flow of the process of generating the second output. The process of generating a virtual viewpoint image generated from the 3D model, or related image information, as the second output, based on imaging data obtained by imaging the first output generated from shape data (3D model) showing the three-dimensional shape of an object, will be explained using diagrams. This series of processes is realized by the CPU 411 executing a predetermined program and operating each functional unit shown in Figure 4(a). Hereafter, steps will be denoted as "S". The same will be used in the following explanation.

[0061] In S601, the image generation unit 404 reads the imaging data obtained by imaging the first output object 107. Specifically, the image generation unit 404 acquires imaging data obtained by imaging the first output object 107 with a mobile terminal 105 or the like via the imaging data processing unit 401. The first output object 107 is, for example, a 3D model figure output by the molding device 106 based on a 3D model in the database 103. An example of the first output object 107 is the 3D model figure shown in Figure 3(a).

[0062] In S602, the image generation unit 404 obtains generation information for the first output (database identifier and time code) and imaging information (position, orientation, and focal length of the imaging device) from the imaging data acquired in processing S601. Examples of imaging data include the image shown in Figure 3(b). Examples of methods for acquiring imaging information include obtaining it from the 2D code described in the configuration of the image generation device above. If the acquired imaging data is the image in Figure 3(b), the 2D code 311 contained in this image will provide the database identifier and time code containing the 3D model that formed the basis of the first output, as well as information regarding the position and orientation of the imaging device.

[0063] In S603, the image generation unit 404 uses the database identifier and time code included in the generation information of the first output obtained in the processing of S602 to retrieve the corresponding 3D model from the database 103.

[0064] In S604, the image generation unit 404 controls the virtual camera by moving and rotating it according to the position, orientation, and focal length of the imaging device included in the imaging information acquired in processing S602. Then, the image generation unit 404 uses the controlled virtual camera and the 3D model acquired in S603 to generate a virtual viewpoint image as a second output. Examples of methods for generating the virtual viewpoint image include the method for generating the virtual viewpoint image described in the configuration of the image generation device described above.

[0065] Here, we will explain the virtual viewpoint image, which is an example of a second output, using Figure 3(c). Figure 3(c) shows a 3D model associated with a time code acquired from the imaging data, and a virtual viewpoint image generated using a virtual camera with a position and orientation acquired from the imaging data. In other words, the 3D model figure in Figure 3(a) (first output) and the virtual viewpoint image in Figure 3(c) (second output) are like the same 3D model associated with the same time code, viewed from the same position and direction. The operation screen 320 has a virtual camera operation area 322 that accepts user operations to set the position and orientation of the virtual camera 110, and a time code operation area 323 that accepts user operations to set the time code. First, we will explain the virtual camera operation area 322. Since the operation screen 320 is displayed on a touch panel, the virtual camera operation area 322 accepts general touch operations 325 such as taps, swipes, and pinch-in / out as user operations. The position and focal length (angle of view) of the virtual camera are adjusted by these touch operations 325. Furthermore, the virtual camera operation area 322 accepts touch operations 324 as user operations, such as pressing and holding each axis of the Cartesian coordinate system. These touch operations 324 cause the virtual camera 110 to rotate around the X, Y, or Z axis, and the orientation of the virtual camera is adjusted. By assigning movement, rotation, and scaling of the virtual camera to each touch operation on the operation screen 320 in this way, the virtual camera 110 can be freely manipulated. These operation methods are publicly known and will not be explained further.

[0066] Next, the timecode operation area 323 will be described. The timecode operation area 323 has a main slider 332 and a knob 342, and has multiple elements for operating the timecode. The timecode operation area 323 also has an output button 350.

[0067] The main slider 332 is an input element that allows you to set a desired timecode from the total timecodes of the imaging data. When the position of the knob 342 is moved to the desired position by dragging or other means, the timecode corresponding to the position of the knob 342 is specified. In other words, any timecode can be specified by adjusting the position of the knob 342 on the main slider 332.

[0068] In the S604 process, instead of using the timecode included in the generation information of the first output directly, a predetermined range of timecodes from a timecode slightly before to a timecode slightly after may be used. In this case, a 3D model corresponding to this predetermined range of timecodes will be obtained. As a result, the decisive scene, which is modeled as the 3D model figure that is the first output, can be viewed as a virtual viewpoint image, which is the second output, with a series of plays starting from a timecode slightly before it. The virtual viewpoint images reproduced in this way are shown in order from Figure 3(d) to Figure 3(f).

[0069] Figure 3(e) shows the same virtual viewpoint image 362 as the virtual viewpoint image in Figure 3(c) described earlier, and is generated using the same 3D model with the same timecode (T1) as the 3D model figure which is the first output in Figure 3(a). Figure 3(d) shows the virtual viewpoint image 361 generated using the 3D model with a timecode (T1-Δt1) slightly earlier than the virtual viewpoint image 362 shown in Figure 3(e). Figure 3(f) shows the virtual viewpoint image 363 generated using the 3D model with a timecode (T1+Δt1) slightly later than the virtual viewpoint image 362 shown in Figure 3(e). When the 3D model figure, which is the first output, is captured by the mobile terminal 105, etc., the virtual viewpoint images of the second output can be played back on the mobile terminal 105, etc., in the order of Figures 3(d), 3(e), and 3(f). Naturally, there are frames not shown between the virtual viewpoint image 361 shown in Figure 3(d) and the virtual viewpoint image 363 shown in Figure 3(f), and the virtual viewpoint images from time code (T1-Δt1) to time code (T1+Δt1) can be played back at 60fps, etc. The width (2Δt1) from the slightly earlier time code (T1-Δt1) to the slightly later time code (T1+Δt1) mentioned above may be included in the generation information of the first output object in advance, or it may be specified by user operation. Regarding the position and orientation of the virtual camera, the position and orientation obtained from the captured image (Figure 3(b), etc.) that captured the first output object may be used for the virtual viewpoint image (Figure 3(e)) of that time code, and different positions and orientations may be used for the virtual viewpoint images of other time codes. The time codes, position and orientation may be adjusted, for example, by adjusting the position of the time code knob 342 by user operation or by tapping the output button 350 on the mobile terminal 105.

[0070] The generated virtual viewpoint image of the second output may be sent to the mobile terminal 105 and displayed on the screen of the mobile terminal 105. In addition to the virtual viewpoint image, other image information related to the virtual viewpoint image, such as match results, may also be displayed.

[0071] As described above, according to this embodiment, a virtual viewpoint image or related image information can be output from an image of an output object that is output based on the same 3D model used to generate the virtual viewpoint image, using the same 3D model used to generate the output object. In other words, a 3D model figure generated based on a scene corresponding to a specific time in the virtual viewpoint image is captured, and an image is obtained. Using this image of the 3D model figure, a virtual viewpoint image of the scene corresponding to the specific time can be generated.

[0072] For example, when using a 3D model of a rugby match, if the 3D model figure is captured by a mobile device, it is possible to display a virtual viewpoint image using the same 3D model at least at the same timecode.

[0073] Furthermore, at that time, it is possible to display the same direction from which the 3D model figure was captured as a virtual viewpoint image viewed from the same direction.

[0074] Furthermore, while the above describes a case where a corresponding 3D model is identified based on a 2D code obtained from imaging data, and a virtual viewpoint image corresponding to the identified 3D model is generated, the method for identifying the corresponding 3D model is not limited to the method using a 2D code. For example, feature quantities of objects (e.g., the color and number of a player's uniform) can be extracted from each scene to create multiple patterns, which can then be managed in a database beforehand. The corresponding 3D model can then be identified based on the results of image processing such as pattern matching. By identifying the corresponding 3D model in this way, a virtual viewpoint image corresponding to the identified 3D model, or image information related to the virtual viewpoint image, can be generated.

[0075] [Embodiment 2] In this embodiment, we will describe a method of obtaining detailed position and orientation from imaging data obtained by capturing a first output object, which is generated based on a 3D model used to generate a virtual viewpoint image, with a mobile terminal or the like, and then generating a second output object using the same 3D model with the same position and orientation.

[0076] In this embodiment, the configuration of the information processing system is the same as in Figure 1, and the configuration of the image generation device is the same as in Figure 4, so their explanations will be omitted, and the differences will be explained.

[0077] The configuration described in Embodiment 1, in which the image data of the first output object captured by the mobile terminal 105 is processed by the image generation device 104 and a second output object is generated based on the information contained in the image data, is the same. In this embodiment as well, the flowchart for the generation process of the second output object can be executed in the same way as in Figure 6, but the method of processing the image data in S604 differs from that of Embodiment 1.

[0078] In this embodiment, the second output object is generated using the field of view of the image data obtained by capturing an image of the first output object with the mobile terminal 105. The generation of a second output object that reflects the field of view of the image data of the first output object will be explained with reference to Figure 7.

[0079] Figure 7 illustrates the process of capturing an image of a 3D model figure and generating related information about the 3D model figure based on this captured image. Figure 7(a) shows an example of a 3D model figure, which is a first output example similar to Figure 3(a). Figure 7(b) shows an example of an image of the 3D model figure from Figure 7(a), and Figure 7(c) shows an example of a virtual viewpoint image in which an object corresponding to the 3D model figure from Figure 7(a) exists.

[0080] As shown in Figure 7(a), the 3D model figure has a base 301, a first figure body 302, a second figure body 303, and a third figure body 304. The first figure body 302, the second figure body 303, and the third figure body 304 each correspond to the individual players, which are objects, and are fixed on the base 301. Markers 701, 702, and 703 are placed on the upper surface of the base 301, near the front, for coordinate recognition. For convenience, these markers 701-703 are called coordinate markers (also called calibration targets, etc.). Note that the coordinate markers 701-703 may be visible or invisible due to watermark information, etc. The number of coordinate markers that can be placed on the first output is not limited to three; there may be fewer or more than three. The location where the coordinate markers 701-703 are placed is not limited to the base 301. The coordinate markers 701-703 may be inconspicuously attached to, for example, the uniforms or numbers of the figures 302-304. Alternatively, the coordinate markers 701-703 may be inconspicuously embedded in the field and lines on the base 301. The shape of each coordinate marker 701-703 is not limited, as long as it can uniquely identify each coordinate marker 701-703.

[0081] As shown in Figure 7(b), the imaging data includes coordinate markers 701-703. Coordinate information can be obtained from these multiple coordinate markers 701-703, and the position and coordinates of the mobile terminal 105 at the time of imaging can be accurately calculated using the obtained coordinate information. This calculation method is called camera calibration, and various methods are publicly known, so a detailed explanation is omitted. Note that the number of coordinate markers required for the calculation (e.g., 6 points) differs depending on the method, so if the required number of coordinate markers are not included when the first output object is imaged by the mobile terminal 105, a warning image may be displayed on the screen of the mobile terminal 105 to notify the user to capture the required number of coordinate markers.

[0082] The coordinate markers 701-703 included in the first output are defined based on the coordinate system shown in Figure 5(a). Since this is the same coordinate system as the 3D data included in database 103, the position and orientation of the imaging camera obtained by camera calibration can be acquired in the same coordinate system as the 3D model. Therefore, in this embodiment, the generated virtual viewpoint image can have the same field of view as the mobile terminal 105 that captured the image, using the position and coordinates obtained from the imaging data and the focal length. An example of a virtual viewpoint image, which is a second output generated in this way, is shown in Figure 7(c). As can be seen from Figures 7(b) and 7(c), the field of view of the 3D model figure (first output) in Figure 7(b) and the field of view of the virtual viewpoint image (second output) in Figure 7(c) are the same.

[0083] Note that coordinate markers 701-703 are attached to the first output and are not included in the 3D model recorded in database 103; therefore, they are not included in the second output, the virtual viewpoint image shown in Figure 7(c).

[0084] Furthermore, instead of using the timecode (T2) included in the generation information of the first output as is, the 3D model may be acquired using the timecode from a little before (T2-Δt2) to a little after (T2+Δt2). In this case, the virtual viewpoint image for all of that timecode may be displayed using the field of view of the imaging data, or the field of view of the imaging data may be used only for the timecode included in the generation information of the first output. In this case, a virtual viewpoint image may be generated with a different field of view for a little before (T2-Δt2), and a virtual viewpoint image may be generated that gradually becomes the field of view of the imaging data towards that timecode (T2).

[0085] Next, we will explain another example in which the first output object in Figure 7(a) is used as the target of imaging by the mobile terminal 105, using Figure 7(d), which shows an example of the imaging data, and Figure 7(e), which shows an example of the second output object generated based on the imaging data in Figure 7(d).

[0086] As shown in Figure 7(d), this is image data captured by zooming in on the third figure body 304 included in the 3D model figure, which is the first output in Figure 7(a), using the zoom function of the mobile terminal 105. If the third figure body 04 in the image data contains multiple coordinate markers (not shown), the position and orientation of the imaging camera that will have the same field of view as in Figure 7(d) can be calculated using the camera calibration described above.

[0087] Then, by controlling the virtual camera according to the calculated position, orientation, and focal length, and generating a virtual viewpoint image, the second output shown in Figure 7(e) can be obtained. As shown in Figures 7(d) and 7(e), the field of view of the first output (3D model figure) in Figure 7(d) and the second output (virtual viewpoint image) in Figure 7(e) are the same.

[0088] Furthermore, when processing image data captured by zooming in on the figure (athlete) using the zoom function of the mobile terminal 105, it is also possible to use a method that does not use the multiple coordinate markers mentioned above.

[0089] For example, the position and posture that result in the recommended zoom angle for the figure (player) may be pre-defined as the zoom angle information for the figure (player), and this information may be added to the figure (player) in the first output using a two-dimensional code or watermark. If the image generation device 104 receives the image data from the mobile terminal 105 and includes the zoom angle information for the figure (player), it may control the virtual camera based on this information to generate a virtual viewpoint image of the figure (player) in a zoomed-in state for the second output. In this case, the image data and the examples of the generated second output will be the same as those shown in Figures 7(d) and 7(e) above.

[0090] As described above, according to this embodiment, a second output object, such as a virtual viewpoint image, can be generated and displayed, which has the same field of view as the captured image obtained by, for example, capturing the output object with a mobile terminal. For example, if the first output object was a 3D model figure, the same 3D model that was created and a virtual viewpoint image with the same field of view as the image can be generated as the second output object.

[0091] For example, if the first output is a 3D model figure generated using a 3D model of a rugby match, the user can take a picture with their mobile device at their preferred angle of view, and a virtual viewpoint image (second output) with the same angle of view can be played back as a video.

[0092] For example, if the first output is a 3D model figure, and you zoom in on one of the players included in it and take an image, you can generate a virtual viewpoint image of that player zoomed in as the second output.

[0093] [Other embodiments] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of symbols]

[0094] 103 Databases 104 Image generation device 604 Image Generation Unit

Claims

1. An acquisition means for acquiring information to identify a specific scene corresponding to a three-dimensional object contained in a captured image, Based on the aforementioned information, an output means outputs an instruction to output three-dimensional shape data that corresponds to the specific scene, which is three-dimensional shape data generated based on multiple captured images. It has, The aforementioned information is a time code associated with the three-dimensional shape data. An information processing device characterized by the following:

2. The information processing device according to claim 1, characterized in that the three-dimensional object is generated based on the three-dimensional shape data.

3. The system includes a generation means for generating a virtual viewpoint image based on the three-dimensional shape data, The output means outputs the virtual viewpoint image to an external device. The information processing apparatus according to claim 1 or 2.

4. The acquisition means acquires a plurality of three-dimensional shape data corresponding to a predetermined time including the time code, The output means outputs a virtual viewpoint image generated based on the three-dimensional shape data corresponding to the time code specified based on user operation, from among the plurality of three-dimensional shape data. The information processing apparatus according to feature 1.

5. The information processing device according to any one of claims 1 to 4, characterized in that the three-dimensional object is a figure.

6. The information processing device according to any one of claims 1 to 5, characterized in that the aforementioned information is obtained based on a two-dimensional code attached to the three-dimensional object.

7. An acquisition means for acquiring information for identifying a specific scene corresponding to a three-dimensional object contained in a captured image, Based on the aforementioned information, an output means outputs an instruction to output three-dimensional shape data that corresponds to the specific scene, which is three-dimensional shape data generated based on multiple captured images. It has, The aforementioned three-dimensional object and the aforementioned three-dimensional shape data represent the same person. An information processing device characterized by the following:

8. A means of photographing three-dimensional objects, An output means for outputting information for identifying a specific scene corresponding to the three-dimensional object, A display control means that controls the display of a virtual viewpoint image generated based on three-dimensional shape data corresponding to the aforementioned specific scene. It has, The aforementioned information is a time code associated with the three-dimensional shape data. An information processing device characterized by the following:

9. The information processing apparatus according to claim 8, characterized in that the virtual viewpoint image is generated by an external device.

10. The information processing apparatus according to claim 8 or 9, characterized in that the virtual viewpoint image is generated based on information indicating the position and orientation of the information processing apparatus at the time of shooting and three-dimensional shape data corresponding to the specific scene.

11. A means for photographing a three-dimensional object, An output means for outputting information for identifying a specific scene corresponding to the three-dimensional object, A display control means that controls the display of a virtual viewpoint image generated based on three-dimensional shape data corresponding to the aforementioned specific scene, It has, The aforementioned three-dimensional object and the aforementioned three-dimensional shape data represent the same person. An information processing device characterized by the following:

12. An acquisition means for acquiring information to identify a specific scene corresponding to a three-dimensional object contained in a captured image, Based on the aforementioned information, an output means for outputting three-dimensional shape data generated from multiple captured images that corresponds to the specific scene, It has, The aforementioned information is a time code associated with the three-dimensional shape data. An information processing system characterized by the following:

13. Acquisition means for acquiring information for identifying a specific scene corresponding to a three-dimensional object contained in a captured image, Based on the aforementioned information, an output means for outputting three-dimensional shape data generated from multiple captured images that corresponds to the specific scene, It has, The aforementioned three-dimensional object and the aforementioned three-dimensional shape data represent the same person. An information processing system characterized by the following:

14. A means of photographing three-dimensional objects, An output means for outputting information for identifying a specific scene corresponding to the three-dimensional object, A display control means that controls the display of a virtual viewpoint image generated based on three-dimensional shape data corresponding to the aforementioned specific scene, It has, The aforementioned information is a time code associated with the three-dimensional shape data. An information processing system characterized by the following:

15. A means for photographing a three-dimensional object, An output means for outputting information for identifying a specific scene corresponding to the three-dimensional object, A display control means that controls the display of a virtual viewpoint image generated based on three-dimensional shape data corresponding to the aforementioned specific scene, It has, The aforementioned three-dimensional object and the aforementioned three-dimensional shape data represent the same person. An information processing system characterized by the following:

16. An acquisition process to obtain information for identifying a specific scene corresponding to a three-dimensional object contained in the captured image, Based on the aforementioned information, an output step is performed to output three-dimensional shape data that corresponds to the specific scene, which is generated based on multiple captured images. It has, The aforementioned information is a timecode associated with the three-dimensional shape data. An information processing method characterized by the following:

17. An acquisition step of acquiring information for identifying a specific scene corresponding to a three-dimensional object contained in a captured image, Based on the aforementioned information, an output step is performed to output three-dimensional shape data that corresponds to the specific scene, which is generated based on multiple captured images. It has, The aforementioned three-dimensional object and the aforementioned three-dimensional shape data represent the same person. An information processing method characterized by the following:

18. A photographing step for photographing a three-dimensional object, An output process that outputs information for identifying a specific scene corresponding to the three-dimensional object, A display control step that controls the display of a virtual viewpoint image generated based on three-dimensional shape data corresponding to the aforementioned specific scene, It has, The aforementioned information is a timecode associated with the three-dimensional shape data. An information processing method characterized by the following:

19. A photographing step for photographing a three-dimensional object, An output process that outputs information for identifying a specific scene corresponding to the three-dimensional object, A display control step that controls the display of a virtual viewpoint image generated based on three-dimensional shape data corresponding to the aforementioned specific scene, It has, The aforementioned three-dimensional object and the aforementioned three-dimensional shape data represent the same person. An information processing method characterized by the following:

20. A program for causing a computer to function as an information processing device according to any one of claims 1 to 11.

Images