Image processing apparatus, image processing method, and program

Abstract

To allow for easily selecting a camera path in a scene when generating a virtual viewpoint image.SOLUTION: An image processing apparatus includes; acquisition means which acquires an input corresponding to a user operation of selecting a specific scene for which a virtual viewpoint image is to be generated, from among a plurality of scenes of an imaging object captured by imaging means; and display control means which controls the apparatus to display information indicating positions and attitudes of a plurality of virtual cameras associated with the specific scene.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for generating a virtual viewpoint image. [Background technology]

[0002] Video content such as sporting events or concerts shot from multiple angles can be recorded, allowing users to play back the video from their desired scene and angle. Patent Document 1 discloses a technology that plays back multiple video contents simultaneously on one screen, allowing the user to select from among them the video from their desired angle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-141895 Summary of the Invention [Problem to be solved by the invention]

[0004] A virtual viewpoint image is an image generated by creating a three-dimensional model (3D model) from images captured by multiple cameras and generating it according to a camera path that indicates the viewpoint (angle) of the camera. Camera paths must be created in advance, but it is not always possible to prepare them in advance for all scenes. Furthermore, because camera paths can be created from any viewpoint, the number of camera paths that include each scene is indefinite. While these camera paths can be created freely, the technology described in Patent Document 1 has the problem of making it difficult to select a desired scene and camera path. An object of the present invention is to make it easy to select a camera path for a scene from which a virtual viewpoint image is generated. [Means for solving the problem]

[0005] The image processing device according to the present invention is characterized by having an acquisition means for acquiring input corresponding to a user operation for selecting a specific scene for generating a virtual viewpoint image from among multiple scenes of an imaging target captured by an imaging means, and a display control means for controlling the display of information indicating the positions and orientations of multiple virtual cameras associated with the specific scene. [Effects of the Invention]

[0006] According to the present invention, it is possible to easily select a camera path in a scene for which a virtual viewpoint video is to be generated. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of an image processing apparatus. [Figure 2] FIG. 2 is a diagram illustrating an example of a functional configuration of an image processing apparatus. [Figure 3] FIG. 2 is a diagram illustrating scene information. [Figure 4] FIG. 10 is a diagram illustrating camera path information. [Figure 5] FIG. 10 is a diagram illustrating a scene selection screen. [Figure 6] FIG. 10 is a diagram illustrating a camera path selection screen. [Figure 7] FIG. 10 is a diagram illustrating a camera path selection screen. [Figure 8] 10 is a flowchart illustrating an example of a process related to scene selection. [Figure 9] 10 is a flowchart illustrating an example of a process related to camera path selection. [Figure 10] FIG. 10 is a diagram illustrating an example of a display screen. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0009] In the following, as an application example of the image processing device in this embodiment, a playback service capable of capturing images of a game such as baseball and playing back a desired scene as a virtual viewpoint image will be described. Here, there are two users of the image processing device: a provider and a user. A provider refers to a person who uses an image processing device to provide a playback service. A provider, for example, captures images of a game such as baseball and operates an image processing device to prepare a three-dimensional model (3D model) required to generate a virtual viewpoint image. A user refers to a person who uses an image processing device to receive the playback service. A user operates an image processing device to view a virtual viewpoint image of a desired scene and camera path.

[0010] 1 is a diagram illustrating an example of the hardware configuration of an image processing device 100 according to this embodiment. The image processing device 100 includes a CPU 111, a ROM 112, a RAM 113, an auxiliary storage device 114, a display unit 115, an operation unit 116, a communication I / F 117, and a bus 118.

[0011] A CPU (Central Processing Unit) 111 controls the entire image processing device 100 using computer programs and data stored in a ROM 112 and a RAM 113, thereby realizing each function of the image processing device 100. Note that the image processing device 100 may have one or more dedicated hardware devices different from the CPU 111, and at least a part of the processing by the CPU 111 may be executed by the dedicated hardware devices. Examples of the dedicated hardware devices include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor).

[0012] ROM (Read Only Memory) 112 stores programs and the like that do not require modification. RAM (Random Access Memory) 113 temporarily stores programs and data supplied from an auxiliary storage device 114, and data and the like supplied from the outside via a communication I / F 117. The auxiliary storage device 114 is configured, for example, by an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various data such as image data and audio data.

[0013] The display unit 115 is configured with, for example, a liquid crystal display or an organic EL display, and displays a GUI (Graphical User Interface) for a user to operate the image processing device 100, images related to image data, etc. The operation unit 116 is configured with, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and receives operations by a user and inputs various instructions to the CPU 111. The communication I / F 117 is used for communication with devices external to the image processing device 100. For example, when the image processing device 100 is connected to an external device via a wired connection, a communication cable is connected to the communication I / F 117. Furthermore, for example, when the image processing device 100 has a function for wireless communication with an external device, the communication I / F 117 is equipped with an antenna. The bus 118 connects each unit of the image processing device 100 to transmit information.

[0014] 2 is a diagram illustrating an example of the functional configuration of the image processing device 100 according to this embodiment. The image processing device 100 includes a capture unit 210, a generation unit 220, a designation unit 230, and a registration unit 250. The capture unit 210 includes an image acquisition unit 211 and a three-dimensional model (3D model) generation unit 212. The generation unit 220 includes a three-dimensional model (3D model) storage unit 221 and a virtual viewpoint image generation unit 222. The designation unit 230 includes a scene information storage unit 231, a camera path information storage unit 232, a selection unit 240, and a viewpoint designation unit 233. The selection unit 240 includes a scene selection unit 241, a camera path selection unit 242, and a camera path counting unit 243. The registration unit 250 includes a playback count update unit 251 and a camera path registration unit 252. The image acquisition unit 211 is an example of an image acquisition means, the virtual viewpoint image generation unit 222 is an example of a generation means, and the 3D model holding unit 221 is an example of a second holding means. The camera path information holding unit 232 is an example of a first holding means. The selection unit 240 and the operation unit 116 are examples of an acquisition means, and the selection unit 240 and the display unit 115 are examples of a display control means.

[0015] The image processing device 100 is communicably connected to a plurality of imaging units 260 that capture images of an imaging area from a plurality of directions. The imaging area is an area to be imaged, such as a stadium where a baseball or soccer game is played, or a stage where a concert or a play is held. The plurality of imaging units 260 are installed at different positions surrounding such an imaging area and capture images in synchronization. A time code indicating the time of capture is assigned to each frame of the captured video. Note that the imaging unit 260 includes a camera, but each imaging unit 260 may be configured to include a camera with different functions, such as a telephoto camera or a wide-angle camera.

[0016] The image acquiring unit 211 acquires a plurality of captured images (including time codes) captured by a plurality of imaging units 260. The image acquiring unit 211 may acquire a plurality of captured images from a plurality of imaging units 260, or may acquire a plurality of captured images captured from a recording medium or the like on which the captured images captured by the plurality of imaging units 260 are recorded.

[0017] The 3D model generation unit 212 generates a three-dimensional model (3D model) from a plurality of captured images captured by the plurality of imaging units 260 and acquired by the image acquisition unit 211. The 3D model is generated, for example, by the following method. First, the 3D model generation unit 212 separates the images captured by the plurality of imaging units 260 into a foreground image and a background image, for example, using a background subtraction method. Here, the foreground image is an image obtained by extracting an area corresponding to a predetermined subject, such as a person or a ball, from the captured image. Furthermore, the background image is an image obtained by excluding the foreground area from the captured image. Next, the 3D model generation unit 212 generates a 3D model of the subject from the separated plurality of foreground images using a volume intersection method.

[0018] The 3D model holding unit 221 holds the 3D model generated by the 3D model generation unit 212 in association with the time code assigned to the captured image. The 3D model holding unit 221 may also hold a 3D model of the object to be captured that serves as the background of a stadium, stage, etc., created in advance using three-dimensional measurement or the like. The 3D model holding unit 221 also holds texture data used to color the 3D model of the subject and the 3D model of the background. The texture data is generated based on the foreground image and the background image.

[0019] The virtual viewpoint image generation unit 222 generates a virtual viewpoint image according to the time code and viewpoint specified by the selection unit 240 or the viewpoint specification unit 233. The virtual viewpoint image generation unit 222 generates a virtual viewpoint image by mapping texture data to a 3D model associated with the specified time code and rendering it according to the specified viewpoint.

[0020] The scene information storage unit 231 stores information about scenes in a game, concert, or the like to be imaged. FIG. 3 shows an example of scene information for a scene in which a baseball game is imaged. The scene information is managed by a scene ID 301 uniquely assigned to each scene, and includes, for example, a start TC 302 (hereinafter, TC refers to time code), an end TC 303, a representative TC 304, a playback count 305, and information 306 describing the scene. The start TC 302 is a time code indicating the start time of the scene. The end TC 303 is a time code indicating the end time of the scene. The representative TC 304 is a time code indicating the time of a representative moment in the scene. For example, in a hit scene, the time when the bat hits the ball may be set as the representative TC. Note that this example shows an example in which the scene has one representative TC 304, but this is not limited to this. The scene information may include multiple representative TCs and each representative TC as the representative TC 304. The number of times of reproduction 305 is the number of times a virtual viewpoint image of the scene has been generated, and can be substituted with the number of times a virtual viewpoint image has been generated by the virtual viewpoint image generation unit 222. The information 306 describing the scene is, for example, information about the inning, pitcher, batter, play, etc. Each of these pieces of information can be input manually or by recognition, etc.

[0021] The scene information stored in the scene information storage unit 231 may be created manually by a provider from a game or concert captured by a scene information acquisition unit (not shown), or may be created automatically from captured images using technology such as image recognition. Information regarding time may be acquired as a time code from the image capture unit 260. While the example shown in FIG. 3 separates scenes for each batter, scenes may be separated for each pitch thrown by the pitcher. There may also be periods during a game where no scenes are registered.

[0022] The camera path information storage unit 232 stores information about camera paths for reproducing scenes of a game, concert, or the like to be imaged using virtual viewpoint images. The camera path is information including the position and direction of the camera (viewpoint) in three-dimensional space when generating a virtual viewpoint image (the position and orientation of the virtual camera). FIG. 4 shows an example of camera path information. The camera path information is managed by a camera path ID 401 uniquely assigned to each camera path, and includes, for example, a start TC 402, an end TC 403, a number of plays 404, creator information 405, and a camera path 406. Each of these pieces of information may be acquired by a camera path information acquisition unit (not shown). For example, the camera path information acquisition unit can input information using a character input means such as a keyboard or a mouse. The start TC 402 is a time code indicating the start time of the camera path. The end TC 403 is a time code indicating the end time of the camera path. The number of plays 404 is the number of times a virtual viewpoint image of the camera path has been generated. The creator information 405 is information identifying the creator of the camera path. "System" in the creator information 405 indicates that the creator of the camera path is a provider that provides a playback service using virtual viewpoint images using the image processing device 100. "User" in the creator information 405 indicates that the creator of the camera path is a user who receives the playback service using virtual viewpoint images.

[0023] The camera path 406 is a pair of a time code and a viewpoint required to generate a virtual viewpoint image. The pair of a time code and a viewpoint is not limited to one, and there may be two or more pairs. When there is one pair of a time code and a viewpoint, an image is generated, and when there are two or more pairs, multiple images, i.e., a video, are generated. Here, the viewpoint indicates the position and orientation of the virtual camera, and is composed of parameters such as the position, direction (pan, tilt, roll), and focal length of the virtual camera.

[0024] The scene selection unit 241 performs processing related to scene selection by a user (not shown). The scene selection unit 241 displays a scene selection screen on the display unit 115 as a UI (User Interface) for the user to select a scene.

[0025] FIG. 5 shows an example of a scene selection screen. The scene selection screen displays scene information in chronological order. The scene playback count graph 501 is a graph plotting the number of playbacks for each scene. The play information 502 is information related to the play of the scene. Predetermined information (here, scores) is displayed as the play information 502 according to the screen size, etc. Scene markers 503 are markers indicating scenes stored in the scene information storage unit 231, and are displayed in chronological order using the representative TC of the scene. In the example shown in FIG. 5, the scene markers 503 are shown as rectangular markers for simplicity, but may also be thumbnail images indicating the representative TC. The thumbnail image shown as the scene marker 503 may also be based on the camera path with the highest number of playbacks, for example. The statistical information is not limited to TC units, and may also be aggregated in minutes or for a fixed time period. The camera path retention count graph 504 is a graph plotting the number of retained camera paths associated with each scene. The number of camera paths held for a scene is the number of camera paths that include that scene among the camera paths held by the camera path information holding unit 232. Here, a camera path that includes a scene is a camera path that has a representative TC of the scene between the start TC and end TC of the camera path. One camera path may include two or more scenes. The number of camera paths held associated with a scene is counted, for example, by the camera path counting unit 243 (described later). The inning information 505 is information about the inning of the scene. The scroll bar 506 is a scroll bar for specifying the display range of the scene selection screen. Based on the various information displayed on the scene selection screen, the user selects a desired scene, for example, by selecting one of the scene markers 503 on the operation unit 116 (for example, by tapping on a touch panel or clicking with a mouse, etc.). When the user selects a desired scene, the scene selection unit 241 sends information about the selected scene, such as scene information, information about camera paths that include the scene, and the number of camera paths held for the scene, to the camera path selection unit 242.

[0026] In addition, when a large number of representative TCs of scenes are collected in a short period of time, all scene markers can be identified and selected by drawing a leader line on the scene selection screen or by displaying only that section separately and enlarging it. Furthermore, when a large number of representative TCs of scenes are collected in a short period of time, those scenes may be combined and treated as a single scene. In this case, only one scene marker 503 may be displayed on the scene selection screen. To make it easier to identify that multiple scenes have been combined, the scene markers 503 may be displayed in a different color or shape from the other scene markers 503. When multiple scenes are combined, for example, the representative TC of the scene is the average of the representative TCs of the combined scenes, the number of times a scene is played is the total number of times the combined scenes are played, and the number of held camera passes is the total number of held camera passes for the combined scenes. Furthermore, scene selection can also be performed by, for example, displaying the entire field, displaying each player, or displaying only the relevant inning or time.

[0027] The camera path counting unit 243 counts the number of camera paths held for each scene using the following method. First, the camera path counting unit 243 receives the representative TC of the scene to be counted from the scene selection unit 241. Next, for each camera path held by the camera path information holding unit 232, if the representative TC of the scene is between the start TC and end TC of the camera path, the camera path counting unit 243 determines that the camera path includes the scene. Then, the camera path counting unit 243 counts the number of camera paths that include the scene and sets this as the number of held camera paths for the scene. The counted number of held camera paths for the scene is sent to the scene selection unit 241. The camera path counting unit 243 also sends information about the camera paths that include the scene to the scene selection unit 241.

[0028] The camera path selection unit 242 performs processing related to the selection of a camera path in the camera path information storage unit 232 in accordance with the selection of a camera path made by a user (not shown). The camera path selection unit 242 displays a camera path selection screen, which is a UI for the user to select a camera path, on the display unit 115. For example, the camera path selection unit 242 performs different processing, such as switching the display of information, as described below, in accordance with the number of camera paths including the selected scene.

[0029] FIG. 6 shows an example of a camera path selection screen displayed when the number of camera paths including a selected scene is equal to or less than a predetermined number. The camera path selection screen displays information about the selected scene, for example, in a bird's-eye view of the imaging area. The subject marker 601 is a marker indicating the position of the subject in the selected scene. The camera path selection unit 242 acquires the position of each subject in the representative TC of the selected scene from the 3D model of the subject stored in the 3D model storage unit 221 and displays it at the corresponding position on the bird's-eye view. The subject marker 601 may be displayed as a specific shape or a 3D model. The viewpoint marker 602 is a marker indicating the angle of view of the camera path associated with the selected scene and including the scene. The angle of view is represented, for example, by the position, direction, and focal length of the viewpoint (virtual camera). In the example shown in FIG. 6, the angle of view of the viewpoint of the camera path in the representative TC of the selected scene is displayed using an isosceles triangle. The vertex of the isosceles triangle (the intersection of the equal sides) is displayed as the viewpoint position, the direction of the perpendicular line from the vertex to the base as the viewpoint direction, and the height as the focal length. Based on this displayed information, the user selects a desired camera path, for example, by selecting one of the viewpoint markers 602 on the operation unit 116 (for example, by tapping in the case of a touch panel or by clicking with a mouse, etc.). Information on the selected camera path is sent to the virtual viewpoint image generation unit 222, which generates a virtual viewpoint image. Note that in this embodiment, a viewpoint marker is shown as representing the virtual viewpoint, but a trajectory before and after the viewpoint marker may also be displayed. Furthermore, the camera path selection unit 242 sends information on the selected scene and information on the camera path to the playback count update unit 251.

[0030] 7(a) to 7(c) are examples of camera path selection screens displayed when the number of camera paths containing a selected scene exceeds a predetermined number. For example, the camera path selection unit 242 first displays the camera path selection screen shown in FIG. 7(a) on the display unit 115. The emphasized object marker 701 is a marker for emphasizing and displaying an object associated with the selected scene and included in the field of view of a camera path including the scene. For each object, if the object is included in the field of view of any one of the camera paths, the object marker 601 is changed to the emphasized object marker 701. Whether or not the object is included in the field of view of a camera path can generally be determined using a camera matrix that can be created from viewpoint information. Based on the displayed information, the user selects a desired object by selecting one of the emphasized object markers 701 on the operation unit 116 (for example, by tapping on a touch panel or clicking with a mouse, etc.). Note that the emphasized object marker 701 may be selected by tapping multiple times, etc.

[0031] When the user selects the desired subject, the camera path selection unit 242 returns the markers among the highlighted subject markers 701 that were not selected by the user to the subject markers 601. Next, the camera path selection unit 242 displays the camera path selection screen of FIG. 7(b) on the display unit 115. The viewpoint direction marker 702 is a marker that indicates the direction from which the camera path views the subject. For each camera path that includes the selected subject in its angle of view, a viewpoint direction marker 702 is displayed at the intersection of a half line extending from the subject to the viewpoint and a circle centered on the subject. Based on this displayed information, the user selects one of the viewpoint direction markers 702 on the operation unit 116 (for example, by tapping on a touch panel or clicking with a mouse, etc.) to select the desired direction from which to view the selected subject. Note that if multiple highlighted subject markers 701 are selected by user operation, a marker for a camera path with an angle of view that includes both subjects is displayed. In this case, since there are multiple intersection points of the circumference, the viewpoint direction marker 702 can be displayed by selecting the center position or by taking the average position. The distance to the subject may also be indicated by changing the color or brightness of the viewpoint direction marker 702. For example, it is possible to indicate an approximate sense of distance by making the marker brighter when the subject is closer and darker when the subject is farther, or by using red when the subject is closer and blue when the subject is farther.

[0032] When the user selects the desired direction, the camera path selection unit 242 deletes all viewpoint direction markers 702 and displays the camera path selection screen of FIG. 7( c) on the display unit 115. The emphasized viewpoint marker 703 is a marker that indicates viewpoint information of the camera path corresponding to the selected viewpoint direction marker 702. The emphasized viewpoint marker 703 is a marker that highlights and displays the viewpoint marker 602. The user checks the subject and the viewpoint direction based on this displayed information. After checking, when the emphasized viewpoint marker 703 is selected on the operation unit 116 (for example, by tapping on a touch panel or clicking with a mouse), the corresponding camera path information is sent to the virtual viewpoint image generation unit 222, and a virtual viewpoint image is generated. The camera path selection unit 242 also sends information on the selected scene and camera path information to the playback count update unit 251.

[0033] The viewpoint specification unit 233 specifies the viewpoint of the virtual viewpoint image to be generated in response to user input. If the desired camera path is not available for the desired scene, the user can manually specify the viewpoint (such as the position and orientation of the virtual camera) and generate a virtual viewpoint image. For example, the user can specify the viewpoint using the following operations on the touch panel of the operation unit 116: Pinch in and out to move the viewpoint forward and backward; Swipe left and right with one finger to move the viewpoint left and right; Swipe up and down with one finger to move the viewpoint up and down; Swipe left and right with two fingers to pan the viewpoint; Swipe with two fingers to tilt the viewpoint; and Draw an arc with two fingers to roll the viewpoint. Operations are not limited to these; a device such as a game controller may also be connected. The time code is automatically advanced according to the frame rate. For example, when generating a virtual viewpoint image at 60 frames per second, the start TC of the scene is set as the initial value, and the time code is advanced by one frame every 1 / 60 seconds. The viewpoint and time code designated by the viewpoint designation unit 233 are sent as a camera path to the virtual viewpoint image generation unit 222, which then generates a virtual viewpoint image. This camera path is also sent to the camera path registration unit 252.

[0034] The play count update unit 251 updates the play count of the scene and camera path selected by the user and played back as a virtual viewpoint image. The play count update unit 251 receives information about the selected scene and information about the camera path from the camera path selection unit 242. The play count update unit 251 acquires the play count of the selected scene from the scene information storage unit 231 based on the scene ID of the received scene information. The play count update unit 251 then adds 1 to the acquired play count and registers this as the updated play count in the scene information storage unit 231. The play count update unit 251 also acquires the play count of the selected camera path from the camera path information storage unit 232 based on the camera path ID of the received camera path information. The play count update unit 251 then adds 1 to the acquired play count and registers this as the updated play count in the camera path information storage unit 232.

[0035] The camera path registration unit 252 newly registers the camera path specified by the viewpoint designation unit 233 in the camera path information storage unit 232. The camera path registration unit 252 receives the camera path from the viewpoint designation unit 233 and generates information to be registered as follows. The camera path registration unit 252 acquires a list of camera path IDs from the camera path information storage unit 232 and generates a new ID that is different from those IDs. The camera path registration unit 252 sets the TC of the first frame of the camera path received from the viewpoint designation unit 233 as the start TC, and sets the TC of the last frame of the camera path received from the viewpoint designation unit 233 as the end TC. The camera path registration unit 252 also sets the number of times the camera path has been played to 1 and the creator to the user. The camera path registration unit 252 newly registers the camera path ID, start TC, end TC, number of times played, creator information, and camera path information in the camera path information storage unit 232.

[0036] FIG. 8 is a flowchart illustrating a processing procedure for scene selection in the image processing device 100 according to this embodiment. In step S801, the scene selection unit 241 acquires scene information from the scene information storage unit 231 and displays it in chronological order on a scene selection screen, as shown in an example in Fig. 5. For example, based on the scene information acquired from the scene information storage unit 231, the scene selection unit 241 displays a scene playback count graph 501, play information 502, scene markers 503, inning information 505, etc. on the scene selection screen.

[0037] In step S802, the camera path counting unit 243 counts the number of retained camera paths that include each scene. The camera path counting unit 243 sends the number of retained camera paths for the counted scenes and information about the camera paths that include the scenes to the scene selection unit 241. The scene selection unit 241 receives the number of retained camera paths for the scene and information about the camera paths that include the scenes sent from the camera path counting unit 243.

[0038] In step S803, the scene selection unit 241 generates a scene selection screen including the information on the number of held camera paths 504 for the scene received in step S802, and sends it to the display unit 115 for display. In step S804, the user selects one of the scene markers 503 on the operation unit 116 to select a desired scene.

[0039] In step S805, the scene selection unit 241 sends, for the selected scene, the scene information acquired in step S801, the camera path information including the scene received in step S802, and the number of camera paths held for the scene to the camera path selection unit 242. Then, the processing shown in Fig. 8 ends.

[0040] FIG. 9 is a flowchart illustrating a processing procedure for selecting a camera path in the image processing device 100 according to this embodiment. In step S901, the camera path selection unit 242 receives, for the scene selected in step S804, scene information, information on the camera paths including the scene, and the number of camera paths held for the scene.

[0041] In step S902, the camera path selection unit 242 generates a camera path selection screen and displays it on the display unit 115. The camera path selection unit 242 displays the camera path selection screen including a subject marker 601 that indicates the position of the subject in the scene selected by the user using the operation unit 116.

[0042] In step S903, the camera path selection unit 242 determines whether the number of held camera paths for the selected scene is equal to or less than a predetermined value. If the camera path selection unit 242 determines that the number of held camera paths is equal to or less than the predetermined value (YES in S903), the process proceeds to step S904. On the other hand, if the camera path selection unit 242 determines that the number of held camera paths is not equal to or less than the predetermined value, that is, that the number exceeds the predetermined value (NO in S903), the process proceeds to step S908.

[0043] In step S904, the camera path selection unit 242 displays a camera path selection screen using a viewpoint marker 602 that indicates the angle of view of the camera path that includes the selected scene. In step S904, for example, a camera path selection screen such as that shown in Fig. 6 is displayed. Then, the process proceeds to step S911.

[0044] In step S905, the camera path selection unit 242 changes the display of the subject that is within the angle of view of the camera path that includes the selected scene to a highlighted subject marker 701. The camera path selection unit 242 generates the changed camera path selection screen and displays it on the display unit 115. In step S905, for example, a camera path selection screen such as that shown in FIG. 7(a) is displayed. In step S906, the user selects one of the displayed highlighted subject markers 701 on the operation unit 116 to select a desired subject.

[0045] In step S907, the camera path selection unit 242 generates a camera path selection screen in which the highlighted object marker 701 that was not selected in step S909 is returned to the object marker 601, and displays the generated screen on the display unit 115. In step S908, the camera path selection unit 242 generates a camera path selection screen including a viewpoint direction marker 702 indicating the direction from which the subject is viewed for the camera path that includes the selected subject in the angle of view, and displays the generated screen on the display unit 115. In step S908, for example, a camera path selection screen such as that shown in FIG. 7(b) is displayed. In step S909, the user selects one of the viewpoint direction markers 702 on the operation unit 116 to select a desired direction from which to view the selected subject.

[0046] In step S910, the camera path selection unit 242 generates a camera path selection screen including an emphasized viewpoint marker 703 corresponding to the selected viewpoint direction marker 702, and displays it on the display unit 115. In step S910, for example, a camera path selection screen such as that shown in Fig. 7(c) is displayed. Then, the process proceeds to step S911.

[0047] In step S911, the user selects a desired camera path by selecting one of the viewpoint markers 602 displayed in step S904 or the highlighted viewpoint markers 703 displayed in step S913 on the operation unit 116. The camera path selection unit 242 sends information about the selected camera path to the virtual viewpoint image generation unit 222.

[0048] In step S912, the virtual viewpoint image generation unit 222 generates a virtual viewpoint image for the selected scene and camera path based on the camera path information received in step S911. The generated virtual viewpoint image can be displayed on the display unit 115.

[0049] In step S913, the playback count update unit 251 increments by 1 the number of playbacks of the scene for which the virtual viewpoint image was generated in step S912, and updates the information in the scene information storage unit 231. The playback count update unit 251 also increments by 1 the number of playbacks of the camera path for which the virtual viewpoint image was generated, and updates the information in the camera path information storage unit 232. Then, the processing shown in FIG. 9 ends.

[0050] Fig. 10 shows an example of the screen configuration of a display screen when a smartphone is used as the display unit 115. In Fig. 10, 1001 is a screen for playing back the generated virtual viewpoint image, 1002 is a camera path selection screen, and 1003 is a scene selection screen.

[0051] In the virtual viewpoint image, a camera path can be created from any viewpoint, so many camera paths can be created for one scene. According to this embodiment, even if there are many camera paths for one scene, it is possible to narrow down the choices by subject, direction of viewing the subject, and viewpoint, and it becomes possible for the user to easily select a camera path for a specific scene selected by the user.

[0052] Furthermore, according to this embodiment, by displaying the camera path retention count graph 504 on the scene selection screen, it is possible to easily check whether or not there is a camera path for playing each scene. For example, a user who is not good at viewpoint control using the operation unit 116 can easily select a scene with a camera path. Also, a user who wants to register a new camera path can easily select a scene with no registered camera path. This makes it easy for various users to select a desired scene when generating a virtual viewpoint video.

[0053] (Other embodiments) The virtual viewpoint image generation unit 222 may set the start time of the virtual viewpoint image to be generated as the start TC of the camera path or as the start TC of the scene. Also, the end time of the virtual viewpoint image to be generated may set the end TC of the camera path or as the end TC of the scene. When the time code of the camera path is used, it is possible to generate video as intended by the creator of the camera path, and when the time code of the scene is used, it is possible to generate video that is extracted from only the scene. The creator of the camera path can create a digest camera path that connects multiple scenes.

[0054] For each scene, the selection unit 240 may display a graph showing the movement amount of the subject in the scene when there is no camera path that includes the scene. The movement amount of the subject can be calculated, for example, by summing the movement amount of the position of the 3D model of the subject for each frame between the start TC and end TC of the scene. In scenes where the movement amount of the subject is small, such as pitching, even a user who is not good at viewpoint control can specify the viewpoint relatively easily, and the movement amount of the subject can serve as a guideline for selecting a scene.

[0055] Although the user selects the desired scene by selecting the scene marker 503 on the operation unit 116, it is also possible to select a scene by performing a selection operation on the corresponding part of the scene playback count graph 501 or the camera path retention count graph 504.

[0056] Note that restrictions may be placed on the scenes for which a user can play virtual viewpoint images. For example, a user may be allowed to choose between paid membership and free membership. Paid members may play all scenes. Free members may only play scenes that have a camera path. Alternatively, free members may only play camera paths created by the provider (where the camera path creator information is "system"), and may not play camera paths created and registered by other users. In other words, free members may not play scenes even if the scene play count graph 501 shows a high number of plays. They may not play camera paths created by other users, even if the scene shows that a camera path has been registered based on the camera path storage count graph 504. These restrictions motivate free members to apply for paid membership.

[0057] Note that the creator information of a camera path may not only be "system" and "user," but also information that can identify the creator (such as a nickname). This allows a user to easily select a camera path of a preferred creator, even when many camera paths are registered. For example, the user inputs desired creator information using the operation unit 116. The input creator information is sent to the scene selection unit 241. The scene selection unit 241 sends the received creator information to the camera path counting unit 243 along with the representative TC of the scene. When selecting a camera path including a scene, the camera path counting unit 243 counts only camera paths that match the received creator information and sends them to the scene selection unit 241. As a result, the number of camera paths held and the camera paths displayed on the camera path selection screen are limited to those that match the creator information entered by the user. Furthermore, the user may be able to specify whether or not to register a camera path, and information on whether or not to make the registered camera path public to other users can also be added.

[0058] The 3D model holding unit 221 may hold a 3D model created using 3D production software or the like, instead of a 3D model generated from images of the imaging target captured by a plurality of cameras. In the above-described embodiment, an example has been described in which a game such as a baseball game is captured and a desired scene is reproduced as a virtual viewpoint image, but the application of the image processing device 100 in this embodiment is not limited to this. The image processing device 100 in this embodiment can be applied to any imaging target that enables the generation of a virtual viewpoint video for a specified scene and camera path.

[0059] (Another embodiment of the present invention) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0060] It should be noted that the above-described embodiments are merely examples of specific embodiments for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be carried out in various forms without departing from its technical concept or main features.

[0061] The disclosure of this embodiment includes the following configurations, methods, etc. (Configuration 1) an acquisition means for acquiring an input corresponding to a user operation for selecting a specific scene for generating a virtual viewpoint image from among a plurality of scenes of an imaging target captured by the imaging means; a display control means for controlling the display of information indicating the positions and orientations of a plurality of virtual cameras associated with the specific scene; 1. An image processing device comprising: (Configuration 2) 2. The image processing device according to configuration 1, wherein the information indicating the position and orientation of the virtual camera includes information indicating the position, orientation, and focal length of the virtual camera. (Configuration 3) 3. The image processing device according to claim 1, wherein the display control means controls display of information indicating the position of the subject in the specific scene together with information indicating the position and orientation of the virtual camera. (Configuration 4) The image processing device according to any one of configurations 1 to 3, wherein the display control means controls switching of display of information depending on the number of pieces of information indicating the positions and orientations of a plurality of virtual cameras associated with the specific scene. (Configuration 5) the display control means controls display of information indicating a position of a subject in the specific scene together with information indicating a position and orientation of the virtual camera; 4. The image processing device according to any one of configurations 1 to 3, characterized in that, when the number of pieces of information indicating the positions and orientations of a plurality of virtual cameras associated with the specific scene exceeds a predetermined value, display of information indicating the position of the subject is switched depending on whether the subject is within the angle of view of the virtual camera associated with the specific scene. (Configuration 6) The image processing device according to any one of configurations 1 to 5, wherein the display control means controls display of the number of pieces of information indicating the positions and orientations of a plurality of virtual cameras associated with the scene for each scene. (Configuration 7) 7. The image processing device according to any one of configurations 1 to 6, wherein the display control means controls display of information indicating the number of times the virtual viewpoint image for each scene has been reproduced. (Configuration 8) a first storage means for storing information about a virtual camera for generating the virtual viewpoint image; The image processing device according to any one of configurations 1 to 7, wherein the display control means controls display of information indicating positions and orientations of a plurality of virtual cameras associated with the specific scene based on information about the virtual cameras held by the first holding means. (Configuration 9) 9. The image processing device according to configuration 8, wherein the information about the virtual camera includes information indicating the position and orientation of the virtual camera in the virtual viewpoint image that has already been generated. (Configuration 10) an image acquisition unit that acquires a plurality of captured images of the imaging target captured by the plurality of imaging units; a second storage means for storing a 3D model generated from the plurality of captured images acquired by the image acquisition means; generating means for generating a virtual viewpoint image of the imaging target based on the 3D model held by the second holding means; 10. The image processing device according to any one of configurations 1 to 9, wherein the generating means generates the virtual viewpoint image of the specific scene at a position and orientation of the virtual camera specified based on a user operation. (Method 1) an acquisition step of acquiring an input corresponding to a user operation of selecting a specific scene for generating a virtual viewpoint image from among a plurality of scenes of an imaging target captured by the imaging means; a display control step of controlling the display of information indicating the positions and orientations of a plurality of virtual cameras associated with the specific scene; An image processing method comprising: (Program 1) The computer of the image processing device an acquisition step of acquiring an input corresponding to a user operation of selecting a specific scene for generating a virtual viewpoint image from among a plurality of scenes of an imaging target captured by the imaging means; a display control step of controlling the display of information indicating the positions and orientations of a plurality of virtual cameras associated with the specific scene; A program that executes the following. [Explanation of symbols]

[0062] 210: Capture unit 211: Image acquisition unit 212: 3D model generation unit 220: Generation unit 221: 3D model storage unit 222: Virtual viewpoint image generation unit 230: Designation unit 231: Scene information storage unit 232: Camera path information storage unit 233: Viewpoint designation unit 240: Selection unit 241: Scene selection unit 242: Camera path selection unit 243: Camera path counting unit 250: Registration unit 251: Playback count update unit 252: Camera path registration unit 260: Imaging unit

Claims

1. an acquisition means for acquiring an input corresponding to a user operation for selecting a specific scene for generating a virtual viewpoint image from among a plurality of scenes of an imaging target captured by the imaging means; a display control means for controlling the display of information indicating the positions and orientations of a plurality of virtual cameras associated with the specific scene; 1. An image processing device comprising:

2. The image processing apparatus according to claim 1 , wherein the information indicating the position and orientation of the virtual camera includes information indicating the position, orientation, and focal length of the virtual camera.

3. 2. The image processing apparatus according to claim 1, wherein the display control means controls the display of information indicating the position of a subject in the specific scene together with information indicating the position and orientation of the virtual camera.

4. The image processing device according to claim 1 , wherein the display control means controls switching of the display of information depending on the number of pieces of information indicating the positions and orientations of a plurality of virtual cameras associated with the specific scene.

5. the display control means controls display of information indicating a position of a subject in the specific scene together with information indicating a position and orientation of the virtual camera; 2. The image processing device according to claim 1, wherein when the number of pieces of information indicating the positions and orientations of a plurality of virtual cameras associated with the specific scene exceeds a predetermined value, the display of the information indicating the position of the subject is switched depending on whether the subject is within the angle of view of the virtual camera associated with the specific scene.

6. 2. The image processing apparatus according to claim 1, wherein the display control means controls display of the number of pieces of information indicating the positions and orientations of a plurality of virtual cameras associated with the scene for each of the scenes.

7. 2. The image processing apparatus according to claim 1, wherein the display control means controls display of information indicating the number of times the virtual viewpoint image for each scene has been reproduced.

8. a first storage means for storing information about a virtual camera for generating the virtual viewpoint image; 2. The image processing device according to claim 1, wherein the display control means controls display of information indicating positions and orientations of a plurality of virtual cameras associated with the specific scene based on information about the virtual cameras held by the first holding means.

9. 9. The image processing apparatus according to claim 8, wherein the information about the virtual camera includes information indicating the position and orientation of the virtual camera in the virtual viewpoint image that has already been generated.

10. an image acquisition unit that acquires a plurality of captured images of the imaging target captured by the plurality of imaging units; a second storage means for storing a 3D model generated from the plurality of captured images acquired by the image acquisition means; a generating means for generating a virtual viewpoint image of the imaging target based on the 3D model held by the second holding means, The image processing apparatus according to claim 1 , wherein the generating means generates the virtual viewpoint image of the specific scene at a position and orientation of the virtual camera specified based on a user operation.

11. an acquisition step of acquiring an input corresponding to a user operation of selecting a specific scene for generating a virtual viewpoint image from among a plurality of scenes of an imaging target captured by the imaging means; a display control step of controlling the display of information indicating the positions and orientations of a plurality of virtual cameras associated with the specific scene; An image processing method comprising:

12. The computer of the image processing device an acquiring step of acquiring an input corresponding to a user operation of selecting a specific scene for generating a virtual viewpoint image from among a plurality of scenes of an imaging target captured by the imaging means; a display control step of controlling the display of information indicating the positions and orientations of a plurality of virtual cameras associated with the specific scene; A program that executes the following.

Images