Display apparatus, control method, and program

Abstract

To reduce the effort of staff for a virtual viewpoint image even when a large number of users sequentially view the virtual viewpoint image on a single device at an exhibition, for example.SOLUTION: A display apparatus includes: display means for displaying a virtual viewpoint image generated using three-dimensional shape data and virtual viewpoint information from a multi-viewpoint image; control means for controlling the virtual viewpoint information; and initializing means which initializes timecode of the virtual viewpoint image to a predetermined start timecode when the display means terminates displaying the image. The display means reproduces the virtual viewpoint image from the predetermined start timecode to a predetermined end timecode. The control means controls the virtual viewpoint information differently when the display means reproduces the virtual viewpoint image and when the initializing means performs the initialization.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a display device that displays a virtual viewpoint image. [Background technology]

[0002] In recent years, attention has been focused on a virtual viewpoint image generation technique that can generate an image seen from an arbitrary viewpoint from a plurality of images captured by a plurality of cameras.

[0003] Any viewpoint used to generate a virtual viewpoint image is called a virtual viewpoint (virtual camera). Unlike a physical camera, a virtual camera can move and rotate in a virtual three-dimensional space without being subject to physical constraints.

[0004] Patent Document 1 discloses a method for controlling various operations of a virtual camera in a configuration in which a virtual viewpoint image is displayed on an image display device equipped with a touch panel, depending on the number of fingers used to tap on the touch panel. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6419278 Summary of the Invention [Problem to be solved by the invention]

[0006] In Patent Document 1, each user has their own device, and each user intuitively operates the virtual camera and time code to preview the virtual viewpoint image. This allows even users who are not familiar with virtual viewpoint images to view the virtual viewpoint images when they are exhibited at an exhibition. For example, an exhibition may allow users to experience viewing a virtual viewpoint image in which a certain scene is repeated, and to enjoy the same scene from different viewpoints. In such an exhibition, users can freely move the virtual camera while viewing.

[0007] However, with the technology described in Patent Document 1, for example at an exhibition, many users would take turns viewing content on one device, and after each user had viewed the content, it would be time-consuming for staff to return the virtual camera of the virtual viewpoint image to its designated initial position. [Means for solving the problem]

[0008] The display device has a display means for displaying a virtual viewpoint image generated using three-dimensional shape data generated from a multi-viewpoint image and virtual viewpoint information, a control means for controlling the virtual viewpoint information, and an initialization means for initializing a time code of the virtual viewpoint image to the predetermined start time code after playback by the playback means is completed, wherein the display means plays back the virtual viewpoint image from the predetermined start time code to a predetermined end time code, and the control means executes different controls on the virtual viewpoint information between the control performed when the virtual viewpoint image is played back by the display means and the control performed when initialization is performed by the initialization means. Effect of the Invention

[0009] For example, even in the case where a large number of users take turns viewing virtual viewpoint images on one device at an exhibition or the like, the work required by staff to view the virtual viewpoint images can be reduced. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a virtual viewpoint image generating system. [Diagram 2] FIG. 1 is a diagram illustrating an example of a configuration of an image display device. [Diagram 3] FIG. 2 is a diagram for explaining the position, orientation, and point of interest of a virtual camera. [Figure 4] 1 is an example of an operation screen of an image display device. [Diagram 5] FIG. 2 illustrates an example of a database configuration. [Figure 6] 6 is a flowchart showing an example of a virtual viewpoint image reproduction process in the first embodiment. [Figure 7] 4 is a diagram showing an example of a display of a virtual viewpoint image in the first embodiment. [Figure 8] 11 is a flowchart showing an example of a reproduction process of a virtual viewpoint image in the second embodiment. [Figure 9] 13 is a diagram showing an example of a display of a virtual viewpoint image in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment described shows an example of a specific implementation of the present invention, and the present invention is not limited to this embodiment.

[0012] [Embodiment 1] In this embodiment, a description will be given of a system for reproducing virtual viewpoint images, particularly with regard to a reproduction process of virtual viewpoint images that is applied to exhibitions and the like.

[0013] The entire virtual viewpoint image generating system is described in Fig. 1, and the image display device 104 therein is described in Fig. 2. This image display device 104 performs the virtual viewpoint image reproduction process, which is a feature of the present invention. The reproduction process is described using Fig. 6, and an example of the reproduction is described using Fig. 7.

[0014] 3 to 5, a virtual camera operation method and a database required for explaining the reproduction process of a virtual viewpoint image will be described.

[0015] (Configuration of the virtual viewpoint image generation system) FIG. 1 shows the configuration of a virtual viewpoint image generating system 100 according to the present invention.

[0016] 1(a) is a block diagram of a configuration example of a virtual viewpoint image generation system 100. The virtual viewpoint image generation system 100 has n sensor systems, from sensor system 101a to sensor system 101n, and each sensor system has at least one camera, which is an image capturing device. Hereinafter, unless otherwise specified, the n sensor systems will not be distinguished from one another and will be referred to as a multiple sensor system 101.

[0017] 1(b) is a diagram showing an example of installation of a multi-sensor system 101. The multi-sensor systems 101 are installed so as to surround an area 120 that is a target area to be photographed, and photograph the area 120 from different directions.

[0018] 1, the area 120 to be photographed is the field of a stadium where a sporting event such as rugby is played, and n (e.g., 100) sensor systems 101 are installed to surround the field. However, the number of installed sensor systems 101 is not limited. Furthermore, the area 120 to be photographed is not limited to the field of a stadium. The area 120 may include stadium seating, or the area 120 may be an indoor studio, stage, or the like.

[0019] Furthermore, the multiple sensor system 101 does not have to be installed all around the area 120, and may be installed only in a part of the periphery of the area 120 due to restrictions on installation space, etc. Furthermore, the cameras possessed by the multiple sensor system 101 may include imaging devices with different functions, such as a telephoto camera or a wide-angle camera.

[0020] The multi-sensor system 101 captures images using multiple cameras in a synchronized manner, and the multiple images obtained by capturing images are called multi-viewpoint images.

[0021] Each of the multi-viewpoint images in this embodiment may be a captured image, or may be an image obtained by performing image processing, such as a process of extracting a predetermined area, on the captured image.

[0022] Each sensor system 101 may have a microphone (not shown) in addition to a camera. The microphones of the multiple sensor systems 101 each pick up sound. These picked up sound signals are time-synchronized and recorded. Based on the picked up sound, an acoustic signal can be generated that is played back together with the display of an image on the image display device 104. In the following, for the sake of simplicity, a description of sound will be omitted, but it is assumed that images and sound are basically processed in parallel.

[0023] Moreover, each sensor system 101 captures images during the target event, and a 3D model is generated using the multi-viewpoint images. For example, if the target event is a rugby match, 3D models of the players and referees, which are the subjects, are generated and recorded from the start of the match (kickoff) to the end of the match (no side).

[0024] The image recording device 102 acquires a three-dimensional model generated using the multi-viewpoint images acquired from the multiple sensor system 101, and stores the model together with the time code used for shooting in a database 103. The configuration of this database will be described later with reference to FIG.

[0025] A time code (hereinafter, also referred to as TC) is information for uniquely identifying the time of shooting, and is expressed in the format of, for example, "day:hour:minute:second.frame number." Here, in this embodiment, a 3D model of the subject is generated for every TC during shooting.

[0026] The image display device 104 receives the multi-viewpoint images for each TC from the database 103 and the user's operation on the virtual camera, and generates a virtual viewpoint image.

[0027] The virtual camera is set in a virtual space associated with the area 120, and can view the area 120 from a viewpoint different from any of the cameras of the multiple sensor systems 101. The viewpoint defined by the position, attitude, etc. of this virtual camera is referred to as a virtual viewpoint. Data including the position, attitude, etc. of the virtual camera is an example of virtual viewpoint information. Details of the virtual camera 110 and its operation will be described later in FIG. 3, and the operation screen will be described later in FIG. 4.

[0028] The image display device 104 executes a virtual viewpoint image playback process and generates a virtual viewpoint image using a three-dimensional model and a virtual camera specified by a time code. The playback process will be described in detail later with reference to FIG. 6, and an example of playback will be described in detail later with reference to FIG.

[0029] The virtual viewpoint image generated by the image display device 104 is an image that represents the view from the virtual camera 110. The virtual viewpoint image in this embodiment is also called a free viewpoint image. The virtual viewpoint image is displayed on a touch panel, a liquid crystal display, or the like of the image display device 104.

[0030] The configuration of the virtual viewpoint image generation system 100 is not limited to that shown in Fig. 1. The image display device 104 may be configured to be separate from the operation device and the display device. Alternatively, a plurality of display devices may be connected to the image display device 104, and a virtual viewpoint image may be output to each of them.

[0031] The database 103 and the image display device 104 may be integrated. Also, only important scenes may be copied in advance from the database 103 to the image display device 104. Depending on the configuration, it may be possible to switch between allowing access to all TCs in a match or only to some TCs.

[0032] (Functional configuration of image display device) The configuration of the image display device 104 will be described with reference to FIG.

[0033] 2A is a block diagram showing an example of the functional configuration of the image display device 104. The playback control unit 203 mainly plays a role in executing the playback process of the virtual viewpoint image, which is a feature of the present invention.

[0034] Each block will be described in turn.

[0035] The model generation unit 201 specifies the TC and generates a three-dimensional model (three-dimensional shape data) representing the three-dimensional shape of the subject from the multi-viewpoint images acquired from the database 103. First, a foreground image in which a foreground region such as a person or a ball included in the subject is extracted from the multi-viewpoint images, and a background image in which a background region other than the foreground is extracted are acquired. Then, the model generation unit 201 generates a three-dimensional model of the foreground using the acquired multiple foreground images. For example, such processing is executed by a shape estimation method such as Visual Hull, and the three-dimensional model is acquired as a point cloud. Note that the format of the three-dimensional model representing the shape of the object is not limited to this.

[0036] The model generation unit 201 may be configured to be included in the image recording device 102 instead of the image display device 104. In that case, the three-dimensional model generated by the image recording device 102 is recorded in the database 103, and the image display device 104 is configured to read out the three-dimensional model from the database 103.

[0037] Incidentally, for the 3D model of the foreground, the coordinates of each object such as a ball or a person may be calculated and stored in the database 103. Incidentally, for the 3D model of the background, the coordinates may be acquired in advance by an external device.

[0038] The virtual camera control unit 202 acquires input information of a user's operation on the virtual camera 110 and controls the position and attitude of the virtual camera. A touch panel, joystick, or the like is used as an operation device for the virtual camera. Note that the operation device is not limited to these. The virtual camera control unit 202 can also control the position and attitude of the virtual camera in accordance with processing of the virtual viewpoint image. The position and attitude of the virtual camera will be described later in FIG. 3, and the operation screen will be described later in FIG. 4.

[0039] The image generator 203 generates a virtual viewpoint image using the position and orientation of the virtual camera and the 3D model. Specifically, for each point constituting the 3D model, appropriate pixel values ​​are acquired from the multi-viewpoint image and coloring processing is performed. The colored 3D model is then placed in a 3D virtual space, projected onto the virtual viewpoint, and rendered to generate a virtual viewpoint image.

[0040] However, the method of generating the virtual viewpoint image is not limited to this, and various methods can be used, such as a method of generating a virtual viewpoint image by projective transformation of a captured image without using a three-dimensional model.

[0041] The initialization unit 205 manages the initial values ​​of the position and orientation of the virtual camera, a predetermined start time code (start TC), and a predetermined end time code (end TC), and sets the initial values ​​for each of them using them. The process of setting an initial value for each is called initialization of each. The position and orientation of the virtual camera will be described later with reference to FIG. 3. The predetermined start TC and end TC will be described later with reference to FIG. 5. Note that the number of start and end TCs managed is not limited to one set, and multiple sets may be managed.

[0042] The playback control unit 204 uses the above-described processing units to play back the virtual viewpoint image as a video. The playback process of the virtual viewpoint image is described later with reference to a flowchart in FIG. 6, and an example of playback is described later with reference to FIG.

[0043] (Hardware configuration of image display device) Next, a description will be given of the hardware configuration of the image display device 104. FIG.

[0044] A CPU (Central Processing Unit) 211 performs processing using programs and data stored in a RAM (Random Access Memory) 212 and a ROM (Read Only Memory) 213 .

[0045] The CPU 211 controls the overall operation of the image display device 104 and executes processes for realizing the functions shown in Fig. 2(a). Note that the image display device 104 may have one or more pieces of dedicated hardware different from the CPU 211, and at least a part of the processes performed by the CPU 211 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

[0046] The ROM 213 holds programs and data. The RAM 212 has a work area for temporarily storing programs and data read from the ROM 213. The RAM 212 also provides a work area used by the CPU 211 when it executes each process.

[0047] The operation input unit 214 acquires information operated by the user, for example, on a touch panel. In this proposal, the operation input unit 214 accepts operations on a virtual camera, a time code, and the like. The operation input unit 214 may be connected to an external controller and accept input information related to the operation. The external controller is, for example, a three-axis controller such as a joystick, a mouse, or the like. The external controller is not limited to these.

[0048] The display unit 215 displays the generated virtual viewpoint image using a touch panel, a screen, etc. In the case of a touch panel, the operation input unit 214 and the display unit 215 are integrated into one unit.

[0049] The external interface 216 transmits and receives information to and from the database 103 via, for example, a LAN or the like. It may also transmit information to an external screen via an image output port such as HDMI (registered trademark) or SDI. It may also transmit image data via Ethernet or the like.

[0050] (Virtual camera position and orientation) The virtual camera 110 (or the virtual viewpoint) will be described with reference to FIG.

[0051] First, an arbitrary coordinate system is used to specify the position and orientation of virtual camera 110. In this embodiment, a general three-dimensional Cartesian coordinate system consisting of X-axis, Y-axis, and Z-axis shown in FIG.

[0052] This coordinate system is set for the subject and used. The subject is a stadium field, a studio, etc. As shown in Fig. 3(b), the subject includes the entire stadium field 391, as well as a ball 392 and players 393 on the field. The subject may also include spectator seats around the field.

[0053] The coordinate system for the subject is set with the center of field 391 as the origin (0,0,0). The X axis is the long side direction of field 391, the Y axis is the short side direction of field 391, and the Z axis is the vertical direction of the field. Note that the method for setting the coordinate system is not limited to this.

[0054] Next, the virtual camera will be described with reference to FIGS.

[0055] 3(c), the vertices of the pyramid represent the virtual camera's position 301, and the vectors extending from the vertices represent the virtual camera's orientation 302. The virtual camera's position is expressed by coordinates (x, y, z) in three-dimensional space, and the orientation i is a unit vector whose components along each axis are scalars.

[0056] The orientation 302 of the virtual camera passes through the center point of the front clip plane 303 and the rear clip plane 304. Moreover, the space 305 sandwiched between the front clip plane 303 and the rear clip plane 304 is called the view frustum of the virtual camera, and is the range in which the image generator 203 generates a virtual viewpoint image (or the range in which the virtual viewpoint image is projected and displayed; hereinafter, this is referred to as the display area of ​​the virtual viewpoint image).

[0057] The virtual camera pose 302 is represented by a vector, which is also called the optical axis vector of the virtual camera.

[0058] The initial values ​​of the virtual camera used in this proposal refer to the initial values ​​of the position, orientation, etc. of the virtual camera. The initial value of the position indicates a specific point in three-dimensional space, and the initial value of the orientation is a specific unit vector in three-dimensional space; for example, the position is set to (x1, y1, z1), the orientation i=(x2, y2, z2), etc. Note that the initial values ​​of the virtual camera may include the focal length in addition to the position and orientation.

[0059] By setting the initial values ​​of the virtual camera, the virtual viewpoint image generated using the virtual camera generates a predetermined area of ​​the subject, and the same virtual viewpoint image can be obtained every time the initial values ​​are set. A specific example of the virtual viewpoint image when the initial values ​​of the virtual camera are set will be described later with reference to FIG.

[0060] The movement and rotation of the virtual camera will be described with reference to FIG. 3(d). The virtual camera moves and rotates in a space expressed by three-dimensional coordinates. The movement 306 of the virtual camera indicates the movement of the position 301 of the virtual camera. The components of the X-axis, Y-axis, and Z-axis of the position 301 are expressed by x, y, and z, respectively. The rotation 307 of the virtual camera is expressed by Euler angles consisting of three angles, namely, Yaw, which is a rotation around the Z-axis, Pitch, which is a rotation around the X-axis, and Roll, which is a rotation around the Y-axis, as shown in FIG. 3(a). Hereinafter, these three rotation angles may be collectively described as, for example, (yaw_1, pitch_1, roll_1). Each of the three angles is expressed in radians or normal angles. The expression of the rotation is not limited to Euler angles, and quaternions, etc. may be used.

[0061] This allows the virtual camera to move and rotate freely in the three-dimensional space of the subject (field), and any area of ​​the subject can be generated as a virtual viewpoint image.

[0062] (Operation screen) Using FIG. 4, the operation method for the virtual camera and the operation method for the time code will be described.

[0063] Fig. 4 shows an operation screen 401 of a tablet in which the operation input unit 214 and the display unit 215 are one and the same in the image display device 104. As shown in Fig. 4, the operation screen 401 is broadly composed of two areas: a virtual camera operation area 402 and a time code display area 403. The virtual camera operation area 402 accepts operations for the virtual camera, and the time code display area 403 displays the TC.

[0064] Since the method of operating the virtual camera is well known, details will be omitted. For example, if the operation input unit 214 is a touch panel, operations for the position and orientation of the virtual camera described in Fig. 3 are assigned according to the number of fingers used to touch the virtual camera operation area 402, and each can be freely operated.

[0065] Next, we will explain the time code display area 403. The time code display area 403 is made up of elements 412 to 415 and 422 to 423 for checking and manipulating the time code.

[0066] The main slider 412 displays the progress of all the time codes of the shooting data.

[0067] The sub-slider 413 can enlarge and display a portion of the total time code, allowing for more detailed time code display. The main slider 412 and the sub-slider 413 have the same length on the screen, but the width of the time code they display is different. For example, the main slider 412 can display three hours, which is the length of one match, whereas the sub-slider 413 can display only a portion of that, such as 30 seconds. In other words, the scales of the sliders are different, and the sub-slider allows for more detailed TC display, such as in frame units.

[0068] The TC displayed using the knob 422 of the main slider 412 and the TC displayed using the knob 423 of the sub-slider 413 may be displayed numerically in the format of "day:hour:minute:second.frame number".

[0069] It is not necessary to always display the sub-slider 413. The time code section that can be displayed by the sub-slider 413 may be variable.

[0070] The playback button 414 is used to start or end the playback process, and provides both instructions as a toggle button. The playback start instruction and end instruction will be described later with reference to FIG.

[0071] The initialization setting button 415 is a button for making settings related to the initialization of the virtual camera.

[0072] The operation screen may have areas other than the virtual camera operation area 402 and the time code display area 403. For example, match information may be displayed in area 411 at the top of the screen. The match information may include the venue, date and time, match cards, and score status.

[0073] Also, an exceptional operation may be assigned to the area 411 at the top of the screen. For example, when the area 411 at the top of the screen receives a double tap, the operation may be to move the position and orientation of the virtual camera to a position where the entire subject can be viewed from above. Operating the virtual camera may be difficult for an inexperienced user, and the user may not know where he or she is. In such a case, an operation may be assigned to return to a bird's-eye view point where the position and orientation are easy for the user to understand. The bird's-eye view image is a virtual viewpoint image in which the subject is viewed down from the Z axis as shown in FIG. 3(b).

[0074] Note that the configuration is not limited to these as long as the virtual camera or the time code can be operated, and the virtual camera operation area 402 and the time code display area 403 do not have to be separated.

[0075] Although the operation input unit 214 and the display unit 215 are described as being the tablet 400, the operation and display devices are not limited to this.

[0076] (Database) The table configuration of database 103 that stores three-dimensional models generated from multi-viewpoint images of a subject will be described with reference to FIG.

[0077] In the table configuration of FIG. 5(a), all time codes (TC) of the events to be filmed are listed on the vertical axis, and 3D model information for each object is stored on the horizontal axis.

[0078] The contents of the three-dimensional model information will be described with reference to Fig. 5(b). The three-dimensional model information mainly consists of the coordinates of each point of the point cloud constituting the three-dimensional model, and a texture image to be added to the three-dimensional model. The three-dimensional model information is not limited to these, and may hold average coordinates, center of gravity coordinates, and maximum and minimum values ​​of each axis in the three-dimensional coordinates as representative values ​​from the coordinates of all points of the point cloud. The three-dimensional model may be stored as one piece of three-dimensional data for each TC, without being divided for each object.

[0079] The example in Figure 5(a) shows a single professional sporting event such as rugby or basketball, with the TC for one game recorded from 17:58:00.000 to 20:20.12.059, which is about two and a half hours. The frame rate is 59.94Hz, and about 550,000 frames of 3D models are recorded.

[0080] By using such a database, when a TC is specified, the image display device 104 can obtain 3D model information for the TC. By using the coordinates of all points for each object contained therein and texture images, the image display device can generate a virtual viewpoint image of the TC, as described in FIG.

[0081] The image display device 104 can use the entire TC for the two and a half hours, but can also select and use only a portion of the TC. For example, if a scoring scene is included in about 30 seconds from 19:01:02.034 to 19:01:31.020, then a virtual viewpoint image of only the scoring scene can be played by using the frames between those times as the start and end TCs. The start and end TCs are data that can be freely set by the provider of the virtual viewpoint image, etc.

[0082] In this proposal, it is assumed that only the scoring scene as described above will be provided as a demo to a large number of users at an exhibition or the like, and the above-mentioned examples of start TC and end TC will be used in the explanation of the playback process and playback example of the virtual viewpoint image in Figures 6 and 7. Note that the start TC and end TC are not limited to the above, and may be included in all TCs. Also, multiple start TCs and end TCs may be set.

[0083] (Reproduction processing of virtual viewpoint images) The virtual viewpoint image reproduction process, which is a feature of the present invention, will be described with reference to the flowchart of FIG.

[0084] This process is executed by the playback control unit 205 in the configuration diagram of Fig. 2 via other processing units. The playback control unit 205 executes the loop process of steps S601 to S610 in units of frames of the time code. For example, if the frame rate of the virtual viewpoint image is 59.94 FPS, one loop (one frame) is processed at intervals of approximately 16.6 [ms].

[0085] It should be noted that the frame rate may be achieved by setting the update rate of the display that displays the images of the image display device 104 to 59.94 FPS and performing processing in synchronization with that.

[0086] In this process, the start TC and end TC to be played back are set in advance. For example, as shown in the database example of FIG. 5, a start TC of 19:01:02.034 and an end TC of 19:01:31.020 are set in advance as a scoring scene in professional sports such as rugby or basketball. Then, about 30 seconds between them are used for the playback process. Note that the start TC and end TC are not limited to these, and any value can be set as long as it is a TC recorded in the database, and other scoring scenes, etc. may also be set.

[0087] Steps S601 to S610 will be explained in order.

[0088] In step S601, the playback control unit 203 acquires a three-dimensional model of the designated TC being processed from the database 103 via the model generation unit 201. As explained in Fig. 5, the database manages a three-dimensional model for each TC.

[0089] In step S602, the playback control unit 203 accepts a user operation of the virtual camera via the virtual camera control unit 202, and changes the position and orientation of the virtual camera. As described in Fig. 3, the user can operate the position and orientation of the virtual camera for each TC.

[0090] In step S603, the playback control unit 203 uses the 3D model acquired in the previous step and the virtual camera operated in the previous step to draw a virtual viewpoint image via the drawing unit 203. The method of drawing the virtual viewpoint image has been described in Fig. 2. An example of drawing the virtual viewpoint image is shown in the playback example in Fig. 7.

[0091] In step S604, the playback control unit 203 judges whether the TC being processed is the end TC. In this example, it judges whether it is 19:01:31.020. If the judgment result is false (N), the process proceeds to step S605, and the user continues previewing. If the judgment result is true (Y), the process proceeds to step S607, and a message is displayed on the screen indicating that playback is stopped (stopped state). At this point, previewing for one user has ended. This branching process will be specifically explained with reference to the playback display example in FIG. 7.

[0092] In step S605, the playback control unit 203 counts up the TC. The TC is counted up in units of frames, and the TC is counted up by one frame each time the loop process of S601 to S609 is performed.

[0093] In step S606, the playback control unit 203 determines whether there is an instruction to end playback. The instruction to end playback can be determined by toggling the playback button 414 shown in FIG. 4. If the determination result is false (NO), the process returns to step S601 and playback processing continues. If the determination result is true (YES), playback processing ends.

[0094] In step S607, the playback control unit 203 displays a message that playback is stopped on the display unit of the image display device 104. If an operation of the virtual camera is input while the message "playback stopped" is being displayed, the operation is invalidated. This allows the user to understand that his or her operation time has ended. An example of the display during the stop state will be described later with reference to FIG. 7(d).

[0095] In step S608, the playback control unit 203 sets the start TC to the current TC. In this example, 19:01:02.034 is set.

[0096] In step S609, the playback control unit 203 initializes the virtual camera. Initializing the virtual camera means setting initial values ​​for the position and orientation of the virtual camera, and has been described with reference to Fig. 3. A specific example of a virtual viewpoint image after the virtual camera has been initialized will be described later with reference to Fig. 7.

[0097] In step S610, the playback control unit 203 determines whether a playback start instruction has been issued. The playback start instruction may be determined by toggling the playback button 414 shown in FIG. 4. If the determination result is true (YES), the process returns to step S601 and playback processing begins. If the determination result is false (NO), the process returns to S610 and waits for a playback start instruction.

[0098] An example of reproducing a virtual viewpoint image using the above flowchart will be described with reference to FIG.

[0099] (Example of virtual viewpoint image playback) A display example in the case where the reproduction process of the virtual viewpoint image in FIG. 6 is used will be described with reference to FIG.

[0100] Take a scene from a rugby match as an example. As explained in Fig. 5, the entire rugby match is filmed, and it is assumed that a scene from the match in which a goal is scored will be used for a demonstration at an exhibition or the like.

[0101] The starting TC of the scoring scene is set to 19:10:02.034 and the ending TC is set to 19:10:31.020, and at the exhibition, each user will be able to preview the TC in between.

[0102] First, a display example in which user A is operating and listening to the content will be described using Fig. 7(a)-(c). After that, the device is handed over to user B, who was next in line. A display example in which user B is operating and listening to the content will be described using Fig. 7(e)-(g). Note that in order to explain the process when users are changed, two users will be taken as an example for explanation, but naturally, this playback process is not limited to two users, and the same playback process can be repeatedly applied regardless of how many users are in line.

[0103] 7(a) to (g) show virtual viewpoint images displayed on the operation screen 401 of the image display device 104 explained in FIG.

[0104] Fig. 7(a) shows a virtual viewpoint image at the start of user A's operation, with the position and orientation of the virtual camera set to initial values ​​at the start TC. In the flowchart of Fig. 6, the start TC is set in step S607, and the position and orientation of the virtual camera are initialized in step S608. In Fig. 7(a), the initial values ​​are the position and orientation of the virtual camera, which is positioned outside the long side of the field and has an orientation slightly to the right, viewing the subject.

[0105] After Fig. 7(a), user A can freely operate the virtual camera and preview the virtual viewpoint image between the start TC and the end TC. These are the processes of steps S602 and S603. As an example, Fig. 7(b) will be described.

[0106] 7(b) is a virtual viewpoint image of a scene where an offload pass (a pass made just before a player falls down) is made. In this figure, user A is operating the virtual camera so as to view the subject from above (Z direction).

[0107] In FIG. 7(b), the TC is 19:10:20.014 as an example, but between the start TC and end TC, the position and orientation of the virtual camera can be freely moved to view from various angles.

[0108] Fig. 7(c) is a virtual viewpoint image of the end TC. Fig. 7(c) shows a scene in which a try is scored in rugby, and user A operates the virtual camera so that the subject is viewed slightly to the left of the long side of the field. If the end TC is determined in step S604, step S605 is not executed, so that TC is not counted up any further, and in step S606 the screen switches to the screen shown in Fig. 7(d) showing "stopped".

[0109] In the example of the pause screen in FIG. 7(d), if an operation is input to the virtual camera while playback stop is being displayed, the operation is invalidated. This allows the user to understand that his / her operation time has ended, and he / she can hand the device over to the next user B or return it to its original position. Note that the pause screen is not limited to this, and it may be anything that indicates that playback has been stopped or that playback instructions are awaited. Also, such a message may be displayed on the screen.

[0110] Next, the operation of user B will be described.

[0111] FIG. 7(e) shows the virtual viewpoint image at the start of operation by user B, with the position and orientation of the virtual camera set to the initial values ​​at the start TC. In the flowchart of FIG. 6, the start TC is set in step S607, and the position and orientation of the virtual camera are initialized in step S608. This is the same screen as FIG. 7(a), and it can be seen that the operation state of the virtual camera at the end of the previous user's operation in FIG. 7(c) has been reset. The processing of steps S607 and S608 shows that it is possible to provide the same state for each user at the start of operation at an exhibition, etc.

[0112] Similarly, after Fig. 7(e), user B can freely operate the virtual camera and preview the virtual viewpoint image between the start TC and the end TC. The operation during this period differs for each user, and Fig. 7(f) will be explained as an example.

[0113] FIG. 7(f) has the same TC as FIG. 7(b), 19:10:20.014, but the virtual camera is positioned outside the long side of the field and oriented slightly to the left to view the subject, creating a virtual viewpoint image.

[0114] This is the position and orientation of the virtual camera, which is different from that of Figure 7(b) for the same time code. In this playback process, the start TC and end TC are common to each user, but the virtual viewpoint between them can be freely operated by each user.

[0115] Figure 7(g) is a virtual viewpoint image at the end TC. It is the same TC as Figure 7(c), but the virtual camera is positioned outside the long side of the field on the opposite side from Figure 7(c), and the direction is slightly to the left.

[0116] In the figure, user B's preview ends, and just like the previous user, the TC is no longer counted up, and the screen display switches to the stopped screen display shown in Fig. 7(d). The subsequent processes are repeated as described above, and can be applied repeatedly even when many users visit an exhibition.

[0117] By using the above process, when many users take turns viewing virtual viewpoint images on one device at an exhibition, etc., the queue can be smoothly advanced and users can operate the appropriate virtual camera, allowing an efficient and satisfying demonstration to be held. In addition, the labor required for staff to initialize the virtual viewpoint images can be reduced.

[0118] In this embodiment, the main slider 412 displays the progress of all the time codes of the shooting data, and the sub-slider 413 displays an enlarged portion of all the time codes, but further improvements may be made to suit the exhibition. For example, the main slider 412 allows the user to select only a plurality of viewable scenes. In this case, the main slider 412 displays brightly so that it is clear that only viewable scenes are selectable among all the time code sections, and displays darkly the other sections so that it is clear that they are not selectable. In addition, the left end of the sub-slider 413 is the start TC of the viewable scene, and the right end is the end TC of the viewable scene. This allows the user to intuitively select viewable scenes, and further makes it easier to select a scene that the user particularly wants to view among those scenes.

[0119] [Embodiment 2] In this embodiment, a case will be described in which the replay process of the virtual viewpoint image shown in the first embodiment includes a repeat process. The repeat process is a process in which one user replays a scene from a predetermined start TC to an end TC multiple times, and is also called repeat playback in this embodiment. Such repeat playback is preferable for the experience of the virtual viewpoint image. This is because it can provide the experience of viewing the same scene from different angles, so that the user can easily experience that the virtual viewpoint image can be viewed from a preferred viewpoint regardless of the position of the real camera.

[0120] The virtual viewpoint image generating system in Fig. 1 and the image display device 104 in Fig. 2 are the same as those described in the first embodiment, and only the virtual viewpoint image playback process including the repeat process will be added and explained. The virtual viewpoint image playback process of this embodiment is explained in Fig. 8, and an example of the playback is explained in Fig. 9.

[0121] (Reproduction processing of virtual viewpoint images) The virtual viewpoint image reproduction process in this embodiment will be described with reference to the flowchart in FIG.

[0122] This process is executed by the playback control unit 205 in the configuration diagram of Fig. 2 via other processing units. The start TC and end TC are the same as those in the first embodiment. Also, the process related to the frame rate is omitted because it is the same as in the first embodiment.

[0123] The flowchart in Fig. 8 is made up of steps S801 to S815, but since S801 to S803 and S805 to S806 are similar to steps S601 to S603 and S605 to S606 in the flowchart in Fig. 6 of the first embodiment, their explanations will be omitted. In addition, steps S813 to S815 are similar to steps S608 to S610, and their explanations will be omitted.

[0124] Here, steps S804 and S807 to S815 will be described in order.

[0125] In step S804, the playback control unit 203 judges whether the TC being processed is the end TC. If the judgment result is false (NO), the process proceeds to step S805, where the user continues viewing without changing the position and orientation of the virtual camera. If the judgment result is true (YES), the process proceeds to step S807, where the number of repeats for the user is judged.

[0126] In step S807, the playback control unit 203 counts up the repeat number.

[0127] In step S808, the playback control unit 203 determines whether the number of repeats has reached the upper limit. If the result of the determination is false (NO), the process proceeds to step S809, and if the result of the determination is true (YES), the process proceeds to step S811.

[0128] In step S809, the playback control unit 203 sets the start TC to the current TC. In this example, 19:01:02.034 is set.

[0129] In step S810, the playback control unit 203 executes a repeat process of the virtual camera. The repeat process of the virtual camera is a process for controlling whether to continue the position and attitude of the virtual camera or to set the initial value according to the state of the initialization setting button 415 shown on the operation screen 401 in Fig. 4. The initialization setting button 415 is a toggle button, and sets either continuation or initialization. In this embodiment, the case where the initialization setting button 415 is set to continue will be mainly described.

[0130] If the initialization setting button 415 is set to continue, in step S810, the start TC is set in the previous step, but the virtual camera is not initialized and the position and orientation of the virtual camera are continued. A specific example of the case where the position and orientation of the virtual camera are continued will be described later.

[0131] In step S811, the playback control unit 203 displays a message indicating that playback is stopped on the display unit of the image display device 104. The display example is similar to the content explained in Fig. 7(d) of the first embodiment, and when this process is applied to an exhibition or the like, the operation is switched to the next user in this step.

[0132] In step S812, the playback control unit 203 resets the repeat count to zero.

[0133] When this process is applied at an exhibition, etc., the repeat count is counted up for each user, and when the user switches, the repeat count is reset to 0.

[0134] In other words, after the same user plays back from the start TC to the end TC, the virtual camera maintains its position and orientation, and the playback process returns to the start TC. This is called repeat playback. When at least two playbacks have been completed, playback for that user ends and the repeat count is reset.

[0135] An example of reproducing a virtual viewpoint image using the above flowchart will be described with reference to FIG.

[0136] (Example of virtual viewpoint image playback) A display example in the case where the reproduction process of the virtual viewpoint image in FIG. 8 is used will be described with reference to FIG.

[0137] As an example, similar to the first embodiment, it is assumed that a scoring scene in a rugby match is used for a demonstration at an exhibition.

[0138] The starting TC of the scoring scene is set to 19:10:02.034 and the ending TC is set to 19:10:31.020, and at the exhibition, each user will be able to preview the TC in between.

[0139] The number of users used in the explanation is two, user A and user B. Naturally, this playback process is not limited to two users, and the same playback process can be repeatedly applied to any number of users in the sequence.

[0140] In this embodiment, repeat playback is included, and as an example, the number of repeats is set to 2. The number of repeats is not limited to 2, and can be changed depending on the congestion situation of an exhibition or the like.

[0141] First, an example of operation and viewing of user A will be described. The first repeat playback of user A is the same as that shown in Figs. 7(a) to (c) described in the first embodiment. Fig. 7(c) shows the end TC at which the first repeat playback has been completed, and the process proceeds from step S804 to S807 and S808. At this time, the number of repeats is 1 and has not yet reached the upper limit, so the process proceeds to S809, where the start TC is set. Then, in the repeat process of the virtual camera in S810, the position and orientation of the virtual camera are maintained. The virtual viewpoint image at this time is shown in Fig. 9(a). In Fig. 9(a), the position and orientation of the virtual camera are the same as those shown in Fig. 7(c) at the previous end TC, and the virtual camera is looking slightly to the right from outside the long side of the field. Alternatively, only the position may be returned to the initial position, and the orientation may remain as in Fig. 7(c). This is because, in the case of a large playing field such as rugby, the virtual camera position at the start TC and the virtual camera position at the end TC may be significantly separated from each other.

[0142] After that, for the next repeat playback, user A can again freely operate the virtual camera until the end TC, and Fig. 9(b) shows the virtual viewpoint image at the end TC. The virtual camera at this point is a virtual camera looking slightly to the left from outside the long side of the field on the opposite side from Fig. 9(a). After the end TC of Fig. 9(b), the process proceeds to S807 and S808, and since the number of repeats has reached the upper limit at 2, the process proceeds to step S811 and transitions to the same stopped screen display as Fig. 7(d).

[0143] Then, the image display device 104 is handed over to the next user B. Since the TC and the virtual camera are initialized in steps S813 and S814 during the stop screen of Fig. 7(d), the screen displayed when user B starts operation is the same as that shown in Fig. 7(e).

[0144] The condition for determining whether to transition to the stopped state in step S808 may be whether the playback time of the user has exceeded a predetermined total playback time, instead of the number of times, or whether the cumulative operation time for the virtual camera has exceeded a predetermined time.

[0145] In addition, in step 810 of FIG. 8, if the state of the initialization setting button 415 is set to initialization, even in repeat playback for one user, the virtual camera is initialized after one playback from the start TC to the end TC, as shown in embodiment 1.

[0146] With the above-described virtual viewpoint image playback process, when a large number of users take turns repeatedly playing back virtual viewpoint images on a single device at an exhibition or the like, the queue can be advanced smoothly and users can be allowed to operate the appropriate virtual camera, enabling an efficient and satisfying demonstration.

[0147] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0148] The disclosure of this embodiment includes the following configurations and methods.

[0149] (Configuration 1) a display means for displaying a virtual viewpoint image generated using the three-dimensional shape data generated from the multi-viewpoint image and the virtual viewpoint information; A control means for controlling the virtual viewpoint information; an initialization means for initializing a time code of the virtual viewpoint image to a predetermined start time code in response to completion of reproduction of the virtual viewpoint image by the display means, The display means reproduces the virtual viewpoint image from the predetermined start time code to a predetermined end time code, The control means performs different controls on the virtual viewpoint information when the display means reproduces the virtual viewpoint image and when the initialization means performs initialization. A display device comprising:

[0150] (Configuration 2) The display device according to configuration 1, characterized in that when initialization is performed by the initialization means, the display means stops playback of the virtual viewpoint image and displays that playback of the virtual viewpoint image is stopped.

[0151] (Configuration 3) 3. The display device according to claim 1, wherein when initialization is performed by the initialization means, the control means sets predetermined values ​​to the position and orientation of the virtual viewpoint included in the virtual viewpoint information.

[0152] (Configuration 4) The device further includes an operation unit for accepting an operation by a user, When the virtual viewpoint image is being reproduced by the display means, the control means controls the virtual viewpoint information in response to an operation received by the operation means; When the initialization is performed by the initialization means, the control means controls the virtual viewpoint information regardless of the operation received from the operation means. 4. The display device according to any one of configurations 1 to 3.

[0153] (Configuration 5) When the display unit repeatedly plays back the virtual viewpoint image, the control unit executes different controls on the virtual viewpoint information between the control performed when the virtual viewpoint image is repeatedly played back and the control performed when the initialization unit performs initialization. 5. The display device according to any one of configurations 1 to 4.

[0154] (Configuration 6) The display device according to any one of configurations 1 to 5, characterized in that, when the virtual viewpoint image is repeatedly played back by the display means, the initialization means performs initialization in response to the repeated playback being performed a predetermined number of times.

[0155] (Configuration 7) The display device according to any one of configurations 1 to 5, characterized in that, when the virtual viewpoint image is repeatedly played back by the display means, the initialization means performs initialization in response to playing back the virtual viewpoint image for a predetermined period of time.

Claims

1. a display means for displaying a virtual viewpoint image generated using the three-dimensional shape data generated from the multi-viewpoint image and the virtual viewpoint information; A control means for controlling the virtual viewpoint information; an initialization means for initializing a time code of the virtual viewpoint image to a predetermined start time code in response to completion of reproduction of the virtual viewpoint image by the display means, The display means reproduces the virtual viewpoint image from the predetermined start time code to a predetermined end time code, The control means performs different controls on the virtual viewpoint information when the display means reproduces the virtual viewpoint image and when the initialization means performs initialization. A display device comprising:

2. 2. The display device according to claim 1, wherein, when the initialization is performed by the initialization means, the display means stops the reproduction of the virtual viewpoint image and displays a message indicating that the reproduction of the virtual viewpoint image is stopped.

3. 2. The display device according to claim 1, wherein, when the initialization is performed by the initialization means, the control means sets predetermined values ​​to the position and orientation of the virtual viewpoint included in the virtual viewpoint information.

4. The device further includes an operation unit for accepting an operation by a user, When the virtual viewpoint image is being reproduced by the display means, the control means controls the virtual viewpoint information in response to an operation received by the operation means; When the initialization is performed by the initialization means, the control means controls the virtual viewpoint information regardless of the operation received from the operation means.

2. The display device according to claim 1 .

5. When the display unit repeatedly plays back the virtual viewpoint image, the control unit executes different controls on the virtual viewpoint information between the control performed when the virtual viewpoint image is repeatedly played back and the control performed when the initialization unit executes initialization.

2. The display device according to claim 1 .

6. 2. The display device according to claim 1, wherein, when the virtual viewpoint image is repeatedly reproduced by the display means, the initialization means performs the initialization in response to the reproduction being repeated a predetermined number of times.

7. 2. The display device according to claim 1, wherein, when the virtual viewpoint image is repeatedly reproduced by the display means, the initialization means performs initialization in response to reproduction of the virtual viewpoint image for a predetermined period of time.

8. a display step of displaying a virtual viewpoint image generated using the three-dimensional shape data generated from the multi-viewpoint image and the virtual viewpoint information; a control step of controlling the virtual viewpoint information; an initialization step of initializing a time code of the virtual viewpoint image to a predetermined start time code in response to completion of the reproduction by the display step, In the display step, the virtual viewpoint image is reproduced from the predetermined start time code to a predetermined end time code, In the control step, different controls are performed on the virtual viewpoint information when the virtual viewpoint image is reproduced by the display step and when the initialization step is performed.

23. A method for controlling a display device comprising:

9. a display step of displaying a virtual viewpoint image generated using the three-dimensional shape data generated from the multi-viewpoint image and the virtual viewpoint information; a control step of controlling the virtual viewpoint information; an initialization step of initializing a time code of the virtual viewpoint image to a predetermined start time code in response to completion of the reproduction by the display step, In the display step, the virtual viewpoint image is reproduced from the predetermined start time code to a predetermined end time code, In the control step, different controls are performed on the virtual viewpoint information when the virtual viewpoint image is reproduced by the display step and when the initialization step is performed. A program that makes a computer do something.

Images