Image processing apparatus, image processing method, and storage medium

Abstract

An image processing apparatus, an image processing method, and a storage medium are provided. The image processing apparatus obtains input of additional information for a two-dimensional virtual viewpoint image based on a virtual viewpoint and a three-dimensional virtual space, converts the input additional information into an object arranged at a three-dimensional position in the virtual space, and displays the virtual viewpoint image based on the virtual space in which the object is arranged at the three-dimensional position on a display member.

Description

Technical Field

[0001] This invention relates to techniques for sophisticating the manipulation of virtual viewpoint images. Background Technology

[0002] In applications used for presentation, the following functionality is known: during image display, input is received of markers with shapes such as circles or lines to represent points of interest in the image; the page image is composited with the markers; and these page images and markers are output. Japanese Patent Application Publication No. 2017-151491 describes a technique for applying this functionality to a remote conferencing system.

[0003] In recent years, techniques for generating images (virtual viewpoint images) of a scene viewed from any viewpoint have received considerable attention, derived from multiple images captured using multiple camera devices. Even in such virtual viewpoint images, it is assumed that markers are added to targets of interest within the scene. When a marker is input into the virtual viewpoint image, it appears in the appropriate location when viewed from the viewpoint where the marker was input. However, if the viewpoint is switched to another viewpoint, the marker may appear in an unexpected location. As described above, when additional information, such as markers, is drawn onto the virtual viewpoint image, this additional information may appear in unexpected locations. Summary of the Invention

[0004] This invention provides a technique for displaying additional information drawn for a virtual viewpoint image in an appropriate location, independent of the viewpoint.

[0005] According to one aspect of the present invention, an image processing apparatus is provided, comprising: an acquisition unit for acquiring input of additional information for a two-dimensional virtual viewpoint image based on a virtual viewpoint and a three-dimensional virtual space; a conversion unit for converting the input additional information into objects arranged at three-dimensional positions in the virtual space; and a display control unit for displaying the virtual viewpoint image based on the virtual space in which the objects are arranged at the three-dimensional positions on a display unit.

[0006] According to another aspect of the present invention, an image processing method performed by an image processing device is provided, comprising: obtaining input of additional information for a two-dimensional virtual viewpoint image based on a virtual viewpoint and a three-dimensional virtual space; converting the input additional information into an object arranged at a three-dimensional position in the virtual space; and displaying the virtual viewpoint image based on the virtual space in which the object is arranged at the three-dimensional position on a display component.

[0007] Other features of the invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0008] Figure 1A and Figure 1B This is a diagram illustrating an example configuration of an image processing system;

[0009] Figure 2A and Figure 2B This is a diagram illustrating an example configuration of an image processing device;

[0010] Figures 3A to 3D It is a diagram used to illustrate a virtual viewpoint;

[0011] Figure 4A and Figure 4B This is a diagram showing an example of the configuration of the operation screen;

[0012] Figures 5A to 5E It is a diagram used to illustrate the marked objects and the areas of interest;

[0013] Figure 6 This is a flowchart illustrating an example of a process performed by an image processing device;

[0014] Figures 7A to 7D This is a diagram showing an example of a displayed screen;

[0015] Figure 8 This is a diagram illustrating an example configuration of an image processing system;

[0016] Figures 9A to 9D This is a diagram illustrating an example of the data configuration for a marked object;

[0017] Figures 10A to 10C This is a diagram showing an example of a display screen; and

[0018] Figures 11A to 11C This is a diagram showing an example of a display screen. Detailed Implementation

[0019] The embodiments will be described in detail below with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. Several features are described in the embodiments, but the invention is not limited to requiring all such features, and multiple such features can be suitably combined. Furthermore, in the drawings, the same reference numerals are given to the same or similar configurations, and redundant descriptions of these configurations are omitted.

[0020] (System Configuration)

[0021] Reference Figure 1A and Figure 1B An example of the configuration of the image processing system 100 according to this embodiment is described. The image processing system 100 includes a plurality of sensor systems (in... Figure 1AIn the example shown, sensor systems 101-1 to 101-n are represented. Each sensor system includes at least one imaging device (e.g., a camera). Note that in the following text, unless otherwise specified, sensor systems 101-1 to 101-n will be collectively referred to as "sensor system 101". In the image processing system 100, virtual viewpoint image data is generated based on image data obtained from the multiple sensor systems 101 and provided to the user.

[0022] Figure 1B An example of the installation of sensor system 101 is shown. Multiple sensor systems 101 are installed around an area of ​​the subject being photographed (hereinafter referred to as the photographed subject area 120), and photograph the photographed subject area 120 from different directions. For example, if the photographed subject area 120 is defined as the field of a stadium where a football or rugby match is played, then n (e.g., up to 100) sensor systems 101 are installed around the field. Note that there is no particular limitation on the number of sensor systems 101 installed, and at least multiple sensor systems are installed. Note that sensor systems 101 do not always need to be installed around the entire perimeter of the photographed subject area 120, and may be installed on a portion of the perimeter of the photographed subject area 120 due to limitations such as installation location. Furthermore, the imaging devices included in each of the multiple sensor systems 101 may include imaging devices with different functions, such as telephoto cameras and wide-angle cameras.

[0023] In addition to the imaging device (camera), the sensor system 101 may also include a sound collection device (microphone). The sound collection devices in the multiple sensor systems 101 collect sound synchronously. Based on the sound data collected by the multiple sound collection devices, the image processing system 100 generates virtual listening point sound data to be reproduced along with the virtual viewpoint image and provides it to the user. Note that although the description of sound will be omitted for simplicity, the image and sound are processed together.

[0024] Note that the photographed subject area 120 can be defined to include not only the stadium grounds but also, for example, the stadium stands. The photographed subject area 120 can be defined as an indoor studio or stage. That is, the area of ​​the photographed subject that serves as the target for generating the virtual viewpoint image can be defined as the photographed subject area 120. Note that "photographed subject" here can be the area defined by the photographed subject area 120 itself, or it can include all photographed subjects present in that area, such as a ball and people such as players and referees. Furthermore, throughout this embodiment, the virtual viewpoint image is a moving image, but it can also be a still image.

[0025] like Figure 1B The multiple sensor systems 101 arranged as shown synchronously capture images of a common subject area 120 using the imaging devices provided in the sensor systems 101. In this embodiment, the images included in the multiple images obtained by synchronously capturing the common subject area 120 from multiple viewpoints are referred to as "multi-viewpoint images". Note that the multi-viewpoint images in this embodiment can be the captured images themselves, but can also be images that have undergone image processing, such as processing to extract a predetermined region from the captured images.

[0026] In addition, the image processing system 100 also includes an image recording device 102, a database 103, and an image processing device 104. The image recording device 102 collects multi-view images obtained through imaging by multiple sensor systems 101 and stores these multi-view images, along with the timecode used in the imaging, in the database 103. Here, the timecode is information used to uniquely identify the time of imaging. For example, the timecode may be information specifying the imaging time in a format such as day:hour:minute:second.frame number.

[0027] Image processing device 104 obtains multiple multi-view images corresponding to a common timecode from database 103, and generates a three-dimensional model of the photographed subject based on the obtained multi-view images. The three-dimensional model is configured to include, for example, shape information (such as a group of points representing the shape of the photographed subject, or faces or vertices defined when the shape of the photographed subject is represented as a set of polygons) and texture information representing the color and texture on the surface of the shape. Note that this is merely an example, and the three-dimensional model can be defined in any format representing the photographed subject in three dimensions. For example, based on a user-specified virtual viewpoint, image processing device 104 uses the three-dimensional model of the photographed subject to generate a virtual viewpoint image corresponding to that virtual viewpoint and outputs the virtual viewpoint image. For example, as... Figure 1B As shown, a virtual viewpoint 110 is specified based on the position and viewing direction of a viewpoint in a virtual space associated with the photographed subject area 120. The user changes the viewing direction by moving the virtual viewpoint in the virtual space, thereby viewing the photographed subject generated based on a 3D model of the photographed subject existing in the virtual space from a viewpoint different from any of the camera devices in the plurality of sensor systems 101. Note that since the virtual viewpoint can move freely in the 3D virtual space, the virtual viewpoint image can also be referred to as a "free viewpoint image."

[0028] Image processing device 104 generates a virtual viewpoint image as an image representing the scene observed from virtual viewpoint 110. Note that the image generated here is a two-dimensional image. Image processing device 104 is, for example, a computer used by a user, and is configured to include a display device such as a touch panel display or a liquid crystal display. Furthermore, image processing device 104 may have display control functions for displaying images on external display devices. Image processing device 104 displays the virtual viewpoint image, for example, on the screen of these display devices. That is, image processing device 104 performs processing to generate an image of the scene within the range visible from the virtual viewpoint as a virtual viewpoint image and displays it on the screen.

[0029] Note that the image processing system 100 may have the same characteristics as... Figure 1A The configurations shown are different. For example, a configuration that includes an operation / display device such as a touch panel display separately from the image processing device 104 can be used. For example, a configuration can be used where a virtual viewpoint operation is performed on a tablet computer including a touch panel display, and in response, a virtual viewpoint image is generated in the image processing device 104 and displayed on the tablet computer. Note that a configuration can be used where multiple tablet computers are connected to the image processing device 104 via a server, and the image processing device 104 outputs the virtual viewpoint image to each of the multiple tablet computers. Furthermore, the database 103 and the image processing device 104 can be integrated. Furthermore, a configuration can be used where processing from multi-viewpoint images to the generation of a 3D model of the photographed subject is performed in the image recording device 102, and the 3D model of the photographed subject is stored in the database 103. In this case, the image processing device 104 reads the 3D model from the database 103 and generates a virtual viewpoint image. Figure 1A An example of multiple sensor systems 101 connected in a daisy-chain configuration is shown. For example, sensor systems 101 can be directly connected to image recording device 102, or they can be connected in other ways. Note that in order for the sensor systems 101 to perform synchronous video recording, image recording device 102 or other devices for time synchronization can notify each sensor system 101 of reference time information.

[0030] In this embodiment, the image processing device 104 also accepts input from the user of markers such as circles or lines for a virtual viewpoint image displayed on the screen, and displays the markers superimposed on the virtual viewpoint image. If such a marker is input, it is appropriately displayed at the virtual viewpoint where the marker was input. However, when the position or orientation of the virtual viewpoint changes, the marker may deviate from the object to which the marker was added, resulting in an unexpected display. Therefore, in this embodiment, the image processing device 104 performs processing to display the accepted markers at the appropriate position on the displayed two-dimensional screen regardless of the movement of the virtual viewpoint. The image processing device 104 converts the two-dimensional markers into three-dimensional marker objects. The image processing device 104 combines the three-dimensional model of the photographed subject and the three-dimensional marker objects to generate a virtual viewpoint image in which the position of the markers is appropriately adjusted according to the movement of the virtual viewpoint. An example of the configuration and processing procedure of the image processing device 104 performing this processing will be described below.

[0031] (Configuration of image processing equipment)

[0032] Next, we will refer to Figure 2A and Figure 2B The configuration of the image processing device 104 is described. Figure 2A An example of the functional configuration of the image processing device 104 is shown. The image processing device 104 includes, for example, a virtual viewpoint control unit 201, a model generation unit 202, an image generation unit 203, a marker control unit 204, and a marker management unit 205 as its functional configuration. Note that these are merely examples. At least some of the functions shown may be omitted, and other functions may be added. If the functions described later can be performed, then... Figure 2A All the functions shown can be replaced by other function blocks. Furthermore, Figure 2A Two or more functional blocks can be integrated into one functional block, or a functional block can be divided into multiple functional blocks.

[0033] The virtual viewpoint control unit 201 accepts user operations related to the virtual viewpoint 110 or timecode and controls the operation of the virtual viewpoint. A touch panel or joystick, etc., is used for user operations on the virtual viewpoint. However, the invention is not limited to this, and user operations can be accepted by any device. The model generation unit 202 obtains multi-viewpoint images corresponding to the timecode specified by the user operation, etc., from the database 103, and generates a three-dimensional model representing the three-dimensional shape of the photographed subject included in the photographed subject area 120. For example, the model generation unit 202 obtains a foreground image from the multi-viewpoint images, extracting the foreground region corresponding to the photographed subject, such as a person or a ball, and a background image, extracting the background region other than the foreground region. The model generation unit 202 generates a three-dimensional model of the foreground based on multiple foreground images. The three-dimensional model is formed, for example, by a group of points generated by a shape estimation method such as a visual hull. Note that the format of the three-dimensional shape data representing the shape of the object is not limited to this, and grids or other specific formats of three-dimensional data can be used. Note that the model generation unit 202 can generate a three-dimensional model of the background in the same manner. Regarding the 3D model of the background, a 3D model pre-generated by an external device can be obtained. In the following text, for ease of description, the 3D model of the foreground and the 3D model of the background will be collectively referred to as the "3D model of the photographed subject," or simply the "3D model."

[0034] Based on the 3D model of the photographed subject and a virtual viewpoint, the image generation unit 203 generates a virtual viewpoint image to reproduce the scene viewed from that virtual viewpoint. For example, the image generation unit 203 obtains appropriate pixel values ​​from a multi-viewpoint image for each point forming the 3D model and performs color processing. Then, the image generation unit 203 arranges the 3D model in a 3D virtual space and projects and renders the 3D model along with its pixel values ​​onto the virtual viewpoint, thereby generating a virtual viewpoint image. Note that the method for generating a virtual viewpoint image is not limited to this, and other methods, such as generating a virtual viewpoint image through projection transformation of the photographed image without using a 3D model, can be used.

[0035] The marker control unit 204 receives input from the user of markers, such as circles or lines, for a virtual viewpoint image. The marker control unit 204 converts the marker input for the two-dimensional virtual viewpoint image into marker objects as three-dimensional data in virtual space. The marker control unit 204 sends an instruction to the image generation unit 203, causing it to generate a virtual viewpoint image that combines the marker objects and a three-dimensional model of the photographed subject based on the position / pose of the virtual viewpoint. Note that the marker control unit 204 provides the marker objects as a three-dimensional model to the image generation unit 203, for example, and the image generation unit 203 processes the marker objects in the same way as the photographed subject to generate the virtual viewpoint image. Based on the marker objects provided by the marker control unit 204, the image generation unit 203 can perform the process of overlaying the marker objects separately from the process for generating the virtual viewpoint image. The marker control unit 204 can perform the process of overlaying markers based on the marker objects onto the virtual viewpoint image provided by the image generation unit 203. The tag management unit 205 performs storage control to store the tag objects of the 3D model converted by the tag control unit 204 in, for example, the storage unit 216 described later. The tag management unit 205 performs storage control such that the tag objects are stored in association with, for example, timecode. Note that the model generation unit 202 can calculate the coordinates of various objects in the foreground, such as figures or balls, accumulate these coordinates in the database 103, and use the coordinates of each object to specify the coordinates of the tag objects.

[0036] Figure 2B An example of the hardware configuration of an image processing device 104 is shown. The image processing device 104 includes, for example, a CPU 211, RAM 212, ROM 213, an operation unit 214, a display unit 215, a storage unit 216, and an external interface 217 as its hardware configuration. Note that CPU is an abbreviation for Central Processing Unit, RAM is an abbreviation for Random Access Memory, and ROM is an abbreviation for Read Only Memory.

[0037] CPU 211 uses programs and data stored in, for example, RAM 212 or ROM 213 to execute the control of the entire image processing device 104 or the processing described later. When CPU 211 executes the program stored in RAM 212 or ROM 213, it can achieve Figure 2AThe functional blocks are shown. Note that the image processing device 104 may include dedicated hardware such as one or more processors in addition to CPU 211, and the hardware performs at least a portion of the processing of CPU 211. The dedicated hardware may be, for example, an MPU (microprocessor unit), an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), or a DSP (digital signal processor). ROM 213 holds programs and data used to perform processing for virtual viewpoint images or markers. RAM 212 provides a working area to temporarily store programs and data read from ROM 213 and used by CPU 211 to perform various processes.

[0038] The operation unit 214 is configured as a device for accepting user operations, including, for example, a touch panel and buttons. The operation unit 214, for example, acquires information indicating user operations on a virtual viewpoint or marker. Note that the operation unit 214 can be connected to an external controller and accept input information from the user related to the operation. The external controller is not particularly limited and can be, for example, a three-axis controller such as a joystick, or a keyboard or mouse. The display unit 215 is configured as a display device, such as a monitor. The display unit 215 displays, for example, a virtual viewpoint image generated by the CPU 211, etc. Furthermore, the display unit 215 can include various types of output devices capable of presenting information to the user, such as speakers for audio output and devices for vibration output. Note that the operation unit 214 and the display unit 215 can be integrally formed using, for example, a touch panel display.

[0039] Storage unit 216 is configured to include, for example, a high-capacity storage device such as an SSD (Solid State Drive) or HDD (Hard Disk Drive). Note that these are merely examples, and storage unit 216 can be configured to include any other storage device. Storage unit 216 records data to be processed by the program. Storage unit 216 stores, for example, a three-dimensional marked object obtained when the CPU 211 transforms the marked input received via operation unit 214. Storage unit 216 may also store other information. External interface 217 is configured to include, for example, an interface device connected to a network such as a LAN (Local Area Network). Information is sent to / received from an external device such as database 103 via external interface 217. Additionally, external interface 217 can be configured to include, for example, an interface device connected to a network such as a LAN (Local Area Network). This could be an image output port such as SDI. Note that HDMI is an abbreviation for High Definition Multimedia Interface, and SDI is an abbreviation for Serial Digital Interface. In this case, information can be sent to an external display device or projection device via external interface 217. Furthermore, the image processing device can use external interface 217 to connect to a network to receive virtual viewpoint or marker operation information or send virtual viewpoint images via the network.

[0040] (Virtual viewpoint and line of sight)

[0041] Next, we will refer to Figures 3A to 3D Virtual viewpoint 110 is described. Virtual viewpoint 110 and its operations are specified using a coordinate system. In this embodiment, a coordinate system is used as follows: Figure 3A The orthogonal coordinate system shown, formed by the X, Y, and Z axes in a general three-dimensional space, is used as a coordinate system. Note that this is merely an example, and any coordinate system capable of representing a position in three-dimensional space can be used. A coordinate system is used to set and utilize the coordinates of the subject being photographed. Subjects being photographed include, for example, a stadium field or studio, and people and objects such as balls existing within the space of that field or studio. For example, in Figure 3B In the example shown, the subject of the photograph includes the entire stadium field 391, as well as the ball 392 and players 393 present on the field 391. Note that the stands and other structures surrounding the field may also be included in the subject. Figure 3B In the example shown, the coordinates of the center of site 391 are set to the origin (0, 0, 0), the X-axis is set to the direction of the longer side of site 391, the Y-axis is set to the direction of the shorter side of site 391, and the Z-axis is set to the direction perpendicular to the site. By setting the coordinates of each photographed subject relative to the center of site 391, the 3D model generated from the photographed subject can be arranged in a 3D virtual space. Note that the method of setting the coordinates is not limited to this.

[0042] Next, we will refer to Figure 3C and Figure 3D Describes the virtual viewpoint. The virtual viewpoint defines the viewpoint and viewing direction used to generate the virtual viewpoint image. (Reference) Figure 3CThe vertices of the quadrangular pyramid represent the position 301 of the virtual viewpoint, and the vectors extending from the vertices represent the viewing direction 302. The position 301 of the virtual viewpoint is represented by coordinates (x, y, z) in three-dimensional virtual space. The viewing direction 302 is represented by a unit vector including components of each axis as scalars, and is also called the optical axis vector of the virtual viewpoint. The viewing direction 302 passes through the centers of the front clip plane 303 and the rear clip plane 304. Note that the clip plane is the plane that defines the area as the drawing target. The space 305 between the front clip plane 303 and the rear clip plane 304 is called the view frustum of the virtual viewpoint, and the virtual viewpoint image is generated (or projected and displayed) within this range. Note that the focal length (not shown) can be set to any value. Similar to a regular camera, the viewing angle changes when the focal length changes. That is, when the focal length is shortened, the viewing angle is widened, and the view frustum is widened. On the other hand, a longer focal length results in a narrower angle of view and allows for a narrower frustum to capture larger subjects. Note that focal length is merely an example, and any parameter that can set the position or size of the frustum can be used.

[0043] The position of the virtual viewpoint and the direction of the line of sight from it can be moved and rotated in a virtual space represented by three-dimensional coordinates. For example... Figure 3D As shown, the movement 306 of the virtual viewpoint is the movement of the virtual viewpoint's position 301, and is represented by the components (x, y, z) of each axis. The rotation 307 of the virtual viewpoint is as follows... Figure 3A The values ​​are represented by yaw (rotation around the Z-axis), pitch (rotation around the X-axis), and roll (rotation around the Y-axis). This allows the virtual viewpoint's position and the direction of the line of sight from it to move and rotate freely in three-dimensional virtual space, and the image processing device 104 can reproduce an image as a virtual viewpoint image, assuming the subject is viewed from any angle at any region. Note that, unless otherwise specified, the position of the virtual viewpoint and the direction of the line of sight from it will be referred to together as the "virtual viewpoint".

[0044] (How to operate virtual viewpoints and markers)

[0045] Reference Figure 4A and Figure 4B Describe the operation methods for virtual viewpoints and markers. Figure 4AThis diagram illustrates the screen displayed by the image processing device 104. The case of using a tablet computer terminal 400 including a touch panel display will be described here. Note that the terminal 400 does not necessarily have to be a tablet computer terminal, and any other type of information processing device can be used as the terminal 400. If the terminal 400 is the image processing device 104, the terminal 400 is configured to generate and display virtual viewpoint images, and also accept operations such as specifying virtual viewpoints and timecodes, as well as tag input. On the other hand, if the terminal 400 is a device connected to the image processing device 104 via a communication network, the terminal 400 sends information representing virtual viewpoints or timecodes to the image processing device 104 and receives provided virtual viewpoint images. Furthermore, the terminal 400 accepts tag input operations for the virtual viewpoint images and sends information representing the accepted tag input to the image processing device 104.

[0046] refer to Figure 4A The display screen 401 on the terminal 400 is roughly divided into two areas, including the virtual viewpoint operation area 402 and the timecode operation area 403.

[0047] In the virtual viewpoint operation area 402, user operations related to the virtual viewpoint are accepted, and a virtual viewpoint image is displayed within the area. That is, in the virtual viewpoint operation area 402, the virtual viewpoint is operated and a virtual viewpoint image is displayed, which reproduces the scene hypothetically observed from the virtual viewpoint after the operation. Furthermore, in the virtual viewpoint operation area 402, marker input for the virtual viewpoint image is accepted. Note that although marker operations and virtual viewpoint operations can be performed together, this embodiment assumes that marker operations are accepted independently of virtual viewpoint operations. In the example, such as... Figure 4B As shown in the example, the virtual viewpoint can be manipulated by touch operations such as tapping and dragging on the terminal 400 using the user's finger, and marking operations can be performed by tapping and dragging using a drawing device such as a pen 450. The user moves or rotates the virtual viewpoint by dragging 431 using, for example, a finger. Furthermore, the user draws markings 451 or 452 on the virtual viewpoint image by dragging using the pen 450. The terminal 400 draws the markings on continuous coordinates using the dragging operations of the pen 450. Note that finger operations can be assigned to marking operations, and pen operations can be assigned to virtual viewpoint operations. Additionally, if the terminal 400 can distinguish between virtual viewpoint operations and marking operations, any other operation method can be used. According to this configuration, the user can easily selectively use virtual viewpoint operations and marking operations.

[0048] Note that when performing virtual viewpoint and marking operations independently, it is not always necessary to use a drawing device such as a pen 450. For example, an ON / OFF button (not shown) for marking operations can be provided on the touch panel, and marking operations can be toggled by button operation. For example, the button is turned on to perform marking operations. Virtual viewpoint operations can be disabled while the button is in the ON state. Conversely, the button is turned off to perform virtual viewpoint operations. Marking operations can be disabled while the button is in the OFF state.

[0049] The timecode operation area 403 is used to specify the timing of the virtual viewpoint image to be viewed. The timecode operation area 403 includes, for example, a main slider 412, a sub-slider 413, a speed specification slider 414, and a cancel button 415. The main slider 412 is used to accept any timecode selected by the user through dragging the position of the button 422, etc. The entire time period from which the virtual viewpoint image can be reproduced is represented by the range of the main slider 412. The sub-slider 413 zooms in and displays a portion of the timecode, allowing the user to perform detailed operations, such as frame-by-frame. The sub-slider 413 is used to accept the user's selection of any detailed timecode through dragging the position of the button 423, etc.

[0050] On terminal 400, an approximate timecode specification is received via main slider 412, and a detailed timecode specification is received via sub-slider 413. For example, main slider 412 and sub-slider 413 can be configured such that main slider 412 corresponds to a 3-hour range corresponding to the entire length of the match, and sub-slider 413 corresponds to a time range of approximately 30 seconds, which is part of that length. For example, a 15-second interval before and after the timecode specified by main slider 412, or a 30-second interval from the timecode, can be represented by sub-slider 413. Furthermore, the time can be pre-divided into intervals of 30 seconds, and sub-slider 413 can represent the intervals that include the timecode specified by main slider 412. As mentioned above, the time scales of main slider 412 and sub-slider 413 are different. Note that the above time lengths are merely examples, and the sliders can be configured to correspond to other time lengths. Note that a user interface capable of changing, for example, the setting of the time length corresponding to sub-slider 413 can be prepared. Furthermore, although... Figure 4AAn example is shown where the main slider 412 and the sub-slider 413 are displayed on the screen with the same length, but their lengths can differ. That is, the main slider 412 can be longer, or the sub-slider 413 can be longer. Furthermore, the sub-slider 413 does not always need to be displayed. For example, it can be displayed after a display instruction is received, or it can be displayed when a specific operation such as pause is indicated. Timecodes can be specified and displayed without using the buttons 422 of the main slider 412 and 423 of the sub-slider 413. For example, timecodes can be specified and displayed using numerical values ​​such as day:hour:minute:second.frame number.

[0051] The speed specification slider 414 is used to accept the user's specification of the playback speed, such as 1x speed playback or slow playback. For example, the playback speed selected by the button 424 using the speed specification slider 414 controls the upward counting interval of the timecode. The cancel button 415 is used to cancel various operations related to the timecode. In addition, the cancel button 415 can be used to clear the pause and return to normal playback. Note that if the button is configured to perform timecode-related operations, it is not limited to canceling.

[0052] Using the screen configuration described above, the user can display a virtual viewpoint image on the terminal 400, showing a 3D model of a photographed subject at any timecode from any position / pose by manipulating the virtual viewpoint and timecode. The user can input markers into the virtual viewpoint image independently of manipulating the virtual viewpoint.

[0053] (Marking objects and areas of interest)

[0054] In this embodiment, two-dimensional markers input by the user into a two-dimensional virtual viewpoint image are converted into three-dimensional marker objects. The converted three-dimensional marker objects are then arranged in the same three-dimensional virtual space as the virtual viewpoint. First, the reference... Figures 5A to 5E Describe the methods for converting marked objects.

[0055] Figure 5A This illustrates the state of inputting markers into the virtual viewpoint image 500 using a pen 450. Here, the vector representing a point on the marker will be called the marker input vector 501. Figure 5B It is schematically shown in the generation Figure 5A The virtual viewpoint image 500 displayed is the image of the virtual viewpoint specified in the image. Let [M] c ]for Figure 5A The marker input vector 501 is defined in [M]. Using camera coordinates relative to the virtual viewpoint position 301, which is the origin, the marker input vector 501 can be represented as [M]. c ] = (a, b, f). Here, "f" is the focal length of the virtual viewpoint. Figure 5B In the diagram, the intersection point between the optical axis of the virtual camera and the focal plane of the virtual viewpoint is (0, 0, f), and the input vector [M] is labeled. c ] = (a, b, f) represents a point that has moved 'a' in the x-direction and 'b' in the y-direction from the intersection point. Regarding the labeled input vector, it is represented in camera coordinates as [M c ], and represented in world coordinates as [M W ]=(m x m y m z World coordinates are coordinates in virtual space. Using the quaternion Qt obtained from the rotation matrix representing the pose of the virtual camera, through [M... W ] = Qt·[M c To calculate the marker input vector 502([M) in world coordinates W Note that the quaternion of a virtual camera is a general technical term, and its detailed description will be omitted here.

[0056] In this embodiment, in order to convert the labeled input for a two-dimensional virtual viewpoint image into a labeled object of three-dimensional data, a method such as... Figure 5C and Figure 5D The area of ​​concern shown is 510. Note that... Figure 5C It is a top view of the entire area of ​​interest, 510, and Figure 5D This is the image viewed from a direction parallel to the plane of interest 510. That is to say, Figure 5C and Figure 5D This illustrates the state of viewing the same subject from different directions. The plane of interest 510 is, for example, a plane parallel to the site 391 (Z=0) and located at a height easily accessible to the viewer (e.g., Z=1.5m). The plane of interest 510 can be composed of, for example, z=z fix It can be expressed by an equation.

[0057] The marker object is generated as 3D data in contact with the surface of interest 510. For example, the intersection point 503 between the marker input vector 502 and the surface of interest 510 is generated as a marker object corresponding to a point on the marker corresponding to the marker input vector 502. That is, a transformation from marker to marker object is performed such that the intersection point between the line passing through the virtual viewpoint and the point on the marker and the predetermined surface prepared as the surface of interest 510 becomes a point in the marker object corresponding to the point on the marker. Note that the marker input vector 502([M W ]=(m x m y m z )) and surface 510 (z=z fixThe intersection point between the two points can be calculated using general mathematical methods, and its detailed description will be omitted. Here, we assume that the intersection point is represented by the intersection point coordinates (A). w =(a x a y a z The marker object 520 is obtained by calculating the intersection coordinates of each continuous point obtained as the marker input, and connecting these intersection coordinates to obtain the three-dimensional data. That is, if the input is a marker of a continuous straight line or curve, the marker object 520 is generated as a straight line or curve on the attention surface 510 corresponding to that continuous straight line or curve. Therefore, the marker object 520 is generated as a three-dimensional object that is easily noticed by the viewer and contacts the attention surface. The attention surface can be set based on the height of the object the viewer is focused on (e.g., based on the height of the ball's main position or the height of the player's center). Note that the conversion from marker to marker object 520 can be accomplished as described above by calculating the intersection points between the vector and the attention surface, or the conversion from marker to marker object can be accomplished by a predetermined matrix operation, etc. Alternatively, the marker object can be generated based on a table corresponding to the positions of the virtual viewpoint and the input marker. Note that regardless of the method used, processing can be performed to generate a marker object on the attention surface.

[0058] In the example, when Figure 5A When the above transformation is performed on the input marker, which is a two-dimensional circle, on the tablet computer shown, the following is generated: Figure 5E The marker object 520 shown has a three-dimensional toroidal shape. The marker object is created from continuous points, and as shown... Figure 5E The data shown is generated as three-dimensional data with a predetermined height added.

[0059] Note that in the reference Figures 5A to 5E In the described example, the surface of interest 510 is a surface parallel to the XY plane, which serves as the site. However, the invention is not limited to this. For example, if the subject being photographed is moving in the vertical direction (such as rock climbing), the surface of interest can be set to be a surface parallel to the XZ or YZ plane. That is, any surface that can be defined in three-dimensional space can be used as the surface of interest. Note that the surface of interest is not limited to a plane and can be a curved surface. That is, even if a curved surface is used, the shape of the surface is not particularly limited if the intersection point between the surface and the marker input vector can be uniquely calculated.

[0060] (The processing procedure)

[0061] Next, we will refer to Figure 6An example describing the process performed by the image processing device 104 is provided. This process consists of a loop (steps S601 and S612) repeating steps S602 to S611, and the loop is executed at a predetermined frame rate. For example, if the frame rate is 60 FPS, one loop (one frame) of processing is performed at an interval of approximately 16.6 [ms]. As a result, in step S611, described later, a virtual viewpoint image is output at this frame rate. The frame rate can be set to synchronize with the update rate in the screen display of the image processing device 104. However, the frame rate can be set according to the frame rate of the camera used to capture multi-viewpoint images or the frame rate of the 3D model stored in the database 103. Note that in the following description, the timecode is counted up by one frame each time the loop is executed. The timecode up-counting interval can be changed according to user operations, etc. For example, if a 1 / 2 playback speed is specified, the timecode frame up-counting can be performed once between two loop processes. Furthermore, for example, if a pause is specified, the timecode up-counting can be stopped.

[0062] In the loop processing, the image processing device 104 updates the timecode of the processing target (step S602). As described above, the timecode here is represented in the form of day:hour:minute:second.frame number, and is updated in units of frames, such as by counting up. The image processing device 104 determines whether the accepted user operation is a virtual viewpoint operation or a marker input operation (step S603). Note that the type of operation is not limited to these. For example, if an operation targeting the timecode is accepted, the image processing device 104 can return the processing to step S602 and update the timecode. If no user operation is accepted, the image processing device 104 can assume that a virtual viewpoint assignment operation was performed in the immediately preceding virtual viewpoint image generation process and proceed with the processing. The image processing device 104 can determine whether other operations have been accepted.

[0063] When it is determined in step S603 that a virtual viewpoint operation has been received, the image processing device 104 obtains the two-dimensional operation coordinates for the virtual viewpoint (step S604). These two-dimensional operation coordinates are, for example, coordinates representing the position where a touch operation on the touch panel is received. Based on the operation coordinates obtained in step S604, the image processing device 104 performs at least one of movement and rotation of the virtual viewpoint in three-dimensional virtual space (step S605). The above relates to... Figure 3DThe movement / rotation of the virtual viewpoint is described, and its repetition will be omitted here. Furthermore, known techniques can be used to perform processing for determining the amount of movement and rotation of the virtual viewpoint in three-dimensional space based on two-dimensional coordinates obtained through touch operations on the touch panel, and its detailed description will also be omitted here. Following or simultaneously with the processing in steps S604 and S605, the image processing device 104 determines whether a marked object exists within the field of view determined by the moved / rotated virtual viewpoint (step S606). If a marked object exists within the field of view determined by the moved / rotated virtual viewpoint ("Yes" in step S606), the image processing device 104 reads the marked object and arranges it in the three-dimensional virtual space (step S607). After arranging the marked object, the image processing device 104 generates a virtual viewpoint image including the marked object and outputs the virtual viewpoint image (step S611). In other words, based on the virtual viewpoint, the image processing device 104 uses the three-dimensional model of the photographed subject corresponding to the timecode updated in step S602 and the marker objects arranged in the virtual space to generate a virtual viewpoint image. Note that if there are no marker objects within the field of view determined by the moved / rotated virtual viewpoint ("No" in step S606), the image processing device 104 generates the virtual viewpoint image without arranging marker objects and outputs the virtual viewpoint image (step S611).

[0064] When it is determined in step S603 that a marker input operation has been accepted, the image processing device 104 obtains the two-dimensional coordinates of the marker operation in the virtual viewpoint image (step S608). That is, the image processing device 104 obtains the coordinates of the marker operation as shown in the image. Figure 4A The two-dimensional coordinates of the marker operation input for the virtual viewpoint operation region 402 are then used. The image processing device 104 then converts the two-dimensional coordinates of the marker input obtained in step S608 into marker objects as three-dimensional data on the plane of interest in virtual space (step S609). (Refer to the above...) Figures 5A to 5E The method for converting from marker input to marker object is described, and its repeated description will be omitted here. When the marker object is obtained in step S609, the image processing device 104 retains the marker object along with the timecode (step S610). Based on the virtual viewpoint, the image processing device 104 generates a virtual viewpoint image using the three-dimensional model of the photographed subject corresponding to the timecode updated in step S602 and the marker object arranged in the virtual space (step S611).

[0065] Based on the above processing, the marker input for a two-dimensional virtual viewpoint image can be converted into three-dimensional data using the plane of interest in virtual space, and the three-dimensional model of the photographed subject and the marker object can be arranged in the same virtual space. Therefore, regardless of the position / pose of the virtual viewpoint, a virtual viewpoint image that maintains the positional relationship between the photographed subject and the marker can be generated.

[0066] (Example shown on screen)

[0067] Reference Figures 7A to 7D Examples of screen displays when the above processing is performed and when the processing is not performed. Figure 7A The image processing device 104 is shown in a state before the display of a virtual viewpoint image. In this state, the user can arbitrarily move / rotate the virtual viewpoint via, for example, touch operation 701. Here, as an example, the timecode is paused by the user operation, and mark input can be accepted in this state. Note that the mark input can correspond to the virtual viewpoint image during a predetermined time period, and the mark input can be maintained for a predetermined number of frames from the frame where the operation was accepted, for example. Furthermore, the input mark can be maintained, for example, until the user inputs an explicit mark erasure instruction. Figure 7B The state during a user's marking operation is shown. An example of receiving two marking inputs (circle 711 and curve 712) via a pen is shown here. When such marking input is received, the image processing device 104 converts these marking inputs into marking objects of a 3D model on the plane of interest and draws these marking objects together with a virtual viewpoint image. Note that in a state where the virtual viewpoint remains unchanged, the same display is performed as when the user's marking input is complete, and the user perceives it as a state of simply entering markings. Note that during the execution of the conversion process to marking objects, the received 2D marking input can be directly displayed, and when the conversion process to marking objects is complete, a redrawing based on the 3D marking object can be performed.

[0068] Figure 7CAn example is shown where, after the input marker is converted into a 3D marker object and drawn, the virtual viewpoint is moved / rotated via a user's touch operation 721. According to the method of this embodiment, since the 3D marker object is arranged in virtual space near the 3D position of the subject to be photographed, the marker remains near the subject even after the virtual viewpoint is moved / rotated. Furthermore, since the marker is converted into a 3D marker object and drawn in the virtual viewpoint image, its position and orientation change according to the movement / rotation of the virtual viewpoint, allowing for observation. As described above, according to the method of this embodiment, consistency between the marker input content and the content of the virtual viewpoint image can be maintained even after the virtual viewpoint is moved / rotated. On the other hand, if the method of this embodiment is not applied, after the virtual viewpoint is moved / rotated, the marker remains displayed at the input position, and only the content of the virtual viewpoint image changes. Therefore, the marker input content is inconsistent with the content of the virtual viewpoint image.

[0069] As described above, according to this embodiment, in a virtual space where a 3D model of the subject is arranged, a marker is generated as a marker object on a surface easily focused on by the viewer for the virtual viewpoint image. Then, the marker object, along with the 3D model of the subject, is drawn together as a virtual viewpoint image. This allows the positional relationship between the subject and the marker to be maintained, and eliminates or reduces discomfort caused by the deviation of the marker relative to the subject when the virtual viewpoint moves to any position / pose.

[0070] (Sharing of marked objects)

[0071] Note that the marked objects generated by the above method can be shared by other devices. Figure 8 This shows an example of the configuration for a system that performs such sharing. (Reference) Figure 8 The virtual viewpoint image generation system 100 is configured to perform operations related to... Figure 1AAt least some of the described functions are performed, including capturing the subject and generating multi-viewpoint images or 3D models. The management server 801 is a storage device configured to manage and store the captured 3D models of the subject or shared marker objects (described later) in timecode units, and may also be a distribution device configured to distribute marker objects in timecode units. Each image processing device in image processing devices 811 to 813 has the same functions as, for example, image processing device 104, obtaining 3D models from the management server 801 and generating and displaying virtual viewpoint images. Furthermore, each image processing device in image processing devices 811 to 813 can receive marker input from a user, generate marker objects, and upload the generated marker objects to the management server 801 for storage. The first image processing device downloads the marker objects uploaded by the second image processing device, thereby generating virtual viewpoint images by combining the markers input by the user of the second image processing device with the 3D model of the subject.

[0072] (Data configuration for marked objects)

[0073] In the example, the image processing device 104 uses the tag management unit 205 to... Figures 9A to 9C The configuration shown maintains the tagged objects. For example, image processing device 104 can upload the data of the tagged objects to management server 801 while maintaining the same data configuration. Furthermore, image processing device 104 downloads the data of the tagged objects with the same data configuration from management server 801. Note that... Figures 9A to 9C The data configuration shown is merely an example, and data in any format that can specify the position and shape of the marked object in virtual space can be used. The management server 801 and the image processing device 104 can use different data formats and can reproduce the marked object data as needed according to predetermined rules.

[0074] Figure 9A This illustrates an example of the configuration of the data 900 for a marker object. The object type is stored in the header 901 at the top of the dataset. For example, information indicating that the dataset is a “marker” dataset is stored in the header 901. Note that the dataset type can be specified as “foreground” or “background”, etc. Furthermore, the data 900 for the marker object includes a frame count 902 and one or more data points 903 corresponding to one or more frames (timecodes).

[0075] The data 903 of each frame (timecode) has, for example, the following characteristics: Figure 9BThe configuration is shown. Data 903 includes, for example, timecode 911 and data size 912. Note that data size 912 allows the specification of the boundary between the data of a frame (timecode) and the data of the next frame. The data size for each frame in a plurality of frames can be prepared outside of data 903. In this case, the data size information can be stored, for example, between header 901 and frame count 902, between frame count 902 and data 903, or after data 903. Data 903 also stores a marker object count 913. The marker object count 913 represents the number of markers included in the timecode. For example, in Figure 7B In the example shown, two markers, including circle 711 and curve 712, are input. Therefore, in this case, marker object count 913 indicates the presence of two marker objects. Data 903 includes data 914 containing the same number of marker objects as indicated by marker object count 913.

[0076] The data 914 of each marked object, for example, has the following characteristics: Figure 9C The configuration shown. The data 914 for each marker object includes a data size 921 and a data type 922. The data size 921 represents the size of each data 914 within one or more data sets 914 and is used to specify the boundaries of the data for each marker object. The data type 922 represents the type of shape of the 3D model. The data type 922 specifies information such as "point group" or "mesh," for example. Note that the shape of the 3D model is not limited to this. As an example, if the information representing a "point group" is stored in the data type 922, then the data 914 subsequently includes a point group count 923 and one or more combinations of the coordinates 924 and texture 934 of all point groups. Note that each data 914 within one or more data sets 914 can include not only the 3D data of the marker object but also the center coordinates of all point group coordinates or the minimum and maximum values ​​on each 3D axis (not shown). Furthermore, each data 914 within one or more data sets 914 can also include other data. Note that the data for the marker objects does not always need to be prepared in frames and can be configured to be animated, for example, in a format such as a scene graph.

[0077] When using, such as Figures 9A to 9C The data shown can be used to manage, for example, multiple marker objects for different timecodes in a sports match. Furthermore, multiple markers can be managed corresponding to a single timecode.

[0078] The following section describes the management of marked objects on management server 801. Figure 9DThis example shows a table configured to manage tagged objects. Timecodes are plotted along the horizontal axis, and information representing the tagged objects corresponding to each timecode is stored along the vertical axis. For example, in the cell corresponding to the row of "First Object," data for multiple timecodes (frames) is stored, as shown in data 903 for "First Object." Figure 9C The data shown is stored in a single cell. When using, as... Figure 9D When configuring the tables in the database shown, if a marker object is specified, the management server 801 can obtain data for all timecodes related to that marker object. Furthermore, when a range of timecodes is specified, the management server 801 can obtain data for all marker objects included within that timecode. Note that these are merely examples, and if marker objects can be managed and shared, the data configuration and management methods described above do not always need to be used.

[0079] (Sharing and displaying marked objects)

[0080] Reference Figures 10A to 10C Examples describing the sharing and display of marked objects. This section will describe the sharing and display of marked objects. Figure 8 The image processing devices 811 to 813 shown share a marked object. Note that here... Figure 10A The operation screen displayed on the image processing device 811 is shown. Figure 10B The operation screen displayed on the image processing device 812 is shown, and Figure 10C The operation screen displayed on the image processing device 813 is shown. As an example, such as... Figure 10A As shown, assuming that in the image processing device 811, a marker with the shape of a circle 1001 is received for a virtual viewpoint image of a certain timecode. For example, based on the user's acceptance of a marker sharing instruction (not shown), the image processing device 811 sends data of the marker object, converted from the input marker in the manner described above, to the management server 801. Furthermore, as... Figure 10B As shown, assuming that in the image processing device 812, for the purpose of... Figure 10A The virtual viewpoint image with the same timecode accepts a marker with a shape having curve 1002. Based on the user's acceptance of a marker sharing instruction (not shown), the image processing device 812 sends a marker object, converted from the input marker in the manner described above, to the management server 801. Note that the marker object is associated with a timecode and sent for storage in the management server 801. For this purpose, the image processing devices 811 to 813 have storage control functions for storing the timecode and marker object in association in the management server 801.

[0081] Assuming that in this state, image processing device 813 receives a marker update instruction (not shown). In this case, regarding the image processing devices 811 and 812, as... Figure 10A and Figure 10B The image processing device 813 obtains the tag object corresponding to the time code from the management server 801, based on the input time code shown. Then, as... Figure 10C As shown, the image processing device 813 uses the 3D model of the photographed subject and the obtained marker objects to generate a virtual viewpoint image. Note that the obtained marker objects here are 3D data arranged in the same virtual space as the 3D model of the photographed subject, as described above. Therefore, even on the image processing device 813 with a shared destination, the 3D model of the photographed subject and the marker objects are arranged in the same positional relationship as when input to the image processing devices 811 and 812 with shared sources. Therefore, even if the position and viewing direction of the virtual viewpoint are arbitrarily manipulated in the image processing device 813, the virtual viewpoint image is generated while maintaining their positional relationship.

[0082] Note that, for example, upon receiving a marker update instruction, image processing device 811 can obtain information about the marker object of the marker (curve 1002) input in image processing device 812. Then, image processing device 811 can draw a virtual viewpoint image of the obtained marker object, in addition to the marker object of the marker (circle 1001) input in its own device and the three-dimensional model of the photographed subject.

[0083] Note that, for example, if the image processing device 813 receives a tag update instruction, the tag objects managed in the management server 801 can be displayed in a list such as thumbnails, and the tag object to be downloaded can be selected. In this case, if the number of tag objects managed in the management server 801 is large, tag objects near the timecode that received the update instruction can be given higher priority and displayed in a list such as thumbnails. That is, at the moment when the image processing device 813 receives a tag update instruction, in order to facilitate the selection of tags entered near the timecode corresponding to that moment, the tag can be displayed first. In addition, the number of times a download instruction is received can be managed for each tag object, and tag objects with a larger count can be given higher priority and displayed in a list such as thumbnails. Note that thumbnails with high display priority can be displayed near the top of the list, or can be displayed in a large size.

[0084] In this manner, the markers input for the virtual viewpoint image can be shared by multiple devices while maintaining the positional relationship between the photographed subject and the markers. As described above, the markers have a format of three-dimensional objects in virtual space. Therefore, even if the position or viewing direction of the virtual viewpoint changes between the sharing source device and the sharing destination device, a virtual viewpoint image can be generated while maintaining the positional relationship between the photographed subject and the markers.

[0085] (Control of the transparency of the markers based on the timecode)

[0086] If the subject being photographed moves over time, the positional relationship between the subject and the marker object displayed as described above changes. In this case, if the marker object remains displayed, the user may experience discomfort. To avoid this, the transparency of the marker object is increased as time passes from the timecode entered, thereby showing the user the change from the original point of interest and reducing discomfort. As described above, the marker object is maintained in association with the timecode. Here, it is assumed that in... Figure 10C In the example shown, the user performs operation 751 on the timecode. In this case, as... Figure 11A As shown, the image processing device 813 can change the transparency (α value) of the marker as the timecode from the generated timecode of the marker object transitions to separate timecodes. Therefore, drawing is performed such that the marker is displayed in a lighter color as the timing deviation relative to the actual marker input increases. This reduces user discomfort. Note that if the user returns the timecode, then... Figure 10C The diagram shows controls for returning to the original transparency. Note that the marker object can be drawn even in a virtual viewpoint image corresponding to a timecode preceding its generation timecode. Similarly, in this case, if the difference between the timecode of the marker object's generation and the timecode of the virtual viewpoint image to be drawn is large, controls can be made to increase the transparency (displaying the marker object in a lighter color). Additionally, controls can be made to make the transparency before the marker object's generation timecode higher than the transparency after the generation timecode (displaying the marker object in a lighter color).

[0087] Note that the examples above illustrate how transparency changes with the timecode transition. However, the invention is not limited to this. For example, control can be implemented to display the marker in a lighter color by changing at least one of the marker object's brightness, color saturation, and chroma.

[0088] (Controlled by the marker object based on the coordinates of the 3D model)

[0089] In addition to or instead of the embodiments described above, the image processing device may accept operations for adding markers to a 3D model of the foreground. For example, such as Figure 11B As shown, the 3D model of the foreground (figure) accepts the addition of indicator 1101. In this case, for example, as... Figure 11C As shown, the image processing device generates a circular marker object 1102, which has a radius of a predetermined size centered on the X and Y coordinates of the plane of interest of the three-dimensional model. Therefore, it is possible to generate a circular marker object with a reference... Figures 5A to 5E The described marker object is the same as the marker object. Note that the shape of the generated marker object is not limited to a circle when an instruction to add a marker object to the foreground 3D model is received (marker addition). The shape of the marker object in this case can be a rectangle, or other shapes of the 3D model that can be of interest.

[0090] Note that the position of markers added to the 3D model of the foreground can change as the figure or ball in that foreground moves. Note that even in the reference... Figures 5A to 5E In the described marker objects, the marker objects can also be associated with the 3D model of the location where the marker was added or the surrounding foreground, and the position of the marker object can change according to the position of the 3D model in the foreground. That is, independent of the method of adding markers, the coordinates of the 3D model in the foreground can be obtained for each timecode, and the coordinates of the marker object can be changed for each timecode. Therefore, if figures or other objects in the foreground move as timecodes pass, the marker object can move along with the 3D model. Alternatively, for example, the position of the marker object can be changed by user operation based on changes in timecodes. In this case, the position of the marker object can be specified for each timecode based on user operation, and this position can be stored / managed.

[0091] Note that the above embodiments describe the process of adding markers to a virtual viewpoint image based on multi-viewpoint images captured by multiple camera devices. However, the invention is not limited to this. That is, for example, even if markers are added to a virtual viewpoint image generated based on a three-dimensional virtual space created entirely manually on a computer, the markers can be converted into three-dimensional objects in that virtual space. Furthermore, the above embodiments describe an example of generating and storing marker objects associated with timecodes corresponding to the virtual viewpoint image with markers. However, timecodes do not always need to be associated with marker objects. For example, if the virtual viewpoint image is a still image, or if the virtual viewpoint image is only used for the purpose of temporarily adding markers in a meeting, the marker objects can be displayed or erased by, for example, user operation, regardless of the timecode. Furthermore, at least some of the image processing devices used in image processing devices such as conference systems do not need to have the ability to specify virtual viewpoints. That is, after markers are added to the virtual viewpoint image, only a specific user, such as a meeting facilitator, needs to be able to specify the virtual viewpoint. Image processing devices maintained by other users do not need to accept virtual viewpoint operations. Similarly, in this case, since the marked objects are drawn according to the virtual viewpoint specified by the specific user, it is possible to prevent the relationship between the marked objects and the photographed subject in the virtual viewpoint image from becoming inconsistent.

[0092] [Other Embodiments]

[0093] In the above embodiments, an example of displaying marked objects as supplementary information on a virtual viewpoint image was described. However, the supplementary information displayed on the virtual viewpoint image is not limited to this. For example, at least one of the following, such as user-specified marks, icons, avatars, and diagrams, can be displayed as supplementary information on the virtual viewpoint image. Multiple pieces of supplementary information can be prepared in advance, and the user can select any one of these pieces of supplementary information and arrange it on the virtual viewpoint image. The user can drag icons, etc., on a touch panel display and arrange them at any position. The arranged supplementary information, such as icons, is converted into three-dimensional data using the same method as in the above embodiments. Note that the method is not limited to this, and the supplementary information of two-dimensional data and the supplementary information of three-dimensional data can be associated in advance, and the supplementary information can be converted into the corresponding supplementary information of three-dimensional data at the timing of arranging the supplementary information of two-dimensional data. As described above, this embodiment can be applied to situations where various kinds of supplementary information are displayed on a virtual viewpoint image.

[0094] Furthermore, in the above embodiments, the additional information (tagged objects) is converted into three-dimensional data. This three-dimensional data does not always need to represent a three-dimensional shape. That is, the three-dimensional data is data that at least has a three-dimensional position in virtual space, and the shape of the additional information can be a surface, line, or point.

[0095] Furthermore, it is not necessary to provide all the functions described in the above embodiments, and the embodiments can be performed by combining any functions.

[0096] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0097] While this disclosure has been described with reference to exemplary embodiments, it should be understood that this disclosure is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An image processing apparatus, comprising: A generation component is used to generate a first virtual viewpoint image based on a virtual viewpoint and the virtual space by generating a three-dimensional model of a first object included in images captured by multiple camera devices and by arranging the first object in a virtual space. Display control unit, used to display a first screen containing the first virtual viewpoint image on a display screen; The component is used to obtain user input for drawing lines on the first screen; as well as A conversion component is used to convert the drawn lines into a three-dimensional form of a second object, and to arrange the three-dimensional form of the second object in the virtual space, wherein the second object is not included in the images captured by the plurality of camera devices. The generation component generates a second virtual viewpoint image based on the virtual space in which the first object and the second object are arranged. When the three-dimensional form of the second object exists within the field of view based on the virtual viewpoint, the second virtual viewpoint image includes the second object. The display control unit displays a second screen containing the second virtual viewpoint image on the monitor.

2. The image processing apparatus according to claim 1, wherein, The conversion component performs the conversion, thereby generating a three-dimensional form of the second object on a predetermined surface in the virtual space.

3. The image processing apparatus according to claim 2, wherein, The conversion component performs the conversion such that at the intersection of the line passing through the virtual viewpoint corresponding to the first virtual viewpoint image when the line is drawn on the first screen and the point of the line drawn with respect to the predetermined surface, a point in the three-dimensional form of the second object corresponding to the point of the line drawn is generated.

4. The image processing apparatus according to claim 1, wherein, When the three-dimensional form of the second object does not exist within the field of view based on the virtual viewpoint, the display control unit displays a second virtual viewpoint image that does not include the second object.

5. The image processing apparatus according to claim 1, wherein, The timecode corresponding to the first virtual viewpoint image when the line is drawn on the first screen is associated with the second object, and When displaying the second virtual viewpoint image corresponding to the timecode associated with the second object, the display control unit displays the second virtual viewpoint image based on a virtual space in which the first object and the second object are arranged.

6. The image processing apparatus according to claim 5, further comprising: A storage control component is used to store the timecode and the second object in association with each other in a storage component.

7. The image processing apparatus according to claim 1, wherein, The display control unit displays a third virtual viewpoint image based on the virtual space, wherein in the virtual space, the three-dimensional forms of other objects based on other lines drawn by other image processing devices are arranged at three-dimensional positions.

8. The image processing apparatus according to claim 1, wherein, The acquiring component obtains information representing the virtual viewpoint specified by the user.

9. The image processing apparatus according to claim 8, wherein, The information representing the virtual viewpoint includes information representing the position of the virtual viewpoint and the direction of the line of sight from the virtual viewpoint.

10. The image processing apparatus according to claim 8, wherein, When the user performs a first operation, the obtaining component obtains line-drawing input corresponding to the first operation, and when the user performs a second operation different from the first operation, the obtaining component obtains information representing the virtual viewpoint specified according to the second operation.

11. The image processing apparatus according to claim 10, wherein, The first operation and the second operation are operations on a touch panel. The first operation is an operation of the user's finger on the touch panel, and the second operation is an operation using a drawing device.

12. The image processing apparatus according to claim 1, wherein, The display control unit controls the display to show the second object in a light color when the difference between the timecode corresponding to the first virtual viewpoint image with the drawn line and the timecode of the second virtual viewpoint image to be displayed is large.

13. The image processing apparatus according to claim 12, wherein, The display control component controls the display of the second object to be in a light color by changing at least one of the transparency, brightness, color saturation, and chroma of the second object.

14. The image processing apparatus according to claim 1, further comprising a designating component, the designating component being used to designate a three-dimensional model of a specific object corresponding to the position of the line drawn in the first virtual viewpoint image. in, The display control unit displays a second virtual viewpoint image of the second object, which is arranged corresponding to the line, at a three-dimensional position corresponding to the three-dimensional model of the specific object in the virtual space.

15. An image processing method performed by an image processing device, comprising: A first virtual viewpoint image is generated based on a virtual viewpoint and the virtual space by generating a three-dimensional model of a first object included in images captured by multiple camera devices and by arranging the first object in a virtual space. The first screen displaying the first virtual viewpoint image will be shown on the monitor; Obtain user input for drawing lines on the first screen; The drawn lines are converted into a three-dimensional form of a second object, and the three-dimensional form of the second object is arranged in the virtual space, wherein the second object is not included in the images captured by the plurality of camera devices; A second virtual viewpoint image is generated based on the virtual space in which the first object and the second object are arranged. The second virtual viewpoint image includes the second object if the three-dimensional form of the second object exists within the field of view based on the virtual viewpoint. A second screen displaying the second virtual viewpoint image will be shown on the display.

16. A computer-readable storage medium storing a program for implementing the image processing method according to claim 15 when executed by a computer.

17. A computer program product comprising a program for implementing the image processing method according to claim 15 when executed by a computer.

Images