Image processing device, image processing method, and program

Abstract

A virtual viewpoint image is generated using an image acquisition device that captures images for generating virtual viewpoint images, and the image obtained from the image is used to generate a virtual viewpoint image that includes an image representing the trajectory of the object's three-dimensional shape. [Solution] The image processing apparatus 102 according to the present disclosure acquires shape data indicating the three-dimensional shape of each object present in the target space, identifies a target three-dimensional shape whose size is within a given range from among the three-dimensional shapes indicated by the shape data for each object, generates trajectory information indicating the trajectory of the target three-dimensional shape by associating information regarding the position of the identified target three-dimensional shape with a corresponding time code, acquires virtual viewpoint information indicating the position of a virtual viewpoint and the direction of the line of sight at the virtual viewpoint, and generates a virtual viewpoint image corresponding to the view from the virtual viewpoint, which includes an image representing the trajectory of the target three-dimensional shape, based on the shape data, the trajectory information, and the virtual viewpoint information.

Description

[Technical Field]

[0001] The present disclosure relates to a technology for generating virtual viewpoint images. [Background Art]

[0002] There is a technology for generating an image corresponding to a view from an arbitrary virtual viewpoint specified by a user or the like (hereinafter referred to as a "virtual viewpoint") (hereinafter referred to as a "virtual viewpoint image") based on a plurality of captured images obtained by imaging with a plurality of imaging devices. According to such a virtual viewpoint image, viewing from a viewpoint that cannot be achieved by imaging with an imaging device can be realized when watching sports or the like. Furthermore, in ball games such as baseball, it has been practiced to identify the trajectory of a ball and superimpose it on broadcast video obtained from captured images. [Prior Art Literature] [Patent Literature]

[0003] [Patent Literature 1] Japanese Unexamined Patent Application Publication No. 2017-211828 [Summary of Invention] [Problem to be Solved by the Invention]

[0004] However, when drawing and superimposing the trajectory of a target object such as a ball on a virtual viewpoint image, conventional methods require an imaging device for specifying the trajectory of the target object separately from the imaging device for generating the virtual viewpoint image. [Means for Solving the Problem]

[0005] The image processing apparatus according to this disclosure includes: shape acquisition means for acquiring shape data indicating the three-dimensional shape of each object present in a target space; identification means for identifying a target three-dimensional shape whose size is within a given range from among the three-dimensional shapes indicated by the shape data for each object, and generating trajectory information indicating the trajectory of the target three-dimensional shape by associating information regarding the position of the identified target three-dimensional shape with a corresponding time code; viewpoint acquisition means for acquiring virtual viewpoint information indicating the position of a virtual viewpoint and the direction of the line of sight at the virtual viewpoint; and image generation means for generating a virtual viewpoint image corresponding to the view from the virtual viewpoint, which includes an image representing the trajectory of the target three-dimensional shape, based on the shape data, the trajectory information, and the virtual viewpoint information. [Brief explanation of the drawing]

[0006] [Figure 1] This is a block diagram showing an example of the configuration of an image processing system according to the first embodiment. [Figure 2] This is a block diagram showing an example of the logical configuration of an image processing device according to the first embodiment. [Figure 3] This is a block diagram showing an example of the hardware configuration of an image processing device according to the first embodiment. [Figure 4] This flowchart shows an example of the processing flow of the image processing apparatus according to the first embodiment. [Figure 5] This flowchart shows an example of the flow of specific processing in the specific unit according to the first embodiment. [Figure 6] This figure illustrates an example of a rule used in the specific processing in the specific unit according to the first embodiment. [Figure 7] This flowchart shows an example of the flow of the object identification process in the specific unit according to the first embodiment. [Figure 8] This figure shows an example of a virtual viewpoint image generated by the image generation unit according to the first embodiment. [Figure 9] This figure shows an example of a GUI according to a modified version of the first embodiment. [Figure 10] This figure shows an example of a virtual viewpoint image according to a modified example of the first embodiment. [Modes for carrying out the invention]

[0007] [First Embodiment] (Configuration of the image processing system) Figure 1 is a block diagram showing an example of the configuration of an image processing system according to the first embodiment. The image processing system includes a plurality of imaging devices 100, a synchronization device 101, an image processing device 102, and a display device 103. The image processing system may have one image processing device 102 or a plurality of image processing devices 102. In the following description, the image processing system will be described as having one image processing device 102.

[0008] Multiple imaging devices 100 are composed of digital video cameras or the like, and they capture images of the space to be captured (hereinafter referred to as the "imaging area") from different directions in synchronous manner. A synchronization device 101 transmits a synchronization signal to each of the multiple imaging devices 100. Specifically, each of the multiple imaging devices 100 receives the synchronization signal from the synchronization device 101 and performs imaging based on the received synchronization signal. The data of multiple captured images (hereinafter referred to as "multi-view images") obtained by the synchronized imaging by the multiple imaging devices 100 is output to the image processing device 102. The multiple imaging devices 100 are arranged, for example, to surround the imaging area so that objects present in the imaging area (hereinafter simply referred to as "objects") can be captured from multiple directions.

[0009] The image processing device 102 acquires data from multi-view images output from multiple imaging devices 100, generates a virtual viewpoint image corresponding to the view from an arbitrary virtual viewpoint, and outputs data of a display image including the generated virtual viewpoint image to the display device 103. The display device 103 is composed of a liquid crystal display or the like, acquires data of the display image output from the image processing device 102, and displays the said display image.

[0010] (Configuration of the image processing device) Figure 2 is a block diagram showing an example of the logical configuration of the image processing device 102 according to the first embodiment. The image processing device 102 has an image acquisition unit 201, a shape estimation unit 202, a viewpoint acquisition unit 203, an image generation unit 204, a specific unit 205, and an output control unit 206 as its logical configuration. The processing of each part of the logical configuration of the image processing device 102 is performed by processing hardware such as an ASIC (Application Specific Integrated Circuit) built into the image processing device 102. This processing may also be performed by software using a computing device such as a CPU (Central Processor Unit) or GPU (Graphics Processor Unit) and memory built into the image processing device 102. Details of the processing of each part of the logical configuration of the image processing device 102 will be described later.

[0011] Referring to Figure 3, the hardware configuration of the image processing device 102 will be described when the processing of each part of the logical configuration of the image processing device 102 is performed by software execution. Figure 3 is a block diagram showing an example of the hardware configuration of the image processing device 102 according to the first embodiment. The image processing device 102 is composed of a computer, and the computer has a hardware configuration of a CPU 301, a GPU 302, a ROM 303, a RAM 304, a VRAM 305, and an auxiliary storage device 306. The computer also has a hardware configuration of a display unit 307, an operation unit 308, a communication unit 309, and a bus 310.

[0012] The CPU 301 controls the computer using programs and data stored in the ROM 303 or auxiliary storage device 306, etc., thereby causing the image processing device 102 to function as each part of its logical configuration. The ROM 303 stores programs and various data that do not require modification. The auxiliary storage device 306 is composed of, for example, a hard disk drive, and stores programs and various data such as image data or audio data. The RAM 304 operates as the work area of ​​the CPU 301 and temporarily stores programs and data supplied from the ROM 303 or auxiliary storage device 306, or data supplied from the outside via the communication unit 309.

[0013] The GPU 302 controls the computer in cooperation with the CPU 301 using programs or data stored in the ROM 303 or auxiliary storage device 306, thereby enabling the image processing device 102 to function as a logical component. The VRAM 305 is memory for graphics processing, operates as the work area of ​​the GPU 302, and temporarily stores programs and data supplied from the ROM 303 or auxiliary storage device 306. The image processing device 102 may have one or more dedicated processing hardware components separate from the CPU 301 and GPU 302, and these processing hardware components may execute at least a portion of the processing performed by the CPU 301 or GPU 302. Examples of dedicated processing hardware include ASICs, FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors).

[0014] The display unit 307 is constituted by a liquid crystal display, an LED (light-emitting diode), or the like. The display unit 307 displays a GUI (Graphical User Interface) for a user to operate the image processing apparatus 102, a GUI for checking the status of the image processing apparatus 102, or the like. The operation unit 308 is constituted by a keyboard, a mouse, a touch panel, or the like, and receives an operation performed by a user and inputs various instructions corresponding to the operation to the CPU 301. The CPU 301 also operates as a display control unit that controls the display unit 307 and an operation control unit that controls the operation unit 308.

[0015] The communication unit 309 is used for communication between the image processing apparatus 102 and an external apparatus. For example, when the image processing apparatus 102 is connected to an external apparatus via a wired connection, a communication cable is connected to the communication unit 309. When the image processing apparatus 102 has a function of performing wireless communication with an external apparatus, the communication unit 309 includes an antenna. A bus 310 communicably connects the above-described hardware configurations of the image processing apparatus 102 and transmits information. Hereinafter, the description is given on the assumption that the display unit 307 and the operation unit 308 are provided inside the image processing apparatus 102, but at least one of the display unit 307 and the operation unit 308 may be provided as a separate apparatus outside the image processing apparatus 102.

[0016] The image acquisition unit 201 acquires data of multi-viewpoint images output from the plurality of imaging apparatuses 100. The image acquisition unit 201 also acquires camera parameters when each of the plurality of imaging apparatuses 100 captures each captured image constituting the multi-viewpoint images. The camera parameters include external parameters related to the position and orientation of the imaging apparatus that captured the captured image, and internal parameters related to focal length, principal point, resolution, and the like. Note that the camera parameters may be stored in advance in the auxiliary storage device 306 or the like, or may be output from each imaging apparatus 100. The data of the multi-viewpoint images acquired by the image acquisition unit 201 and the camera parameters corresponding to each captured image obtained by imaging by each imaging apparatus or the plurality of imaging apparatuses are stored in the RAM 304, the auxiliary storage device 306, or the like.

[0017] The shape estimating unit 202 estimates the three-dimensional shape of an object by using the multi-viewpoint image acquired by the image acquiring unit 201 and the camera parameters corresponding to each captured image constituting the multi-viewpoint image. Specifically, for example, the shape estimating unit 202 first extracts a region including the image of the object in each captured image as a silhouette by performing foreground-background separation on each captured image. Since foreground-background separation techniques are well-known, description thereof will be omitted. It should be noted that when foreground-background separation is performed by a background subtraction method or the like, it is assumed that background image data corresponding to each captured image is stored in advance in the RAM 304, the auxiliary storage device 306, or the like. Subsequently, the shape estimating unit 202 estimates the three-dimensional shape of the object by a technique such as the visual hull intersection method using the camera parameters of each imaging device 100 acquired by the image acquiring unit 201. Since the visual hull intersection method is well-known, description thereof will be omitted. Data relating to the three-dimensional shape of the object estimated by the shape estimating unit 202 is output to and stored in the auxiliary storage device 306 or the like. Here, the object is a natural person, an article handled by a natural person, or the like that is an object for three-dimensional shape estimation.

[0018] The viewpoint acquiring unit 203 acquires information related to a virtual viewpoint (hereinafter referred to as "virtual viewpoint information"). The virtual viewpoint information includes information on the position of the virtual viewpoint corresponding to the external parameters of the imaging device and the direction of the line of sight at the virtual viewpoint, information on the focal length, principal point, and the like corresponding to the internal parameters, and information on the time code of the virtual viewpoint image to be generated. Hereinafter, the direction of the line of sight at the virtual viewpoint is referred to as "posture of the virtual viewpoint". Specifically, for example, a user operates the operation unit 308 to input the position and posture of the virtual viewpoint, the focal length and principal point at the virtual viewpoint, the time code of the virtual viewpoint image to be generated, and the like. The viewpoint acquiring unit 203 acquires the virtual viewpoint information by receiving a signal corresponding to the input from the operation unit 308.

[0019] The image generation unit 204 generates a virtual viewpoint image. Specifically, the image generation unit 204 first acquires the necessary material data for generating a virtual viewpoint image corresponding to a time code, based on the time code included in the virtual viewpoint information acquired by the viewpoint acquisition unit 203, by reading it from the auxiliary storage device 306 or the like. Here, the material necessary for generating a virtual viewpoint image includes, for example, data relating to the three-dimensional shape of an object, captured image data, and camera parameters corresponding to the time code, as well as data for a background model and background texture image.

[0020] Next, the image generation unit 204 uses the acquired material data to generate a virtual viewpoint image corresponding to the view from the virtual viewpoint indicated by the virtual viewpoint information. The method for generating a virtual viewpoint image corresponding to an arbitrary virtual viewpoint using data related to the three-dimensional shape of an object, captured image data, camera parameters, and background model and background texture image data is well known, so its explanation will be omitted. The virtual viewpoint image data generated by the image generation unit 204 is output to the output control unit 206. The image generation unit 204 also draws the trajectory of a predetermined object, such as a ball, on the virtual viewpoint image based on the trajectory information described later, but the details of this drawing process will be described later.

[0021] The identification unit 205 uses the three-dimensional shape data estimated by the shape estimation unit 202 to identify a predetermined object, such as a ball, present in the imaging area. Details of the identification process in the identification unit 205 will be described later. The result of the identification of the predetermined object by the identification unit 205 is associated with an identifier that can uniquely identify the object and stored as trajectory information in the auxiliary storage device 306 or the like. The output control unit 206 generates a display image including the virtual viewpoint image generated by the image generation unit 204, outputs the data of the generated display image to the display device 103, and displays the display image on the display device 103.

[0022] (Operation of the image processing device) An example of the operation of the image processing device 102 will be described with reference to Figures 4 and 5. Figure 4 is a flowchart showing an example of the processing flow of the image processing device 102 according to the first embodiment. In the following description, the symbol "S" means step. First, in S401, the image acquisition unit 201 acquires multi-view image data and camera parameters corresponding to each captured image that makes up the multi-view image. The multi-view image data and camera parameters acquired in S401 are stored in the auxiliary storage device 306, etc. Next, in S402, the shape estimation unit 202 estimates the three-dimensional shape of the object using the multi-view image data acquired in S401 and the camera parameters corresponding to each captured image. The three-dimensional shape data estimated in S402 is stored in the auxiliary storage device 306, etc. Next, in S403, the viewpoint acquisition unit 203 acquires virtual viewpoint information. Next, in S404, the identification unit 205 executes the identification process described later.

[0023] (Specific processing in a specific section) Figure 5 is a flowchart showing an example of the flow of the identification process in the identification unit 205 according to the first embodiment, that is, the identification process in S404 shown in the flowchart of Figure 4. In this embodiment, as an example, the imaging target is a baseball game, and the object of identification (hereinafter referred to as the "target object") is a baseball used in a baseball game.

[0024] In the processing of S404, first, in S501, the identification unit 205 acquires data for all three-dimensional shapes that correspond to the period indicated by the time code included in the virtual viewpoint information acquired in S403, from among the three-dimensional shapes obtained by the estimation processing in S402. Specifically, the identification unit 205 acquires data for all three-dimensional shapes that meet the above conditions by reading them from the auxiliary storage device 306, etc.

[0025] Next, in S502, the identification unit 205 identifies the three-dimensional shape corresponding to the target object from among the three-dimensional shapes acquired in S501. Specifically, the identification unit 205 determines whether the size of each three-dimensional shape acquired in S501 is within a predetermined range, and identifies the three-dimensional shape whose size is determined to be within that range as the three-dimensional shape corresponding to the target object (ball). Here, the size of the three-dimensional shape is at least one of the volume, surface area, and external dimensions of the three-dimensional shape. For example, if the data of the three-dimensional shape is represented by a dense point cloud, the identification unit 205 can obtain the approximate volume of the object based on the number of points included in the point cloud. Also, if the data of the three-dimensional shape is represented by a point cloud indicating the surface shape of the object, the identification unit 205 can obtain the approximate surface area of ​​the object based on the number of points included in the point cloud of the three-dimensional shape. Regarding the external dimensions, at least one of the width, depth, and height of the three-dimensional shape can be used.

[0026] Next, in S503, the identification unit 205 obtains coordinates indicating the position of each of the three-dimensional shapes corresponding to the target object identified in S502 (hereinafter referred to as "coordinates of the three-dimensional shape"). Specifically, for example, the identification unit 205 obtains coordinates corresponding to the position of the centroid or center of the three-dimensional shape as the coordinates of the three-dimensional shape. The coordinates of the three-dimensional shape are not limited to coordinates corresponding to the position of the centroid or center of the three-dimensional shape, but may also be, for example, coordinates indicating the position of the bounding box containing the three-dimensional shape.

[0027] Next, in S504, the identification unit 205 identifies candidates for three-dimensional shapes corresponding to the same target object based on the coordinates and time codes of the three-dimensional shapes corresponding to each target object acquired in S503. Specifically, for example, the identification unit 205 determines whether the coordinates acquired in S503 at an arbitrary point in time and a point in time close to that point in time are within a predetermined range within the period indicated by the time code. If it is determined that the coordinates at the two points in time are within the predetermined range, the identification unit 205 identifies the two three-dimensional shapes at the two points in time as candidates for three-dimensional shapes corresponding to the same target object. The process in S504 is repeatedly executed for the entire period indicated by the time code. Through this repeated process, candidates for time-series three-dimensional shapes corresponding to the same target object are identified for the entire period indicated by the time code.

[0028] Next, in S505, the identification unit 205 acquires the direction of movement and velocity for each candidate three-dimensional shape corresponding to the same target object identified in S504. Specifically, the identification unit 205 calculates the direction of movement and velocity of the three-dimensional shape of interest based on the difference between the coordinates and timecode of the three-dimensional shape of interest and the coordinates and timecode of the three-dimensional shape with a timecode close to the timecode of the three-dimensional shape of interest. The method for acquiring the direction of movement and velocity of the three-dimensional shape is not limited to this. For example, the identification unit 205 may calculate the direction of movement and velocity of the three-dimensional shape of interest based on the difference between the coordinates and timecodes of the three-dimensional shapes with timecodes before and after the timecode of the three-dimensional shape of interest.

[0029] Next, in S506, the identification unit 205 identifies a three-dimensional shape corresponding to the same target object from among the candidate three-dimensional shapes identified in S504, based on at least one of the direction of movement and velocity acquired in S505, according to a predetermined rule. The identification unit 205 also assigns an identifier to each identified three-dimensional shape corresponding to the same target object that can uniquely identify the target object.

[0030] In this embodiment, in S506, the identification unit 205 identifies a three-dimensional shape corresponding to the same target object from among the candidates for three-dimensional shape based on at least one of the direction of movement and the speed. However, the method for identifying a three-dimensional shape corresponding to the same target object is not limited to this. For example, the identification unit 205 may identify a three-dimensional shape corresponding to the same target object based on the coordinates and time code of the three-dimensional shape corresponding to each target object acquired in S503. That is, the identification unit 205 may identify the candidates for three-dimensional shapes corresponding to the same target object identified in S504 as the three-dimensional shape corresponding to the same target object, rather than treating them as candidates.

[0031] Next, in S507, the identification unit 205 stores the time code, coordinates, direction of movement, velocity, and identifier information for each three-dimensional shape corresponding to the same target object identified in S506 in the auxiliary storage device 306 or the like as trajectory information indicating the trajectory of the target object. After S507, the identification unit 205 completes the processing shown in the flowchart in Figure 5, that is, the processing shown in S404 in Figure 4.

[0032] (Continuation of the explanation of the image processing device's operation) Following S404, in S405, the image generation unit 204 generates a virtual viewpoint image based on the multi-view image data and camera parameters acquired in S401, the three-dimensional shape data estimated in S402, and the virtual viewpoint information acquired in S403. Details of the virtual viewpoint image generation process in S405 will be described later. Next, in S406, the output control unit 206 generates a display image including the virtual viewpoint image generated in S405, outputs the data of the generated display image to the display device 103, and displays the display image on the display device 103. After S406, the image processing device 102 completes the process shown in the flowchart in Figure 4.

[0033] In this embodiment, a configuration has been described in which data of multi-view images obtained by imaging over the entire period of the processing target is pre-stored in an auxiliary storage device 306 or the like before processing in S404. However, the scope of application of the technology of this disclosure is not limited to this. For example, the technology of this disclosure can also be applied when each imaging device 100 continuously captures moving images and continuously outputs frame data sequentially. In this case, for example, the image processing device 102 repeatedly executes the flowchart shown in Figure 4 each time frame data is output from each imaging device 100. Specifically, in this case, each time frame data is output from each imaging device 100, frame data is acquired in S401, and processing from S402 onwards is executed for data relating to the latest time code.

[0034] Referring to Figures 6 and 7, the above-mentioned rules for when the identification unit 205 identifies the three-dimensional shape corresponding to the same target object will be explained. Figure 6 is a diagram illustrating an example of the rules used in the identification process in the identification unit 205 according to the first embodiment. Specifically, Figure 6 shows, as an example, an overhead view of a baseball field when the target of imaging is a baseball game. In the baseball field, which is the imaging area, three areas 601 to 603, shown as an example in Figure 6, are pre-set as identification areas. The identification unit 205 identifies the target object moving in a predetermined direction in each of the three areas 601 to 603. The specific identification process will be explained with reference to Figure 7.

[0035] Figure 7 is a flowchart showing the flow of the object identification process in the identification unit 205 according to the first embodiment, that is, an example of the processing flow of S506 shown in Figure 5. First, in S701, the identification unit 205 identifies a three-dimensional shape that moves in a predetermined direction (direction indicated by arrow 611) in the first region (region 601 shown in Figure 6) from among the candidates for three-dimensional shapes corresponding to the same target object identified in S504. Next, in S702, the identification unit 205 assigns an identifier corresponding to a pitch to the three-dimensional shape identified in S701, assuming that the three-dimensional shape corresponds to a ball thrown by a pitcher (hereinafter referred to as "a pitch").

[0036] The determination of whether the three-dimensional shape in S701 exists in region 601 is performed based on the coordinates of the three-dimensional shape obtained in S503. The determination of the direction of movement in S701 is performed by comparing the direction of movement obtained in S505 with the vector or angle indicating arrow 611, and is based on whether the difference between the direction of movement and the vector or angle is within a predetermined range. In addition, conditions may be added in the specific processing of S701, such as determining that only three-dimensional shapes whose speed is in the speed range of 140 km / h to 160 km / h are three-dimensional shapes corresponding to a pitch.

[0037] Following S702, in S703, the identification unit 205 identifies a three-dimensional shape from among the candidates for three-dimensional shapes corresponding to the same target object identified in S504 that moves in a direction in which both the x-axis and y-axis directions are positive in the second region (region 602 shown in Figure 6). Specifically, the identification process in S703 may be performed, for example, on three-dimensional shapes other than the three-dimensional shape corresponding to the pitch from among the candidates for three-dimensional shapes corresponding to the same target object identified in S504. Note that the method for determining whether a three-dimensional shape exists in region 602 in S703, and the method for determining the direction of movement in S703 are the same as the determination method in S701, so an explanation is omitted.

[0038] Next, in S704, the identification unit 205 assigns an identifier to the three-dimensional shape identified in S703, assuming that it is the three-dimensional shape corresponding to the ball hit back by the batter (hereinafter referred to as "hit ball"). In addition, in the processing of S704, the identification unit 205 also assigns an identifier to the hit ball, assuming that all temporally continuous trajectories of the three-dimensional shape, even after the three-dimensional shape corresponding to the hit ball has moved from inside to outside the region 602, are the three-dimensional shapes corresponding to the hit ball.

[0039] Next, in S705, the identification unit 205 identifies a three-dimensional shape that exists in the third region (region 603 shown in Figure 6) from among the candidate three-dimensional shapes corresponding to the same target object identified in S504, excluding three-dimensional shapes other than pitches and batted balls. The method for determining whether or not the three-dimensional shape in S705 exists in region 603 is the same as the method for determining in S701, so the explanation is omitted. Next, in S706, the identification unit 205 assigns an identifier corresponding to a throw to the three-dimensional shape identified in S705, assuming that the three-dimensional shape corresponds to a ball thrown by a fielder (hereinafter referred to as a "throw").

[0040] Next, in S707, the identification unit 205 excludes from the candidates for three-dimensional shapes corresponding to the same target object identified in S504 any three-dimensional shapes other than pitches, batted balls, and throws. Specifically, the identification unit 205 deletes the data related to the excluded three-dimensional shapes from the auxiliary storage device 306, etc. The identification unit 205 may also assign an identifier to the excluded three-dimensional shapes to indicate that they are excluded three-dimensional shapes. After S707, the identification unit 205 completes the processing shown in the flowchart in Figure 7, that is, the processing shown in S506 in Figure 5.

[0041] In the estimation process of the shape estimation unit 202 at a specific point in time, the shape of the target object (ball) may not be estimated if it comes into contact with a white line or the like on the field, resulting in the loss of data for the three-dimensional shape corresponding to the target object. Hereinafter, the specific point in time mentioned above will be referred to as the "specific point in time." Here, the ball moves according to the laws of physics. Therefore, for example, the approximate position (coordinates) of the three-dimensional shape at the specific point in time can be estimated based on the coordinates, direction of movement, and velocity of the three-dimensional shape corresponding to the point in time immediately preceding the specific point in time, as well as the acceleration due to gravity. On the other hand, if the ball is caught by a fielder or the like, or bounces off the field surface or fence, the estimated position at the specific point in time may change significantly. Therefore, the specific unit 205 determines that data for the three-dimensional shape corresponding to the target object at the specific point in time is missing if the three-dimensional shape corresponding to the target object was accurately estimated at the points immediately preceding and immediately following the specific point in time.

[0042] If the identification unit 205 determines that data for the three-dimensional shape corresponding to the target object is missing, it performs the following processing, for example. For example, in this case, the identification unit 205 estimates and interpolates the position of the target object at the specified time based on the coordinates, direction of movement, and velocity of the three-dimensional shape corresponding to the target object at at least one of the points immediately before and immediately after the specified time, as well as the acceleration due to gravity. Specifically, the identification unit 205 estimates the position to which the three-dimensional shape can move in a period of one frame relative to the coordinates of the three-dimensional shape, based on the direction of movement, velocity, and acceleration due to gravity of the three-dimensional shape corresponding to the target object at the immediately preceding time. The method of interpolating the missing data for the three-dimensional shape corresponding to the target object is not limited to this. For example, the identification unit 205 may estimate an intermediate position between the positions of the three-dimensional shape corresponding to the target object at the points immediately before and immediately after the specified time as the position of the three-dimensional shape corresponding to the target object at the specified time.

[0043] (Drawing process in the image generation unit) The image generation unit 204 first generates a virtual viewpoint image corresponding to a period indicated by a time code included in the virtual viewpoint information, based on the virtual viewpoint information. In this case, it obtains trajectory information corresponding to the said period by reading it from the auxiliary storage device 306 or the like. In addition to the trajectory information corresponding to the said period, the image generation unit 204 may also obtain trajectory information corresponding to a past period equivalent to a predetermined number of frames from the start of the said period by reading it from the auxiliary storage device 306 or the like.

[0044] Next, when the image generation unit 204 generates (also called "drawing") a virtual viewpoint image, it places a mesh model or a two-dimensional billboard at the coordinates of each three-dimensional shape corresponding to the target object indicated by the acquired trajectory information, and then draws the virtual viewpoint image. Here, the mesh model or billboard placed at the coordinates of each three-dimensional shape is, for example, a virtual object that mimics the shape of the target object. By performing this drawing process, the trajectory of the image of the virtual object placed at the position of the three-dimensional shape corresponding to the target object is drawn on the virtual viewpoint image. The aforementioned mesh model or two-dimensional billboard is placed in three-dimensional space. Therefore, even if the virtual viewpoint is moved, rotated, or turned, the trajectory of the image of the virtual object placed at the position of the three-dimensional shape corresponding to the target object is appropriately drawn on the virtual viewpoint image.

[0045] Figure 8 shows an example of a virtual viewpoint image generated by the image generation unit 204 according to the first embodiment. Specifically, Figure 8(a) shows an example of a virtual viewpoint image in which the trajectory of a virtual object placed at a three-dimensional position corresponding to a target object to which an identifier corresponding to a pitch and a batted ball has been assigned by the identification unit 205 is drawn. Figure 8(b) also shows an example of a virtual viewpoint image in which the trajectory of a virtual object placed at a three-dimensional position corresponding to a target object to which an identifier corresponding to a pitch has been assigned by the identification unit 205 is drawn. Each three-dimensional shape corresponding to a target object is pre-assigned an identifier that can identify its type. Therefore, for example, as shown in Figure 8(b), it is possible to draw the trajectory of a virtual object placed at a three-dimensional position corresponding to a target object only in a specific state or period, such as a pitch. Note that when the image generation unit 204 acquires trajectory information from the auxiliary storage device 306, etc., if the identifier of the target for which the trajectory of the target object is to be drawn in the virtual viewpoint image is predetermined, it may acquire only the trajectory information corresponding to that identifier.

[0046] According to the image processing apparatus 102 of this embodiment, the position of a target object can be determined based on the three-dimensional shape estimated in the intermediate processing of the virtual viewpoint image generation, without using a dedicated imaging device or the like for determining the position of the target object. Furthermore, by using the trajectory information determined based on the three-dimensional shape corresponding to the target object, a virtual object corresponding to the target object can be placed at a position corresponding to the position of the target object in the same coordinate system as the virtual viewpoint corresponding to the virtual viewpoint image. As a result, the trajectory of the image of the virtual object corresponding to the target object can be drawn on the virtual viewpoint image without using a dedicated imaging device or the like for determining the position of the target object.

[0047] [Modified version of the first embodiment] In the first embodiment, an example was described in which the image processing system is applied to baseball, and the position of a three-dimensional shape corresponding to the target object, a ball, is identified, and the trajectory of the position of the three-dimensional shape is drawn on a virtual viewpoint image. However, the application of the image processing system is not limited to baseball. For example, the image processing system can be applied to other ball games such as soccer or table tennis. Furthermore, the target object is not limited to a ball, but may be a shuttlecock in badminton, etc.

[0048] Furthermore, in the first embodiment, as an example of the process of acquiring three-dimensional shape data by the identification unit 205, a method of acquiring the data by reading it from an auxiliary storage device 306 or the like was described. However, the method of acquiring three-dimensional shape data by the identification unit 205 is not limited to this. For example, the shape estimation unit 202 may store the three-dimensional shape data obtained by the estimation process in an auxiliary storage device 306 or the like and output it to the identification unit 205, and the identification unit 205 may acquire the three-dimensional shape data output by the shape estimation unit 202.

[0049] Furthermore, in the first embodiment, an example of the generation (drawing) process by the image generation unit 204 was described in which trajectory information is acquired based on the time code included in the virtual viewpoint information. However, the method of acquiring trajectory information by the image generation unit 204 is not limited to this. For example, the image generation unit 204 may acquire trajectory information from the auxiliary storage device 306 or the like for a period corresponding to a predetermined number of frames from an arbitrary point in time different from the time of the time code included in the virtual viewpoint information, or for a period from that arbitrary point in time to any other arbitrary time.

[0050] Furthermore, in the first embodiment, as an example of the generation (drawing) process by the image generation unit 204, a method of placing virtual objects such as mesh models or two-dimensional billboards at the coordinates of the three-dimensional shape indicated by the trajectory information was described. However, the method of placing virtual objects by the image generation unit 204 is not limited to this. For example, the image generation unit 204 may place mesh models as virtual objects corresponding to lines connecting the coordinates of each three-dimensional shape indicated by the acquired trajectory information.

[0051] In the first embodiment, the size of the virtual objects such as the mesh model to be placed is not specifically mentioned, but it is desirable that the size of the virtual object representing the trajectory be the same as or approximately the same as the size of the target object (ball). This allows the image generation unit 204 to generate a virtual viewpoint image that is easy for viewers of the virtual viewpoint image to recognize as an image showing the trajectory of the target object.

[0052] On the other hand, depending on the field of view of the virtual viewpoint, or the distance from the virtual viewpoint to the three-dimensional shape corresponding to the target object, the image of the trajectory far from the virtual viewpoint may become small in the virtual viewpoint image, making it difficult for viewers to see. In such cases, for example, the image generation unit 204 may place a virtual object that is larger than the actual size of the target object. Alternatively, the image generation unit 204 may automatically determine the size of the virtual object to be placed based on the field of view of the virtual viewpoint, or the distance from the virtual viewpoint to the three-dimensional shape corresponding to the target object.

[0053] Furthermore, the image processing device 102 may have a function to display a GUI on the display device 103 that accepts instructions from the user regarding the types of trajectories of target objects to be drawn on the virtual viewpoint image, or the method of drawing the trajectories. Figure 9 is a diagram showing an example of a GUI 900 according to a modification of the first embodiment. The GUI 900 includes, as an example, an area 901 that accepts the setting of the selection of the type of object to be drawn for the trajectory, and an area 902 that accepts the setting of the selection of the trajectory drawing method. The GUI 900 also includes an area 903 that accepts the setting of the size of a virtual object used for drawing the trajectory, and an area 904 that accepts the setting of the color of the virtual object. The GUI 900 also includes an area 905 that accepts the setting of the frame rate of the virtual viewpoint image. The user can make various settings to the image processing device 102 by making inputs to each of the areas 901 to 905 included in the GUI 900 using the operation unit 308.

[0054] Furthermore, in the first embodiment, the type of identifier was described simply as "throw" by a fielder, but the type of identifier is not limited to such categories. For example, the region 603 shown in Figure 6 may be divided or limited to smaller regions, and, similar to the identification of a batted ball, the trajectory of a three-dimensional shape corresponding to a target object starting from a predetermined region may be further specified, such as a throw from a predetermined infielder such as a third baseman or a return throw from an outfielder.

[0055] Furthermore, in the first embodiment, as an example, a description was given of how the movement trajectory of a three-dimensional shape corresponding to a ball in a baseball game is classified into pitching, batting, or throwing. However, the classification of the movement trajectory of a three-dimensional shape corresponding to a target object is not limited to this. For example, the image processing apparatus 102 according to this disclosure can be applied to other ball games such as soccer. For example, if the object to be imaged is a soccer game, the following areas may be set in advance, and the identification unit 205 may classify the movement trajectory of a three-dimensional shape corresponding to the target object based on the pre-set areas. Specifically, for example, areas that identify the movement trajectory of a three-dimensional shape corresponding to the target object may be pre-set in areas including the goal and goal area, or areas including points where the starting point for restarting the game, such as a corner kick or throw, is predetermined. In this case, for example, the identification unit 205 identifies the movement trajectory of a three-dimensional shape corresponding to the target object for each of these pre-set areas.

[0056] Furthermore, in the first embodiment, as an example, a method was described in which the trajectory of the movement of a three-dimensional shape corresponding to a target object is drawn on the virtual viewpoint image using trajectory information. However, what is drawn on the virtual viewpoint image is not limited to the trajectory of the movement of a three-dimensional shape corresponding to a target object. For example, in addition to drawing the trajectory, the identification unit 205 may superimpose an image showing the velocity of the batted ball onto the virtual viewpoint image based on the velocity information of the three-dimensional shape identified by the batted ball. Figure 10 is a diagram showing an example of a virtual viewpoint image according to a modification of the first embodiment. The virtual viewpoint images shown in Figures 10(a) and (b) include an image showing the velocity of the batted ball in addition to an image of the trajectory of the batted ball.

[0057] In the explanation so far, the image generation unit 204 has been described as generating a virtual viewpoint image that includes the image of the trajectory of the target object (ball). However, the image generation unit 204 may decide whether or not to include the image of the target object's trajectory in the virtual viewpoint image based on instructions from the user or others. Specifically, for example, if an instruction is received to not include the image of the target object's trajectory in the virtual viewpoint image, the image generation unit 204 will generate a virtual viewpoint image that does not include the image of the target object's trajectory.

[0058] Furthermore, in the first embodiment, as an example of the acquisition process by the viewpoint acquisition unit 203, a method was described in which virtual viewpoint information, including information regarding the position and orientation of the virtual viewpoint, is acquired by receiving a signal corresponding to the input from the operation unit 308. However, the viewpoint acquisition unit 203 may also acquire information regarding the position and orientation of the virtual viewpoint by determining the position or orientation of the virtual viewpoint based on information regarding the trajectory of a three-dimensional shape corresponding to a target object included in the trajectory information. For example, the viewpoint acquisition unit 203 determines the orientation of the virtual viewpoint so that the coordinates of the three-dimensional shape corresponding to the target object (ball) at the time the virtual viewpoint image is generated, which are included in the trajectory information, are captured in the center of the field of view from the virtual viewpoint. This makes it possible to realize virtual camera work that does not take a fast-moving ball or the like out of the field of view of the virtual viewpoint image. In addition, it is possible to realize virtual camera work that is not possible in reality, such as following a ball or the like from behind in the direction of movement.

[0059] Furthermore, in the first embodiment, a configuration was described in which the identification unit 205 performs a process to identify the three-dimensional shape corresponding to the target object, a process to identify the trajectory, and a process to assign an identifier based on the identification result. However, the processing performed by the identification unit 205 is not limited to this. For example, the identification unit 205 may only perform the process to identify the three-dimensional shape corresponding to the target object, and the image generation unit 204 may determine whether or not to draw the trajectory of the target object based on the position or velocity of each three-dimensional shape corresponding to the target object included in the trajectory information.

[0060] Furthermore, in the first embodiment, an embodiment was described in which the identification unit 205 performs identification processing of the three-dimensional shape corresponding to the target object for all three-dimensional shapes estimated by the shape estimation unit 202. However, the target of the identification processing of the three-dimensional shape corresponding to the target object in the identification unit 205 is not limited to all three-dimensional shapes estimated by the shape estimation unit 202. For example, in the case of an application where only pitching needs to be identified, the identification unit 205 performs identification processing of the three-dimensional shape corresponding to the target object only for the three-dimensional shapes that exist within the region 601 shown in Figure 6. In this case, since all the identified three-dimensional shapes corresponding to the target object correspond to pitching, the identifier assignment process in the process S506 shown in Figure 5 can be omitted.

[0061] [Other embodiments] The technology of this disclosure can also be realized by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a processing circuit such as an ASIC (Application-Specific Integrated Circuit) that implements one or more functions.

[0062] Furthermore, within the scope of this disclosure, the technologies described herein allow for free combination of each embodiment, modification of any component of each embodiment, or omission of any component in each embodiment.

[0063] [Technical Features of This Disclosure] This disclosure includes the following configurations, methods, and programs.

[0064] (Composition 1) A shape acquisition means for acquiring shape data that shows the three-dimensional shape of each object present in the target space, A means for identifying a three-dimensional shape of an object whose size falls within a given range from among the three-dimensional shapes indicated by the shape data for each object, and for generating trajectory information that shows the trajectory of the three-dimensional shape of the object by associating information regarding the position of the identified three-dimensional shape of the object with a corresponding time code. A viewpoint acquisition means that acquires virtual viewpoint information indicating the position of a virtual viewpoint and the direction of the line of sight in the said virtual viewpoint, Image generation means that generates a virtual viewpoint image corresponding to the view from the virtual viewpoint, which includes an image representing the trajectory of the three-dimensional shape of the object, based on the shape data, the trajectory information, and the virtual viewpoint information. An image processing apparatus characterized by having

[0065] (Configuration 2) The aforementioned identification means identifies the three-dimensional shape of the object using at least one of the volume, surface area, and external dimensions of the three-dimensional shape as the size of the three-dimensional shape. An image processing apparatus according to configuration 1, characterized by the above.

[0066] (Composition 3) The identifying means generates trajectory information corresponding to an object, assuming that the three-dimensional shapes of the object are the same object if the distance between the position of the three-dimensional shape of the object at any given time and the position of the three-dimensional shape of the object at a time near that time is within a specific range, An image processing apparatus according to configuration 1 or 2, characterized by the above.

[0067] (Composition 4) One or more specific regions are pre-defined within the aforementioned target region. The identification means performs a process to identify the target three-dimensional shape with respect to only the three-dimensional shapes that exist within the specified region from among the three-dimensional shapes indicated by the shape data for each object. An image processing apparatus according to any one of configurations 1 to 3 characterized by the above.

[0068] (Composition 5) The image generation means places a three-dimensional shape representing the trajectory of the three-dimensional shape of the target at the position of the three-dimensional shape of the target indicated by the trajectory information, and generates a virtual viewpoint image including an image representing the trajectory of the three-dimensional shape of the target by performing drawing processing based on the shape data and the virtual viewpoint information. An image processing apparatus according to any one of configurations 1 to 4 characterized by the above.

[0069] (Composition 6) The image generation means arranges a three-dimensional shape that represents the trajectory of the three-dimensional shape of the target, which includes line segments connecting the positions of the three-dimensional shape of the target indicated by the trajectory information. The image processing apparatus described in configuration 5, characterized by the above.

[0070] (Composition 7) The image generation means places a virtual object, composed of a two-dimensional image representing the trajectory of the three-dimensional shape of the target, at the position of the three-dimensional shape of the target indicated by the trajectory information, and generates the virtual viewpoint image including an image representing the trajectory of the three-dimensional shape of the target by performing drawing processing based on the shape data and the virtual viewpoint information. An image processing apparatus according to any one of configurations 1 to 4 characterized by the above.

[0071] (Composition 8) The image generation means arranges the virtual object composed of the two-dimensional image in a plane orthogonal to the direction of the virtual viewpoint indicated by the virtual viewpoint information. The image processing apparatus according to configuration 7, characterized by the above.

[0072] (Composition 9) One or more specific regions are pre-defined within the aforementioned target region. The identification means assigns a different identifier to the trajectory information corresponding to the three-dimensional shape of the object for each of the specific regions in which the three-dimensional shape of the object exists. An image processing apparatus according to any one of configurations 1 to 8 characterized by the above.

[0073] (Composition 10) The identifying means identifies at least one of the movement direction and velocity of the three-dimensional shape of the target corresponding to the same object based on the difference in position between the positions of the three-dimensional shapes of the target corresponding to the same object at a point in time when they are in close proximity to each other, and generates the trajectory information including information regarding the movement direction or velocity of the identified three-dimensional shape of the target. An image processing apparatus according to any one of configurations 1 to 9 characterized by the above.

[0074] (Composition 11) The identifying means assigns different identifiers to the trajectory information corresponding to the three-dimensional shape of the object, based on at least one of the position, direction of movement, and velocity of the three-dimensional shape of the object. An image processing apparatus according to configuration 10, characterized by the above.

[0075] (Composition 12) The image generation means determines whether or not to include an image representing the trajectory of the three-dimensional shape of the object in the virtual viewpoint image, based on the position, direction of movement, and velocity of the three-dimensional shape of the object. An image processing apparatus according to configuration 10 or 11, characterized by the above.

[0076] (Composition 13) The image generation means determines, based on the identifier assigned to the trajectory information, whether or not to include in the virtual viewpoint image an image representing the trajectory of the three-dimensional shape of the target based on the trajectory information to which the identifier is assigned, for each identifier. An image processing apparatus according to configuration 9 or 11, characterized by the above.

[0077] (Composition 14) The image generation means determines, based on instructions from the user, whether or not to include an image representing the trajectory of the three-dimensional shape of the object in the virtual viewpoint image. An image processing apparatus according to any one of configurations 1 to 13 characterized by the above.

[0078] (Composition 15) The image generation means, based on instructions from the user, changes at least one of the length, size, and color of the trajectory of the image representing the trajectory of the three-dimensional shape of the target. An image processing apparatus according to any one of configurations 1 to 14 characterized by the above.

[0079] (method) A shape acquisition process that acquires shape data showing the three-dimensional shape of each object present in the target space, A selection step of identifying a three-dimensional shape of an object whose size falls within a given range from among the three-dimensional shapes indicated by the shape data for each object, and generating trajectory information that shows the trajectory of the three-dimensional shape of the object by associating information regarding the position of the identified three-dimensional shape of the object with a corresponding time code, A viewpoint acquisition step that acquires virtual viewpoint information indicating the position of a virtual viewpoint and the direction of the line of sight at the said virtual viewpoint, An image generation step that generates a virtual viewpoint image corresponding to the view from the virtual viewpoint, which includes an image representing the trajectory of the three-dimensional shape of the target, based on the shape data, the trajectory information, and the virtual viewpoint information; Image processing methods including [specific details omitted].

[0080] (program) A program for causing a computer to function as an image processing device described in any one of configurations 1 to 15. [Explanation of symbols]

[0081] 102 Image Processing Device 201 Image Acquisition Unit 202 Shape estimation section 203 Viewpoint acquisition unit 204 Image Generation Unit 205 Specific section

Claims

1. A shape acquisition means for acquiring shape data that shows the three-dimensional shape of each object present in the target space, A means for identifying a three-dimensional shape of an object whose size falls within a given range from among the three-dimensional shapes indicated by the shape data for each object, and for generating trajectory information that shows the trajectory of the three-dimensional shape of the object by associating information regarding the position of the identified three-dimensional shape of the object with a corresponding time code. A viewpoint acquisition means that acquires virtual viewpoint information indicating the position of a virtual viewpoint and the direction of the line of sight in the said virtual viewpoint, Image generation means for generating a virtual viewpoint image corresponding to the view from the virtual viewpoint, which includes an image representing the trajectory of the three-dimensional shape of the object, based on the shape data, the trajectory information, and the virtual viewpoint information. An image processing apparatus characterized by having

2. The aforementioned identification means identifies the three-dimensional shape of the object using at least one of the volume, surface area, and external dimensions of the three-dimensional shape as the size of the three-dimensional shape. The image processing apparatus according to claim 1, characterized in that

3. The identifying means generates trajectory information corresponding to an object, assuming that the three-dimensional shapes of the object are the same object if the distance between the position of the three-dimensional shape of the object at any given time and the position of the three-dimensional shape of the object at a time near that time is within a specific range, The image processing apparatus according to claim 1, characterized in that

4. One or more specific regions are pre-defined within the aforementioned target region. The identification means performs a process to identify the target three-dimensional shape with respect to only the three-dimensional shapes that exist within the specified region from among the three-dimensional shapes indicated by the shape data for each object. The image processing apparatus according to claim 1, characterized in that

5. The image generation means places a three-dimensional shape representing the trajectory of the three-dimensional shape of the target at the position of the three-dimensional shape of the target indicated by the trajectory information, and generates a virtual viewpoint image including an image representing the trajectory of the three-dimensional shape of the target by performing drawing processing based on the shape data and the virtual viewpoint information. The image processing apparatus according to claim 1, characterized in that

6. The image generation means arranges a three-dimensional shape that represents the trajectory of the three-dimensional shape of the target, which includes line segments connecting the positions of the three-dimensional shape of the target indicated by the trajectory information. The image processing apparatus according to claim 5, characterized by the following:

7. The image generation means places a virtual object, composed of a two-dimensional image representing the trajectory of the three-dimensional shape of the target, at the position of the three-dimensional shape of the target indicated by the trajectory information, and generates the virtual viewpoint image including an image representing the trajectory of the three-dimensional shape of the target by performing drawing processing based on the shape data and the virtual viewpoint information. The image processing apparatus according to claim 1, characterized in that

8. The image generation means arranges the virtual object composed of the two-dimensional image in a plane orthogonal to the direction of the virtual viewpoint indicated by the virtual viewpoint information. The image processing apparatus according to claim 7, characterized by the following:

9. One or more specific regions are pre-defined within the aforementioned target region. The identification means assigns a different identifier to the trajectory information corresponding to the three-dimensional shape of the target for each of the specific regions in which the three-dimensional shape of the target exists. The image processing apparatus according to claim 1, characterized in that

10. The identifying means identifies at least one of the movement direction and velocity of the three-dimensional shape of the target corresponding to the same object based on the difference in position between the positions of the three-dimensional shapes of the target corresponding to the same object at a point in time when they are in close proximity to each other, and generates the trajectory information including information regarding the movement direction or velocity of the identified three-dimensional shape of the target. The image processing apparatus according to claim 1, characterized in that

11. The identifying means assigns different identifiers to the trajectory information corresponding to the three-dimensional shape of the object, based on at least one of the position, direction of movement, and velocity of the three-dimensional shape of the object. The image processing apparatus according to claim 10, characterized by the above.

12. The image generation means determines whether or not to include an image representing the trajectory of the three-dimensional shape of the object in the virtual viewpoint image, based on the position, direction of movement, and velocity of the three-dimensional shape of the object. The image processing apparatus according to claim 10, characterized by the above.

13. The image generation means determines, based on the identifier assigned to the trajectory information, whether or not to include in the virtual viewpoint image an image representing the trajectory of the three-dimensional shape of the target based on the trajectory information to which the identifier is assigned, for each identifier. The image processing apparatus according to claim 9, characterized by the above.

14. The image generation means determines, based on instructions from the user, whether or not to include an image representing the trajectory of the three-dimensional shape of the object in the virtual viewpoint image. The image processing apparatus according to claim 1, characterized in that

15. The image generation means, based on instructions from the user, changes at least one of the length, size, and color of the trajectory of the image representing the trajectory of the three-dimensional shape of the target. The image processing apparatus according to claim 1, characterized in that

16. A shape acquisition process that acquires shape data showing the three-dimensional shape of each object present in the target space, A selection step of identifying a target three-dimensional shape whose size falls within a given range from among the three-dimensional shapes indicated by the shape data for each object, and generating trajectory information that shows the trajectory of the target three-dimensional shape by associating the position information of the identified target three-dimensional shape with a corresponding time code, A viewpoint acquisition step that acquires virtual viewpoint information indicating the position of a virtual viewpoint and the direction of the line of sight at the said virtual viewpoint, Image generation step: Based on the shape data, the trajectory information, and the virtual viewpoint information, generates a virtual viewpoint image corresponding to the view from the virtual viewpoint, which includes an image representing the trajectory of the three-dimensional shape of the target. Image processing methods including [specific details omitted].

17. A program for causing a computer to function as an image processing device according to any one of claims 1 to 15.

Images