Image processing apparatus, control method, and program

Abstract

To provide viewers with information related to players in a virtual viewpoint video without interfering with a viewing experience of the virtual viewpoint video.SOLUTION: An image processing apparatus comprises: detection means for detecting a position of a subject; video generation means for generating a virtual viewpoint video from virtual viewpoint materials; and display means for displaying, at the position based on subject position information detected by the detection means, information related to movements of the subject as subject information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device that generates a virtual viewpoint video. [Background technology]

[0002] There is a virtual viewpoint video generation system that generates a virtual viewpoint video, which is an image seen from a virtual viewpoint specified by a user, based on an image captured by a shooting system using multiple cameras. Patent Document 1 describes a system in which, after transmitting images captured by multiple cameras, an image computing server (image processing device) extracts images with large changes from the captured images as foreground images and images with small changes as background images.

[0003] In sports, the position information of players is detected from sensors attached to the players or from images taken in multiple directions. The position information of players is used, for example, for coaching the players or for commentary in broadcasting. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-211828 A Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, information such as the speed and direction of a player's movements fluctuates rapidly. Therefore, if the player's information is shown as a number, it may be difficult for the viewer to understand at a glance. This may also hinder the user's viewing experience. [Means for solving the problem]

[0006] The image processing device includes a detection means for detecting the position of a subject, an image generation means for generating a virtual viewpoint image from a virtual viewpoint material, and a display means for displaying information relating to the movement of the subject as subject information at a position based on the subject position information detected by the detection means. Effect of the Invention

[0007] According to the present invention, information about the players in the virtual viewpoint video can be provided to the viewer without interfering with the viewing experience of the virtual viewpoint video. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example of an image processing system. [Diagram 2] FIG. 1A is a diagram showing an example of the position of a subject; FIG. 1B is a diagram showing an example of a subject extracted by a shape extraction unit; FIG. 1C is a diagram showing an example of the position of a subject in a different state; FIG. 1D is a diagram showing an example of a subject extracted by a shape extraction unit; and FIG. 1E is a diagram showing an example of an extracted shape. [Diagram 3] 1A is an example of an extracted shape having an identifier attached thereto, and FIG. 1B is an example of a graphical user interface for displaying the extracted shape and the identifier. [Figure 4] FIG. 13 is a diagram showing an example of a representative position. [Diagram 5] 13 is a flowchart showing an example of a tracing analysis process performed by a tracing unit. [Figure 6] 11 is a flowchart showing an example of a subject information superimposing process performed by a video generating unit. [Figure 7] 4 shows an example of an image generated by the image generation unit. [Figure 8] An example of markings that indicate the subject's speed and direction of travel. [Figure 9] FIG. 2 is a block diagram showing an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [First embodiment] (System configuration and operation of image processing device) FIG. 1 shows an example of the configuration of an image processing system for generating a virtual viewpoint video according to this embodiment. The image processing system includes, for example, an imaging unit 1, a synchronization unit 2, a three-dimensional shape estimation unit 3, a storage unit 4, a viewpoint instruction unit 5, a video generation unit 6, a display unit 7, and a subject position detection unit 8. The video generation unit 6 includes a foreground video generation unit 61, a background video generation unit 62, a subject information video generation unit 63, and a video synthesis unit 64, and the subject position detection unit 8 includes a shape extraction unit 11, a tracking unit 12, a subject position calculation unit 13, and an identification setting unit 14. The image processing system may be configured by one image processing device, or may be a system configured by multiple image processing devices. In the following description, the image processing system will be described as being one image processing device.

[0010] The operation of each component in an image processing device that generates a virtual viewpoint video to which this system is applied will be outlined below. First, multiple imaging units 1 perform imaging in synchronization with each other based on a synchronization signal from a synchronization unit 2. The imaging units 1 output the captured images to a three-dimensional shape estimation unit 3. The imaging units 1 are installed so as to surround an imaging area including the subject so that the subject can be photographed from multiple directions. The three-dimensional shape estimation unit 3 uses the input images taken from multiple viewpoints to extract, for example, the silhouette of the subject, and then generates a three-dimensional shape of the subject using a volume intersection method or the like. The three-dimensional shape estimation unit 3 also outputs the generated three-dimensional shape of the subject and the captured images to a storage unit 4. Here, the subject is an object that is the subject of three-dimensional shape generation, and includes people and items handled by people.

[0011] The subject position detection unit 8 detects the position of the subject within the shooting area, and outputs the detected subject position information to the storage unit 4, as will be described in detail later.

[0012] The storage unit 4 saves and accumulates a group of data (material data) used to generate a virtual viewpoint video. The data used to generate a virtual viewpoint video specifically includes the captured image and the three-dimensional shape of the subject input from the three-dimensional shape estimation unit 3, camera parameters such as the position and orientation of each imaging unit and optical characteristics, and subject position information acquired by the subject position detection unit 8. Note that a background model and a background texture image are saved (recorded) in advance in the storage unit 4 as data used to generate the background of the virtual viewpoint video.

[0013] The viewpoint instruction unit 5 is composed of a viewpoint operation unit, which is a physical user interface such as a joystick or jog dial (not shown), and a display unit for displaying a virtual viewpoint image. The virtual viewpoint of the virtual viewpoint image displayed here can be changed by the viewpoint operation unit. In response to a change in the virtual viewpoint by the viewpoint operation unit, a virtual viewpoint image is generated by the image generation unit 6 (described later) at any time and displayed on the display unit. This display unit may share the display unit 7 (described later) or may have a separate display device. The viewpoint instruction unit 5 generates virtual viewpoint information based on an input from the viewpoint operation unit, and outputs the generated virtual viewpoint information to the image generation unit 6. The virtual viewpoint information includes information equivalent to external parameters of the camera such as the position and orientation of the virtual viewpoint, information equivalent to internal parameters of the camera such as the focal length and angle of view, and time information specifying the shooting time to be played back.

[0014] Based on the time information included in the input virtual viewpoint information, the video generation unit 6 acquires the material data of the shooting time from the storage unit 4. The video generation unit 6 generates a virtual viewpoint video at the set virtual viewpoint using the three-dimensional shape of the subject and the shot image from the acquired material data, and outputs the generated video to the display unit 7.

[0015] The display unit 7 is a display means for displaying the image input from the image generating unit 6. The display unit 7 is configured with a display, a head mounted display (HMD), or the like.

[0016] (How to track the subject position) Next, a method for tracking the subject position in this embodiment will be described.

[0017] First, three-dimensional shape estimation unit 3 generates a three-dimensional shape of the subject, and outputs the generated three-dimensional shape to storage unit 4 and also outputs the generated three-dimensional shape to shape extraction unit 11.

[0018] The shape extraction unit 11 cuts out the lower part of the three-dimensional shape of the subject as shown in FIG. 2(b) from the three-dimensional shape of the subject as shown in FIG. 2(a). In this embodiment, the cutout is made from the bottom surface of the circumscribed rectangular parallelepiped of the three-dimensional shape of the subject to a predetermined height (e.g., a height equivalent to 50 cm). For example, as shown in FIG. 2(c), when one of the subjects is standing and the other subject is away from the floor of the shooting area by jumping or the like, the three-dimensional shape of the subject is cut out as shown in FIG. 2(d). That is, for both of the three-dimensional shapes of the subjects, the three-dimensional shape is cut out from the part equivalent to the feet to a predetermined height.

[0019] Next, as shown in FIG. 2(e), the shape extraction unit 11 projects the cut-out three-dimensional shape onto a plane viewed from directly above the three-dimensional shape of the subject to generate a two-dimensional image. In this embodiment, the shape extraction unit 11 parallel projects the cut-out three-dimensional shape onto a two-dimensional plane equivalent to the feet (floor surface). In this embodiment, the image projected onto the plane is a binary image in which the cut-out three-dimensional shape is white and the other parts are black. The shape extraction unit 11 divides this two-dimensional image into independent regions and obtains circumscribing rectangles 201 to 204 shown in FIG. 2(e). The shape extraction unit 11 outputs vertex information of this circumscribing rectangle as an extracted three-dimensional shape (extracted shape). Here, the shape extraction unit 11 converts the vertex information of the circumscribing rectangle into the same coordinate system and unit as the three-dimensional space of the shooting region before outputting it. In addition, to determine the independent shapes in the shape extraction unit 11, a method such as continuous component analysis is used for the projected two-dimensional image. By using such a method, the shape extraction unit 11 can divide the three-dimensional shape into individual regions.

[0020] The identification setting unit 14 assigns an identifier to the extracted shape output by the shape extraction unit 11. Specifically, the identification setting unit 14 calculates the distance between each extracted shape, and assigns an identifier according to the distance between the extracted shapes. For example, as shown in FIG. 3(a), the identification setting unit 14 assigns the same identifier to extracted shapes whose distance between the extracted shapes is less than a predetermined distance (solid arrow), and assigns a different identifier to extracted shapes whose distance between the extracted shapes is equal to or greater than the predetermined distance (dashed arrow). The threshold value of the predetermined distance used as the criterion for judgment is preferably a distance equivalent to the distance between the legs of the subject when standing. In this embodiment, the explanation will be given assuming that the threshold value of the predetermined distance is set to 50 cm.

[0021] The identification setting unit 14 displays the assigned identifiers on a display unit provided in the identification setting unit 14 by a graphical interface (GUI) as shown in FIG. 3(b). The user operates the image processing system while viewing this GUI. Specifically, the identification setting unit 14 displays the current identifier assignment (identifier assignment in the initial state) on the graphical interface by distinguishing it with at least one of characters and color coding. In FIG. 3(b), the identification setting unit 14 displays each identifier by using both characters and color coding. The user checks the GUI and confirms whether the desired identifier assignment is set as the initial state. If the desired identifier assignment is not set, the user instructs the subject to change its standing position or close its legs, and repeats this until the desired assignment is set. Alternatively, the user operates the image processing system via the GUI and issues a change instruction so that the desired identifier assignment is set. If the desired identifier assignment is set, the user presses, for example, a decision button (initial identifier decision button) on the graphical interface as shown in FIG. 3(b). In response to this operation, the identification setting unit 14 decides the initial state identifier. Then, the identification setting unit 14 outputs the identifier assigned to each extracted shape to the tracking unit 12.

[0022] In response to receiving an identifier from the identification setting unit 14, the tracking unit 12 assigns the identifier to each extracted shape as an initial state. After that, the tracking unit 12 tracks the extracted shape to which the identifier has been assigned. The identifier assigned to the extracted shape during tracking is not the identifier determined by the identification setting unit 14, but an identifier determined based on the tracking result of the position of each extracted shape by the tracking unit 12. In tracking (tracking analysis) of the extracted shape, the tracking unit 12 tracks the extracted shape based on the position of each extracted shape at the time immediately before the shooting time of the extracted shape, the identifier of each extracted shape, and information on the subject position input from the subject position calculation unit described later. The specific processing of tracking by the tracking unit 12 will be described later. The tracking unit 12 assigns an identifier to each extracted shape at the time based on the result of the tracking analysis, and outputs each extracted shape to the subject position calculation unit 13.

[0023] The subject position calculation unit 13 obtains a representative position for each extracted shape to which an identifier has been assigned, input by the tracking unit 12. For example, as shown in Fig. 4, the subject position calculation unit 13 obtains a position indicating each extracted shape group for each extracted shape group to which the same identifier has been assigned, such as representative positions 401 and 402. In this embodiment, the representative position is the center position of the extracted shape group.

[0024] However, since this representative position is affected by shape estimation errors and fluctuations in the boundary parts when the shape is cut out by the shape extraction unit 11, the position may fluctuate at each time even if the subject is stationary. Therefore, in this embodiment, the subject position calculation unit 13 performs processing such as low-pass filtering and moving average in the time direction on the center position information at each time to generate position information with high frequency components suppressed. Then, the subject position calculation unit 13 outputs the position information of the representative position together with the identifier as the position of the subject to the tracking unit 12. In addition, the subject position calculation unit 13 records (accumulates) information in which information on the time when the three-dimensional shape that was the basis of the tracking analysis was photographed is added to the position information of the representative position as subject position information (subject position information) in the accumulation unit 4.

[0025] (Tracking analysis process by tracking unit 12) Next, an example of the tracking and analysis process of the extracted shape position in the tracking section 12 will be described with reference to the flowchart of FIG.

[0026] In step S501, the tracking unit 12 performs an initialization process upon receiving an input from the identification setting unit 14. Specifically, the tracking unit 12 acquires the identifiers of the respective extracted shapes input from the identification setting unit 14.

[0027] In step S 502 , the tracking unit 12 acquires the next extracted shape input from the shape extraction unit 11 .

[0028] In step S503, the tracking unit 12 assigns the identifiers acquired from the identification setting unit 14 to the acquired extracted shapes, and outputs the extracted shapes to which the identifiers have been assigned to the subject position calculation unit 13.

[0029] In step S504 , the subject position calculation unit 13 obtains the subject position from the group of extracted shapes to which the same identifier is assigned, and outputs the subject position to the tracking unit 12 .

[0030] The above-mentioned processes in steps S501 to S504 correspond to the initialization process.

[0031] The subsequent steps S505 to S509 are executed every time while the imaging unit 1 is imaging the subject. When the imaging process of the subject by the imaging unit 1 is completed, the process of step S509 is completed and the process of this flowchart is terminated.

[0032] In step S505, the tracking unit 12 acquires the extracted shape input from the shape extraction unit 11 and the subject position at the previous time (previous time) calculated by the subject position calculation unit 13. The previous time is, for example, the shooting time of the extracted shape generated one frame before the extracted shape currently being processed. Here, for comparison, the current time is also referred to as the current time. Here, the current time refers to the shooting time of the image used to generate the extracted shape currently being processed.

[0033] In step S506, when the subject position at the previous time overlaps with the representative position of each extracted shape at the current time, the tracking unit 12 assigns to the extracted shape an identifier that is assigned to the subject position that overlaps with the representative position. Here, in step S506, when the representative position of one extracted shape overlaps with multiple subject positions, the tracking unit 12 assigns to the extracted shape an identifier indicating "undeterminable" at the current time. This is because, for example, multiple extracted shapes with different identifiers assigned may overlap at the current time, such as when two subjects are close to each other, so an identifier indicating "undeterminable" is assigned in the processing of this step. The extracted shape to which an identifier including an identifier indicating "undeterminable" is assigned is subjected to the processing of S509 described later.

[0034] In step S507, if the representative position of an extracted shape to which an identifier has not yet been assigned overlaps with the extracted shape of the previous time, the tracking unit 12 assigns the identifier assigned to the extracted shape of the previous time to the extracted shape of the current time.

[0035] In step S508, if there is another extracted shape to which an identifier has already been assigned at the current time within a predetermined range from an extracted shape to which an identifier has not yet been assigned, the tracking unit 12 assigns the identifier assigned to the other extracted shape. The predetermined range is preferably a range equivalent to the width of the subject's legs when standing. For example, the predetermined range is a range of a radius of 50 cm from the center of the extracted shape. Here, if there are multiple other extracted shapes to which an identifier has been assigned within a predetermined range from a certain extracted shape, the tracking unit 12 assigns the identifier of the extracted shape that is closest among the other extracted shapes to the extracted shape. For an extracted shape to which an identifier has not been assigned at the stage of completing the processing up to step S508, the tracking unit 12 determines that the extracted shape is not a tracking target. In this case, the tracking unit 12 does not output the extracted shape determined to be not a tracking target to the subject position calculation unit 13.

[0036] In step S509, the tracking unit 12 outputs to the subject position calculation unit 13 the extracted shape to which an identifier has been assigned in the processes of steps S506 to S508, and the identifier assigned thereto.

[0037] In step S510, a control unit (not shown) determines whether the imaging process of the subject by the imaging unit 1 has ended. If it is determined that the imaging process of the subject by the imaging unit 1 has not ended, the process of step S508 is executed. If it is determined that the imaging process of the subject by the imaging unit 1 has ended, the process of this flowchart ends.

[0038] Note that the processes from step S506 to step S508 are performed for each extracted shape. By repeating the processes from step S506 to step S509, the identifier set by the identification setting unit 14 is linked to the extracted shape at each time. Using this identifier, the subject position calculation unit 13 can distinguish between each subject and find the subject position.

[0039] In addition, when the tracking unit 12 assigns an "undeterminable" identifier to the extracted shape, there is a possibility that some of the identifiers set in the initial setting will not be assigned at a certain time. In such a case, the subject position calculation unit 13 does not update the subject position information that is the same as the identifier that is not assigned to the extracted shape. As a result, even if the extracted shapes overlap due to multiple subjects approaching each other, the multiple subject position information will not be in the same position. In this case, the multiple subject positions are maintained at their respective positions up to the previous time. After that, when the subjects move away from each other and the overlapped multiple extracted shapes are separated again, an identifier is assigned to each extracted shape based on the most recent subject position. In other words, in response to the elimination of the overlap of the multiple extracted shapes, the update of each subject position information is resumed.

[0040] By the above processing, the image processing system can track each subject and obtain position information of each subject even when multiple subjects are in the shooting area. Furthermore, by the above processing, the image processing system can track each subject even when the subjects approach or move away from each other, causing overlapping or separation in the generated three-dimensional shape model.

[0041] (Display processing of subject information) Next, a method of calculating and displaying subject information in this embodiment will be described with reference to the flowchart in FIG.

[0042] For the sake of explanation, the image generating unit 6 will be described here as being composed of four units: a foreground image generating unit 61, a background image generating unit 62, a subject information image generating unit 63, and an image synthesizing unit 64. However, the image generating unit 6 does not necessarily have to be composed of these four units, and may be substantially composed as a single image generating unit.

[0043] First, in S601, the foreground image generating unit 61 acquires material data (virtual viewpoint material) stored in the storage unit 4 based on the time information included in the virtual viewpoint information. In S602, an image of the foreground (subject) is generated based on the acquired virtual viewpoint material. FIG. 7(a) is an example of the generated foreground image. Next, in S603, the background image generating unit 62 generates a background image. In this embodiment, the background to be drawn at this time is generated using three-dimensional model data (mesh) and texture data stored in advance. FIG. 7(b) is an example of the generated background image. Although details will be described later, in the process from S604 to S610, the subject information image generating unit 63 generates an image in which information about the subject is drawn at a position based on the subject position information. FIG. 7(c) is an example of the subject information image generated at this time. In S611, the image synthesizing unit 64 synthesizes the generated foreground image, background image, and subject information image based on the depth information of each image, and outputs the resulting image to the display unit 7. Fig. 7(d) is an example of the virtual viewpoint image generated after synthesis at this time. By performing the processes of S601 to S611 every time, an image on which subject information is superimposed is output. That is, when the virtual viewpoint image is a video, the processes of S601 to S611 are performed for each frame (image) constituting the video.

[0044] A specific operation of the subject information image generating section 63 for generating the subject information image at this time will be described.

[0045] First, in S604, the subject information video generating unit 63 acquires position information of all subjects included in the shooting area from the storage unit 4. At this time, the subject information video generating unit 63 acquires not only subject position information at the time included in the virtual viewpoint information, but also subject position information for several frames to several tens of frames before and after the time. From S605 to S610, processing is performed for each subject. First, in S605, the subject information video generating unit 63 judges whether to perform subsequent processing based on an identifier for identifying the subject included in the position information and display target information described later. The display target information is data used to specify a target for displaying subject information, and is data linked to the identifier of the subject. For example, the display target information is generated as data to be displayed if the player is a participating player based on the information of the participating players, and is recorded in the storage unit 4. As a result, the subject information is not displayed to referees and the like who are not included in the display target information. Alternatively, a configuration may be adopted in which a subject to be displayed is selected from an interface (not shown) to generate display target information. In S606, a filtering process is performed on the position information of the subject determined to be a display target. Specifically, position information for multiple frames acquired in S604 is averaged to calculate position information, thereby making it possible to reduce blurring caused by detection errors of the subject position detection unit 8.

[0046] Next, in S607, the subject information video generating unit 63 acquires subject position information of the display target subject from a time in the past, for example, one second ago, and several to several tens of frames before and after the time, and calculates the average value. Note that in S607, if the average value of the subject position information at the past time obtained in this step corresponds to the averaged position information previously calculated in S606, the subject information video generating unit 63 may perform a process of reading the information. This makes it possible to reduce the processing load in S607.

[0047] Next, in S608, the position information after averaging at the past time is subtracted from the position information after averaging at the present time, and then the time difference between the present time and the past time, here 1 second, is divided to obtain the position deviation at the present time. The magnitude of this position deviation at the unit time is the speed, and the direction of the vector on the two-dimensional plane is the moving direction of the subject.

[0048] In S609, the subject information video generating unit 63 generates a subject information video by drawing a triangular moving direction mark, for example, as shown in Fig. 8(a), based on the calculated speed and moving direction of each subject. In S610, the subject information video generating unit 63 determines whether the processes from S605 to S609 have been completed for all subjects. If the processes from S605 to S609 have not been completed for all subjects, the processes from S605 are executed again. If the processes from S605 to S609 have been completed for all subjects, the process of S611 is executed.

[0049] Finally, in S611, the image synthesis unit 64 synthesizes the generated foreground image, background image, and object information image based on the depth information of each image, and outputs the resulting image to the display unit 7.

[0050] The travel direction mark 81 displayed in the subject information video is drawn in the direction of travel of the subject as shown in FIG. 8. The size of the travel direction mark varies depending on the speed of the subject, and for example, the travel direction mark 82 is drawn longer in the travel direction as the speed increases. If the travel direction mark is placed in the center of the subject, it is blocked by the subject's feet, making it difficult to understand the travel direction. Therefore, in this embodiment, it is desirable to draw the travel direction mark at a certain distance from the subject position as shown in the travel direction mark 83 in FIG. 8(a) and rotate the drawing direction around the subject position in accordance with the travel direction. In addition, as shown in FIG. 8(a), the travel direction mark 81 is drawn at a certain distance from the subject position, so that it is drawn on a circumference centered on the subject position. Therefore, by drawing a circular mark 84 or the like concentric with the drawing position of the travel direction mark, it is desirable to make it easier for the viewer, the user, to visually recognize the correspondence between the travel direction mark of the subject and the subject. Also, as shown in FIG. 8(a), the speed information may be displayed as a numerical value.

[0051] Also, information based on past data such as shooting success rate corresponding to the subject's position on the court or field may be displayed around the player.

[0052] With the above system configuration, even if the viewpoint changes significantly in the virtual viewpoint video, it is possible to display the speed, direction of travel, and other additional information near the position of the subject displayed within the field of view. This allows the viewer to check the direction and speed of each subject even when performing operations such as changing the viewpoint while the time in the virtual viewpoint video is stopped, improving the viewer experience. It can also be used for coaching, commentary, etc.

[0053] (Other aspects of the first embodiment) In this embodiment, the subject position detection unit 8 is described as a means for detecting the subject position based on the shape estimation result, but the present invention is not limited to this method of detecting the subject position. For example, a configuration may be adopted in which a position sensor such as a GPS is attached to the player and the sensor value is acquired, or a configuration may be adopted in which the subject position is detected using image recognition technology from images obtained by multiple imaging means.

[0054] In this embodiment, the mark indicating the speed and the direction of travel is a triangular icon (mark), but the present invention is not limited to this. The mark may be in the shape of an arrow as shown in FIG. 8(b) or in other shapes. For example, in this embodiment, the mark is displayed near the floor, but a cone-shaped mark may be displayed at waist height of the subject as shown in FIG. 8(c). In addition, in FIG. 8(a), only the size (length) in the direction of travel is changed, but the size of the mark itself may be changed, and the color may also be changed, for example, to a blue color when the speed is low and to a red color when the speed is high. In addition, the mark may have multiple information by changing the shape of the mark according to the speed and changing the color according to the acceleration.

[0055] In this embodiment, if there is a subject that is not to be displayed among multiple subjects, calculations of the position information and speed of that subject are not performed, but the calculations themselves may be performed but the display may not be performed.

[0056] In addition, in this embodiment, the subject information video generating unit 63 is configured to calculate the speed and traveling direction for each frame when generating the virtual viewpoint video, but this is not necessarily limited to this. For example, the subject position detecting unit 8 may detect the position, calculate the speed and traveling direction, and record them in the storage unit 4. In this case, the subject information video generating unit 63 may obtain information on the speed and traveling direction as well as the position information, and use them to draw a mark.

[0057] In this embodiment, the filtering process of the position information is performed by averaging, but the present invention is not limited to this. For example, a low-pass filter such as an IIR filter or an FIR filter may be used. However, in the case of a configuration in which the speed or the like is calculated each time, if a low-pass filter is used, the value becomes incorrect when the time at which the virtual viewpoint video is played is changed discontinuously. Therefore, in such a configuration, it is desirable to obtain information on nearby times and then average the information.

[0058] In addition, when the speed information is calculated in advance and stored in the storage unit 4 in this manner, the speed information of a specific player may be displayed in graph form. In addition, for example, the acceleration may be calculated from the speed history, and it may be determined that the acceleration of the player is decreasing, and the fatigue level of the player may be calculated and displayed in a simple manner.

[0059] (Other configurations) In the above embodiment, each processing unit shown in Fig. 1 is configured as hardware. However, the processing performed by each processing unit shown in these figures may be configured as a computer program.

[0060] FIG. 9 is a block diagram showing an example of the hardware configuration of a computer applicable to the indirect position estimation device according to each of the above embodiments.

[0061] The CPU 901 controls the entire computer using computer programs and data stored in the RAM 902 and the ROM 903, and executes the processes described above as being performed by the indirect position estimation device according to each of the above embodiments. That is, the CPU 901 functions as each of the processing units shown in FIG.

[0062] The RAM 902 has an area for temporarily storing computer programs and data loaded from an external storage device 906, data acquired from the outside via an I / F (interface) 907, etc. Furthermore, the RAM 902 has a work area used when the CPU 901 executes various processes. That is, the RAM 902 can be allocated as a frame memory, for example, or can provide various other areas as appropriate.

[0063] The ROM 903 stores setting data for the computer, a boot program, and the like. The operation unit 904 is composed of a keyboard, a mouse, and the like, and can be operated by a user of the computer to input various instructions to the CPU 901. The output unit 905 displays the results of processing by the CPU 901. The output unit 905 is composed of, for example, a liquid crystal display. For example, the viewpoint instruction unit 5 is composed of the operation unit 904, and the display unit 7 is composed of the output unit 905.

[0064] The external storage device 906 is a large-capacity information storage device, such as a hard disk drive. The external storage device 906 stores an operating system (OS) and computer programs for causing the CPU 901 to realize the functions of each unit shown in Fig. 1. Furthermore, the external storage device 906 may store image data to be processed.

[0065] Computer programs and data stored in the external storage device 906 are loaded into the RAM 902 as appropriate under the control of the CPU 901, and become the subject of processing by the CPU 901. A network such as a LAN or the Internet, and other devices such as a projector or display device can be connected to the I / F 907, and the computer can obtain and transmit various information via the I / F 907. In the first embodiment, the imaging unit 1 is connected to this, and the captured images are input and each of them is controlled. 908 is a bus connecting the above-mentioned respective units.

[0066] The operation of the above-mentioned configuration is controlled mainly by the CPU 901 as described in the previous embodiment.

[0067] In another configuration, the above-mentioned functions can be achieved by supplying a storage medium on which a computer program code that realizes the above-mentioned functions is recorded to a system, and the system reads and executes the computer program code. In this case, the computer program code read from the storage medium itself realizes the functions of the above-mentioned embodiments, and the storage medium on which the computer program code is stored constitutes the present invention. Also included is a case in which an operating system (OS) running on a computer performs part or all of the actual processing based on the instructions of the program code, and the above-mentioned functions are realized by that processing.

[0068] Furthermore, the present invention may be realized in the following form: That is, the computer program code read from the storage medium is written into a memory provided in a function expansion card inserted into a computer or a function expansion unit connected to the computer. Then, based on the instructions of the computer program code, a CPU provided in the function expansion card or function expansion unit performs part or all of the actual processing to realize the above-mentioned functions.

[0069] When the present invention is applied to the above-mentioned storage medium, the storage medium stores computer program code corresponding to the processes described above.

[0070] The disclosure of this embodiment includes the following configurations and methods.

[0071] (Configuration 1) A detection means for detecting the position of a subject; an image generating means for generating a virtual viewpoint image from the virtual viewpoint material; an image processing device comprising: a display means for displaying information relating to movement of a subject as subject information at a position based on the subject position information detected by said detection means;

[0072] (Configuration 2) 2. The image processing device according to configuration 1, wherein the display means comprises a calculation means for calculating a speed and a traveling direction of the subject from the subject position information at a plurality of times, and displays the speed and the traveling direction as the subject information.

[0073] (Configuration 3) 3. The image processing device according to configuration 2, wherein the display means indicates the moving direction of the subject by using an arrow, a triangle or an icon similar thereto near the subject position as the subject information.

[0074] (Configuration 4) 4. The image processing device according to configuration 3, wherein the display means changes at least one of the size, length and color of the icon indicating the traveling direction based on the calculated speed of the subject as the subject information.

[0075] (Configuration 5) 4. The image processing device according to configuration 3, wherein the display means displays, as the subject information, a circle or another icon surrounding the subject near the feet of the subject based on the subject position information.

[0076] (Configuration 6) 2. The image processing device according to configuration 1, wherein when the virtual viewpoint material includes a plurality of subjects, the display means displays information for each of the subjects based on the subject position information.

[0077] (Configuration 7) 7. The image processing device according to configuration 6, wherein the display means does not display subject information for a subject that is not designated in advance, even if the subject is included in the virtual viewpoint material.

[0078] (Configuration 8) a detection step of detecting the position of the subject; an image generation step of generating a virtual viewpoint image from the virtual viewpoint material; A control method comprising the steps of: displaying information relating to movement of the subject as subject information at a position based on the subject position information detected in the detection step.

[0079] (Configuration 9) a detection step of detecting the position of the subject; an image generation step of generating a virtual viewpoint image from the virtual viewpoint material; a display step of displaying information relating to the movement of the subject as subject information at a position based on the subject position information detected in the detection step.

Claims

1. A detection means for detecting the position of a subject; an image generating means for generating a virtual viewpoint image from the virtual viewpoint material; an image processing device comprising: a display means for displaying information relating to movement of a subject as subject information at a position based on the subject position information detected by said detection means;

2. 2. The image processing apparatus according to claim 1, wherein said display means comprises a calculation means for calculating a speed and a traveling direction of said subject from said subject position information at a plurality of times, and displays the speed and the traveling direction as said subject information.

3. 3. The image processing apparatus according to claim 2, wherein said display means indicates the moving direction of the subject by using an arrow, a triangle or an icon similar thereto in the vicinity of the subject position as said subject information.

4. 4. The image processing apparatus according to claim 3, wherein said display means changes at least one of a size, a length and a color of said icon indicating a traveling direction based on the calculated speed of the subject as said subject information.

5. 4. The image processing apparatus according to claim 3, wherein said display means displays, as said subject information, a circle or another icon surrounding the subject near the feet of said subject based on said subject position information.

6. 2. The image processing apparatus according to claim 1, wherein when the virtual viewpoint material includes a plurality of subjects, the display means displays information for each of the subjects based on the subject position information.

7. 7. The image processing apparatus according to claim 6, wherein said display means does not display subject information for a subject that is not designated in advance even if the subject is included in the virtual viewpoint material.

8. a detection step of detecting the position of the subject; an image generation step of generating a virtual viewpoint image from the virtual viewpoint material; A control method comprising the steps of: displaying information relating to movement of the subject as subject information at a position based on the subject position information detected in the detection step.

9. a detection step of detecting the position of the subject; an image generation step of generating a virtual viewpoint image from the virtual viewpoint material; a display step of displaying information relating to the movement of the subject as subject information at a position based on the subject position information detected in the detection step.

Images