Image processing apparatus, control method, and program

Abstract

To facilitate tracking a person's position in generating a virtual viewpoint image.SOLUTION: An image processing apparatus includes: imaging means for capturing images of a subject from a plurality of directions; estimation means for estimating a three-dimensional shape of the subject using a captured image acquired by the imaging means; viewpoint instruction means for outputting virtual viewpoint information; image generation means for generating a virtual viewpoint image on the basis of the captured image, the three-dimensional shape, and the virtual viewpoint information; shape extraction means for extracting a part of the three-dimensional shape to output it as an extracted shape; identifier setting means for assigning an identifier to the extracted shape; tracking means for tracking a position of the extracted shape; subject position calculation means for obtaining a representative position of the extracted shape to which the same identifier is assigned on the basis of a tracking result of the tracking means and outputting it as a subject position.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device that generates a virtual viewpoint video. [Background technology]

[0002] There is a virtual viewpoint video generation system that generates a virtual viewpoint video, which is an image seen from a virtual viewpoint specified by a user, based on an image captured by a shooting system using multiple cameras. Patent Document 1 describes a system in which, after transmitting images captured by multiple cameras, an image computing server (image processing device) extracts images with large changes from the captured images as foreground images and images with small changes as background images.

[0003] In recent years, virtual reality (VR) using head-mounted displays (HMD) has made it possible to have a viewing experience that makes one feel as if they are in a virtual space created using 3DCG. In addition, virtual artists using avatars have begun to hold concerts in this virtual reality. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-211828 A Summary of the Invention [Problem to be solved by the invention]

[0005] In such concerts, there is a demand to recreate the concert of an actual artist in a virtual 3D space using the aforementioned virtual viewpoint video. When recreating a concert of an actual artist in a virtual 3D space using virtual viewpoint video, it is necessary to track the position of the artist, for example, in order to synthesize a sound source according to the position of the artist. However, especially when there are multiple artists, it can be difficult to track the position of each individual person. [Means for solving the problem]

[0006] The image processing device of the present invention comprises an imaging means for imaging a subject from multiple directions, an estimation means for estimating a three-dimensional shape of the subject using the captured images acquired by the imaging means, a viewpoint indicating means for outputting virtual viewpoint information, an image generation means for generating a virtual viewpoint image based on the captured image, the three-dimensional shape, and the virtual viewpoint information, a shape extraction means for cutting out a portion of the three-dimensional shape and outputting it as an extracted shape, an identification setting means for setting an identifier to the extracted shape, a tracking means for tracking the position of the extracted shape, and a subject position calculation means for determining a representative position of the extracted shape that has been assigned the same identifier based on the tracking result of the tracking means, and outputting this as the subject position. Effect of the Invention

[0007] According to the present invention, tracking of a person's position is facilitated in generating a virtual viewpoint video. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example of an image processing system. [Diagram 2] FIG. 1A is a diagram showing an example of the position of a subject; FIG. 1B is a diagram showing an example of a subject extracted by a shape extraction unit; FIG. 1C is a diagram showing an example of the position of a subject in a different state; FIG. 1D is a diagram showing an example of a subject extracted by a shape extraction unit; and FIG. 1E is a diagram showing an example of an extracted shape. [Diagram 3]1A is an example of an extracted shape having an identifier attached thereto, and FIG. 1B is an example of a graphical user interface for displaying the extracted shape and the identifier. [Figure 4] FIG. 13 is a diagram showing an example of a representative position. [Diagram 5] 13 is a flowchart showing an example of a tracing analysis process performed by a tracing unit. [Figure 6] 13 is an example of a method for assigning an identifier to an extracted shape. [Figure 7] FIG. 13 is a diagram for explaining reassignment of identifiers from a graphical user interface. [Figure 8] 11 is a diagram for explaining how a viewpoint instruction unit uses subject position information; FIG. [Figure 9] FIG. 2 is a block diagram showing an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [First embodiment] (System configuration and operation of image processing device) An example of the configuration of an image processing system for generating a virtual viewpoint video according to this embodiment is shown in Fig. 1. The image processing system includes, for example, an imaging unit 1, a synchronization unit 2, a three-dimensional shape estimation unit 3, an accumulation unit 4, a viewpoint instruction unit 5, a video generation unit 6, a display unit 7, a sound collection unit 8, an audio signal generation unit 9, an audio output unit 10, a shape extraction unit 11, a tracking unit 12, a subject position calculation unit 13, and an identification setting unit 14. The image processing system may be configured by one image processing device, or may be a system configured by multiple image processing devices. In the following description, the image processing system will be described as being one image processing device.

[0010] The operation of each component in an image processing device that generates a virtual viewpoint video to which this system is applied will be outlined below. First, multiple imaging units 1 perform imaging in synchronization with each other based on a synchronization signal from a synchronization unit 2. The imaging units 1 output the captured images to a three-dimensional shape estimation unit 3. The imaging units 1 are installed so as to surround an imaging area including the subject so that the subject can be photographed from multiple directions. The three-dimensional shape estimation unit 3 uses the input images taken from multiple viewpoints to extract, for example, the silhouette of the subject, and then generates a three-dimensional shape of the subject using a volume intersection method or the like. The three-dimensional shape estimation unit 3 also outputs the generated three-dimensional shape of the subject and the captured images to a storage unit 4. Here, the subject is an object that is the subject of three-dimensional shape generation, and includes people and items handled by people.

[0011] The sound collection unit 8 collects sounds from a plurality of locations within the shooting area or the sounds of each subject, and outputs the collected sound data to the storage unit 4 .

[0012] The storage unit 4 saves and accumulates the following data group as data (material data) used to generate the virtual viewpoint video. The data used to generate the virtual viewpoint video specifically includes the captured image and the three-dimensional shape of the subject input from the three-dimensional shape estimation unit 3. The data used to generate the virtual viewpoint video also includes camera parameters such as the position and orientation and optical characteristics of each imaging unit, and audio data acquired by the sound collection unit 8. Note that a background model and a background texture image are saved (recorded) in advance in the storage unit 4 as data used to generate the background of the virtual viewpoint video.

[0013] The viewpoint instruction unit 5 is made up of a viewpoint operation unit, which is a physical user interface such as a joystick or jog dial (not shown), and a display unit for displaying a virtual viewpoint video. The virtual viewpoint of the virtual viewpoint image displayed here can be changed by the viewpoint operation unit. In response to changes in the virtual viewpoint by the viewpoint operation unit, a virtual viewpoint image is generated as needed by the image generation unit 6, which will be described later, and displayed on the display unit. This display unit may share the display unit 7, which will be described later, or may be provided with a separate display device. The viewpoint instruction unit 5 generates virtual viewpoint information based on input from the viewpoint operation unit, and outputs the generated virtual viewpoint information to the image generation unit 6. The virtual viewpoint information includes information equivalent to external parameters of the camera, such as the position and orientation of the virtual viewpoint, information equivalent to internal parameters of the camera, such as the focal length and angle of view, and time information specifying the shooting time to be played back.

[0014] Based on the time information included in the input virtual viewpoint information, the video generation unit 6 acquires the material data of the shooting time from the storage unit 4. The video generation unit 6 generates a virtual viewpoint video at the set virtual viewpoint using the three-dimensional shape of the subject and the shot image from the acquired material data, and outputs the generated video to the display unit 7.

[0015] The display unit 7 is a display means for displaying the image input from the image generating unit 6. The display unit 7 is configured with a display, a head mounted display (HMD), or the like.

[0016] The sound signal generating unit 9 acquires sound data at the shooting time from the storage unit 4 based on the time information included in the input virtual viewpoint information. The sound signal generating unit 9 arranges the acquired sound data as a virtual sound source in three-dimensional space. The sound signal generating unit 9 also generates a sound signal based on the distance between the virtual sound source and the virtual viewpoint position and the direction of the virtual viewpoint, and outputs the sound signal to the sound output unit 10.

[0017] The audio output unit 10 is composed of a speaker, a headphone, etc., and performs audio output (reproduction) based on an input audio signal.

[0018] (How to track the subject position) Next, a method for tracking the subject position in this embodiment will be described.

[0019] First, three-dimensional shape estimation unit 3 generates a three-dimensional shape of the subject, and outputs the generated three-dimensional shape to accumulation unit 4 and also outputs the generated three-dimensional shape to shape extraction unit 11.

[0020] The shape extraction unit 11 cuts out the lower part of the three-dimensional shape of the subject as shown in FIG. 2(b) from the three-dimensional shape of the subject as shown in FIG. 2(a). In this embodiment, the cutout is made from the bottom surface of the circumscribed rectangular parallelepiped of the three-dimensional shape of the subject to a predetermined height (e.g., a height equivalent to 50 cm). For example, as shown in FIG. 2(c), when one of the subjects is standing and the other subject is away from the floor of the shooting area by jumping or the like, the three-dimensional shape of the subject is cut out as shown in FIG. 2(d). That is, for both of the three-dimensional shapes of the subjects, the three-dimensional shape is cut out from the part equivalent to the feet to a predetermined height.

[0021] Next, as shown in FIG. 2(e), the shape extraction unit 11 projects the cut-out three-dimensional shape onto a plane viewed from directly above the three-dimensional shape of the subject to generate a two-dimensional image. In this embodiment, the shape extraction unit 11 parallel projects the cut-out three-dimensional shape onto a two-dimensional plane equivalent to the feet (floor surface). In this embodiment, the image projected onto the plane is a binary image in which the cut-out three-dimensional shape is white and the other parts are black. The shape extraction unit 11 divides this two-dimensional image into independent regions and obtains circumscribing rectangles 201 to 204 shown in FIG. 2(e). The shape extraction unit 11 outputs vertex information of this circumscribing rectangle as an extracted three-dimensional shape (extracted shape). Here, the shape extraction unit 11 converts the vertex information of the circumscribing rectangle into the same coordinate system and unit as the three-dimensional space of the shooting region before outputting it. In addition, to determine the independent shapes in the shape extraction unit 11, a method such as continuous component analysis is used for the projected two-dimensional image. By using such a method, the shape extraction unit 11 can divide the three-dimensional shape into individual regions.

[0022] The identification setting unit 14 assigns an identifier to the extracted shape output by the shape extraction unit 11. Specifically, the identification setting unit 14 calculates the distance between each extracted shape, and assigns an identifier according to the distance between the extracted shapes. For example, as shown in FIG. 3(a), the identification setting unit 14 assigns the same identifier to extracted shapes whose distance between the extracted shapes is less than a predetermined distance (solid arrow), and assigns a different identifier to extracted shapes whose distance between the extracted shapes is equal to or greater than the predetermined distance (dashed arrow). The threshold value of the predetermined distance used as the criterion for judgment is preferably a distance equivalent to the distance between the legs of the subject when standing. In this embodiment, the explanation will be given assuming that the threshold value of the predetermined distance is set to 50 cm.

[0023] The identification setting unit 14 displays the assigned identifiers on a display unit provided in the identification setting unit 14 by a graphical interface (GUI) as shown in FIG. 3(b). The user operates the image processing system while viewing this GUI. Specifically, the identification setting unit 14 displays the current identifier assignment (identifier assignment in the initial state) on the graphical interface by distinguishing it with at least one of characters and color coding. In FIG. 3(b), the identification setting unit 14 displays each identifier by using both characters and color coding. The user checks the GUI and confirms whether the desired identifier assignment is set as the initial state. If the desired identifier assignment is not set, the user instructs the subject to change its standing position or close its legs, and repeats this until the desired assignment is set. Alternatively, the user operates the image processing system via the GUI and issues a change instruction so that the desired identifier assignment is set. If the desired identifier assignment is set, the user presses, for example, a decision button (initial identifier decision button) on the graphical interface as shown in FIG. 3(b). In response to this operation, the identification setting unit 14 decides the initial state identifier. Then, the identification setting unit 14 outputs the identifier assigned to each extracted shape to the tracking unit 12.

[0024] In response to receiving an identifier from the identification setting unit 14, the tracking unit 12 assigns the identifier to each extracted shape as an initial state. After that, the tracking unit 12 tracks the extracted shape to which the identifier has been assigned. The identifier assigned to the extracted shape during tracking is not the identifier determined by the identification setting unit 14, but an identifier determined based on the tracking result of the position of each extracted shape by the tracking unit 12. In tracking (tracking analysis) of the extracted shape, the tracking unit 12 tracks the extracted shape based on the position of each extracted shape at the time immediately before the shooting time of the extracted shape, the identifier of each extracted shape, and information on the subject position input from the subject position calculation unit described later. The specific processing of tracking by the tracking unit 12 will be described later. The tracking unit 12 assigns an identifier to each extracted shape at the time based on the result of the tracking analysis, and outputs each extracted shape to the subject position calculation unit 13.

[0025] The subject position calculation unit 13 obtains a representative position for each extracted shape to which an identifier has been assigned, input by the tracking unit 12. For example, as shown in Fig. 4, the subject position calculation unit 13 obtains a position indicating each extracted shape group for each extracted shape group to which the same identifier has been assigned, such as representative positions 401 and 402. In this embodiment, the representative position is the center position of the extracted shape group.

[0026] However, since this representative position is affected by shape estimation errors and fluctuations in the boundary parts when the shape is cut out by the shape extraction unit 11, the position may fluctuate at each time even if the subject is stationary. Therefore, in this embodiment, the subject position calculation unit 13 performs processing such as low-pass filtering and moving average in the time direction on the center position information at each time to generate position information with high frequency components suppressed. Then, the subject position calculation unit 13 outputs the position information of the representative position together with the identifier as the position of the subject to the tracking unit 12. In addition, the subject position calculation unit 13 records (accumulates) information in which information on the time when the three-dimensional shape that was the basis of the tracking analysis was photographed is added to the position information of the representative position as subject position information (subject position information) in the accumulation unit 4.

[0027] The sound signal generating unit 9 acquires the sound data at the corresponding shooting time and the subject position information from the storage unit 4 based on the time information included in the virtual viewpoint information. In this embodiment, since the subject position information does not include information in the height direction, the sound signal generating unit 9 adds height information corresponding to the subject to the subject position information. For example, in the case of this embodiment, the sound signal generating unit 9 adds height information corresponding to the subject's mouth and musical instrument to the subject position information. Then, the sound signal generating unit 9 links the sound data collected by the sound collecting unit (microphone) corresponding to the identifier added to the subject position information including the height information. This allows the sound signal generating unit 9 to add the position of a virtual sound source that reproduces the sound data to the position indicated by the subject position information.

[0028] With the above system configuration, the image processing system 1 can generate three-dimensional sound based on the viewpoint position of the virtual viewpoint video and the position of the subject in the virtual viewpoint video. The virtual viewpoint video generated by the image processing system 1 allows the user to experience highly realistic video and audio.

[0029] (Tracking analysis process by tracking unit 12) Next, an example of the tracking and analysis process of the extracted shape position in the tracking section 12 will be described with reference to the flowchart of FIG.

[0030] In step S501, the tracking unit 12 performs an initialization process upon receiving an input from the identification setting unit 14. Specifically, the tracking unit 12 acquires the identifiers of the respective extracted shapes input from the identification setting unit 14.

[0031] In step S 502 , the tracking unit 12 acquires the next extracted shape input from the shape extraction unit 11 .

[0032] In step S503, the tracking unit 12 assigns the identifiers acquired from the identification setting unit 14 to the acquired extracted shapes, and outputs the extracted shapes to which the identifiers have been assigned to the subject position calculation unit 13.

[0033] In step S504 , the subject position calculation unit 13 obtains the subject position from the group of extracted shapes to which the same identifier is assigned, and outputs the subject position to the tracking unit 12 .

[0034] The above processing in steps S501 to S504 corresponds to the initialization processing.

[0035] The subsequent steps S505 to S509 are executed every time while the imaging unit 1 is imaging the subject. When the imaging process of the subject by the imaging unit 1 is completed, the process of step S509 is completed and the process of this flowchart is terminated.

[0036] In step S505, the tracking unit 12 acquires the extracted shape input from the shape extraction unit 11 and the subject position at the previous time (previous time) calculated by the subject position calculation unit 13. The previous time is, for example, the shooting time of the extracted shape generated one frame before the extracted shape currently being processed. Here, for comparison, the current time is also referred to as the current time. Here, the current time refers to the shooting time of the image used to generate the extracted shape currently being processed.

[0037] In step S506, when the subject position at the previous time overlaps with the representative position of each extracted shape at the current time, the tracking unit 12 assigns to the extracted shape an identifier that is assigned to the subject position that overlaps with the representative position. Here, in step S506, when the representative position of one extracted shape overlaps with multiple subject positions, the tracking unit 12 assigns to the extracted shape an identifier indicating "undeterminable" at the current time. This is because, for example, multiple extracted shapes with different identifiers assigned may overlap at the current time, such as when two subjects are close to each other, so an identifier indicating "undeterminable" is assigned in the processing of this step. The extracted shape to which an identifier including an identifier indicating "undeterminable" is assigned is subjected to the processing of S509 described later.

[0038] In step S507, if the representative position of an extracted shape to which an identifier has not yet been assigned overlaps with the extracted shape of the previous time, the tracking unit 12 assigns the identifier assigned to the extracted shape of the previous time to the extracted shape of the current time.

[0039] In step S508, if there is another extracted shape to which an identifier has already been assigned at the current time within a predetermined range from an extracted shape to which an identifier has not yet been assigned, the tracking unit 12 assigns the identifier assigned to the other extracted shape. The predetermined range is preferably a range equivalent to the width of the subject's legs when standing. For example, the predetermined range is a range of a radius of 50 cm from the center of the extracted shape. Here, if there are multiple other extracted shapes to which an identifier has been assigned within a predetermined range from a certain extracted shape, the tracking unit 12 assigns the identifier of the extracted shape that is closest among the other extracted shapes to the extracted shape. For an extracted shape to which an identifier has not been assigned at the stage of completing the processing up to step S508, the tracking unit 12 determines that the extracted shape is not a tracking target. In this case, the tracking unit 12 does not output the extracted shape determined to be not a tracking target to the subject position calculation unit 13.

[0040] In step S509, the tracking unit 12 outputs to the subject position calculation unit 13 the extracted shape to which an identifier has been assigned in the processes of steps S506 to S508, and the identifier assigned thereto.

[0041] In step S510, a control unit (not shown) determines whether the imaging process of the subject by the imaging unit 1 has ended. If it is determined that the imaging process of the subject by the imaging unit 1 has not ended, the process of step S508 is executed. If it is determined that the imaging process of the subject by the imaging unit 1 has ended, the process of this flowchart ends.

[0042] Note that the processes from step S506 to step S508 are performed for each extracted shape. By repeating the processes from step S506 to step S509, the identifier set by the identification setting unit 14 is linked to the extracted shape at each time. Using this identifier, the subject position calculation unit 13 can distinguish between each subject and find the subject position.

[0043] In addition, when the tracking unit 12 assigns an "undeterminable" identifier to the extracted shape, there is a possibility that some of the identifiers set in the initial setting will not be assigned at a certain time. In such a case, the subject position calculation unit 13 does not update the subject position information that is the same as the identifier that is not assigned to the extracted shape. As a result, even if the extracted shapes overlap due to multiple subjects approaching each other, the multiple subject position information will not be in the same position. In this case, the multiple subject positions are maintained at their respective positions up to the previous time. After that, when the subjects move away from each other and the overlapped multiple extracted shapes are separated again, an identifier is assigned to each extracted shape based on the most recent subject position. In other words, in response to the elimination of the overlap of the multiple extracted shapes, the update of each subject position information is resumed.

[0044] By the above processing, the image processing system can track each subject and obtain position information of each subject even when multiple subjects are in the shooting area. Furthermore, by the above processing, the image processing system can track each subject even when the subjects approach or move away from each other, causing superimposition or separation in the generated three-dimensional shape model. As a result, the image processing system can generate a virtual viewpoint image having a three-dimensional sound as if the sound is coming from the mouth of each subject, even if multiple artists change positions to sing or hold hands in close proximity. By watching such a virtual viewpoint image, a user can have a highly realistic video experience and audio experience.

[0045] (Other aspects of the first embodiment) In the first embodiment, a specific embodiment of the image processing system 1 is shown, but the image processing system 1 is not necessarily limited to the specific form shown here.

[0046] For example, in the first embodiment, the shape extraction unit 11 is configured to receive the shape estimation result of the three-dimensional shape estimation unit 3 from the three-dimensional shape estimation unit 3, but this is not limited to this. For example, the shape extraction unit 11 may obtain the shape estimation result of the three-dimensional shape estimation unit 3 from the shape estimation result that the three-dimensional shape estimation unit 3 has accumulated in the accumulation unit 4.

[0047] Additionally, in the first embodiment, the predetermined height cut out by the shape extraction unit 11 from the bottom surface of the circumscribing rectangular parallelepiped of the three-dimensional shape is set to 50 cm, but it is not necessarily required to be 50 cm. For example, the predetermined height may be set in a range of 20 cm to 100 cm, etc. Also, the predetermined height may be automatically set to a height between the knees and waist based on the height of the subject.

[0048] In the first embodiment, the shape extraction unit 11 processes to extract a predetermined height from the bottom surface of the circumscribed rectangular parallelepiped, but this is not limited thereto, and the shape extraction unit 11 may extract a predetermined height from the floor surface of the shooting range. In this case, if the subject is standing on a structure such as a step stool in the shooting area, the shape extraction unit 11 may extract a predetermined height from the top surface of the structure when extracting the subject.

[0049] In the first embodiment, the shape extraction unit 11 divides the extracted shape into independent shapes by projecting it onto a two-dimensional image and processing it, but this is not limited to this. For example, the shape extraction unit 11 may divide the extracted shape into three-dimensional shapes after determining whether the shapes are independent. In this case, in the subsequent processing, the tracking unit 12 tracks the subject based on the three-dimensional shape of the subject.

[0050] In the first embodiment, the distance at which the identification setting unit 14 assigns the same identifier to multiple extracted shapes is preferably a distance (50 cm) equivalent to the distance between the legs of a subject standing, but is not necessarily limited to this. For example, when there are many subjects and the subjects are close to each other, the subjects narrow the distance between their legs, the distance at which the same identifier is assigned to multiple extracted shapes may be a short distance such as 30 cm. On the other hand, when the shooting area is wide, or when the distance between the legs can be wide, the distance at which the same identifier is assigned to multiple extracted shapes may be a long distance such as 100 cm.

[0051] In the first embodiment, the identification setting unit 14 is configured to assign identifiers based on the distance between each extracted shape, but this is not necessarily limited to this. For example, the identification setting unit 14 may assign identifiers by determining whether or not the extracted shapes are included in any of the circumscribed rectangular parallelepipeds of the three-dimensional shape. For example, in the example shown in FIG. 6(a), the identification setting unit 14 assigns identifier A to extracted shapes 603 and 604 included in the circumscribed rectangular parallelepiped 601, and assigns identifier B to extracted shapes 605 and 606 included in the circumscribed rectangular parallelepiped 602. Alternatively, a user may determine in advance the area to which a predetermined identifier is assigned to the identification setting unit 14. For example, as shown in FIG. 6(b), the user first inputs information on a judgment area 611 to which identifier A is assigned and a judgment area 612 to which identifier B is assigned from an input unit (not shown) to the image processing system 1. Then, the identification setting unit 14 assigns identifier A to extracted shapes 613 and 614 included in the judgment area 611, and identifier B to extracted shapes 605 and 606 included in the judgment area 612.

[0052] In the first embodiment, the identifiers are automatically set by the identification setting unit 14, but the present invention is not limited to this. The user may determine the identifier to be assigned to the extracted shape. For example, a case will be described in which, in a state where there are multiple subjects, the user assigns an identifier to each subject after the identifiers are assigned. In this case, for example, the user instructs a transition to an identifier reallocation mode from a graphical user interface as shown in FIG. 7(a). Then, the user specifies the identifier to be assigned by, for example, clicking in order on each of the subject positions 701 to 703 displayed on the screen as shown in FIG. 7(a). For example, it is assumed that identifier C is assigned to subject position 701, identifier B is assigned to subject position 702, and identifier C is assigned to subject position 703. On the other hand, when identifiers A to C are assigned to subject positions 701 to 703 in this order, the user clicks subject positions 701 to 703 in order. The identification setting unit 14 receives the clicked order as an input and reallocates identifiers based on the order as shown in FIG. 7(b). Then, the identification setting unit 14 outputs the extracted shape and the identifier reassigned to the extracted shape to the tracking unit 12.

[0053] In the first embodiment, the information on the subject position obtained by the tracking analysis is used as the sound source playback position, but the present invention is not limited to this use. For example, the viewpoint instruction unit 5 may acquire and use the subject position information stored in the storage unit 4. In this case, the viewpoint instruction unit 5 may generate a virtual viewpoint that can always rotate around the subject even if the subject moves, with the position of the rotation center of the virtual viewpoint set to the subject position 800 as shown in FIG. 8(a), for example. The viewpoint instruction unit 5 may also be configured to set the line of sight of the virtual viewpoint to the subject position 800 as shown in FIG. 8(b). In this case, the image processing system 1 can generate a virtual viewpoint video of a viewpoint in which the virtual viewpoint arranged at a semi-fixed position automatically rotates horizontally as the subject moves.

[0054] (Other configurations) In the above embodiment, each processing unit shown in Fig. 1 is configured as hardware. However, the processing performed by each processing unit shown in these figures may be configured as a computer program.

[0055] FIG. 9 is a block diagram showing an example of the hardware configuration of a computer applicable to the indirect position estimation device according to each of the above embodiments.

[0056] The CPU 901 controls the entire computer using computer programs and data stored in the RAM 902 and the ROM 903, and executes the processes described above as being performed by the indirect position estimation device according to each of the above embodiments. That is, the CPU 901 functions as each of the processing units shown in FIG.

[0057] The RAM 902 has an area for temporarily storing computer programs and data loaded from an external storage device 906, data acquired from the outside via an I / F (interface) 907, etc. Furthermore, the RAM 902 has a work area used when the CPU 901 executes various processes. That is, the RAM 902 can be allocated as a frame memory, for example, or can provide various other areas as appropriate.

[0058] The ROM 903 stores setting data for the computer, a boot program, and the like. The operation unit 904 is composed of a keyboard, a mouse, and the like, and can be operated by a user of the computer to input various instructions to the CPU 901. The output unit 905 displays the results of processing by the CPU 901. The output unit 905 is composed of, for example, a liquid crystal display. For example, the viewpoint instruction unit 5 is composed of the operation unit 904, and the display unit 7 is composed of the output unit 905.

[0059] The external storage device 906 is a large-capacity information storage device, such as a hard disk drive. The external storage device 906 stores an operating system (OS) and computer programs for causing the CPU 901 to realize the functions of each unit shown in Fig. 1. Furthermore, the external storage device 906 may store image data to be processed.

[0060] Computer programs and data stored in the external storage device 906 are loaded into the RAM 902 as appropriate under the control of the CPU 901, and become the subject of processing by the CPU 901. A network such as a LAN or the Internet, and other devices such as a projector or display device can be connected to the I / F 907, and the computer can obtain and transmit various information via the I / F 907. In the first embodiment, the imaging unit 1 is connected to this, and the captured images are input and each of them is controlled. 908 is a bus connecting the above-mentioned respective units.

[0061] The operation of the above-mentioned configuration is controlled mainly by the CPU 901 as described in the previous embodiment.

[0062] In another configuration, the above-mentioned functions can be achieved by supplying a storage medium on which a computer program code that realizes the above-mentioned functions is recorded to a system, and the system reads and executes the computer program code. In this case, the computer program code read from the storage medium itself realizes the functions of the above-mentioned embodiments, and the storage medium on which the computer program code is stored constitutes the present invention. Also included is a case in which an operating system (OS) running on a computer performs part or all of the actual processing based on the instructions of the program code, and the above-mentioned functions are realized by that processing.

[0063] Furthermore, the present invention may be realized in the following form: That is, the computer program code read from the storage medium is written into a memory provided in a function expansion card inserted into a computer or a function expansion unit connected to the computer. Then, based on the instructions of the computer program code, a CPU provided in the function expansion card or function expansion unit performs part or all of the actual processing to realize the above-mentioned functions.

[0064] When the present invention is applied to the above-mentioned storage medium, the storage medium stores computer program code corresponding to the processes described above.

[0065] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) An imaging means for imaging a subject from a plurality of directions; an estimation means for estimating a three-dimensional shape of the subject using the captured image acquired by the imaging means; A viewpoint specifying means for outputting virtual viewpoint information; an image generating means for generating a virtual viewpoint image based on the captured image, the three-dimensional shape, and the virtual viewpoint information; A shape extraction means for cutting out a part of the three-dimensional shape and outputting it as an extracted shape; an identification setting means for setting an identifier to the extracted shape; a tracking means for tracking the position of the extracted shape; a subject position calculation means for calculating a representative position of the extracted shape to which the same identifier is assigned based on the tracking result of the tracking means, and outputting the representative position as the subject position; An image processing device comprising: (Configuration 2) 2. The image processing device according to configuration 1, wherein the identification setting means sets an initial state identifier for the extracted shape. (Configuration 3) 3. The image processing device according to configuration 1 or 2, wherein the identification setting means sets an identifier for each extracted shape based on a distance between the extracted shapes. (Configuration 4) The image processing device according to any one of configurations 1 to 3, characterized in that the identification setting means determines whether each extracted shape is included in a circumscribed rectangular parallelepiped of any of the three-dimensional shapes, and sets the same identifier when the extracted shapes are included in the same circumscribed rectangular parallelepiped. (Configuration 5) The image processing device according to any one of configurations 1 to 4, wherein the identification setting means sets an area to which a predetermined identifier is to be assigned, and sets the predetermined identifier set in the area for an extracted shape included in the area. (Configuration 6) 6. The image processing device according to any one of configurations 1 to 5, wherein the identification setting means reassigns an identifier to an assigned identifier in response to an instruction from a user. (Configuration 7) The image processing device according to any one of configurations 1 to 6, wherein the shape extraction means converts the extracted shape into a two-dimensional image by parallel projection onto a two-dimensional plane corresponding to a floor surface, cuts out each independent area from the two-dimensional image, and outputs a circumscribing rectangle of each area on the two-dimensional image as the extracted shape. (Configuration 8) 8. The image processing device according to any one of configurations 1 to 7, wherein the shape extraction means cuts out a shape from a bottom surface of a circumscribed rectangular parallelepiped of each three-dimensional shape to a predetermined height. (Configuration 9) 9. The image processing device according to any one of configurations 1 to 8, wherein the shape extraction means cuts out the shape from a floor surface of the shooting area to a predetermined height. (Configuration 10) The image processing device according to any one of configurations 1 to 9, characterized in that, when the three-dimensional shape is located on a structure installed in a shooting area, the shape extraction means extracts a shape from an upper surface of the structure to a predetermined height. (Configuration 11) The image processing device according to any one of configurations 1 to 10, characterized in that the tracking means determines, at each time, an identifier of the extracted shape at the time based on the extracted shape at the time and the position and identifier of the extracted shape at the previous time. (Configuration 12) The image processing device according to any one of configurations 1 to 11, characterized in that the tracking means determines, at each time, an identifier of the extracted shape at the time based on the extracted shape at the time and the subject position and its identifier at the previous time. (Configuration 13) The image processing device described in any one of configurations 1 to 12, characterized in that the tracking means assigns an identifier of the subject position to the extracted shape located at the subject position at the previous time, checks whether the extracted shapes to which no identifier has been assigned overlap with the extracted shape at the previous time, and if there is an overlap, assigns the identifier of the extracted shape at the previous time, and assigns the identifier of the extracted shape that is closest within a predetermined range to the extracted shapes to which no identifier has still been assigned. (Configuration 14) The image processing device according to any one of configurations 1 to 13, characterized in that, when the extracted shape overlaps with two or more subject positions at a previous time, the tracking means does not assign an identifier to the extracted shape and does not update position information of all of the subject positions. (Configuration 15) 15. The image processing device according to any one of configurations 1 to 14, further comprising an audio signal generating means for generating an audio signal based on a playback position of a virtual sound source and a position of a virtual viewpoint, wherein the audio signal generating means updates the virtual playback position of the sound source to be added to the virtual viewpoint image based on the subject position. (Configuration 16) 16. The image processing device according to any one of configurations 1 to 15, wherein the viewpoint specifying means updates at least one of a direction and a position of a virtual viewpoint based on the subject position. (Method 1) An imaging step of imaging a subject from a plurality of directions; an estimation step of estimating a three-dimensional shape of the subject using the photographed image acquired in the imaging step; a viewpoint designation step of outputting virtual viewpoint information; an image generating step of generating a virtual viewpoint image based on the captured image, the three-dimensional shape, and the virtual viewpoint information; a shape extraction step of cutting out a part of the three-dimensional shape and outputting it as an extracted shape; an identification setting step of setting an identifier for the extracted shape; a tracking step of tracking the position of the extracted shape; a subject position calculation step of determining a representative position of the extracted shape to which the same identifier is assigned based on the tracking result of the tracking step, and outputting the representative position as the subject position; A control method comprising the steps of: (Program 1) An imaging step of imaging a subject from a plurality of directions; an estimation step of estimating a three-dimensional shape of the subject using the photographed image acquired in the imaging step; a viewpoint designation step of outputting virtual viewpoint information; an image generating step of generating a virtual viewpoint image based on the captured image, the three-dimensional shape, and the virtual viewpoint information; a shape extraction step of cutting out a part of the three-dimensional shape and outputting it as an extracted shape; an identification setting step of setting an identifier for the extracted shape; a tracking step of tracking the position of the extracted shape; a subject position calculation step of determining a representative position of the extracted shape to which the same identifier is assigned based on the tracking result of the tracking step, and outputting the representative position as the subject position; A program for causing a computer to execute the following.

Claims

1. An imaging means for imaging a subject from a plurality of directions; an estimation means for estimating a three-dimensional shape of the subject using the captured image acquired by the imaging means; A viewpoint specifying means for outputting virtual viewpoint information; an image generating means for generating a virtual viewpoint image based on the captured image, the three-dimensional shape, and the virtual viewpoint information; A shape extraction means for cutting out a part of the three-dimensional shape and outputting it as an extracted shape; an identification setting means for setting an identifier to the extracted shape; a tracking means for tracking the position of the extracted shape; a subject position calculation means for calculating a representative position of the extracted shape to which the same identifier is assigned based on the tracking result of the tracking means, and outputting the representative position as the subject position; An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein said identification setting means sets an initial state identifier for said extracted shape.

3. 3. The image processing apparatus according to claim 1, wherein said identification setting means sets an identifier for each extracted shape based on a distance between the extracted shapes.

4. 3. The image processing device according to claim 1, wherein the identification setting means determines whether each extracted shape is contained within a circumscribed rectangular parallelepiped of any of the three-dimensional shapes, and sets the same identifier when the extracted shapes are contained within the same circumscribed rectangular parallelepiped.

5. 3. The image processing apparatus according to claim 1, wherein the identification setting means sets an area to which a predetermined identifier is to be assigned, and sets the predetermined identifier set for the area for an extracted shape included in the area.

6. 3. The image processing apparatus according to claim 1, wherein said identification setting means reassigns an identifier to the assigned identifier in response to an instruction from a user.

7. 2. The image processing device according to claim 1, wherein the shape extraction means converts the extracted shape into a two-dimensional image by parallel projection onto a two-dimensional plane corresponding to a floor surface, cuts out each independent area from the two-dimensional image, and outputs a circumscribing rectangle for each area on the two-dimensional image as the extracted shape.

8. 2. The image processing apparatus according to claim 1, wherein said shape extracting means cuts out a shape from a bottom surface of a circumscribed rectangular parallelepiped of each three-dimensional shape to a predetermined height.

9. 2. The image processing apparatus according to claim 1, wherein said shape extracting means extracts the shape of a portion of the photographed area from a floor surface to a predetermined height.

10. 2. The image processing device according to claim 1, wherein the shape extraction means extracts the shape from a top surface of a structure installed in a shooting area to a predetermined height when the three-dimensional shape is located on the structure.

11. 3. The image processing apparatus according to claim 2, wherein said tracing means determines an identifier of said extracted shape at each time point based on said extracted shape at said time point and the position and identifier of the extracted shape at the immediately preceding time point.

12. 3. The image processing apparatus according to claim 2, wherein said tracking means determines an identifier of said extracted shape at each time point based on said extracted shape at that time point and said subject position and its identifier at the immediately preceding time point.

13. The image processing device described in claim 2, characterized in that the tracking means assigns an identifier of the subject position to the extracted shape located at the subject position at the previous time, checks whether the extracted shapes to which no identifier has been assigned overlap with the extracted shape at the previous time, and if there is an overlap, assigns the identifier of the extracted shape at the previous time, and for extracted shapes to which no identifier has still been assigned, assigns the identifier of the extracted shape that is closest within a specified range.

14. The image processing device according to claim 12, characterized in that, when the extracted shape overlaps with two or more subject positions at the previous time, the tracking means does not assign an identifier to the extracted shape and does not update the position information of all of the subject positions.

15. 2. The image processing device according to claim 1, further comprising an audio signal generating means for generating an audio signal based on a playback position of a virtual sound source and a virtual viewpoint position, wherein the audio signal generating means updates the virtual playback position of the sound source to be added to the virtual viewpoint image based on the subject position.

16. 2. The image processing apparatus according to claim 1, wherein said viewpoint specifying means updates at least one of the direction and the position of a virtual viewpoint based on the subject position.

17. An imaging step of imaging a subject from a plurality of directions; an estimation step of estimating a three-dimensional shape of the subject using the photographed image acquired in the imaging step; a viewpoint designation step of outputting virtual viewpoint information; an image generating step of generating a virtual viewpoint image based on the captured image, the three-dimensional shape, and the virtual viewpoint information; a shape extraction step of cutting out a part of the three-dimensional shape and outputting it as an extracted shape; an identification setting step of setting an identifier for the extracted shape; a tracking step of tracking the position of the extracted shape; a subject position calculation step of determining a representative position of the extracted shape to which the same identifier is assigned based on the tracking result of the tracking step, and outputting the representative position as the subject position; A control method comprising the steps of:

18. An imaging step of imaging a subject from a plurality of directions; an estimation step of estimating a three-dimensional shape of the subject using the photographed image acquired in the imaging step; a viewpoint designation step of outputting virtual viewpoint information; an image generating step of generating a virtual viewpoint image based on the captured image, the three-dimensional shape, and the virtual viewpoint information; a shape extraction step of cutting out a part of the three-dimensional shape and outputting it as an extracted shape; an identification setting step of setting an identifier for the extracted shape; a tracking step of tracking the position of the extracted shape; a subject position calculation step of determining a representative position of the extracted shape to which the same identifier is assigned based on the tracking result of the tracking step, and outputting the representative position as the subject position; A program for causing a computer to execute the following.

Images