Image processing device and its control method, program

Abstract

To provide a technology which supports switching operation to a video in which a desired subject is photographed.SOLUTION: On the basis of information on a three-dimensional shape corresponding to a subject which exists in a photographing area photographed by a plurality of photograph devices and is designated by a user, a video processing device identifies a photograph device including the subject in a photographing range and outputs display information including presentation of the photograph device.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a video processing apparatus, a control method thereof, and a program.

Background Art

[0002] In recent years, a virtual viewpoint video generation technology has been realized that synthesizes a video captured from an arbitrary viewpoint (hereinafter referred to as a virtual viewpoint video) from multi-viewpoint images synchronously captured by a plurality of cameras installed at different positions. For example, by using the virtual viewpoint video generation technology to shoot sports such as soccer and basketball, it is possible to view a video as if it were shot from inside the field during the game, which was impossible for a cameraman to shoot. Thus, according to the virtual viewpoint video generation technology, it is possible to generate a video control that gives a higher sense of presence to viewers and to generate a video with a higher degree of freedom.

[0003] On the other hand, a system for generating a virtual viewpoint video has a configuration in which a plurality of cameras are installed. Depending on the size of the shooting range and the required image quality, a large number of cameras may be installed. Therefore, in a system that generates a virtual viewpoint video, it is difficult for a user to check what kind of video each camera is outputting. For example, when switching from a virtual viewpoint video to a video of an actual camera (hereinafter referred to as an actual camera) that is actually shooting, the user needs to check the video from the actual camera. However, as described above, when the number of cameras is large, it is difficult for the user to check the video of the actual camera intended by the user.

[0004] Patent Document 1 discloses a technique for identifying an actual camera used for virtual viewpoint video generation from the positions of a plurality of actual cameras used for virtual viewpoint video generation and the position of the virtual viewpoint, and displaying the video of the identified actual camera.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] However, the technology disclosed in Patent Document 1 can only display the video from the actual camera used to generate the virtual viewpoint image. For example, in a situation where multiple performers are acting in a studio, if the operator wants to switch from the virtual viewpoint image to the video from the actual camera showing a specific performer, the operator needs to check which video from among the videos from multiple actual cameras shows the specific performer and then switch. In this case, it is necessary to switch to the video from the actual camera showing a specific subject, regardless of the generation of the virtual viewpoint image, but the technology in Patent Document 1 is not intended to support such switching to an actual camera.

[0007] This disclosure provides a technology that assists in switching to video footage of a desired subject. [Means for solving the problem]

[0008] An image processing apparatus according to one aspect of this disclosure has the following configuration. That is, the image processing apparatus is Acquisition means for acquiring a first image captured by a first shooting device for the user to specify a subject, and a plurality of second images captured by a plurality of second shooting devices for generating a virtual viewpoint image, and in the first image User-specified The aforementioned Corresponding to the subject The subject in the virtual space generated based on the plurality of second images Based on the information of the three-dimensional shape, Of the aforementioned plurality of second imaging devices, Include the subject in the shooting range. The third Imaging device to special A specific means to determine, The first video mentioned above, The identified The third Imaging device To display the second video captured by on one screen It has output means for outputting display information. [Effects of the Invention]

[0009] According to this disclosure, it becomes easier to switch to video footage of the desired subject. [Brief explanation of the drawing]

[0010] [Figure 1] Block diagram showing a configuration example of an image processing system according to the first embodiment. [Figure 2] Diagram showing an arrangement example of a photographing device according to the first embodiment. [Figure 3] Block diagram showing a hardware configuration example of an image processing device according to the first embodiment. [Figure 4] Diagram for explaining the functions of an image processing device according to the first embodiment. [Figure 5] Block diagram showing a functional configuration example of an image processing device according to the first embodiment. [Figure 6] Diagram for explaining the bounding box set for a subject. [Figure 7] Schematic diagram showing the shooting range of a shooting device. [Figure 8] Diagram for explaining a method for determining a shooting device that shoots a subject. [Figure 9] Diagram for explaining a method for specifying a bounding box corresponding to a specified subject. [Figure 10] Flowchart showing the processing of an image processing device according to the first embodiment. [Figure 11] Diagram representing a list of bounding boxes and shooting devices. [Figure 12] Diagram showing a display example according to the first embodiment. [[ID=X]] [[ID=Y]] [Figure 13] Block diagram showing a functional configuration example of an image processing device according to the second embodiment. [Figure 14] Schematic diagram showing the shooting ranges of a shooting device and a virtual viewpoint. [Figure 15] Flowchart showing the processing of an image processing device according to the second embodiment. [Figure 16] Diagram representing a list of bounding boxes, shooting devices, and virtual viewpoints. [Figure 17] Diagram showing a display example according to the second embodiment.

Embodiments for Carrying Out the Invention

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments are not limiting to this disclosure. While the embodiments describe multiple features, not all of these features are essential to this disclosure, and the features may be combined in any way. Furthermore, in the attached drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] <First Embodiment> Figure 1 is a block diagram showing an example configuration of an image processing system according to the first embodiment. The image processing system 100 comprises N actual cameras (shooting devices 110), an image processing device 120, and a user terminal 130. The shooting device 110 includes a lens (not shown), a shooting sensor such as a CCD or CMOS, an image processing circuit, etc., and converts optical information from the subject to be photographed into an electrical signal, which is then transmitted to the image processing device 120 as image information in a predetermined format such as RAW. Figure 2 shows an example of the arrangement of the N shooting devices 110 in the image processing system 100. The shooting devices 110a to 110b are arranged to surround the shooting area 200, and they capture images of subjects within the shooting area 200, providing multi-view images for generating a virtual viewpoint image. The number and placement of the shooting devices 110 can be appropriately set according to the shooting area, the subject to be photographed, and the required image quality.

[0013] The video processing device 120 aggregates video from the camera 110, processes it appropriately, and generates and outputs a virtual viewpoint video. The video processing device 120 can also select video captured by the camera 110, convert it into data in an appropriate format, and output it. Figure 3 shows an example of the hardware configuration of the video processing device 120. The video processing device 120 includes a CPU 121, main memory 122, auxiliary memory 123, external I / F 124, and GPU 125, which are interconnected via an internal bus 126 for communication.

[0014] The CPU 121 is the central processing unit and controls the video processing unit 120, performing various calculations, video processing, and data input / output. The main memory 122 functions as the work area of ​​the CPU 121 and as a temporary storage area for data. The main memory 122 is implemented using a storage medium such as Dynamic Random Access Memory (DRAM) or Static Random Access Memory (SRAM). The auxiliary storage device 123 stores information such as various programs, various setting information, various image data, camera parameters, three-dimensional shape information, and two-dimensional maps. The auxiliary storage device 123 can be implemented using a storage medium such as various Read Only Memory (ROM), non-volatile memory such as flash memory, hard disk drives (HDDs), solid-state drives (SSDs), or tape media. The auxiliary storage device 123 may be configured by combining multiple storage media to increase capacity and speed, for example, by combining multiple storage media into one logically using RAID.

[0015] External I / F124 includes a communication interface used for communication with external devices such as cameras, PCs, and other equipment. External I / F124 also includes a general-purpose input / output interface for connecting to external devices that accept various user operations, such as joysticks, keyboards, and mice, and for connecting external storage for file input / output. External I / F124 consists of interfaces with physical cable connection terminals, such as InfiniBand, Ethernet, and USB, or wireless interfaces such as Wi-Fi and Bluetooth. Note that USB stands for Universal Serial Bus. In the video processing system 100, video information output from N shooting devices 110 is aggregated by a switching hub (not shown) and input to External I / F124.

[0016] The GPU125 is a processing unit for high-speed execution of video processing calculations, and has functions such as rendering images from a virtual viewpoint and generating shape data and color information necessary for virtual viewpoint video. The GPU125 may include a separate main memory separate from the main memory 122. In addition, the GPU125 may perform some or all of the various calculations performed by the CPU121 other than those described above. The internal bus 126 is configured by a bus such as PCI Express, and each part of the hardware block can communicate bidirectionally with each other. In addition, any type of internal bus 126 that enables bidirectional communication between each part of the hardware block may be used.

[0017] The user terminal 130 is equipped with a display device such as an LCD monitor (not shown), and input devices such as a keyboard, mouse, and joystick, and is connected to the video processing device 120 via an external I / F 124. The user terminal 130 displays the video output from the video processing device 120 on the display device and converts the input from the user's input device into an appropriate signal and outputs it to the video processing device 120. The hardware configuration of the user terminal 130 can be, for example, the same as that of the video processing device 120 (Figure 3). Alternatively, the video processing device 120 may also have the functions of the user terminal 130. For example, the video processing device 120 and the user terminal 130 may be implemented in a single device.

[0018] The configuration of the image processing system of this embodiment has been described above with reference to Figures 2 and 3. However, this configuration is just one example and can be appropriately modified and configured depending on the subject being filmed and the function.

[0019] Next, with reference to Figures 4 to 11, a method for identifying and displaying the video from a shooting device containing a specific subject from among multiple shooting devices, according to the first embodiment, will be described.

[0020] First, an overview of the functions realized in this embodiment will be described with reference to Figure 4. In this embodiment, when it is desired to switch the streamed video from a state where virtual viewpoint video is being streamed to a state where video from a shooting device showing a specific subject is to be streamed, a function is realized to assist the operator during video switching by displaying the video from the shooting device that is shooting the specific subject as a candidate video.

[0021] In Figure 4, subjects 410 to 412 are present in the shooting area 200. For example, suppose we want to switch the streamed video from a state where virtual viewpoint video 402 is being shot and streamed from a virtual viewpoint 401 to the video of a shooting device that is showing subject 410. When the operator switching the video selects subject 410 as the subject to be focused on (hereinafter referred to as the subject to focus on), the video of the shooting device that is shooting subject 410 is extracted from the videos of multiple shooting devices. Here, suppose shooting devices 110a, 110b, and 110c are extracted from N shooting devices 110. In this case, the video 421 shot by shooting device 110a, the video 422 shot by shooting device 110b, and the video 423 shot by shooting device 110c are displayed as candidate videos. In this embodiment, candidate videos are extracted and displayed from a large number of videos, making it possible to select the intended video from a smaller number of videos. Therefore, compared to operations that require selecting the desired image from a large number of images output from multiple installed shooting devices, the burden on the operator in reviewing the images can be significantly reduced.

[0022] Next, the functional configuration of the video processing device 120 according to the first embodiment will be described. Figure 5 is a block diagram showing an example of the functional configuration of the video processing device 120. Each functional block shown in Figure 5 is realized, for example, by the CPU 121 (Figure 3) loading a computer program stored in the auxiliary storage device 123 into the main storage device 122 and executing it. Note that some or all of the functional blocks may be realized by dedicated hardware, or they may be realized through the cooperation of a computer and hardware.

[0023] The video receiving unit 500 receives video from the shooting device 110 via a LAN cable or the like, converts it to an appropriate data format, and sends it to the separation unit 501 and the video selection unit 508. The separation unit 501 separates the area of ​​the object from the video input from the video receiving unit 500 by cutting it out to an appropriate size, and generates a foreground image. In this embodiment, the separation unit 501 separates the object using, for example, the background subtraction method described later, and generates a foreground image. The separation unit 501 sends the generated foreground image to the shape generation unit 502.

[0024] In this embodiment, an object refers to a dynamic object (a moving body) that is in motion (its absolute position and shape may change) when photographed from the same direction over time. For example, in sports competitions, people such as players and referees on the field where the competition takes place, or the ball if the competition is a ball game, would be considered objects. The background subtraction method is a technique that uses an arbitrary image obtained from the camera as a reference image, compares the reference image with one or more images that are temporally close to the reference image, and extracts the region with change as the foreground image. In the background subtraction method, for example, the pixel values ​​of each pixel in the image are compared, and a set of pixels with small differences (pixels that are not moving) is extracted as the background image. Then, by subtracting the region of the generated background image from the reference image, the region with change is extracted, and the foreground image is generated.

[0025] The shape generation unit 502 generates three-dimensional shape information based on the foreground image input from the separation unit 501 using three-dimensional shape reconstruction methods such as the viewing volume cross-eyed method and Multi-View-Stereo (MVS). The coordinate information of the positions of the N imaging devices 110 and the field of view information necessary for reconstructing the three-dimensional shape of the subject are pre-stored in, for example, the auxiliary storage device 123. Furthermore, the shape generation unit 502 defines a rectangular prism that encloses (for example, circumscribing) the three-dimensional shape represented by the generated three-dimensional shape information. Hereinafter, this rectangular prism will be referred to as the bounding box. Figure 6 shows the relationship between the subject and the bounding box. In the example in Figure 6, a bounding box 610 is defined for the subject 410 within the imaging area 200. The bounding box 610 is defined as a rectangular prism with vertices at points a, b, c, d, e, f, g, and h. Furthermore, information regarding the three-dimensional shape or bounding box 610 of the subject is also acquired in terms of its position in three-dimensional space.

[0026] The virtual viewpoint video generation unit 503 receives information on the three-dimensional shape generated by the shape generation unit 502 and information on the position and direction of the virtual viewpoint input from the user terminal 130 via the input unit 504 described later. Based on this received information, the virtual viewpoint video generation unit 503 calculates how the three-dimensional shape appears from the position of the virtual viewpoint, colors it using the image of the corresponding shooting device, generates a virtual viewpoint video, and outputs it to the video distribution unit 509.

[0027] The input unit 504 receives signals input from the user terminal 130 via the external I / F 124 and transmits them to the virtual viewpoint image generation unit 503. The input unit 504 also inputs coordinates in the displayed image specified by the user to identify the object of focus. The input unit 504 also inputs viewpoint information indicating the position and orientation of the virtual viewpoint for generating the virtual viewpoint image. The viewpoint information includes, for example, parameters representing the three-dimensional position of the virtual viewpoint and parameters representing the orientation of the virtual viewpoint in the pan, tilt, and roll directions. The content of the viewpoint information is not limited to the above and may include any information indicating the position and orientation of the virtual viewpoint.

[0028] The shooting range determination unit 505 determines that an object located within the shooting range 700 of the shooting device 110a, as shown in Figure 7, is an object being photographed by the shooting device 110a. The shooting range 700 of the shooting device 110a is defined as a three-dimensional figure (for example, a truncated square pyramid) with vertices o, p, q, r, s, t, u, and v. An object located within the shooting range 700 is determined to be being photographed by the shooting device 110a. The shooting range 700 is a range that can be photographed calculated based on the position, orientation, focal length, F-number, and other shooting parameters of the shooting device 110a, and the three-dimensional coordinate information representing the shooting range 700 is stored in advance in, for example, the auxiliary storage device 123. Furthermore, such shooting ranges are pre-set for each of the multiple shooting devices 110 according to their respective shooting parameters and stored in the auxiliary storage device 123.

[0029] Furthermore, the shooting range determination unit 505 determines whether each of the multiple shooting devices is shooting the bounding box acquired by the shape generation unit 502, and extracts the shooting device that is shooting the bounding box. This determination method will be explained using Figure 8. Here, the shooting device 110 that is the subject of the determination is the shooting device that is shooting each surface of the bounding box other than the bottom surface. As mentioned above, the position of the bounding box (the three-dimensional shape of the subject) in three-dimensional space is acquired by the shape generation unit 502. If part or all of the surface abfe is inside the shooting range 700, as in the shooting device 110a in Figure 8, the shooting range determination unit 505 determines that the shooting device 110a is the shooting device that is shooting the bounding box 610. The shooting range determination unit 505 performs this determination for each surface of the bounding box 610 that is the subject of the determination, and identifies the shooting device that is shooting the bounding box 610. The shooting range determination unit 505 performs the above process for each bounding box present in the shooting area 200 and obtains a determination result for the bounding boxes present in the shooting area 200 at a given time and the shooting device that is photographing them. The shooting range determination unit 505 generates a list (described later in Figure 11) that associates the bounding boxes with the shooting devices that are photographing them at each time point in time, according to this determination result.

[0030] The subject identification unit 506 identifies the bounding box corresponding to the subject of focus specified by the user and notifies the shooting device selection unit 507. The subject of focus is specified, for example, by the user specifying the position of the subject they wish to focus on in the video display for selecting a subject. The coordinates specified by the user on the video (coordinates for identifying the subject of focus) are provided from the input unit 504 to the subject identification unit 506. The subject identification unit 506 identifies the bounding box of the subject of focus from these coordinates.

[0031] Here, using Figure 9, we will explain how to specify a subject of focus and how to identify a bounding box corresponding to that subject of focus. As shown in Figure 9, a camera 900 is installed for subject confirmation, and the captured image 910 from the camera 900 is distributed to the user terminal 130 via a cable (not shown). Alternatively, the video from the camera 900 may be distributed to the user terminal 130 via the video processing device 120, similar to the camera 110. The user can input coordinates corresponding to the subject of focus while confirming the captured image 910. The subject identification unit 506 projects the three-dimensional shape generated by the shape generation unit 502 onto the captured image 910 to generate a subject area image 920. Information necessary for projection, such as the coordinates of the shooting position of the camera 900 and field of view information, is pre-stored in, for example, the auxiliary storage device 123. The coordinates in the subject area image 920 correspond to the coordinates of the image captured by the camera 900, and are further linked to the information of the projected three-dimensional shape. For example, in Figure 9, subjects 410, 411, and 412 in the shooting area 200 correspond to areas 930, 931, and 932 in the subject area image 920, respectively.

[0032] When a user specifies an arbitrary position in the captured image 910, those coordinates are notified to the subject identification unit 506 via the input unit 504. The subject identification unit 506 determines which subject the notified coordinates belong to by referring to the subject area image 920. For example, when coordinates P(xp,yp) are input to specify subject 410, the subject identification unit 506 determines the area of ​​the subject containing coordinates P(xp,yp) from the subject area image 920. In the example in Figure 9, the subject identification unit 506 identifies subject 410, which corresponds to area 930, because area 930 contains coordinates P(xp,yp). The subject identification unit 506 then identifies the bounding box 610 generated for the identified subject 410 as the bounding box of the subject being watched. Note that the method for identifying the bounding box of a subject specified by the user is not limited to the above. For example, a GPS could be attached to each subject to maintain information associating the position and identifier of each subject, and the user could specify the identifier of the desired subject. This allows the bounding box to be identified based on the GPS location of the designated subject. In this embodiment, the image from a single camera 900 was used as the image for specifying the subject, but the user may obtain the image for specifying the subject by having multiple cameras share the shooting area. Also, in this embodiment, the image captured by the physical camera 900 was used as the image for specifying the subject to be focused on, but a virtual viewpoint image from a virtual viewpoint, such as an overhead view of the shooting area 200, may also be used.

[0033] The shooting device selection unit 507 selects the shooting device used to photograph the bounding box identified by the subject identification unit 506 by referring to a list generated by the shooting range determination unit 505, which will be described later in Figure 11. In this way, the shooting device selected by the shooting device selection unit 507 from the multiple shooting devices 110 is the shooting device that is photographing the subject being watched. The video selection unit 508 selects the video of the shooting device selected by the shooting device selection unit 507 from the video received from the video receiving unit 500 and sends it to the video distribution unit 509. The video distribution unit 509 converts the virtual viewpoint video generated by the virtual viewpoint video generation unit 503 and the video sent from the video selection unit 508 into a predetermined display format and output it.

[0034] Next, the operation of the image processing device 120 according to the first embodiment will be described. Figure 10 is a flowchart showing the operation of the image processing device 120 according to the first embodiment. In S110, the separation unit 501 acquires a foreground image corresponding to a subject present in the shooting area 200. Next, in S111, the shape generation unit 502 estimates the three-dimensional shape of the subject based on the foreground image acquired by the separation unit 501 and generates information on the three-dimensional shape of the subject and a corresponding bounding box.

[0035] In S112, the shooting range determination unit 505 determines, for each bounding box generated in S111, which of the multiple shooting devices 110 is photographing the bounding box. Using the determination result, the shooting range determination unit 505 generates a list that associates the bounding box with the shooting device photographing it at each time point. Figure 11 shows an example of the list (list 1100) generated by the shooting range determination unit 505. In list 1100, the correspondence between the bounding box and the number of the shooting device photographing it at each time point is listed. Note that the data format of list 1100 is not limited to that shown. List 1100 only needs to include information that identifies the bounding box and the shooting device photographing that bounding box in time series.

[0036] In S113, the subject identification unit 506 determines whether a subject to be focused on has been specified via the input unit 504. If no subject to be focused on has been specified (NO in S113), the process returns to S110. If it is determined that a subject to be focused on has been specified (YES in S113), the process proceeds to S114. In S114, the subject identification unit 506 identifies the bounding box corresponding to the specified subject to be focused on. The shooting device selection unit 507 then selects the shooting device that is capturing the bounding box identified by the subject identification unit 506 by referring to the list 1100 output in S112. In S115, the video selection unit 508 selects the video obtained from the shooting device selected by the shooting device selection unit 507. If the video selection unit 508 selects multiple videos, it may select the videos in order of priority according to user settings.

[0037] In S116, the video distribution unit 509 outputs display information, including the video obtained by converting the video selected by the video selection unit 508 into an appropriate output format, to the user terminal 130, and terminates this process. The display information provides the video to be presented to the user when switching the distributed video (switching the shooting device) on the user terminal 130. An example of the video displayed by the user terminal 130 according to the display information output in S116 is shown in Figure 12. Figure 12 is an image displayed on the display device of the user terminal 130 that has received the output display information. Here, the subject 410 is designated as the subject to be watched, and the video from the shooting device (the shooting device selected by the shooting device selection unit 507) that is shooting the subject 410 is displayed. The display screen 1200 shows the video 1201 for subject confirmation (video obtained from the shooting device 900) and the videos 1210, 1211, and 1212 of the shooting device that is shooting the subject 410 selected by the video selection unit 508. The display conditions for these videos 1210-1212, such as their position and size, can be pre-set by the user. For example, if the size of the subject in the video is prioritized, the larger the subject, the higher the ranking of the video, and the videos are arranged and displayed according to the assigned ranking. Figure 12 shows a state where the size of subject 410 in the video is prioritized, and the videos are arranged from the center in order of the size of the subject 411. Specifically, the video distribution unit 509 generates and distributes display information (video) such that the captured videos are arranged in order of the proportion of the bounding box area corresponding to the specified subject to the entire video of the shooting device. In this way, when display conditions are set in advance by user operation, the video distribution unit 509 sets the display order and size, etc., according to the set display conditions for the captured videos selected by the video selection unit 508, generates and outputs display information (video). A predetermined number of videos may be displayed in order of highest ranking. Furthermore, the user may be able to set this predetermined number.

[0038] Of course, the display conditions are not limited to those described above, and various conditions can be applied. For example, if the subject in the video is an object with a front, such as a person, a higher ranking may be set for videos where the subject is facing forward. The closer the subject is to the center of the image, the higher the ranking set for that video. Also, if similar videos are determined to be the same, instead of displaying all the videos selected by the video selection unit 508, the shooting conditions may be judged based on brightness, contrast, etc., and the video with the best shooting conditions may be displayed as a representative. For example, the size and position of the subject in the video may be quantified and calculated as numerical values, and videos where the difference in the calculated values ​​is less than or equal to a predetermined value may be classified as similar videos and into groups of similar videos. Then, from each group of videos, the video that is judged to have good shooting conditions based on brightness, contrast, etc., is selected as a representative and displayed. The above example shows the display of videos to indicate the shooting device selected by the shooting device selection unit 507, but it is not limited to this. For example, the display of identification information may indicate the shooting device selected by the shooting device selection unit 507. In this case, the ranking set for the videos is also the ranking of the corresponding shooting devices. Therefore, the arrangement of the displayed identification information of the shooting device may be determined according to the above order. Note that these exemplified display conditions may be set according to the usage conditions of the video processing system 100, etc.

[0039] When the user selects a desired video from the candidate videos (video 1210, video 1211, video 1212), the video processing device 120 is instructed to distribute the video from the camera that is capturing the selected video. This instruction is notified to the video selection unit 508 via the input unit 504, and the video selection unit 508 selects the video from the specified camera. The video distribution unit 509 stops distributing the video that was being distributed up to that point and starts distributing the video selected by the video selection unit 508. In this way, the switching of the video to be distributed is performed.

[0040] As described above, according to the first embodiment, the video from a camera that is capturing a specific subject is displayed as a candidate video. The user can select the camera that is capturing a specific subject by selecting the desired candidate video from the displayed candidate videos. This makes it easier to switch between streamed videos. In the above, the video from the selected camera (candidate video) was displayed, but information identifying the selected camera (e.g., camera number) may also be displayed. The user can select the desired camera from the displayed identification information. In this case, the user cannot select while viewing the video, but the number of video devices to switch to can be narrowed down, thus improving operability.

[0041] <Second Embodiment> Referring to Figures 13 to 17, a method for identifying and displaying video from a shooting device or virtual viewpoint video containing a specific subject, according to the second embodiment, will be described. In the first embodiment, video footage containing a specified subject was identified from video footage captured by a shooting device, which is a real camera. In the second embodiment, video footage containing a specified subject is identified from video footage captured by the shooting device and virtual video from a virtual viewpoint. That is, in the second embodiment, a configuration will be described in which, in addition to the video from the shooting device, a virtual viewpoint video different from the distributed and displayed virtual viewpoint video is used as a candidate for the switching video.

[0042] The functional configuration of the video processing device 120 according to the second embodiment will now be described. Figure 13 is a block diagram showing an example of the functional configuration of the video processing device 120 according to the second embodiment. Each functional block can be realized by the CPU 121 loading a computer program stored in the auxiliary storage device 123 into the main storage device 122 and executing it, similar to the first embodiment (Figure 3). In the functional blocks of the video processing device 120 of the second embodiment, the virtual viewpoint video generation unit 503a is connected to the shooting range determination unit 505a and the video selection unit 508a. Functional blocks similar to the functional blocks of the video processing device 120 of the first embodiment (Figure 5) are given the same reference numbers. The functions of the virtual viewpoint video generation unit 503a, the shooting range determination unit 505a, and the video selection unit 508a will be mainly described below.

[0043] The virtual viewpoint video generation unit 503a receives the specification of the position and direction of one or more virtual viewpoints and generates one or more virtual viewpoint videos according to the received virtual viewpoint information. The video selection unit 508a selects the video to be distributed from the videos of multiple shooting devices received by the video receiving unit 500 and the virtual viewpoint videos generated by the virtual viewpoint video generation unit 503a. The virtual viewpoint video generation unit 503a also provides information on the position and shooting angle of the virtual viewpoint to the shooting range determination unit 505a.

[0044] Similar to the first embodiment, the shooting range determination unit 505a pre-calculates and stores information on the shooting range of each of the multiple shooting devices 110 in memory. The shooting range determination unit 505a also calculates the shooting range of the virtual viewpoint from the position and shooting angle information of the virtual viewpoint from the virtual viewpoint image generation unit 503a and stores it in memory in addition to the shooting ranges of each of the multiple shooting devices 110. Figure 14 shows an example of the shooting range calculated by the shooting range determination unit 505a in the second embodiment. In addition to the shooting range information of the shooting devices 110 calculated in the first embodiment, the shooting range of the virtual viewpoint 1401 (the range of the virtual viewpoint image generated by the virtual viewpoint 1401) is calculated. Here, the shooting range 1400 defined by a three-dimensional figure (frustum of a square pyramid) with vertices o2, p2, q2, r2, s2, t2, u2, and v2 is calculated as the shooting range of the virtual viewpoint 1401. Furthermore, if there are multiple virtual viewpoints, the shooting range determination unit 505a calculates the shooting range for all virtual viewpoints and stores it in memory. In addition, the shooting range determination unit 505a determines whether or not the bounding box generated by the shape generation unit 502 is being photographed for all shooting devices and virtual viewpoints. The shooting range determination unit 505a generates a list 1600, as shown in Figure 16, which associates the bounding box with the shooting device and virtual viewpoint that is photographing the bounding box.

[0045] The shooting device selection unit 507a, the video selection unit 508a, and the video distribution unit 509 perform the same processing as in the first embodiment, but they treat the pre-prepared virtual viewpoint video in the same way as the video from the shooting device.

[0046] Next, the operation of the video processing device 120 in the second embodiment will be described. Figure 15 is a flowchart showing the operation of the video processing device 120 according to the second embodiment.

[0047] In S210, the separation unit 501 generates a foreground image of a subject present in the shooting area 200. Next, in S211, the shape generation unit 502 estimates the three-dimensional shape based on the foreground image generated by the separation unit 501 and generates information on the three-dimensional shape and a corresponding bounding box. In S212, the virtual viewpoint image generation unit 503a generates a virtual viewpoint image from the virtual viewpoint input to the input unit 504. The virtual viewpoint image generation unit 503a outputs virtual viewpoint information, including the position of the virtual viewpoint and the shooting angle of view, to the shooting range determination unit 505a.

[0048] In S213, the shooting range determination unit 505a determines the imaging device and virtual viewpoint that are imaging the bounding box generated in S211. If there are multiple bounding boxes in the shooting area 200, the shooting range determination unit 505a determines the imaging device and virtual viewpoint that are imaging all of those bounding boxes. Based on the results of these determinations, the shooting range determination unit 505a generates a list of bounding boxes and the imaging devices that are imaging them at each time point. An example of this list (list 1600) is shown in Figure 16. List 1600 lists the bounding boxes and the imaging devices and virtual viewpoints that are imaging them at each time point. Note that the data format of the list is not limited to that shown in Figure 16. List 1600 only needs to include information that identifies the bounding boxes and the imaging devices and virtual viewpoints that are imaging those bounding boxes in at least one time series, and its data format is not restricted.

[0049] Next, in S214, the subject identification unit 506 determines whether a subject of focus has been specified via the input unit 504. If it is determined that no subject of focus has been specified (NO in S214), the process returns to S211. On the other hand, if it is determined that a subject of focus has been specified (EYS in S214), the process proceeds to S215. In S215, the subject identification unit 506 identifies the bounding box corresponding to the specified subject of focus. In S216, the shooting device selection unit 507a refers to the list 1600 generated in S213 and selects the shooting device and virtual viewpoint that are shooting the bounding box identified in S215. Then, the video selection unit 508a selects the video of the shooting device and virtual viewpoint selected by the shooting device selection unit 507a as the video to be output. The process when display conditions are set by the user is the same as in the first embodiment.

[0050] In S217, the video distribution unit 509 generates display information (presentation video) by converting the selected video into an appropriate output format and outputs it to the user terminal 130. Figure 17 shows an example of the output video from the user terminal 130 that has received the display information according to the second embodiment. Figure 17 is an example of an image displayed by the display device of the user terminal 130 that has received the output display information (video). Here, the subject 410 is designated as the subject to be watched, and the video of the shooting device and virtual viewpoint that are shooting the designated subject to be watched are displayed. Specifically, the display screen 1200 shows video 1201 (video obtained from the shooting device 900), the video 1710 of the virtual viewpoint selected by the video selection unit 508, and the videos 1711 and 1712 of the shooting device. Videos 1710, 1711 and 1712 are candidate videos for switching to show the subject 410, which is the subject to be watched. As with the first embodiment, the display position and size of these images 1710-1711 shall conform to the display conditions set in advance by the user. Furthermore, as shown in Figure 17, the display information (images) may be configured so that the user can distinguish whether it is an image from a virtual viewpoint or an image from the shooting device.

[0051] As described above, according to the second embodiment, an image from a shooting device containing a specific subject or a virtual viewpoint image from a virtual viewpoint is identified and displayed as a switching candidate. The user can select the desired image from the images displayed as candidates, thus improving the operability when switching between images.

[0052] (Other examples) This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of Symbols]

[0053] 100: Video processing system, 110: Shooting device, 120: Video processing device, 130: User terminal, 500: Image receiving unit, 501: Separation unit, 502: Shape generation unit, 503: Virtual viewpoint video generation unit, 504: Input unit, 505: Shooting range determination unit, 506: Subject identification unit, 507: Shooting device selection unit, 508: Video selection unit, 509: Video distribution unit

Claims

1. An image processing apparatus, An acquisition means for acquiring a first image captured by a first shooting device for the user to specify a subject, and a plurality of second images captured by a plurality of second shooting devices for generating a virtual viewpoint image. Based on the information of the three-dimensional shape of the subject in a virtual space generated based on the plurality of second images corresponding to the subject specified by the user in the first image, a selection means for selecting a third shooting device among the plurality of second shooting devices that includes the subject in its shooting range, An image processing apparatus characterized by having an output means for outputting display information for displaying the first image and the second image captured by the identified third shooting device on a single screen.

2. The image processing apparatus according to claim 1, characterized in that the identifying means identifies an imaging device that includes at least a portion of the bounding box surrounding a subject represented by the three-dimensional shape information as the third imaging device.

3. The image processing apparatus according to claim 1 or 2, characterized in that the identifying means acquires the shooting range of each of the plurality of second shooting devices based on the position and orientation of each of the plurality of second shooting devices.

4. The image processing apparatus according to any one of claims 1 to 3, characterized in that the display information is information for further displaying information indicating the third shooting device identified by the identifying means on one screen.

5. The system further includes setting means for assigning a ranking to the identified third shooting device or the images of the identified third shooting device based on the images of the third shooting device identified by the identification means, The video processing apparatus according to any one of claims 1 to 4, characterized in that the display information is information that causes the second video to be arranged on the one screen based on the order set by the setting means.

6. The image processing apparatus according to claim 5, characterized in that the setting means sets a rank for the specified third shooting device based on at least one of the following conditions: setting a higher rank for subjects that appear in the image as they are larger; setting a higher rank for images in which subjects are facing forward; and setting a higher rank for images in which the position of subjects appears in the image is closer to the center of the image.

7. The third imaging device identified by the identification means is further classified into groups of imaging devices that have captured similar images by quantifying the size or position of the subject in the image, and a determination means is made to determine one imaging device for each group. The image processing apparatus according to any one of claims 1 to 6, characterized in that the display information includes the presentation of one imaging device determined for each group by the determination means.

8. The video processing apparatus according to any one of claims 1 to 7, further comprising a switching means for switching the video to be distributed to a video selected by the user from among the second videos displayed according to the display information.

9. The aforementioned identification means further identifies a virtual viewpoint from among the pre-set virtual viewpoints from which a virtual viewpoint image including the subject specified by the user is generated, The image processing apparatus according to any one of claims 1 to 7, characterized in that the display information is information that causes the display of the second image captured by the third shooting device identified by the identifying means and the virtual viewpoint image viewed from the virtual viewpoint.

10. The display information of the second image captured by the third shooting device and the virtual viewpoint image viewed from the virtual viewpoint is such that the user can distinguish whether the displayed image is the second image or the virtual viewpoint image, as described in Claim 9.

11. The video processing apparatus according to claim 9 or 10, further comprising switching means for switching the video to be distributed to the second video or the virtual viewpoint video selected by the user from among the second video captured by the third shooting device and the virtual viewpoint video viewed from the virtual viewpoint, as displayed according to the aforementioned display information.

12. A control method performed by an image processing device, An acquisition step of acquiring a first image captured by a first shooting device for the user to specify a subject, and a plurality of second images captured by a plurality of second shooting devices for generating a virtual viewpoint image, A selection step of identifying a third shooting device among the plurality of second shooting devices that includes the subject in its shooting range, based on information of the three-dimensional shape of the subject in a virtual space generated based on the plurality of second images corresponding to the subject specified by the user in the first image, A control method characterized by comprising: an output step of outputting display information for displaying the first image and the second image captured by the identified third shooting device on a single screen.

13. A program for causing a computer to function as one of the means of the video processing apparatus described in any one of claims 1 to 11.

Images