Wide-field-of-view video capture support device and program thereof, and wide-field-of-view video capture system

Abstract

To provide a wide visual field video shooting support device which can visually present an overlapping area and information that a subject only exists in one video of the overlapping area to a cameraman when shooting a wide visual field video with a plurality of cameras.SOLUTION: A wide visual field video shooting support device 1 comprises: wide-angle image correction means 10 which corrects a wide-angle image of a wide-angle camera 3 into the whole image of the whole visual field by plain-expanding the wide-angle image; whole image feature point extraction means 11 which extracts a feature point from the whole image; partial image feature point extraction means 12 which extracts the feature point from the partial image being the image of the camera 2 for each of the plurality of cameras 2; visualization area searching means 14 which makes the feature point of the whole image match the feature point for each partial image and searches for the overlapping area of the partial image and the subject area existing only in one of the overlapping area in the whole image as a visualization area; and area composition means 15 which combines the visualization area on the whole image to display it.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a wide-field video capture support device and a program therefor that support wide-field video capture using multiple cameras, and a wide-field video capture system. [Background technology]

[0002] In recent years, virtual reality (VR) services that use head-mounted displays (HMDs) to allow users to experience remote locations as if they were there have become popular. VR is a technology that gives users a sense of immersion by presenting images that correspond to the user's head movements from 360-degree images captured in advance to the user wearing the HMD, making them feel as if they are virtually in that location. Conventionally, a technology for capturing 360° images for VR has been disclosed in which two 180° images captured by two ultra-wide-angle lens (fisheye lens) cameras are stitched together using image processing (see Patent Document 1).

[0003] The reflected light from objects in a 180° field of view captured by this ultra-wide-angle lens camera is converted into an electric charge by a single image sensor and recorded. For example, even if an image sensor with approximately 33 million pixels for SVH (Super Hi-Vision) is used, the number of horizontal pixels is 7680. On the other hand, if the upper visual frequency limit for a person with 1.0 vision is 30 cpd (cycles per degree) (60 pixels / °), then within a 180° field of vision, unless there are 60 pixels x 180° = 10,800 pixels or more arranged on a semicircle, the person will be able to detect the pixel structure, and a single image sensor would not have enough pixels. In response to this, a technology has been disclosed for capturing high-resolution 180° video by stitching together partially overlapping video images captured by multiple cameras (see Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-117330 A [Patent Document 2] Special Publication No. 2018-518078 Summary of the Invention [Problem to be solved by the invention]

[0005] Even if an ultra-wide-angle lens camera is used as in the invention described in Patent Document 1, there is a problem in that the resolution is insufficient to capture a wide-angle image. Also, as in the invention described in Patent Document 2, it is possible to capture a wide-field image with increased resolution by stitching images captured by multiple cameras.

[0006] However, when stitching together footage shot with multiple cameras, if the subject is only visible in one camera and exists in the area to be stitched (overlapping area), the stitching is performed between footage that includes the subject and footage that does not include the subject, which can result in the stitching not being performed correctly and part of the subject disappearing. Furthermore, the stitching technique causes image degradation the larger the area to be stitched.

[0007] The present invention has been made in consideration of the problems with the conventional technology, and has an objective of providing a wide-field video shooting support device and a program therefor, as well as a wide-field video shooting system, that, when shooting wide-field video with multiple cameras, can visually indicate to the cameraman that the subject is present only in the overlapping area to be stitched and only in one of the images in that overlapping area. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the wide-field-of-view video capture support device of the present invention is a wide-field-of-view video capture support device that visualizes overlapping areas in a wide-field-of-view image captured by multiple cameras from images from each camera in which parts of the field of view are overlapped and images from a wide-angle camera that captures the entire field of view of the multiple cameras, and is configured to include a wide-angle image correction means, an overall image feature point extraction means, a partial image feature point extraction means, a visualization area search means, and an area synthesis means.

[0009] In this configuration, the wide-field-of-view video capture support device generates an overall image of the entire field of view by correcting the wide-angle images of each frame constituting the image of the wide-angle camera to a normal image that is expanded on a plane by the wide-angle image correction means. This makes it easier to compare the overall image with images captured by multiple cameras. Then, the wide-field-of-view video photography support device extracts feature points such as SIFT and SURF from the entire image using an entire image feature point extraction means. In addition, the wide-field video shooting support device extracts feature points from partial images, which are images for each frame constituting the video of the camera, for each of the multiple cameras by using a partial image feature point extraction means. Since the images captured by the cameras correspond to partial images of the entire image captured by the wide-angle camera, the feature points of the partial images and the feature points of the entire image have similar feature amounts.

[0010] The wide-field-of-view video photography support device then uses the visualization area search means to match the feature points of the overall image with the feature points of each partial image, and to identify overlapping areas of the partial images in the overall image, Partial image In the overlapping region, a subject region that exists only in one of the partial images is searched for as a visualization region. The wide-field image capture support device then uses the area synthesis means to synthesize a visualization area on the entire image. A figure showing are synthesized and displayed.

[0011] This allows the wide-field video shooting support device to visually show the cameraman where the overlapping area to be stitched exists on the entire image, and whether or not a subject exists in only one of the partial images in that overlapping area.

[0012] The wide-field-of-view video capture support device may further include a stitching means for stitching together areas where feature points of the partial images match to connect the partial images, thereby generating a wide-field-of-view video. Furthermore, the wide-field video photography support device can be operated by a wide-field video photography support program that causes a computer to function as each of the above-mentioned means.

[0013] In addition, in order to solve the above-mentioned problems, the wide-field video shooting system of the present invention is a wide-field video shooting system that captures wide-field video, and is configured to include multiple cameras, a wide-angle camera, a wide-field video shooting support device, and a camera operation device.

[0014] In such a configuration, the wide-field-of-view video capture system captures a portion of a desired field of view with each camera using a plurality of cameras arranged so that their fields of view partially overlap. Furthermore, the wide-field video capture system captures the entire field of view of multiple cameras using a wide-angle camera.

[0015] The wide-field video capture system then uses the wide-field video capture support device to display, as visualized areas, overlapping areas of partial images and subject areas that exist in only one of the overlapping areas from images captured by multiple cameras and images captured by a wide-angle camera, and generates a wide-field video. This allows the wide-field video capture system to visually show the cameraman where the overlapping area to be stitched exists on the entire image, and whether or not a subject exists in only one of the partial images in that overlapping area.

[0016] The wide-field video shooting system controls the orientation and zoom of the multiple cameras and the wide-angle camera through the operation of a cameraman using a camera operation device. This allows the cameraman to perform camera control such as narrowing the overlapping area, capturing the subject in both of the two partial images of the overlapping area, or removing the subject from the overlapping area. Effect of the Invention

[0017] The present invention provides the following excellent effects. According to the present invention, when generating wide-angle images using multiple cameras, it is possible to visually present to the cameraman overlapping areas where stitching is performed between camera images, and subject areas where the subject appears in only one of the camera images in the overlapping area. As a result, the present invention can adjust the size of the overlapping area to reduce image quality degradation during stitching when creating wide-angle images using multiple cameras, and can provide assistance to the cameraman with camerawork when operating the camera to avoid the presence of a subject that appears in only one of the camera images in the overlapping area. [Brief description of the drawings]

[0018] [Figure 1] 1 is a block diagram showing a configuration of a wide-field video shooting system including a wide-field video shooting support device according to an embodiment of the present invention. [Diagram 2] 1 is an explanatory diagram for explaining the relationship between images captured by a plurality of cameras and an overlapping area. [Diagram 3] 11 is an explanatory diagram for explaining an image in which an overlapping region and a subject region are combined on an entire image. FIG. [Figure 4] This is an explanatory diagram to explain camera work, where (a) shows a state in which a subject is captured by only one camera in an overlapping area captured by two cameras, and (b) shows a state in which, by zooming out, both cameras are capturing images of the subject in the overlapping area captured by both cameras. [Diagram 5]4 is a flowchart showing the operation of the wide-field video shooting support device according to the embodiment of the present invention. [Figure 6] FIG. 13 is a diagram showing a modified example of the camera arrangement, in which a plurality of cameras are arranged on a circumference. [Figure 7] FIG. 13 is an explanatory diagram for explaining camera work (zoom out) when cameras are arranged on a circumference. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] <Configuration of wide-field imaging system> First, with reference to FIG. 1, the overall configuration of a wide-field video capture system 100 according to an embodiment of the present invention will be described.

[0020] The wide-field video capture system 100 is a system that captures wide-field video using multiple cameras. The wide-field video shooting system 100 includes a wide-field video shooting support device 1, a plurality of cameras (video cameras) 2, a wide-angle camera 3, a camera operation device 4, a display device 5, and a recording device 6.

[0021] The wide-field-of-view video capture support device 1 visualizes overlapping areas in a wide-field-of-view video captured by multiple cameras 2 from images captured by each camera 2, in which portions of the field of view are overlapped and captured by the multiple cameras 2 for capturing wide-field-of-view video, and from images captured by a wide-angle camera 3, which captures the entire fields of view of the multiple cameras 2. The wide-field video shooting support device 1 supports the cameraman M's camerawork by displaying an image in which the overlapping area is visualized on the display device 5. In addition, the wide-field video shooting support device 1 also stitches together camera images captured by a plurality of cameras 2 to generate a wide-angle video image, and outputs the video image to a recording device 6. The configuration and operation of this wide-field-of-view video shooting support device 1 will be described later.

[0022] The camera 2 is a video camera that captures video images. For example, the camera 2 is a video camera for SVH (Super Hi-Vision). Here, multiple cameras 2 (2a, 2b, 2c) are arranged in the horizontal direction so that their fields of view partially overlap.

[0023] For example, the three cameras 2a, 2b, and 2c are arranged so that the image Ia captured by camera 2a and the image Ib captured by camera 2b overlap in an area D1, and the image Ib captured by camera 2b and the image Ic captured by camera 2c overlap in an area D2, as shown in Fig. 2. Note that the areas D1 and D2 may be the same width (equally spaced between cameras) or different widths (unequally spaced between cameras). Although three cameras 2 are shown in FIG. 1, the present invention is not limited to this, and the number of cameras may be two, four or more as long as it is sufficient to capture a desired wide field of view. Furthermore, the camera 2 pans, tilts, and zooms in response to instructions from the camera operation device 4. The camera 2 may be equipped with a movement mechanism and have a function of moving its position in response to instructions from the camera operation device 4. The cameras 2 (2a, 2b, 2c) output the captured images to the wide-field image capture support device 1.

[0024] The wide-angle camera 3 is a wide-field camera that captures the entire fields of view of the multiple cameras 2. For example, the wide-angle camera 3 is a camera equipped with a wide-angle lens (fisheye lens) with a field of view of 180° or more. Furthermore, the wide-angle camera 3 performs zooming in response to an instruction from the camera operation device 4. Note that the wide-angle camera 3 may be provided with a movement mechanism and have a function of moving its position in response to an instruction from the camera operation device 4. The resolution of this wide-angle camera 3 may be the same as the resolution of camera 2 or may be lower. The wide-angle camera 3 outputs the captured wide-angle image to the wide-angle image capture support device 1.

[0025] The camera operation device 4 controls the cameras 2 through the operation of the cameraman M. The camera operation device 4 controls the pan, tilt, and zoom of each of the cameras 2 (2a, 2b, 2c) via an operation panel (not shown). The camera operation device 4 may also control the movement of the cameras 2 so that they are positioned at regular intervals. When controlling the zoom of the camera 2, the camera operation device 4 also controls the zoom of the wide-angle camera 3 in conjunction with the camera operation device 4.

[0026] The display device 5 displays an image in which the overlapping region is visualized by the wide-field-of-view video shooting support device 1. The display device 5 is a general display, such as a liquid crystal display. The recording device 6 records the wide-angle image generated by the wide-angle image shooting support device 1. This recording device 6 records the wide-angle image on a general recording medium such as a hard disk.

[0027] <Configuration of wide-field video capture support device> Next, the configuration of a wide-field-of-view video shooting support device 1 according to an embodiment of the present invention will be described with reference to FIG.

[0028] The wide-field image capture support device 1 comprises a wide-angle image correction means 10, an overall image feature point extraction means 11, a partial image feature point extraction means 12, a stitching means 13, a visualization area search means 14, and an area synthesis means 15. It is preferable to provide a distortion correction means (not shown) for correcting the lens distortion of each of the camera 2 and the wide-angle camera 3, at a stage preceding the wide-angle image correction means 10 and at a stage preceding the partial image feature point extraction means 12. In this case, the distortion correction means may hold correction coefficients obtained by performing a calibration in advance, and may correct the lens distortion for each captured frame.

[0029] The wide-angle image correction means 10 generates an overall image of the entire field of view by correcting the wide-angle image for each frame constituting the image of the wide-angle camera 3 into a normal image obtained by expanding the image on a plane. Note that the method of converting the wide-angle image into a normal image can be a general method such as equirectangular projection. The wide-angle image correction means 10 outputs the corrected whole image to the whole image feature point extraction means 11 and the region synthesis means 15 .

[0030] The whole image feature point extraction means 11 extracts feature points from the whole image corrected by the wide-angle image correction means 10 . To extract these feature points, a common method such as Scale-Invariant Feature Transform (SIFT) or Speeded-Up Robust Features (SURF) may be used. The whole image feature point extraction means 11 outputs the positions and feature amounts (SIFT features, etc.) of the extracted feature points to the visualization area search means 14.

[0031] The partial image feature point extraction means 12 extracts feature points from partial images, which are images for each frame constituting the video of the cameras 2, for each of the multiple cameras 2. The frame captured by camera 2 is a partial image of the field of view of the subject. Here, the wide-field video shooting support device 1 includes a plurality of partial image feature point extraction means 12 (12a, 12b, 12c) corresponding to a plurality of cameras 2 (2a, 2b, 2c). Note that it is not necessary to physically include a plurality of partial image feature point extraction means 12, and one partial image feature point extraction means 12 may be used to extract feature points by switching between the plurality of cameras 2 in sequence. The partial image feature point extraction means 12 extracts feature points using the same method as the whole image feature point extraction means 11 .

[0032] The partial image feature point extraction means 12 outputs the positions and feature amounts (SIFT features, etc.) of the extracted feature points to the visualization region searching means 14 and the stitching means 13. Furthermore, the partial image feature point extraction means 12 outputs the partial image from which the feature points have been extracted to the stitching means 13.

[0033] The stitching means 13 synthesizes overlapping areas from the partial images and their feature points for each camera 2 input from the partial image feature point extraction means 12 . A general stitching method may be used for combining the overlapping regions in the stitching means 13. For example, the stitching means 13 overlaps the overlapping regions by alpha blending the range of feature points with similar feature amounts between the two partial images. Alpha blending is a method of multiplying one image and the other image by a predetermined transparency (α value) and calculating the average. In this way, the stitching means 13 overlaps and stitches the partial images together to generate a panoramic image with an expanded field of view. The stitching means 13 outputs to the recording device 6 a wide-angle video composed of the successively generated panoramic images.

[0034] The visualization area search means 14 matches the feature points of the overall image extracted by the overall image feature point extraction means 11 with the feature points of each partial image extracted by the partial image feature point extraction means 12, and searches for overlapping areas of the partial images in the overall image and subject areas that exist in only one of the partial images in the overlapping areas as visualization areas. Here, the visualization region searching means 14 includes an overlap region searching means 140 and a subject region searching means 141 .

[0035] The overlapping area searching means 140 searches for areas in the entire image where partial images overlap, based on the feature points of the entire image extracted by the entire image feature point extraction means 11 and the feature points of each partial image extracted by the partial image feature point extraction means 12. The overlapping area search means 140 searches for overlapping areas between partial images by matching feature points between the partial images. Then, the overlapping area searching means 140 matches the feature points of the overlapping area of ​​the partial images with the feature points of the entire image, thereby specifying the extent of the overlapping area in the entire image.

[0036] Specifically, the overlapping region searching means 140 calculates the similarity (distance) of the feature amounts at the feature points between the partial images, and associates the most similar feature points. Then, the overlapping region searching means 140 determines the horizontal region including the matching feature points in the partial images as the overlapping region between the partial images. Furthermore, the overlap region search means 140 calculates the similarity (distance) of the feature amounts at the feature points between the overlap region between the partial images and the entire image, and associates the most similar feature points. Then, the overlap region search means 140 specifies a horizontal region including the matching feature points in the entire image as an overlap region corresponding to the entire image. The overlapping region searching means 140 outputs the overlapping region between the partial images thus found to the subject region searching means 141 , and also outputs the overlapping region within the entire image to the region synthesis means 15 .

[0037] The subject region searching means 141 searches for a subject region that exists in one partial image but does not exist in the other partial image, within the overlapping region searched for by the overlapping region searching means 140 . The subject region searching means 141 searches for a region having dissimilar image features in the same overlapping region between the partial images searched for by the overlapping region searching means 140 as a subject region in the partial image. Then, the subject region searching means 141 identifies the range of the subject region in the entire image by matching the feature points of the subject region in the partial image with the feature points of the entire image.

[0038] Specifically, the subject region searching means 141 calculates the similarity for each block of a predetermined size between the same overlapping regions of different partial images, and determines the region consisting of blocks whose similarity is smaller than a predetermined threshold as the subject region in the partial image. This similarity for each block can be determined by a general image similarity determination method such as SSD (Sum of Squared Difference), which is the sum of squares of the error between images. Furthermore, the subject region searching means 141 calculates the similarity of the feature amounts at the feature points between the subject region in the partial image and the entire image, and associates the most similar feature points. Then, the subject region searching means 141 specifies the region including the matching feature points in the entire image as the subject region corresponding to the entire image.

[0039] It should be noted that the subject area does not include a corresponding subject in one of the two partial images, and it is unclear in which partial image the subject is captured. However, the subject region searching means 141 can specify the region of the subject by matching feature points between the subject regions of the two partial images and the entire image. The subject region searching means 141 outputs the searched subject region within the entire image to the region synthesis means 15 .

[0040] The area synthesis means 15 synthesizes the entire image corrected by the wide-angle image correction means 10 with the overlapping area and the subject area, which are the visualization areas searched for by the visualization area search means 14 . The area synthesis means 15 synthesizes, on the entire image, figures, images, etc., which indicate areas such as frames and hatching, as the overlapping areas and the subject area, respectively. For example, as shown in FIG. 3, the area synthesis means 15 draws dashed lines as overlapping areas Dab, Dbc, and a dotted line as a subject area F on the entire image ALL. The area synthesis means 15 outputs the entire image with the visualization areas synthesized to the display device 5.

[0041] With the configuration described above, when generating wide-angle images using multiple cameras 2, the wide-field video shooting support device 1 can visually present to the photographer M areas of overlapping images between the cameras 2 and areas of subjects that exist in only one of the images.

[0042] This allows the cameraman M to operate the cameras 2 to make the overlapping area smaller (narrower) or to move the subject to be photographed away from the overlapping area. Even when photographing a subject in the overlapping area, the cameraman M can operate the cameras 2 so that the subject falls within the photographing range of the two cameras 2. The wide-field video shooting support device 1 can be operated by a program (wide-field video shooting support program) for causing a computer (not shown) to function as each of the above-mentioned means.

[0043] (About the camera work) Now, camera work when capturing wide-angle video using multiple cameras will be described with reference to Fig. 4. Here, the description will be given assuming that subject O is captured by two cameras 2a and 2b.

[0044] FIG. 4 shows a state in which a cameraman M photographs a subject O at an angle of view W where the angles of view of the cameras 2a and 2b overlap, in a photographing area V that is farther than the depth where the photographing ranges of the cameras 2a and 2b intersect. At this time, the wide-field video shooting support device 1 stitches the overlapping area D captured by the cameras 2a and 2b to generate a wide-angle video.

[0045] As shown in FIG. 4(a), when a subject O is present in an overlapping area D that is captured by the camera 2a but not by the camera 2b, even if stitching processing is performed, the subject O may be missing. In this case, the wide-field video shooting support device 1 displays the entire image ALL by combining the overlap area D (Dab) and the subject area F, as shown in Figure 3, thereby informing the cameraman M that there is a possibility that the subject O will be missing from the wide-angle image.

[0046] Then, by zooming out cameras 2a and 2b as shown in FIG. 4(b), cameraman M can place subject O in the overlapping area D captured by cameras 2a and 2b, thereby preventing subject O from being lost in the wide-angle image. In order to prevent the subject O from being omitted, the cameraman M may change the shooting direction (pan direction) of the cameras 2a and 2b to shoot the subject O at a position that is not in the overlapping area D. Furthermore, the cameraman M may perform camerawork so as to make the overlapping area D as small as possible. In this way, the wide-field video shooting support device 1 can assist cameraman M in shooting wide-angle images while preventing the causes of image degradation by presenting overlapping areas and subject areas that cause image degradation.

[0047] <Operation of the wide-field video capture support device> Next, the operation of the wide-field-of-view video shooting support device 1 according to the embodiment of the present invention will be described with reference to Fig. 5 (for the configuration, refer to Fig. 1 as appropriate). Note that the following operation is an operation for each frame of a shot video.

[0048] In step S1, the wide-angle image correction means 10 corrects the wide-angle image captured by the wide-angle camera 3 into a normal image developed on a plane, thereby generating a whole image. In step S2, the whole image feature point extraction means 11 extracts feature points from the whole image generated by correction in step S1 using SIFT, SURF, or the like.

[0049] In step S3, the partial image feature point extraction means 12 extracts feature points from frames (partial images) constituting the video captured by the camera 2, in the same manner as in step S2. Note that step S3 may be performed in parallel with steps S1 and S2. In step S4, overlapping area searching means 140 of visualization area searching means 14 searches for overlapping areas of partial images in the entire image by matching the feature points of the entire image extracted in step S2 with the feature points of the partial images extracted in step S3. Here, overlapping area searching means 140 searches for overlapping areas between partial images, and then identifies an area of ​​the entire image corresponding to the overlapping area as an overlapping area to be a visualized area.

[0050] In step S5, the subject region searching means 141 of the visualization region searching means 14 searches for a region of a subject that exists only in one partial image and does not exist in the other partial image within the overlapping region searched for in step S4, as a subject region. Here, the subject region searching means 141 searches for a dissimilar region in the overlapping region between the partial images, and then specifies the region of the entire image corresponding to that region as a subject region to be a visualized region.

[0051] In step S6, the area synthesis means 15 synthesizes the entire image generated in step S1 with the overlapping area found in step S4 and the subject area found in step S5 as a visualization area. This synthesized image is output to the display device 5 and displayed.

[0052] In step S7, the stitching means 13 synthesizes overlapping areas of the partial images from each camera 2 and the feature points extracted for each partial image in step S3 by a stitching method to connect the partial images and generate a wide-field image (panoramic image). The panoramic images are recorded in the recording device 6 in sequence. It should be noted that step S7 may be performed in parallel with steps S4 to S6.

[0053] In step S8, the wide-field video shooting support device 1 determines the end of shooting based on whether or not video is input from the camera 2 and the wide-angle camera 3, and if video is input (No in step S8), it returns to step S1 to continue operation. On the other hand, if the image input has ended (Yes in step S8), the wide-field image capture support device 1 ends its operation. The termination determination may be made based on whether or not an instruction to terminate has been issued using a switch or the like (not shown).

[0054] Through the above operations, the wide-field video shooting support device 1 can visually present to the cameraman M the overlapping areas of the images that overlap between the cameras 2 when wide-angle images are generated by the multiple cameras 2. Furthermore, when a subject is present in the overlapping area, the wide-field video photography support device 1 can present to the cameraman M whether or not the subject is present in only one of the images from the cameras 2.

[0055] The configuration of the wide-field video shooting system 100 according to the embodiment of the present invention and the configuration and operation of the wide-field video shooting support device 1 have been described above, but the present invention is not limited to this embodiment.

[0056] <Modification> Here, as shown in FIG. 1 and FIG. 4, the multiple cameras 2 of the wide-field video capture system 100 are arranged in a straight line, but the arrangement of the cameras 2 is not limited to this. For example, multiple cameras 2 (2a, 2b, 2c, 2d, 2e) may be arranged on a circumference as shown in Fig. 6. In this case, the wide-angle camera 3 may be arranged in a position that captures the entire field of view of the multiple cameras 2 (2a, 2b, 2c, 2d, 2e).

[0057] Even when the cameras 2 are arranged on a circle, there are cases where the subject O is captured in one image and not captured in the other image in the overlapping area of ​​the shooting range. The example of Fig. 6 shows a state where the subject O is captured by the camera 2b but not by the camera 2c in the overlapping area of ​​the cameras 2b and 2c. In this case, photographer M can zoom out cameras 2 (2a, 2b, 2c, 2d, 2e) as in Figure 4, so that subject O is captured in both of the overlapping areas of cameras 2b and 2c, as shown in Figure 7.

[0058] In addition, here, the wide-field video shooting support device 1 displays on the display device 5 an image in which the visualization area is synthesized onto an overall image generated from the wide-angle image captured by the wide-angle camera 3. However, the display device 5 may display the wide-angle image that is actually generated and output by the stitching means 13 together with the image into which the visualization area is synthesized. This allows the wide-field video shooting support device 1 to present the cameraman M with information necessary for shooting, such as overlapping areas, while also accurately presenting the shooting results.

[0059] Further, here, the wide-field image capture support device 1 is configured to include the stitching means 13, but the stitching means 13 may be separated as an external dedicated device. This allows the wide-field image capture support device 1 to reduce the load. [Explanation of symbols]

[0060] 100 Wide-field imaging system 1. Wide-field imaging support device 10 Wide-angle image correction means 11 Whole image feature point extraction method 12. Partial image feature point extraction method 13 Stitching Method 14 Visualization Area Exploration Tools 140 Overlapping area search means 141 Subject area search means 15 Area synthesis means 2 Camera 3. Wide-angle camera 4 Camera control device 5 Display device 6 Recording Devices

Claims

1. A wide-field-of-view video shooting support device that visualizes an overlapping area in a wide-field-of-view video captured by a plurality of cameras from an image of each of the cameras, the image being captured by the plurality of cameras with a portion of the field of view overlapped with the other images, and an image of a wide-angle camera capturing the entire field of view of the plurality of cameras, the device comprising: a wide-angle image correction means for correcting the wide-angle images of each frame constituting the image of the wide-angle camera into a normal image obtained by expanding the image on a plane, thereby generating an overall image of the entire field of view; an entire image feature point extraction means for extracting feature points from the entire image; a partial image feature point extraction means for extracting feature points from partial images, which are images for each frame constituting an image of the camera, for each of the plurality of cameras; a visualization area search means for searching, in the entire image, an overlapping area of ​​the partial images and a subject area that exists in only one of the partial images in the overlapping area of ​​the partial images by matching the feature points of the entire image with the feature points of each of the partial images, as a visualization area; an area synthesis means for synthesizing and displaying a figure indicating the visualization area on the entire image; A wide-field video shooting support device comprising:

2. The wide-field video shooting support device of claim 1, characterized in that the visualization area search means identifies areas where partial images overlap by matching feature points between the partial images, and searches for the overlapping area in the overall image by matching feature points of the identified area with feature points of the overall image.

3. The wide-field video shooting support device of claim 1 or claim 2, characterized in that the visualization area search means identifies an area of ​​a subject that exists only in one of the partial images by calculating the similarity for each block between partial images having the same overlapping area, and searches for the subject area in the overall image by matching feature points of the identified area with feature points of the overall image.

4. The wide-field-of-view video shooting support device of any one of claims 1 to 3, further comprising a stitching means for stitching areas where feature points of the partial images match to connect the partial images, thereby generating the wide-field-of-view video.

5. A wide-field-of-view video shooting support program for causing a computer to function as the wide-field-of-view video shooting support device according to any one of claims 1 to 4.

6. A wide-field-of-view imaging system for capturing a wide-field-of-view image, comprising: A plurality of cameras arranged so that their fields of view overlap; a wide-angle camera that captures the entire field of view of the plurality of cameras; 5. The wide-field video shooting support device according to claim 4, which displays an overlapping area of ​​the partial images and a subject area that exists only in one of the partial images in the overlapping area as a visualization area from the images captured by each of the plurality of cameras and the image captured by the wide-angle camera, and generates the wide-field video; a camera operation device that controls the orientation and zoom of the plurality of cameras and the wide-angle camera by operation of a cameraman; A wide-field video capture system comprising:

Images