Image synthesis assistance device and its program

Abstract

To generate an image which facilitates adjustment of the position and orientation of an object reflected in an image.SOLUTION: An image composition assisting device 100 comprises: an extraction unit 111 which acquires a plurality of images and extracts edges of an object out of the images; and a composition unit 112 which outputs an image obtained by superimposing images where the edges extracted from the images are displayed. The composition unit 112 may output edges extracted from different images, in different colors or different line types.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image synthesis assistance device that generates an image for assisting in adjusting the position and orientation of an object shown in a synthesized image, and a program thereof.

Background Art

[0002] In a TV program in which a plurality of performers appear, there are many images in which the performers are lined up. However, during a period when an infectious disease is spreading, it is not possible to closely adjacent the performers to each other, and they are often photographed while being separated from each other or with partitions placed between them. When photographed in this way, in order to make it appear on the screen that the performers are lined up adjacent to each other, it may be necessary to synthesize the images of the respective performers that have been photographed. At this time, it is desirable that not only the performers are simply lined up, but also the height position and orientation of an object at the boundary between adjacent images are shown in alignment.

[0003] As a method for generating an image in which the height position and orientation of an object at the boundary are aligned and connected, a method of converting the images so that the feature points of the images are matched and making them appear connected can be considered. As a method of matching the feature points of images, there is a technique described in Non-Patent Document 1. In addition, there is also a method of manually adjusting the position of the camera or adjusting the position and orientation of an object shown in the video while viewing the synthesized image using a switcher used at the broadcast site.

[0004] The situation in conventional image synthesis will be described with reference to FIGS. 11 and 12. As shown in FIG. 11, TV program performers 301 and 302 are seated apart in a shooting studio. Desks 321 and 322 are arranged in front of the performers 301 and 302, and whiteboards 311 and 312, which are background items, are arranged behind them, respectively. Hereinafter, objects to be photographed other than the performers 301 and 302, such as the desks 321 and 322 and the whiteboards 311 and 312, will also be referred to as objects.

[0005] The first camera 210 and the second camera 220 each photograph the performers 301 and 302, and output the photographed images to the switcher 230. A mask image 260 (see FIG. 12 described later) is set in the switcher 230. The switcher 230 refers to the mask image 260, synthesizes the photographed images of the first camera 210 and the second camera 220, and outputs a synthesized image 250.

[0006] FIG. 12 is a diagram showing a mask image 260 that designates the regions of the images to be synthesized in the image synthesis of the conventional switcher 230. The mask image 260 designates the regions of the images to be synthesized (the photographed images of the first camera 210 and the second camera 220, which are the input images to the switcher 230) in the output image (synthesized image) for broadcasting by the luminance of the image. In the left half region of the mask image 260, the luminance is high, indicating the photographed image of the first camera 210. In the right half region, the luminance is low, indicating the photographed image of the second camera 220. The switcher 230 synthesizes the left half of the photographed image of the first camera 210 to be the left half of the synthesized image 250 (see FIG. 11), and the right half of the photographed image of the second camera 220 to be the right half of the synthesized image 250.

[0007] As shown in FIG. 11, as it is, in the synthesized image 250, the displacement between the desks 321 and 322 and the displacement between the whiteboards 311 and 312 at the joints (boundaries) of the images to be synthesized are prominent and the appearance is poor. In order to eliminate / reduce / make less prominent the displacement, the assistant director and the cameraman correct the positions and orientations of the objects (desks 321 and 322, whiteboards 311 and 312), the first camera 210, and the second camera 220 while looking at the synthesized image 250.

Prior Art Documents

Non-Patent Documents

[0008]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0009] Even when using the method of Non-Patent Document 1, there are problems such as specific feature amounts on the image affecting and making it difficult to match the entire image, and the processing time being too long resulting in delays. Also, even when manually adjusting the position and orientation of an object reflected in a camera or video, the degree of freedom of adjustment items is high, and it is difficult to make the joints unobtrusive. Therefore, an object of the present invention is to provide an image synthesis assistance device and a program that generate an image for facilitating adjustment of the position and orientation of an object reflected in a video.

Means for Solving the Problems

[0010] To solve the above-described problems, an image synthesis assistance device according to the present invention includes To be synthesized an extraction unit that acquires a plurality of images and extracts the edges of an object from the plurality of images, and a synthesis unit that outputs to a monitor an image obtained by overlapping images showing the edges extracted from the plurality of images. When adjusting the positions and orientations of the plurality of cameras that captured the plurality of images, or the positions and orientations of the objects shown in the plurality of images, the edges output to the monitor can be referred to .

Effects of the Invention

[0011] The invention according to the present application overlays and displays the edges of an object reflected in the synthesized image. By adjusting the position and orientation of the object while viewing this display image, it becomes possible to make the height position and orientation of the object in the synthesized image match and the image appear to be connected without a sense of incongruity.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments according to the present invention will be described with reference to the drawings. ≪First Embodiment: Configuration of Image Composition Auxiliary Device≫ FIG. 1 is a functional block diagram of an image composition auxiliary device 100 according to the first embodiment. The image composition auxiliary device 100 is a computer and includes a control unit 110, a storage unit 120, and an input / output unit 130. In addition to the switcher 230, user interface devices such as a monitor, a keyboard, and a mouse are connected to the input / output unit 130. Note that the switcher 230 outputs photographed images of the first camera 210 and the second camera 220 here, rather than outputting a composite image 250 (see FIG. 11) to the image composition auxiliary device 100.

[0014] The storage unit 120 is configured to include storage devices such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive). A program 121 is stored in the storage unit 120. The program 121 includes descriptions of the processes of the extraction unit 111 and the synthesis unit 112 described later. The control unit 110 is configured to include a CPU (Central Processing Unit), and is provided with an extraction unit 111 and a synthesis unit 112.

[0015] The extraction unit 111 performs edge extraction processing on the captured images (input images from the switcher 230) of the input first camera 210 and second camera 220, respectively. In the edge extraction processing, processing for extracting the outer contour line of an object is performed. Note that, as an example, the extraction unit 111 may perform edge extraction processing on all objects shown in the captured image, may perform edge extraction processing only on a specified object, or may perform edge extraction processing on a preset object.

[0016] FIG. 2 is a diagram showing a captured image 410 of the first camera 210 according to the first embodiment. Note that in the captured image 410, only the object to be described, omitting small objects and the like in the video captured on the screen, is shown. Among the captured image 410, an image 411 is an image of a whiteboard 311 which is a background small object, and an image 412 is an image of a desk 321. For such a captured image 410, the extraction unit 111 performs edge extraction processing to output an edge image 420 (see FIG. 3 described later).

[0017] FIG. 3 is a diagram showing an edge image 420 for the captured image 410 according to the first embodiment. The edge image 420 includes (displays) an edge 421 of the image 411 showing the whiteboard 311 and an edge 422 of the image 412 showing the desk 321. Note that, to distinguish from the captured image of the object, the edges 421, 422 are described as dotted lines. The captured image 410 is a captured image when adjusting the position and orientation of the object, and the performer 301 is not present (in FIG. 1, the performer 301 is shown by a dotted line).

[0018] Returning to FIG. 1, the continuation of the configuration will be described. The synthesizing unit 112 generates an auxiliary image 430 (see FIG. 4 described later) in which two edge images output by the extracting unit 111 are overlapped, and outputs it to a monitor connected to the input / output unit 130.

[0019] FIG. 4 is a diagram showing the auxiliary image 430 according to the first embodiment. The auxiliary image 430 includes (displays) edges 421 and 422 extracted from the captured image of the first camera 210 and edges 423 and 424 extracted from the captured image of the second camera 220. In this auxiliary image 430, the edge 421 of the object shown in the captured image of the first camera 210 and the edge 423 of the object shown in the captured image of the second camera 220 are displayed so as to overlap (be superimposed). The desks 321 and 322 are approximately at the same height position in the captured image, and the edges 422 and 424 are displayed overlapping. The same applies to the whiteboards 311 and 312, and the edges 421 and 423 are displayed overlapping.

[0020] Note that the synthesizing unit 112 may display such that a part of the edges of the common object (for example, desks 321 and 322) in the auxiliary image 430 to be synthesized overlaps, and basically there is no need to deform the image to improve the alignment accuracy. The synthesizing unit 112 may calculate, for example, similar objects from the videos from the first camera 210 and the second camera 220 based on the feature amounts of the objects, align the positions and orientations on the images of the common objects, and synthesize them so that at least a part of the edges overlaps.

[0021] ≪Auxiliary Image Generation Process≫ FIG. 5 is a flowchart of the auxiliary image generation process according to the first embodiment. In step S11, the extraction unit 111 extracts the edges of the objects from the captured images of the input first camera 210 and second camera 220 (see FIG. 2) respectively, and generates edge images (see FIG. 3). In step S12, the composition unit 112 generates an auxiliary image 430 (see FIG. 4) by overlapping the edge images, and outputs it to the monitor.

[0022] <<Features of the auxiliary image generation process>> By referring to the auxiliary image 430 displayed on the monitor, the assistant director and the cameraman (hereinafter also referred to as the user) can easily adjust the position and orientation of the object and the camera. That is, by looking at the auxiliary image 430 (see FIG. 4), the user can grasp the position and orientation of the entire object shown in the captured images of the first camera 210 and the second camera 220, and the positional and directional relationship (deviation from the target) of the object that is desired to appear connected. Therefore, since the deviation in position and orientation can be grasped, it becomes clear in which direction and by how much the object, for example, the desks 321, 322 should be moved in order for the desks 321, 322 to be horizontal, and it becomes easier to adjust the position and orientation.

[0023] On the other hand, in the composite image 250 (see FIG. 11), only the part of the object that is broadcast (for example, the left half of the captured image of the first camera 210) is shown, and the part hidden in other images (for example, the right half of the captured image of the first camera 210) is not shown. For this reason, it is difficult to grasp in which direction and by how much the object (for example, the desks 321, 322) that is desired to appear connected should be moved with respect to the positional relationship.

[0024] <<Modification example: Edge display form>> In the auxiliary image 430, the edges 421 to 424 extracted from the captured images of the first camera 210 and the second camera 220 are directly superimposed. Therefore, just by looking at the auxiliary image 430, it is difficult for the user to distinguish which of the edges 421 to 424 are the edges of the object photographed by which of the first camera 210 and the second camera 220. To make it easier to distinguish, the display form of the edge may be changed according to the captured image from which the edge is extracted. For example, the color of the edge may be changed according to the captured image from which the edge is extracted.

[0025] FIG. 6 is a diagram showing an auxiliary image 440 according to a modified example of the first embodiment. The synthesizing unit 112 makes the edges 441 and 442 extracted from the captured image of the first camera 210 red (described by a dotted line in FIG. 6) and the edges 443 and 444 extracted from the captured image of the second camera 220 blue (described by a broken line in FIG. 6), and generates and outputs a superimposed auxiliary image 440.

[0026] The colors of the edges 441 to 444 are different according to the captured images from which the edges are extracted, making it easier to distinguish which of the edges 441 to 444 are the objects photographed by either the first camera 210 or the second camera 220, and thus making it easier to adjust the position and orientation of the object. Also, by looking at the degree of color overlap (color mixing), it becomes easier to grasp the degree of edge overlap (the degree to which the joints are not noticeable). For example, it becomes possible to confirm that the adjustment of the position and orientation is progressing by the increase in the purple edges due to the overlap of the red edges and the blue edges. That is, the synthesizing unit 112 may display the overlapping portions of the colors of the edges 443 and 444 as changed to the color of the mixed result with reference to a chromaticity diagram or the like. In addition to the color, the synthesizing unit 112 may change the line type of the edge (for example, solid line, dotted line, broken line, dash-dot line, multiple line, thick line, etc.) according to the captured image from which the edge is extracted.

[0027] <<Modified Example: Emphasized Display of Feature Points>> When the user adjusts the position and orientation of the object so that the edges overlap while viewing the auxiliary images 430 and 440 (see FIGS. 4 and 6), if the feature points of the edges can be identified, it becomes easier to make further adjustments. In the auxiliary images, the feature points of the edges (for example, points corresponding to corners on the image, points where the edges intersect) may be displayed so as to be easily grasped. Specifically, the extraction unit 111 extracts edges from the input image and further extracts the feature points of the edges. The synthesis unit 112 overlaps the edges and generates and outputs an image in which the feature points are emphasized.

[0028] FIG. 7 is a diagram showing an auxiliary image 450 according to a modification of the first embodiment. The auxiliary image 450 is an image obtained by emphasizing the feature points of the edges 441 to 444 in the auxiliary image 440 (see FIG. 6) by adding circles 445 and 446. Instead of adding the circles 445 and 446, the edges near the feature points may be displayed thicker, the color may be changed (to a darker color), or they may be displayed blinking. By paying attention to the feature points, it becomes even easier to adjust the position and orientation of the object. Also, when the desks 321 and 322 or the whiteboards 311 and 312 are of the same size and are adjusted to overlap, they may be adjusted so that the feature points overlap, which makes it even easier to adjust. Also, the corresponding feature points determined by calculation may be connected by lines and displayed.

[0029] <<Second Embodiment>> In the first embodiment, the input to the image synthesis assistance device 100 is the captured images of the first camera 210 and the second camera 220. The input to the image synthesis assistance device 100A according to the second embodiment (see FIG. 8 described later) further includes the mask image 260 (see FIG. 12) output from the switcher 230. Setting information of the composite image 250 in place of the mask image 260 may be input from a device connected to the input / output unit 130. The image synthesis assistance device 100A generates an auxiliary image with reference to the boundary in the composite image 250 (see FIG. 11).

[0030] FIG. 8 is a functional block diagram of the image synthesis assistance device 100A according to the second embodiment. In addition to the edge extraction process for the captured image, the extraction unit 111A extracts the luminance boundary (the boundary of the image to be synthesized) from the mask image 260.

[0031] FIG. 9 is a diagram for explaining the boundary and the vicinity of the boundary according to the second embodiment. The boundary in the mask image 260 is a vertical line that divides the mask image 260 into left and right halves, and corresponds to the boundary 271 that divides the image 270 into left and right halves. The vicinity of the boundary 272 is a region composed of points where the distance to the boundary 271 is within a predetermined value.

[0032] Returning to FIG. 8, the description of the configuration continues. The extraction unit 111A extracts the edges included in the vicinity of the boundary 272 among the extracted edges as the edges in the vicinity of the boundary. The synthesis unit 112A changes the color of the edges extracted by the extraction unit 111A according to the extraction source image, overlays them, and further generates and outputs an image in which the edges in the vicinity of the boundary are emphasized. The synthesis unit 112A also calculates the overlapping degree L using the following formula.

[0033]

Equation

[0034] Here, IA and IB are input images (captured images), and IAe and IBe are the edge images of IA and IB, respectively. (x, y) indicates the coordinates of the pixel, and Δ is the vicinity of the boundary 272. c is the channel into which the image is input, and C is the number of channels. IA and IB are color images with 3 channels, and IAe and IBe are black-and-white images with 1 channel. N is the number of pixels in the vicinity of the boundary Δ, and α is an adjustable parameter between 0 and 1. The synthesis unit 112A includes the calculated overlapping degree L in the image as, for example, a meter 466 (see FIG. 10 described later).

[0035] FIG. 10 is a diagram showing the auxiliary image 460 according to the second embodiment. In the auxiliary image 460, portions 462 and 464 of the edges 461 and 463 near the boundary 272 are thickened and emphasized. Also, a meter 466 is arranged on the lower left side of the auxiliary image 460. The higher the degree of overlap L, the higher the value indicated by the meter 466. Note that in order to emphasize the edges in the vicinity of the boundary 272 (see FIG. 9), the extraction unit 111A may apply a morphological process to the vicinity of the boundary 272 to dilate the edges. In this case, the composition unit 112A overlays the edge image with the edges in the vicinity of the boundary 272 emphasized as it is. Alternatively, the edges in the vicinity of the boundary may be emphasized by changing the color to a color different from that of the edges outside the vicinity of the boundary or changing the line type.

[0036] ≪Second Embodiment: Features of Image Composition Assistance Device≫ The image composition assistance device 100A emphasizes and displays portions 462 and 464 (see FIG. 10), which are the boundaries of the images to be composed in the composite image 250 and are the joints of the objects, or displays the degree of overlap (see the meter 466). The boundary is not necessarily at the center of the image. For example, when three captured images are equally composed on the left, in the center, and on the right, the boundary is not at the center, and it becomes difficult for the user to grasp the position of the boundary. By emphasizing the edges near the boundary, the user can grasp the boundary (joint) and it becomes easier to make adjustments. Also, since the user can know the degree of overlap, it becomes easier to know whether the adjustment is being made in the correct direction (so that the joint becomes less conspicuous) or in the reverse direction, and thus easier to make adjustments.

[0037] ≪Modification Example: Output of Degree of Overlap≫ In the second embodiment described above, the degree of overlap L is displayed as the meter 466, but it may be output in other formats. The composition unit 112A may output, for example, as a higher sound as the degree of overlap L increases to a speaker connected to the input / output unit 130, or as a sound with a faster tempo, or may output it numerically.

[0038] ≪Modification Example: Selection of Objects for Extracting Edges≫ In the above-described embodiment, edges are extracted from the captured image, and as a result, the outlines of all the objects shown in the captured video are included as edges in the auxiliary images 430, 440, 450, and 460. It is also possible to select the objects that are the sources of the edges included in the auxiliary images 430, 440, 450, and 460.

[0039] For example, the user may select the objects. The extraction units 111, 111A extract the areas of the objects included in the input image (captured image) using object recognition technology and output them to the touch panel display connected to the input / output unit 130. The extraction units 111, 111A perform edge extraction processing on the objects specified by the user.

[0040] It is also possible to allow the user to specify the area within the input image from which edges are to be extracted. The extraction units 111, 111A perform edge extraction processing on the area specified by the user.

[0041] The type of the object (for example, a desk, a whiteboard, etc.) may be set, and the edges of the objects of that type may be extracted or not extracted. The extraction units 111, 111A recognize the type of the objects included in the input image using object recognition technology and may extract or not extract the edges of the objects of the set type.

[0042] There are no joints in the objects that do not cross the boundary 271 (see FIG. 9) and they are not subject to adjustment, so there is no need to extract edges in the first place. The extraction units 111, 111A may extract the areas of the objects included in the input image (captured image) and target the objects whose areas overlap the boundary 271 for edge extraction. Alternatively, the extraction units 111, 111A may delete the edges that do not overlap the boundary 271 (do not cross the boundary 271) and extract only the edges that cross (intersect) the boundary 271.

[0043] ≪Other Modification Examples≫ As described above, some embodiments of the present invention have been explained. However, these embodiments are merely examples and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and various changes such as omission and substitution can be made without departing from the gist of the present invention.

[0044] For example, the function of the image synthesis assistance device may be incorporated into a switcher. Also, the camera may be provided with an extraction unit, and the switcher may be provided with a synthesis unit. These embodiments and their modifications are included in the scope and gist of the invention described in this specification and the like, and are also included in the invention described in the claims and its equivalent scope.

Explanation of Reference Numerals

[0045] 100, 100A Image synthesis assistance device 111, 111A Extraction unit 112, 112A Synthesis unit 121 Program 250 Synthesized image 271 Boundary 272 Vicinity of the boundary 430, 440, 450, 460 Auxiliary image 421~424, 441~444 462, 464 Portion (object edge that becomes the vicinity of the boundary) 466 Meter

Claims

1. An extraction unit that acquires a plurality of images to be synthesized and extracts object edges from the plurality of images, and a synthesis unit that outputs, to a monitor, an image obtained by overlapping images showing the edges extracted from the plurality of images, wherein the edges output to the monitor can be referred to when adjusting the positions and orientations of a plurality of cameras that captured the plurality of images, or the positions and orientations of objects shown in the plurality of images. An image synthesis assistance device characterized by this.

2. The image synthesis assistance device according to claim 1, wherein the extraction unit extracts the edges of the object specified via an input / output unit.

3. The image synthesis assistance device according to claim 1, wherein the synthesis unit outputs, to the monitor, an image obtained by overlapping images showing the edges such that at least a part of the edges overlap.

4. The image synthesis assistance device according to claim 1, wherein the extraction unit acquires boundaries of the plurality of images in a composite image of the plurality of images and extracts the edges that cross the boundaries.

5. The image synthesis assistance device according to claim 1, wherein the extraction unit extracts edges of a region specified in advance from the plurality of images.

6. The image synthesis assistance device according to claim 1, wherein the synthesis unit outputs edges extracted from different images in different colors or line types.

7. The extraction unit further extracts feature points of the edges, and the synthesis unit outputs while emphasizing the feature points. The image synthesis assistance device according to claim 1, characterized by this.

8. The extraction unit acquires boundaries of the plurality of images in a composite image of the plurality of images, and extracts object edges that are part of the edges and are in the vicinity of the boundary where the distance from the boundary is within a predetermined value, and the synthesis unit outputs object edges in the vicinity of the boundary extracted from different images in a color or line type different from that of the edges. The image synthesis assistance device according to claim 1, characterized by this.

9. The synthesis unit calculates the degree of overlap of object edges in the vicinity of the boundary, and outputs a sound corresponding to the degree of overlap to a speaker, or outputs a meter corresponding to the degree of overlap to a monitor. The image synthesis assistance device according to claim 8, characterized by this.

10. A program for causing a computer to function as the image synthesis assistance device according to any one of claims 1 to 9.

Images