Image synthesis device, image synthesis method, and image synthesis program
The video synthesis device addresses double image issues in composite videos by extracting and selecting foregrounds based on size and deformation, ensuring clear object representation in the superimposed region.
Patent Information
- Application Number
- JP2024552328
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-12-28
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-12-28
AI Technical Summary
Existing video compositing devices display tall objects as double images in the superimposed region of composite videos due to camera perspectives.
A video synthesis device that extracts foregrounds from multiple camera videos, determines overlapping areas, and selects either the first or second foreground based on size and deformation to generate a composite video without double images.
Generates a composite video that effectively eliminates double images of objects in the superimposed region by prioritizing foregrounds with minimal deformation.
Smart Images

Figure 0007584723000001 
Figure 0007584723000002 
Figure 0007584723000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an image synthesis device, an image synthesis method, and an image synthesis program. [Background technology]
[0002] In order to widen the angle of view of an image, an image synthesizing device is known that synthesizes multiple images (i.e., multiple camera images) captured by multiple imaging devices to generate one synthetic image. For example, Patent Document 1 describes an image synthesizing device that generates a synthetic image using a blending process. This image synthesizing device creates a reference table including weighting coefficients at each pixel position from the camera parameters of each imaging device and the pixel position information calculated using the camera parameters, and performs image synthesis processing by calculating the color value of each pixel position with reference to the reference table. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 6513305 Summary of the Invention [Problem to be solved by the invention]
[0004] However, in the above image synthesis device, when a tall object such as a person is captured in the overlapping area between the camera images, the camera image of the object extends in a direction away from the position of the imaging device on a straight line connecting the position of the imaging device and the object, so that the object captured from a different position is displayed as a double image in the overlapping area of the synthesis image.
[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to generate a composite image in which objects within the overlapping area are not displayed as double images. [Means for solving the problem]
[0006] The image synthesizing device of the present disclosure includes an image receiving unit that receives a first image and a second image captured from different positions; a foreground extraction unit that extracts a first foreground in the first image and a second foreground in the second image; an overlapping area determination unit that associates a first foreground portion of the first foreground and a second foreground portion of the second foreground, the first and second images showing the same object, in an overlapping area between the first and second images; a selection process that selects, when the first and second foreground portions showing the same object exist, either the first or second foreground portion as a foreground portion to be displayed in the overlapping area based on sizes of the first and second foreground portions; and a synthesis process that displays the foreground portion selected in the selection process in an image obtained by synthesizing the first and second images, thereby generating a synthetic image. and in the selection process, when the first foreground portion and the second foreground portion showing the same object are present and neither the first foreground portion nor the second foreground portion is in contact with the boundary of the superimposed area, the image processing unit selects either the first foreground portion or the second foreground portion as the foreground portion based on sizes of the first foreground portion and the second foreground portion, and when another first foreground portion and another second foreground portion showing another object are present and one of the other first foreground portion and the other second foreground portion is not in contact with the boundary of the superimposed area and the other is in contact with the boundary of the superimposed area, the image processing unit selects the one foreground portion. .
[0007] The image synthesis method of the present disclosure is a method executed by an image synthesis device, and includes the steps of: receiving a first image and a second image captured from different positions; extracting a first foreground in the first image and a second foreground in the second image; associating a first foreground portion of the first foreground and a second foreground portion of the second foreground, which indicate a same object, in an overlapping area between the first image and the second image; performing a selection process of selecting, when the first foreground portion and the second foreground portion indicating the same object exist, either the first foreground portion or the second foreground portion as a foreground portion to be displayed in the overlapping area based on sizes of the first foreground portion and the second foreground portion; and performing a synthesis process of displaying the foreground portion selected in the selection process in an image obtained by synthesizing the first image and the second image, thereby generating a synthetic image. and generating the composite image includes, in the selection process, selecting either the first foreground part or the second foreground part as the foreground part based on sizes of the first foreground part and the second foreground part when the first foreground part and the second foreground part showing the same object are present and neither the first foreground part nor the second foreground part is in contact with the boundary of the superimposed area, and selecting one of the first foreground part and the second foreground part when another first foreground part and another second foreground part showing another object are present and one of the other first foreground part and the other second foreground part is not in contact with the boundary of the superimposed area and the other is in contact with the boundary of the superimposed area. It is characterized by: Effect of the Invention
[0008] According to the present disclosure, it is possible to generate a composite image in which objects within the overlap region are not displayed as double images. [Brief description of the drawings]
[0009] [Figure 1] 1 is a functional block diagram illustrating a schematic configuration of a video synthesizing device according to a first embodiment. [Diagram 2] 13A to 13C are explanatory diagrams showing a synthesis process in a superimposed area of camera images. [Diagram 3] 1 is a diagram illustrating an example of a hardware configuration of an image synthesizing device according to a first embodiment. [Figure 4] 4 is a flowchart showing an image synthesis process of the image synthesizing device according to the first embodiment. [Diagram 5] 10 is a flowchart showing overlapping region processing in the first embodiment. [Figure 6] 13A to 13E are diagrams illustrating overlapping region processing according to the first embodiment. [Figure 7] 13(A) and 13(B) are diagrams illustrating overlapping region processing in the second embodiment. [Figure 8] 13 is a flowchart showing overlapping region processing in the second embodiment. [Figure 9] 13(A) and 13(B) are diagrams illustrating overlapping region processing in the third embodiment. [Figure 10] 13 is a flowchart showing overlapping region processing in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, a video synthesis device, a video synthesis method, and a video synthesis program according to embodiments will be described with reference to the drawings. The following embodiments are merely examples, and the embodiments can be appropriately combined and each embodiment can be appropriately modified.
[0011] 1. First embodiment 1-1 Configuration of the First Embodiment FIG. 1 is a functional block diagram showing a schematic configuration of an image synthesizing device 10 according to the first embodiment. The image synthesizing device 10 is a device that generates a composite image Ic from a plurality of camera images (for example, a first image Ia and a second image Ib) captured by a plurality of imaging devices 20a, 20b, ... (i.e., a plurality of cameras) and displays the composite image on a display device 30. A plurality of imaging devices 20a, 20b, ... such as a plurality of cameras, the image synthesizing device 10, and a display device 30 such as a liquid crystal display constitute an image system. The image system is, for example, a monitoring system for monitoring a monitoring area with images. The image synthesizing device 10 is a device that can execute the image synthesizing method according to the first embodiment. Also, the image synthesizing device 10 is, for example, a computer serving as an information processing device that can execute the image synthesizing program according to the first embodiment.
[0012] 1, image synthesizing device 10 includes image receiving unit 11 that receives images, parameter input unit 12 that receives camera parameters or receives and stores camera parameters, image processing unit 13, foreground extraction unit 14, overlapping area determination unit 15, and display processing unit 16 that transmits composite image data based on composite image Ic to display device 30. However, foreground extraction unit 14 and overlapping area determination unit 15 may be part of image processing unit 13.
[0013] 2A to 2C are explanatory diagrams showing a synthesis process in an overlapping region Ov of a first image Ia and a second image Ib, which are camera images. The image receiving unit 11 receives the first image Ia and the second image Ib captured from different positions. Usually, the first image Ia and the second image Ib shown in FIG. 2A are images after conversion using camera parameters. The camera parameters include internal parameters that are parameters unique to each of the imaging devices 20a and 20b, external parameters that are camera parameters indicating the position and orientation of each of the imaging devices 20a and 20b in the world coordinate system, and lens distortion correction coefficients (e.g., lens distortion correction maps) used to correct distortion unique to the lenses of each of the imaging devices 20a and 20b.
[0014] The foreground extraction unit 14 extracts the foreground in the first image Ia (also referred to as the "first foreground") and the foreground in the second image Ib (also referred to as the "second foreground") The overlapping area determination unit 15 associates a first foreground portion F1 of the first foreground and a second foreground portion F2 of the second foreground (shown in FIG. 2(B)) showing the same object (e.g., a person) in the overlapping area Ov (shown in FIG. 2(A)) between the first image Ia and the second image Ib. The overlapping area determination unit 15 associates the first foreground portion F1 with the second foreground portion F2, for example, based on the coordinate (t) of the lowest position of the first foreground portion F1 and the coordinate of the lowest position of the second foreground portion F2. The coordinate of the lowest position is, for example, the coordinate of a body part that is in contact with the ground, for example, the coordinate of the position of a person's feet. If the coordinate of the lowest position of the first foreground portion F1 and the coordinate of the lowest position of the second foreground portion F2 match (or are within a predetermined threshold), the overlapping area determination unit 15 determines that the first foreground portion F1 and the second foreground portion F2 are foreground portions indicating the same object, and associates the first foreground portion F1 with the second foreground portion F2.
[0015] When a first foreground portion F1 and a second foreground portion F2 showing the same object are present in the overlapping area Ov, the image processing unit 13 performs a selection process to select either the first foreground portion F1 or the second foreground portion F2 as the foreground portion to be displayed in the overlapping area Ov based on the sizes of the first foreground portion F1 and the second foreground portion F2, and a synthesis process to display the foreground portion selected in the selection process in an image obtained by synthesizing the first image Ia and the second image Ib, thereby generating a synthetic image Ic.
[0016] As shown in FIG. 2B, the length L1 of the first foreground portion F1 and the length L2 of the second foreground portion F2 are used as the size of the first foreground portion F1 and the second foreground portion F2. The deformation rate (i.e., the degree of deformation or the stretch rate) of the first foreground portion F1 with respect to the actual dimensions of the object and the deformation rate (i.e., the degree of deformation or the stretch rate) of the second foreground portion F2 with respect to the actual dimensions of the object may be used as the size of the first foreground portion F1 and the second foreground portion F2. In addition, the length of the foreground portion in the diagonal direction of the rectangular bounding box B1 surrounding the first foreground portion F1 and the rectangular bounding box B2 surrounding the second foreground portion F2 may be used as the size of the first foreground portion F1 and the second foreground portion F2.
[0017] Furthermore, the size of the first foreground portion F1 and the second foreground portion F2 may be expressed by the area of a bounding box B1 surrounding the first foreground portion F1 and a bounding box B2 surrounding the second foreground portion F2. The area of each of the bounding boxes B1 and B2 may be expressed by the number of pixels in the bounding box. The size of the first foreground portion F1 and the second foreground portion F2 may be expressed by the deformation rate of the first foreground portion F1 with respect to the actual dimensions of the object (i.e., the degree of deformation or the magnification rate), and the deformation rate of the second foreground portion F2 with respect to the actual dimensions of the object (i.e., the degree of deformation or the magnification rate).
[0018] The image processor 13 composites the background of the first image Ia and the background of the second image Ib. For example, alpha blending is performed to composite the backgrounds. The image processor 13 uses the camera parameters to set a composite weight for each pixel of the image data, and performs alpha blending as a composite process by referring to the weight information.
[0019] The foreground extraction unit 14 performs background subtraction processing on the input video to obtain the foreground portion (information that is not the background, such as people). At this time, image processing such as opening processing and closing processing is performed to remove noise and fill in pixel holes.
[0020] In the selection process, the image processing unit 13 selects the smaller foreground portion of the first foreground portion F1 or the second foreground portion F2, and performs a synthesis process in which the foreground portion selected in the selection process (for example, the second foreground portion F2 in FIG. 2(C)) is displayed on an image obtained by synthesizing the first image Ia and the second image Ib, thereby generating a synthetic image Ic. Note that the number of images input to the image receiving unit 11 may be three or more, and two images having an overlapping area Ov among the multiple images are synthesized in a similar manner. The display processing unit 16 outputs the synthetic image data to the display device 30. The display device 30 displays an image based on the received synthetic image data.
[0021] 3 is a diagram showing an example of a hardware configuration of the image synthesizing device 10 according to the first embodiment. The image synthesizing device 10 includes, for example, a main processor 101 as a processor such as a CPU (Central Processing Unit), a main memory 102, an auxiliary memory 103, an image processor 104 as a parallel arithmetic device such as a GPU (Graphics Processing Unit), an image processing memory 105, an input interface 106, a file interface 107, a display interface 108, and an image input interface 109. Note that the processes of the image processor 104 and the image processing memory 105 may be executed by the main processor 101 and the main memory 102.
[0022] In FIG. 3, each part constituting the image synthesizing device 10 is realized by, for example, a processing circuit. The processing circuit may be dedicated hardware, or may include a processor that executes a program (for example, an image synthesizing program) stored in the main memory 102. The main memory 102 is, for example, a semiconductor memory such as a random access memory (RAM). The auxiliary memory 103 is, for example, a non-volatile storage device such as a semiconductor memory or a magnetic disk. The image synthesizing device 10 may be a mixture of a configuration including a processing circuit and a configuration having a main processor. The image synthesizing device 10 may be a server computer on a network. The image synthesizing program is provided by downloading via the network, or by a storage medium such as a USB memory that stores information.
[0023] In the example of Fig. 3, a series of processes is controlled by main processor 101, and video-related processes are processed by video processor 104 to speed up. Input video data is stored in main memory 102 and transferred to video processing memory 105 for processing by video processor 104. Also, composite video data created by the video processor is saved in video processing memory 105. The processes that can be speeded up by the video processor correspond to the processes by video processing unit 13, foreground extraction unit 14, and overlapping area determination unit 15. Note that in foreground extraction unit 14 and overlapping area determination unit 15, the processes are limited to the overlapping area Ov portion, so that it is possible to speed up by transferring and processing data only for the relevant area, rather than data for the entire camera video.
[0024] The video receiving unit 11 includes a video input interface 109. The parameter input unit 12 includes a file interface 107, acquires camera parameter information from the auxiliary memory 103, and writes it to the main memory 102. The display processing unit 16 includes a display interface 108, and outputs composite video data based on the created composite video to the display device 30. The input interface 106 accepts device inputs such as mouse input, keyboard input, and touch panel input, and sends the input information to the main processor 101. Note that the hardware configuration in FIG. 3 is an example and can be changed.
[0025] 1-2 Operation of the First Embodiment 4 is a flowchart showing the image synthesis process of the image synthesizer 10 according to embodiment 1. First, in the initialization process of step S11, the image processor 13 sets in advance the synthesis weights of the camera images (for example, the first image Ia and the second image Ib) in the overlapping area Ov using input camera parameter information.
[0026] In the video input process of step S12, the video receiving unit 11 receives the video from the imaging devices 20a, 20b, ... and stores the video data in the main memory. In the video synthesis process of step S13, the video processing unit 13 transfers the data to the video processing memory 105 for processing by the video processor 104. A synthetic video is generated for the background of the video by referring to the synthesis weight information created in step S11.
[0027] In the overlapping area processing of step S14, the foreground extraction unit 14 determines whether or not there is a foreground and extracts the foreground, the overlapping area determination unit 15 determines the overlapping area Ov, and the image processing unit 13 synthesizes the images. The processing of step S14 will be described in detail with reference to Fig. 5 and Figs. 6(A) to (E). In step S15, the processing of S12 to S14 is repeated until the input of the images is completed.
[0028] Fig. 5 is a flowchart showing the overlapping area processing (step S14) in embodiment 1. Figs. 6(A) to (E) are explanatory diagrams showing the overlapping area processing S14 in embodiment 1. In step S101, foreground extraction unit 14 receives camera video stored in video processing memory 105 as input, and extracts a foreground portion from the camera video by, for example, background difference processing.
[0029] In step S102, the foreground extraction unit 14 removes small patterns and noise by performing an opening process that performs an equal number of expansion and contraction processes on the foreground portion acquired in step S101. In addition, the foreground extraction unit 14 fills in the detected object (e.g., a person) by performing a closing process that performs an equal number of contraction and expansion processes, so that the object is not represented as being hollow.
[0030] In step S103, the overlapping region determination unit 15 receives the foreground portion acquired in step S102 and determines whether the target foreground portion (e.g., the first foreground portion F1 or the second foreground portion F2) exists in the overlapping region Ov. If a foreground portion (e.g., the first foreground portion F1 and the second foreground portion F2) exists (YES in step S103), the process proceeds to step S104, and if no foreground portion exists (NO in step S103), the composition result based on the above-mentioned composition weight is adopted as it is, and the overlapping region process ends.
[0031] In step S104, overlap area determination unit 15 determines whether a foreground portion is present in only one of the camera images, and if a foreground portion is present in only one of the camera images as shown in FIG. 6(A) (YES in step S104), processing proceeds to step S105; otherwise (NO in step S104), processing proceeds to step S106.
[0032] In step S105, when only one of the camera images includes a foreground portion, the image processor 13 preferentially selects the camera image including the foreground portion. For example, in the case of Fig. 6(A), the image processor 13 selects the first image Ia, which is the camera image including the first foreground portion F1, as the image of the overlapping region Ov.
[0033] In step S106, overlapping area determination unit 15 determines whether the foreground part of one camera image abuts the boundary of overlapping area Ov. If the foreground part abuts the boundary of overlapping area Ov (YES in step S106), the process proceeds to step S107, and if not (NO in step S106), the process proceeds to step S108. For example, in the cases of Figures 6(B) and 6(E), the process proceeds to step S107, and in the cases of Figures 6(C) and 6(D), the process proceeds to step S108.
[0034] In step S107, when the foreground portion of one camera image abuts the boundary of the overlapping area Ov and the foreground portion of the other camera image does not abut the boundary of the overlapping area Ov, the image processor 13 preferentially selects the camera image including the foreground portion that does not abut the boundary of the overlapping area Ov. For example, in the case of Fig. 6(B), the image processor 13 selects the first image Ia which is the camera image including the first foreground portion F1. Also, in the case of Fig. 6(E), the image processor 13 selects the second image Ib which is the camera image including the first foreground portion F1.
[0035] In step S108, the image processor 13 calculates the degree of deformation of the foreground portion, and in step S109, the image processor 13 preferentially selects a camera image including a foreground portion with a small degree of deformation among the corresponding foreground portions. For example, in the case of FIG. 6(C), the image processor 13 selects a first image Ia which is a camera image including a first foreground portion F1 with a small area of a bounding box B1. Also, in the case of FIG. 6(D), the image processor 13 selects a second image Ib which is a camera image including a second foreground portion F2 with a small area of a bounding box B2. The degree of deformation refers to the rate of change in the area occupied by the foreground portion (area of the bounding box), the rate of extension of the length of the foreground portion (the length of the foreground portion in the diagonal direction of the bounding box), and the like. The area can be calculated from the number of pixels of the bounding box of the foreground portion. In the case of elongation of the foreground part, the bounding box of the foreground part is obtained, and the diagonal line of the bounding box is drawn starting from the vertex of the bounding box that is closest to the position of the imaging device, and the first point where the diagonal line intersects with the foreground is obtained as viewed from both ends of the diagonal line, and the distance between the two points is calculated. However, the method of calculating the area or elongation is not limited to these methods.
[0036] In step S110, the image processor 13 refers to and reflects pixel values of the camera image selected in steps S105, S107, and S109 with respect to pixels of the foreground portion extracted within the overlapping area Ov in each camera image.
[0037] 1-3. Advantages of the First Embodiment According to the first embodiment, it is possible to generate a composite image Ic that does not display objects in the overlapping area Ov as double images. Moreover, according to the first embodiment, since foreground parts with a small degree of deformation are preferentially displayed and foreground parts with a large degree of deformation are not displayed (i.e., only foreground parts with a small degree of deformation are displayed in the overlapping area Ov), it is possible to make the display device 30 properly display objects in the overlapping area Ov.
[0038] 2. Second embodiment In the second embodiment, an image synthesizing device and an image synthesizing method will be described for a case where multiple objects exist in the overlapping area Ov. The functional blocks and hardware configuration of the image synthesizing device according to the second embodiment are similar to those of the first embodiment (shown in Figs. 1 and 3). Therefore, Figs. 1 and 3 will be referred to in describing the second embodiment.
[0039] 7(A) and (B) are explanatory diagrams showing the overlapping area processing in embodiment 2. In the selection process, when a first foreground portion F11 and a second foreground portion F12 showing the same object are present and neither the first foreground portion F11 nor the second foreground portion F12 is in contact with the boundary of the overlapping area Ov, the video processing unit 13 selects either the first foreground portion F11 or the second foreground portion F12 as the foreground portion to be displayed in the overlapping area Ov of the composite video based on the sizes of the first foreground portion F11 and the second foreground portion F12. In addition, in the selection process, when there are another first foreground part F21 and another second foreground part F22 indicating another object, and one of the other first foreground part F21 and the other second foreground part F22 does not abut the boundary of the overlapping area Ov, and the other abuts the boundary of the overlapping area Ov, the image processing part 13 selects the foreground part abutting the boundary of the overlapping area Ov as the foreground part to be displayed in the overlapping area Ov of the composite image.
[0040] Fig. 8 is a flowchart showing overlapping area processing in embodiment 2. The flowchart in Fig. 8 differs from the flowchart in Fig. 5 in that the processes of steps S204 and S205 and the loop of step S206 are added. The processes of steps S201 to S203 and S207 to S213 in Fig. 8 are the same as the processes of steps S101 to S103 and S104 to S110 in Fig. 5.
[0041] In step S204, if the overlap area Ov includes a foreground portion, the overlap area determination unit 15 performs object detection processing using the video data as input, obtains the number of objects (i.e., the number of targets) detected in the overlap area Ov, as well as the coordinates of the bounding box of each object and the type of object, and classifies the foreground portion.
[0042] In step S205, the overlapping area determination unit 15 calculates the coordinates corresponding to the feet of each foreground part recognized in step S204. First, a diagonal line of the bounding box is drawn starting from the vertex closest to the position of the imaging device among the four vertices of the bounding box. Then, the first point that intersects with the foreground part as viewed from the starting point is obtained as the foot coordinate. The calculated coordinates are associated between the cameras. The difference in the foot coordinates after synthesis between both cameras is confirmed, and if it is equal to or less than a certain threshold and the target type is the same, it is determined that it is the same foreground part.
[0043] In step S206, the image processing unit 13 performs the processes of steps S207 to S213 on all the foreground parts classified and associated in steps S204 and S205, and performs image synthesis in the overlapping region Ov.
[0044] According to the second embodiment, it is possible to generate a composite image Ic that does not display objects in the overlapping area Ov as double images. Also, according to the second embodiment, it is possible to display objects in the overlapping area Ov appropriately (i.e., an image of the foreground part with little distortion) on the display device 30, as in the case of the first embodiment.
[0045] Other than the above, the second embodiment is the same as the first embodiment.
[0046] 3. Third embodiment In the third embodiment, a video synthesis device and a video synthesis method are described for a case where multiple objects exist in the overlapping area Ov and the foreground parts of the multiple objects in the overlapping area Ov have overlapping parts. The functional blocks and hardware configuration of the video synthesis device according to the third embodiment are similar to those of the first embodiment (shown in Figs. 1 and 3). Therefore, Figs. 1 and 3 will be referred to in describing the third embodiment.
[0047] 9(A) and 9(B) are explanatory diagrams showing overlapping area processing in embodiment 3. When there is a foreground part (e.g., the first foreground part F11 and the other first foreground part F21) in the overlapping area Ov of the first image Ia that does not correspond to any of the foreground parts (e.g., the second foreground part F12 and the other second foreground part F22) in the overlapping area Ov of the second image Ib, the overlapping area determination unit 15 integrates the foreground part (e.g., the first foreground part F11 and the other first foreground part F21) that does not correspond to any of the foreground parts in the overlapping area Ov of the first image Ia into one foreground part. Also, as shown in FIG. 9(A), when there is a foreground portion within the overlapping area Ov of the second image Ib (e.g., the second foreground portion F12 and the other second foreground portion F22) that does not correspond to any of the foreground portions within the overlapping area Ov of the first image Ia (e.g., the first foreground portion F11 and the other first foreground portion F21), the overlapping area determination unit 15 integrates the foreground portion that does not correspond to any of the foreground portions within the overlapping area Ov of the second image Ib and the other foreground portions (e.g., the second foreground portion F12 and the other second foreground portion F22) into a single foreground portion.
[0048] Fig. 10 is a flowchart showing overlapping area processing in embodiment 3. The flowchart in Fig. 10 differs from the flowchart in Fig. 8 in that steps S306, S307, and S308 are added. The processes in steps S301 to S305 and S309 to S316 in Fig. 10 are the same as the processes in steps S201 to S205 and S206 to S213 in Fig. 8.
[0049] In step S304, if the overlapping region Ov contains a foreground portion, the overlapping region determination unit 15 receives the foreground portion acquired in step S302, extracts the contour of the foreground portion, and classifies the foreground portion. At this time, the overlapping region determination unit 15 also acquires the coordinates of the bounding box of the contour.
[0050] In step S305, if each foreground part recognized in step S304 is a person, the overlapping area determination unit 15 calculates coordinates corresponding to the person's feet. First, the overlapping area determination unit 15 draws a diagonal line of the bounding box starting from the vertex closest to the camera direction among the four vertices of the bounding box, and obtains the first point of intersection with the foreground from the starting point as the foot coordinate. The overlapping area determination unit 15 associates the calculated coordinates between the cameras. For example, the overlapping area determination unit 15 checks the difference in foot coordinates after synthesis between both cameras, and determines that they are the same foreground part if the difference is equal to or less than a certain threshold.
[0051] In step S306, the overlapping area determination unit 15 checks whether there is a corresponding one for all of the foreground parts when the foreground parts are associated between the cameras. If there is a corresponding one for all of the foreground parts (YES in step S306), the process proceeds to step S309, and if there is no corresponding one for a part of the foreground parts (e.g., the second foreground part F12 and the other second foreground part F22 in FIG. 9A) (NO in step S306), the process proceeds to step S307.
[0052] In step S307, overlapping region determination unit 15 determines whether or not the foreground portion with no correspondence in step S306 (e.g., another second foreground portion F22 in FIG. 9(A)) is included in another foreground portion of the same image (e.g., the second foreground portion F12 in FIG. 9(A)). If the foreground portion with no correspondence (e.g., the other second foreground portion F22 in FIG. 9(A)) is included in another foreground portion of the same image (e.g., the second foreground portion F12 in FIG. 9(A)) (YES in step S307), the process proceeds to step S308, and if the foreground portion with no correspondence (e.g., F22 in FIG. 9(A)) is not included in another foreground portion (NO in step S307), the process proceeds to step S309.
[0053] In step S308, if an uncorresponding foreground part (for example, the other second foreground part F22 in FIG. 9(A)) is not included in another foreground part (for example, the second foreground part F12 in FIG. 9(A)), the overlapping area determination unit 15 determines that the uncorresponding foreground part (for example, the other second foreground part F22 in FIG. 9(A)) has been combined in the foreground part on the other camera side (for example, the first foreground part F11 and the other first foreground part F21 in FIG. 9(A)), and accordingly integrates the foreground parts (for example, the second foreground part F12 and the other second foreground part F22 in FIG. 9(A)).
[0054] Specifically, in the case of Figure 9 (A), the overlap area determination unit 15 detects one foreground portion (one foreground portion consisting of a first foreground portion F11 and another first foreground portion F21) from the image of the imaging device 20a, detects a second foreground portion F12 and another second foreground portion F22 from the image of the imaging device 20b, and determines that there is no corresponding foreground portion in one of the second foreground portion F12 and the other second foreground portion F22. Therefore, the overlap area determination unit 15 determines whether the second foreground portion F12 and the other second foreground portion F22 (e.g., the coordinates of the feet) are included in each foreground portion on the imaging device 20a side (in this example, one foreground portion consisting of the first foreground portion F11 and the other first foreground portion F21), and if included, integrates the "second foreground portion F12 of the image captured by the imaging device 20b" and the "other second foreground portion F22 of the image captured by the imaging device 20b."
[0055] In step S309, the image processing unit (i.e., the image of the foreground portion with little deformation) performs the processes of steps S310 to S316 on all the foreground portions that have been classified, associated, and integrated, and performs image synthesis in the overlapping area Ov.
[0056] According to the third embodiment, it is possible to generate a composite image Ic that does not display objects in the overlapping area Ov as double images. Also, according to the third embodiment, it is possible to cause the display device 30 to properly display objects in the overlapping area Ov, as in the cases of the first and second embodiments, and it is also possible to properly display the foreground portion (i.e., an image of the foreground portion with little distortion) even when the foreground portion is overlapped.
[0057] Except for the above, the third embodiment is the same as the first or second embodiment. [Explanation of symbols]
[0058] 10 image synthesis device, 11 image receiving unit, 12 parameter input unit, 13 image processing unit, 14 foreground extraction unit, 15 overlapping area determination unit, 16 display processing unit, 20a, 20b imaging device, 30 display device, B1, B2 bounding box, Ia first image (camera image), Ib second image (camera image), Ic synthesized image, F1, F11 first foreground part (foreground part), F2, F12 second foreground part (foreground part), F21 other first foreground part (foreground part), F22 other second foreground part (foreground part), Ov overlapping area.
Claims
1. an image receiving unit that receives a first image and a second image captured from different positions; a foreground extraction unit that extracts a first foreground in the first image and a second foreground in the second image; an overlapping area determination unit that associates a first foreground portion of the first foreground and a second foreground portion of the second foreground, the first and second foreground portions indicating the same object, in an overlapping area between the first and second images; an image processing unit that performs a selection process to select either the first foreground portion or the second foreground portion as a foreground portion to be displayed in the overlapping area based on sizes of the first foreground portion and the second foreground portion when the first foreground portion and the second foreground portion indicating the same object are present, and a synthesis process to display the foreground portion selected in the selection process in an image obtained by synthesizing the first image and the second image, thereby generating a synthetic image; having In the selection process, the image processing unit when the first foreground portion and the second foreground portion indicating the same object are present and neither the first foreground portion nor the second foreground portion is in contact with the boundary of the overlapping region, selecting either the first foreground portion or the second foreground portion as the foreground portion based on sizes of the first foreground portion and the second foreground portion; When another first foreground portion and another second foreground portion showing another object are present, and one of the other first foreground portion and the other second foreground portion does not contact the boundary of the overlapping region, and the other contacts the boundary of the overlapping region, the one foreground portion is selected.
1. A video synthesizing device comprising:
2. In the selection process, the image processing unit uses, as the size, a length of the first foreground portion and the second foreground portion or an area of a bounding box of the first foreground portion and the second foreground portion.
2. The image synthesizing apparatus according to claim 1.
3. an image receiving unit that receives a first image and a second image captured from different positions; a foreground extraction unit that extracts a first foreground in the first image and a second foreground in the second image; an overlapping area determination unit that associates a first foreground portion of the first foreground and a second foreground portion of the second foreground, the first and second foreground portions indicating the same object, in an overlapping area between the first and second images; an image processing unit that performs a selection process to select either the first foreground portion or the second foreground portion as a foreground portion to be displayed in the overlapping area based on sizes of the first foreground portion and the second foreground portion when the first foreground portion and the second foreground portion indicating the same object are present, and a synthesis process to display the foreground portion selected in the selection process in an image obtained by synthesizing the first image and the second image, thereby generating a synthetic image; having The overlapping region determination unit If there is a first foreground part that does not correspond to any of the second foreground parts among the plurality of first foreground parts, the first foreground part that does not correspond to any of the second foreground parts and the other first foreground parts are integrated into one foreground part; When there is a second foreground portion that does not correspond to any of the first foreground portions among the plurality of second foreground portions, the second foreground portion that does not correspond to any of the first foreground portions and the other second foreground portions are integrated into one foreground portion.
1. A video synthesizing device comprising:
4. In the selection process, the image processing unit uses, as the size, a length of the first foreground portion and the second foreground portion or an area of a bounding box of the first foreground portion and the second foreground portion.
4. The image synthesizing apparatus according to claim 3.
5. In the selection process, the image processing unit selects a smaller foreground portion out of the first foreground portion or the second foreground portion.
5. The image synthesizing apparatus according to claim 3 or 4.
6. In the selection process, the image processing unit does not display a larger foreground portion of the first foreground portion or the second foreground portion in the composite image.
6. The image synthesizing apparatus according to claim 5.
7. In the selection process, the image processing unit when the first foreground portion and the second foreground portion indicating the same object are present and neither the first foreground portion nor the second foreground portion is in contact with the boundary of the overlapping region, selecting either the first foreground portion or the second foreground portion as the foreground portion based on sizes of the first foreground portion and the second foreground portion; When another first foreground portion and another second foreground portion showing another object are present, and one of the other first foreground portion and the other second foreground portion does not contact the boundary of the overlapping region, and the other contacts the boundary of the overlapping region, the one foreground portion is selected.
5. The image synthesizing apparatus according to claim 3 or 4.
8. When the object is a person, the overlapping area determination unit associates the first foreground part with the second foreground part based on coordinates of a position of the person's feet in the first foreground part and coordinates of a position of the person's feet in the second foreground part.
8. The image synthesizing apparatus according to claim 7.
9. 1. A video compositing method executed by a video compositing device, comprising: receiving a first image and a second image taken from different positions; extracting a first foreground in the first image and a second foreground in the second image; Correlating a first foreground portion of the first foreground and a second foreground portion of the second foreground, the first and second foreground portions indicating the same object, in an overlapping area between the first and second images; a step of performing a selection process of selecting either the first foreground portion or the second foreground portion as a foreground portion to be displayed in the overlapping area based on sizes of the first foreground portion and the second foreground portion when the first foreground portion and the second foreground portion indicating the same object are present, and a synthesis process of displaying the foreground portion selected in the selection process in an image obtained by synthesizing the first image and the second image, thereby generating a synthetic image; having The step of generating the composite image includes the steps of: when the first foreground portion and the second foreground portion indicating the same object are present and neither the first foreground portion nor the second foreground portion is in contact with the boundary of the overlapping region, selecting either the first foreground portion or the second foreground portion as the foreground portion based on sizes of the first foreground portion and the second foreground portion; When another first foreground portion and another second foreground portion showing another object are present, and one of the other first foreground portion and the other second foreground portion does not contact the boundary of the overlapping region, and the other contacts the boundary of the overlapping region, the one foreground portion is selected.
13. A video compositing method comprising:
10. On the computer, receiving a first image and a second image taken from different positions; extracting a first foreground in the first image and a second foreground in the second image; Correlating a first foreground portion of the first foreground and a second foreground portion of the second foreground, the first and second foreground portions indicating the same object, in an overlapping area between the first and second images; a step of performing a selection process of selecting either the first foreground portion or the second foreground portion as a foreground portion to be displayed in the overlapping area based on sizes of the first foreground portion and the second foreground portion when the first foreground portion and the second foreground portion indicating the same object are present, and a synthesis process of displaying the foreground portion selected in the selection process in an image obtained by synthesizing the first image and the second image, thereby generating a synthetic image; Run the command, The step of generating the composite image includes the steps of: when the first foreground portion and the second foreground portion indicating the same object are present and neither the first foreground portion nor the second foreground portion is in contact with the boundary of the overlapping region, selecting either the first foreground portion or the second foreground portion as the foreground portion based on sizes of the first foreground portion and the second foreground portion; When another first foreground portion and another second foreground portion showing another object are present, and one of the other first foreground portion and the other second foreground portion does not contact the boundary of the overlapping region, and the other contacts the boundary of the overlapping region, the one foreground portion is selected. A video compositing program comprising:
Citation Information
Patent Citations
Image processing device
JP2010250640A
Monitoring camera terminal
JP2011193187A
Image processing device, image processing method and computer program
JP2017092543A
Imaging apparatus, control method of the same, and program
JP2019201325A
Apparatus and method for displaying panorama image of vehicle
KR1020160139606A
