Image processing apparatus and image processing method

Abstract

To reduce a sense of discomfort of a joint portion when images are overlapped and joined together.SOLUTION: An image processing apparatus includes: first acquisition means of acquiring a first image; second acquisition means of acquiring a second image representing a background of the first image; and correction means of performing correction based on the second image with respect to an object region representing a region of an object having a short distance from an imaging position. The object region is present in an overlapping portion of the first image when performing stitch processing by connecting the first image with another image at the overlapping portion.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to image processing. [Background technology]

[0002] In recent years, systems that stitch together images captured in multiple directions to generate a wide-angle image (stitched image) have become popular. Stitching is a process of overlapping and joining images. One method of capturing images for stitching is to install multiple cameras and capture images in multiple directions simultaneously. Patent Document 1 describes a method of geometrically transforming images using a homography matrix calculated so that images of nearby objects in images captured by multiple cameras at different positions match, and stitching the images using the overlapping portions where the images of nearby objects match. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-189576 Summary of the Invention [Problem to be solved by the invention]

[0004] In a method of taking images using multiple cameras, the multiple cameras cannot be installed at the same nodal point, so the images of objects that are close to the imaging position at the seams do not match between the images, and the seams can appear unnatural. In the method described in Patent Document 1, a shift occurs in distant scenes in order to prevent the images of close objects from being distorted.

[0005] Therefore, an object of the present invention is to reduce the sense of incongruity at the seam when overlapping and joining images. [Means for solving the problem]

[0006] The image processing device of the present invention is characterized by having a first acquisition means for acquiring a captured image, a second acquisition means for acquiring a background image representing the background of the captured image, and a correction means for performing correction based on the background image on an object area representing an object area that is close to the imaging position and exists in the overlapping area of ​​the captured image when stitching the captured image together with another image at the overlapping area. [Effects of the Invention]

[0007] According to the present invention, when images are superimposed and joined together, it is possible to reduce the sense of incongruity that appears at the joint. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of the internal configuration of an image processing device. [Figure 2] 4 is a flowchart showing main processing of the image processing device. [Figure 3] FIG. 10 is a diagram illustrating a method for correcting an image. [Figure 4] 10 is a flowchart showing a distance image generation process. [Figure 5] FIG. 1 is a diagram illustrating the appearance of an imaging device. [Figure 6] 10 is a flowchart showing an image storage process. [Figure 7] FIG. 10 is a diagram illustrating a first modified example. [Figure 8] FIG. 10 is a diagram illustrating a second modified example. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] In this embodiment, a method for correcting an image that is the target of a process (stitching process) in which overlapping portions of images captured in a plurality of different directions are superimposed and stitched together will be described.

[0011] FIG. 1 shows an example of the internal configuration of an image processing device. 1(a) shows an example of the hardware configuration of an image processing device 100. The image processing device 100 has a CPU 101, a non-volatile memory 102, a memory 103, and an interface 104. These components are connected to each other via a bus 110. In this embodiment, the image processing device 100 is an information processing device such as a PC (personal computer). Note that the image processing device may be configured integrally with an imaging device.

[0012] The CPU (Central Processing Unit) 101 controls the entire image processing device 100. The non-volatile memory 102 is a ROM (Read Only Memory), an HDD (Hard Disk Drive), or the like, and stores programs and various data. The memory 103 is a RAM (Random Access Memory), or the like, and temporarily stores acquired images and corrected images. The memory 103 also functions as a work area for the CPU 101. The interface 104 is an interface for connecting to an external device. The interface 104 is an interface that supports at least one of wired communication and wireless communication, and communicates with the external device via a communication path that corresponds to the communication mode used.

[0013] 1(b) shows an example of the functional configuration of the image processing device 100. The image processing device 100 functions as an image acquisition unit 111, an image storage unit 112, an image correction unit 113, and an image output unit 114 by the CPU 101 reading out a program stored in the non-volatile memory 102 or the like into the memory 103 and executing the program.

[0014] The image acquisition unit 111 acquires captured images from an imaging device or an external storage device via the interface 104. The captured images acquired by the image acquisition unit 111 are subject to stitching processing. In order to output the stitched images as a video, the image acquisition unit 111 may acquire the captured images in chronological order. Hereinafter, the captured images acquired by the image acquisition unit 111 will be referred to as first images. The image storage unit 112 stores a background image that represents the background of the first image. In this embodiment, a captured image previously acquired by the image acquisition unit 111, which has the same angle of view as the first image and does not include an object that is close to the image capture position, is stored as the second image in the non-volatile memory 102 or the like.

[0015] The image correction unit 113 corrects the first image by performing a predetermined process on an area of ​​an object that is close to the imaging position in the first image (hereinafter referred to as a short-distance object) and combining the area with the second image. The image output unit 114 outputs the corrected image corrected by the image correction unit 113 to a device that performs stitching processing. Furthermore, if no close-range object is present in the first image, the image output unit 114 may output the first image as is to the device. Note that if the image processing device further has a stitch processing unit that performs stitching processing, the image output from the image output unit 114 may be input to the stitch processing unit.

[0016] <Main processing of image processing device> Next, the main processing of the image processing device 100 according to this embodiment will be described with reference to Figures 2 and 3. The main processing is processing for correcting an image (first image) that is the target of stitching processing. Figure 2 is a flowchart showing the main processing of the image processing device 100 according to this embodiment. The CPU 101 reads a program stored in the non-volatile memory 102 or the like into the memory 103 and executes it, thereby realizing the processing of this flowchart. Hereinafter, each step (process) will be represented by adding an S before the reference number.

[0017] In S201, the CPU 101 acquires an image (first image) to be subjected to stitching processing from an imaging device. FIG. 3(a) shows an example of the first image. The first image 311 shown in FIG. 3(a) shows people 301 and 302 with the wall of a room in the background. The person 301 is located near the center of the first image 311, and the person 302 is located near the right edge of the first image 311. If the image acquired from the imaging device is a fisheye lens image, the CPU 101 may perform processes such as distortion correction and projective transformation on the image.

[0018] In S202, CPU 101 acquires a first distance image, which is a distance image of the first image. FIG. 3B shows an example of the first distance image. In first distance image 312 shown in FIG. 3B, areas close to the imaging position are indicated in bright colors, and areas farther away are indicated in dark colors. First distance image 312 is a distance image of first image 311. People 321 and 322 in first distance image 312 correspond to people 301 and 302 in first image 311. Since the areas of people 321 and 322 are indicated in white, it can be seen that people 301 and 302 in first image 311 are located close to the imaging position. A method for acquiring the first distance image will be described later with reference to FIG. 4.

[0019] In S203, CPU 101 separates objects in the first image from the background and detects areas of objects that are close to the imaging position in the first image (short-distance objects) based on the first distance image. Here, areas of person 301 and person 302 in first image 311 are detected.

[0020] For example, CPU 101 detects objects that are close to the image capture position by binarizing the first distance image so that objects that are close to the image capture position are included. The binarization threshold is set so that objects that are close to the image capture position are included. The binarization threshold depends on the distance between the nodal points of the cameras that captured the images to be stitched. For example, if the distance between the nodal points is 2 centimeters, the threshold can be set to 1 meter.

[0021] Note that because distance images generally have low resolution, close-range objects may not be detected correctly. Therefore, CPU 101 may perform noise reduction processing, such as median filtering or blurring, on the first distance image. This lowers the threshold value, making it possible to detect areas that are slightly larger than the actual size. Note that the processing of S203 may be performed simultaneously with the processing of S205 (later described).

[0022] In S204, CPU 101 determines whether the object area detected in S203 exists in the overlapping portion of the first image when performing a process of overlapping and stitching the first image with another image (stitching process). If CPU 101 determines that the object area exists in the overlapping portion, the process proceeds to S205, and if CPU 101 determines that the object area does not exist in the overlapping portion, the process proceeds to S208. Here, the vicinity of the right edge of first image 311 corresponds to the overlapping portion.

[0023] In the method of acquiring the first distance image shown in FIG. 4, a first distance image is generated that represents the distance in the overlapping portion between the first image and another image to be stitched together. Since S203 detects a nearby object based on the first distance image, it is assumed that the detected object area exists in the overlapping portion. Therefore, the determination process of S204 may be skipped. On the other hand, as shown in Modification 2 described below, the determination process of S204 is required when detecting a nearby object based on a first distance image 312 of the entire area of ​​the first image, as shown in FIG. 3(b), acquired using an external device such as a LiDAR.

[0024] In S205, the CPU 101 determines an extracted portion of the first image to be processed in the subsequent step S207. In this embodiment, the region of a close-range object present in the overlapping portion is extracted. FIG. 3(c) shows an example of an extracted portion in the first image 311. In the image 313 shown in FIG. 3(c), only a region 331 of a person 302 present near the right edge of the first image 311 is extracted, and the person 301 present near the center is not extracted.

[0025] In S206, the CPU 101 acquires an image (second image) representing the background of the first image from the non-volatile memory 102 or the like. In this embodiment, the second image is acquired from the imaging device that acquired the first image, is captured at the same angle of view as the first image, and does not show any nearby objects. FIG. 3(d) shows an example of the second image. The second image 314 shown in FIG. 3(d) shows only the wall of the room, and does not show the person who was in the foreground.

[0026] In S207, CPU 101 performs a predetermined process on the extracted portion of the first image determined in S205 and combines it with the second image to generate a corrected image. The process of this flowchart then ends. CPU 101 then inputs this corrected image to a device that performs stitching. This makes it possible to make the misalignment of people at close range at the seams less noticeable in the wide-angle image obtained by stitching.

[0027] As a method for combining the first image and the second image, for example, the CPU 101 averages the extracted portion of the first image determined in S205 and the portion of the second image corresponding to the extracted portion. For example, the CPU 101 sets a predetermined transmittance value for the extracted portion of the first image and superimposes it on the second image. Note that the transmittance is not particularly limited. For example, the CPU 101 increases the transmittance as the distance from the imaging position decreases. This allows the object to become gradually more transparent as it moves and gradually approaches the imaging position.

[0028] Fig. 3(e) shows an example of a corrected image. In the corrected image 315 shown in Fig. 3(e), an area 342 corresponding to the person 302 in the first image 311 is made semi-transparent and is superimposed on the second image 314. This allows the room in the background to be seen through the area 342. Note that the person 341 in the corrected image 315 corresponds directly to the person 301 in the first image 311. Fig. 3(f) shows another example of a corrected image. In the corrected image 316 shown in Fig. 3(f), the area corresponding to the person 302 in the first image 311 is replaced with the corresponding area in the second image 314. Note that the person 351 in the corrected image 316 corresponds directly to the person 301 in the first image 311.

[0029] In S208, the CPU 101 updates the second image stored in the non-volatile memory 102 or the like with the first image acquired in S201 so that the first image can be used in the processing of the flowchart in Fig. 2 that is executed next and thereafter. Details of the processing for updating the second image will be described later with reference to Fig. 6. Thereafter, this flowchart ends, and the CPU 101 inputs the first image acquired in S201 to a device that performs stitching processing without correcting it.

[0030] <Distance image generation processing> Next, the distance image generation process executed in S202 will be described with reference to the flowchart of FIG. In S401, the CPU 101 acquires a first image from the imaging device. Here, the acquired image is the same as the first image acquired in S201. In S402, the CPU 101 acquires an image (third image) captured in a direction different from the first image from the imaging device. The first image and the third image have overlapping portions. In the stitching process, the overlapping portions of the first image and the third image are superimposed and stitched together to generate a stitched image.

[0031] FIG. 5 shows the appearance of an imaging device that captures the first and third images. The imaging device includes a lens 501 and a lens 502. In this embodiment, the first image is captured by the lens 501, and the third image is captured by the lens 502. The lens 501 faces left in the drawing, and the lens 502 faces in the opposite direction from the lens 501. The lenses 501 and 502 are fisheye lenses with a horizontal angle of view of 180° or more. Therefore, even though the lenses face in opposite directions, they can capture overlapping portions.

[0032] For example, if the horizontal angle of view is 190°, there will be a 10° overlap in the horizontal direction (towards the rear and towards the front in the drawing). However, because lenses 501 and 502 capture images in different directions, perspective conflict will occur, and the images of close-up objects captured in the overlapping area will not match between the images, resulting in misalignment when the images are stitched together. This is particularly noticeable when a fisheye lens is used. Therefore, in this embodiment, correction is performed on the images before stitching to make the misalignment of close-up objects less noticeable.

[0033] In this embodiment, images captured by an imaging device equipped with multiple lenses are the subject of stitching processing, but images captured in multiple imaging directions by a single imaging device may also be the subject of stitching processing. Furthermore, in this embodiment, a stitched image is generated from two images captured by an imaging device equipped with two lenses, but a stitched image may also be generated from three or more images captured by an imaging device equipped with three or more lenses. When generating a stitched image from images captured by an imaging device equipped with three or more lenses, correction may be performed on the overlapping portions of each image.

[0034] In S403, the CPU 101 performs distortion correction on the first image acquired in S401 and the third image acquired in S402, and performs projective transformation in the same direction to generate a first projectively transformed image and a third projectively transformed image. Projective transformation methods that can be used include perspective projection, cylindrical transformation, and equirectangular transformation.

[0035] The processing from S404 to S406, which will be described next, is processing for generating a distance image (parallax image) by stereo matching processing, and StereoBM of OpenCV or the like is used. In S404, the CPU 101 detects feature points in the first projectively transformed image and the third projectively transformed image generated in S403. SIFT, FAST, or the like is used for the feature point detection. In S405, the CPU 101 performs matching of the feature points detected in S404. In S406, CPU 101 generates a first distance image from the distances between the feature points matched in S405. This completes the processing of this flowchart.

[0036] In this embodiment, the third image is an image captured in the opposite direction to the first image at a horizontal angle of view of 180° or more, but this is not limiting. The first distance image may be a distance image of the entire area of ​​the first image, or a distance image of the overlapping portion between the first and third images. Furthermore, because stereo matching of the overlapping portion is also a necessary step in the stitching process, some or all of the processing shown in the flowchart of FIG. 4 may be incorporated as part of the stitching process.

[0037] <Image retention processing> Next, the image holding process executed in S208 will be described. Fig. 6 is a flowchart showing the image holding process. The process shown in Fig. 6 is a process for holding images that were previously captured and do not include close-range objects. In S601, the CPU 101 acquires a first image from the imaging device. Here, the acquired image is the same as the first image acquired in S201. In S602, CPU 101 acquires a first distance image, which is the same as the first distance image acquired in S202. In S603, the CPU 101 reads and acquires the second image from the non-volatile memory 102 etc. The processing in this step is similar to that in S207.

[0038] In S604, the CPU 101 reads and acquires the second distance image from the non-volatile memory 102 or the like. Fig. 3(g) shows an example of the second distance image. In the second distance image 318 shown in Fig. 3(g), areas that are close to the imaging position are shown in light colors, and areas that are farther away are shown in dark colors. The second distance image 318 is a distance image of the second image 314. In S605, CPU 101 compares the size of the area of ​​the close-range object between the first distance image acquired in S602 and the second distance image acquired in S604. Specifically, the average pixel value of the first distance image is compared with the average pixel value of the second distance image.

[0039] In S606, CPU 101 determines whether the area of ​​close-range objects is smaller in the first distance image or the second distance image. If the area of ​​close-range objects is smaller in the first distance image, the process proceeds to S606. If the area of ​​close-range objects is smaller in the second distance image, the process ends. In S607, CPU 101 stores the first image acquired in S601 and the first distance image acquired in S602 as the second image and second distance image. That is, CPU 101 updates the second image and second distance image with the newly acquired first image and first distance image. Then, the processing of this flowchart ends.

[0040] When first images are acquired in chronological order by a video camera or the like, the flowchart of Fig. 6 allows the second image to be updated with a newly acquired first image at any time, even if lighting conditions, etc., change. Note that, in an environment where lighting conditions, etc., do not change, a captured image obtained by shooting when there is no object in the close range at the start of shooting may be stored as the second image and continue to be stored without updating. In this case, the image storage process of S208 may be skipped. Furthermore, since the flowchart of Fig. 6 is a process that is executed when it is determined in S204 that there is no object in the close range in the overlapping portion, CPU 101 may skip the processes of S601 to S606 and execute only the process of S607.

[0041] According to the present embodiment described above, when performing stitching processing, objects that are present at the seam and are close to the imaging position are made translucent or transparent, thereby reducing the sense of incongruity at the seam of the stitched image.

[0042] [Variation 1] In the above explanation, when combining images in S207, the transmittance of the extracted portion is set to a predetermined value. However, in this modified example, a method of generating an alpha channel image in which the alpha value decreases as the distance from the imaging position decreases will be described. In S206, CPU 101 generates an alpha channel image from the first distance image to be alpha blended in S207. Alpha blending is a process in which the first image and the second image are superimposed and combined based on the alpha value set for each pixel. CPU 101 generates an alpha channel image in which the proportion of the first image mixed increases as the value decreases (farther away) in the first distance image.

[0043] For example, an alpha channel image is generated for each pixel in the first distance image using a formula such as 100% - pixel value of the first distance image divided by 2. Here, the pixel value of the first distance image is 100% at the largest (closest) part and 0% at the smallest (farthest) part. That is, according to the above formula, the proportion of the first image mixed in the closest part of the first distance image is 50%, and the proportion of the first image mixed in the farthest part of the first distance image is 100%. FIG. 7(a) shows an example of an alpha channel image generated by applying the above formula to the first distance image 312 in FIG. 3(b). In the alpha channel image 701 shown in FIG. 7(a), the alpha value in the areas of people 321 and 322 is 50%, and the alpha value increases toward the back of the room, reaching 100% at the very back. Note that because person 321 exists outside the stitching joints, the alpha value of person 321 may be set to be greater than the alpha value of person 322, making person 322 more transparent. An alpha channel image 701 shown in FIG. 7(a) represents the proportion of the first image 311 to be mixed. An alpha channel image 702 shown in FIG. 7(b) represents the proportion of the second image 314 to be mixed. For each pixel of the images in FIGS. 7(a) and 7(b), the greater the proportion of mixing, the brighter the color.

[0044] At the seams of stitching, the closer an object is to the imaging position, the greater the misalignment between the images, making it more likely that an unnatural appearance will occur. According to this modification, the closer the object is to the imaging position, the greater the transparency, making it possible to make the unnatural appearance less noticeable. Furthermore, as the object moves and gradually approaches the imaging position, the transparency can be gradually reduced, thereby reducing the unnatural appearance caused by sudden changes.

[0045] [Variation 2] In the above description, transmittance was set only for close-range objects present at stitching seams. However, in this modified example, a method of setting transmittance also for close-range objects present at areas other than stitching seams will be described. In this modified example, to obtain a distance image for the entire area of ​​the first image, the distance image is obtained from, for example, an external device such as a LiDAR (light detection and ranging) or an on-chip phase-difference AF (autofocus) device. In this modified example, transmittance is set also for close-range objects present at areas other than stitching seams, thereby generating a corrected image 801 as shown in FIG. 8. In corrected image 801 shown in FIG. 8, regions 802 and 803 corresponding to people 301 and 302 in first image 311 are made semi-transparent and superimposed on second image 314. Note that CPU 101 may change the transmittance for the close-range objects in the first image depending on the positional relationship between the close-range objects and the stitching seams. For example, the area of ​​a nearby object may be blurred in stages as the object approaches the seam.

[0046] A method for generating a distance image for the entire area of ​​the first image may be to estimate the distance from the imaging position to the object based on the size of the object detected by template matching such as face detection in the first image. Alternatively, a method for estimating the distance using deep learning-based image processing may be used. Alternatively, a distance image may be generated by stereo matching processing using multiple cameras arranged so that all parts of the first image overlap.

[0047] According to this modification, moving objects in a video become semi-transparent even in scenes other than when they pass through the joints of stitched images, thereby reducing the overall sense of incongruity.

[0048] [Variation 3] Furthermore, the sense of incongruity caused by misalignment between images at the seams of stitching may vary depending on the type of object. For example, a sense of incongruity is likely to occur when a human face is present at the seam. Therefore, in this modification, the CPU 101 may control the transmittance of the area of ​​the object to be changed depending on the type of the close-up object detected in S203. Specifically, if the close-up object detected in S203 is a human face, the transmittance of the area of ​​the person may be set higher than normal so that the background can be more easily seen through. For example, the area of ​​the person may be made transparent.

[0049] The present invention can also be realized by providing a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. The above-described embodiments are merely illustrative examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0050] The disclosure of each of the above-described embodiments includes the following configurations, methods, and programs. (Configuration 1) a first acquisition means for acquiring a captured image; a second acquisition means for acquiring a background image representing a background of the captured image; a correction means for correcting, based on the background image, an object region that represents an object region that is close to an imaging position and exists in the overlapping portion of the captured image when stitching the captured image together with another image at the overlapping portion; 1. An image processing device comprising: (Configuration 2) The image processing device according to configuration 1, wherein the correction means performs correction on the object region that exists in the overlapping portion in the captured image, and does not perform correction on the object region that exists outside the overlapping portion in the captured image. (Configuration 3) 3. The image processing device according to claim 1, wherein the correction means makes the object region in the captured image semitransparent and superimposes it on the background image. (Configuration 4) 3. The image processing device according to configuration 1 or 2, wherein the correction means replaces the object region in the captured image with a corresponding region in the background image. (Configuration 5) The image processing device according to configuration 1, wherein the correction means performs correction on the object region that exists in the overlapping portion in the captured image as well as on the object region that exists outside the overlapping portion in the captured image. (Configuration 6) Further, the image processing device includes a storage means for storing the background image. the first acquisition means acquires captured images in time series, 6. The image processing device according to any one of configurations 1 to 5, wherein the storage means updates the background image using a captured image newly acquired by the first acquisition means. (Configuration 7) The image processing device according to configuration 6, wherein the holding means updates the background image when an area in the captured image newly acquired by the first acquisition means that is closer to the imaging position is smaller than an area in the background image that is closer to the imaging position. (Configuration 8) a third generation means for generating a distance image representing a distance from an imaging position in the overlapping portion of the captured image using the other image; 8. The image processing device according to any one of configurations 1 to 7, wherein the correction means detects the area of ​​the object at a short distance in the captured image based on the distance image. (Configuration 9) 9. The image processing device according to configuration 8, wherein the third generating means generates the distance image by stereo matching. (Configuration 10) a third acquisition means for acquiring a distance image representing a distance from an imaging position in the entire area of ​​the captured image; 8. The image processing device according to any one of configurations 1 to 7, wherein the correction means detects the area of ​​the object at a short distance in the captured image based on the distance image. (Configuration 11) 11. The image processing device according to claim 10, wherein the third acquisition means acquires the distance image from a LiDAR or an autofocus device. (Configuration 12) The image processing device according to any one of configurations 1 to 7, wherein the correction means generates an alpha channel image based on the distance in the captured image, and synthesizes the captured image and the background image based on the alpha channel image. (Configuration 13) 4. The image processing device according to configuration 3, wherein the correction means changes the transmittance of the object region in accordance with the distance from the imaging position. (Configuration 14) 4. The image processing device according to configuration 3, wherein the correction means changes the transmittance of the object region based on the type of object that is close to the imaging position. (Configuration 15) 4. The image processing device according to configuration 3, wherein the correction means changes the transmittance of the object region in accordance with the positional relationship between the object region and the overlapping portion. (method) a first acquisition step of acquiring a captured image; a second acquisition step of acquiring a background image representing a background of the captured image; a correction step of correcting, based on the background image, an object region that represents a region of an object that is close to an imaging position and exists in the overlapping portion of the captured image when stitching the captured image together with another image at the overlapping portion; An image processing method comprising: (program) A program for causing a computer to function as the image processing device according to any one of configurations 1 to 15.

Claims

1. a first acquisition means for acquiring a captured image; a second acquisition means for acquiring a background image representing a background of the captured image; a correction means for correcting, based on the background image, an object region that represents an object region that is close to an imaging position and exists in the overlapping portion of the captured image when stitching the captured image together with another image at the overlapping portion; 1. An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the correction means performs correction on the object area that exists in the overlapping portion in the captured image, and does not perform correction on the object area that exists outside the overlapping portion in the captured image.

3. 2. The image processing apparatus according to claim 1, wherein the correction means makes the object area in the captured image semitransparent and superimposes it on the background image.

4. 2. The image processing apparatus according to claim 1, wherein the correction means replaces the object region in the captured image with a corresponding region in the background image.

5. The image processing device according to claim 1 , wherein the correction means performs correction on the object region that exists in the overlapping portion in the captured image as well as on the object region that exists outside the overlapping portion in the captured image.

6. Further, the image processing device includes a storage means for storing the background image. the first acquisition means acquires captured images in time series, 2. The image processing apparatus according to claim 1, wherein the storage unit updates the background image using a captured image newly acquired by the first acquisition unit.

7. The image processing device according to claim 6, characterized in that the storage means updates the background image when an area in the captured image newly acquired by the first acquisition means that is closer to the imaging position is smaller than an area in the background image that is closer to the imaging position.

8. a third generation means for generating a distance image representing a distance from the imaging position in the overlapping portion of the captured image using the other image, 2. The image processing apparatus according to claim 1, wherein the correction means detects the area of ​​the object located at a short distance in the captured image based on the distance image.

9. 9. The image processing apparatus according to claim 8, wherein the third generating means generates the distance image by stereo matching.

10. a third acquisition means for acquiring a distance image representing a distance from an imaging position in the entire area of ​​the captured image; 2. The image processing apparatus according to claim 1, wherein the correction means detects the area of ​​the object located at a short distance in the captured image based on the distance image.

11. The image processing device according to claim 10 , wherein the third acquisition means acquires the distance image from a LiDAR or an autofocus device.

12. 2. The image processing device according to claim 1, wherein the correction means generates an alpha channel image based on the distance in the captured image, and combines the captured image with the background image based on the alpha channel image.

13. 4. The image processing apparatus according to claim 3, wherein the correction means changes the transmittance of the object region in accordance with the distance from the image capturing position.

14. 4. The image processing apparatus according to claim 3, wherein the correction means changes the transmittance of the object region based on the type of object located close to the imaging position.

15. 4. The image processing apparatus according to claim 3, wherein the correction means changes the transmittance of the object region in accordance with the positional relationship between the object region and the overlapping portion.

16. a first acquisition step of acquiring a captured image; a second acquisition step of acquiring a background image representing a background of the captured image; a correction step of correcting, based on the background image, an object region that represents a region of an object that is close to an imaging position and exists in the overlapping portion of the captured image when stitching the captured image together with another image at the overlapping portion; An image processing method comprising:

17. A program for causing a computer to function as the image processing device according to claim 1.

Images