Image processing device, image processing method, and program

Abstract

Because the reference image may lack sufficient feature points, it may not be possible to accurately detect the field of view shift. [Solution] The image processing device is an image processing device that generates a reference image for detecting angle of view shift, and includes input means for acquiring a first image generated by photographing a predetermined area and a second image generated by photographing an area that includes at least a part of the area after the first image has been photographed, composite area setting means for setting a composite area which is an area for updating the first image, and image updating means for updating the reference image by compositing the image of the composite area of ​​the second image with the first image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] When the posture of a camera changes due to the influence of wind or vibration, an angular deviation occurs in the captured image. To solve this problem, there is a method of extracting a plurality of feature points from a reference image captured in a reference camera posture, and estimating the angular deviation of the entire image from the amount of movement of the position of an area having similar features within the input image at the time of shooting and the extracted feature points.

[0003] In Patent Document 1, a method is disclosed in which a plurality of linear subject (also referred to as an object) contours existing in a captured image are extracted as patches, and high-precision alignment is performed using these plurality of patches between a reference image and a target image captured at different times. In the technique of Patent Document 1, since the patch cannot detect a movement parallel to the contour of the subject, the amount of change of the entire image is obtained by combining the amounts of movement detected by a plurality of patches in different directions extracted from the contour of the subject.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] To solve this problem, for example, the image processing apparatus of the present invention has the following configuration. That is, An image processing device that generates a reference image for detecting angle of view shift, An input means for acquiring a first image generated by photographing a predetermined region, and a second image generated by photographing a region that includes at least a part of the region after the first image has been photographed, A composite region setting means for setting a composite region which is an area for updating the first image, Image updating means for updating a reference image by combining the image of the composite region of the second image with the first image, It holds. [Effects of the Invention]

[0008] According to the present invention, the number of feature points in the reference image can be increased, enabling accurate detection of field-of-view shifts. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram showing the functions of the image processing apparatus according to the embodiment. [Figure 2] A block diagram showing the hardware configuration of an image processing device. [Figure 3] A diagram showing examples of reference images and additional input images. [Figure 4] A diagram showing a flowchart of the reference image update process performed by the image processing unit. [Figure 5] A diagram showing an example of a shape mask for an object mask. [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] <Embodiment> The embodiments will be described below with reference to the drawings. In this embodiment, a method for updating a portion of a reference image (also called a composite region) using one or more additional images that include changes in the object that occurred after the reference image was taken will be described. The reference image here is, for example, a reference image for the image that is the target of alignment. The additional images are, for example, images used to update a portion of the reference image, and are images taken after the reference image but before the image that is the target of alignment. In other words, the reference image is updated by the additional images before the time of alignment.

[0012] (Configuration of the image processing device) Figure 1 is a block diagram illustrating the functions of an image processing device 100 according to this embodiment. This embodiment relates to a technique for detecting and aligning the field of view shift of an input image using, for example, feature points of multiple images taken at different times. The image processing device 100 receives a reference image and additional images taken after the reference image, and outputs an updated reference image by compositing the composite region of the additional image onto the corresponding region of the reference image. The image processing device 100 may also detect the field of view shift and perform alignment based on the updated reference image and the input image taken during alignment. The reference image is an example of a first image. The additional image is an example of a second image. The additional image is an image taken of a region that includes at least a part of the region captured by the reference image.

[0013] The image processing apparatus 100 includes an input unit 101, an image analysis unit 102, a candidate region setting unit 103, a composite region setting unit 104, and an image update unit 105.

[0014] The input unit 101 acquires input images to be used as reference images and additional images from a photographing device or the like. The input unit 101 outputs the acquired input images to the image analysis unit 102, the image update unit 105, and the like.

[0015] The image analysis unit 102 determines whether an additional image can be combined with a reference image based on whether there is an angular deviation between the reference image and the additional image. The image analysis unit 102 outputs the additional image determined to be combinable to the candidate region setting unit 103.

[0016] The candidate region setting unit 103 detects objects in the reference image and the additional image. The candidate region setting unit 103 sets, as an update candidate region that is a candidate for the composite region, a region including the object based on the shape and region of the object and the like. The candidate region setting unit 103 obtains information regarding the shape and region of the detected object and the like, and outputs information regarding the shape and region of the detected object and the like and information on the update candidate region to the composite region setting unit 104.

[0017] The composite region setting unit 104 sets a composite region, which is a region to be combined with the reference image, in the additional image or the reference image. The composite region setting unit 104 sets the composite region using, for example, the relationship between the update candidate region, the object updated with the additional image, the shape of the object in the reference image, and the type of the object, and outputs information on the composite region to the image update unit 105. The type of the object may be determined using the means by which the shape of the object was detected and the result of classification by analysis of the object shape. The type of the object may include information on whether the object is an object whose position and shape change, together with a name such as a base, a person, or an antzuka. Note that the composite region setting unit 104 may set the composite region based on the dilated update candidate region in order to suppress false detection of feature points during alignment.

[0018] The image update unit 105 synthesizes and updates the image included in the composition area of the additional image with the reference image to generate an updated reference image. For example, the image update unit 105 synthesizes and updates the image cut out from the composition area of the additional image with the reference image to generate an updated reference image.

[0019] FIG. 2 is a block diagram showing the hardware configuration of the image processing apparatus 100. The hardware configuration of the image processing apparatus 100 will be described with reference to FIG. 2. The image processing apparatus 100 includes a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a display unit 205, an operation unit 206, a communication I / F 207, and a bus 208.

[0020] CPU201 stands for Central Processing Unit and is a processor. CPU201 controls the entire image processing device 100. CPU201 reads computer programs stored in ROM202 and auxiliary storage device 204, loads them into RAM203, and implements various functions. For example, by executing computer programs, CPU201 functions as an input unit 101, an image analysis unit 102, a candidate region setting unit 103, a composite region setting unit 104, and an image update unit 105. CPU201 also operates as a display control unit that controls the display unit 205 and an operation control unit that controls the operation unit 206. The image processing device 100 may have other processors such as an MPU (Micro Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), and QPU (Quantum Processing Unit) in place of or in addition to CPU201. The image processing device 100 may have one or more dedicated hardware components distinct from the CPU 201, and at least a portion of the functions and processing performed by the CPU 201 may be executed by the dedicated hardware. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors).

[0021] ROM202 stands for Read Only Memory and is a type of non-volatile memory. ROM202 is used to store programs and other data that do not require modification.

[0022] RAM203 stands for Random Access Memory and is a high-speed read and write memory. RAM203 temporarily stores data such as programs and the targets of program processing supplied from auxiliary storage device 204, as well as data supplied from external sources via communication interface 207. When the CPU 201 executes a program, RAM203 functions as a work area.

[0023] The auxiliary storage device 204 may be a large-capacity non-volatile storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive). The auxiliary storage device 204 stores various data such as computer programs executed by the CPU 201, data such as parameters necessary for the execution of computer programs, and image data and audio data that are processed by computer programs.

[0024] The display unit 205 may be, for example, a liquid crystal display and an LED (Light Emitting Diode). The display unit 205 displays a GUI (Graphical User Interface) or the like for the user to operate the image processing device 100.

[0025] The control unit 206 may be, for example, a keyboard, mouse, joystick, or touch panel. The control unit 206 receives operations from the user and inputs various instructions to the CPU 201.

[0026] The communication interface 207 is used for communication between the image processing device 100 and external devices. For example, if the image processing device 100 is connected to an external device by a wired connection, a cable for wired communication is connected to the communication interface 207. If the image processing device 100 has a function for wireless communication with an external device, the communication interface 207 is equipped with an antenna for wireless communication.

[0027] Bus 208 connects the various parts of the image processing device 100 to transmit information.

[0028] In this embodiment, the display unit 205 and the operation unit 206 are assumed to be located inside the image processing device 100, but at least one of the display unit 205 and the operation unit 206 may be located outside the image processing device 100 as a separate device.

[0029] (Reference image update process) Figure 3 shows examples of input images that will serve as a reference image and an additional image. Figure 4 shows a flowchart of the reference image update process executed by the image processing device 100. The reference image update process is executed by the CPU 201 reading a computer program stored in an auxiliary storage device 204 or the like. The operation of the image processing device 100 of the present invention will be shown below using each step in Figure 4 and the corresponding images in Figure 3.

[0030] Figure 3(A) is an example of a reference image taken before the baseball field was prepared. Reference image 300 includes images of objects such as white lines 303, letters 304, letters 305, cinder field 302, and net 301. Net 301 is an example of an object whose position and shape change. Bases and white lines are added to cinder field 302 during the preparation after the reference image was taken. Figure 3(B) is additional image 312 taken when base 313 was added to cinder field 302. Figure 3(C) is additional image 317 taken when white lines 318 were further added to cinder field 302. Base 313 and white lines 318 are examples of objects whose position and shape do not change. The image processing device 100 generates an updated reference image by combining a portion of the additional image with the reference image from the reference image 300 and the additional image 312 or additional image 317, in order to select feature points used for image-based angle shift correction and alignment.

[0031] In S401, the input unit 101 acquires an input image from the camera. The input image is the image used as the reference image 300 in Figure 3(A), the additional image 312 in Figure 3(B), and the additional image 317 in Figure 3(C).

[0032] In S402, the image analysis unit 102 determines whether a reference image has already been set. The image analysis unit 102 may determine whether a reference image has already been set by, for example, whether a reference image is stored in an auxiliary storage device 204. If the image analysis unit 102 has already set a reference image by storing it in the auxiliary storage device 204, it proceeds to S406. On the other hand, if the image analysis unit 102 has not set a reference image, it proceeds to S403.

[0033] In S403, the candidate region setting unit 103 performs object detection processing to detect objects from the reference image. The candidate region setting unit 103 analyzes the reference image 300 for color, frequency components, and contour shape, and describes an example of detecting objects by performing region division on the reference image. For example, the candidate region setting unit 103 performs region division so as not to include nets 301 containing meshes whose shape changes in the updated reference image. On the other hand, the candidate region setting unit 103 performs region division in order to include objects with a stable shape in the updated reference image.

[0034] In the example of the reference image 300 in Figure 3(a), the candidate area setting unit 103 detects the cinder 302, characters 305, and white lines 303 as objects to be included in the updated reference image by analyzing their color and shape, and detects the shape of these objects. The candidate area setting unit 103 does not recognize the characters 304 in the shape analysis of the reference image because the net 301 covers the characters 304.

[0035] On the other hand, the candidate region setting unit 103 distinguishes between the region of net 301 and regions other than net 301 based on the differences in the frequency components of the image of net 301. For example, the candidate region setting unit 103 may determine the region of net 301, which is an object not to be included in the updated reference image, based on whether the density of high-frequency components extracted by the high-pass filter exceeds a predetermined threshold, and then determine the region 306 from the shape of net 301.

[0036] In S404, the candidate area setting unit 103 sets update candidate areas based on objects extracted from the reference image 300. The update candidate areas are candidates for areas to be updated by compositing a part of the additional image onto the reference image. In the example of the reference image 300, the candidate area setting unit 103 uses the shape information of each object in the reference image, namely net 301, character 305, cinder 302, and white line 303, to generate a shape mask in which areas where these objects exist are set to 1 and other areas are set to 0. Figure 5 is an example of a shape mask of an object in the reference image generated by the candidate area setting unit 103. Shape mask 500 is a mask for cinder 302. Shape mask 503 is a mask for net 301. The candidate area setting unit 103 sets update candidate areas when compositing the reference image and the additional image using the shape mask, for example.

[0037] Depending on the object, the candidate area setting unit 103 may set the update candidate area using a bounding box surrounding the object instead of a shape mask. For example, the candidate area setting unit 103 may set the bounding box 310 surrounding the white line 303, the bounding box 307 surrounding the character 305, and the bounding box 309 surrounding the cinder 302 as update candidate areas. The candidate area setting unit 103 outputs information such as the shape of the detected object, the bounding box, and the update candidate area to the composite area setting unit 104.

[0038] In the processing described later in S406 to S410, the image processing device 100 uses these update candidate regions to select or generate the region that maximizes the area of ​​the object from which feature points are to be extracted, and selects it as the update candidate region to be used when combining the additional image and the reference image in S411.

[0039] In S405, the image update unit 105 sets the acquired input image as the reference image. Once the reference image setting is complete, the process returns to S401, and the input unit 101 waits for a new input image to be used as an additional image.

[0040] Next, we will explain the case where the image analysis unit 102 determines in S402 that a valid reference image has been set and proceeds to S406.

[0041] In S406, the image analysis unit 102 extracts feature points from the additional image.

[0042] In S407, the image analysis unit 102 determines whether the newly acquired input image can be used as an additional image. For example, the image analysis unit 102 may determine whether the input image can be used as an additional image based on whether the field of view of the newly acquired input image differs from the field of view of the set reference image.

[0043] Specifically, the image analysis unit 102 extracts feature points of objects in the input image and feature points of objects in the reference image. The image analysis unit 102 compares the coordinates of the feature points in the reference image with the coordinates of the corresponding feature points in the input image. If the coordinates of the feature points in the reference image and the input image are nearly identical, the image analysis unit 102 determines that there is no field of view shift between the reference image and the input image and determines that the input image can be used as an additional image. Alternatively, the image analysis unit 102 may determine the presence or absence of a field of view shift by checking whether four or more non-collinear coordinates of feature points extracted from common objects in the two images are identical.

[0044] If the image analysis unit 102 determines that there is a field of view misalignment between the reference image and the input image and that the input image cannot be used as an additional image, the process proceeds to S412. On the other hand, if the image analysis unit 102 determines that there is no field of view misalignment between the reference image and the input image and that the input image can be used as an additional image, the process proceeds to S408 using the input image as an additional image.

[0045] In S408, the candidate area setting unit 103 sets update candidate areas in the additional image. For example, the candidate area setting unit 103 analyzes the shape of the contours of objects in the additional image and the reference image, and also detects objects such as people and equipment. The candidate area setting unit 103 may set update candidate areas in the additional image based on the relationship between the results of the analysis and detection and the update candidate areas of the reference image 300 detected in S403.

[0046] As an example, the operation of the candidate region setting unit 103 when the additional image 312 shown in Figure 3(B) is input after the input of the reference image 300 will be explained. In the additional image 312, there are people 314 and base 313 in the cinder 302 that were not present in the reference image 300. The candidate region setting unit 103 extracts the contours of objects in the reference image 300 and the additional image 312 and examines their relationship with the updated candidate region of the reference image 300 obtained in S404. The objects that have differences in contours, etc., between the reference image 300 and the additional image 312 are people 314 and base 313. In the additional image 312, the candidate region setting unit 103 obtains a bounding box 316 from the detected people 314 and a bounding box 315 from the contour shape mask of base 313. The candidate region setting unit 103 records at least one of the newly detected objects, object shape masks, and bounding boxes in the additional image 312 as updated candidate regions for the additional image 312.

[0047] As another example, the operation of the candidate area setting unit 103 when the additional image 317 shown in Figure 3(C) is input after the input of the reference image 300 will be explained. The additional image 317 contains a person 314, a base 313, a white line 318, and a net 319 in which the shape of the net 301 of the reference image 300 has been changed, which were not present in the reference image 300. In the additional image 317, the character 304 that was hidden by the net 301 can be recognized due to the change in the shape of the net. Therefore, the objects that have differences in outlines, etc., between the reference image 300 and the additional image 317 are the person 314, the base 313, the net 319, the character 304, and the white line 318. The candidate area setting unit 103 sets at least one of the bounding boxes 308 of the character 304, 310 of the white line 303, 322 of the white line 318 and base 313, and 316 of the person 314 as update candidate areas. The candidate area setting unit 103 also detects a shape mask for the shape of the net 319 in the same way as it did for the reference image 300, and sets the area 321 of the shape mask as an update candidate area.

[0048] The candidate region setting unit 103 sets regions where objects with differences in contours, etc., exist in the reference image 300 and the additional image 312 as update candidate regions, outputs information on the update candidate regions of the additional image to the composite region setting unit 104, and proceeds to S409.

[0049] In S409, the composite region setting unit 104 determines, for example, whether there are any areas among the update candidate areas set for the additional image in S408 that can be used to update the reference image. The composite region setting unit 104 may exclude objects whose position and shape change, such as people and nets, from the update candidate areas of the additional image. The composite region setting unit 104 determines that the remaining update candidate areas of the additional image are update candidate areas that can be used for updating. In other words, the composite region setting unit 104 determines that update candidate areas that include objects whose position and shape do not change or change less than people, such as bases, white lines, and cinders, are update candidate areas that can be used for updating. Therefore, the composite region setting unit 104 may not determine that update candidate areas that include objects whose position and shape change is greater than a predetermined threshold, but may determine that update candidate areas that include objects whose position and shape change is smaller than the threshold are usable. The composite region setting unit 104 may also determine whether there are any objects to be added to the update reference image that include areas that can be used for updating.

[0050] If the update candidate areas for the additional image only include update candidate areas such as the person 314 and the net 319, and if there are no changes in the outlines of objects between the additional image and the reference image, then there are no objects to add to the updated reference image. In this case, the composite area setting unit 104 determines that there are no available update candidate areas and proceeds to S412. If there are update candidate areas that include objects that can be used as update candidates, such as the base 313, the white line 318, and the character 304, the composite area setting unit 104 proceeds to S410.

[0051] In S410, the composite region setting unit 104 composites the updated objects detected from the additional image onto the reference image to set a composite region, which is the region for updating the reference image. The composite region setting unit 104 may set the composite region based on the update candidate regions that were determined to be available in S409. Here, since the capture times of the reference image and the additional image are different, the objects at the edges (or boundaries) may not necessarily match.

[0052] When using a synthesized updated reference image for feature point extraction, if there are discontinuities in image color and object shape between the reference image and the added image at the edges of the added image synthesized with the reference image, a problem arises in which non-existent contours are selected as feature points. To avoid this problem, the synthesis region setting unit 104 uses a region where there is little overlap between the boundary between the two images and the objects used for feature point selection (for example, a region including available update candidate regions) as the synthesis region, and performs processing to exclude the edges of the images from the feature point selection target.

[0053] The following describes how to determine the synthesis region when compositing the base image 313 of the additional image 312 onto the reference image 300.

[0054] The composite region setting unit 104 uses the shape mask of the object detected from the reference image 300 to determine the bounding box of the shape mask, and checks whether all pixels corresponding to the bounding box exist in the additional image and are available for updating. If all pixels corresponding to the bounding box of the object in the reference image do not overlap with the shape mask or bounding box of a changing object such as a person or a net in the additional image, the bounding box of the reference image is set as the composite region.

[0055] In the additional image 312, the bounding box 316 of a person is located within a portion of the bounding box 309. Since the person 314 in the additional image is an object whose shape changes, the composite region setting unit 104 does not use the pixels within the bounding box 316 of the person 314 in the additional image 312 to update the reference image 300. Thus, since the bounding box 309 of the cinder surface 302 does not contain pixels from the additional image in areas that overlap with the bounding box 316, the composite region setting unit 104 does not use the bounding box 309 of the cinder surface 302 as a composite region for compositing the additional image.

[0056] If the bounding box 309 cannot be used as a composite region, the composite region setting unit 104 determines whether the bounding box 315 of the base 313 within the bounding box 309 can be used as the shape of the composite region by checking whether the bounding box 315 of the base 313 overlaps with the shape mask of the cinder 302. When making this determination, the composite region setting unit 104 uses the contour of the bounding box 315 of the base 313, which has been expanded by a predetermined number of pixels. The predetermined number of pixels is an example of a predetermined expansion condition. This suppresses the extraction of features from the contour of the bounding box 315 during feature extraction. The predetermined number of pixels to expand the bounding box 315 may be the sum of the number of pixels referenced by the feature selection algorithm during feature selection and the width of the region excluded from feature selection when the reference image and the additional image are combined.

[0057] In additional image 312, if the entire area of ​​the expanded bounding box 315 of base 313 is included in the shape mask of cinder 302, the composite region setting unit 104 sets the bounding box 315 area as the composite region.

[0058] In additional image 312, if the bounding box 315 of the inflated base 313 overlaps with the contour of the shape mask of the cinder 302, the composite region setting unit 104 checks whether there is an overlap between the inflated region of the base 313's shape mask and the contour of the cinder 302's shape mask. If the inflated region of the base 313's shape mask and the contour of the cinder 302's shape mask overlap, the composite region setting unit 104 uses the inflated region of the base 313's shape mask as the composite region.

[0059] If there is an overlap between the shape mask of base 313 and the contour of the shape mask of cinder 302, the composite region setting unit 104 determines whether or not there is an overlap between the bounding box 316 of person 314 and the contour of the region obtained by expanding the shape mask of cinder 302. If the composite region setting unit 104 determines that there is no overlap, it uses the region obtained by expanding the shape mask of cinder 302 as the composite region.

[0060] The composite region setting unit 104 determines that if there is no overlap of all available regions and no region available for use as a composite region, it will not add the objects of the added additional image to the reference image.

[0061] In S411, the image update unit 105 acquires the composite region created by the composite region setting unit 104 and generates an updated reference image by compositing the additional image and the reference image. For example, the image update unit 105 may generate an updated reference image by cutting out the composite region of the additional image and compositing the resulting image into the corresponding region of the reference image.

[0062] Here, it is preferable to exclude a certain range centered on the boundary between the combined reference image and the additional image from the acquisition of feature points. For this reason, the image update unit 105 replaces the pixel values ​​of the bounding box or shape mask region detected in S410, which the feature selection algorithm refers to when selecting feature points, with the pixel values ​​of the expanded region of the additional image. Furthermore, the image update unit 105 may apply Gaussian blur to the outer edge for a predetermined number of pixels from the outer edge. This makes it possible to exclude the contour of the boundary between the two images before combination from the target of feature selection.

[0063] Through the above processing, the updated reference image is output as the updated reference image. This makes it possible to generate an updated reference image with increased feature points that reflect the desired object regions from additional images taken after the reference image was captured. This embodiment can increase the feature points of objects required at the time of shooting in the reference image by adding objects that are common to both the original and current images. As a result, this embodiment can more accurately detect field of view shifts and improve the accuracy of alignment when the field of view shifts based on the increased feature points in the reference image. Furthermore, this embodiment can suppress changes in the position of notable objects in consecutive captured images due to alignment failures, and reduce unnatural vibrations of objects when viewed as a video.

[0064] As described above, the reference image is captured before the capture time of the input image for which the field of view shift is to be corrected. On the other hand, the state of an object may change by the time the input image is captured, and the object used for feature extraction in the reference image may no longer be present in the input image for which the field of view shift is to be detected and aligned. In this case, the feature points of the reference image do not match the feature points of the input image at the time of capture, so the amount of movement between the reference image and the input image cannot be obtained. Even in such cases, this embodiment can increase the number of feature points by adding an object that can become a stable feature to the input image after the reference image has been captured, as preparation for shooting progresses. As a result, even if an object that was present when the reference image was captured is no longer present, this embodiment can improve the accuracy of detecting the field of view shift by using the feature points of the added object.

[0065] For example, in this embodiment, the number of feature points in the reference image can be increased by the presence of a base 313, etc., which was not present when the reference image was taken but is present when the additional image was taken.

[0066] Furthermore, when photographing event venues or sports competitions, if setup and rehearsals continue until just before shooting, it is difficult to create a situation where there are no obstacles such as people within a wide shooting range. For example, objects such as the aforementioned characters 304 are photographed with other obstructions such as the net 301 in the reference image. Even in such cases, if the net 301 deforms or otherwise changes during the shooting of the additional image and the characters 304 are included in the additional image, this embodiment can synthesize the characters 304 into the reference image to increase the number of feature points.

[0067] When a reference image includes background objects visible through a net 301 whose position and shape change, combining the additional image and the reference image results in a problem where the net shape does not match at the boundary of the combined image. Furthermore, when combining an additional image and a reference image with different color tones due to changes in ambient light or camera settings, a boundary is created between the additional image and the reference image, which can be mistakenly recognized as an outline shape. This embodiment sets the composite region based on an expanded update candidate region, thus suppressing an increase in inappropriate feature points due to the boundary of the composite region and other factors.

[0068] This embodiment adds objects such as lines and bases that do not change much to the reference image. This allows the embodiment to reduce feature points that are present only in one of the input images captured when detecting the field of view shift.

[0069] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0070] The disclosures herein include the following image processing apparatus, image processing methods, and programs. (Item 1) An image processing device that generates a reference image for detecting angle of view shift, An input means for acquiring a first image generated by photographing a predetermined region, and a second image generated by photographing a region that includes at least a part of the region after the first image has been photographed, A composite region setting means for setting a composite region which is an area for updating the first image, Image updating means for updating a reference image by combining the image of the composite region of the second image with the first image, An image processing apparatus characterized by having (Item 2) Image analysis means for determining whether or not to combine the second image with the first image, Candidate region setting means for detecting objects included in at least one of the first image and the second image and setting update candidate regions that are candidates for the composite region, An image processing apparatus as described in item 1, having the characteristic of having the following features. (Item 3) The image analysis means determines whether or not to combine the images based on the difference between the field of view of the first image and the field of view of the second image. The image processing apparatus according to item 2, characterized in that (Item 4) The image analysis means compares the feature points of the object extracted from the first image with the feature points of the object extracted from the second image to determine the shift in the field of view and decide whether or not to combine them. The image processing apparatus according to item 3, characterized in that (Item 5) The candidate region setting means detects the object from the first image and the second image based on at least one of the color, contour, or frequency components of the second image. The image processing apparatus according to item 2 or item 3, characterized in that it is an image processing apparatus. (Item 6) The candidate region setting means is The update candidate region is set by detecting objects including a first object whose position and shape change in at least one of them, and a second object whose change is less than that of the first object. The aforementioned synthesis region setting means is The update candidate region containing the first object is not set as the composite region. The update candidate region including the second object is set as the composite region. An image processing apparatus according to any one of items 2 to 5, characterized in that it is an image processing apparatus. (Item 7) The second object has less change in position and / or shape than the first object. The image processing apparatus according to item 6, characterized in that (Item 8) The composite region setting means sets the composite region based on the update candidate region obtained by expanding the update candidate region, which includes the object, based on predetermined expansion conditions. An image processing apparatus according to any one of items 2 to 7, characterized in that it is an image processing apparatus. (Item 9) The candidate region setting means is Based on the bounding box generated based on the shape mask generated based on the shape of the object, the update candidate region is set. The aforementioned synthesis region setting means is The composite region is set based on the expanded update candidate region and the contour of the shape mask. The image processing apparatus according to item 8, characterized in that (Item 10) An image processing method for generating a reference image for detecting angle of view shift, A first image is obtained by photographing a predetermined region, and a second image is obtained by photographing a region that includes at least a part of the region after the first image has been taken. A composite region is set, which is the area for updating the first image described above. The reference image is updated by combining the image of the composite region of the second image with the first image. An image processing method characterized by the following: (Item 11) A program to cause a computer to function as one of the means of an image processing device described in any one of items 1 through 9.

[0071] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0072] 100...Image processing unit, 101...Input unit, 102...Image analysis unit, 103...Candidate region setting unit, 104...Composite region setting unit, 105...Image update unit, 300...Reference image, 312...Additional image, 317...Additional image, 500, 503...Shape mask, 302, 304, 309, 310, 315, 316, 322...Bounding box.

Claims

1. An image processing device that generates a reference image for detecting angle of view shift, An input means for acquiring a first image generated by photographing a predetermined region, and a second image generated by photographing a region that includes at least a part of the region after the first image has been taken, A composite region setting means for setting a composite region which is an area for updating the first image, Image updating means for updating a reference image by combining the image of the composite region of the second image with the first image, An image processing apparatus characterized by having

2. Image analysis means for determining whether or not to combine the second image with the first image, Candidate region setting means for detecting objects included in at least one of the first image and the second image and setting update candidate regions that are candidates for the composite region, The image processing apparatus according to claim 1, characterized by having the following features.

3. The image analysis means determines whether or not to combine the images based on the difference between the field of view of the first image and the field of view of the second image. The image processing apparatus according to claim 2.

4. The image analysis means compares the feature points of the object extracted from the first image with the feature points of the object extracted from the second image to determine the shift in the field of view and decide whether or not to combine them. The image processing apparatus according to claim 3.

5. The candidate region setting means detects the object from the first image and the second image based on at least one of the color, contour, or frequency components of the second image. The image processing apparatus according to claim 2.

6. The candidate region setting means is The update candidate region is set by detecting an object that includes a first object whose position and shape change in at least one way, and a second object whose change is less than that of the first object. The aforementioned synthesis region setting means is The update candidate region including the first object is not set as the composite region. The update candidate region including the second object is set as the composite region. The image processing apparatus according to claim 2.

7. The second object exhibits less change in position and / or shape than the first object. The image processing apparatus according to claim 6.

8. The composite region setting means sets the composite region based on the update candidate region obtained by expanding the update candidate region, which includes the object, based on predetermined expansion conditions. The image processing apparatus according to claim 2.

9. The candidate region setting means is Based on the bounding box generated based on the shape mask generated based on the shape of the object, the update candidate region is set. The aforementioned synthesis region setting means is The composite region is set based on the expanded update candidate region and the contour of the shape mask. The image processing apparatus according to feature 8.

10. An image processing method for generating a reference image for detecting angle of view shift, A first image is obtained by photographing a predetermined area, and a second image is obtained by photographing an area that includes at least a portion of the area after the first image has been taken. A composite region is set, which is the region for updating the first image described above. The reference image is updated by combining the image of the composite region of the second image with the first image. An image processing method characterized by the following:

11. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 9.

Images