IMAGE SYNTHESIS DEVICE, IMAGE SYNTHESIS METHOD, AND PROGRAM

Abstract

To provide an image synthesizing apparatus and the like capable of suitably synthesizing images.SOLUTION: An image synthesizing apparatus sets a template area in the Nth image and searches for a template corresponding area corresponding to the template area from the (N+1)th image. Then, the apparatus groups the template areas based on the amount of movement from the template area to the template corresponding area and selects a group for specifying the amount of movement from the Nth image to the (N+1)th image. After that, the apparatus specifies the amount of movement from the Nth image to the (N+1)th image based on the amount of movement of the template area within the selected group.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image synthesis device, an image synthesis method, and a program. [Background technology]

[0002] A technology has been proposed that extracts coordinate data of feature points of a first image detected in a second image, and combines the first and second images so that the coordinates of the feature points stored with the image data of the first image match the coordinates of the feature points detected in the image data of the second image, thereby generating a panoramic image (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-282100 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology described in Patent Document 1, for example, if a repeated pattern is present in the second image, feature points may not be detected appropriately, and an appropriate composite image may not be generated.

[0005] The present invention has been made in consideration of the above-mentioned circumstances, and has an object to provide an image synthesis device and the like that can suitably synthesize images. [Means for solving the problem]

[0006] In order to achieve the above object, an image synthesis device according to the present invention comprises: 1. An image synthesis device that synthesizes a first image and a second image that is an image that at least partially overlaps with the first image and is located at a position moved a predetermined amount from a position of the first image, A determination means for determining n (n is an integer equal to or greater than 2) template regions from the first image; a detection means for detecting, from the second image, template corresponding regions corresponding to each template region in the first image; an acquisition means for acquiring a movement amount from each template region to the template corresponding region detected by the detection means; a grouping means for grouping template regions whose movement amount acquired by the acquisition means is within a predetermined range; a group selection means for selecting a group from the groups generated by the grouping means based on the number of template regions included therein; a movement amount specification means for specifying a movement amount from a position of the first image to a position of the second image based on a movement amount from each template region included in the group selected by the group selection means to a template corresponding region; an image synthesis means for synthesizing the first image and the second image based on the amount of movement specified by the amount of movement specification means; Equipped with 、 The image synthesis means includes: a coefficient setting means for setting a coefficient between 0 and 1 for image synthesis to each of a plurality of images including the first image and the second image; a synthesis means for weighting the image by multiplying the pixel value of each image by a coefficient set for each image, and synthesizing the plurality of images; The coefficient setting means For the plurality of images, coefficients of a decreasing function are set so that the center of each image is 1 and the end is 0 in the synthesis direction; When a center of one of the multiple images is located closer to the center than an edge of the one of the multiple images with respect to the synthesis direction, a coefficient of the one image is reset so that a coefficient of a position in the one image corresponding to the center of the other image is set to 0; The coefficients of each image at each synthesis position in the synthesis direction are reset by normalizing the coefficients of each image with respect to the sum of the coefficients of all images at that synthesis position.

[0007] The detection means may detect, in the second image, an area in which a correlation value between each template area and the image is equal to or greater than a predetermined value, as the template corresponding area.

[0008] The group selection means includes: selecting a group including a maximum number of template regions from the groups generated by the grouping means; When there are multiple groups each containing the maximum number of template regions, the group having the highest correlation value between the image of each template region and the image of each template corresponding region within the group may be selected.

[0010] The image synthesis method of the present invention further comprises: 1. An image synthesis method for an image synthesis device that synthesizes a first image with a second image that is an image that at least partially overlaps with the first image and is located at a position shifted a predetermined amount from a position of the first image, comprising: a determining step of determining n (n is an integer equal to or greater than 2) template regions from the first image; a detection step of detecting, from the second image, template corresponding regions corresponding to each template region in the first image; an acquisition step of acquiring a movement amount from each template region to the template corresponding region detected in the detection step; a grouping step of grouping template regions whose movement amounts obtained in the obtaining step are within a predetermined range; a group selection step of selecting a group from the groups generated in the grouping step based on the number of template regions included therein; a movement amount specifying step of specifying a movement amount from a position of the first image to a position of the second image based on a movement amount from each template region included in the group selected in the group selection step to a template corresponding region; an image synthesis step of synthesizing the first image and the second image based on the movement amount specified in the movement amount specification step; Equipped with 、 The image synthesis step includes: a coefficient setting step of setting a coefficient of 0 to 1 for image synthesis for each of a plurality of images including the first image and the second image; a synthesis step of weighting the image by multiplying the pixel value of each image by a coefficient set for each image, and synthesizing the plurality of images; In the coefficient setting step, For the plurality of images, coefficients of a decreasing function are set so that the center of each image is 1 and the end is 0 in the synthesis direction; When a center of one of the multiple images is located closer to the center than an edge of the one of the multiple images with respect to the synthesis direction, a coefficient of the one image is reset so that a coefficient of a position in the one image corresponding to the center of the other image is set to 0; The coefficients of each image at each synthesis position in the synthesis direction are reset by normalizing the coefficients of each image with respect to the sum of the coefficients of all images at that synthesis position.

[0011] In addition, the program of the present invention is Computer, A program that functions as an image synthesis device that synthesizes a first image and a second image that is an image that at least partially overlaps with the first image and is located at a position moved a predetermined amount from a position of the first image, A determination means for determining n (n is an integer equal to or greater than 2) template regions from the first image; a detection means for detecting, from the second image, template corresponding regions corresponding to each template region in the first image; an acquisition means for acquiring a movement amount from each template region to the template corresponding region detected by the detection means; a grouping means for grouping template regions whose movement amount acquired by said acquisition means is within a predetermined range; a group selection means for selecting a group from the groups generated by the grouping means based on the number of template regions included therein; a movement amount specification means for specifying a movement amount from a position of the first image to a position of the second image based on a movement amount from each template region included in the group selected by the group selection means to a template corresponding region; an image synthesis means for synthesizing the first image and the second image based on the amount of movement specified by the amount of movement specification means; Function as 、 The image synthesis means includes: a coefficient setting means for setting a coefficient between 0 and 1 for image synthesis to each of a plurality of images including the first image and the second image; a synthesis means for weighting the image by multiplying the pixel value of each image by a coefficient set for each image, and synthesizing the plurality of images; The coefficient setting means For the plurality of images, coefficients of a decreasing function are set so that the center of each image is 1 and the end is 0 in the synthesis direction; When a center of one of the multiple images is located closer to the center than an edge of the one of the multiple images with respect to the synthesis direction, a coefficient of the one image is reset so that a coefficient of a position in the one image corresponding to the center of the other image is set to 0; The coefficients of each image at each synthesis position in the synthesis direction are reset by normalizing the coefficients of each image with respect to the sum of the coefficients of all images at that synthesis position. Effect of the Invention

[0012] According to the present invention, images can be suitably synthesized. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing an example of a configuration of an image synthesis device according to an embodiment; [Diagram 2] 13 is a flowchart illustrating an example of an image synthesis process. [Diagram 3] 13 is a flowchart illustrating an example of a movement amount determination process. [Figure 4] FIG. 13 is a diagram showing an example of pre-combine image data. [Diagram 5] 13A to 13D are diagrams showing examples of determining a target range, a template area, and a search range. [Figure 6] 13A and 13B are diagrams illustrating a specific example of grouping processing. [Figure 7] 13 is a diagram showing a composite position of the first image and the second image based on a specified amount of movement. FIG. [Figure 8] 13 is a flowchart illustrating an example of a synthesis process. [Figure 9] FIG. 13 is a diagram showing an example of setting weighting coefficients when synthesizing images. [Figure 10] FIG. 13 is a diagram showing an example of setting weighting coefficients when synthesizing images. [Figure 11] 13(A) to 13(C) are diagrams showing examples of setting weighting coefficients when synthesizing images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An image synthesis device according to an embodiment of the present invention will now be described with reference to the drawings.

[0015] 1 is a block diagram showing an example of a configuration of an image synthesis device 10 according to an embodiment of the present invention. The image synthesis device 10 is a device that synthesizes a plurality of images to generate one synthetic image (panoramic image). The image synthesis device 10 includes, for example, a display 11, an operation reception unit 12, a communication interface 13, a CPU (Central Processing Unit) 14, a ROM (Read Only Memory) 15, a RAM (Read Only Memory) 16, and an auxiliary storage device 17.

[0016] The display 11 includes a display device such as a liquid crystal display, an organic EL (Electro Luminescence) display, etc. Under the control of the CPU 14, the display 11 displays various images (pre-compositing images, composite images, etc.) and an operation screen for image composition, etc.

[0017] The operation reception unit 12 is configured with input devices such as a keyboard, a mouse, and a touch panel, receives operation input from a user, and sends a signal related to the received operation to the CPU 14.

[0018] The communication interface 13 is hardware for communicating via a network such as the Internet, and is, for example, a wired or wireless LAN (Local Area Network) module.

[0019] The CPU 14 performs overall control of the image synthesis device 10. The ROM 15 stores various data and operation programs executed by the CPU 14. The RAM 16 is used as a working area for the CPU 14.

[0020] The auxiliary storage device 17 is composed of a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) or a readable / writable nonvolatile semiconductor memory. The readable / writable nonvolatile semiconductor memory is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or a flash memory. The auxiliary storage device 17 stores a program for synthesizing a plurality of images to generate one composite image (panoramic image). That is, the auxiliary storage device 17 stores a program for causing an information processing device such as a personal computer to function as the image synthesis device 10 of the present embodiment. The program for causing the information processing device to function as the image synthesis device 10 may be stored in the ROM 15.

[0021] The program for causing an information processing device to function as the image synthesis device 10 of this embodiment may be made downloadable via a network, or may be stored in a computer-readable recording medium such as a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc), a magneto-optical disk, a USB (Universal Serial Bus) memory, a memory card, a HDD, or an SSD and distributed.

[0022] Further, the auxiliary storage device 17 stores a plurality of pre-composition image data 171. The plurality of pre-composition image data 171 is image data before panoramic composition, and is image data of images obtained by continuously photographing a common photographing subject while moving the photographing position in a predetermined direction. Therefore, adjacent pre-composition image data 171 are image data that at least partially overlap, and are image data at positions shifted by a predetermined distance or a predetermined angle.

[0023] The multiple pre-combination image data 171 are panorama-combined by an image combining process executed by the CPU 14 to generate one composite image data 172. When the image combining process is executed, the composite image data 172 is stored in the auxiliary storage device 17.

[0024] The pre-composite image data 171 may be stored in the auxiliary storage device 17 via a USB memory or a memory card, or may be stored in the auxiliary storage device 17 via the communication interface 13. The image synthesis device 10 may be provided with an imaging means constituted by an image sensor or the like, and the pre-composite image data 171 may be image data continuously captured by the imaging means. The imaging means may be a digital camera capable of continuously capturing images, or may be an oral cavity camera, an X-ray camera, an endoscope, or the like.

[0025] It should be noted that the configuration of the image synthesis device shown in FIG. 1 is merely an example, and additional components may be added or some components may be omitted.

[0026] Furthermore, the image synthesis device 10 may be configured as a general-purpose information processing device (such as a personal computer, a smartphone, or a tablet) that stores a program for functioning as the image synthesis device 10, or may be configured as dedicated hardware.

[0027] Next, the operation of the image synthesis device 10 will be described. FIG. 2 is a flow chart showing an example of the image synthesis process executed by the CPU 14 of the image synthesis device 10. The CPU 14 executes the image synthesis process based on a program stored in the auxiliary storage device 17. In the image synthesis process, first, the CPU 14 acquires pre-composition image data 171 from the auxiliary storage device 17 (step S1). In step S1, for example, a plurality of pre-composition image data 171 designated by a user's operation of the operation reception unit 12 is acquired. As shown in FIG. 4, the plurality of pre-composition image data 171 are image data of images (first image, second image, and third image in FIG. 4) obtained by continuously photographing a common photographing subject while moving the photographing position in a predetermined direction, and adjacent pre-composition image data 171 at least partially overlap. In addition, the photographing order of the plurality of pre-composition image data 171 can be specified from the first image, which is the N-th pre-composition image data 171. The plurality of pre-composition image data 171 may be different in size, and may be shifted in vertical position (up-down position) as shown in FIG. 4.

[0028] Next, CPU 14 executes a movement amount specification process for specifying the amount of movement of the photographing positions between each image based on pre-combination image data 171 (step S2). In order to generate a panoramic image by combining a plurality of images, it is important to specify the accuracy of the amount of movement of the photographing positions between adjacent images. In the image composition device 10 of this embodiment, a template area is set in each image, and the amount of movement between adjacent images is specified based on the amount of movement of the template area.

[0029] 3 is a flow chart showing an example of the movement amount specification process executed in step S2. In the movement amount specification process, the CPU 14 first sets 1 to N corresponding to the image number (step S201).

[0030] Then, CPU 14 determines the target range of the Nth pre-composition image data 171 (Nth image) (step S202). The target range is a range in pre-composition image data 171 that determines a template region, which will be described later. The target range may be at least a partial region in the Nth image, such as a predetermined position, a predetermined portion obtained by dividing the Nth image, a predetermined portion excluding the outer edge of the Nth image, or the entire range of the Nth image. The target range may also be automatically determined depending on the content of the Nth image (smoothness, number of colors, etc.).

[0031] After determining the target range in the Nth image, CPU 14 determines n template regions from the target range (step S203). The template regions are regions between the Nth image and the (N+1)th pre-combination image data 171 (N+1th image) for comparing image correlation (matching portions). For example, n regions at predetermined positions in the target range are determined as template regions. n may be a predetermined integer of 2 or more (for example, 5).

[0032] Fig. 5(A) is a diagram showing the target range of the first image as pre-synthesis image data 171 and five template regions. As shown in Fig. 5(A), a predetermined portion of pre-synthesis image data 171 (here, the first image) is determined as the target range, and five template regions (templates 1 to 5) are determined in a predetermined arrangement (for example, the arrangement shown in Fig. 5(A)) within the target range.

[0033] The value of n and the position of the template region may be changed according to the image quality (smoothness, resolution, number of colors, etc.) of the Nth image.

[0034] In this manner, in this embodiment, the target area is determined in pre-composition image data 171 (Nth image), and then the template area is determined from the target area. This reduces the processing load and improves the processing speed, compared to when the template area is determined from the entire range of pre-composition image data 171.

[0035] After determining the template region in the Nth image, the CPU 14 determines a search range for the template corresponding region corresponding to each template region in the N+1th image (step S204). In this embodiment, a template corresponding region having a high correlation with the template region is searched (detected) from the N+1th image, and the amount of movement from the position of the Nth image to the position of the N+1th image (also simply referred to as the amount of movement from the Nth image to the N+1th image) is specified based on the amount of movement from the template region to the template corresponding region. In step S204, a search range for searching the template corresponding region from the N+1th image is determined. The search range may be an area in the N+1th image where the probability of detecting the template corresponding region is high, and may be determined to be a position corresponding to the position of each template region, for example. The search range may be different depending on the shooting interval (frame rate) of the pre-combination image data 171 and the image quality. For example, the search range may be wider since it is expected that the amount of movement is larger as the shooting interval becomes longer.

[0036] For example, with respect to the arrangement of the template regions (templates 1, 2, 5) of the first image shown in Fig. 5(A), a predetermined portion in the upper left of the second image shown in Fig. 5(B) is determined as the search range for template 1, a predetermined portion in the lower left of the second image shown in Fig. 5(C) is determined as the search range for template 2, and a predetermined portion in the lower right of the second image shown in Fig. 5(D) is determined as the search range for template 5. The search ranges for templates 3 and 4 may be determined similarly.

[0037] After determining the search range for the template region, CPU 14 searches for a template corresponding region corresponding to the template region from the search range of each template region in the (N+1)th image (step S205). In step S205, a region in the search range where the template region and the image (pixel value of each pixel) match or a region where the correlation value is a predetermined value (0.95, etc.) or more may be detected as a template corresponding region. The threshold value (predetermined value) of the correlation value for detecting a template corresponding region may be lowered since the longer the shooting interval (frame rate), the more likely it is that the shooting subject will change.

[0038] In this manner, in this embodiment, the search range for searching for the template region in the (N+1)th image is determined, and then the template corresponding region is searched for within that search range. This reduces the processing load and improves the processing speed compared to searching for the template corresponding region within the entire range of the (N+1)th image.

[0039] After searching for the template corresponding region corresponding to each template region, the CPU 14 calculates the amount of movement (horizontal and vertical movement distances) from each template region to the template corresponding region (step S206).

[0040] Then, CPU 14 executes a grouping process of the template regions based on the amount of movement of each template region (step S207). In the grouping process of the template regions in step S207, template regions whose correlation values ​​of the amount of movement (movement direction, movement distance) of the template regions are in a predetermined range are grouped. For example, if the amount of movement of the template regions of templates 1 to 5 is as shown in FIG. 6(A), templates 2 to 4, which have a high correlation value of the amount of movement, are grouped. As a result, as shown in FIG. 6(B), template 1 is group A, templates 2 to 4 are group B, and template 5 is group C.

[0041] After executing the grouping process of the template regions, CPU 14 determines whether or not there are two or more maximum groups, which are groups each having the maximum number of template regions (step S208).

[0042] If there is one maximum group (step S208; No), the maximum group is selected as the group for identifying the amount of movement (step S209).

[0043] If there are two or more maximum groups (step S208; Yes), the group with the highest average (or median, etc.) of the correlation value between the template region and the template corresponding region in the group is are selected as a group for identifying the amount of movement (step S210).

[0044] Here, the median of the correlation values, etc. may be any representative value that can be used as a basis for selecting a group, such as the median of the correlation values ​​within a group, a value determined by majority vote within the group, an average value excluding the maximum and minimum correlation values ​​within the group, or the correlation value of a template region selected at random from within a group.

[0045] After selecting the group in step S209 or step S210, CPU 14 saves the average (or median, etc.) of the movement amounts of the template regions in the selected group as the movement amount from the Nth image to the N+1th image (step S211). In this way, the movement amount from the Nth image to the N+1th image is determined based on the movement amount of the template regions in the selected group.

[0046] When the average movement amount of group B shown in FIG. 6(B) is saved as the movement amount from the first image to the second image, the composite position of the second image relative to the first image will be a position moved from the first image by the average movement amount, as shown in FIG. 7.

[0047] Note that the median value of the movement amount of the template regions in the group in step S211 may be any representative value that can be used as a basis for specifying the movement amount of a template region, such as the median value of the movement amount of the template regions in the group, a value determined by majority vote in the group, an average value excluding the maximum and minimum values ​​of the movement amount of the template regions in the group, a movement amount of a template region randomly selected from the group, etc. In Fig. 7, the composite position of the second image relative to the first image is a position moved from the first image by the average movement amount, but it may be a position moved by the representative value of the movement amount of group B.

[0048] Next, CPU 14 adds 1 to N (step S212) and determines whether the Nth image after the increment is the last image (step S213). If the Nth image is not the last image (step S213; No), CPU 14 returns to the process of step S202 and repeats the processes of steps S202 to S212 (processing for specifying the amount of movement from the Nth image to the N+1th image). If the Nth image is the last image (step S213; Yes), CPU 14 ends the process for specifying the amount of movement.

[0049] In this way, multiple template areas are grouped based on the amount of movement, a group is selected, and the amount of movement of the pre-composition image data 171 is determined based on the amount of movement of the template areas of the selected group, so that the amount of movement can be suitably determined while removing noise such as a group including a falsely detected template-corresponding area. This improves the accuracy of determining the amount of movement of the shooting positions between adjacent pre-composition image data 171, and as a result, the pre-composition image data 171 can be suitably combined, and the combined image data 172 can be suitably generated.

[0050] When the movement amount specification process ends, the process returns to FIG. 2, and the CPU 14 arranges all pre-combine image data 171 to be composited at the composite position (eg, FIG. 7) based on the specified movement amount (step S3).

[0051] Then, the CPU 14 executes a synthesis process for synthesizing the pre-synthesis image data 171 to generate synthetic image data 172 (step S4).

[0052] 8 is a flow chart showing an example of the compositing process executed in step S4. In the compositing process, the CPU 14 first sets 1 to N corresponding to the image number (step S401).

[0053] Then, CPU 14 sets coefficients according to a decreasing function with the center in the synthesis direction (X-axis direction) of the Nth image being 1 and the ends (both ends) being 0 (step S402). Specifically, as shown in FIG. 9, linear coefficients are set so that the image center is 1 and both ends of the image are 0 in the synthesis direction. Note that the coefficients are not limited to linear coefficients, and may be set so as to be a curved decreasing function from the image center to the ends. These coefficients are weighting coefficients when synthesizing a plurality of pre-synthetic image data 171, and take values ​​from 0 to 1. When synthesizing images, the pixel values ​​of each pre-synthetic image data 171 are weighted by multiplying them by the coefficient set for that position (X coordinate in the synthesis direction), and image synthesis is performed.

[0054] In this embodiment, for simplicity, an example in which the synthesis direction is the X-axis direction (horizontal direction with the first image as a reference) will be described, but the synthesis direction may be the Y-axis direction (vertical direction with the first image as a reference) or the XY-axis direction (diagonal direction with the first image as a reference). In these cases, a weighting coefficient may also be set for the Y-axis direction, and image synthesis may be performed by weighting pixel values ​​of pre-synthetic image data 171 by the coefficient set for that position (XY coordinates with respect to the synthesis direction).

[0055] Next, the CPU 14 determines whether the Nth image is the last image (step S403). If the Nth image is not the last image (step S403; No), the CPU 14 adds 1 to N (step S404), returns to the process of step S402, and repeats the process of step S402 to initialize the coefficients for all pre-combination images.

[0056] If the Nth image is the last image (step S403; Yes), the CPU 14 sets N corresponding to the image number to 1 again (step S405).

[0057] Then, CPU 14 judges whether or not the center of the N+1th image is located inside (closer to the center) than the edge of the Nth image at the synthesis position (step S406). If the center of the N+1th image is located inside the edge of the Nth image (step S406; Yes), the coefficient of the position in the Nth image corresponding to the center of the N+1th image is reset to 0 (step S407). Between the center of the Nth image and the position where the coefficient has been reset to 0, the coefficient is reset using a decreasing function with the center being 1 and the reset position being 0.

[0058] If the center of the (N+1)th image is not located inside the edge of the Nth image (step S406; No), or after the process of step S407 has been executed, it is determined whether or not the center of the (N-1)th image is located inside (closer to the center) than the edge of the Nth image at the synthesis position (step S408). If the center of the (N-1)th image is located inside the edge of the Nth image (step S408; Yes), the coefficient of the position in the Nth image corresponding to the center of the (N-1)th image is reset to 0 (step S409). Between the center of the Nth image and the position where the coefficient has been reset to 0, the coefficient is reset using a decreasing function with the center at 1 and the reset position at 0.

[0059] FIG. 10 is a diagram showing an example of resetting coefficients according to the arrangement of the second image as an example of the Nth image. The dotted lines in FIG. 10 show the coefficients before resetting. As shown in FIG. 10, when the image center of the third image is located inside the right edge of the second image, the coefficient of the position in the second image corresponding to the center of the third image is reset to 0, and between the center of the second image and the position where the coefficient is reset to 0, the coefficient is reset by a decreasing function with the center being 1 and the reset position being 0. On the other hand, since the image center of the first image is located outside the left edge of the second image, the coefficients on the left side of the center of the second image are not reset.

[0060] If the center of the (N-1)th image is not located inside the edge of the Nth image (step S408; No) or after executing the process of step S409, CPU 14 adds 1 to N (step S410) and determines whether the Nth image after the increment of 1 is the last image (step S411). If the Nth image is not the last image (step S411; No), the process returns to step S406 and repeats the processes of steps S406 to S410 (resetting of coefficients according to the arrangement of each image).

[0061] If the Nth image is the last image (step S411; Yes), the CPU 14 sets N corresponding to the image number to 1 again (step S412).

[0062] Then, CPU 14 resets the coefficient of the Nth image to the coefficient normalized with respect to the sum of the coefficients of all images (step S413). That is, the coefficient of each image is normalized so that the sum of the coefficients of all images at a synthesis position (X coordinate) in the synthesis direction (X axis direction) becomes 1. Specifically, at each synthesis position, the coefficient of the Nth image is divided by the sum of the coefficients of all images, and the resulting value is reset to the coefficient of the Nth image.

[0063] 11(A) to 11(C) are diagrams showing examples of resetting by normalizing the coefficients of the first image, the second image, and the third image. FIG. 11(A) to 11(C) show the composite positions (arrangements) of the first image, the second image, and the third image. When coefficients as shown in FIG. 11(B) are set before normalization (after resetting by arrangement), the coefficients of each image are normalized as shown in FIG. 11(C). Specifically, when the coefficient of another image is not set (there is no other image at the composite position) at a composite position (X coordinate) relative to the composite direction (X axis direction), the coefficient at that composite position becomes 1, and when another coefficient is set at the composite position (X coordinate), a value obtained by dividing the coefficient of one image by the sum of the coefficients of all images is reset as the coefficient of that one image.

[0064] The method of normalizing the coefficients is not limited to division of the sum of the coefficients, and the coefficients may be normalized using a predetermined formula or the like.

[0065] After executing the process of step S413, the CPU 14 determines whether the Nth image is the last image (step S414). If the Nth image is not the last image (step S414; No), the CPU 14 adds 1 to N (step S415), returns to the process of step S413, and repeats the process of step S413 to normalize and reset the coefficients of all pre-combination image data 171.

[0066] If the Nth image is the last image (step S414; Yes), CPU 14 weights each pre-composition image data 171 by multiplying the pixel value of each pre-composition image data 171 by a coefficient set for each pre-composition image data 171 in the composition direction, composes all pre-composition image data 171, and generates composite image data 172 (step S416). The generated composite image data 172 may be stored in auxiliary storage device 17. Thereafter, the composition process is terminated. When the composition process (step S4) is terminated, the image composition process shown in FIG. 2 is terminated.

[0067] After generating the composite image data 172, the composite image data 172 may be displayed on the display 11. In this case, the composite image data 172 may be arranged in a predetermined shape (for example, a rectangular panoramic image) and displayed on the display 11. Also, the user may be allowed to specify the display format and output format by operating the operation receiving unit 12. In addition, when the image synthesis process is executed, the display 11 may be appropriately displayed with respect to the image synthesis process, such as the pre-synthesis image data 171, the result of identifying the amount of movement of the pre-synthesis image data 171, and information on the synthesis position.

[0068] As described above, according to the image synthesis device 10 of the above embodiment, a template area is set in the Nth image, and a template corresponding area corresponding to the template area is searched for in the N+1th image. Then, based on the amount of movement from the template area to the template corresponding area, the template areas are grouped, and a group is selected for identifying the amount of movement from the Nth image to the N+1th image. After that, based on the amount of movement of the template areas in the selected group, the amount of movement from the Nth image to the N+1th image is identified. By grouping the template areas and selecting the groups, it is possible to eliminate erroneous detection due to the presence of repeated patterns, etc., and therefore it is possible to suitably identify the amount of movement of the images.

[0069] Furthermore, according to the image synthesis device 10, a coefficient is set for each pre-synthetic image data 171 according to the amount of movement thus determined (the arrangement of each image), and synthetic image data 172 is generated based on the normalized coefficient, so that the images can be suitably synthesized to generate a panoramic image.

[0070] The present invention is not limited to the above-mentioned embodiment, and various modifications and applications are possible, and further features may be added. For example, the processing contents and determination methods shown in the above-mentioned embodiment are merely examples, and any other method may be used as long as it can achieve the same functions and effects as the above-mentioned embodiment. In addition, not all of the configurations described in the above-mentioned embodiment are essential, and some of them may be missing.

[0071] In the above embodiment, the amount of movement between adjacent images in multiple pre-combination image data 171 is identified based on the movement amount identification process shown in FIG. 3, and the combination process shown in FIG. 8 is executed to generate the composite image data 172. However, as long as the images can be combined suitably, the combination process is not limited to the processing method shown in FIG. 8 (a method of setting a coefficient for each image and combining them), and any processing method may be adopted.

[0072] Furthermore, as long as the amount of movement between adjacent images can be suitably identified, the movement amount identification process is not limited to the process method shown in FIG. 3 (a method of identifying the amount of movement using a template region), and any process method may be employed.

[0073] That is, an image synthesis apparatus according to another aspect of the present invention comprises: An image synthesis device that synthesizes a plurality of images along a predetermined synthesis direction, A coefficient setting means for setting a coefficient between 0 and 1 for image synthesis for each of the plurality of images; a synthesis means for weighting the image by multiplying the pixel value of each image by a coefficient set for each image, and synthesizing the plurality of images; The coefficient setting means For the plurality of images, coefficients of a decreasing function are set so that the center of each image is 1 and the end is 0 in the synthesis direction; When a center of one of the multiple images is located closer to the center than an edge of the one of the multiple images with respect to the synthesis direction, a coefficient of the one image is reset so that a coefficient of a position in the one image corresponding to the center of the other image is set to 0; The coefficients of each image at each synthesis position in the synthesis direction may be reset by normalizing the coefficients of each image with respect to the sum of the coefficients of all images at that synthesis position.

[0074] Also, it is possible to execute only the determination of the amount of movement between images without synthesizing the images. For example, the process of step S4 shown in FIG. 2 may be omitted and only the processes of steps S1 to S3 may be executed by the user's operation of the operation reception unit 12. In this case, the amount of movement between each image may be output instead of the process of step S3. In this way, it is possible to execute the process desired by the user and to grasp the amount of movement between the images. Also, the image synthesis device 10 may output only the result of the determination of the amount of movement between each image or the synthesized image data in response to the input of pre-synthetic image data from a camera or the like. In this case, the display 11, the operation reception unit 12, and the auxiliary storage device 17 may be omitted.

[0075] In the above embodiment, the target area is determined in pre-composition image data 171 (Nth image), and then the template area is determined from the target area. However, the present invention is not limited to this, and the template area may be determined without determining the target area in pre-composition image data 171 (Nth image).

[0076] In the above embodiment, the search range for searching for the template region in the (N+1)th image is determined, and then the template corresponding region is searched for from the search range. However, the present invention is not limited to this, and it is also possible to search for the template corresponding region corresponding to each template region from the entire (N+1)th image without determining the search range.

[0077] In the above embodiment, a coefficient is set in the synthesis direction in the pre-synthesis image data 171, and the pixel values ​​of each pre-synthesis image data 171 are weighted by multiplying them by the coefficient set for that position to perform image synthesis. However, for parts that do not overlap with other images, the parts may be synthesized as is without being weighted by a coefficient.

[0078] In the above embodiment, a coefficient is set in the horizontal direction (X-axis direction) in the pre-composition image data 171, and the pixel value of each pre-composition image data 171 is weighted by multiplying it by the coefficient set for that position (X coordinate) to perform image synthesis, but a coefficient may also be set in the vertical direction (Y-axis direction), and image synthesis may also be performed by weighting the image in the vertical direction using the coefficient. That is, the synthesis direction is the horizontal direction (X-axis direction) based on the first image, but the synthesis direction may be the XY axis direction based on the first image. In other words, the synthesis direction is not limited to a one-dimensional direction, and may be a two-dimensional direction.

[0079] In this case, for each pre-combination image data 171, the same processes as in steps S401 to S411 in FIG. 8 are executed to set a coefficient in the vertical direction (Y-axis direction). Then, in step S413, the coefficient of each image is normalized (e.g., the coefficient of each image at the combination position is divided by the sum of the coefficients of all images) so that the total of the coefficients of all images at the combination position (XY coordinates) in the combination direction (XY axis direction) becomes 1. In this way, the coefficient of the combination position (XY coordinates) in the combination direction (XY axis direction) can be set. After that, in the process of step S416, the pixel value of each pre-combination image data 171 is multiplied by the coefficient set for each pre-combination image data 171 to weight each pre-combination image data 171, and all pre-combination image data 171 are combined to generate the combined image data 172.

[0080] That is, the coefficient setting means For the plurality of images, coefficients of a decreasing function are set so that the center of each image is 1 and the ends are 0 in the synthesis direction (X-axis direction, Y-axis direction, respectively); When a center of one of the multiple images is located closer to the center than an edge of the other image in the synthesis direction (X-axis direction, Y-axis direction), the coefficient of the one image is reset so that the coefficient of the position in the one image corresponding to the center of the other image is 0; The coefficients of each image at each synthesis position (XY coordinates) in the synthesis direction may be reset by normalizing the coefficients of each image with respect to the sum of the coefficients of all images at that synthesis position. [Explanation of symbols]

[0081] 10 Image synthesis device 11 Display 12 Operation reception section 13 Communication Interface 14 CPU 15 ROM 16 RAM 17 Auxiliary storage

Claims

1. 1. An image synthesis device that synthesizes a first image and a second image that is an image that at least partially overlaps the first image and is located at a position moved a predetermined amount from a position of the first image, A determination means for determining n (n is an integer equal to or greater than 2) template regions from the first image; a detection means for detecting, from the second image, template corresponding regions corresponding to each template region in the first image; an acquisition means for acquiring a movement amount from each template region to the template corresponding region detected by the detection means; a grouping means for grouping template regions whose movement amount acquired by the acquisition means is within a predetermined range; a group selection means for selecting a group from the groups generated by the grouping means based on the number of template regions included therein; a movement amount specifying means for specifying a movement amount from a position of the first image to a position of the second image based on a movement amount from each template region included in the group selected by the group selection means to a template corresponding region; an image synthesis means for synthesizing the first image and the second image based on the amount of movement specified by the amount of movement specification means; Equipped with The image synthesis means includes: a coefficient setting means for setting a coefficient between 0 and 1 for image synthesis to each of a plurality of images including the first image and the second image; a synthesis means for weighting the image by multiplying the pixel value of each image by a coefficient set for each image, and synthesizing the plurality of images; The coefficient setting means For the plurality of images, coefficients of a decreasing function are set so that the center of each image is 1 and the end of each image is 0 in the synthesis direction; When a center of one of the plurality of images is located closer to the center than an edge of the other image of the plurality of images with respect to the synthesis direction, a coefficient of the one image is reset so that a coefficient of a position in the one image corresponding to the center of the other image is set to 0; The coefficients of each image at each synthesis position in the synthesis direction are reset by normalizing the coefficients of each image with respect to the sum of the coefficients of all images at that synthesis position.

1. An image synthesis device comprising:

2. The detection means detects regions in the second image in which a correlation value between each template region and the image is equal to or greater than a predetermined value as the template corresponding regions.

2. The image synthesis device according to claim 1.

3. The group selection means includes: selecting a group including a maximum number of template regions from the groups generated by the grouping means; If there are multiple groups each containing the maximum number of template regions, the group with the highest correlation value between the image of each template region and the image of each template corresponding region in the group is selected.

3. The image synthesis device according to claim 1, wherein the image synthesis device is a digital camera.

4. 1. An image synthesis method for an image synthesis device that synthesizes a first image with a second image that is an image that at least partially overlaps with the first image and is located at a position moved a predetermined amount from a position of the first image, the method comprising: A determination step of determining n (n is an integer equal to or greater than 2) template regions from the first image; a detection step of detecting, from the second image, template corresponding regions corresponding to each template region in the first image; an acquisition step of acquiring a movement amount from each template region to the template corresponding region detected in the detection step; a grouping step of grouping template regions whose movement amounts obtained in the obtaining step are within a predetermined range; a group selection step of selecting a group from the groups generated in the grouping step based on the number of template regions included therein; a movement amount specifying step of specifying a movement amount from a position of the first image to a position of the second image based on a movement amount from each template region included in the group selected in the group selection step to a template corresponding region; an image synthesis step of synthesizing the first image and the second image based on the movement amount specified in the movement amount specification step; Equipped with The image synthesis step includes: a coefficient setting step of setting a coefficient of 0 to 1 for image synthesis for each of a plurality of images including the first image and the second image; a synthesis step of weighting the image by multiplying the pixel value of each image by a coefficient set for each image, and synthesizing the plurality of images; In the coefficient setting step, For the plurality of images, coefficients of a decreasing function are set so that the center of each image is 1 and the end of each image is 0 in the synthesis direction; When a center of one of the plurality of images is located closer to the center than an edge of the other image of the plurality of images with respect to the synthesis direction, a coefficient of the one image is reset so that a coefficient of a position in the one image corresponding to the center of the other image is set to 0; The coefficients of each image at each synthesis position in the synthesis direction are reset by normalizing the coefficients of each image with respect to the sum of the coefficients of all images at that synthesis position.

13. An image synthesis method comprising:

5. Computer, A program that functions as an image synthesis device that synthesizes a first image and a second image that is an image that at least partially overlaps with the first image and is located at a position moved a predetermined amount from a position of the first image, A determination means for determining n (n is an integer equal to or greater than 2) template regions from the first image; a detection means for detecting, from the second image, template corresponding regions corresponding to each template region in the first image; an acquisition means for acquiring a movement amount from each template region to the template corresponding region detected by the detection means; a grouping means for grouping template regions whose movement amount acquired by said acquisition means is within a predetermined range; a group selection means for selecting a group from the groups generated by the grouping means based on the number of template regions included therein; a movement amount specification means for specifying a movement amount from a position of the first image to a position of the second image based on a movement amount from each template region included in the group selected by the group selection means to a template corresponding region; an image synthesis means for synthesizing the first image and the second image based on the amount of movement specified by the amount of movement specification means; Functioning as a The image synthesis means includes: a coefficient setting means for setting a coefficient between 0 and 1 for image synthesis to each of a plurality of images including the first image and the second image; a synthesis means for weighting the image by multiplying the pixel value of each image by a coefficient set for each image, and synthesizing the plurality of images; The coefficient setting means For the plurality of images, coefficients of a decreasing function are set so that the center of each image is 1 and the end of each image is 0 in the synthesis direction; When a center of one of the plurality of images is located closer to the center than an edge of the other image of the plurality of images with respect to the synthesis direction, a coefficient of the one image is reset so that a coefficient of a position in the one image corresponding to the center of the other image is set to 0; The coefficients of each image at each synthesis position in the synthesis direction are reset by normalizing the coefficients of each image with respect to the sum of the coefficients of all images at that synthesis position. A program characterized by:

Images