Image stitching device, image stitching method, and program

Abstract

To obtain a learning device that can obtain a learning result by which a coupled disordered region in connected images can be specified.SOLUTION: A learning device 40 includes: a data acquisition unit 41 that acquires learning data including a learning connected image generated by joining a plurality of learning still images and a mask image obtained by labelling a coupled disordered region which is a portion different in condition of a subject from the still images before coupling in the learning connected image; and a model generation unit 42 that generates a learned model 43 for inferring the coupled disordered region in the connected image by using a plurality of learning data sets, each including the learning connected image and a mask image which is correct data corresponding to the learning connected image.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to , painting an image connection processing apparatus and an image connection processing method for performing image connection processing, this and a program for causing a computer system to execute the method.

Background Art

[0002] Conventionally, there has been known a technique for performing image connection processing on a plurality of still images captured by a camera and creating a new still image that covers the shooting ranges of the plurality of still images before connection. A connected image, which is a new still image that covers the shooting ranges of the plurality of still images before connection, is also referred to as a panoramic image. For example, a connected image is created by connecting a plurality of still images captured by a camera mounted on a moving body such as a helicopter, a ship, and a vehicle.

[0003] When connecting a plurality of still images, based on the information on the position and orientation of the camera, after determining the approximate positional relationship of the plurality of still images, the position with the highest correlation between the still images to be connected is derived and connected, or based on the correspondence of feature points between the plurality of still images, a method of deriving the relative position between the still images and connecting them is used.

[0004] In image connection processing, the connection of still images may not be appropriately performed, and there may be a place where the images in the connected image become discontinuous. For example, the portions corresponding to the same location in a plurality of images cannot be correctly joined and may be displaced.

[0005] Patent Document 1 discloses a learning device that performs learning for determining whether the connection of a connected image is appropriate by learning the connected image with information on whether the connection is appropriate as the correct answer. Based on the learning result by the learning device disclosed in Patent Document 1, when it is determined that the connection of the connected image is not appropriate, measures such as outputting another connected image instead can be taken. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] International Publication No. 2020 / 230891 [Overview of the project] [Problems that the invention aims to solve]

[0007] In image stitching, if the process is not performed properly, parts of the stitched image may differ in appearance from the original still images. For example, the number of objects in the still images may differ from the original number, or the shape of the subject in the still images may differ from its original shape. For instance, a building that is originally one may appear as two in the stitched image, or a straight road may become distorted and misaligned in the stitched image. Such parts of the stitched image where the appearance of the subject differs from the original still images are hereinafter referred to as "stitched image regions." Even in stitched images containing stitched image regions, there may be no discontinuities in the image. For example, if a straight road is misaligned and not connected in the stitched image, a discontinuity will occur. However, if the discontinuity is only in the overlapping area of ​​multiple still images, and the subject is misaligned and appears as a double, rather than at the seam, the seam of the still images will not be discontinuous. Therefore, even in stitched images that have been judged to be appropriately stitched based on the learning results of the learning device disclosed in Patent Document 1, stitched image regions may still exist in the image. Therefore, the learning device disclosed in Patent Document 1 was unable to obtain learning results that made it possible to identify which parts of the concatenated image differ from the still image before concatenation.

[0008] This disclosure has been made in view of the above, and concerns the concatenation of a concatenation failure region in a concatenation image. Image stitching device capable of reducing the size to below a predetermined threshold. The objective is to obtain. [Means for solving the problem]

[0009] In order to solve the aforementioned problems and achieve the objectives, the disclosure relating to this disclosure Image stitching device This is a training data acquisition unit that acquires training data including a concatenated training image generated by stitching together multiple training still images, and a mask image that labels the concatenation failure regions in the concatenated training image, which are areas where the state of the subject differs from the still image before concatenation. and, Model generation unit generates a trained model for inferring concatenation regions in concatenated images using multiple training datasets, each containing a concatenated training image and corresponding mask images that are the ground truth data for that concatenated training image. and Equipped with It has a learning device. ru. The image concatenation processing device includes an image concatenation processing unit that concatenates still images to generate a concatenated image to be inferred, an inference target data acquisition unit that acquires the concatenated image to be inferred, and an inference unit that uses a trained model to output concatenation mismatch regions in the concatenated image to be inferred. The image concatenation processing device also includes a concatenation parameter modification unit that changes the concatenation parameters used by the image concatenation processing unit when generating the concatenated image to be inferred if the size of the concatenation mismatch region output by the inference unit is greater than a preset threshold, and repeats the generation of the concatenated image to be inferred and the modification of the concatenation parameters until the concatenation mismatch region in the concatenated image output by the inference unit is less than or equal to the preset threshold. [Effects of the Invention]

[0010] According to this disclosure, the concatenation failure region in the concatenated image Image stitching device capable of reducing the size to below a predetermined threshold. This has the effect of allowing you to obtain [something]. [Brief explanation of the drawing]

[0011] [Figure 1] A diagram showing the configuration of the image stitching processing device according to Embodiment 1. [Figure 2] This figure shows an example of image stitching processing by the image stitching processing unit according to Embodiment 1. [Figure 3] This figure shows an example configuration of a learning device that performs machine learning to realize the learning unit for detecting misaligned regions in the image stitching processing device according to Embodiment 1. [Figure 4] This figure shows an example of the configuration of the neural network used for learning in the learning unit for detecting misaligned regions of image stitching, which is part of the image stitching processing device according to Embodiment 1. [Figure 5] A flowchart illustrating the procedure for learning by the learning unit that detects and learns the linking failure region of the image linking processing device according to Embodiment 1. [Figure 6] This figure shows an example of the configuration of an inference device that realizes the image stitching failure region detection unit of the image stitching processing device according to Embodiment 1. [Figure 7]Flowchart showing the inference procedure of the connection failure area detection unit included in the image connection processing apparatus according to Embodiment 1 [Figure 8] Diagram showing the configuration of the image connection processing apparatus according to Embodiment 2 [Figure 9] Diagram showing the configuration of the learning data generation unit of the image connection processing apparatus according to Embodiment 2 [Figure 10] Diagram showing the processing of the learning data generation unit of the image connection processing apparatus according to Embodiment 2 [Figure 11] Diagram showing the processing of the learning data generation unit of the image connection processing apparatus according to Embodiment 2 [Figure 12] Diagram showing the configuration of the learning data generation unit of the image connection processing apparatus according to Embodiment 3 [Figure 13] Diagram showing the configuration of the learning data generation unit of the image connection processing apparatus according to Embodiment 4 [Figure 14] Diagram showing the configuration of the image connection processing apparatus according to Embodiment 5 [Figure 15] Diagram showing the relationship between the connection parameters changed by the connection parameter change unit according to Embodiment 5 and the size of the connection failure area [Figure 16] Diagram showing the configuration of the image connection processing apparatus according to Embodiment 6 [Figure 17] Diagram showing the configuration of the control circuit for realizing the functions of the image connection processing apparatus, learning apparatus, and inference apparatus according to Embodiments 1 to 6

Embodiments for Carrying Out the Invention

[0012] Hereinafter, regarding the embodiments The picture image connection processing apparatus , painting The image connection processing method and program will be described in detail based on the drawings.

[0013] Embodiment 1. Figure 1 is a diagram showing the configuration of an image concatenation processing device according to Embodiment 1. The image concatenation processing device 10 according to Embodiment 1 includes an image concatenation processing unit 11 that performs image concatenation processing to concatenate a plurality of still images to generate a concatenated image, a concatenation failure region detection learning unit 12 that learns the relationship between the concatenated image and the concatenation failure region in the concatenated image, a learned model storage unit 13 that stores a learned model 43 which is the learning result of the concatenation failure region detection learning unit 12, a concatenation failure region detection unit 14 that detects the concatenation failure region in the concatenated image, and a detection result image generation unit 15 that generates a detection result screen showing the detection result of the concatenation failure region.

[0014] The image stitching processing unit 11 stitches together multiple still images to create a stitched image, which is a new still image that covers the shooting range of the multiple still images before stitching. For example, the image stitching processing unit 11 determines the approximate positional relationship of the multiple still images based on the camera's position and orientation information, and then stitches the still images together by performing scaling, rotation, and projection transformation on at least one of the multiple still images to derive the position with the highest correlation between the still images to be stitched together. Alternatively, the image stitching processing unit 11 determines the approximate positional relationship of the multiple still images based on the camera's position and orientation information, and then stitches the still images together by performing scaling, rotation, and projection transformation on at least one of the multiple still images to derive the relative positions between the still images based on the correspondence of feature points between the multiple still images.

[0015] Figure 2 shows an example of image stitching processing by the image stitching processing unit according to Embodiment 1. Here, the case of stitching together still images taken from a helicopter 95 is used as an example. The shooting range includes a straight road 88 in a grassland 87. Still image A1 taken at time t and still image A2 taken at time t+Δt overlap in part of their shooting ranges. The position and attitude of the camera mounted on the helicopter 95 are different at time t and time t+Δt. Therefore, the image stitching processing unit 11 performs processing to convert still image A1 into a still image equivalent to the ground plane based on the camera's position and attitude information at time t, and generates still image B1. Similarly, the image stitching processing unit 11 performs processing to convert still image A2 into a still image equivalent to the ground plane based on the camera's position and attitude information at time t+Δt, and generates still image B2.

[0016] The image stitching processing unit 11 performs the above-mentioned scaling, rotation, and projection transformation on at least one of the still images B1 and B2 corresponding to the ground plane to generate still images C1 and C2 for stitching. In Figure 2, scaling, rotation, and projection transformation are performed only on still image B2, so the still image C1 for stitching is the same image as still image B1. The image stitching processing unit 11 stitches the still images C1 and C2 for stitching. By performing this processing, the stitched image becomes an image in which the road 88 is connected without being shifted or distorted in the parts where the shooting ranges overlap. The stitching failure region here refers to the region in which the road 88 is shifted or distorted when the ideal state of still image C2 described above cannot be obtained. In other words, it refers to the region in which the positions of the two images that are originally at the same location are shifted, and the surrounding area.

[0017] When stitching together multiple aerial images, the above method, which involves converting them to still images equivalent to the ground plane before stitching, is commonly used, but other methods may also be used.

[0018] Here, we have explained the process of creating a panoramic image that covers a wide area by stitching together a series of still images acquired from an aerial photograph as an example, but this is just one example and can be applied to other uses. In other words, the still images that the image stitching device 10 stitches together are not limited to still images taken from a helicopter 95 or the like.

[0019] Furthermore, the image stitching processing unit 11 performs blending on the parts where the shooting ranges of the multiple still images to be stitched overlap. For blending, in addition to alpha blending, which superimposes two images and combines them based on the alpha value, which is the transparency set for each pixel, known blending modes such as dithering, multiplication, screen, and overlay can be used.

[0020] The concatenation failure region detection learning unit 12 is a learning device that uses a concatenated learning image and a mask image in which the concatenation failure region of the concatenated image has been labeled as learning data.

[0021] Figure 3 is a diagram showing an example configuration of a learning device that performs machine learning to realize the learning unit for detecting misconnection regions in the image stitching processing device according to Embodiment 1. The learning device 40 includes a data acquisition unit 41, which is a learning data acquisition unit, and a model generation unit 42. The data acquisition unit 41 receives a learning stitched image and a mask image as input. The data acquisition unit 41 creates learning data using the input learning stitched image and mask image. In Embodiment 1, the learning data is data that associates the learning stitched image and the misconnection regions within the learning stitched image with each other. In this way, the data acquisition unit 41 acquires learning data that includes the learning stitched image and the learning misconnection regions.

[0022] The model generation unit 42 generates a trained model 43 for inferring concatenation regions in concatenated images using multiple training datasets, each containing a training concatenated image and a mask image which is the correct data corresponding to the training concatenated image. The trained model storage unit 13 stores the generated trained model 43. The trained model storage unit 13 shown in Figure 3 is an external storage unit of the learning device 40. The trained model storage unit 13 may also be located inside the learning device 40.

[0023] The model generation unit 42 can use any known supervised learning algorithm as its learning algorithm. As an example, the case where a neural network is applied will be described.

[0024] The model generation unit 42 learns the disconnected regions within the connected images using so-called supervised learning, for example, according to a neural network model. Here, supervised learning is a method in which a pair of input and result data is provided to the learning device 40, thereby learning features in the training data and inferring the result from the input.

[0025] Training data includes inputs and labels that correspond to the inputs. The concatenated training images correspond to the inputs, and the disconnected regions within the concatenated training images correspond to the labels. A neural network consists of an input layer composed of multiple neurons, a hidden layer (an intermediate layer) composed of multiple neurons, and an output layer composed of multiple neurons. The hidden layer can be one or more.

[0026] Figure 4 shows an example of the configuration of a neural network used for learning in the learning unit for detecting misconnection regions of the image stitching processing device according to Embodiment 1. The neural network shown in Figure 4 is, for example, a neural network consisting of a convolutional neural network 202 and a deconvolutional neural network 203. For an input image 201, the output image 204 is obtained. The convolutional neural network 202 and the deconvolutional neural network 203 have adjustable neural network weights.

[0027] In Embodiment 1, the neural network learns the connectivity mismatch region through so-called supervised learning, according to the training data acquired by the data acquisition unit 41. That is, the neural network learns the connectivity mismatch region by inputting a connected image into the convolutional neural network 202 and adjusting the weights W1 and W2 so that the result output from the deconvolutional neural network 203 approaches the connectivity mismatch region. The model generation unit 42 generates a trained model 43 by performing the above-described learning and outputs the trained model 43. The trained model storage unit 13 stores the trained model 43 output from the model generation unit 42. The model generation unit 42 may read the already generated trained model 43 from the trained model storage unit 13 and update the trained model 43 by retraining according to the training data.

[0028] Next, the learning process by the linkage failure region detection learning unit 12 will be described. Figure 5 is a flowchart showing the procedure of the learning process by the linkage failure region detection learning unit provided in the image linkage processing device according to Embodiment 1. In step S11, the learning device 40 acquires learning data with the data acquisition unit 41. The data acquisition unit 41 associates the learning linkage image and the mask image, which are both acquired simultaneously, to create learning data. Note that the data acquisition unit 41 only needs to be able to create learning data in which the learning linkage image and the linkage failure regions in the learning linkage image are associated with each other, and does not necessarily need to acquire the learning linkage image and the mask image at the same time.

[0029] In step S12, the learning device 40 uses a model generation unit 42 to learn the misconnected regions according to the training data through so-called supervised learning, and generates or updates a trained model 43. In step S13, the model generation unit 42 outputs the generated or updated trained model 43. With this, the learning device 40 completes the learning process according to the procedure shown in Figure 5. The trained model storage unit 13 stores the trained model 43 obtained through the learning process.

[0030] Here, we have described the case where supervised learning is applied to the learning algorithm used by the model generation unit 42, but other learning methods may also be applied to the learning algorithm. The model generation unit 42 may perform machine learning using learning algorithms such as deep learning, genetic programming, inductive logic programming, or support vector machines.

[0031] The learning device 40 may be a device connected to the image stitching processing device 10 via a network. The learning device 40 may also be a device located on a cloud server.

[0032] Machine learning may also be performed using a convolutional neural network that includes convolutional layers and pooling layers. Alternatively, semantic segmentation may be performed, which classifies all pixels in an image to create a segmented image for each class.

[0033] The concatenation failure region detection unit 14 is an inference device that detects concatenation failure regions included in the concatenated image generated by the image concatenation processing unit 11, based on the trained model 43 which is the learning result of the concatenation failure region detection learning unit 12.

[0034] Figure 6 shows an example of the configuration of an inference device that realizes the image stitching processing device according to Embodiment 1, specifically the stitching failure region detection unit. The inference device 50 includes a data acquisition unit 51 and an inference unit 52, which are data acquisition units for inference.

[0035] The data acquisition unit 51 receives the concatenated image to be inferred as input; that is, the data acquisition unit 51 acquires the concatenated image to be inferred.

[0036] The inference unit 52 uses the trained model 43 stored in the trained model storage unit 13 to infer the concatenation disorder region in the concatenated image. That is, the inference unit 52 inputs the concatenated image acquired by the data acquisition unit 51 into the trained model 43 stored in the trained model storage unit 13 and performs inference, obtaining the concatenation disorder region as the inference result.

[0037] The inference device 50 is used to infer misaligned regions in the concatenated image, but it may be connected to the image concatenation processing device 10 via a network and operate as a separate device from the image concatenation processing device 10. Furthermore, the inference device 50 may reside on a cloud server.

[0038] Next, the inference process by the inference device 50 will be described. Figure 7 is a flowchart showing the procedure of the inference process by the linkage failure region detection unit of the image linkage processing device according to Embodiment 1. In step S21, the inference device 50 acquires a linked image using the data acquisition unit 51. In step S22, the inference device 50 inputs the linked image to the trained model 43 using the inference unit 52. In step S23, the inference unit 52 outputs the linkage failure region. With this, the inference device 50 completes the inference process according to the procedure shown in Figure 7.

[0039] In Embodiment 1, the inference device 50 is described as inferring the linkage mismatch region using a trained model 43 generated by a learning device 40 that constitutes the linkage mismatch region detection learning unit 12. However, the inference device 50 may acquire a trained model 43 generated outside the image linkage processing device 10 and use this trained model 43 to infer the linkage mismatch region.

[0040] The inference device 50 uses a pre-trained model 43, generated using multiple training datasets that include training concatenated images and mask images, which are ground truth data corresponding to the training concatenated images, to infer the concatenation-defect regions in the concatenated images to be inferred. Therefore, there is a correlation between the concatenated images to be inferred input to the inference device 50 and the concatenation-defect regions output from the inference device 50. For this reason, by using a pre-trained model 43 with high learning accuracy for concatenation-defect regions, the concatenation-defect regions in the concatenated images to be inferred can be inferred more accurately.

[0041] The detection result image generation unit 15 generates a detection result image by overlaying the linkage discrepancy region detected by the linkage discrepancy region detection unit 14 onto the linked image. For example, by overlaying the linkage discrepancy region as a monochrome silhouette onto the linked image, a detection result image can be generated that allows identification of which part of the linked image is the linkage discrepancy region. Although the method of overlaying the linkage discrepancy region as a monochrome silhouette is shown here, the linkage discrepancy region may also be overlaid on the linked image by displaying a dashed or dotted line frame that follows the outline of the linkage discrepancy region on the linked image.

[0042] The image stitching processing device 10 according to Embodiment 1 generates a trained model 43 for inferring stitching defects from stitched images using a model generation unit 42 that learns stitching defect regions in stitched images, thereby obtaining a training result that can identify stitching defect regions in stitched images. Furthermore, by inferring stitching defect regions using the trained model 43 learned by machine learning and generating a detection result image by overlaying the inferred stitching defect regions onto the stitched images, it is possible to easily identify which part of the stitched image is a stitching defect region.

[0043] In the first embodiment, the image stitching processing device 10 receives the stitched image as input to the stitching failure region detection unit 14. However, it is also possible to configure the device to simultaneously input the image before stitching in addition to the stitched image. By inputting the image before stitching in addition to the stitched image, it becomes easier to identify the stitching failure region based on the comparison result between the state before stitching and the state after stitching.

[0044] In the above description, an image stitching processing device 10 was described that includes both a stitching disorder detection learning unit 12 for learning stitching disorder regions and a stitching disorder detection unit 14 for detecting stitching disorder regions. However, the stitching disorder detection learning unit 12 and the stitching disorder detection unit 14 may be provided in separate devices. That is, the learning device 40 that constitutes the stitching disorder detection learning unit 12 and the inference device 50 that constitutes the stitching disorder detection unit 14 may each be implemented independently.

[0045] Embodiment 2. Figure 8 shows the configuration of the image linking processing device according to Embodiment 2. The image linking processing device 10 according to Embodiment 2 differs from the image linking processing device 10 according to Embodiment 1 in that it includes a learning data generation unit 16. The other parts are the same as those of the image linking processing device 10 according to Embodiment 1, so redundant explanations are omitted.

[0046] Figure 9 shows the configuration of the learning data generation unit of the image stitching processing device according to Embodiment 2. The learning data generation unit 16 comprises a stitching parameter noise addition unit 161, a stitching processing unit 162, and a stitching mismatch region label generation unit 163.

[0047] The concatenation parameter noise addition unit 161 generates noise-added concatenation parameters with values ​​different from the correct concatenation parameters in the image concatenation process. The concatenation processing unit 162 performs concatenation processing using the noise-added concatenation parameters generated by the concatenation parameter noise addition unit 161 and concatenation processing using the correct concatenation parameters. The concatenation mismatch region label generation unit 163 compares the noise-added concatenated image, which has been concatenated using the noise-added concatenation parameters, with the training concatenated image, which has been concatenated using the correct concatenation parameters, and generates a mask image that labels the concatenation mismatch regions.

[0048] Figure 10 shows the processing of the training data generation unit of the image concatenation processing device according to Embodiment 2. Performing concatenation processing using noise-adding concatenation parameters is equivalent to applying deformation to the still images before concatenation and then concatenating them. In the example shown in Figure 10, when concatenating two rectangular still images D1 and D2, the noise-adding concatenation parameters are equivalent to performing deformation by moving vertex A of still image D2 along edge AB in the direction of extending edge AB to become vertex A1, and then performing concatenation processing. In such deformation, the positions of vertices B, C, and D remain unchanged, so the entire still image is not uniformly deformed, and the amount of deformation increases closer to vertex A. Therefore, the difference between the training concatenated image processed using the correct concatenation parameters and the noise-adding concatenated image increases closer to vertex A. The concatenation mismatch region label generation unit 163 extracts pixels where the difference between the noise-adding concatenated image and the training concatenated image is greater than a preset judgment threshold as concatenation mismatch regions 61, and generates a mask image by superimposing the extracted concatenation mismatch regions 61 on top of the training concatenated image. Furthermore, the concatenation failure region 61 may include the area surrounding the concatenation failure region periphery 62, which is the area formed by pixels where the difference between the noisy concatenated image and the training concatenated image is greater than the judgment threshold. This is because the periphery of the image also appears as image displacement. In Figure 10, the concatenation failure region periphery 62 is shown as a dashed rectangle. By making the concatenation failure region 61 an area slightly larger than the area formed by pixels where the difference between the noisy concatenated image and the training concatenated image is greater than the judgment threshold, it becomes possible to detect displacement around the area formed by pixels where the difference between the noisy concatenated image and the training concatenated image is greater than the judgment threshold.

[0049] In this example, a noise-adding coupling parameter is used that corresponds to a deformation where vertex A is moved to extend edge AB. However, the noise-adding coupling parameter may correspond to a deformation other than that exemplified here. For example, the noise-adding coupling parameter may correspond to a deformation where vertex B is moved to extend edge AB, or to a deformation where vertex A is moved to shorten edge AB, or to vertex A and vertex D being moved to extend edges AB and DC. Thus, the noise-adding coupling parameter may correspond to a deformation that changes at least one of the area and shape of one or both of the concatenated still images.

[0050] Figure 11 shows the processing of the learning data generation unit of the image stitching processing device according to Embodiment 2. When stitching aerial images, noise can be added in the process of converting a still image corresponding to the ground plane into a still image for stitching. The method of adding noise in the process of converting a still image corresponding to the ground plane into a still image for stitching is the same as the example shown in Figure 10, and can be done by changing the parameters used in processing such as scaling, rotation, and projection transformation to values ​​different from the correct parameters. In addition, noise can also be added to the camera position and orientation in the process of converting aerial images into a still image corresponding to the ground plane. That is, by changing the camera position and orientation to values ​​different from the correct values ​​and performing the process of converting to a still image corresponding to the ground plane, the ground plane equivalent still images B11 and B21 converted by adding noise to the camera position and orientation become different still images from the ground plane equivalent still images B1 and B2 converted with the camera position and orientation set to the correct values. Therefore, the still images C11 and C21 used for stitching, which are based on the ground plane equivalent still images B11 and B21 transformed by adding noise to the camera's position and orientation, will also contain noise. As a result, the stitched image obtained by stitching together still images C11 and C21 will be a different image from the stitched image obtained by stitching together images without adding noise.

[0051] Aerial photography is performed using cameras mounted on mobile devices such as helicopters, balloons, and drones. However, since mobile devices flying in the air are affected by wind, errors are likely to occur in the camera's position and orientation. Therefore, by adding noise to the camera's position and orientation when generating training data, it is possible to generate training data that contains noise similar to the noise that occurs during aerial photography.

[0052] Identifying areas of poor image concatenation can be difficult by human intervention, and creating mask images for training data is not always easy. Therefore, using manually created mask images as training data takes time to prepare, and as a result, it takes time to improve the learning accuracy. The image concatenation processing device 10 according to Embodiment 2 does not require manual preparation of training data, so it can easily learn areas of poor image concatenation. For this reason, the image concatenation processing device 10 according to Embodiment 2 can easily improve the learning accuracy of areas of poor image concatenation.

[0053] Embodiment 3. The image linking processing device 10 according to Embodiment 3 is the same as the image linking processing device 10 according to Embodiment 2, and includes a training data generation unit 16. The configuration of the training data generation unit 16 in the image linking processing device 10 according to Embodiment 3 differs from that of the image linking processing device 10 according to Embodiment 2. The other parts are the same as those of the image linking processing device 10 according to Embodiment 2, so redundant explanations are omitted.

[0054] Figure 12 shows the configuration of the learning data generation unit of the image stitching processing device according to Embodiment 3. The learning data generation unit 16 comprises a pre-stitch image noise addition unit 164, a stitching processing unit 162, and a stitching mismatch region label generation unit 163.

[0055] The pre-concatenation still image noise addition unit 164 adds noise to at least one of the multiple still images before concatenation to generate a noise-added pre-concatenation still image. Examples of noise added to the still image by the pre-concatenation still image noise addition unit 164 include impulse noise and Gaussian noise.

[0056] The concatenation processing unit 162 performs concatenation processing using a noisy pre-concatenation still image to which noise has been added by the pre-concatenation still image noise addition unit 164, and concatenation processing using a pre-concatenation still image without added noise. The concatenation mismatch region label generation unit 163 compares the noisy concatenation image, which has been concatenated using a noisy pre-concatenation still image, with the training concatenation image, which has been concatenated using a pre-concatenation still image without added noise, and generates a mask image with concatenation mismatch regions labeled.

[0057] When concatenating images using noise-added still images, the locations of highly correlated positions or feature points extracted during the concatenation process change compared to when concatenating images using non-noise-added still images. As a result, continuous dissonant regions are formed in the noise-added concatenated image.

[0058] Similar to the image linking processing device 10 according to Embodiment 2, the image linking processing device 10 according to Embodiment 3 does not require manual preparation of training data, and therefore can easily learn regions of linking failure. For this reason, the image linking processing device 10 according to Embodiment 3, like the image linking processing device 10 according to Embodiment 2, can easily improve the learning accuracy of regions of linking failure.

[0059] Embodiment 4. The image linking processing device 10 according to Embodiment 4 is the same as the image linking processing device 10 according to Embodiment 2 and the image linking processing device 10 according to Embodiment 3, and includes a learning data generation unit 16. The configuration of the learning data generation unit 16 in the image linking processing device 10 according to Embodiment 4 differs from that of the image linking processing device 10 according to Embodiment 2 and the image linking processing device 10 according to Embodiment 3. Since the other parts are the same as those of the image linking processing device 10 according to Embodiment 2 and the image linking processing device 10 according to Embodiment 3, redundant explanations will be omitted.

[0060] Figure 13 shows the configuration of the learning data generation unit of the image stitching processing device according to Embodiment 4. The learning data generation unit 16 includes a stitching parameter noise addition unit 161, a pre-stitch image noise addition unit 164, a stitching processing unit 162, and a stitching mismatch region label generation unit 163.

[0061] When a region of misalignment occurs in a concatenated image, it may be caused not only by noise in the still image or noise in the concatenation parameters, but by both. The image concatenation processing device 10 according to Embodiment 4 creates training data by adding noise to both the concatenation parameters and the still image, so it can generate training data that contains noise similar to the noise that occurs in the concatenated image.

[0062] Identifying areas of poor linkage can be difficult by human intervention, making it challenging to create mask images for training data. Therefore, using manually created mask images for training data requires considerable time to improve learning accuracy. The image linking processing device 10 according to Embodiment 4, like the image linking processing device 10 according to Embodiment 2 and Embodiment 3, does not require manual preparation of training data, thus enabling easy learning of areas of poor linkage. Consequently, the image linking processing device 10 according to Embodiment 4, like the image linking processing device 10 according to Embodiment 2 and Embodiment 3, can easily improve the learning accuracy of areas of poor linkage.

[0063] Embodiment 5. Figure 14 shows the configuration of the image stitching device according to Embodiment 5. The image stitching device 10 according to Embodiment 5 differs from the image stitching device 10 according to Embodiment 1 in that it includes a stitching parameter changing unit 17. The stitching parameter changing unit 17 changes the stitching parameters of the stitching process when the size of the stitching defect region detected by the stitching defect region detection unit 14 is larger than a preset determination threshold. The other parts are the same as the image stitching device 10 according to Embodiment 1, so redundant explanations are omitted.

[0064] Figure 15 shows the relationship between the connection parameter changed by the connection parameter changing unit according to Embodiment 5 and the size of the connection failure region. The size of the connection failure region changes as the connection parameter changing unit 17 changes the connection parameter. The connection parameter changing unit 17 changes the connection parameter so that the connection failure region becomes larger, and terminates the change of the connection parameter when the size of the connection failure region becomes less than or equal to a preset threshold. In the example shown in Figure 15, the connection parameter is changed in the order P→Q→R→S, and since the size of the connection failure region at connection parameter S is less than or equal to the threshold, the change of the connection parameter is terminated.

[0065] Note that changing the linking parameter may result in a larger area of ​​mislinking than before the change. For example, in the example shown in Figure 15, changing the linking parameter from R to T results in a larger area of ​​mislinking than before the change. If the area of ​​mislinking becomes larger than before after changing the linking parameter, the linking parameter changing unit 17 changes the amount of change to the linking parameter and redoes the linking of the still images. For example, the linking parameter changing unit 17 changes the linking parameter from R to U and causes the image linking processing unit 11 to redo the linking of the still images. In this way, if the size of the area of ​​mislinking does not decrease even after changing the amount of change to the linking parameter, the area of ​​mislinking can be made smaller than before the change to the linking parameter by further changing the amount of change to the linking parameter and redoing the linking of the still images. Therefore, the image linking processing device 10 according to Embodiment 5 can converge the size of the area of ​​mislinking to a preset determination threshold or less.

[0066] Furthermore, the larger the overlap between the two still images before concatenation, the larger the area of ​​concatenation failure tends to be. For this reason, the threshold for determining the size of the concatenation failure area may be set as a ratio to the overlap between the two still images before concatenation.

[0067] The image stitching processing device 10 according to Embodiment 5 can obtain a stitched image in which the size of the stitching failure region is less than or equal to a preset threshold.

[0068] Embodiment 6. Figure 16 shows the configuration of the image concatenation processing device according to Embodiment 6. The image concatenation processing device 10 according to Embodiment 6 includes a concatenation parameter changing unit 17, similar to the image concatenation processing device 10 according to Embodiment 5 shown in Figure 14. However, the image concatenation processing device 10 according to Embodiment 6 differs from the image concatenation processing device 10 according to Embodiment 5 in that the learning data of the learning device 40 constituting the concatenation failure region detection learning unit 12 includes concatenation parameter errors, the trained model 43 includes concatenation parameter errors, and the output of the inference device 50 constituting the concatenation failure region detection unit 14 includes concatenation parameter errors. The other parts are the same as the image concatenation processing device 10 according to Embodiment 5, so redundant explanations are omitted. Note that the concatenation parameter error is the difference between the concatenation parameter and the parameter value when no concatenation failure region occurs.

[0069] The image stitching processing device 10 according to Embodiment 6's learning unit 12 for detecting stitching defects also learns stitching parameter errors. Therefore, the stitching defect detection unit 14 not only detects which part of the stitched image is a stitching defect region, but can also infer which stitching parameter is causing the stitching defect region. The stitching defect detection unit 14 outputs the inferred stitching parameter that is causing the stitching defect region to the stitching parameter modification unit 17. The stitching parameter modification unit 17 modifies the stitching parameter that is causing the stitching defect region based on the input from the stitching defect detection unit 14.

[0070] The connection malfunction area detection unit 14 may infer the item name of the connection parameter that is the cause of the connection malfunction area, or it may infer the difference between the parameter value when no connection malfunction area occurs and the parameter value when no connection malfunction area occurs. If the connection malfunction area detection unit 14 infers the item name of the connection parameter that is the cause of the connection malfunction area, the connection parameter modification unit 17, as in Embodiment 5, repeatedly modifies the parameter value of the connection parameter that was inferred to be the cause of the connection malfunction area until the size of the connection malfunction area is less than or equal to a preset determination threshold. If the connection malfunction area detection unit 14 infers the difference between the parameter value when no connection malfunction area occurs and the parameter value when no connection malfunction area occurs, the connection parameter modification unit 17 determines the amount of change to the connection parameter based on the input from the connection malfunction area detection unit 14.

[0071] The image stitching processing device 10 according to Embodiment 6 changes the stitching parameters and redoes the image stitching process if the size of the stitching failure region is larger than a preset threshold, thereby obtaining a stitched image in which the size of the stitching failure region is less than or equal to the preset threshold. Therefore, when using the stitched image output from the image stitching processing device 10, it is possible to suppress the occurrence of problems caused by the existence of stitching failure regions larger than a preset threshold. In addition, since the stitching failure region detection unit 14 infers the item name of the stitching parameter that is causing the stitching failure region, or the difference between the parameter value when no stitching failure region occurs and the actual stitching failure region, it is less likely that the amount of change in the stitching parameters will be changed and the stitching of still images will be redone. Therefore, compared to the image stitching processing device 10 according to Embodiment 5, the image stitching processing device 10 according to Embodiment 6 is more likely to converge the size of the stitching failure region to less than or equal to a preset determination threshold.

[0072] Next, the hardware configurations of the image stitching processing device 10, learning device 40, and inference device 50 according to Embodiments 1 to 6 of this disclosure will be described. The image stitching processing device 10 is realized by executing a program on a computer system that describes the processing performed by the image stitching processing device 10. When either the learning device 40 or the inference device 50 is implemented independently, the learning device 40 or the inference device 50 is realized by executing a program on a computer system that describes the processing performed by the learning device 40 or the inference device 50.

[0073] Figure 17 is a diagram showing the configuration of a computer system for realizing the functions of the image stitching processing device, learning device, and inference device according to Embodiments 1 to 6. As shown in Figure 17, the computer system 91 comprises a processor 92 and memory 93. The processor 92 is a CPU (Central Processing Unit), also called a central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, DSP (Digital Signal Processor), etc. The memory 93 is, for example, a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Registered Trademark) (Electrically EPROM), magnetic disk, flexible disk, optical disk, compact disk, minidisc, DVD (Digital Versatile Disk), etc.

[0074] The computer system 91 described above realizes the functions of the image stitching device 10, the learning device 40, or the inference device 50 by having the processor 92 read and execute programs corresponding to the processing of each component stored in memory 93. Memory 93 is also used as temporary memory for each process performed by the processor 92. The programs executed by the processor 92 may be provided stored on a storage medium or provided via a network.

[0075] The configurations shown in the above embodiments are merely examples of the content, and can be combined with other known technologies. It is also possible to omit or modify parts of the configuration without departing from the gist of the invention. [Explanation of symbols]

[0076] 10 Image stitching processing unit, 11 Image stitching processing unit, 12 Stitching defect region detection learning unit, 13 Trained model storage unit, 14 Stitching defect region detection unit, 15 Detection result image generation unit, 16 Training data generation unit, 17 Stitching parameter modification unit, 40 Learning device, 41, 51 Data acquisition unit, 42 Model generation unit, 43 Trained model, 50 Inference device, 52 Inference unit, 61 Stitching defect region, 87 Grassland, 88 Road, 91 Computer system, 92 Processor, 93 Memory, 95 Helicopter, 161 Stitching parameter noise addition unit, 162 Stitching processing unit, 163 Stitching defect region label generation unit, 164 Pre-stitch still image noise addition unit, 201 Input image, 202 Convolutional neural network, 203 Deconvolutional neural network, 204 Output image.

Claims

1. A learning device comprising: a learning data acquisition unit that acquires learning data including a learning concatenated image generated by stitching together multiple learning still images, and a mask image that labels the concatenation region, which is a part of the learning concatenated image where the state of the subject differs from that of the still image before concatenation; and a model generation unit that generates a trained model for inferring the concatenation region in the concatenated image using multiple learning datasets, each containing the learning concatenated image and the mask image, which is the correct answer data corresponding to the learning concatenated image. An image concatenation processing unit that concatenates still images to generate a concatenated image for inference, An inference device comprising: an inference target data acquisition unit that acquires the concatenated image to be inferred; and an inference unit that outputs the concatenation mismatch region in the concatenated image to be inferred using the trained model; The system includes a linking parameter modification unit that modifies the linking parameters when the image linking processing unit generates the linked image to be inferred if the size of the linking failure region output by the inference device is greater than a preset threshold, An image merging apparatus characterized by repeatedly generating the merging target merging image and changing the merging parameters until the merging defect region in the merging image output by the inference unit falls below a preset threshold.

2. The steps include acquiring training data that includes a concatenated training image generated by stitching together multiple still training images, and a mask image that labels the concatenation failure regions in the concatenated training image, which are areas where the state of the subject differs from the still images before concatenation; The steps include generating a trained model for inferring the concatenation region in a concatenated image using multiple training datasets, each of which includes the training concatenated image and the mask image, which is the correct data corresponding to the training concatenated image; The steps include:

1. Concatenating still images to generate a concatenated image for inference; A data acquisition unit for inference that acquires the concatenated image to be inferred, and a step of outputting the concatenation failure region in the concatenated image to be inferred using the trained model, The step of outputting the concatenation failure region in the concatenated image to be inferred includes a step of changing the concatenation parameters when generating the concatenated image to be inferred if the size of the concatenation failure region output is greater than a preset threshold. An image stitching method characterized by repeating the steps of generating the stitched image to be inferred and changing the stitching parameters until the number of stitching errors in the output stitched image is less than or equal to a preset threshold, in the step of outputting the stitching error region in the stitched image to be inferred.

3. In a computer system, The steps include acquiring training data that includes a concatenated training image generated by stitching together multiple still training images, and a mask image that labels the concatenation failure regions in the concatenated training image, which are areas where the state of the subject differs from the still images before concatenation; The steps include generating a trained model for inferring the concatenation region in a concatenated image using multiple training datasets, each of which includes the training concatenated image and the mask image, which is the correct data corresponding to the training concatenated image; The steps include:

1. Concatenating still images to generate a concatenated image for inference; A data acquisition unit for inference that acquires the concatenated image to be inferred, and a step of outputting the concatenation failure region in the concatenated image to be inferred using the trained model, In the step of outputting the concatenation failure region in the concatenated image to be inferred, if the size of the concatenation failure region output is greater than a preset threshold, the concatenation parameters used when generating the concatenated image to be inferred are to be executed. A program characterized by repeatedly performing the steps of generating the concatenated image to be inferred and changing the concatenation parameters until the concatenation region in the concatenated image to be inferred is less than or equal to a preset threshold, in the step of outputting the concatenation region in the concatenated image to be inferred.

Images