Feature map generation device, image matching system, feature map generation method, and program

Abstract

To perform convolution operation using a kernel which can be deformed as intended.SOLUTION: A feature map generation apparatus includes: a kernel deformation method determination unit which determines a plurality of deformation methods for a kernel corresponding to a target image, using geometric transformation on the target image; a kernel deformation unit which generates a plurality of deformed kernels by deforming a criterial kernel with the deformation methods; a convolution operation unit which extracts feature quantities in the target image by convolution operation on the target image, using the deformed kernels, to generate a plurality of feature maps corresponding to the target image using the extracted feature quantities; a feature integration method determination unit which determines an integration method for integrating the feature maps generated by the convolution operation unit; and a feature map integration unit which generates an integrated feature map by integrating the feature maps with the integration method determined by the feature integration method determination unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to a feature map generation device, an image matching system, a feature map generation method, and a program. [Background technology]

[0002] Machine learning is beginning to be used in image matching. In image matching, features are extracted from an image, and the similarity between two images is calculated by comparing the extracted features with those of other images, thereby matching (associating) the two images. Image matching technology is used for identifying objects in images, segmentation, obtaining three-dimensional information through stereo matching, and image generation.

[0003] For example, Non-Patent Document 1-2 discloses a technique for extracting features from an image. In Non-Patent Document 1-2, feature extraction is performed by concatenating convolution operations. The kernel used in the convolution operation is a square grid. When a convolution operation is performed using a square grid kernel, the extracted features are not robust to the imaging angle.

[0004] As a countermeasure, one could prepare a large number of images taken at different imaging angles as training data and have a machine learning model learn how to handle changes in imaging angle. Non-patent document 3 proposes a method for training a machine learning model to modify the kernel used in convolution operations. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Karen Simonyan, Andrew Zisserman, "VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION," [online], [Accessed June 30, 2022], Internet<URL:https: / / arxiv.org / pdf / 1409.1556.pdf> . [Non-Patent Document 2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, "Deep Residual Learning for Image Recognition," [online], [Accessed June 7, 2022], Internet<URL:https: / / arxiv.org / pdf / 1512.03385.pdf> . [Non-Patent Document 3] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei, "Deformable Convolutional Networks", [online], [Retrieved June 7, 2022], Internet <URL:https: / / openaccess.thecvf.com / content_ICCV_2017 / papers / Dai_Deformable_Convolutional_Networks_ICCV_2017_paper.pdf> . [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] However, in the method described in Non-Patent Document 3, while the kernel deformation method is learned, there is a risk that an unintended deformation method may be estimated as a result of the learning, and a deformation method that is not robust to the imaging angle may be estimated. For example, when the imaging angle changes, it is expected that a method of performing a projection transformation according to the imaging angle will be estimated as the kernel deformation method, but the offset amount of the square grid may be estimated, and the result may not be a projection transformation.

[0007] The present invention has been made based on the above-mentioned problems, and aims to provide a feature map generation device, an image matching system, a feature map generation method, and a program that can perform convolution operations using a kernel that allows for intended deformation. [Means for solving the problem]

[0008] The feature map generation device of the present invention is a feature map generation device that generates feature maps for one or more target images, comprising: a kernel deformation method determination unit that determines a plurality of deformation methods for a kernel corresponding to the target image using geometric transformations for the target image; a kernel deformation unit that generates a plurality of deformed kernels, which are kernels used for convolution operations on the target image, by deforming a reference kernel using the deformation methods; a convolution operation unit that extracts feature quantities from the target image by performing a convolution operation on the target image using the deformed kernels, and generates a plurality of feature maps corresponding to the target image using the extracted feature quantities; a feature integration method determination unit that determines an integration method for integrating the plurality of feature maps generated by the convolution operation unit; and a feature map integration unit that generates an integrated feature map by integrating the plurality of feature maps using the integration method determined by the feature integration method determination unit.

[0009] The image matching system of the present invention matches images using an integrated feature map generated by the feature map generation device described above.

[0010] The present invention relates to a feature map generation method performed by a feature map generation device that generates feature maps for one or more target images, wherein a kernel deformation method determination unit determines a plurality of deformation methods for a kernel corresponding to the target image using geometric transformations for the target image, a kernel deformation unit generates a plurality of deformed kernels which are kernels used for convolution operations on the target image by deforming a reference kernel using the deformation methods, a convolution operation unit extracts feature quantities from the target image by performing a convolution operation on the target image using the deformed kernels, generates a plurality of feature maps corresponding to the target image using the extracted feature quantities, a feature integration method determination unit determines an integration method for integrating the plurality of feature maps generated by the convolution operation unit, and a feature map integration unit generates an integrated feature map by integrating the plurality of feature maps using the integration method determined by the feature integration method determination unit.

[0011] The present invention is a program that causes a feature map generation device, which generates feature maps for one or more target images, to generate feature maps, wherein the device determines multiple methods for transforming a kernel corresponding to the target image using geometric transformations, transforms a reference kernel using each of the transformation methods to generate multiple transformed kernels which are kernels used for convolution operations on the target image, extracts feature quantities from the target image by performing a convolution operation on the target image using the transformed kernels, generates multiple feature maps corresponding to the target image using the extracted feature quantities, determines a method for integrating the multiple feature maps, and generates an integrated feature map by integrating the multiple feature maps using the integration method. [Effects of the Invention]

[0012] According to the present invention, convolution operations can be performed using a kernel that allows for the intended deformation. [Brief explanation of the drawing]

[0013] [Figure 1] It is a block diagram showing a configuration common to the feature map generation device 1 of the embodiment. [Figure 2] It is a diagram for explaining the processing performed by the feature map generation device 1 of the embodiment. [Figure 3] It is a diagram for explaining the processing performed by the feature map generation device 1 of the embodiment. [Figure 4] It is a diagram for explaining the processing performed by the feature map generation device 1 of the embodiment. [Figure 5A] It is a diagram for explaining the processing according to the embodiment. [Figure 5B] It is a diagram for explaining the processing according to the embodiment. [Figure 6] It is a block diagram showing the configuration of the feature map generation device 1 of the first embodiment. [Figure 7A] It is a diagram for explaining the processing according to Examples 1 to 2 and Examples 4 to 5 of the first embodiment. [Figure 7B] It is a diagram for explaining the processing according to Examples 1 to 2 and Examples 4 to 5 of the first embodiment. [Figure 8A] It is a diagram for explaining the processing according to Example 3 of the first embodiment. [Figure 8B] It is a diagram for explaining the processing according to Example 3 of the first embodiment. [Figure 9A] It is a block diagram showing the configuration of the feature map generation device 1 of the second embodiment. [Figure 9B] It is a diagram for explaining the processing according to the second embodiment. [Figure 9C] It is a diagram for explaining the processing according to the second embodiment. [Figure 10A] It is a block diagram showing the configuration of the feature map generation device 1 of the third embodiment. [Figure 10B] It is a diagram for explaining the processing according to the third embodiment. [Figure 10C] It is a diagram for explaining the processing according to the third embodiment. [Figure 11A] It is a block diagram showing the configuration of the feature map generation device 1 of the fourth embodiment. [Figure 11B] This diagram illustrates the process according to the fourth embodiment. [Figure 11C] This diagram illustrates the process according to the fourth embodiment. [Figure 12A] This block diagram shows the configuration of the feature map generation device 1 of the fifth embodiment. [Figure 12B] This diagram illustrates the process according to the fifth embodiment. [Figure 12C] This diagram illustrates the process according to the fifth embodiment. [Figure 12D] This diagram illustrates the process according to the fifth embodiment. [Figure 12E] This diagram illustrates the process according to the fifth embodiment. [Modes for carrying out the invention]

[0014] The feature map generation device 1 of the embodiment will be described below with reference to the drawings. The following sections will describe examples of how the feature map generation device 1 can be applied to the following multiple embodiments.

[0015] First Embodiment: Classification System Second Embodiment: Stereo Matching System Third Embodiment: Single-View Depth Map Generation System Fourth Embodiment: Character Recognition System Fifth Embodiment: Style Transfer System

[0016] In the first embodiment, an embodiment is described in which the feature map generation device 1 is applied to a class classification system that classifies the entire image or a part of an image into classes. For example, the class classification system can be used to recognize whether an animal captured in an image is a cat or to identify the pixels in which a cat is captured. In the second embodiment, we will describe an embodiment in which the feature map generation device 1 is applied to a stereo matching system that generates a depth map from stereo images of the same object taken from different viewpoints. In the third embodiment, we will describe an embodiment in which the feature map generation device 1 is applied to a single-view depth map generation system that generates a depth map from a single image. In the fourth embodiment, an embodiment in which the feature map generation device 1 is applied to a character recognition system that recognizes characters shown in an image will be described. In the fifth embodiment, we will describe an embodiment in which the feature map generation device 1 is applied to a style transfer system that generates a post-transfer image by transferring the style shown in the style image to the destination image.

[0017] <Configuration common to each embodiment> First, the configuration common to each embodiment will be explained using Figures 1 to 4. In the configuration common to each embodiment, the feature map generation device 1 performs feature extraction and feature map generation using a deformed kernel on the image to be used for generating the feature map (referred to as the target image or target viewpoint image). In the feature map generation device 1, the deformed kernel is generated by deforming the reference kernel according to rules determined by the deformation method. As a result, in the feature map generation device 1 of this embodiment, the deformation of the kernel itself can be excluded from the learning of machine learning. Therefore, the method for deforming the kernel does not fall into local optima, and the intended deformation can be performed. This makes it possible to improve robustness to the imaging angle.

[0018] Figure 1 is a block diagram showing an example of the configuration of the feature map generation device 1 according to an embodiment. Figure 1 also shows an example of the basic configuration 10, which is common to each embodiment.

[0019] As shown in Figure 1, the feature map generation device 1 includes, for example, a kernel deformation unit 101, a convolution operation unit 102, a feature integration method determination unit 103, a feature map integration unit 104, an image storage unit 105, a feature map storage unit 106, a reference kernel storage unit 107, and a deformed kernel storage unit 108.

[0020] The kernel deformation method determination unit 100 determines a deformation method for deforming the kernel. Here, the kernel is an array in which convolution coefficients are associated with two-dimensional image coordinates (see, for example, equation (5)). The kernel is used when performing convolution operations with each pixel in the image.

[0021] Here, using Figures 2 and 3, we will explain the process by which the kernel deformation method determination unit 100 determines the kernel deformation method. Figures 2 and 3 are diagrams illustrating the process performed by the feature map generation device 1 of this embodiment.

[0022] Figure 2 schematically shows an example where the target object is imaged from different viewpoints (View1 to View3). The camera parameters at viewpoint View1 are {K1, R1, t1}. The camera parameters at viewpoint View2 are {K2, R2, t2}. The camera parameters at viewpoint View3 are {K3, R3, t3}. Here, K represents the intrinsic parameter of the camera parameters. R represents the rotation matrix of the camera parameters. t represents the translation vector of the camera parameters.

[0023] Furthermore, for an object, a pair {M, n} can be defined consisting of a three-dimensional coordinate M on the object's surface and a normal direction n.

[0024] As shown in Figure 2, the appearance of an object changes depending on the camera's imaging position and orientation (imaging angle) in a multi-view image. A multi-view image here refers to multiple images of the same object taken from different viewpoints (imaging position and orientation). For example, when the object is imaged from viewpoint View1, an image of the object seen from the front is captured. When the object is imaged from viewpoint View2, an image is captured that is inverted vertically and horizontally compared to viewpoint View1, and is also enlarged. When the object is imaged from viewpoint View3, an image of the object seen from an oblique angle is captured.

[0025] Thus, in multi-view images of the same object captured from different viewpoints, image deformations other than translation, such as rotation, scaling, and projection transformation, often occur between images. When image deformations other than translation occur between images, the results of the convolution operation (features) extracted from each image using a general convolution operation will have different values ​​at corresponding points in each image, resulting in a significant decrease in the accuracy of three-dimensional reconstruction.

[0026] In contrast, in each embodiment, the kernel used for the convolution operation is deformed according to the image deformation, and the convolution operation is performed on the image using the deformed kernel (deformed kernel described later). By doing this, even if the image deformation between images is large, the calculation results (features) of the convolution operation extracted from each image can be made to have similar values ​​at corresponding points in each image, thereby suppressing a decrease in the accuracy of three-dimensional reconstruction.

[0027] The kernel deformation method determination unit 100 determines a method for deforming the kernel according to the image deformation. Specifically, the kernel deformation method determination unit 100 determines a method for deforming the kernel used in the convolution operation according to the relationship between the viewpoint in the image and the set {M, n} of three-dimensional coordinates M and their normal direction n on the surface of the target object. The following describes the process by which the kernel deformation method determination unit 100 determines a method for deforming the kernel used in the convolution operation.

[0028] Figure 3 schematically shows an example of a deformed kernel. The kernel deformation method determination unit 100 determines a deformation method for deforming the common kernel such that the two-dimensional coordinates set using, for example, a square grid in the common kernel become the two-dimensional coordinates of the corresponding points in each image.

[0029] For example, the kernel deformation method determination unit 100 determines a deformation method corresponding to the viewpoint View1 such that each of the two-dimensional coordinates P00, P10, P20, and P30 in the reference kernel is transformed into two-dimensional coordinates P01, P11, P21, and P31 corresponding to the positions of the right eye, left eye, right corner of the mouth, and left corner of the mouth of the target object, respectively, in the viewpoint View1.

[0030] The kernel deformation method determination unit 100 determines a deformation method corresponding to the viewpoint View2 such that each of the two-dimensional coordinates P00, P10, P20, and P30 in the reference kernel is transformed into the respective two-dimensional coordinates P02, P12, P22, and P32 of the corresponding points in the viewpoint View2.

[0031] The kernel deformation method determination unit 100 determines a deformation method corresponding to the viewpoint View3 such that each of the two-dimensional coordinates P00, P10, P20, and P30 in the reference kernel is transformed into the respective two-dimensional coordinates P03, P13, P23, and P33 of the corresponding points in the viewpoint View3.

[0032] Furthermore, the kernel deformation method determination unit 100 may determine the deformation method according to the image deformation between the two images.

[0033] For example, the kernel deformation method determination unit 100 determines a deformation method such that the coordinates in viewpoint View1 are transformed into coordinates where the corresponding points in viewpoint View2 exist. Specifically, the kernel deformation method determination unit 100 determines a deformation method corresponding to viewpoint View2 such that the two-dimensional coordinates P01, P11, P21, and P31 in viewpoint View1 are transformed into the corresponding points in viewpoint View2, i.e., the two-dimensional coordinates P02, P12, P22, and P32. The kernel deformation method determination unit 100 determines a deformation method corresponding to viewpoint View3 such that the two-dimensional coordinates P01, P11, P21, and P31 in viewpoint View1 are transformed into the corresponding points in viewpoint View3, i.e., the two-dimensional coordinates P03, P13, P23, and P33.

[0034] Returning to Figure 1, the kernel deformation method determination unit 100 obtains camera parameters corresponding to each viewpoint in the image to be convolutional (target viewpoint image) from the image storage unit 105. The kernel deformation method determination unit 100 obtains the coordinates (position coordinates) and normal directions of the three-dimensional points from the image storage unit 105. The kernel deformation method determination unit 100 determines the kernel deformation method corresponding to the target viewpoint image from the camera parameters corresponding to each viewpoint, the coordinates and normal directions of the three-dimensional points.

[0035] If, for example, projection transformation is used as a method for transforming the kernel, the kernel transformation method determination unit 100 calculates the projection transformation matrix H. The projection transformation matrix H can be calculated using equation (1).

[0036]

number

[0037] Here, the coordinates M and normal direction n of the three-dimensional point in equation (1) are given in the camera coordinate system of the reference viewpoint image. The reference viewpoint image here is an image in which the same object as the target viewpoint image is captured, but it is a different image from the target viewpoint image. The projection transformation matrix H shown in equation (1) corresponds to a coordinate transformation matrix that transforms the two-dimensional image coordinates in the reference viewpoint image to the two-dimensional image coordinates in the target viewpoint image via a three-dimensional plane defined by the coordinates M and normal direction n of the three-dimensional point.

[0038] By performing a convolution operation using a kernel deformed by the kernel deformation method determined in this way, when the coordinates and normal directions of a three-dimensional point are close to those of the actual target surface, the change in feature quantities due to image deformation between multi-view images is reduced, and similar feature quantities can be extracted at corresponding points in each viewpoint even in multi-view images with large image deformation.

[0039] The kernel deformation method determination unit 100 determines the kernel deformation method using, for example, the camera parameters of the reference viewpoint image and the camera parameters of the target viewpoint image, which are camera parameters at two viewpoints. Alternatively, the kernel deformation method determination unit 100 may determine the kernel deformation method using only the camera parameters of the target viewpoint image, that is, the camera parameters at a single viewpoint.

[0040] When using only the camera parameters of one viewpoint in the target viewpoint image, the kernel deformation method determination unit 100 uses the coordinates and normal direction of a three-dimensional point given in the camera coordinate system in the target viewpoint image.

[0041] When using only camera parameters from a single viewpoint, for example, the kernel deformation method determination unit 100 determines a projection transformation matrix as the kernel deformation method that rotates the plane facing the camera at the target viewpoint in the direction of a given normal. When using only camera parameters from a single viewpoint, there is an advantage that feature extraction can be performed independently using each viewpoint image in a multi-view image, that is, independently of the reference viewpoint image.

[0042] On the other hand, using camera parameters from two viewpoints—camera parameters from a reference viewpoint image and camera parameters from a target viewpoint image—has the advantage of being able to extract consistent features across multiple viewpoints.

[0043] The kernel transformation method determination unit 100 reads, for example, the coordinates and normal directions of a pair of three-dimensional points from the image storage unit 105 for the camera parameters of a pair of reference viewpoint images and the camera parameters of a target viewpoint image, and determines a projection transformation matrix.

[0044] Alternatively, the kernel deformation method determination unit 100 may read the coordinates and normal directions of different three-dimensional points for each pixel in the target viewpoint image from the image storage unit 105 and determine a different projection transformation matrix for each pixel in the target viewpoint image.

[0045] Generally, the three-dimensional coordinates and normal directions of an object on its surface differ for each pixel in the viewpoint image. Therefore, determining a different projection transformation matrix for each pixel using the coordinates and normal directions of the three-dimensional points that differ for each pixel is likely to result in a smaller final three-dimensional reconstruction error. On the other hand, transforming the kernel using different transformation methods for each pixel increases the computational cost. For this reason, using a single transformation method per viewpoint can shorten processing time.

[0046] The kernel transformation method determination unit 100 reads, for example, the camera parameters of a set of reference viewpoint images, the camera parameters of a target viewpoint image, and the coordinates of a three-dimensional point from the image storage unit 105, and determines a projection transformation matrix.

[0047] Alternatively, the kernel deformation method determination unit 100 may read two or more normal directions from the image storage unit 105 for a set of camera parameters of a reference viewpoint image, camera parameters of a target viewpoint image, and coordinates of a three-dimensional point, and determine multiple projection transformation matrices corresponding to each normal direction.

[0048] Generally, the more normal directions are considered, the higher the probability that the feature map calculated by the convolution unit 102 (described later) will contain feature maps that are consistent with the multi-viewpoint feature map, each corresponding to one of the multiple normal directions. On the other hand, the fewer normal directions are considered, the lower the computational cost.

[0049] The kernel deformation unit 101 generates a deformed kernel. The kernel deformation unit 101 obtains a reference kernel, for example, by reading the reference kernel from the reference kernel storage unit 107. The kernel deformation unit 101 generates a deformed kernel by deforming the reference kernel according to the kernel deformation method determined by the kernel deformation method determination unit 100. The kernel deformation unit 101 writes the generated deformed kernel to the deformed kernel storage unit 108.

[0050] For example, if the kernel deformation method is a projection transformation represented by the projection transformation matrix H, the kernel deformation unit 101 generates the deformed kernel by deforming the base kernel using equation (2).

[0051]

number

[0052] The kernel deformation unit 101 may, when given multiple different kernel deformation methods, deform the same reference kernel using each of the different kernel deformation methods, thereby generating multiple different deformed kernels.

[0053] In other words, the kernel deformation unit 101 generates different deformed kernels for the same reference kernel depending on the combination of the target viewpoint image, the coordinates of the three-dimensional points, and the normal direction. Furthermore, the kernel deformation unit 101 generates deformed kernels tailored to each target viewpoint image, the coordinates of the three-dimensional points, and the normal direction using the same common reference kernel for different target viewpoint images, different coordinates of the three-dimensional points, and different normal directions.

[0054] Even if the reference kernel has integer two-dimensional image coordinates arranged in a square grid, depending on the transformation method, the two-dimensional image coordinates of the transformed kernel may be real numbers. In response to this, the kernel transformation unit 101 may generate a transformed kernel with integer two-dimensional image coordinates by interpolating the transformed kernel with real coordinates using integer two-dimensional image coordinates arranged in a square grid. Here, a transformed kernel with real two-dimensional image coordinates is an example of a "temporary transformed kernel". For example, bilinear interpolation can be used as the interpolation method.

[0055] Here, using Figure 4, we will explain an example in which the kernel deformation unit 101 generates a deformed kernel having integer two-dimensional image coordinates. Figure 4 is a diagram illustrating the processing performed by the feature map generation device 1 of this embodiment.

[0056] In Figure 4, the second from the left for each viewpoint (View 1 to View 3) schematically shows an example of mapping the deformed kernel corresponding to each viewpoint onto a 5x5 square grid. As shown in this example, when the two-dimensional image coordinates of the deformed kernel are real numbers, the two-dimensional image coordinates of the deformed kernel are mapped to any point on the square grid.

[0057] In Figure 4, the third from the left for each viewpoint (View 1 to View 3) schematically shows an example in which a transformed kernel with integer two-dimensional image coordinates is generated by applying bilinear interpolation to a transformed kernel with real number coordinates.

[0058] This allows the transformed kernel to have integer two-dimensional image coordinates, similar to the kernels used in general convolution operations. Therefore, convolution operations using the transformed kernel can be performed using general convolution processing.

[0059] If the transformed kernel has real coordinates, the convolution operation performed by the convolution unit 102 (described later) requires interpolation for each pixel involved in the convolution operation, resulting in a high computational cost. In contrast, by making the transformed kernel have integer coordinates, the need for interpolation for each pixel is eliminated. Therefore, the processing time required by the convolution unit 102 can be reduced.

[0060] The convolution unit 102 performs a convolution operation. The convolution unit 102 reads and obtains a deformed kernel from, for example, the deformed kernel storage unit 108. The convolution unit 102 also reads and obtains the target viewpoint image from the image storage unit 105. The convolution unit 102 performs a convolution operation on the target viewpoint image using the deformed kernel. The convolution unit 102 generates a feature map that associates the results of the convolution operation with the target viewpoint image as features. The convolution unit 102 stores the generated feature map in the feature map storage unit 106.

[0061] By performing convolution operations using the deformed kernel, even when there is significant image deformation between multi-view images, the degree to which the feature quantities (the result of the convolution operation) change due to image deformation is reduced, and the feature quantities extracted for each corresponding view from each viewpoint become similar in value.

[0062] If the deformed kernel is a kernel with integer two-dimensional image coordinates arranged in a square grid, the convolution unit 102 performs a convolution operation on the target viewpoint image using a general convolution operation. On the other hand, if the deformed kernel is a kernel with real two-dimensional image coordinates, the convolution unit 102 calculates, for example, the pixel value of a real two-dimensional image coordinate in the target viewpoint image by interpolating the pixel values ​​of integer two-dimensional image coordinates surrounding it, and then performs a convolution operation on the calculated pixel value of a real two-dimensional image coordinate using a kernel with real two-dimensional image coordinates.

[0063] For example, bilinear interpolation can be used as an interpolation method. Here, in the convolution operation, performing the convolution operation on the pixel values ​​of the real two-dimensional image coordinates obtained by interpolation using a kernel with real two-dimensional image coordinates, and interpolating the kernel with real two-dimensional image coordinates in an integer coordinate system and then performing the convolution operation on the pixel values ​​of the integer two-dimensional image coordinates using the interpolated kernel, will yield the same result.

[0064] If the deformed kernel is a single kernel determined for a single target viewpoint image, the convolution unit 102 performs the convolution operation using the same deformed kernel common to the target viewpoint images.

[0065] On the other hand, if the deformed kernel is a different deformed kernel determined for each pixel in the target viewpoint image, the convolution unit 102 performs the convolution operation using the different deformed kernels for each pixel in the target viewpoint image.

[0066] The convolution unit 102 may perform a convolution operation on a single target viewpoint image using multiple deformed kernels to generate a feature map. For example, the convolution unit 102 may read multiple deformed kernels from the deformed kernel storage unit 108, read a single target viewpoint image from the multi-view image storage unit, and then sequentially perform a convolution operation on the read target viewpoint image using multiple different deformed kernels to generate a feature map. In this case, multiple feature maps corresponding to each of the multiple deformed kernels are generated for a single target viewpoint image.

[0067] The feature integration method determination unit 103 determines a method (integration method) for integrating the multiple feature maps generated by the convolution operation unit 102. The feature integration method determination unit 103 determines the integration method so that features whose conditions (for example, normal direction) set when extracting features from the target image are close to those of the actual target object are emphasized.

[0068] For example, the feature integration method determination unit 103 determines the integration method by determining the weight coefficients of each feature map. By setting the conditions when extracting features from the target image so that the weight coefficients of feature maps that are closer to the actual target object are large, it is possible to generate an integrated feature map in which feature maps that are closer to the actual target object are emphasized. The specific method by which the feature integration method determination unit 103 determines the weight coefficients of each feature map will be explained in the first embodiment described later.

[0069] Furthermore, the feature integration method determination unit 103 may determine a method for selecting the maximum or minimum value as the integration method. This allows the map containing the feature quantity with the maximum or minimum value among multiple feature maps to be used as the integrated feature map.

[0070] The feature map integration unit 104 generates an integrated feature map. The feature map integration unit 104 integrates the multiple feature maps generated by the convolution operation unit 102 using a method determined by the feature integration method determination unit 103. The feature map integration unit 104 generates the integrated feature map obtained by integrating the multiple feature maps as an integrated feature map. For example, the feature map integration unit 104 generates the integrated feature map using weight coefficients determined by the feature integration method determination unit 103. For example, if the weight coefficients for feature maps A, B, and C are α, β, and γ, the integrated feature map is generated by multiplying each feature map as a weighted sum by the weight coefficients and combining them, i.e., αA + βB + γC. Note that the weight coefficients may be determined on an image (feature map) basis or on a pixel basis. The feature map integration unit 104 stores the integrated feature map in the feature map storage unit 106.

[0071] The image storage unit 105 stores multiple target viewpoint images to be used for generating feature maps, and camera parameters corresponding to each viewpoint in those target viewpoint images.

[0072] Furthermore, the image storage unit 105 stores a pre-prepared set of three-dimensional point coordinates {M} and a set of normals {n}. The image storage unit 105 may also use a set of three-dimensional point coordinates and normals selected from the set of three-dimensional coordinates and normals to determine the kernel deformation method.

[0073] Here, the set of coordinates {M} of three-dimensional points that are prepared in advance is given by, for example, equation (3).

[0074]

number

[0075] Furthermore, the pre-defined group of normals {n} is given by, for example, equation (4).

[0076]

number

[0077] The feature map storage unit 106 stores multiple feature maps generated by the convolution operation unit 102. Here, the feature map storage unit 106 may also store the feature map associated with the viewpoint corresponding to the source image from which the feature map was generated, as well as the coordinates and normal direction of the three-dimensional points used when generating the feature map.

[0078] Furthermore, the feature map storage unit 106 stores the integrated feature map generated by the feature map integration unit 104. The feature map storage unit 106 may also store the integrated feature map in association with the viewpoint corresponding to the image from which the integrated feature map was generated, as well as the coordinates and normal directions of the three-dimensional points used when generating the integrated feature map.

[0079] The reference kernel storage unit 107 stores a reference kernel. The reference kernel is a kernel that serves as the basis for generating the transformed kernel, and is represented, for example, by a set of pairs of two-dimensional image coordinates and weight coefficients. The reference kernel is shown, for example, by equation (5). The reference kernel storage unit 107 may store multiple reference kernels that are different from each other.

[0080]

number

[0081] The modified kernel storage unit 108 stores the modified kernel. The modified kernel is a kernel generated by the kernel modification unit 101, which modifies the reference kernel using the method determined by the kernel modification method determination unit 100. The modified kernel, like the reference kernel, is represented, for example, as a set of pairs of two-dimensional image coordinates and weight coefficients. The modified kernel is shown, for example, by equation (6).

[0082]

number

[0083] The two-dimensional image coordinates of the deformed kernel shown in equation (6) may be constructed as integer coordinates arranged in a square grid, for example, or as real number coordinates. When the kernel deformation unit 101 deforms the reference kernel, if the reference kernel is deformed while interpolating with integer coordinates arranged in a square grid, the number of elements K' and weight coefficient (a' of the deformed kernel will be... i ) is the number of elements K of the reference kernel and the weight coefficient (a i ) takes a different value. On the other hand, if the kernel deformation unit 101 deforms the reference kernel without interpolation, the number of elements K' and the weight coefficient (a') of the deformed kernel will be different. i ) is the number of elements K of the reference kernel and the weight coefficient (a i This matches the following: where i is a variable corresponding to the elements that make up the kernel, and 1 ≤ i ≤ K'.

[0084] The modified kernel memory unit 108 may store the modified kernel in association with the reference kernel from which it was generated. In other words, if multiple reference kernels are available, the modified kernel memory unit 108 stores information indicating which reference kernel each modified kernel was generated from. The modified kernel memory unit 108 may also store the modified kernel in association with the coordinates and normal direction of the three-dimensional points used to generate that modified kernel.

[0085] Furthermore, the images stored in the image storage unit 105 include not only color images in which RGB values ​​are associated with the coordinates of each pixel, but also monochrome, multiband, feature data, etc. In other words, the images in this embodiment only need to be data in which at least one channel is associated with a two-dimensional array (for example, an array corresponding to the coordinates of each pixel).

[0086] Furthermore, the kernel deformation method determination unit 100 only needs to determine a deformation method that is not estimated by machine learning, and that uses at least some kind of geometric deformation method. For example, the kernel deformation method determination unit 100 can determine a deformation method that uses projection transformation, affine transformation, or a transformation using the curvature of a sphere or cylinder.

[0087] Furthermore, when the kernel deformation unit 101 interpolates so that the deformed kernel has integer coordinates, it may apply a common interpolation method regardless of the kernel deformation method. For example, the kernel deformation unit 101 may employ any image interpolation method, including bilinear interpolation, that is, an interpolation method that uses the four surrounding pixels to interpolate so that the deformed kernel has integer coordinates.

[0088] <Processing common to each embodiment> Here, the processes common to each embodiment will be explained using Figure 5 (Figures 5A and 5B). Figure 5 is a diagram illustrating the processes according to the embodiments. In the following description, the image on which the convolution operation is performed, that is, the target viewpoint image, will be simply referred to as the "target image".

[0089] As shown in Figure 5A, the process common to each embodiment consists of phases PH1 to PH3. In Phase PH1, an integrated feature map is generated. Phase PH1 receives information from Phase PH2 regarding the deformed kernels A-C and the target image. Additionally, Phase PH1 receives information from Phase PH3 regarding the integration method. In Phase PH1, based on the deformed kernels and the target image, feature quantities corresponding to each of the deformed kernels A-C in the target image are extracted, and feature maps A-C are generated according to the extracted feature quantities. In Phase PH1, feature maps A-C are integrated according to the integration method to generate an integrated feature map. In Phase PH2, the transformed kernels A to C are generated. In Phase PH2, the transformation methods A to C are determined. In Phase PH2, the base kernel is transformed according to each of the transformation methods A to C, thereby generating the transformed kernels A to C. In Phase PH3, the integration method will be determined.

[0090] Figure 5B shows a flowchart illustrating the processing flow common to each embodiment. In this flowchart, the processing flow in phase PH1 is shown in steps S10 to S12. The processing flow in phase PH2 is shown in steps S20 to S21. The processing flow in phase PH3 is shown in step S30.

[0091] First, let's explain the processing flow in Phase PH1. Step S10: The convolution unit 102 of the feature map generation device 1 acquires the target image. The feature map generation device 1 acquires the target image from the image storage unit 105 by referring to the image storage unit 105. Step S11: The convolution unit 102 extracts features from the target image using the deformed kernel and generates feature maps A to C. The convolution unit 102 obtains the deformed kernels A to C from the deformed kernel storage unit 108 by referring to the deformed kernel storage unit 108. The convolution unit 102 performs a convolution operation on the target image using the deformed kernel A and generates feature map A by associating the calculation result of the convolution operation at each pixel with the coordinates of that pixel. The convolution unit 102 performs a convolution operation on the target image using the deformed kernel B and generates feature map B by associating the calculation result of the convolution operation at each pixel with the coordinates of that pixel. The convolution unit 102 performs a convolution operation on the target image using the deformed kernel C and generates feature map C by associating the calculation result of the convolution operation at each pixel with the coordinates of that pixel. The convolution unit 102 stores each of the generated feature maps A to C in the feature map storage unit 106. Step S12: The feature map integration unit 104 of the feature map generation device 1 integrates feature maps A to C using an integration method to generate an integrated feature map. The feature map integration unit 104 retrieves each of feature maps A to C from the feature map storage unit 106 by referring to the feature map storage unit 106. The feature map integration unit 104 integrates feature maps A to C using the integration method determined by the feature integration method determination unit 103, and generates an integrated feature map by making the integrated map an integrated feature map. The feature map integration unit 104 stores the generated integrated feature map in the feature map storage unit 106.

[0092] Next, we will explain the processing flow in Phase PH2. Step S20: The kernel deformation unit 101 of the feature map generation device 1 acquires a reference kernel. The kernel deformation unit 101 acquires a reference kernel from the reference kernel storage unit 107 by referring to the reference kernel storage unit 107. Step S21: The kernel deformation unit 101 generates deformed kernels A to C by deforming the reference kernel based on the deformation method determined by the kernel deformation method determination unit 100. The kernel deformation unit 101 deforms the reference kernel based on deformation method A determined by the kernel deformation method determination unit 100 and generates deformed kernel A. The kernel deformation unit 101 deforms the reference kernel based on deformation method B determined by the kernel deformation method determination unit 100 and generates deformed kernel B. The kernel deformation unit 101 deforms the reference kernel based on deformation method C determined by the kernel deformation method determination unit 100 and generates deformed kernel C. The kernel deformation unit 101 stores each of the generated deformed kernels A to C in the deformed kernel storage unit 108.

[0093] Finally, we will explain the processing flow in Phase PH3. Step S30: The feature integration method determination unit 103 of the feature map generation device 1 determines the integration method. The specific details of the process by which the feature integration method determination unit 103 determines the integration method will be explained in the first embodiment described later.

[0094] <Regarding the first embodiment> Here, we will describe the first embodiment. The first embodiment is a configuration in which the feature map generation device 1 is applied to a classification system. In the classification system, classification is performed based on the feature quantities in each image, i.e., the feature map.

[0095] In classification, when extracting features by performing a convolution operation using the same kernel (e.g., a reference kernel), different features may be extracted even for pixels that capture the same region of the same object, depending on the image capture conditions. For example, if the capture position is different, or if the object is not directly facing the camera and is not captured from the front, the pixels surrounding the corresponding points will differ from image to image. In this way, when the features of pixels capturing the same region of the same object differ from image to image, it becomes difficult to classify the same object into the same class, and the accuracy of the classification decreases.

[0096] To address this decrease in classification accuracy, in the first embodiment, a convolution operation is performed using a deformed kernel that has been modified according to the image. This makes it possible to extract similar features from pixels where the same region of the same object has been captured, thereby suppressing a decrease in classification accuracy.

[0097] In the first embodiment, it is desirable to determine multiple transformation methods for a single target image. By extracting feature quantities using multiple transformed kernels with transformation methods corresponding to each of the multiple normals, it is possible to reduce errors associated with projection transformations and other operations performed when transforming the kernels.

[0098] Furthermore, in the first embodiment, image alignment may be performed in advance. By performing alignment, corresponding points can be clearly identified. When performing alignment, by corresponding each of the multiple deformation methods to a deformation method that applies a small normal change, it is possible to reduce alignment errors and make the accuracy of class classification more precise.

[0099] However, when determining the deformation method, the kernel deformation method determination unit 100 may determine not only projection transformations corresponding to the normal direction, but also transformations that do not use normals, such as scale transformations and affine transformations, as deformation methods. Even when such transformations that do not use normals are used, determining multiple deformation methods makes it possible to reduce errors associated with transformations performed when deforming the kernel (e.g., scale transformations, affine transformations, etc.).

[0100] <Configuration of the first embodiment> Here, the configuration of the first embodiment will be described with reference to Figure 6. Figure 6 is a block diagram showing the configuration of the feature map generation device 1 of the first embodiment. As shown in Figure 6, the feature map generation device 1 in the first embodiment comprises a basic configuration 10, a class classification unit 109, a normal map generation unit 110, and a normal map storage unit 111.

[0101] In the first embodiment, examples (Examples 1 to 5) of how class classification is implemented using multiple methods will be described. Regarding the configuration of the map generation device 1 in the first embodiment, the normal map generation unit 110 and the normal map storage unit 111 are configurations that are only present in Example 3 of the first embodiment. That is, Examples 1 to 2 and Examples 4 to 5 of the first embodiment include a basic configuration 10 and a class classification unit 109. Example 3 of the first embodiment includes a basic configuration 10, a class classification unit 109, a normal map generation unit 110, and a normal map storage unit 111.

[0102] The classification unit 109 performs classification by classifying each image into its respective class. Based on the integrated feature map of each image, the classification unit 109 performs classification so that images with similar features belong to the same class.

[0103] Alternatively, the classification unit 109 may perform a convolution operation on the integrated feature map, generate feature vectors after the convolution operation via pooling (Global Average Pooling), and then perform classification through a fully connected layer.

[0104] The normal map generation unit 110 generates a normal map from an image (including a feature map). The normal map generation unit 110 may generate a normal map from the feature map D using any conventional method.

[0105] In Example 3 of the first embodiment, the normal map generation unit 110 generates a normal map from the feature map D. The normal map generation unit 110 stores the generated normal map in the normal map storage unit 111. In Example 3 of the first embodiment, the normal map storage unit 111 stores the normal map generated from the feature map D.

[0106] <Examples of the First Embodiment> Here, each example (Examples 1 to 5) of the first embodiment will be explained using Figures 7 (Figures 7A and 7B) and 8 (Figures 8A and 8B). Figure 7 is a diagram illustrating the processing according to Examples 1 to 2 and Examples 4 to 5 of the first embodiment. Figure 8 is a diagram illustrating the processing according to Example 3 of the first embodiment.

[0107] <Examples 1-2 and 4-5 of the first embodiment> As shown in Figure 7A, in Examples 1-2 and 4-5 of the first embodiment, class classification using an integrated feature map is performed in phase PH1. Furthermore, feature map D is generated in phase PH3.

[0108] In Examples 1 to 2 and Examples 4 to 5 of the first embodiment, the method by which the feature integration method determination unit 103 determines the integration method in phase PH3 will be described below.

[0109] Example 1 of the First Embodiment: The feature integration method determination unit 103 generates a feature map D from the target image. The feature integration method determination unit 103 may generate the feature map D from the target image using any conventional method. The feature integration method determination unit 103 determines the weight coefficients of each of the multiple feature maps (for example, feature maps A to C) generated by the convolution operation unit 102 from the generated feature map D by inferring using attention (learning). In attention, the weight coefficients are estimated so as to minimize the error in class classification. The feature map integration unit 104 generates an integrated feature map by weighting and summing the weight coefficients calculated by the feature integration method determination unit 103.

[0110] Example 2 of the First Embodiment: The feature integration method determination unit 103 generates a feature map D from an additional image. The additional image is an image of the same target object as the target image, but taken under different imaging conditions (imaging position and orientation) than the target image. Similar to Example 1 of the First Embodiment, the feature integration method determination unit 103 may generate the feature map D from the additional image using any conventional method. The feature integration method determination unit 103 determines the weight coefficients of each of the multiple feature maps (for example, feature maps A to C) generated by the convolution operation unit 102 from the generated feature map D by inferring using attention (learning). In attention, the weight coefficients are estimated so as to minimize the error in class classification. The feature map integration unit 104 generates an integrated feature map by weighting and summing the weight coefficients calculated by the feature integration method determination unit 103.

[0111] Example 4 of the First Embodiment: The feature integration method determination unit 103 generates a feature map D from the additional image. The feature integration method determination unit 103 calculates the dot product of the feature vectors of the generated feature map D and the multiple feature maps (for example, feature maps A to C) generated by the convolution operation unit 102. The feature integration method determination unit 103 determines the calculated dot products as the weight coefficients for each feature map (for example, feature maps A to C). The feature map integration unit 104 generates an integrated feature map by weighting and summing the features using the weight coefficients calculated by the feature integration method determination unit 103.

[0112] Example 5 of the First Embodiment: The feature integration method determination unit 103 generates feature map D from a plurality of feature maps (e.g., feature maps A to C) generated by the convolution operation unit 102. Similar to Example 1 of the First Embodiment, the feature integration method determination unit 103 may generate feature map D from a plurality of feature maps (e.g., feature maps A to C) using any conventional method. The feature integration method determination unit 103 determines the weight coefficients of each of the plurality of feature maps (e.g., feature maps A to C) generated by the convolution operation unit 102 from the generated feature map D by inferring using attention (learning). In attention, the weight coefficients are estimated so as to minimize the error in class classification. The feature map integration unit 104 generates an integrated feature map by weighting and summing the weight coefficients calculated by the feature integration method determination unit 103.

[0113] In the above, the feature integration method determination unit 103 may determine weight coefficients for each image (feature map), or it may determine weight coefficients for each pixel.

[0114] Furthermore, by using an additional image, for example, an image of the target object taken from the front, it is possible to generate a feature map D that contains the features of the target object when it is facing directly. In this case, it becomes possible to obtain an integrated feature map in which the feature maps A to C that have features that are close to the state in which the target object is facing directly are emphasized.

[0115] For example, when considering product inspection, an additional image is prepared, which is an image of the object to be inspected taken from an ideal imaging position that facilitates inspection. In this case, the target image is the image of the object to be inspected taken during the actual inspection. For example, when inspecting by imaging the object as it moves on a conveyor belt, vibrations of the conveyor belt may result in slightly different imaging conditions than those of the additional image. It is usually difficult to distinguish whether such differences in imaging conditions are due to vibration or to a defective product. As a countermeasure against such difficult-to-distinguish phenomena, in this embodiment, it is possible to correct differences in imaging conditions due to vibration by kernel deformation. Therefore, it becomes possible to accurately extract product defects based on the target image.

[0116] In this way, by using representative images from each class classified by the class classification unit 109 as additional images, it becomes possible to accurately determine whether or not the target image belongs to that class.

[0117] Figure 7B shows flowcharts illustrating the processing flow performed by the feature map generation device 1 in Examples 1-2 and 4-5 of the first embodiment. In this flowchart, the processing flow in phase PH1 is shown in steps S110-S113. The processing flow in phase PH2 is shown in steps S120-S121. The processing flow in phase PH3 is shown in steps S130-S131. Steps S110-S112 and S120-S121 in this flowchart are the same as steps S10-S12 and S20-S21 in Figure 5B, so their explanation is omitted.

[0118] Step S113: The classification unit 109 performs classification using the integrated feature map. The classification unit 109 acquires the target image. The feature map generation device 1 acquires the target image from the image storage unit 105 by referring to the image storage unit 105.

[0119] Step S130: The feature integration method determination unit 103 generates feature map D. In Example 1 of the first embodiment, the feature integration method determination unit 103 generates feature map D from the target image. In Examples 2 and 4 of the first embodiment, the feature integration method determination unit 103 generates feature map D from additional images. In Example 5 of the first embodiment, the feature integration method determination unit 103 generates feature map D from feature maps A to C.

[0120] Step S131: The feature integration method determination unit 103 determines the integration method. In Examples 1 to 2 and 5 of the first embodiment, the feature integration method determination unit 103 determines the integration method by determining weight coefficients corresponding to the feature quantities in feature map D. In Example 4 of the first embodiment, the feature integration method determination unit 103 determines the integration method by determining weight coefficients corresponding to the dot product values ​​of feature map D and each of feature maps A to C.

[0121] <Regarding Example 3 of the First Embodiment> As shown in Figure 8A, in Example 3 of the first embodiment, in phase PH1, class classification is performed using an integrated feature map. In phase PH3, a feature map D is generated, and a normal map is generated from the generated feature map D.

[0122] In Example 3 of the first embodiment, the method by which the feature integration method determination unit 103 determines the integration method in phase PH3 will be described. In Example 3 of the first embodiment, the feature integration method determination unit 103 generates a feature map D from the target image. Furthermore, the normal map generation unit 110 generates a normal map from the feature map D. The feature integration method determination unit 103 calculates the dot product of the normal vector in the normal map and the normal direction corresponding to the multiple deformation methods (e.g., deformation methods A to C) determined by the kernel deformation method determination unit 100. The feature integration method determination unit 103 determines each of the calculated dot products as the weight coefficients for each feature map (e.g., feature maps A to C). The feature map integration unit 104 generates an integrated feature map by weighting and summing using the calculated weight coefficients.

[0123] In Example 3 of the first embodiment, the feature integration method determination unit 103 may determine weight coefficients for each image (feature map), or it may determine weight coefficients for each pixel.

[0124] Furthermore, in Example 3 of the first embodiment, the larger the dot product value calculated, the more the direction of the normal vector in the normal map and the normal direction corresponding to each of the multiple deformation methods are in the same direction. Therefore, in Example 3 of the first embodiment, it is possible to obtain an integrated feature map in which the feature map calculated in accordance with the deformation method determined using the normal direction set in the same direction as the normal map is emphasized.

[0125] Figure 8B shows a flowchart illustrating the processing flow performed by the feature map generation device 1 in Example 3 of the first embodiment. In this flowchart, the processing flow in phase PH1 is shown in steps S210 to S213. The processing flow in phase PH2 is shown in steps S220 to S221. The processing flow in phase PH3 is shown in steps S230 to S232.

[0126] Steps S210-S212 and S220-S221 in this flowchart are the same as steps S10-S12 and S20-S21 in Figure 5B, so their explanation is omitted. Also, step S213 in this flowchart is the same as step S113 in Figure 7B, so its explanation is omitted.

[0127] Step S230: The feature integration method determination unit 103 generates a feature map D from the target image. Step S231: The feature integration method determination unit 103 generates a normal map from the feature map D. Step S232: The feature integration method determination unit 103 calculates the dot product value of the normal map and each of the feature maps A to C, and determines the integration method by determining the weight coefficient corresponding to the calculated dot product.

[0128] As described above, the feature map generation device 1 of the first embodiment is a device that generates feature maps for one or more target images. The feature map generation device 1 comprises a kernel deformation method determination unit 100, a kernel deformation unit 101, a convolution operation unit 102, a feature integration method determination unit 103, and a feature map integration unit 104. The kernel deformation method determination unit 100 determines a deformation method for the kernel corresponding to the target image using geometric transformations for the target image. The kernel deformation method determination unit 100 determines multiple deformation methods for the kernel corresponding to one target image. The kernel deformation unit 101 generates multiple deformed kernels for one target image by deforming a reference kernel using each deformation method. The deformed kernels are kernels used in convolution operations to extract feature quantities from the target image. The convolution operation unit 102 extracts feature quantities from the target image by performing a convolution operation on the target image using the deformed kernels. The convolution unit 102 generates multiple feature maps corresponding to a single target image using the extracted features. The feature integration method determination unit 103 determines an integration method for integrating the multiple feature maps generated by the convolution unit 102. The feature map integration unit 104 generates an integrated feature map by integrating the multiple feature maps according to the integration method determined by the feature integration method determination unit 103.

[0129] As a result, in the feature map generation device 1 of this embodiment, the deformation of the kernel itself can be excluded from the machine learning training. Therefore, the method for deforming the kernel does not get stuck in a local minimum, and the intended deformation can be performed. This improves robustness to the imaging angle. Furthermore, in the feature map generation device 1 of this embodiment, multiple feature maps can be generated for a single target image and these multiple feature maps can be integrated. Therefore, it is possible to reduce errors associated with transformations performed when deforming the kernel (e.g., scale transformation, affine transformation, etc.).

[0130] Furthermore, in the feature map generation device 1 of the first embodiment, the kernel deformation method determination unit 100 determines a deformation method using one of the following as the geometric transformation: projection transformation, affine transformation, sphere, or cylindrical curvature. As a result, the feature map generation device 1 of the embodiment can perform the intended deformation according to the condition of the target object and the imaging environment of the target image, thereby achieving the same effects as described above.

[0131] Furthermore, in the feature map generation device 1 of the first embodiment, the kernel deformation unit 101 deforms the reference kernel using the deformation method to obtain a temporary deformed kernel, and generates a deformed kernel by performing interpolation processing on the temporary deformed kernel using coordinates arranged in a square grid. As a result, in the feature map generation device 1 of the embodiment, the deformed kernel can be a deformed kernel with integer two-dimensional coordinates, and the increase in computational cost can be suppressed by not having real two-dimensional coordinates in the deformed kernel, which would increase the computational cost of the convolution operation.

[0132] Furthermore, in the feature map generation device 1 of the first embodiment, the feature integration method determination unit 103 determines a weight coefficient for each of the multiple feature maps for each pixel of the feature map. The feature map integration unit 104 uses the weight coefficient to weight each of the multiple feature maps for each pixel of the feature map and generates a map that is combined as an integrated feature map. As a result, the feature map generation device 1 of the embodiment can generate an integrated feature map that is a weighted sum of each of the multiple feature maps, and can generate an integrated feature map that emphasizes a specific feature map, for example, a feature map in which features have been extracted under conditions close to those of an actual target object.

[0133] Furthermore, in the feature map generation device 1 of the first embodiment, the feature integration method determination unit 103 generates a feature map D (second feature map) from the target image that is different from feature maps A to C. The feature integration method determination unit 103 determines a weight coefficient for each pixel of feature maps A to C according to the feature quantity in feature map D. As a result, the feature map generation device 1 of the embodiment can generate an integrated feature map in which feature maps A to C that have similar feature quantities to feature map D are emphasized.

[0134] Furthermore, in the feature map generation device 1 of the first embodiment, the feature integration method determination unit 103 acquires an additional image (second target image) and generates a feature map D (second feature map) from the acquired additional image, which is different from feature maps A to C. The feature integration method determination unit 103 determines a weight coefficient for each pixel of feature maps A to C, corresponding to the feature quantities in feature map D. As a result, the feature map generation device 1 of the embodiment can generate an integrated feature map in which feature maps A to C that have similar feature quantities to feature map D are emphasized. By using, for example, an image of the target object taken from the front as the additional image used to generate feature map D, it becomes possible to generate an integrated feature map in which feature maps A to C that have feature quantities close to those of the target object facing directly are emphasized.

[0135] Furthermore, in the feature map generation device 1 of the first embodiment, the feature integration method determination unit 103 generates a feature map D (second feature map) from a plurality of feature maps A to C. The feature integration method determination unit 103 determines a weight coefficient for each pixel of feature maps A to C according to the feature amount in feature map D. As a result, the feature map generation device 1 of the embodiment makes it possible to generate an integrated feature map in which common features among feature maps A to C are emphasized.

[0136] Furthermore, in the feature map generation device 1 of the first embodiment, the feature integration method determination unit 103 acquires a feature map D (reference feature vector) generated from the additional image, and determines a weight coefficient for each pixel of feature maps A to C, corresponding to the inner product of the feature vector in feature map D and the feature vector in feature map A to C. As a result, the feature map generation device 1 of the embodiment can generate an integrated feature map in which features having the same direction as the feature vector in feature map D are emphasized.

[0137] Furthermore, in the feature map generation device 1 of the first embodiment, the feature integration method determination unit 103 determines a method for selecting the maximum or minimum value as the integration method. The feature map integration unit 104 generates an integrated feature map for each pixel of feature maps A to C, in which the maximum or minimum value among the feature quantities in each pixel of feature maps A to C is selected. As a result, the feature map generation device 1 of the embodiment can generate an integrated feature map having the maximum or minimum value among the feature quantities in each pixel of feature maps A to C.

[0138] Furthermore, the feature map generation device 1 of the first embodiment further includes a class classification unit 109. The class classification unit 109 performs class classification using the integrated feature map generated by the feature map integration unit 104. As a result, the feature map generation device 1 of the embodiment can perform class classification using the integrated feature map. In the integrated feature map, errors in each feature map are reduced by integrating multiple feature maps, making it possible to perform class classification with high accuracy.

[0139] Furthermore, the feature map generation device 1 of the first embodiment further includes a class classification unit 109. The kernel deformation method determination unit 100 determines multiple deformation methods based on each of multiple normal directions, and the feature integration method determination unit 103 determines weight coefficients that minimize the error in class classification by the class classification unit 109 based on the multiple normal directions by inference using attention (learning). As a result, the feature map generation device 1 of the embodiment can generate an integrated feature map that can perform accurate class classification using weight coefficients that minimize the error in class classification.

[0140] Furthermore, the feature map generation device 1 of the first embodiment further includes a class classification unit 109. The kernel deformation method determination unit 100 determines multiple deformation methods based on each of multiple normal directions. The feature integration method determination unit 103 uses a feature map D (feature vector) generated from representative images in each class classified by the class classification unit 109 as a reference feature vector. As a result, the feature map generation device 1 of the embodiment can generate an integrated feature map such that the features of the representative images in each class are emphasized. Therefore, it is possible to accurately determine whether or not an image is classified into each class using the integrated feature map generated in this way.

[0141] Alternatively, an image matching system may be constructed using the feature map generation device 1 of the first embodiment. In the image matching system, images are matched using the integrated feature map generated by the feature map generation device 1. This makes it possible to associate images with similar features.

[0142] <Regarding the second embodiment> A second embodiment will now be described. The second embodiment is a configuration in which the feature map generation device 1 is applied to a stereo matching system. In the stereo matching system, a depth map is generated by calculating depth values ​​using stereo images.

[0143] In stereo matching, depth values ​​are calculated using a stereo-parallelized image, which consists of an image of the object being photographed taken from one imaging position and an image taken from another imaging position moved horizontally (or vertically) from that position. In such a stereo-parallelized image, in order to extract features similar to those of pixels that photograph the same region of the same object, it is necessary to generate a deformed kernel that takes skew (parallelogram) into account.

[0144] Therefore, the kernel deformation method determination unit 100 determines a deformation method using an affine transformation with specified skew, rather than a projection transformation. The kernel deformation method determination unit 100 may, for example, determine the skew based on the normal direction of the target object, or it may set the angle of the parallelogram to an appropriate value. In addition, the kernel deformation method determination unit 100 may determine a deformation method that adds a deformation to expand or contract in the direction of the epipolar lines in the stereo-parallelized image, in addition to the deformation method that deforms the square grid into a parallelogram.

[0145] A second embodiment will be described using Figure 9 (Figures 9A, 9B, and 9C).

[0146] Figure 9A is a block diagram showing the configuration of the feature map generation device 1 in the second embodiment. As shown in Figure 9A, the feature map generation device 1 in the second embodiment comprises a basic configuration 10, a cost value calculation unit 112, a depth map generation unit 113, and a depth map storage unit 114.

[0147] The cost value calculation unit 112 calculates cost values ​​from multiple feature maps. The cost value is the degree to which corresponding points in the feature maps are similar, and is a value that indicates, for example, the variance or correlation between features at corresponding points.

[0148] Here, consider the case where a set {M1, n1} of the coordinates and the normal direction of a certain three-dimensional point is given. In this case, for each of the plurality of target viewpoint images {I0, I1, …} included in the multi-viewpoint image, by collaborating each of the kernel deformation method determination unit 100, the kernel deformation unit 101, and the convolution operation unit 102, the feature maps {f 0,1 、f 1,1 、…} of each target viewpoint image when the set {M1, n1} of the coordinates and the normal direction of the three-dimensional point is applied are generated.

[0149] At this time, the cost value C1 at the three-dimensional coordinates M = [X, Y, Z] T is a value indicating the variance of the feature amounts of the corresponding points corresponding to the three-dimensional coordinates M in the feature maps {f 0,1 、f 1,1 、…} of each target viewpoint image.

[0150] Similarly, consider the case where a set {M1, n2} of different normal directions is given for the same coordinates of the three-dimensional point. In this case, the feature maps {f 0,2 、f 1,2 、…} of each of the plurality of target viewpoint images included in the multi-viewpoint image are generated. The cost value calculation unit 112 calculates, as the cost value C2, a value indicating the variance of the feature amounts at the corresponding points of the feature maps {f 0,2 、f 1,2 、…} of each target viewpoint image.

[0151] The cost value calculation unit 112 sets the minimum value among the elements constituting the set {C1, C2} of the cost values for each of the two normal directions for the same coordinates of the three-dimensional point as the final cost value C.

[0152] The above explanation illustrated the case of calculating the final cost value C for each of the two normal directions for the same three-dimensional point's coordinates. However, even when three or more sets of normal directions are given for the same three-dimensional point's coordinates, a set of cost values ​​{C1, C2, ...} can be generated in a similar manner. The cost value calculation unit 112 takes the minimum value among the elements constituting the set of cost values ​​{C1, C2, ...} as the final cost value.

[0153] When the assumed normal direction, i.e., the given normal direction, is close to the true normal direction, the feature quantities of the corresponding points in the feature map will have similar values. In this case, the set of cost values, i.e., the variance between feature quantities for each normal direction, will be small. Therefore, calculating cost values ​​for each of the multiple normal directions and adopting the minimum of these as the final cost value is equivalent to selecting the cost value calculated for the normal direction that best matches the true normal direction from among the multiple assumed normal directions.

[0154] In the above, the cost value calculation unit 112 may calculate the variance of features at corresponding points in the feature maps of multiple target viewpoint images, or it may calculate the correlation of features at corresponding points in the feature maps of multiple target viewpoint images, as the cost value. The cost value calculation unit 112 only needs to be able to calculate the degree of similarity of the features at corresponding points in the feature maps of at least multiple target viewpoint images as the cost value. That is, the cost value calculation unit 112 may calculate a value that represents either variance or correlation as the cost value.

[0155] When calculating the correlation of feature quantities as a cost value, the cost value calculation unit 112 uses the maximum value among each element in the set of cost values ​​corresponding to the normal direction as the final cost value. When the feature quantities of corresponding points in the feature map of each target viewpoint image take similar values, the correlation between the feature quantities with respect to each normal direction takes a large value. Therefore, by adopting the maximum value of the cost value for each normal direction as the final cost value, it is possible to select a cost value calculated using the normal direction that is most consistent with the true normal direction.

[0156] The cost value calculation unit 112 may calculate the cost value based on any of the variables (three-dimensional coordinate M, normal direction n) associated with the feature map. For example, as described above, the cost value may be calculated for each of the multiple normal directions n based on the same three-dimensional coordinate, or the cost value may be calculated for each pair of three-dimensional coordinate and normal direction n. Alternatively, the cost value may be calculated for each of the multiple three-dimensional coordinates based on the same normal direction.

[0157] Furthermore, the cost value calculation unit 112 generates a cost volume. The cost volume is a set of multiple planes facing the reference viewpoint image, each having discrete depths, and the cost values ​​are reflected in these planes. For example, the cost value calculation unit 112 generates the cost volume by calculating the cost value in the three-dimensional coordinates corresponding to each voxel for all voxels of a three-dimensional voxel set up in three-dimensional space beforehand. The cost value calculation unit 112 stores the generated cost volume in the depth map storage unit 114.

[0158] In this case, the cost value calculation unit 112 may set the three-dimensional voxel as a cube, or it may set it as a square swing with the faces and bottom faces of the reference viewpoint image facing each other.

[0159] In the second embodiment, the depth map generation unit 113 generates a depth map based on the cost volume.

[0160] The depth map generation unit 113 first normalizes the cost volume. The depth map generation unit 113 normalizes the cost volume by, for example, performing a three-dimensional convolution operation on the generated cost volume. The depth map generation unit 113 stores the normalized cost volume in the depth map storage unit 114. At this time, the depth map generation unit 113 may apply different kernels multiple times in the three-dimensional convolution operation.

[0161] The depth map generation unit 113 then extracts a depth map from the cost volume in the reference viewpoint image, for example. The depth map generation unit 113 stores the extracted depth map in the depth map storage unit 114.

[0162] The depth map generation unit 113 extracts the cost value of the cost volume on the line of sight corresponding to each pixel in the reference viewpoint image, and selects the depth of the cost value that best matches the extracted cost value as the depth value of the depth map.

[0163] For example, if the three-dimensional voxels in the cost volume are set up on a square swing with the face and bottom of the reference viewpoint image facing each other, the line of sight of each pixel will coincide with a row of voxels on the cost volume. When the cost value is represented by the variance of the features, the depth map generation unit 113 selects the depth at which the cost value is smallest for each pixel in the reference viewpoint image as the depth value of the depth map. On the other hand, when the cost value is represented by the correlation of the features, the depth map generation unit 113 selects the depth at which the cost value is largest for each pixel in the reference viewpoint image as the depth value of the depth map.

[0164] The depth map storage unit 114 stores cost volumes. Cost volumes are information in which cost values ​​are associated with each voxel in a three-dimensional voxel set up in three-dimensional space.

[0165] Furthermore, the depth map storage unit 114 stores the depth map for each viewpoint of the multi-view image. Here, the size of the depth map matches the image size of the corresponding viewpoint (image). Also, the depth value of each pixel in the depth map is a value that indicates the depth to the target object in the two-dimensional coordinates of each pixel of the corresponding viewpoint.

[0166] Figure 9B is a diagram illustrating the processing according to the second embodiment. As shown in Figure 9B, in the second embodiment, a depth map is generated in phase PH1 using target image 1 and target image 2 as two stereo-parallelized images.

[0167] In phase PH1 of the second embodiment, multiple cost volumes (cost volumes A to C) are generated using each of the multiple feature maps (feature maps A to C) generated by the convolution operation unit 102. The multiple cost volumes are integrated by the feature map integration unit 104 to generate an integrated feature map. That is, in the second embodiment, the integrated feature map is a map in which multiple cost volumes are integrated. The integrated feature map generated in this way is input into an existing network (3D U-Net) for generating depth maps to generate a depth map.

[0168] Figure 9C is a flowchart showing the processing flow performed by the feature map generation device 1 in the second embodiment. In this flowchart, the processing flow in phase PH1 is shown in steps S310 to S314. The processing flow in phase PH2 is shown in steps S320 to S321. The processing flow in phase PH3 is shown in step S330.

[0169] First, let's explain the processing flow in Phase PH1. Step S310: The convolution unit 102 of the feature map generation device 1 acquires target image 1 and target image 2. Step S311: The convolution unit 102 extracts features from the target image 1 using the deformed kernel and generates feature maps 1A to 1C. The convolution unit 102 extracts features from the target image 2 using the deformed kernel and generates feature maps 2A to 2C. The convolution unit 102 stores the generated feature maps 1A to 1C and feature maps 2A to 2C in the feature map storage unit 106. Step S312: The cost value calculation unit 112 of the feature map generation device 1 calculates cost values ​​using feature maps 1A to 1C and feature maps 2A to 2C, respectively, and generates cost volumes A to C. For example, cost volume A is generated based on the correlation between feature map 1A and feature map 2A. Cost volume B is generated based on the correlation between feature map 1B and feature map 2B. Cost volume C is generated based on the correlation between feature map 1C and feature map 2C. Step S313: The feature map integration unit 104 integrates cost volumes A to C using an integration method and generates an integrated feature map. The feature map integration unit 104 retrieves each of cost volumes A to C from the depth map storage unit 114 by referring to the depth map storage unit 114. The feature map integration unit 104 integrates cost volumes A to C using the integration method determined by the feature integration method determination unit 103 and generates an integrated feature map by making the integrated map an integrated feature map. The feature map integration unit 104 stores the generated integrated feature map in the feature map storage unit 106. Step S314: The depth map generation unit 113 generates a depth map using the integrated feature map.

[0170] Next, we will explain the processing flow in Phase PH2. Step S320: The kernel deformation unit 101 of the feature map generation device 1 acquires a reference kernel. The kernel deformation unit 101 acquires a reference kernel from the reference kernel storage unit 107 by referring to the reference kernel storage unit 107. Step S321: The kernel deformation unit 101 generates deformed kernels A to C by deforming the reference kernel based on the deformation method determined by the kernel deformation method determination unit 100.

[0171] Finally, we will explain the processing flow in Phase PH3. Step S30: The feature integration method determination unit 103 of the feature map generation device 1 determines the integration method. The feature integration method determination unit 103 may use any of the methods described in each example of the first embodiment as the integration method.

[0172] <Regarding the third embodiment> Now, a third embodiment will be described. The third embodiment is a configuration in which the feature map generation device 1 is applied to a single-view depth map generation system.

[0173] In a single-view depth map generation system, a depth map is generated by estimating the depth value from a single image using machine learning. Depending on the image, the appearance of the same object may differ depending on the imaging angle. Therefore, a large training dataset is required to train the system to estimate the depth value from images of the object taken from various angles. When using a square grid kernel, features are extracted that ignore the influence of the object's normal direction. For this reason, even for the same object, it is necessary to prepare a large amount of data taken from different imaging angles.

[0174] To address these challenges, this embodiment enables high-accuracy estimation of depth maps from a single image using the smallest possible number of training datasets.

[0175] A third embodiment will be described using Figure 10 (Figures 10A, 10B, and 10C).

[0176] Figure 10A is a block diagram showing the configuration of the feature map generation device 1 according to the third embodiment. As shown in Figure 10A, the feature map generation device 1 in the third embodiment comprises a basic configuration 10, a normal map generation unit 110, a normal map storage unit 111, a depth map generation unit 113, and a depth map storage unit 114.

[0177] In the third embodiment, the kernel deformation method determination unit 100 determines a plurality of deformation methods corresponding to a plurality of directions that can be the normal direction of the target object.

[0178] The feature integration method determination unit 103 determines the integration method for multiple feature maps corresponding to each of the multiple deformation methods, such that the feature map in which an appropriate direction is set as the normal direction of the target object is emphasized.

[0179] Specifically, if a normal map does not exist, the feature integration method determination unit 103 determines integration method 1 using any method. The feature integration method determination unit 103 may use any of the methods described in each example of the first embodiment as integration method 1.

[0180] On the other hand, if a normal map exists, the feature integration method determination unit 103 determines the integration method 2 based on that normal map.

[0181] Similar to Example 3 in the first embodiment, the feature integration method determination unit 103 calculates the dot product of the normal vector in the normal map and the normal direction corresponding to the multiple deformation methods (e.g., deformation methods A to C) determined by the kernel deformation method determination unit 100. The feature integration method determination unit 103 determines the integration method 2 by using each of the calculated dot products as the weight coefficients for each feature map (e.g., feature maps A to C).

[0182] Alternatively, the feature integration method determination unit 103 may determine the integration method 2 by inferring the weight coefficients of each of the multiple feature maps (e.g., feature maps A to C) generated by the convolution operation unit 102 from the normal map using attention (learning), similar to Examples 1 and 5 in the first embodiment.

[0183] As a result, in the third embodiment, feature maps can be integrated while taking into account the normal direction of the target object, making it possible to generate a trained model that can estimate depth values ​​with high accuracy from a small number of training datasets.

[0184] Figure 10B is a diagram illustrating the processing according to the third embodiment. As shown in Figure 10B, in the third embodiment, a depth map is generated using one target image in phase PH1. The depth map generation unit 113 inputs an integrated feature map, which is formed by integrating multiple feature maps (feature maps A to C), into an existing network (3D U-Net) for generating depth maps, thereby generating a depth map.

[0185] The existing networks used here are not limited to 3D U-Net. Encoder-Decoder networks can also be used instead of 3D U-Net. The Encoder-Decoder network described here takes images or feature maps (hereinafter referred to as "images, etc.") as input and outputs images or maps transformed from the input images, etc.

[0186] Furthermore, the normal map generation unit 110 generates a normal map from the depth map generated by the feature integration method determination unit 103. The normal map thus generated is used in determining the integration method 2 in phase PH3.

[0187] Figure 10C is a flowchart showing the processing flow performed by the feature map generation device 1 in the third embodiment. In this flowchart, the processing flow in phase PH1 is shown in steps S410 to S414. The processing flow in phase PH2 is shown in steps S420 to S421. The processing flow in phase PH3 is shown in steps S430 to S431.

[0188] Steps S410-S412 and S420-S421 in this flowchart are the same as steps S10-S12 and S20-S21 in Figure 5B, so their explanation is omitted. Also, step S430 in this flowchart is the same as step S330 in Figure 9B, so its explanation is omitted.

[0189] Step S413: The depth map generation unit 113 of the feature map generation device 1 generates a depth map using the integrated feature map. Step S414: The normal map generation unit 110 generates a normal map using the depth map. Step S431: The feature integration method determination unit 103 determines the integration method 2 using the normal map.

[0190] <Regarding the fourth embodiment> Now, a fourth embodiment will be described. The fourth embodiment is a configuration in which the feature map generation device 1 is applied to a character recognition system.

[0191] In character recognition systems, characters are recognized when they are captured in an image. If the characters in the image are not facing directly towards the subject, for example, if they are captured from an oblique angle, or if the characters are printed on a non-flat surface such as a beverage can, the accuracy of character recognition will decrease.

[0192] When training a machine learning model for character recognition, the images used for training are typically those in which the characters are photographed from the front. Therefore, to train a model to recognize characters that are not directly facing the model, such as those shown in images where the characters are not directly facing the model, it is necessary to expand the training data by including images in which the characters are not directly facing the model. If the training data is not sufficiently expanded, it will be difficult for the trained model to recognize characters that are not directly facing the model. On the other hand, expanding the training data can result in an enormous number of training datasets, which may make training difficult.

[0193] To address these challenges, the fourth embodiment aims to improve the accuracy of character recognition in images where characters are not directly facing the subject, and in images where characters printed on a non-planar surface are captured.

[0194] A fourth embodiment will be described using Figure 11 (Figures 11A, 11B, and 11C).

[0195] Figure 11A is a block diagram showing the configuration of the feature map generation device 1 according to the fourth embodiment. As shown in Figure 11A, the feature map generation device 1 in the fourth embodiment comprises a basic configuration 10, a class classification unit 109, and a class classification storage unit 115.

[0196] In the fourth embodiment, the kernel deformation method determination unit 100 determines deformation methods corresponding to various normal directions. The kernel deformation method determination unit 100 may employ a determination method other than one based on the normal direction. For example, the kernel deformation method determination unit 100 may determine a deformation method corresponding to characters printed on a canned beverage by performing deformations into spherical or cylindrical shapes corresponding to various curvatures.

[0197] The kernel deformation unit 101 deforms the reference kernel using the deformation method determined by the kernel deformation method determination unit 100, thereby generating a deformed kernel corresponding to each deformation method. The convolution operation unit 102 generates a feature map corresponding to each deformed kernel using the deformed kernel generated by the kernel deformation unit 101.

[0198] The classification unit 109 classifies the multiple feature maps (for example, feature maps A to C) generated by the convolution operation unit 102. The method by which the classification unit 109 performs the classification is the same as the method by which the classification unit 109 performs the classification in the first embodiment. That is, the classification unit 109 performs the classification in each feature map such that feature maps with similar features belong to the same class.

[0199] The feature integration method determination unit 103 determines weight coefficients corresponding to the scores of each transformed kernel based on the results of the feature map classification performed by the classification unit 109. Alternatively, instead of determining weight coefficients, the feature integration method determination unit 103 may select the feature map to be used in the integrated feature map by majority vote.

[0200] The feature map integration unit 104 generates an integrated feature map by performing a weighted sum using the weighting coefficients determined by the feature integration method determination unit 103. The classification unit 109 uses an integrated feature map to classify the characters captured in the image.

[0201] In the fourth embodiment, the feature integration method determination unit 103 can determine weight coefficients so that feature maps classified into a specific class are emphasized. For example, the feature integration method determination unit 103 can determine weight coefficients so that feature maps associated with a kernel deformation method suitable for deforming a front-facing character into a rear-facing character are emphasized. In this case, the weight coefficients can be determined so that feature maps generated using a deformed kernel suitable for rear-facing characters are emphasized. The weight coefficients can also be determined so that feature maps generated using a deformed kernel suitable for a character image printed on a non-planar surface are emphasized.

[0202] Figure 11B is a diagram illustrating the processing according to the fourth embodiment. As shown in Figure 11B, in the third embodiment, character recognition is performed in phase PH1 by classifying the target image in which characters have been captured. A unified feature map is generated by integrating multiple feature maps (feature maps A to C) using an integration method determined according to the results of classifying them, and class classification of the target image is performed based on the unified feature map.

[0203] Figure 11C is a flowchart showing the processing flow performed by the feature map generation device 1 in the fourth embodiment. In this flowchart, the processing flow in phase PH1 is shown in steps S510 to S513. The processing flow in phase PH2 is shown in steps S520 to S521. The processing flow in phase PH3 is shown in steps S530 to S531.

[0204] Steps S510-S512 and S520-S521 in this flowchart are the same as steps S10-S12 and S20-S21 in Figure 5B, so their explanation is omitted. Also, step S513 in this flowchart is the same as step S113 in Figure 7B, so its explanation is omitted.

[0205] Step S530: The class classification unit 109 of the feature map generation device 1 classifies feature maps A to C into classes. Step S531: The feature integration method determination unit 103 determines the integration method using the class classification results from step S530. The feature integration method determination unit 103 determines the integration method so that feature maps classified into a specific class are highlighted.

[0206] Thus, in the fourth embodiment, multiple normal directions are set as deformation methods, and when integrating feature maps, the appropriate normal direction is emphasized as the deformation method. As a result, the feature map calculated from the deformed kernel corresponding to the appropriate normal direction is selected as the integrated feature map, and an integrated feature map that takes the influence of normals into account can be generated. In this case, in locally planar regions, the normal direction does not change, resulting in almost the same features, and the feature map corresponding to the same normal direction is selected as the map that constitutes the integrated feature map in that region.

[0207] <Regarding the fifth embodiment> Now, a fifth embodiment will be described. The fifth embodiment is a configuration in which the feature map generation device 1 is applied to a style transfer system.

[0208] Conventional style transfer systems assume that the source image for the style is a painting or similar image taken from the front. They do not anticipate the use of images of three-dimensional objects taken from an oblique angle, or images of paintings or similar images drawn on a non-planar surface. Therefore, for example, if an image of an object taken from an oblique angle is used as the style image, the system will treat the shape that appears as a projection of the object due to the oblique angle as the style, resulting in the generation of an image with a distorted style.

[0209] To address these challenges, the fifth embodiment enables the transfer of the style of a three-dimensional object as seen from the front, even when using an image of the target object taken from an oblique angle.

[0210] A fifth embodiment will be described using Figure 12 (Figures 12A, 12B, 12C, 12D, and 12E).

[0211] Figure 12A is a block diagram showing the configuration of the feature map generation device 1 in the fifth embodiment. As shown in Figure 12A, the feature map generation device 1 in the fifth embodiment comprises a plurality of basic configurations 10 (basic configurations 10-1, 10-2, ..., 10-k), where k is an integer of 2 or more. The style transfer system is constructed by combining a style extraction network N1 and a feature extraction network N2 (see Figure 12D). The basic configurations 10 are used in both the style extraction network N1 and the feature extraction network N2.

[0212] Figure 12B is a diagram illustrating the processing according to the fifth embodiment. Figure 12B shows an example in which the basic configuration 10 is used in the style extraction network N1. As shown in Figure 12B, in the fifth embodiment, in phase PH1, an integrated feature map is generated using one target image or feature map. In phase PH3, the integration method is determined using a feature map D generated based on another image. The other image here can be the same image as the target image or a different image, depending on each stage of style transfer. The method for generating feature map D is the same as in Example 2 of the first embodiment. The parameters for generating feature map D may be common to all levels in style transfer.

[0213] Figure 12C is a diagram illustrating the processing according to the fifth embodiment. Figure 12C shows an example where the basic configuration 10 is used in the feature extraction network N2. As shown in Figure 12C, in the fifth embodiment, separately from phases PH1 to PH3, a feature map is generated in the image or feature map by calculating the feature quantities in the image or feature map using a reference kernel. A general CNN may be used for the feature extraction process here. Also, the reference kernel here may be the same kernel as that used in the style extraction network N1.

[0214] Figure 12D is a diagram illustrating the processing according to the fifth embodiment. As shown in Figure 12D, a style transfer system is constructed by combining the style extraction network N1 and the feature extraction network N2.

[0215] The style transfer system uses a style image, a post-transfer image, a destination image, and a reference kernel as inputs. The style image is an image of a painting or other artwork that has the style to be transferred. The post-transfer image is the image after the style has been transferred. The destination image is the image before the style has been transferred.

[0216] In the first layer ST1, the top-level style extraction network N1 outputs an integrated feature map T1 using the style image and a reference kernel as inputs. The second-level style extraction network N1 outputs an integrated feature map T2 using the transcribed image and a reference kernel as inputs. The third-level feature extraction network N2 outputs a feature map T3 using the transcribed image and a reference kernel as inputs. The bottom-level feature extraction network N2 outputs a feature map T4 using the target image and a reference kernel as inputs.

[0217] In the integrated feature map T1 and integrated feature map T2, the transcribed image is optimized to minimize the loss (error) between integrated feature map T1 and integrated feature map T2.

[0218] In the second layer ST2, the same processing as in the first layer ST1 is performed, but instead of an image, the integrated feature map or feature map output from the first layer ST1 is input to the second layer ST2. Specifically, the style image, integrated feature map T1, and reference kernel are input to the style extraction network N1 located at the top. The transcribed image, integrated feature map T2, and reference kernel are input to the style extraction network N1 located second from the top. The transcribed image, feature map T3, and reference kernel are input to the feature extraction network N2 located third from the top. The destination image, feature map T4, and reference kernel are input to the feature extraction network N2 located at the bottom. Similarly, in the third layer ST3, the integrated feature map or feature map output from the second layer ST2 is input to the third layer ST3. In the fourth layer, ST4, the transcribed image is optimized to minimize the loss (error) between the feature map output from the third feature extraction network N2 from the top and the feature map output from the bottom feature extraction network N2.

[0219] Figure 12E is a flowchart showing the processing flow performed by the feature map generation device 1 in the fourth embodiment. In this flowchart, the processing flow in phase PH1 is shown in steps S610 to S612. The processing flow in phase PH2 is shown in steps S620 to S621. The processing flow in phase PH3 is shown in steps S630 to S532.

[0220] Steps S610-S612 and S620-S621 in this flowchart are the same as steps S10-S12 and S20-S21 in Figure 5B, so their explanation is omitted. Also, steps S620-S621 in this flowchart are the same as in Example 2 in steps S130-S131 in Figure 7B, so their explanation is omitted.

[0221] By performing this process, the features of the transferred image can be made to possess features from both the style image and the target image, thereby generating an image with the style transferred. Moreover, the transformed kernel can be used when outputting the integrated feature map from the style image. Therefore, even if an image of a painting or other artwork with a style taken from an oblique angle is used as the style image, the style of that painting or artwork as seen from the front can be transferred.

[0222] In the fifth embodiment, the kernel deformation method determination unit 100 may employ a determination method other than the normal direction determination method. For example, the kernel deformation method determination unit 100 may determine a deformation method corresponding to the characters printed on the canned beverage by performing deformation into spherical or cylindrical shapes corresponding to various curvatures. For example, by performing deformation into spherical or cylindrical shapes with various curvatures, it can also be used for style transfer from canned beverages.

[0223] The feature map generation device 1 in the above-described embodiment may be implemented in whole or in part by a computer. In that case, the program for implementing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. Here, "computer system" includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into a computer system. Moreover, "computer-readable recording medium" may also include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside a computer system that acts as a server or client in such cases. Furthermore, the above-mentioned program may be for implementing a part of the above-mentioned function, or it may be a program that can implement the above-mentioned function in combination with a program already recorded in the computer system, or it may be implemented using a programmable logic device such as an FPGA.

[0224] The computer mentioned above may be a quantum computer. A quantum computer is a computer that performs parallel computations using the principle of quantum mechanical superposition, and is capable of computations exponentially faster than conventional computers. By using a quantum computer, it becomes possible to perform computations at high speed when executing operations such as convolution.

[0225] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and includes designs, device configurations, etc., that do not depart from the spirit of this invention.

[0226] Furthermore, the following inventions are also included in the present invention.

[0227] (Invention 1) A feature map generation device that generates feature maps for one or more target images, A kernel deformation method determination unit determines multiple methods for deforming the kernel corresponding to the target image using geometric transformations for the target image, A kernel deformation unit generates multiple deformed kernels, which are kernels used for convolution operations on the target image, by deforming a reference kernel using the deformation method described above. A convolution unit that extracts feature quantities from the target image by performing a convolution operation on the target image using the deformed kernel, and generates multiple feature maps corresponding to the target image using the extracted feature quantities, A feature integration method determination unit determines an integration method for integrating a plurality of feature maps generated by the convolution calculation unit, A feature map integration unit generates an integrated feature map by integrating a plurality of feature maps according to the integration method determined by the feature integration method determination unit, A feature map generation device equipped with the following features.

[0228] (Invention 2) The kernel deformation method determination unit determines the deformation method to be a transformation using one of the following geometric transformations: a projection transformation, an affine transformation, a sphere, or the curvature of a cylinder. A feature map generation device as described in Invention 1.

[0229] (Invention 3) The kernel deformation unit generates the deformed kernel by first deforming the reference kernel using the deformation method, and then performing interpolation on the provisionally deformed kernel using coordinates arranged in a square grid. A feature map generation device according to Invention 1 or Invention 2.

[0230] (Invention 4) The feature integration method determination unit determines a weight coefficient for each of the multiple feature maps for each pixel of the feature map, The feature map integration unit generates the integrated feature map by multiplying each of the multiple feature maps by the weight coefficient for each pixel of the feature map, and compositing the resulting map. A feature map generation device according to any one of Inventions 1 to 3.

[0231] (Invention 5) The feature integration method determination unit generates a second feature map from the target image that is different from the first feature map, and determines the weight coefficient for each pixel of the feature map according to the feature amount in the second feature map. A feature map generation device according to any one of Inventions 1 to 4.

[0232] (Invention 6) The feature integration method determination unit acquires a second target image different from the target image, generates a second feature map different from the feature map from the acquired second target image, and determines a value corresponding to the feature amount in the second feature map as the weight coefficient for each pixel of the feature map. A feature map generation device according to any one of Inventions 1 to 4.

[0233] (Invention 7) The feature integration method determination unit generates a second feature map different from the first feature map from a plurality of feature maps, and determines a value corresponding to the feature amount in the second feature map as the weight coefficient for each pixel of the feature map. A feature map generation device according to any one of Inventions 1 to 4.

[0234] (Invention 8) The feature integration method determination unit acquires a reference feature vector and determines the weight coefficient for each pixel in each of the plurality of feature maps, according to the dot product of the feature quantity of each pixel and the reference feature vector. The feature map generation device according to any one of Inventions 1 to 4.

[0235] (Invention 9) The feature integration method determination unit determines, as the integration method, a method of selecting a maximum value or a minimum value. The feature map integration unit generates, as the integrated feature map, a map in which the maximum value or the minimum value of the feature amounts at each pixel of a plurality of the feature maps is selected for each pixel of the feature map. The feature map generation device according to any one of Inventions 1 to 4.

[0236] (Invention 10) Further comprising a class classification unit that performs class classification using the integrated feature map generated by the feature map integration unit. The feature map generation device according to any one of Inventions 1 to 9.

[0237] (Invention 11) Further comprising a class classification unit that performs class classification using the integrated feature map generated by the feature map integration unit, The kernel deformation method determination unit determines a plurality of the deformation methods based on each of a plurality of normal directions. The feature integration method determination unit determines, based on the normal direction, by estimating, through learning, the weight coefficient such that the error of class classification in the class classification unit is minimized. The feature map generation device according to any one of Inventions 4 to 7.

[0238] (Invention 12) Further comprising a class classification unit that performs class classification using the integrated feature map generated by the feature map integration unit. The kernel deformation method determination unit acquires a plurality of normal vectors and determines a plurality of the deformation methods based on each of the plurality of normal vectors. The feature integration method determination unit uses, as the reference feature vector, a feature vector generated from a representative image in each class classified by the class classification unit. The feature map generation device according to Invention 8.

Explanation of Signs

[0239] 1... Feature map generation device 100... Kernel transformation method determination unit 101... Kernel transformation unit 102... Convolution operation unit 103... Feature integration method determination unit 104... Feature map integration unit 109... Class classification unit

Claims

1. A feature map generation device that generates feature maps for one or more target images, A kernel deformation method determination unit determines multiple methods for deforming the kernel corresponding to the target image using geometric transformations for the target image, A kernel deformation unit generates multiple deformed kernels, which are kernels used for convolution operations on the target image, by deforming a reference kernel using the deformation method described above. A convolution unit that extracts feature quantities from the target image by performing a convolution operation on the target image using the deformed kernel, and generates multiple feature maps corresponding to the target image using the extracted feature quantities, A feature integration method determination unit determines an integration method for integrating a plurality of feature maps generated by the convolution calculation unit, A feature map integration unit generates an integrated feature map by integrating a plurality of feature maps according to the integration method determined by the feature integration method determination unit, A feature map generation device equipped with the following features.

2. The kernel deformation method determination unit determines the deformation method to be a transformation using one of the following geometric transformations: a projection transformation, an affine transformation, a sphere, or the curvature of a cylinder. A feature map generation device according to claim 1.

3. The kernel deformation unit generates the deformed kernel by first deforming the reference kernel using the deformation method, and then performing interpolation on the provisionally deformed kernel using coordinates arranged in a square grid. A feature map generation device according to claim 1.

4. The feature integration method determination unit determines a weight coefficient for each of the multiple feature maps for each pixel of the feature map, The feature map integration unit generates the integrated feature map by multiplying each of the multiple feature maps by the weight coefficient for each pixel of the feature map, and compositing the resulting map. A feature map generation device according to claim 1.

5. The feature integration method determination unit generates a second feature map from the target image that is different from the first feature map, and determines the weight coefficient for each pixel of the feature map according to the feature amount in the second feature map. A feature map generation device according to claim 4.

6. The feature integration method determination unit acquires a second target image different from the target image, generates a second feature map different from the feature map from the acquired second target image, and determines a value corresponding to the feature amount in the second feature map as the weight coefficient for each pixel of the feature map. A feature map generation device according to claim 4.

7. The feature integration method determination unit generates a second feature map different from the first feature map from a plurality of feature maps, and determines a value corresponding to the feature amount in the second feature map as the weight coefficient for each pixel of the feature map. A feature map generation device according to claim 4.

8. The feature integration method determination unit acquires a reference feature vector and determines the weight coefficient for each pixel in each of the plurality of feature maps, according to the dot product of the feature quantity of each pixel and the reference feature vector. A feature map generation device according to claim 4.

9. The feature integration method determination unit determines a method for selecting the maximum or minimum value as the integration method. The feature map integration unit generates the integrated feature map by selecting the maximum or minimum value of each feature quantity in each pixel of the multiple feature maps for each pixel of the feature map. A feature map generation device according to any one of claims 1 to 3.

10. The system further comprises a classification unit that performs classification using the integrated feature map generated by the feature map integration unit. A feature map generation device according to claim 1.

11. The system further comprises a classification unit that performs classification using the integrated feature map generated by the feature map integration unit, The kernel deformation method determination unit determines a plurality of deformation methods based on each of the plurality of normal directions, The feature integration method determination unit determines the weight coefficients by learning, based on the normal direction, such that the error in the classification of the classification unit is minimized. A feature map generation device according to any one of claims 4 to 7.

12. The system further comprises a classification unit that performs classification using the integrated feature map generated by the feature map integration unit, The kernel deformation method determination unit acquires a plurality of normal vectors and determines a plurality of deformation methods based on each of the plurality of normal vectors. The feature integration method determination unit uses the feature vector generated from representative images in each class classified by the class classification unit as the reference feature vector. A feature map generation device according to claim 8.

13. The integrated feature map generated by the feature map generation device described in claim 1 is used to match the image. Image matching system.

14. A feature map generation method performed by a feature map generation device that generates feature maps for one or more target images, The kernel deformation method determination unit determines multiple methods for deforming the kernel corresponding to the target image using geometric transformations for the target image. The kernel deformation unit deforms a reference kernel using the deformation method to generate multiple deformed kernels, which are kernels used for convolution operations on the target image. The convolution unit performs a convolution operation on the target image using the deformed kernel to extract features from the target image, and generates multiple feature maps corresponding to the target image using the extracted features. The feature integration method determination unit determines an integration method for integrating the multiple feature maps generated by the convolution calculation unit, The feature map integration unit generates an integrated feature map by integrating a plurality of feature maps according to the integration method determined by the feature integration method determination unit. Method for generating feature maps.

15. A program that causes a feature map generation device, which generates feature maps for one or more target images, to generate feature maps, Using geometric transformations, multiple methods for deforming the kernel corresponding to the target image are determined for the target image. By transforming the reference kernel using the transformation method described above, multiple transformed kernels are generated, which are kernels used for convolution operations on the target image. By performing a convolution operation on the target image using the deformed kernel, the feature quantities in the target image are extracted, and multiple feature maps corresponding to the target image are generated using the extracted feature quantities. Determine an integration method for integrating multiple feature maps. The aforementioned integration method generates an integrated feature map by integrating multiple feature maps. program.

Images