Image processing apparatus, image processing method, and program

Abstract

To enable estimating a coordinate system to be referenced in detecting feature points from an image.SOLUTION: An image processing apparatus performs horizontal inversion processing on each of a plurality of original images including a subject such as a vehicle and the face of a person, to generate a different image. Then, the image processing apparatus detects feature points from the original and different images, respectively. The image processing apparatus also acquires a plurality of coordinate system candidates to be referenced, calculates a positional error between feature points obtained by inversely converting feature points detected from the different image with each coordinate system and feature points detected from the original image, and estimates a coordinate system having the highest evaluation value as a coordinate system to be referenced.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] In particular, the present invention relates to an image processing device, an image processing method, and a program suitable for use in detecting feature points from an image. [Background technology]

[0002] Conventionally, when detecting or recognizing an object in an image, estimating a depth value, etc., a method of detecting and using feature points is sometimes used. In such a case, when analyzing or matching feature points, the accuracy of the coordinate values ​​of the feature points on the image is important. Patent Document 1 discloses a method of measuring the three-dimensional shape of a measurement object by photographing the measurement object from multiple different positions using a photographing device, extracting and matching feature points from the photographed images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-212430 A [Non-patent literature]

[0004] [Non-Patent Document 1] Joseph Redmon, et al., "You Only Look Once: Unified, Real-Time Object Detection", in arXiv, Submitted on 8 Jun 2015 (v1), last revised 9 May 2016 (this version, v5),<URL: https: / / arxiv.org / abs / 1506.02640> [Non-Patent Document 2] Wenhao Ding, et al., "Vehicle Pose and Shape Estimation through Multiple Monocular Vision", in arXiv, Submitted on 10 Feb 2018 (v1), last revised 11 Nov 2018 (this version, v5),<URL:https: / / arxiv.org / abs / 1802.03515>

Non-Patent Document 3

Non-Patent Document 4

Non-Patent Document 5

Summary of the Invention

[0005] In general, a feature point detector outputs the coordinate values ​​of feature points on an image based on the position and orientation of a predetermined coordinate system. However, the position and orientation of the reference coordinate system may be unclear, which may result in an error in the position of the feature point.

[0006] In view of the above-mentioned problems, an object of the present invention is to make it possible to estimate a coordinate system that should be used as a reference when detecting feature points from an image. [Means for solving the problem]

[0007] The image processing device of the present invention is characterized in having a generation means for performing a predetermined conversion process on a first image including a subject to generate a second image, a detection means for detecting feature points related to the subject from the first image and the second image, and a determination means for determining a coordinate system serving as a reference for a detector for detecting the feature points based on the relationship between the detection result in the first image by the detection means and the detection result in the second image. Effect of the Invention

[0008] According to the present invention, it is possible to estimate a coordinate system that should be used as a reference when detecting feature points from an image. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of a functional configuration of an image processing apparatus according to an embodiment; [Diagram 2] 13 is a flowchart illustrating an example of a processing procedure for determining a coordinate system. [Diagram 3] FIG. 2 is a diagram showing an example of an original image included in an image group. [Figure 4] 11A and 11B are diagrams illustrating examples of detection results of feature points from an original image and another image. [Diagram 5]FIG. 13 is a diagram showing an example of an image in which the coordinate axes of the coordinate system W0 and W1 are superimposed on the original image. [Figure 6] 13 is a flowchart illustrating an example of a detailed processing procedure for calculating an evaluation value for a coordinate system candidate. [Figure 7] FIG. 13 is a diagram showing an example of the result of transforming the coordinates of feature points in the coordinate system W0. [Figure 8] FIG. 13 is a diagram showing an example of the result of transforming the coordinates of feature points in the coordinate system W1. [Figure 9] FIG. 13 is a diagram showing an example of a result of associating feature points using labels. [Figure 10] FIG. 13 is a diagram for explaining a position error in one pair of feature points. [Figure 11] FIG. 13 is a diagram showing position errors for pairs of feature points in one image. [Figure 12] FIG. 13 is a diagram for explaining an integrated value of position errors in each image. [Figure 13] FIG. 13 is a diagram showing an example of detecting a face area from an image. [Figure 14] FIG. 13 is a diagram showing an example of the result of generating a different image by using left-right reversal of an image. [Figure 15] 11A and 11B are diagrams illustrating examples of detection results of feature points from an original image and another image. [Figure 16] FIG. 13 is a diagram showing an example of an image in which the coordinate axes of the coordinate system W0 and W1 are superimposed on a face image. [Figure 17] FIG. 13 is a diagram showing an example of the result of transforming the coordinates of feature points in the coordinate system W0. [Figure 18] FIG. 13 is a diagram showing an example of the result of transforming the coordinates of feature points in the coordinate system W1. [Figure 19] FIG. 13 is a diagram showing an example of a result of associating feature points using labels. [Figure 20] FIG. 13 is a diagram for explaining a position error in one pair of feature points. [Figure 21] FIG. 13 is a diagram showing position errors for pairs of feature points in one image. [Figure 22] FIG. 13 is a diagram for explaining an integrated value of position errors in each image. [Diagram 23] FIG. 13 is a diagram for explaining a procedure for determining corresponding feature points when using natural feature points for which no unique label exists. [Figure 24] 13 is a flowchart showing an example of a processing procedure for calculating a position error of a certain feature points pair in Modification 5. [Diagram 25] FIG. 13 is a diagram showing an example of a result of transforming the coordinates of feature points in the coordinate system Wp in Modification 5. [Figure 26] 1A to 1C are diagrams for explaining different interpretation results of the positions of feature points on an image according to different coordinate systems. [Figure 27] FIG. 23 is a diagram showing an example of a different image created from an original image in Modification 8. [Figure 28] 23 is a flowchart showing an example of a processing procedure for further optimizing a reference coordinate system in Modification 9. [Figure 29] FIG. 11 is a diagram showing an example of a detection result of feature points for a face image. [Diagram 30] FIG. 13 is a diagram showing an example of a feature point detection result manually corrected; [Diagram 31] 1 is a block diagram illustrating an example of a hardware configuration of an image processing apparatus according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, a detailed description will be given of preferred embodiments of the present invention with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings.

[0011] First, a problem with a feature point detector used for detecting and recognizing objects in an image, estimating depth values, and the like will be described. Conventionally, the coordinate values ​​of feature points on an image are output based on the position and orientation of a predetermined coordinate system, but the position and orientation of the coordinate system to be used as the reference may be unclear. For example, when an existing detector is obtained, the position and orientation of the coordinate system to be used as the reference may not be specified in the detector, such as when information on the coordinate system to be used as the reference is not attached. As another example, even in a detector created by machine learning, the position and orientation of the reference coordinate system may be unclear because multiple types of coordinate systems are mixed as the reference for the data used for learning. As another example, even in a detector created by machine learning, a systematic error may occur in the estimated coordinate values ​​because the characteristics of the image used for learning and the image actually processed are different.

[0012] In addition, various other operations, such as mistakes and variations in accuracy when creating the true values ​​of the learning data, rounding errors when saving the values ​​indicating the feature point positions, and image cropping and scaling, can cause the position and orientation of the reference coordinate system to be unclear. In such cases, the coordinate values ​​of the feature points output by the detector are interpreted based on a certain coordinate system, but depending on the coordinate system used, errors in the analysis results and matching results of the feature points, or the survey results of the 3D shape, can become large. Below, an example of a large error in the output result is explained with reference to Figure 26.

[0013] First, in FIG. 26, there are two coordinate systems W0 and W1, and an example is shown in which the position of coordinate system W1 is shifted by α to the right of the image and β downward from the position of coordinate system W0. Then, consider a case in which a certain feature point P0 is detected from image 2600, and the two-dimensional coordinates of feature point P0 on the image are output as (x0, y0). When coordinate system W0 is interpreted as the reference, the position of feature point P0 is the position of point 2601, but when coordinate system W1 is interpreted as the reference, the position of feature point P0 is the position of point 2602. It is unclear at this stage which of coordinate systems W0 and W1 is more appropriate, but if we assume that coordinate system W1 is the appropriate coordinate system, then when coordinate system W0 is used, the position of feature point P0 will be sqrt(α 2 +β 2 ) distance.

[0014] If the coordinate system used as the reference is different in this way, the interpretation result of the position of the feature point on the image will differ, and an error may occur in the output result. Therefore, a method for identifying the position and orientation of the coordinate system to be used as the reference is desired. Alternatively, when a correct coordinate system cannot be uniquely determined, a method for optimizing the coordinate system so as to minimize the error is desired. Therefore, in the following embodiment, an example of estimating the coordinate system to be used as the reference will be specifically described.

[0015] (First embodiment) In this embodiment, an image processing device that estimates a reference coordinate system and detects feature points from a vehicle shown in an image will be described.

[0016] 31 is a block diagram showing an example of a hardware configuration of an image processing device 100 according to this embodiment. The image processing device 100 has a CPU 111, a ROM 112, a RAM 113, a HDD 114, an output I / F 115, an input I / F 116, and a communication I / F 117, which are connected by a bus 118. The CPU 111 reads out a control program stored in the ROM 112 and executes various processes. The RAM 113 is used as a temporary storage area such as a main memory and a work area for the CPU 111. The HDD 114 stores various data, various programs, and the like.

[0017] The output I / F 115 is an interface for outputting information to a display device that displays various types of information. The input I / F 116 is an interface for inputting various operations by a user via a keyboard or mouse. The communication I / F 117 is an interface for performing communication processing with an external device such as an image forming device via a network, wirelessly or wired.

[0018] Note that the functions and processes of the image processing device 100 described below are realized by the CPU 111 reading out a program stored in the ROM 112 or the HDD 114 and executing the program. As another example, the CPU 111 may read out a program stored in a recording medium such as an SD card instead of the ROM 112 or the like.

[0019] In this embodiment, the image processing device 100 is configured such that one processor (CPU 111) executes each process shown in the flowcharts described later using one memory (ROM 112), but other configurations are also possible. For example, each process shown in the flowcharts described later can be executed by using multiple processors, multiple RAMs, ROMs, and storages in cooperation with each other. Also, some of the processes can be executed using hardware circuits. Also, functions and processes of the image processing device 100 described later can be realized using a processor other than a CPU (for example, a GPU (Graphics Processing Unit) can be used instead of a CPU).

[0020] 1 is a block diagram showing an example of the functional configuration of an image processing device 100 according to this embodiment. The image processing device 100 includes a separate image generating unit 101, a feature point detecting unit 102, a coordinate system estimating unit 103, an output unit 104, a receiving unit 105, and a storage unit 106.

[0021] The receiving unit 105 receives the detector 107, the image 108, and the coordinate system candidate 109 via the communication I / F 117, and stores them in the storage unit . The separate image generating unit 101 generates a separate image from the image 108 stored in the storage unit 106 , and stores the separate image in the storage unit 106 . The feature point detection unit 102 uses a detector 107 stored in the memory unit 106 to detect feature points from the image 108 and the image generated by the separate image generation unit 101, and stores the coordinates of the detected feature points in the memory unit 106.

[0022] The coordinate system estimation unit 103 determines one of the coordinate system candidates 109 stored in the storage unit 106 as a reference coordinate system. The output unit 104 outputs information 110, which includes the coordinate system to be used as a reference and the coordinate values ​​of the feature points calculated using the coordinate system to be used as a reference, stored in the memory unit 106, to an external display device via the output I / F 115.

[0023] 2 is a flow chart showing an example of a processing procedure for determining a coordinate system to be used as a reference in this embodiment. Each process shown in FIG. 2 will be described in detail below using specific data. First, in S200, the separate image generating unit 101 acquires an image to be processed to detect feature points from the storage unit 106. Hereinafter, the image acquired in S200 is called an original image, and the image generated in S201 (described later) is called an alternative image. In this embodiment, a group of images of a vehicle photographed by a general camera is used as the original images. FIG. 3 shows an example of an image included in the image group. Generally, the number of images included in the image group is N (N is an integer equal to or greater than 1), but in this embodiment, N=10 is used for the sake of simplicity. In addition, generally, the number of vehicles appearing in each image is unknown, but in this embodiment, the number of vehicles appearing in each image is assumed to be 1 for the sake of simplicity.

[0024] If the acquired images include images of two or more vehicles, the areas in which the vehicles appear may be detected to generate cropped images, and the cropped images may be used as processing targets. The cropping of areas in which vehicles appear may be performed manually, for example, using a user interface for selecting rectangular areas on an image. Alternatively, rectangular areas in which vehicles appear may be automatically detected from an image, for example, using the method described in Non-Patent Document 1.

[0025] In S201, the different image generating unit 101 generates a different image by inputting the original image acquired in S200. In this embodiment, the different image is generated by flipping the image horizontally, but the different image may be generated by any method as long as the coordinates of corresponding feature points between the original image and the different image can be calculated. For example, the different image may be generated by using homography transformation, since the correspondence of coordinates before and after transformation can be calculated by calculation using a matrix.

[0026] In S202, the feature point detection unit 102 acquires a detector for detecting feature points from an image from the storage unit 106. In this embodiment, a feature point detector equipped with a DNN (Deep Convolutional Neural Network) generated by the method described in Non-Patent Document 2 is used, but a feature point detector prepared by another method may also be used.

[0027] In S203, the feature point detection unit 102 detects feature points from each of the original image and the separate image using the feature point detector acquired in S202. An example of the feature point detection result is shown in FIG. 4. In the method described in Non-Patent Document 2, 12 feature points are detected from one vehicle, and a unique label is assigned to each feature point. In the example shown in FIG. 4, 12 feature point labels A, B, C, J, K, and L are assigned to each image. In FIG. 4, the labels and coordinates 402 of the feature points detected from the original image 400, and the labels and coordinates 403 of the feature points detected from the separate image 401 are shown.

[0028] In S204, the coordinate system estimation unit 103 acquires a plurality of candidates for the coordinate system to be used as a reference from the storage unit 106. In this embodiment, for the sake of simplicity, two coordinate systems W0 and W1 are acquired as candidates, but three or more candidates may be acquired. The candidates for the coordinate system may be generated, for example, by analogy from data used for the learning of the detector, or may be generated by generating detection results of feature points using several images and generating the candidates by analogy based on the detection results.

[0029] Fig. 5 is a diagram showing an example of an image in which the coordinate axes of the coordinate systems W0 and W1 are superimposed on the original image acquired in S200. In the example of Fig. 5, the positions and orientations of the coordinate systems W0 and W1 are as follows. W0: The origin position is the top left position of the top left pixel of the image. The X-axis direction is the positive direction in the downward direction of the image. The Y-axis direction is the positive direction in the right direction of the image. W1: The origin position is the center position of the top left pixel of the image. The X-axis direction is the positive direction in the downward direction of the image. The Y-axis direction is the positive direction in the right direction of the image.

[0030] In S205, the coordinate system estimation unit 103 calculates an evaluation value for each of the coordinate system candidates acquired in S204. In this embodiment, the coordinate system estimation unit 103 calculates evaluation values ​​for the two coordinate systems W0 and W1 acquired in S204.

[0031] Fig. 6 is a flowchart showing an example of a detailed processing procedure for calculating the evaluation value of the coordinate system in S205 of Fig. 2. Each process shown in Fig. 6 will be described in detail below. In S600, the coordinate system estimation unit 103 converts the coordinates of the feature points detected from the other image in the candidate coordinate system for all images. Here, the conversion of coordinates refers to a process of calculating coordinates on the original image corresponding to the coordinates of the feature points detected from the other image, and the method of converting the coordinates is based on an inverse conversion that returns the other image to the original image. In this embodiment, in S201, the other image is generated by a process of flipping the original image from left to right. Therefore, as a method of inversely converting the other image to return it to the original image, a method of flipping the other image again from left to right is used. Below, conversion examples for two coordinate systems W0 and W1 as candidates for the coordinate system are shown.

[0032] FIG. 7 is a diagram showing an example of the results of generating another image 701 from an original image 700 for a coordinate system W0, detecting feature points, and converting the coordinates of the feature points. In FIG. 7, an image 702 shows an image obtained by again flipping the left and right sides of another image 701. Furthermore, coordinates (X0, Y0) represent the coordinates of the feature points detected from the original image 700. If the coordinates of the feature points detected from another image 701 are (X0', Y0'), the width of the image is L, and the coordinates after conversion are (X0", Y0"), then (X0", Y0") = (L - X0', Y0').

[0033] FIG. 8 is a diagram showing an example of the results of generating another image 801 from an original image 800 for coordinate system W1, detecting one feature point, and converting the coordinates of the feature point. Image 802 shows an image obtained by again flipping the other image 801 horizontally. Furthermore, coordinates (X1, Y1) represent the coordinates of the feature point detected from the original image 800. If the coordinates of the feature point detected from the other image 801 are (X1', Y1'), the horizontal width of the image is L, the pixel radius is α, and the coordinates after conversion are (X1", Y1"), then (X1", Y1") = (L-X1'-2α, Y1').

[0034] In S601, the coordinate system estimation unit 103 performs correspondence between feature points detected from the original image detected in S203 and feature points detected from another image for all images. In general, there are two methods for detecting feature points: a method for determining the type of feature point (for example, labels such as the right eye and nose for a face) and a method for not determining the type. When the type of feature point is determined, the feature points are associated using the type. Note that, in the case where the type of feature point is not determined, a method for performing correspondence between feature points will be described later in Modification 4.

[0035] As described above, since each feature point detected in S203 is assigned a unique label, in this embodiment, the feature points detected from the original image and the feature points detected from the different image are associated with each other using the unique labels. Since the different image is generated by the process of flipping the original image horizontally in S201, the feature points are associated with each other taking into consideration that the positions of the detected feature points are flipped horizontally.

[0036] Fig. 9 is a diagram showing an example of the result of matching 12 detected feature points using labels A, B, C, ..., J, K, and L. In Fig. 9, table 902 shows the result of matching feature points. The feature point pairs P0, P1, P2, ..., P included in table 902 are 11 is generated taking into consideration that another image 901 is generated by flipping the original image 900 horizontally. For example, feature point A included in the original image 900 corresponds to feature point B included in the another image 901 when the horizontal flip is taken into consideration. Therefore, pair P0 included in table 902 is a pair of feature point A included in original image 900 and feature point B included in another image 901. By performing a similar process to associate feature points, pairs P1, P2, ..., P included in table 902 are generated. 11 is also generated.

[0037] In S602, the coordinate system estimation unit 103 calculates a position error for each pair of feature points associated in S601 in all candidate coordinate systems. The position error is calculated by squaring the Euclidean distance between the position of the feature point detected from the original image and the position obtained by converting the position of the feature point detected from another image in S600.

[0038] Fig. 10 is a diagram for explaining a position error in a certain pair of feature points. In Fig. 10, point 1000 indicates a feature point detected from the original image. Point 1001 indicates a point obtained by converting the position of the feature point paired with point 1000 by the process of S600. Furthermore, arrow 1002 indicates the Euclidean distance between point 1000 and point 1001. In this embodiment, a value obtained by squaring the Euclidean distance is calculated as the position error.

[0039] In S603, the coordinate system estimation unit 103 calculates the evaluation value of the coordinate system in all candidate coordinate systems. The evaluation value of the coordinate system is set to be higher as the position error calculated in S602 is smaller. In this embodiment, the position error calculated in S602 is integrated for all images and all feature point pairs, and the negative number is used as the evaluation value. In the following, for the sake of simplicity, the description will be given assuming that the number of feature point pairs in each image is 12 and the number of images is 10. Note that the number of feature point pairs in each image and the number of images are arbitrary, and calculations can be performed in the same way even if the number of feature point pairs and the number of images are other values. For example, there are cases where the number of feature point pairs differs for each image because images in which the entire vehicle is captured and images in which the entire vehicle is not captured are mixed. In addition, the number of images to be calculated may be reduced from the number of images initially prepared due to factors such as image quality and detection performance. In such cases, calculations can be performed in the same way.

[0040] Fig. 11 is a diagram showing position errors for each pair of feature points in one image. In Fig. 11, arrows 1100 to 1111 indicate the Euclidean distance between two points in each of the 12 pairs of feature points calculated in S602. First, an integrated value E of position errors in one image is calculated by adding up all the squares of the 12 Euclidean distances (position errors).

[0041] Next, the sum of the integrated values ​​of the 10 images with the negative sign is used as the evaluation value of the coordinate system. As shown in Fig. 12, for example, if the integrated values ​​of the position errors of the 10 images 1200, 1201, . . . 1208, 1209 are E0, E1, . . . , E8, E9, and the evaluation value of the coordinate system is V, then V = -ΣE i (i=0,1,···9).

[0042] Returning to the explanation of FIG. 2, in S206, the coordinate system estimation unit 103 selects one coordinate system to be used as a reference. Specifically, one coordinate system having the largest evaluation value among the evaluation values ​​calculated in S205 is selected. In this embodiment, of the two coordinate systems W0 and W1 prepared in S204, the one having the larger evaluation value calculated in S205 is selected. Through the above process, it is possible to select a coordinate system to be used as a reference from a plurality of coordinate system candidates prepared in advance.

[0043] As described above, when the position and orientation of the reference coordinate system is unclear, an error occurs in the interpretation of the coordinates output by the feature point detector, and the accuracy of the detected positions of the feature points decreases. According to this embodiment, the evaluation value of each of the different candidate coordinate systems is calculated by the above procedure, and the coordinate system with the largest evaluation value is selected. This makes it possible to efficiently estimate the position and orientation of the reference coordinate system, and to suppress a decrease in the accuracy of the detected positions of the feature points.

[0044] Second Embodiment In the first embodiment, an example of a method for selecting one coordinate system to be used as a reference from candidates of coordinate systems that are used as a reference when interpreting the coordinates of feature points in a case where feature points are detected from an image of a vehicle captured by a general camera has been described. In this embodiment, a method for selecting a coordinate system to be used as a reference for detecting feature points with high accuracy from a face image will be described. The internal configuration of the image processing device in this embodiment is similar to the configuration shown in FIG. 1 and FIG. 31 as in the first embodiment, and therefore a description thereof will be omitted. In addition, in this embodiment, the procedure for selecting a coordinate system for detecting feature points with high accuracy is basically the same as the procedure shown in FIG. 2 as in the first embodiment. Below, the process shown in FIG. 2 will be described in detail, including specific data examples.

[0045] In S200, the separate image generating unit 101 acquires an image to be processed. In this embodiment, a face area is detected from a group of images captured by photographing a person with a general camera and stored to acquire a face image. In this embodiment, the face area of ​​a person is detected using the method described in Non-Patent Document 3, but another method may be used as long as it can detect the face area of ​​a person.

[0046] Fig. 13 is a diagram showing an example of detecting a face region from an image. In Fig. 13, image 1300 represents an image included in an image group obtained by photographing a person with a general camera, and face regions 1301, 1302, and 1303 are detected by the method described in Non-Patent Document 3. Furthermore, face images 1304, 1305, and 1306 are images obtained by cutting out and saving face regions 1301, 1302, and 1303, respectively. Generally, the number of face images included in an image group is N (N is an integer equal to or greater than 1), but in this embodiment, N=10 is used to simplify the explanation.

[0047] In S201, the separate image generating unit 101 generates separate images by using the face image (original image) acquired in S200. The method of generating separate images is basically the same as in the first embodiment.

[0048] Fig. 14 is a diagram showing an example of the result of generating a different image by using left-right reversal of an image. In Fig. 14, images 1400, 1401, and 1402 each represent an example of a face image (original image) acquired in S200, and images 1403, 1404, and 1405 each represent an example of a different image generated in S201.

[0049] In S202, the feature point detection unit 102 acquires a detector for detecting facial organ points as feature points from the image from the storage unit 106. This process is basically the same as in the first embodiment except that the type of detector is different.

[0050] In S203, the feature point detection unit 102 detects feature points from the original image acquired in S200 and the separate image generated in S201 using the feature point detector acquired in S202. In this embodiment, feature points are detected from a face image using the method described in Non-Patent Document 3. An example of a feature point detection result is shown in FIG. 15. In the method described in Non-Patent Document 3, five feature points (center of the left eye, center of the right eye, center of the nose, right end point of the mouth, left end point of the mouth) are detected from one face region, and a unique label is assigned to each of the feature points. Hereinafter, each unique label is referred to as Left eye, Right eye, Nose, Mouth left, and Mouth right. In FIG. 15, table 1502 shows the labels of the five feature points detected from the original image 1500 and the coordinate values ​​of each of the feature points. Also, table 1503 shows the labels of the five feature points detected from the separate image 1501 and the coordinate values ​​of each of the feature points.

[0051] In S204, the coordinate system estimation unit 103 acquires a plurality of candidates for the coordinate system to be used as a reference from the storage unit 106. As in the first embodiment, to simplify the explanation, two coordinate systems W0 and W1 are acquired as candidates, but three or more candidates may be acquired. Note that FIG. 16 shows an example of an image in which the coordinate axes of the coordinate systems W0 and W1 are superimposed on the face image acquired in S200. The position and orientation of the coordinate systems W0 and W1 are the same as in the first embodiment.

[0052] In S205, the coordinate system estimation unit 103 calculates an evaluation value for each of the coordinate system candidates acquired in S204. As in the first embodiment, the processing procedure in S205 basically follows the procedure shown in Fig. 6. Details will be described below with reference to the drawings.

[0053] In S600, the coordinate system estimation unit 103 converts the coordinates of the feature points detected from the other image. FIG. 17 is a diagram showing an example of a case where, for the coordinate system W0, another image 1701 is generated from an original image 1700, feature points are detected, and the coordinates of the feature points are converted. In FIG. 17, an image 1702 shows an image obtained by again flipping the other image 1701 horizontally. Furthermore, coordinates (X0, Y0) represent the coordinates of the feature points detected from the original image 1700. If the coordinates of the feature points detected from the other image 1701 are (X0', Y0'), the horizontal width of the image is L, and the coordinates after conversion are (X0", Y0"), then (X0", Y0") = (L-X0', Y0').

[0054] FIG. 18 is a diagram showing an example of a case where, for coordinate system W1, another image 1801 is generated from an original image 1800, one feature point is detected, and the coordinates of the feature point are transformed. Image 1802 shows an image obtained by again flipping another image 1801 horizontally. Furthermore, coordinates (X1, Y1) represent the coordinates of the feature point detected from original image 1800. If the coordinates of the feature point detected from another image 1801 are (X1', Y1'), the pixel radius is α, the horizontal width of the image is L, and the transformed coordinates are (X1", Y1"), then (X1", Y1") = (L-X1'-2α, Y1').

[0055] In S601, the coordinate system estimation unit 103 associates feature points detected from an original image with feature points detected from another image. Fig. 19 is a diagram showing the result of associating five detected feature points A to E using the labels Left eye, Right eye, Nose, Mouth left, and Mouth right, respectively. In Fig. 19, table 1902 shows the association between the label names of feature points and the symbols of feature points, and table 1903 shows the result of the feature point association.

[0056] In S602, the coordinate system estimation unit 103 calculates a position error for each pair of feature points associated in S601. As in the first embodiment, the calculated position error is the square of the Euclidean distance between the feature points. An arrow 2002 in Fig. 20 indicates the Euclidean distance between a feature point 2000 detected from the original image and a point 2001 obtained by conversion in S600.

[0057] In S603, the coordinate system estimation unit 103 calculates an evaluation value of the coordinate system. As shown in Fig. 21 and Fig. 22, by using the same procedure as in the first embodiment, integrated values ​​E0, E1, ... E9 of position errors indicated by arrows 2100-2104 are calculated for ten face images 2200-2209. Then, an evaluation value V is obtained for each coordinate system.

[0058] In S206, the coordinate system estimation unit 103 can efficiently estimate a reference coordinate system for the face image as described above, as in the first embodiment, and can suppress a decrease in accuracy of the detection positions of feature points.

[0059] <Variation 1> In the first and second embodiments, the images for which the feature point detection unit 102 detects feature points are an original image input from outside and a left-right inverted separate image generated by the separate image generation unit 101. Therefore, the resolution of the images for which the feature point detection unit 102 detects feature points is only the resolution of the original image input from outside.

[0060] On the other hand, the accuracy of detecting feature points by the detector may vary depending on the resolution of the input image. Therefore, an image with a changed resolution of the image input from the outside may be generated and used as a new image to be processed. For example, when a specific subject is detected from an image received by the receiving unit 105 and the detection of the subject is successful, an image with the length and width of the image 108 multiplied by 0.5 and an image with the length and width multiplied by 2.0 are generated as new input images and stored in the storage unit 106. Then, the same subject may be detected again in the newly generated image, and multiple images to be processed may be generated by repeating this process until the subject cannot be detected.

[0061] In addition, when the range of image resolution that can be processed is known as the specifications of the detector input from outside, the image received by the receiving unit 105 may be enlarged or reduced so that it falls within the range of resolution that can be processed by the detector, thereby generating a new image to be processed.

[0062] <Variation 2> In the first and second embodiments, the images for which the feature point detection unit 102 detects feature points are an original image input from outside and a left-right inverted separate image generated by the separate image generation unit 101. Therefore, the resolution of the images for which the feature point detection unit 102 detects feature points is only the resolution of the original image input from outside.

[0063] On the other hand, when detecting feature points, there are cases where it is desired to detect optimal feature points for an image with a certain resolution. For example, if a certain resolution at which it is desired to detect optimal feature points is R and the resolution of an original image input from outside matches the resolution R, the separate image generating unit 101 generates a separate image using only a conversion method that does not change the resolution of the original image. On the other hand, if the resolution of an original image input from outside does not match the resolution R, the separate image generating unit 101 may generate a separate image using only a conversion method that makes the resolution of the separate image the resolution R.

[0064] <Variation 3> In the first and second embodiments, the number of original images input from the outside is arbitrary, and one reference coordinate system is determined for all target images. On the other hand, it may be more effective to classify the original images input from the outside into one of a plurality of sets and determine one reference coordinate system for each set. In such a case, the CPU 111 may function as a classification unit to classify the original images input from the outside into a plurality of image groups and estimate the reference coordinate system and detect feature points for each image group.

[0065] For example, the original image input from the outside may include an image with a general angle of view and an image with a wide angle of view. In such a case, the image may be divided into a plurality of image groups based on the aspect ratio of the image included in the image information, the type of lens used for shooting, and other shooting conditions, and the coordinate system to be used as a reference and the feature points may be detected for each image group. Also, the direction of the light source may be estimated from the image, and the image may be divided into a plurality of image groups based on the light source direction, and the coordinate system to be used as a reference and the feature points may be detected for each image group. Furthermore, the image may be divided into a plurality of image groups based on the orientation and size of the face in the image, and the coordinate system to be used as a reference and the feature points may be detected for each image group.

[0066] <Modification 4> In the first and second embodiments, each of the detected feature points has a unique label, and the feature points are detected and associated with each other. However, a method for detecting feature points that does not have a unique label may be used.

[0067] In the fields of camera position and orientation estimation, 3D reconstruction, stereo matching, and the like, a method is sometimes used to detect natural feature points that do not have unique labels without assuming a specific subject. For example, as in the method described in Non-Patent Document 4, a method may be used that uses an image containing an arbitrary subject as input and detects natural feature points that do not have unique labels.

[0068] On the other hand, when using natural feature points for which no unique label exists, it is necessary to associate feature points using a method different from the methods of the first and second embodiments. In such a case, for example, for each feature point detected from the original image, the feature point with the closest coordinates among those obtained by converting feature points detected from another image may be determined as the corresponding feature point.

[0069] FIG. 23 is a diagram for explaining a procedure for determining corresponding feature points. In FIG. 23, feature point 2300 indicates a feature point detected from an original image. Points 2301 to 2306 indicate the positions of feature points obtained by detecting feature points from another image and then performing coordinate conversion in S600 in FIG. 6. In the example of FIG. 23, point 2301, which is the smallest distance from feature point 2300 detected from the original image, is selected as the point corresponding to feature point 2300 detected from the original image. In addition, when a unique feature amount or feature vector is defined for each feature point, the feature point with the smallest difference in feature amount or feature vector may be determined as the corresponding feature point.

[0070] <Variation 5> In the first and second embodiments, a plurality of coordinate system candidates are set in advance, and one coordinate system is selected from among them. However, instead of setting a plurality of coordinate system candidates, a reference coordinate system may be set by defining one coordinate system having variable parameters for position and orientation, and optimizing the parameters through analytical calculation.

[0071] Also, similarly to the first and second embodiments, a plurality of coordinate system candidates may be set in advance, and a coordinate system having position and orientation parameters may be included as one of the candidates. In this case, in the coordinate system having no position and orientation parameters, an evaluation value is calculated by the method described in the first and second embodiments, and in the coordinate system having position and orientation parameters, an evaluation value is calculated by optimizing parameters by a method described later. A method for optimizing parameters of a coordinate system having position and orientation parameters will be described in detail below.

[0072] First, we consider the coordinate system W0, which is the basis for analytical calculations, and the coordinate system W with two parameters (α, β). p Here, the coordinate system W0 is the same as in the first and second embodiments. On the other hand, the coordinate system W p is a coordinate system that is the reference for the coordinates of feature points output by the feature point detector, and is defined as follows: W p:The X-axis direction is positive when facing downward in the image. The Y-axis direction is positive when facing right in the image. The origin position (α, β) is shifted α in the X-axis direction and β in the Y-axis direction from the origin of the coordinate system W0.

[0073] Coordinate system W p In this case, if a general affine transformation is used to generate another image, the integrated value E of the position error calculated in S603 in Fig. 6 will be a quadratic expression of the parameters α and β. Here, since the parameters α and β are independent of each other, the parameters α and β that minimize the position error can be found by solving a general quadratic function minimization problem. Below, a method of expressing the position error for one feature point as a quadratic function with the parameters α and β as variables and calculating the parameters α and β that minimize the integrated value E of the position error will be described.

[0074] Fig. 24 is a flowchart showing an example of a processing procedure for calculating a position error of a certain feature points pair. Each process shown in Fig. 24 is performed as a part of the process of S602 in Fig. 6. Each process shown in Fig. 24 will be described below with reference to Fig. 25.

[0075] In S2400, the feature point detection unit 102 detects feature points from the original image. In FIG. 25, the detector detects feature points in the coordinate system W p If the coordinates of the feature point 2503 outputted based on the coordinate system W0 are (X1, Y1), then the coordinates of the feature point 2503 when the coordinate system W0 is used as the reference are (X1+α, Y1+β).

[0076] In S2401, the different image generating unit 101 generates a different image from an original image. In this modified example, the different image is generated using a general affine transformation. If a point on the original image 2500 is (x, y), a point on the different image 2501 is (x', y'), and an affine transformation matrix for generating the different image 2501 from the original image 2500 is A, the following formula (1) is established.

[0077]

number

[0078] Moreover, the matrix A in the formula (1) can generally be defined by the following formula (2).

[0079]

number

[0080] In S2402, the feature point detection unit 102 detects feature points from another image. In FIG. 25, it is assumed that a feature point 2504 detected from another image 2501 corresponds to a feature point 2503 detected from the original image. Here, the detector detects feature points in the coordinate system W p If the coordinates of the feature point 2504 outputted based on the coordinate system W0 are (X1', Y1'), then the coordinates of the feature point 2504 when the coordinate system W0 is used as the reference are (X1'+α, Y1'+β).

[0081] In S2403, the coordinate system estimation unit 103 transforms the coordinates of the feature points detected from the different image. In FIG. 25, an image 2502 is an image generated by an affine transformation (hereinafter, referred to as an inverse affine transformation) that returns another image 2501 to an original image 2500. Here, if an affine transformation matrix for the inverse affine transformation is A', then the matrix A' is the inverse matrix of the matrix A. Furthermore, a feature point 2505 is obtained by transforming the feature point 2504 by the inverse affine transformation. Therefore, if the coordinates of the feature point 2505 in the case where the coordinate system W0 is used as a reference are set to (X1", Y1"), then the following formula (3) is established. Furthermore, by defining the matrix A' in formula (3) by the following formula (4), the following two formulas (5) and (6) are obtained.

[0082]

number

[0083] On the other hand, the coordinate system W p When the coordinates of the feature point 2505 are taken as (X1''',Y1''') with respect to the reference point, the following equations (7) and (8) hold.

[0084]

number

[0085] In S2404, the coordinate system estimation unit 103 calculates a position error. If the value of the position error is denoted as Diff, the magnitude of the position error is the square of the distance between the feature point 2503 and the feature point 2505, and therefore the following equation (9) holds.

[0086]

number

[0087] Here, since the right-hand sides of equations (5) to (8) are linear expressions of the parameters α and β, when equations (5) to (8) are substituted into equation (9), the right-hand side of equation (9) becomes a quadratic function of the two parameters α and β.

[0088] Next, we will explain how to calculate the parameters α and β that minimize the integrated value E of the position error. When the number of feature points to be calculated is m (m is an integer equal to or greater than 1), and the position error for each feature point is Diff_i (i=1, 2, , m), the integrated value E of the position error can be defined by the following formula (10).

[0089]

number

[0090] Here, the integrated value E is the sum of the position errors of each feature point, so it is understood to be a quadratic function of two parameters α and β, similar to equation (9). Since the parameters α and β are independent, the parameters α and β that minimize the integrated value E can be found as a general quadratic function minimization problem. Note that, although it is possible to find the parameters α and β that minimize the integrated value E of the position error if there is at least one feature point, from a statistical point of view, it is possible to find more optimal parameters α and β by using as many feature points as possible.

[0091] In addition, in this modified example, an example has been shown in which two parameters α and β representing a position shift are optimized among the parameters that characterize the coordinate system, but an affine parameter with six degrees of freedom including a shear deformation as shown in the following formula (11) may be the target of optimization. In this case, six variables can be optimized by using three or more feature points. As described above, various forms are possible for optimizing the coordinate system, and the method is not limited to a specific method.

[0092]

number

[0093] <Variation 6> In the first and second embodiments, the processing target is an image of a specific object captured by a general camera. However, depending on the subject, it may be difficult to collect a sufficient number and variety of images. In such a case, in addition to the images captured by the general camera, artificially generated images may be used, or the reference coordinate system may be estimated only using the artificially generated images.

[0094] For example, when acquiring a face image of a person, a face image taken from the front direction can be easily acquired, but a face image taken from an oblique direction or a face image with a strong cast shadow due to the position of the light source may not be easily acquired. Therefore, a face image taken from an oblique direction or a face image with a strong cast shadow may be artificially generated, and the face image taken from the front direction and the artificially generated face image may be used simultaneously. The artificial image is generated, for example, by rendering a 3DCG model of a person with various settings.

[0095] <Variation 7> In the first and second embodiments, in order to determine a reference coordinate system, a position error is calculated for each image in the acquired image group. Here, an image with a large position error calculated using the reference coordinate system is likely to have low position accuracy of detected feature points. Therefore, this may be applied to select images from the image group based on the position error.

[0096] For example, when generating face images for training or evaluating a DNN for face recognition, the face images may be generated based on the positions of feature points detected from the face. For example, in the method described in Non-Patent Document 5, an affine transformation is performed so that five feature points detected from a face image are as close as possible to five predetermined coordinates, and a group of normalized images with the same size is generated.

[0097] Here, as face images used for learning or evaluation, images with feature point position accuracy less than a predetermined value may be removed as noise. For example, in the second embodiment, face images 2200-2209 shown in Fig. 22 may be subjected to noise determination based on integrated values ​​E0-E9 of position errors of the respective images. In this case, since face images 2200-2209 have different image sizes, integrated values ​​E0-E9 of position errors are normalized by the image size, respectively, and normalized integrated values ​​E0'-E9' are calculated. Images with integrated values ​​equal to or greater than a predetermined value are then determined as noise.

[0098] Furthermore, normalization for calculating the integrated values ​​E0' to E9' may be unnecessary by receiving an input image that has been normalized to a specific size in advance at the receiving unit 105. For example, if a group of normalized images with the same size has been generated in advance by the method described in Non-Patent Document 5, the group of normalized images may be treated as input images. Then, an integrated value E of the position error of each image may be calculated according to the procedure shown in FIG. 2, and an image with an integrated value E equal to or greater than a predetermined value may be determined to be noise.

[0099] <Variation 8> In the first and second embodiments, when generating another image from an original image, a conversion process for flipping the image horizontally is used. However, the other image may be generated by any method as long as the coordinates of corresponding feature points between the original image and the other image can be calculated. For example, the other image may be generated by using up-down flipping, translation, enlargement or reduction, rotation, shearing, etc. Also, the other image may be generated by a conversion (hereinafter referred to as trapezoidal conversion) in which a rectangular area on the original image becomes a trapezoid on the other image. Furthermore, a conversion that combines these conversions may be used.

[0100] Fig. 27 is a diagram showing an example of another image generated from an original image. In Fig. 27, another image 2701 is an image generated by translating an original image 2700. Another image 2702 is an image generated by enlarging the original image 2700, and another image 2703 is an image generated by reducing the original image 2700. Another image 2704 is an image generated by rotating the original image 2700, and another image 2705 is an image generated by shearing the original image 2700. Also, another image 2706 is an image generated by subjecting the original image 2700 to trapezoidal transformation.

[0101] <Variation 9> In the first and second embodiments, the feature points detected from the original image and another image are associated with each other, and the coordinate system to be used as a reference is determined based on the distance between the corresponding feature points. As another modification, the reference coordinate system may be further optimized by using the true values ​​of the coordinates at this time. Below, a procedure for determining a further optimal coordinate system using the coordinate system determined in the second embodiment as an input will be described.

[0102] Fig. 28 is a flowchart showing an example of a processing procedure for further optimizing the reference coordinate system. Each process shown in Fig. 28 will be described below using a specific example. In S2800, the coordinate system estimation unit 103 acquires an image that is a target of processing for detecting feature points. In this modification, for example, face images 2200 to 2209 in FIG.

[0103] In S2801, the coordinate system estimation unit 103 acquires a previously determined reference coordinate system. In this modification, it is assumed that the coordinate system W1 in the second embodiment is the optimal coordinate system, and the coordinate system W1 is acquired.

[0104] In S2802, the coordinate system estimation unit 103 acquires detection results of feature points previously detected. In this modification, the detection results of feature points based on the coordinate system W1 in the face images 2200 to 2209 in Fig. 22 are acquired. Here, Fig. 29 is a diagram showing an example of the detection results of feature points previously detected for the face image 2200. In Fig. 29, feature points 2900 to 2904 indicate feature points previously detected.

[0105] In S2803, the coordinate system estimation unit 103 generates true values ​​of the feature points. In this modification, information on the correct positions of the feature points is generated for each of the face images 2200 to 2209 by manual work using a user interface via the input I / F 116. Feature points 3000 to 3004 shown in Fig. 30 indicate the positions of five feature points determined manually using a user interface.

[0106] In S2804, the coordinate system estimation unit 103 acquires a coordinate system having parameters. p is used as a parameterized coordinate system.

[0107] In S2805, the coordinate system estimation unit 103 calculates the parameters that maximize the evaluation value. p In this modification, however, the coordinate system W is calculated by using pairs of the feature points acquired in S2802 and the true values ​​prepared in S2803. p Optimize the parameters.

[0108] In S2806, the coordinate system estimation unit 103 determines one reference coordinate system. Specifically, the coordinate system W having the parameters obtained in S2805 is p is the reference coordinate system.

[0109] <Modification 10> In the first and second embodiments, the reference coordinate system is determined using the original image and the other image. On the other hand, a plurality of types of other images may be generated from the original image, and the reference coordinate system may be determined using the plurality of types of other images. For example, in the second embodiment, the target for detecting feature points is a person's face, but both a first other image obtained by performing homography transformation so that the face in the original image is at the center of the image, and a second other image generated by flipping the first other image horizontally may be generated. Then, the reference coordinate system may be determined using the first other image and the second other image.

[0110] <Modification 11> In the first and second embodiments, feature points are detected from a vehicle or a person's face, but other objects may be used as the detection target for feature points. For example, joint positions such as elbows, hips, and knees may be detected as feature points from the whole body of a person, or a method of detecting natural feature points that do not have a specific subject and do not have a unique label may be used.

[0111] <Modification 12> In the first and second embodiments, one reference coordinate system is determined to be applied to all feature points detected by the feature point detector. However, depending on the characteristics of the feature point detector, the tendency of the coordinate positions may differ for each subset of feature points. Therefore, the detected feature points may be divided into a plurality of subsets, and one reference coordinate system may be determined for each subset of feature points. For example, when detecting feature points from a person's face, two subsets are formed: a set of only feature points detected from the right and left eyes, and a set of only feature points detected from the left and right ends of the mouth. Then, a reference coordinate system may be determined for each of these subsets. Also, as shown in Modification 11, when detecting feature points from the entire body of a person, a set of only feature points detected from the upper body may be treated as one subset, and one reference coordinate system may be determined.

[0112] <Modification 13> In the first and second embodiments, the feature point detector only has a process of detecting feature points from an input image, but a detector including multiple types of processes may be used. For example, the detector may include a process of generating a partial image by cutting out a specific object area from the input image, or a normalization process of converting the input image or the partial image to a predetermined size, orientation, etc. Furthermore, the detector may include a process of converting the number of digits of a numerical value indicating the coordinates of a detected feature point to a predetermined number of digits. Note that in a detector including multiple types of processes, the accuracy, error, characteristics, etc. in each process affect the detection result of feature points, but by using a procedure similar to that of the first and second embodiments, one reference coordinate system can be determined taking all of the processes into consideration.

[0113] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0114] The disclosure of this embodiment includes the following configuration, method, and program.

[0115] (Configuration 1) a generating means for performing a predetermined conversion process on a first image including a subject to generate a second image; a detection means for detecting feature points related to the subject from the first image and the second image; a determination means for determining a coordinate system serving as a reference for a detector for detecting the feature points, based on a relationship between a detection result in the first image by the detection means and a detection result in the second image; 13. An image processing device comprising:

[0116] (Configuration 2) The image processing device according to configuration 1, wherein the determining means determines the reference coordinate system based on a difference between a result of performing an inverse transformation of the predetermined transformation process on the feature points detected from the second image by the detecting means and a detection result from the first image by the detecting means. (Configuration 3) The image processing device according to configuration 2, wherein the determination means performs correspondence between a result of performing an inverse transformation of the predetermined transformation process on the feature points detected from the second image by the detection means and a detection result from the first image by the detection means, and determines the reference coordinate system based on a positional difference between the corresponding feature points. (Configuration 4) The detection means detects feature points related to the subject and assigns labels to the detected feature points; 4. The image processing device according to configuration 3, wherein the determining means performs correspondence between feature points based on the assigned labels. (Configuration 5) 4. The image processing device according to configuration 3, wherein the determining means associates feature points with each other such that a distance between the coordinates of the feature points is the smallest. (Configuration 6) 4. The image processing device according to configuration 3, wherein the determining means associates feature points having a smallest difference in feature amount or feature vector between the feature points.

[0117] (Configuration 7) The coordinate system further includes an acquisition unit that acquires a candidate for the coordinate system determined by the determination unit, The image processing device according to any one of configurations 2 to 6, wherein the determination means performs an inverse transformation of the predetermined transformation process for each candidate acquired by the acquisition means, and determines the candidate having the smallest difference in the detection results as the reference coordinate system. (Configuration 8) The candidate coordinate systems include a coordinate system having a variable parameter, 8. The image processing device according to claim 7, wherein the determining means determines the reference coordinate system by calculating the parameters so as to minimize a difference in the detection results. (Configuration 9) 9. The image processing device according to any one of configurations 1 to 8, wherein the determining means determines the reference coordinate system by using true values ​​of the coordinates of the feature points. (Configuration 10) The detection means detects feature points related to the subject and classifies the detected feature points into a plurality of groups; 9. The image processing device according to any one of configurations 1 to 8, wherein the determining means determines the reference coordinate system for each of the sets.

[0118] (Configuration 11) further comprising a classification means for classifying the first image into one of a plurality of groups based on a photographing condition of the first image; 9. The image processing device according to any one of configurations 1 to 8, wherein the determining means determines the reference coordinate system for each set classified by the classifying means. (Configuration 12) The image processing device according to any one of configurations 1 to 8, wherein the generating means enlarges or reduces the first image, and performs the predetermined conversion processing on the enlarged or reduced first image to generate a second image. (Configuration 13) 9. The image processing device according to any one of configurations 1 to 8, wherein the generating means generates the second image so as to have a predetermined resolution. (Configuration 14) 9. The image processing device according to any one of configurations 1 to 8, wherein the predetermined transformation process is horizontal inversion, vertical inversion, translation, enlargement, reduction, rotation, shear, trapezoid transformation, or a combination thereof. (Configuration 15) 9. The image processing device according to any one of configurations 1 to 8, wherein the predetermined transformation process is homography transformation. (Configuration 16) The image processing device according to any one of configurations 1 to 8, further comprising a selection means for selecting a first image in which an integrated value of the difference between a result of performing an inverse transformation of the predetermined transformation process on feature points detected from the second image by the detection means and a detection result from the first image by the detection means is less than a predetermined value, as an image to be used when the determination means determines the reference coordinate system.

[0119] (method) a generating step of performing a predetermined conversion process on a first image including a subject to generate a second image; a detection step of detecting feature points related to the subject from the first image and the second image; a determination step of determining a coordinate system serving as a reference for a detector for detecting the feature points, based on a relationship between a detection result in the first image and a detection result in the second image in the detection step; 13. An image processing method comprising:

[0120] (program) a generating step of performing a predetermined conversion process on a first image including a subject to generate a second image; a detection step of detecting feature points related to the subject from the first image and the second image; a determination step of determining a coordinate system serving as a reference for a detector for detecting the feature points, based on a relationship between a detection result in the first image and a detection result in the second image in the detection step; A program for causing a computer to execute the following. [Explanation of symbols]

[0121] 101 separate image generation unit, 102 feature point detection unit, 103 coordinate system estimation unit

Claims

1. a generating means for performing a predetermined conversion process on a first image including a subject to generate a second image; a detection means for detecting feature points related to the subject from the first image and the second image; a determination means for determining a coordinate system serving as a reference for a detector for detecting the feature points, based on a relationship between a detection result in the first image by the detection means and a detection result in the second image; 13. An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the determining means determines the reference coordinate system based on a difference between a result of performing an inverse transformation of the predetermined transformation process on the feature points detected from the second image by the detecting means and a detection result from the first image by the detecting means.

3. 3. The image processing device according to claim 2, wherein the determining means performs a correspondence between a result of performing an inverse transformation of the predetermined transformation process on the feature points detected from the second image by the detecting means and a detection result from the first image by the detecting means, and determines the reference coordinate system based on a positional difference between the corresponding feature points.

4. The detection means detects feature points related to the subject and assigns labels to the detected feature points; The image processing apparatus according to claim 3 , wherein the determining means performs correspondence between feature points based on the assigned labels.

5. 4. The image processing apparatus according to claim 3, wherein the determining means associates feature points with each other such that a distance between the coordinates of the feature points is the smallest.

6. 4. The image processing apparatus according to claim 3, wherein the determining means performs correspondence between the feature points having the smallest difference in feature amount or feature vector between the feature points.

7. The coordinate system further includes an acquisition unit that acquires a candidate for the coordinate system determined by the determination unit, 4. The image processing device according to claim 2, wherein the determining means performs an inverse transformation of the predetermined transformation process for each candidate acquired by the acquiring means, and determines the candidate having the smallest difference in the detection results as the reference coordinate system.

8. The candidate coordinate systems include a coordinate system having a variable parameter, 8. The image processing apparatus according to claim 7, wherein said determining means determines said reference coordinate system by calculating said parameters so as to minimize a difference in said detection results.

9. 2. The image processing apparatus according to claim 1, wherein said determining means determines said reference coordinate system by using true values ​​of the coordinates of said feature points.

10. The detection means detects feature points related to the subject and classifies the detected feature points into a plurality of groups; 2. The image processing apparatus according to claim 1, wherein said determining means determines the reference coordinate system for each of said sets.

11. The method further comprises: classifying the first image into one of a plurality of groups based on a photographing condition of the first image; 2. The image processing apparatus according to claim 1, wherein the determining means determines the reference coordinate system for each set classified by the classifying means.

12. 2 . The image processing device according to claim 1 , wherein the generating means enlarges or reduces the first image, and performs the predetermined conversion processing on the enlarged or reduced first image to generate a second image.

13. 2. The image processing apparatus according to claim 1, wherein the generating means generates the second image so as to have a predetermined resolution.

14. 2. The image processing device according to claim 1, wherein the predetermined transformation process is horizontal inversion, vertical inversion, translation, enlargement, reduction, rotation, shear, trapezoid transformation, or a combination thereof.

15. 2. The image processing apparatus according to claim 1, wherein the predetermined transformation process is a homography transformation.

16. 2. The image processing device according to claim 1, further comprising a selection means for selecting a first image in which an integrated value of the difference between the result of performing an inverse transformation of the predetermined transformation process on the feature points detected from the second image by the detection means and the detection result from the first image by the detection means is less than a predetermined value, as an image to be used when the determination means determines the reference coordinate system.

17. A generating step of performing a predetermined conversion process on a first image including a subject to generate a second image; a detection step of detecting feature points related to the subject from the first image and the second image; a determination step of determining a coordinate system serving as a reference for a detector for detecting the feature points, based on a relationship between a detection result in the first image and a detection result in the second image in the detection step; 13. An image processing method comprising:

18. A generating step of performing a predetermined conversion process on a first image including a subject to generate a second image; a detection step of detecting feature points related to the subject from the first image and the second image; a determination step of determining a coordinate system serving as a reference for a detector for detecting the feature points, based on a relationship between a detection result in the first image and a detection result in the second image in the detection step; A program for causing a computer to execute the following.

Images