Information Processing Apparatus, Control Method, Program, and Storage Medium

Abstract

To provide an information processing apparatus that can appropriately calculate the normal line of an object having a flat surface.SOLUTION: An on-vehicle machine 1 has image acquisition means, projective transformation means, and normal line calculation means. The image acquisition means acquires a first image including an object having a flat surface and a second image including the object. The projective transformation means performs projective transformation of the first image based on the relative photographing position and photographing posture of the second image to the first image, the distance from the photographing position of the first image to the object, and a provisional value of the normal line of the object. The normal line calculation means calculates the normal line based on the degree of correlation of the second image and a projective transformation image obtained by performing the projective transformation of the first image with the object.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a technique for calculating the normal line of an object.

Background Art

[0002] Conventionally, a technique for calculating the normal line of a planar object has been known. For example, Patent Document 1 discloses a normal line estimation device that calculates the normal line of a planar object using an image in which three or more planar objects are captured.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, the normal line of a planar object is calculated using an image in which three or more planar objects are captured. However, depending on the shooting situation, it may not be possible to obtain an image in which three or more planar objects are captured, and it is desirable to be able to calculate the normal line of a planar object with a smaller number of images.

[0005] The present invention has been made to solve the above problems, and a main object thereof is to provide an information processing apparatus capable of suitably calculating the normal line of an object having a plane.

Means for Solving the Problems

[0006] The invention according to the claims is image acquisition means for acquiring a first image including an object having a plane and a second image including the object, the relative shooting position and shooting posture of the second image with respect to the first image, the distance from the shooting position of the first image to the object, the provisional value of the normal line of the object, and projection conversion means for performing projection conversion on the first image based on ; normal line calculation means for calculating the normal line based on the degree of correlation of the object between the projection conversion image obtained by performing projection conversion on the first image and the second image; and has the projection conversion means sets an initial value of the provisional value based on a vector determined from the imaging position of the first image and the imaging position of the second image. It is an information processing apparatus. Further, the invention according to the claim is image acquisition means for acquiring a first image including an object having a plane and a second image including the object; the relative imaging position and imaging posture of the second image with respect to the first image; the distance from the imaging position of the first image to the object; the provisional value of the normal line of the object, and projection conversion means for performing projection conversion on the first image based on ; normal line calculation means for calculating the normal line based on the degree of correlation of the object between the projection conversion image obtained by performing projection conversion on the first image and the second image; and has the projection conversion means sets an initial value of the provisional value based on a vector determined from the imaging position of the first image or the second image and the position of the object. It is an information processing apparatus. Further, the invention according to the claim is image acquisition means for acquiring a first image including an object having a plane and a second image including the object; geometric information acquisition means for acquiring geometric information which is geometric information of the object; the relative imaging position and imaging posture of the second image with respect to the first image; the distance from the imaging position of the first image to the object; the provisional value of the normal line of the object, and projection conversion means for performing projection conversion on the first image based on ; Normal calculation means for calculating the normal based on the degree of correlation of the object between the projective transformation image obtained by projective-transforming the first image and the second image; having the normal calculation means determines an updated value of the provisional value based on the geometric information. It is an information processing device.

[0007] Further, the invention according to the claims is implemented by a computer to obtain a first image including an object having a plane and a second image including the object, the relative photographing position and photographing posture of the second image with respect to the first image, the distance from the photographing position of the first image to the object, a provisional value of the normal of the object, based on which, projective-transform the first image, calculate the normal based on the degree of correlation of the object between the projective transformation image obtained by projective-transforming the first image and the second image, set an initial value of the provisional value based on a vector determined from the photographing position of the first image or the second image and the position of the object. It is a control method.

[0008] Further, the invention according to the claims is image acquisition means for acquiring a first image including an object having a plane and a second image including the object, the relative photographing position and photographing posture of the second image with respect to the first image, the distance from the photographing position of the first image to the object, a provisional value of the normal of the object, based on which, projective transformation means for projective-transforming the first image, normal calculation means for calculating the normal based on the degree of correlation of the object between the projective transformation image obtained by projective-transforming the first image and the second image function the computer as The projective transformation means sets the initial value of the provisional value based on a vector determined from the imaging position of the first image or the second image and the position of the object. It is a program.

Brief Description of Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0010] According to a preferred embodiment of the present invention, an information processing apparatus includes an image acquisition unit that acquires a first image including an object having a plane and a second image including the object, a relative shooting position and shooting posture of the second image with respect to the first image, a distance from the shooting position of the first image to the object, and a provisional value of the normal line of the object, and a projective transformation unit that performs projective transformation on the first image based on these values, and a normal line calculation unit that calculates the normal line based on the degree of correlation of the object between the projective transformation image obtained by projective-transforming the first image and the second image. With this configuration, the information processing apparatus can suitably calculate the normal line of the object based on two images in which the object is captured.

[0011] In one aspect of the above information processing apparatus, when the degree of correlation is calculated a plurality of times by changing the provisional value of the normal line, the normal line calculation unit calculates the provisional value at which the degree of correlation becomes the highest as the normal line. The higher the estimation accuracy of the normal line of the object, the higher the accuracy of the projective transformation and the higher the degree of correlation. Therefore, the information processing apparatus can suitably search for the optimal normal line by calculating the degree of correlation a plurality of times by changing the provisional value of the normal line and determining the normal line using the calculated degree of correlation as an index.

[0012] In another aspect of the above information processing apparatus, the projective transformation unit sets the initial value of the provisional value based on a vector determined from the shooting position of the first image and the shooting position of the second image. Thereby, when the shooting position moves toward the front direction of the object, the information processing apparatus can suitably set the initial value of the provisional value of the normal line.

[0013] In another aspect of the above information processing apparatus, the projective transformation unit sets the initial value of the provisional value based on a vector determined from the shooting position of the first image or the second image and the position of the object. Thereby, when the object is photographed from the front direction, the information processing apparatus can suitably set the initial value of the provisional value of the normal line.

[0014] In another aspect of the information processing apparatus, there is geometric information acquisition means for acquiring geometric information which is geometric information of the object, and the normal line calculation means extracts the region of the object from the projective transformation image and the second image based on the geometric information, and calculates the degree of correlation based on the regions extracted from each of the projective transformation image and the second image. According to this aspect, the information processing apparatus can accurately extract the region of the object from the projective transformation image and the second image, and preferably suppress the occurrence of an error in the degree of correlation caused by using a pixel region that is not the object as a comparison target.

[0015] In another aspect of the information processing apparatus, there is geometric information acquisition means for acquiring geometric information which is geometric information of the object, and the normal line calculation means determines an updated value of the provisional value based on the geometric information. According to this aspect, the information processing apparatus can preferably set an updated value of the provisional value.

[0016] In another aspect of the information processing apparatus, the projective transformation means calculates the position of the object based on the first image and the second image, and calculates the distance based on the position of the object, the shooting position of the first image, and the provisional value. According to this aspect, the information processing apparatus can preferably calculate the distance required for projective transformation.

[0017] In another aspect of the information processing apparatus, the first image and the second image are generated by a camera that moves together with the vehicle, and the object is a sign. According to this aspect, the information processing apparatus can preferably calculate the normal line of the sign photographed from the vehicle.

[0018] According to another preferred embodiment of the present invention, the control method includes a computer acquiring a first image including an object having a plane and a second image including the object, and determining the relative shooting position and shooting posture of the second image with respect to the first image, the distance from the shooting position of the first image to the object, and a provisional value of the normal line of the object. Based on these, the first image is subjected to projective transformation, and the normal line is calculated based on the degree of correlation of the object between the projective transformation image obtained by projective-transforming the first image and the second image. By executing this control method, the computer can suitably calculate the normal line of the object based on two images in which the object is captured.

[0019] According to another preferred embodiment of the present invention, the program causes a computer to function as an image acquisition means for acquiring a first image including an object having a plane and a second image including the object, a projective transformation means for performing projective transformation on the first image based on the relative shooting position and shooting posture of the second image with respect to the first image, the distance from the shooting position of the first image to the object, and a provisional value of the normal line of the object, and a normal line calculation means for calculating the normal line based on the degree of correlation of the object between the projective transformation image obtained by projective-transforming the first image and the second image. By executing this program, the computer can suitably calculate the normal line of the object based on two images in which the object is captured. Preferably, the above program is stored in a storage medium.

Example

[0020] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0021] (1) System Overview FIG. 1 is a schematic configuration diagram of a normal line calculation system according to this embodiment. The normal line calculation system shown in FIG. 1 is a system for calculating the normal line of an object existing on or around a road, and includes an in-vehicle device 1 and a camera 2. These are mounted on the same vehicle. Hereinafter, the object for which the normal line is to be calculated will be referred to as an "object".

[0022] The in-vehicle device 1 calculates the normal line of the object based on the data generated by the camera 2. For example, the in-vehicle device 1 regards the sign as the object, and calculates the normal line of the sign based on the image of the camera 2 including the sign. Note that the object may be any ground feature having a plane such as a direction signboard, a kilometer post, a 100m post, a delineator, etc. in addition to the sign. The in-vehicle device 1 is an example of an "information processing device".

[0023] The camera 2 is installed in the vehicle and generates an image (also referred to as a "captured image") of the scenery seen from the vehicle. Note that the captured image includes date and time data indicating the date and time of shooting (generation time) of the captured image as metadata. The camera 2 supplies the captured image including the date and time data to the in-vehicle device 1.

[0024] Note that the configuration of the normal line calculation system shown in FIG. 1 is an example, and various modifications may be made to the configuration shown in FIG. 1. For example, the in-vehicle device 1 and the camera 2 may be integrally configured. In this case, the in-vehicle device 1 and the camera 2 may be configured as a single drive recorder. Also, the in-vehicle device 1 may be composed of a plurality of devices. In this case, the plurality of devices execute the pre-assigned processing and exchange the necessary data with each other between the devices.

[0025] (2) Device Configuration FIG. 2 is a block diagram showing the functional configuration of the in-vehicle device 1. The in-vehicle device 1 mainly includes an interface 11, a memory 12, and a controller 15. These elements are interconnected via a bus line.

[0026] The interface 11 performs an interface operation regarding the exchange of data between the in-vehicle device 1 and an external device. In the present embodiment, the interface 11 acquires the data output from the camera 2 etc. and supplies it to the memory 12.

[0027] The memory 12 is composed of various memories such as RAM (Random Access Memory), ROM (Read Only Memory), and non-volatile memory (including hard disk drives, flash memories, etc.). The memory 12 stores a program for the controller 15 to execute predetermined processes. Also, the memory 12 is used as a working memory for the controller 15. Note that the program executed by the controller 15 may be stored in a storage medium other than the memory 12.

[0028] Functionally, the memory 12 also has a captured image storage unit 16 and a normal line information storage unit 18. The captured image storage unit 16 stores the captured image generated by the camera 2. This captured image includes date and time data indicating the capture date and time as metadata. The normal line information storage unit 18 stores information regarding the normal line of the object (also referred to as "normal line information"). The normal line information is, for example, information in which the identification information of the object (including information indicating the type or / and position, etc.) is associated with the calculated normal vector (direction information).

[0029] Note that at least one of the captured image storage unit 16 and the normal line information storage unit 18 may be stored in an external storage device of the in-vehicle unit 1, such as a hard disk, connected to the in-vehicle unit 1 via the interface 11.

[0030] The controller 15 includes processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and controls the entire in-vehicle unit 1. In this case, the controller 15 performs processes such as normal line calculation processing of the object by executing a program stored in the memory 12 and the like. The controller 15 functions as "image acquisition means", "projection conversion means", "normal line calculation means", and a computer that executes a program.

[0031] Note that the processing executed by the controller 15 is not limited to being realized by software based on a program, and may be realized by any combination of hardware, firmware, and software, etc. Further, the processing executed by the controller 15 may be realized using a user-programmable integrated circuit such as, for example, an FPGA (Field-Programmable Gate Array) or a microcomputer. In this case, the program executed by the controller 15 in this embodiment may be realized using this integrated circuit. Thus, the controller 15 may be realized by hardware other than a processor.

[0032] (3) Functional Block The normal line calculation process executed by the controller 15 will be described. Generally speaking, the controller 15 sets a provisional value of the normal line of the object for one of the two captured images including the object, performs projective transformation, and optimizes the normal line so that the degree of correlation between the captured image after projective transformation and the other captured image becomes maximum.

[0033] FIG. 3 is a block diagram showing a functional configuration of the controller 15 regarding the normal line calculation process. As shown in FIG. 3, functionally, the controller 15 includes an image acquisition unit 21, a shooting parameter calculation unit 22, a projective transformation unit 23, an image comparison unit 24, and an optimization unit 25. Note that the captured images generated by the camera 2 are stored in the captured image storage unit 16.

[0034] The image acquisition unit 21 extracts two captured images including the object from the captured image storage unit 16. In this case, the image acquisition unit 21 detects the object, for example, by performing image recognition processing on each captured image stored in the captured image storage unit 16. In this case, the image acquisition unit 21 may detect the object based on pattern matching, or may detect the object using an inference device that outputs information on the types of feature objects existing in the input image and the pixel regions thereof. The above inference device is, for example, a learning model based on machine learning such as deep learning. The image acquisition unit 21 constitutes the above inference device by reading the parameters of the inference device obtained by prior machine learning from the memory 12 or the like, and inputs the captured image into the inference device. When the image acquisition unit 21 detects an object from the captured image, the image acquisition unit 21 extracts two captured images including the object from the captured image storage unit 16. These two captured images are not limited to the continuously captured images, and may be captured images with a predetermined frame number interval so as to generate a sufficient parallax. Hereinafter, the two captured images acquired by the image acquisition unit 21 are referred to as the "first image" and the "second image", respectively.

[0035] Note that the in-vehicle device 1 may obtain a plurality of sets of captured images including the object in the image acquisition unit 21, and obtain the normal line of the object stored in the normal line information storage unit 18 by statistically processing the calculation results of the normal lines of the objects for each set.

[0036] The imaging parameter calculation unit 22 calculates the imaging position and imaging posture of the camera 2 that captured the captured image as imaging parameters (so-called external parameters of the camera 2). In this case, the imaging parameter calculation unit 22 recognizes the imaging parameters of each captured image by a method used in various SLAM (Simultaneous Localization and Mapping). Further, the imaging parameter calculation unit 22 may calculate the imaging parameters from at least four or more captured images with different viewpoints using a method such as SfM (Structure from Motion).

[0037] The imaging parameter calculation unit 22 supplies information indicating the calculated imaging parameters of the first image and the second image to the projective transformation unit 23. Note that instead of calculating the imaging parameters only for the first image and the second image, the imaging parameter calculation unit 22 may calculate the imaging parameters for all the captured images generated by the camera 2. In this case, the imaging parameter calculation unit 22 may store the calculated imaging parameters in the captured image storage unit 16 in association with the target captured image.

[0038] The projective transformation unit 23 performs projective transformation so that one of the first image and the second image becomes an image captured at the shooting position and shooting posture of the other. In the present embodiment, the projective transformation unit 23 generates a first image (also referred to as a "projectively transformed image") transformed to the shooting position and shooting posture of the second image by performing projective transformation on the first image. In this case, the projective transformation unit 23 performs projective transformation on the first image by setting the normal vector of the object required for projective transformation to a predetermined provisional value. The projective transformation by the projective transformation unit 23 will be described later.

[0039] The image comparison unit 24 calculates the degree of correlation (degree of matching) of the object between the projectively transformed image obtained by projectively transforming the first image and the second image. The degree of correlation used in this case may be various indexes such as SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), NCC (Normalized Cross-Correlation), and ZNCC (Zero-mean Normalized Cross-Correlation). This degree of correlation may be calculated using color information. Further, when calculating the degree of correlation, the image comparison unit 24 extracts the region in which the object is represented (also referred to as the "object region") from among the first image and the second image by performing arbitrary image recognition processing, and calculates the degree of correlation with the extracted object region as the target. Note that when the image acquisition unit 21 detects the object region on the first image and the second image, the image comparison unit 24 may recognize the above object region based on the detection result by the image acquisition unit 21.

[0040] The optimization unit 25 optimizes the normal vector of the object used by the projective transformation unit 23 for projective transformation based on the correlation degree calculated by the image comparison unit 24. Specifically, the optimization unit 25 changes the provisional value of the normal vector of the object used by the projective transformation unit 23 for projective transformation so that the correlation degree calculated by the image comparison unit 24 becomes the maximum. Then, the optimization unit 25 changes the provisional value of the normal vector and causes the image comparison unit 24 to execute projective transformation, and regards the provisional value of the normal vector that results in the highest correlation degree when the correlation degree based on the execution result is calculated multiple times as the normal vector of the object. Here, for example, the optimization unit 25 continues the process of changing the provisional value of the normal vector and calculating the correlation degree until the correlation degree becomes equal to or higher than a predetermined threshold value, or until the amount of change in the correlation degree due to repeated calculation of the correlation degree becomes equal to or lower than a predetermined threshold value.

[0041] Note that various optimization methods can be used for such optimization of the normal vector. For example, the optimization unit 25 may use the LM method (Levenberg-Marquardt Method), which is a non-linear optimization method, the Newton method, etc., to set the provisional value of the normal vector to be searched next. The optimization unit 25 stores the normal vector information in which the normal vector is associated with the identification information of the object in the normal vector information storage unit 18.

[0042] The in-vehicle device 1 may transmit the normal vector information stored in the normal vector information storage unit 18 to a server device or the like that manages map data at predetermined intervals. In this case, the normal vector information generated by the in-vehicle device 1 is used for the update process of the map data.

[0043] (4) Details of Projection Transformation Next, the projective transformation process executed by the projective transformation unit 23 will be described. FIG. 4 shows an overhead view of the camera 2 and the object 50 at the time of acquisition of each of the first image and the second image on the road. The object 50 is a planar sign provided on the road. In FIG. 4, the position "P1" of the camera 2 at the time of acquisition of the first image, the position "P2" of the camera 2 at the time of acquisition of the second image, and the center position "P3" of the object 50 are clearly shown. Also, here, it is assumed as a premise that the second image is acquired after the first image is acquired.

[0044] In this case, based on the calculation result of the imaging parameter calculation unit 22, the projective transformation unit 23 recognizes a relative vector (translation vector) "t" indicating the relative imaging position of the second image with respect to the first image, and a rotation matrix "R" indicating the relative imaging posture of the second image with respect to the first image. Then, the projective transformation unit 23 calculates a projective transformation matrix "H" for converting the first image into the second image based on the following formula.

[0045]

Equation

[0046] First, a method for setting an initial value of the normal vector n, which is a provisional value of the normal vector n, will be described.

[0047] In the first example, the projective transformation unit 23 sets the initial value of the normal vector n to the reverse vector (-t) of the translation vector t. Generally, in the case of a sign provided on a road, the sign is often oriented in the opposite direction of the vehicle's traveling direction so that it is easily visible to the driver. Considering the above, the projective transformation unit 23 can set the initial value of the normal vector n to a highly accurate value by setting the reverse vector (-t) of the translation vector t as the initial value of the normal vector n.

[0048] In the second example, the projective transformation unit 23 sets the initial value of the normal vector n to the vector from the center position P3 of the object 50 to the position P2 of the camera 2 at the time of acquisition of the second image (i.e., P2 - P3). When it is assumed that the sign is perpendicular to the horizontal plane, the projective transformation unit 23 may calculate the above vector by setting the height (elevation) components of the positions P2 and P3 to 0 without considering them. As described above, in the case of a sign provided on a road, since the sign is generally directed toward the vehicle so that the driver can easily visually recognize it, also in the second example, the projective transformation unit 23 can set the initial value of the normal vector n to a highly accurate value. Note that, in the second example, the projective transformation unit 23 may set the initial value of the normal vector n to the vector from the center position P3 of the object 50 to the position P1 of the camera 2 at the time of acquisition of the first image (i.e., P1 - P3).

[0049] Next, a method for calculating the object distance h will be described. The projective transformation unit 23 calculates the center position P3 of the object 50, which is a three-dimensional position, based on the stereo measurement method using triangulation, using the first image and the second image. Then, preferably, the projective transformation unit 23 optimizes the calculated three-dimensional points based on an optimization method such as bundle adjustment. In this case, the projective transformation unit 23 may calculate the center position P3 of the object 50 using, in addition to or instead of the first image and the second image, other captured images in which the object 50 is displayed. Then, the projective transformation unit 23 calculates the object distance h based on the distance formula between a point and a plane, using the calculated center position P3 of the object 50, the shooting position P1 at the time of acquisition of the first image, and the provisional value of the normal vector n.

[0050] As described above, the projective transformation unit 23 calculates the translation vector t, the rotation matrix R, the provisional value (initial value) of the normal vector n, and the object distance h, respectively, and calculates the projective transformation matrix based on the above formula. Thereby, the projective transformation unit 23 can suitably perform the projective transformation onto the first image.

[0051] Next, a specific example of the projective transformation by the projective transformation unit 23 will be described.

[0052] FIG. 5(A) shows an example of the first image in the example of FIG. 4, and FIG. 5(B) shows an example of the second image in the example of FIG. 4. In the first image shown in FIG. 5(A) and the second image shown in FIG. 5(B), an object 50 which is a sign regarding speed limit is depicted. FIG. 6(A) shows a pixel region within the dashed frame F1 of the first image shown in FIG. 5(A). FIG. 6(B) shows a pixel region corresponding to the region within the dashed frame F1 when the first image is subjected to projective transformation. FIG. 6(C) shows a pixel region within the dashed frame F2 of the second image shown in FIG. 5(B).

[0053] The projective transformation unit 23 applies a projective transformation matrix calculated using a translation vector t indicating the relative shooting position of the second image with respect to the first image, a rotation matrix R indicating the relative shooting posture of the second image with respect to the first image, etc. to the first image. In this case, as shown in FIG. 6(B), in the projective transformation image obtained by projective-transforming the first image, the size and posture of the object change so as to look the same as the second image. Note that, in this case, since the normal vector n used in the projective transformation matrix is a provisional value, a deviation from the appearance of the second image occurs according to the accuracy (precision) of the provisional value of the normal vector n. Thereafter, the image comparison unit 24 extracts the pixel region indicated by the dashed frame F3 as the object region with respect to the projective transformation image, and calculates the correlation degree by comparing the pixel region within the extracted dashed frame F3 and the pixel region within the dashed frame F2 extracted as the object region from the second image. In this case, the higher the accuracy (i.e., closer to the true value) of the provisional value of the normal vector n, the higher the corresponding correlation degree. Therefore, the in-vehicle device 1 can obtain the accurate normal vector n by obtaining the provisional value of the normal vector n with a high correlation degree by optimization.

[0054] (5) Processing Flow FIG. 7 is an example of a flowchart showing the procedure of the normal vector calculation process executed by the in-vehicle device 1. The in-vehicle device 1 executes the process of the flowchart shown in FIG. 7, for example, when detecting an object for which a normal vector is to be calculated from a captured image generated by the camera 2.

[0055] First, the in-vehicle device 1 acquires a first image and a second image that include the object (step S11). In this case, the in-vehicle device 1 extracts any two captured images including the object to be calculated for the normal vector from the captured image storage unit 16 that stores the captured images generated by the camera 2 as the first image and the second image, respectively. Note that the in-vehicle device 1 may acquire a plurality of sets of captured images including the object, calculate the normal vector for each set of the acquired captured images by performing steps S12 to S18 described later, and calculate a representative value such as the average of the calculated normal vectors as the normal of the object to be obtained.

[0056] Next, the in-vehicle device 1 calculates a translation vector t and a rotation matrix R corresponding to the relative shooting positions and shooting postures of the first image and the second image (step S12). In this case, the in-vehicle device 1 calculates the translation vector t and the rotation matrix R by analyzing, for example, a plurality of images including the first image and the second image.

[0057] Next, the in-vehicle device 1 sets a provisional value of the normal vector n of the object (step S13). In this case, in the first-time processing, the in-vehicle device 1 sets a provisional value of the normal vector n based on any two relative positions among the shooting position of the first image, the shooting position of the second image, or the position of the object. Also, in the processing after the second time, the in-vehicle device 1 changes the provisional value of the normal vector n from the previous value based on the optimization method.

[0058] Next, the in-vehicle device 1 calculates the object distance h that is the distance between the shooting position of the first image and the object (step S14). In this case, for example, the in-vehicle device 1 calculates the object distance h based on the provisional value of the normal vector set in step S13 and the center position of the object calculated based on the first image and the second image. Note that in the processing after the second time, the in-vehicle device 1 calculates the object distance h using the normal vector updated based on the optimization method. This processing is executed by the projective transformation unit 23 in FIG. 3.

[0059] Then, the in-vehicle device 1 calculates a projective transformation matrix based on the translational vector t, rotation matrix R, provisional values of the normal vector n, and object distance h calculated in steps S12 to S14 (step S15). Then, the in-vehicle device 1 calculates the correlation degree of the object between the projective transformation image, which is the first image projective-transformed by the calculated projective transformation matrix, and the second image (step S16).

[0060] Then, the in-vehicle device 1 determines whether to terminate the optimization of the normal vector (step S17). For example, when the correlation degree calculated in step S16 is equal to or greater than a predetermined threshold value, when the processes of steps S13 to S16 are repeated a specified number of times, or when the change width of the correlation degree calculated in step S16 is less than a predetermined threshold value, etc., the in-vehicle device 1 determines that the optimization should be terminated. Then, when the in-vehicle device 1 terminates the optimization (step S17; Yes), it determines the normal vector of the object (step S18). In this case, the in-vehicle device 1 determines the provisional value of the normal vector with the highest correlation degree as the normal vector of the object. After that, the in-vehicle device 1 stores the information regarding the determined normal vector of the object in the normal vector information storage unit 18. On the other hand, when the in-vehicle device 1 determines that the optimization should not be terminated (step S17; No), it returns the process to step S13. In this case, in step S13, the in-vehicle device 1 changes the provisional value of the normal vector n from the previous value based on the optimization method to be applied, and executes the processes of steps S14 to S16.

[0061] As described above, the in-vehicle device 1 according to the embodiment includes an image acquisition unit, a projective transformation unit, and a normal vector calculation unit. The image acquisition unit acquires a first image including an object having a plane and a second image including the object. The projective transformation unit projective-transforms the first image based on the relative shooting position and shooting posture of the second image with respect to the first image, the distance from the shooting position of the first image to the object, and the provisional value of the normal vector of the object. The normal vector calculation unit calculates the normal vector based on the correlation degree of the object between the projective transformation image obtained by projective-transforming the first image and the second image. Thereby, the in-vehicle device 1 can accurately calculate the normal vector of the object based on two images including the object.

[0062] (6) Modification Example A modification example suitable for the above-described embodiment will be described. The following modification examples may be arbitrarily combined and applied to the above-described embodiment.

[0063] (Modification Example 1) The in-vehicle device 1 may further perform a normal vector calculation process with reference to geometric information regarding the shape of the object.

[0064] FIG. 8 shows a block configuration diagram of the in-vehicle device 1A according to Modification Example 1. The in-vehicle device 1A includes an interface 11, a memory 12, and a controller 15, similar to the block configuration of the in-vehicle device 1 shown in FIG. 2. The memory 12 has a photographed image storage unit 16, a normal vector information storage unit 18, and a geometric information storage unit 19. The controller 15 of the in-vehicle device 1A functionally has a geometric information acquisition unit 26 in addition to an image acquisition unit 21, a photographing parameter calculation unit 22, a projective conversion unit 23, an image comparison unit 24, and an optimization unit 25.

[0065] The geometric information storage unit 19 stores geometric information of a ground object that is an object for which normal vector detection is performed. This geometric information is geometric information (i.e., information indicating the shape) of the target ground object, and exists for each type (category) of ground object having a different shape, for example. For example, when the object includes a sign, for each type of sign, information indicating the shape of the sign corresponding to that type is stored in the geometric information storage unit 19 as geometric information. For example, if the speed limit sign is circular, the crosswalk sign is triangular, and the intersection sign is diamond-shaped, this information is stored in the geometric information storage unit 19 as geometric information. Note that the geometric information may include size information.

[0066] The geometric information acquisition unit 26 extracts the geometric information corresponding to the object detected by the image acquisition unit 21 from the geometric information storage unit 19. For example, when the image acquisition unit 21 inputs the captured image to the inference device obtained by learning or the like, the image acquisition unit 21 supplies the information on the type of the object output at this time to the geometric information acquisition unit 26, and the geometric information acquisition unit 26 extracts the geometric information associated with the type of the object supplied from the image acquisition unit 21 from the geometric information storage unit 19. Then, the geometric information acquisition unit 26 supplies the extracted geometric information to at least one of the image comparison unit 24 or the optimization unit 25. In this way, the controller 15 functions as a "geometric information acquisition means".

[0067] The utilization of the geometric information by the image comparison unit 24 will be described. The image comparison unit 24 calculates the degree of correlation based on the geometric information supplied from the geometric information acquisition unit 26. For example, the image comparison unit 24 uses the geometric information in the process of extracting the object regions from the projective transformation image obtained by projective-transforming the first image and the second image, respectively. For example, in the case of the object 50 shown in FIG. 5, the image comparison unit 24 recognizes that the object 50 is circular based on the geometric information, and sets the range to be extracted as the object region (see the dashed frames F1 to F3 in FIGS. 5 and 6) to a circular shape. In this case, the image comparison unit 24 extracts the circular pixel region as the object region. Thereby, the image comparison unit 24 can accurately extract the object region and preferably suppress the occurrence of an error in the degree of correlation caused by using the pixel region that is not the object as the comparison target.

[0068] Next, the utilization of geometric information by the optimization unit 25 will be described. The optimization unit 25 may determine an updated value (a value to be changed) of a provisional value of the normal vector used for the projective transformation by the projective transformation unit 23 based on the geometric information supplied from the geometric information acquisition unit 26. For example, the optimization unit 25 limits the direction in which the provisional value changes (i.e., the solution search direction) in the direction approaching the shape of the object indicated by the geometric information. In this case, the optimization unit 25 sets, for example, a predetermined number of provisional values changed in directions different from the initial value of the normal vector. Then, the optimization unit 25 recognizes as the direction in which the provisional value changes (i.e., the direction in which the solution should be searched) the provisional value for which the object in the projective transformation image based on the set provisional value is closest to the shape indicated by the geometric information. In this way, by using the geometric information, the optimization unit 25 can suitably limit the search range of the provisional value of the normal vector for which optimization is performed, and can reduce the processing load.

[0069] (Modification Example 2) The normal vector calculation process executed by the in-vehicle device 1 may be executed by a device different from the in-vehicle device 1.

[0070] FIG. 9 is a schematic configuration diagram of a normal vector calculation system according to Modification Example 2. As shown in FIG. 9, the normal vector calculation system includes an in-vehicle device 1X, a camera 2, and a map generation device 4. The in-vehicle device 1X has the same configuration as the in-vehicle device 1 shown in FIG. 2 and is capable of data communication with the map generation device 4. Note that the in-vehicle device 1X does not have the normal vector information storage unit 18. Then, the in-vehicle device 1X transmits the captured image supplied from the camera 2 to the map generation device 4 as measurement data "Im". In this case, the in-vehicle device 1X may temporarily store the captured image, or may transmit the captured image supplied from the camera 2 to the map generation device 4 without storing it.

[0071] The map generation device 4 receives the measurement data Im including the captured image from the in-vehicle device 1 and stores the received measurement data Im. Then, the map generation device 4 executes the normal vector calculation process by the in-vehicle device 1 described in the embodiment based on the stored measurement data Im. In this case, the map generation device 4 may calculate the normal vector of the object by processing the measurement data Im received from the in-vehicle device 1X in real time, or may calculate the normal vector at a predetermined timing based on user input or the like. The map generation device 4 is an example of an "information processing device".

[0072] FIG. 10 is a block diagram showing the functional configuration of the map generation device 4. The map generation device 4 includes an interface 41, a memory 42, and a controller 45. These elements are interconnected via a bus line.

[0073] The interface 41 performs an interface operation regarding data transfer between the map generation device 4 and an external device. The memory 42 is composed of various memories such as a RAM, a ROM, and other non-volatile memories (including a hard disk drive, a flash memory, etc.). The memory 42 stores a program for the controller 45 to execute a predetermined process. Also, the memory 42 is used as a working memory for the controller 45. Note that the program executed by the controller 45 may be stored in a storage medium other than the memory 42.

[0074] Also, the memory 42 functionally has a measurement data storage unit 46 and a normal vector information storage unit 47. The measurement data storage unit 46 stores the measurement data Im received from the in-vehicle device 1. The normal vector information storage unit 47 stores the normal vector information generated by the map generation device 4. Note that at least one of the measurement data storage unit 46 and the normal vector information storage unit 47 may be stored in an external storage device of the map generation device 4 such as a hard disk connected to the map generation device 4 via the interface 41. The above storage device may be a server device that communicates with the map generation device 4. Also, the above storage device may be composed of a plurality of devices.

[0075] The controller 45 includes processors such as a CPU and a GPU, etc., and controls the entire in-vehicle device 1. In this case, the controller 45 performs processing related to detecting change points of feature objects by executing a program stored in the memory 42 or the like. The controller 45 functions as an "image acquisition means", "projection conversion means", "normal calculation means", and a computer that executes a program, etc.

[0076] In this way, the normal calculation process may be performed by any device other than the in-vehicle device 1X. Instead of the in-vehicle device 1 and the map generation device 4 exchanging measurement data Im through data communication, the map generation device 4 may obtain the measurement data Im stored in the storage medium of the in-vehicle device 1 by reading it. In this case, the above storage medium is electrically connected to the in-vehicle device 1 during measurement, and the in-vehicle device 1 writes the measurement data Im.

[0077] (Modification Example 3) The object is not limited to ground features such as signs on or around the road, and may be any object having a planar shape.

[0078] In this case, the camera 2 may be provided on a moving body other than the vehicle. For example, the camera 2 may be provided on a self-propelled robot. Even in this case, the self-propelled robot or the information processing device that has obtained the detection data of the camera 2 from the self-propelled robot calculates the normal of the object by executing the normal calculation process described in the embodiment.

[0079] (Modification Example 4) The in-vehicle device 1 may recognize the shooting parameters (shooting position and shooting posture) using the outputs of various sensors other than the camera.

[0080] In this case, the in-vehicle device 1 is electrically connected to the sensor unit, for example, by wire or wirelessly. The sensor unit is a group of sensors that detect information regarding the position and attitude (traveling direction) of the vehicle. The sensor unit includes, for example, a plurality of sensors such as a GPS receiver, an acceleration sensor, a gyro sensor, and an IMU (Inertial Measurement Unit). The above GPS receiver may generate high-precision position information indicating the absolute position of the vehicle (for example, the three-dimensional position of latitude, longitude, and altitude) based on the RTK positioning method (i.e., the interference positioning method). Note that the sensor unit may be a sensor provided in the camera 2 so as to directly detect the position and shooting attitude of the camera 2. The sensor unit supplies information (also referred to as "sensor information") in which the detection result by the sensor is associated with the date and time data indicating the detection date and time to the in-vehicle device 1. Then, the in-vehicle device 1 calculates the shooting position and shooting attitude with respect to the first image and the second image by referring to the sensor information including the date and time data that matches the date and time indicated by the date and time data added to the first image and the second image. Here, when the referred sensor information indicates the position and traveling direction of the vehicle, the in-vehicle device 1 further refers to the camera installation information indicating the relative position and orientation of the camera 2 with respect to the vehicle, thereby specifying the position and shooting attitude of the camera 2 that captured the target shooting image. The above camera installation information is stored in advance in the memory 12, for example. Also with this aspect, the in-vehicle device 1 can suitably calculate the shooting parameters.

[0081] In the above-described embodiments, the program can be stored using various types of non-transitory computer readable media and supplied to a controller or the like that is a computer. The non-transitory computer readable media include various types of tangible storage media. Examples of the non-transitory computer readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memories (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)).

[0082] The present invention has been described with reference to the embodiments, but the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. That is, the present invention naturally includes various modifications and corrections that those skilled in the art could make in accordance with the entire disclosure including the claims and the technical idea. Also, each disclosure of the above-cited patent documents and the like is incorporated herein by reference.

Explanation of Reference Numerals

[0083] 1 Vehicle-mounted device 2 Camera 4 Map generation device 16 Captured image storage unit 18, 47 Normal line information storage unit 46 Measurement data storage unit

Claims

1. an image capture means for capturing a first image including an object having a plane and a second image including the object; A relative photographing position and photographing attitude of the second image with respect to the first image; A distance from a photographing position of the first image to the object; A provisional value of a normal of the object; A projection transformation means for projectively transforming the first image based on the projection transformation means; a normal calculation means for calculating the normal based on a correlation degree of the object between a projectively transformed image obtained by projectively transforming the first image and the second image; having the projective transformation means sets an initial value of the provisional value based on a vector determined from a photographing position of the first image and a photographing position of the second image; Information processing device.

2. an image capture means for capturing a first image including an object having a plane and a second image including the object; A relative photographing position and photographing attitude of the second image with respect to the first image; A distance from a photographing position of the first image to the object; A provisional value of a normal of the object; A projection transformation means for projectively transforming the first image based on the projection transformation means; a normal calculation means for calculating the normal based on a correlation degree of the object between a projectively transformed image obtained by projectively transforming the first image and the second image; having The projective transformation means sets an initial value of the provisional value based on a vector determined from a photographing position of the first image or the second image and a position of the object. Information processing device.

3. an image capture means for capturing a first image including an object having a plane and a second image including the object; A geometric information acquisition means for acquiring geometric information that is geometric information of the object; A relative photographing position and photographing attitude of the second image with respect to the first image; A distance from a photographing position of the first image to the object; A provisional value of a normal of the object; A projection transformation means for projectively transforming the first image based on the projection transformation means; a normal calculation means for calculating the normal based on a correlation degree of the object between a projectively transformed image obtained by projectively transforming the first image and the second image; having The normal calculation means determines an update value of the provisional value based on the geometric information. Information processing device.

4. The information processing apparatus according to claim 1 , wherein the normal calculation means calculates, as the normal, the provisional value that gives the highest correlation degree when the provisional value of the normal is changed and the correlation degree is calculated a plurality of times.

5. A geometric information acquisition means for acquiring geometric information that is geometric information of the object, The information processing device according to any one of claims 1 to 4, wherein the normal calculation means extracts a region of the object from the projectively transformed image and the second image based on the geometric information, and calculates the degree of correlation based on the regions extracted from each of the projectively transformed image and the second image.

6. The information processing device according to any one of claims 1 to 5, wherein the projective transformation means calculates the position of the object based on the first image and the second image, and calculates the distance based on the position of the object, the shooting position of the first image, and the provisional value.

7. the first image and the second image are generated by a camera that moves with the vehicle; The information processing device according to claim 1 , wherein the object is a sign.

8. By computer, acquiring a first image including an object having a plane and a second image including the object; A relative photographing position and photographing attitude of the second image with respect to the first image; A distance from a photographing position of the first image to the object; A provisional value of a normal of the object; projecting the first image based on the Calculating the normal based on a correlation degree of the object between a projectively transformed image obtained by projectively transforming the first image and the second image; An initial value of the provisional value is set based on a vector determined from a photographing position of the first image or the second image and a position of the object. Control methods.

9. an image capture means for capturing a first image including an object having a plane and a second image including the object; A relative photographing position and photographing attitude of the second image with respect to the first image; A distance from a photographing position of the first image to the object; A provisional value of a normal of the object; A projection transformation means for projectively transforming the first image based on the projection transformation means; a normal calculation means for calculating the normal based on a correlation degree of the object between a projectively transformed image obtained by projectively transforming the first image and the second image; The computer functions as The projective transformation means sets an initial value of the provisional value based on a vector determined from a photographing position of the first image or the second image and a position of the object. program.

10. A storage medium storing the program according to claim 9.

Images