Information processing apparatus, control method, program, and storage medium
The information processing device calculates the normal line of a planar object using two images with projective transformation and iterative optimization, addressing the need for fewer images and improving accuracy.
Patent Information
- Application Number
- JP2024151429
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-03-31
AI Technical Summary
Existing methods for calculating the normal line of a planar object require multiple images, which can be impractical in certain shooting situations.
An information processing device that calculates the normal line using a first and second image, employing projective transformation based on relative photographing position, posture, and distance, with iterative optimization of the normal vector to maximize correlation.
Accurately determines the normal line of an object using only two images, enhancing precision and reducing errors through iterative optimization and geometric information utilization.
Smart Images

Figure 2024160014000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for calculating normals of an object. [Background technology]
[0002] Conventionally, there have been known techniques for calculating normals of planar objects. For example, Patent Document 1 discloses a normal estimation device that calculates normals of a planar object using images in which three or more planar objects are captured. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2018-97795 A Summary of the Invention [Problem to be solved by the invention]
[0004] In Patent Document 1, the normals of a planar object are calculated using images that include three or more planar objects. However, depending on the shooting conditions, it may not be possible to obtain an image that includes three or more planar objects, and it is desirable to be able to calculate the normals of a planar object using a smaller number of images.
[0005] The present invention has been made in order to solve the above-mentioned problems, and has as its main object to provide an information processing device capable of suitably calculating the normal of an object having a plane. [Means for solving the problem]
[0006] The claimed invention is An image capture means for capturing a first image including an object having a plane and a second image including the object; A relative photographing position and photographing attitude of the second image with respect to the first image; The distance from the capture position of the first image to the object; and A provisional value of a normal of the object; A projection transformation means for projectively transforming the first image based on the projection transformation means; a normal calculation means for calculating the normal based on a correlation degree of the object between a projectively transformed image obtained by projectively transforming the first image and the second image; having The projective transformation means sets an initial value of the provisional value based on a vector determined from a shooting position of the first image and a shooting position of the second image. It is an information processing device. The claimed invention also includes: An image capture means for capturing a first image including an object having a plane and a second image including the object; A relative photographing position and photographing attitude of the second image with respect to the first image; The distance from the capture position of the first image to the object; and A provisional value of a normal of the object; A projection transformation means for projectively transforming the first image based on the projection transformation means; a normal calculation means for calculating the normal based on a correlation degree of the object between a projectively transformed image obtained by projectively transforming the first image and the second image; having The projective transformation means sets an initial value of the provisional value based on a vector determined from a photographing position of the first image or the second image and a position of the object. It is an information processing device. The invention described in the claims also includes an image acquisition means for acquiring a first image including an object having a plane and a second image including the object; A geometric information acquisition means for acquiring geometric information that is geometric information of the object; A relative photographing position and photographing attitude of the second image with respect to the first image; The distance from the capture position of the first image to the object; and A provisional value of a normal of the object; A projection transformation means for projectively transforming the first image based on the projection transformation means; a normal calculation means for calculating the normal based on a correlation degree of the object between a projectively transformed image obtained by projectively transforming the first image and the second image; having The normal calculation means determines an update value of the provisional value based on the geometric information. It is an information processing device.
[0007] The claimed invention can also be implemented by a computer as follows: acquiring a first image including an object having a plane and a second image including the object; A relative photographing position and photographing attitude of the second image with respect to the first image; The distance from the capture position of the first image to the object; and A provisional value of a normal of the object; projecting the first image based on the Calculating the normal based on a correlation degree of the object between a projectively transformed image obtained by projectively transforming the first image and the second image; An initial value of the provisional value is set based on a vector determined from a photographing position of the first image or the second image and a position of the object. It is a control method.
[0008] The claimed invention also includes: An image capture means for capturing a first image including an object having a plane and a second image including the object; A relative photographing position and photographing attitude of the second image with respect to the first image; The distance from the capture position of the first image to the object; and A provisional value of a normal of the object; A projection transformation means for projectively transforming the first image based on the projection transformation means; a normal calculation means for calculating the normal based on a correlation degree of the object between a projectively transformed image obtained by projectively transforming the first image and the second image; The computer functions as The projective transformation means sets an initial value of the provisional value based on a vector determined from a photographing position of the first image or the second image and a position of the object. It is a program. [Brief description of the drawings]
[0009] [Figure 1] 1 is a schematic configuration of a normal calculation system. [Diagram 2] The block diagram of the on-board device is shown. [Diagram 3] A functional block diagram of the on-board device is shown. [Figure 4] 3 shows an overhead view of a camera and an object when a first image and a second image are captured on a road. [Diagram 5] (A) shows an example of a first image, (B) shows an example of a second image. [Figure 6] (A) shows the pixel area within the dashed frame of the first image shown in Figure 5(A). (B) shows the pixel area within the dashed frame when the first image is projectively transformed. (C) shows the pixel area within the dashed region of the second image shown in Figure 5(B). [Figure 7] 13 is an example of a flowchart illustrating a procedure for a normal line calculation process. [Figure 8] 2 shows a block configuration diagram of an on-vehicle device according to a first modified example. [Figure 9] FIG. 11 is a schematic configuration diagram of a normal calculation system according to Modification 2. [Figure 10] 13 shows a block configuration of a map generating device according to a second modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] According to a preferred embodiment of the present invention, an information processing device includes an image acquisition means for acquiring a first image including an object having a plane and a second image including the object, a projection transformation means for projectively transforming the first image based on a relative shooting position and shooting attitude of the second image with respect to the first image, a distance from the shooting position of the first image to the object, and a provisional value of a normal of the object, and a normal calculation means for calculating the normal based on a correlation degree of the object between the projection transformation image obtained by projectively transforming the first image and the second image. With this aspect, the information processing device can preferably calculate the normal of the object based on two images in which the object is captured.
[0011] In one aspect of the information processing device, the normal calculation means calculates the provisional value that gives the highest correlation degree as the normal when the provisional value of the normal is changed and the correlation degree is calculated multiple times. The higher the estimation accuracy of the normal of the object, the higher the accuracy of the projective transformation and the higher the correlation degree. Therefore, the information processing device can suitably search for an optimal normal by changing the provisional value of the normal and calculating the correlation degree multiple times, and determining the normal using the calculated correlation degree as an index.
[0012] In another aspect of the information processing device, the projective transformation means sets an initial value of the provisional value based on a vector determined from the shooting position of the first image and the shooting position of the second image, whereby the information processing device can suitably set an initial value of the provisional value of the normal when the shooting position moves toward the front of the object.
[0013] In another aspect of the information processing device, the projective transformation means sets the initial value of the provisional value based on a vector determined from the photographing position of the first image or the second image and the position of the object, whereby the information processing device can suitably set the initial value of the provisional value of the normal when the object is photographed from the front direction.
[0014] In another aspect of the information processing device, the information processing device includes a geometric information acquisition means for acquiring geometric information that is geometric information of the object, and the normal calculation means extracts an area of the object from the projectively transformed image and the second image based on the geometric information, and calculates the correlation degree based on the areas extracted from the projectively transformed image and the second image. With this aspect, the information processing device can accurately extract the object area from the projectively transformed image and the second image, and preferably suppress the occurrence of an error in the correlation degree caused by comparing a pixel area that is not the object.
[0015] In another aspect of the information processing device, the information processing device includes a geometric information acquisition means for acquiring geometric information that is geometric information of the object, and the normal calculation means determines an update value of the provisional value based on the geometric information. With this aspect, the information processing device can suitably set the update value of the provisional value.
[0016] In another aspect of the information processing device, the projective transformation means calculates the position of the object based on the first image and the second image, and calculates the distance based on the position of the object, the shooting position of the first image, and the provisional value. With this aspect, the information processing device can suitably calculate the distance required for projective transformation.
[0017] In another aspect of the information processing device, the first image and the second image are generated by a camera moving with a vehicle, and the object is a sign. With this aspect, the information processing device can suitably calculate the normal of the sign photographed from the vehicle.
[0018] According to another preferred embodiment of the present invention, the control method includes a computer acquiring a first image including an object having a plane and a second image including the object, projecting the first image based on a relative photographing position and photographing attitude of the second image with respect to the first image, a distance from the photographing position of the first image to the object, and a provisional value of a normal of the object, and calculating the normal based on a correlation degree of the object between the projectively transformed image obtained by projecting the first image and the second image. By executing this control method, the computer can suitably calculate the normal of the object based on two images in which the object is captured.
[0019] According to another preferred embodiment of the present invention, the program causes a computer to function as an image acquisition means for acquiring a first image including an object having a plane and a second image including the object, a projection transformation means for projectively transforming the first image based on the relative photographing position and photographing attitude of the second image with respect to the first image, the distance from the photographing position of the first image to the object, and a provisional value of the normal of the object, and a normal calculation means for calculating the normal based on the correlation degree of the object between the projection transformation image obtained by projecting the first image and the second image. By executing this program, the computer can preferably calculate the normal of the object based on two images in which the object is captured. Preferably, the program is stored in a storage medium. EXAMPLES
[0020] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.
[0021] (1) System Overview Fig. 1 is a schematic diagram of a normal calculation system according to the present embodiment. The normal calculation system shown in Fig. 1 is a system that calculates the normal of an object existing on or around a road, and includes an on-vehicle device 1 and a camera 2. These are mounted on the same vehicle. Hereinafter, an object for which a normal is to be calculated is referred to as an "object".
[0022] The vehicle-mounted device 1 calculates the normal of the object based on the data generated by the camera 2. For example, the vehicle-mounted device 1 regards a sign as the object, and calculates the normal of the sign based on an image of the camera 2 that includes the sign. Note that the object may be any feature having a flat surface, such as a direction sign, a kilometer post, a 100m post, a delineator, etc., in addition to a sign. The vehicle-mounted device 1 is an example of an "information processing device."
[0023] The camera 2 is installed in the vehicle and generates images (also called "captured images") of the scenery captured from the vehicle. The captured images contain date and time data indicating the capture date and time (generation date and time) of the captured images as metadata. The camera 2 supplies the captured images containing the date and time data to the vehicle-mounted device 1.
[0024] The configuration of the normal calculation system shown in Fig. 1 is an example, and various modifications may be made to the configuration shown in Fig. 1. For example, the vehicle-mounted device 1 and the camera 2 may be integrated. In this case, the vehicle-mounted device 1 and the camera 2 may be configured as one drive recorder. The vehicle-mounted device 1 may also be composed of multiple devices. In this case, the multiple devices execute pre-assigned processes and exchange necessary data between each other.
[0025] (2) Device configuration 2 is a block diagram showing the functional configuration of the vehicle-mounted device 1. The vehicle-mounted device 1 mainly includes an interface 11, a memory 12, and a controller 15. These elements are connected to each other via a bus line.
[0026] The interface 11 performs an interface operation related to the transmission and reception of data between the vehicle-mounted device 1 and an external device. In this embodiment, the interface 11 acquires data output from the camera 2 and the like, and supplies the data to the memory 12.
[0027] The memory 12 is composed of various types of memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), and a non-volatile memory (including a hard disk drive, a flash memory, etc.). The memory 12 stores programs for the controller 15 to execute predetermined processes. The memory 12 is also used as a working memory for the controller 15. The programs executed by the controller 15 may be stored in a storage medium other than the memory 12.
[0028] Moreover, functionally, the memory 12 has a captured image storage unit 16 and a normal information storage unit 18. The captured image storage unit 16 stores the captured image generated by the camera 2. This captured image includes date and time data indicating the date and time of capture as metadata. The normal information storage unit 18 stores information on the normal of an object (also referred to as "normal information"). The normal information is, for example, information in which identification information of the object (including information indicating the type and / or position, etc.) is associated with a calculated normal vector (orientation information).
[0029] At least one of the captured image storage unit 16 and the normal line information storage unit 18 may be stored in a storage device external to the vehicle-mounted device 1, such as a hard disk connected to the vehicle-mounted device 1 via the interface 11.
[0030] The controller 15 includes a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and controls the entire vehicle-mounted device 1. In this case, the controller 15 performs processing such as calculation of normals of an object by executing a program stored in the memory 12, etc. The controller 15 functions as an "image acquisition means", a "projection transformation means", a "normal calculation means", a computer that executes the program, etc.
[0031] The processing executed by the controller 15 is not limited to being realized by software using a program, but may be realized by any combination of hardware, firmware, and software. The processing executed by the controller 15 may be realized by using an integrated circuit programmable by a user, such as an FPGA (Field-Programmable Gate Array) or a microcomputer. In this case, the program executed by the controller 15 in this embodiment may be realized by using this integrated circuit. In this way, the controller 15 may be realized by hardware other than a processor.
[0032] (3) Functional Blocks The following describes the normal calculation process executed by the controller 15. In summary, the controller 15 performs projective transformation on one of two captured images including an object by setting a provisional value of the normal of the object, and optimizes the normal so that the correlation between the captured image after projective transformation and the other captured image is maximized.
[0033] Fig. 3 is a block diagram showing the functional configuration of the controller 15 related to normal calculation processing. As shown in Fig. 3, the controller 15 functionally has an image acquisition unit 21, an imaging parameter calculation unit 22, a projective transformation unit 23, an image comparison unit 24, and an optimization unit 25. The captured image storage unit 16 accumulates captured images generated by the camera 2.
[0034] The image acquisition unit 21 extracts two captured images including an object from the captured image storage unit 16. In this case, the image acquisition unit 21 detects the object by, for example, performing an image recognition process on each captured image stored in the captured image storage unit 16. In this case, the image acquisition unit 21 may detect the object based on pattern matching, or may detect the object using an inference device that outputs information on the type of feature present in the input image and its pixel area from the input image. The inference device is, for example, a learning model based on machine learning such as deep learning, and the image acquisition unit 21 configures the inference device by reading out parameters of the inference device obtained by prior machine learning from the memory 12 or the like, and inputs the captured image to the inference device. Then, when the image acquisition unit 21 detects an object from the captured image, it extracts two captured images including the object from the captured image storage unit 16. The two captured images are not limited to captured images captured consecutively, but may be captured images spaced apart by a predetermined number of frames so that sufficient parallax is generated. Hereinafter, the two captured images acquired by the image acquisition unit 21 will be referred to as a "first image" and a "second image", respectively.
[0035] In addition, the vehicle-mounted device 1 may obtain the normals of the object to be stored in the normal information storage unit 18 by acquiring multiple sets of captured images including the object using the image acquisition unit 21 and performing statistical processing on the calculation results of the normals of the object for each set.
[0036] The shooting parameter calculation unit 22 calculates the shooting position and shooting attitude of the camera 2 that captured the captured image as shooting parameters (so-called external parameters of the camera 2). In this case, the shooting parameter calculation unit 22 recognizes the shooting parameters of each captured image by a method used in various SLAM (Simultaneous Localization and Mapping). The shooting parameter calculation unit 22 may also calculate the shooting parameters from at least four or more captured images with different viewpoints using a method such as SfM (Structure from Motion).
[0037] The shooting parameter calculation unit 22 supplies information indicating the calculated shooting parameters of the first image and the second image to the projective transformation unit 23. Note that the shooting parameter calculation unit 22 may calculate shooting parameters for all captured images generated by the camera 2, instead of calculating shooting parameters only for the first image and the second image. In this case, the shooting parameter calculation unit 22 may store the calculated shooting parameters in the captured image storage unit 16 in association with the captured image of the target.
[0038] The projection transformation unit 23 performs projection transformation on one of the first image and the second image so that the image is captured at the shooting position and shooting attitude of the other image. In this embodiment, the projection transformation unit 23 performs projection transformation on the first image to generate a first image (also called a "projection transformed image") that is transformed to the shooting position and shooting attitude of the second image. In this case, the projection transformation unit 23 performs projection transformation on the first image by setting the normal vector of the object required for projection transformation to a predetermined provisional value. The projection transformation by the projection transformation unit 23 will be described later.
[0039] The image comparison unit 24 calculates the degree of correlation (degree of matching) between the projectively transformed image obtained by projectively transforming the first image and the object in the second image. The degree of correlation used in this case may be various indices such as SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), NCC (Normalized Cross-Correlation), and ZNCC (Zero-mean Normalized Cross-Correlation). This degree of correlation may be calculated using color information. When calculating the degree of correlation, the image comparison unit 24 extracts an area in which the object is represented (also called an "object area") in the first image and the second image by performing an arbitrary image recognition process, and calculates the degree of correlation for the extracted object area. When the image acquisition unit 21 detects the object area in the first image and the second image, the image comparison unit 24 may recognize the object area based on the detection result by the image acquisition unit 21.
[0040] The optimization unit 25 optimizes the normal vector of the object used by the projection transformation unit 23 for projection transformation based on the degree of correlation calculated by the image comparison unit 24. Specifically, the optimization unit 25 changes the provisional value of the normal vector of the object used by the projection transformation unit 23 for projection transformation so that the degree of correlation calculated by the image comparison unit 24 is maximized. The optimization unit 25 then changes the provisional value of the normal vector and has the image comparison unit 24 execute projection transformation, and when the degree of correlation based on the execution result is calculated multiple times, the provisional value of the normal vector that gives the highest degree of correlation is regarded as the normal vector of the object. Here, the optimization unit 25 continues the process of calculating the degree of correlation by changing the provisional value of the normal vector until the degree of correlation becomes equal to or greater than a predetermined threshold value, or until the amount of change in the degree of correlation due to repeated calculations of the degree of correlation becomes equal to or less than a predetermined threshold value, for example.
[0041] For optimizing the normal vector, various optimization methods can be used. For example, the optimization unit 25 may use a nonlinear optimization method such as the Levenberg-Marquardt method (LM method) or the Newton method to set a provisional value of the normal vector to be searched next. The optimization unit 25 stores normal information in which the normal vector is associated with the identification information of the object in the normal information storage unit 18.
[0042] The vehicle-mounted device 1 may transmit the normal line information stored in the normal line information storage unit 18 at a predetermined interval to a server device or the like that manages map data. In this case, the normal line information generated by the vehicle-mounted device 1 is used for updating the map data.
[0043] (4) Projection transformation details Next, the projective transformation process executed by the projective transformation unit 23 will be described. Fig. 4 shows an overhead view of the camera 2 and the object 50 when the first image and the second image on the road are acquired. The object 50 is a planar sign provided on the road. Fig. 4 clearly shows the position "P1" of the camera 2 when the first image is acquired, the position "P2" of the camera 2 when the second image is acquired, and the center position "P3" of the object 50. In addition, it is assumed here that the second image is acquired after the first image is acquired.
[0044] In this case, the projective transformation unit 23 recognizes a relative vector (translation vector) "t" indicating the relative shooting position of the second image with respect to the first image and a rotation matrix "R" indicating the relative shooting attitude of the second image with respect to the first image based on the calculation result of the shooting parameter calculation unit 22. Then, the projective transformation unit 23 calculates a projective transformation matrix "H" for transforming the first image into the second image based on the following formula.
[0045]
number
[0046] First, a method for setting the initial value of the normal vector n, which is a provisional value of the normal vector n, will be described.
[0047] In a first example, the projective transformation unit 23 sets the initial value of the normal vector n to the reverse vector (-t) of the translation vector t. Generally, in the case of signs provided on roads, the signs are often oriented in the opposite direction to the vehicle's traveling direction so that the driver can easily see them. In consideration of the above, the projective transformation unit 23 sets the reverse vector (-t) of the translation vector t as the initial value of the normal vector n, thereby making it possible to set the initial value of the normal vector n to a highly accurate value.
[0048] In the second example, the projection transformation unit 23 sets the initial value of the normal vector n to a vector (i.e., P2-P3) from the center position P3 of the object 50 to the position P2 of the camera 2 when the second image is acquired. If it is assumed that the sign is perpendicular to the horizontal plane, the projection transformation unit 23 may calculate the above vector by setting the height (altitude) components of the positions P2 and P3 to 0 without considering these values. As described above, in the case of signs provided on roads, the signs are generally oriented toward the vehicle so that the driver can easily see them. Therefore, even in the second example, the projection transformation unit 23 can set the initial value of the normal vector n to a highly accurate value. In the second example, the projection transformation unit 23 may set the initial value of the normal vector n to a vector (i.e., P1-P3) from the center position P3 of the object 50 to the position P1 of the camera 2 when the first image is acquired.
[0049] Next, a calculation method of the object distance h will be described. Using the first image and the second image, the projective transformation unit 23 calculates the center position P3 of the object 50, which is a three-dimensional position, based on a stereo measurement method using triangulation. After that, preferably, the projective transformation unit 23 optimizes the calculated three-dimensional points based on an optimization method such as bundle adjustment. In this case, in addition to or instead of the first image and the second image, the projective transformation unit 23 may calculate the center position P3 of the object 50 using another captured image in which the object 50 is displayed. Then, the projective transformation unit 23 calculates the object distance h based on a point-plane distance formula using the distance between the calculated center position P3 of the object 50 and the capture position P1 at the time of acquiring the first image, and the provisional value of the normal vector n.
[0050] As described above, the projective transformation unit 23 calculates the translation vector t, the rotation matrix R, the provisional value (initial value) of the normal vector n, and the object distance h, and calculates the projective transformation matrix based on the above formula. This allows the projective transformation unit 23 to suitably execute projective transformation onto the first image.
[0051] Next, a specific example of the projective transformation performed by the projective transformation unit 23 will be described.
[0052] FIG. 5(A) shows an example of the first image in the example of FIG. 4, and FIG. 5(B) shows an example of the second image in the example of FIG. 4. In the first image shown in FIG. 5(A) and the second image shown in FIG. 5(B), an object 50, which is a speed limit sign, is shown. FIG. 6(A) shows a pixel area within a dashed frame F1 of the first image shown in FIG. 5(A). FIG. 6(B) shows a pixel area equivalent to the area within the dashed frame F1 when the first image is projectively transformed. FIG. 6(C) shows a pixel area within a dashed frame F2 of the second image shown in FIG. 5(B).
[0053] The projective transformation unit 23 applies a projective transformation matrix calculated using a translation vector t indicating the relative photographing position of the second image with respect to the first image, and a rotation matrix R indicating the relative photographing orientation of the second image with respect to the first image, to the first image. In this case, as shown in FIG. 6B, in the projective transformation image obtained by projectively transforming the first image, the size and orientation of the object change so that the object appears similar to the second image. In this case, since the normal vector n used in the projective transformation matrix is a provisional value, a deviation occurs in the appearance of the second image depending on the precision (accuracy) of the provisional value of the normal vector n. After that, the image comparison unit 24 extracts the pixel area indicated by the dashed frame F3 from the projective transformation image as the object area, and calculates the correlation degree by comparing the pixel area in the extracted dashed frame F3 with the pixel area in the dashed frame F2 extracted as the object area from the second image. In this case, the higher the accuracy of the provisional value of the normal vector n (i.e., the closer to the true value), the higher the corresponding correlation degree. Therefore, the vehicle-mounted device 1 can obtain an accurate normal vector n by obtaining a provisional value of the normal vector n with a high correlation degree through optimization.
[0054] (5) Processing Flow Fig. 7 is an example of a flowchart showing the procedure of normal line calculation processing executed by the vehicle-mounted device 1. The vehicle-mounted device 1 executes the processing of the flowchart shown in Fig. 7, for example, when an object for which a normal line is to be calculated is detected from a photographed image generated by the camera 2.
[0055] First, the vehicle-mounted device 1 acquires a first image and a second image including the object (step S11). In this case, the vehicle-mounted device 1 extracts any two captured images including the object for which the normal is to be calculated as the first image and the second image, respectively, from the captured image storage unit 16 that stores the captured images generated by the camera 2. Note that the vehicle-mounted device 1 may acquire a plurality of sets of captured images including the object, calculate a normal vector for each set of the acquired captured images by performing steps S12 to S18 described below, and calculate a representative value such as an average of the calculated normal vectors as the normal of the object to be obtained.
[0056] Next, the vehicle-mounted device 1 calculates a translation vector t and a rotation matrix R corresponding to the relative photographing position and photographing attitude of the first image and the second image (step S12). In this case, the vehicle-mounted device 1 calculates the translation vector t and the rotation matrix R by, for example, analyzing a plurality of images including the first image and the second image.
[0057] Next, the vehicle-mounted device 1 sets a provisional value of the normal vector n of the object (step S13). In this case, in the first processing, the vehicle-mounted device 1 sets the provisional value of n of the normal vector based on any two relative positions of the photographing position of the first image, the photographing position of the second image, or the position of the object. In the second and subsequent processing, the vehicle-mounted device 1 changes the provisional value of n of the normal vector from the previous value based on an optimization technique.
[0058] Next, the vehicle-mounted device 1 calculates the object distance h, which is the distance between the shooting position of the first image and the object (step S14). In this case, for example, the vehicle-mounted device 1 calculates the object distance h based on the provisional value of the normal vector set in step S13 and the center position of the object calculated based on the first image and the second image. Note that in the second and subsequent processing, the vehicle-mounted device 1 calculates the object distance h using the normal vector updated based on the optimization method. This processing is executed by the projective transformation unit 23 in FIG. 3.
[0059] Then, the vehicle-mounted device 1 calculates a homography matrix based on the translation vector t, the rotation matrix R, the provisional value of the normal vector n, and the object distance h calculated in steps S12 to S14 (step S15).The vehicle-mounted device 1 then calculates the correlation between the object between the homography transformed image, which is the first image that has been homography transformed using the calculated homography matrix, and the second image (step S16).
[0060] Then, the vehicle-mounted device 1 judges whether or not the optimization of the normal vector should be terminated (step S17). For example, the vehicle-mounted device 1 judges that the optimization should be terminated when the correlation calculated in step S16 becomes equal to or greater than a predetermined threshold, when the processes of steps S13 to S16 have been repeated a specified number of times, or when the change in the correlation calculated in step S16 is less than a predetermined threshold. Then, when the vehicle-mounted device 1 judges that the optimization should be terminated (step S17; Yes), it determines the normal vector of the object (step S18). In this case, the vehicle-mounted device 1 determines the provisional value of the normal vector with the highest correlation as the normal vector of the object. Thereafter, the vehicle-mounted device 1 stores information on the determined normal vector of the object in the normal information storage unit 18. On the other hand, when the vehicle-mounted device 1 judges that the optimization should not be terminated (step S17; No), it returns the process to step S13. In this case, in step S13, the vehicle-mounted device 1 changes the provisional value of n of the normal vector from the previous value based on the applied optimization method, and executes the processes of steps S14 to S16.
[0061] As described above, the vehicle-mounted device 1 according to the embodiment includes an image acquisition means, a projective transformation means, and a normal calculation means. The image acquisition means acquires a first image including an object having a plane and a second image including the object. The projective transformation means performs projective transformation on the first image based on the relative photographing position and photographing attitude of the second image with respect to the first image, the distance from the photographing position of the first image to the object, and a provisional value of the normal of the object. The normal calculation means calculates the normal based on the correlation of the object between the projectively transformed image obtained by projectively transforming the first image and the second image. This allows the vehicle-mounted device 1 to accurately calculate the normal of the object based on two images including the object.
[0062] (6) Variations Preferred modifications of the above-mentioned embodiment will be described. The following modifications may be applied to the above-mentioned embodiment in any combination.
[0063] (Variation 1) The vehicle-mounted device 1 may perform the normal calculation process by further referring to geometric information relating to the shape of the object.
[0064] Fig. 8 shows a block configuration diagram of an in-vehicle device 1A according to the first modification. The in-vehicle device 1A includes an interface 11, a memory 12, and a controller 15, similar to the block configuration of the in-vehicle device 1 shown in Fig. 2. The memory 12 includes a captured image storage unit 16, a normal information storage unit 18, and a geometric information storage unit 19. The controller 15 of the in-vehicle device 1A functionally includes an image acquisition unit 21, a shooting parameter calculation unit 22, a projective transformation unit 23, an image comparison unit 24, an optimization unit 25, and a geometric information acquisition unit 26.
[0065] The geometric information storage unit 19 stores geometric information of a feature that is the target of normal detection as the target object. This geometric information is geometric information (i.e., information indicating the shape) of the target feature, and exists for each type (category) of features with different shapes, for example. For example, if the target object includes a sign, information indicating the shape of the sign corresponding to each type of sign is stored as geometric information in the geometric information storage unit 19. For example, if a speed limit sign is round, a pedestrian crossing sign is triangular, and an intersection sign is diamond-shaped, this information is stored as geometric information in the geometric information storage unit 19. Note that the geometric information may include size information.
[0066] The geometric information acquisition unit 26 extracts geometric information corresponding to the object detected by the image acquisition unit 21 from the geometric information storage unit 19. For example, the image acquisition unit 21 supplies information on the type of object output when a captured image is input to an inference device obtained by learning or the like to the geometric information acquisition unit 26, and the geometric information acquisition unit 26 extracts geometric information associated with the type of object supplied from the image acquisition unit 21 from the geometric information storage unit 19. Then, the geometric information acquisition unit 26 supplies the extracted geometric information to at least one of the image comparison unit 24 and the optimization unit 25. In this way, the controller 15 functions as a "geometric information acquisition means."
[0067] The use of geometric information by the image comparison unit 24 will be described. The image comparison unit 24 calculates the degree of correlation based on the geometric information supplied from the geometric information acquisition unit 26. For example, the image comparison unit 24 uses the geometric information in the process of extracting object regions from a projection-transformed image obtained by projection-transforming a first image and a second image. For example, in the case of the object 50 shown in FIG. 5, the image comparison unit 24 recognizes that the object 50 is round based on the geometric information, and sets the range to be extracted as the object region (see the dashed frames F1 to F3 in FIG. 5 and FIG. 6) to a round shape. In this case, the image comparison unit 24 extracts a pixel region of a round shape as the object region. This allows the image comparison unit 24 to accurately extract the object region and suitably suppress the occurrence of an error in the degree of correlation caused by using a pixel region that is not the object as a comparison target.
[0068] Next, the use of geometric information by the optimization unit 25 will be described. The optimization unit 25 may determine an update value (a value to be changed) of the provisional value of the normal vector used in the projective transformation by the projective transformation unit 23 based on the geometric information supplied from the geometric information acquisition unit 26. For example, the optimization unit 25 limits the direction in which the provisional value is changed (i.e., the direction in which the solution is searched) to a direction in which the provisional value approaches the shape of the object indicated by the geometric information. In this case, the optimization unit 25 sets a predetermined number of provisional values changed in different directions from the initial value of the normal vector, for example. Then, the optimization unit 25 recognizes the provisional value in which the object of the projective transformation image based on the set provisional values approaches the shape indicated by the geometric information most closely as the direction in which the provisional value is changed (i.e., the direction in which the solution should be searched). In this way, the optimization unit 25 can appropriately limit the search range of the provisional value of the normal vector to be optimized by using the geometric information, thereby reducing the processing load.
[0069] (Variation 2) The normal line calculation process executed by the vehicle-mounted device 1 may be executed by a device other than the vehicle-mounted device 1.
[0070] FIG. 9 is a schematic diagram of a normal calculation system according to the second modification. As shown in FIG. 9, the normal calculation system includes an on-vehicle device 1X, a camera 2, and a map generating device 4. The on-vehicle device 1X has a configuration equivalent to that of the on-vehicle device 1 shown in FIG. 2, and is capable of data communication with the map generating device 4. The on-vehicle device 1X does not include a normal information storage unit 18. The on-vehicle device 1X transmits the captured image provided by the camera 2 to the map generating device 4 as measurement data "Im". In this case, the on-vehicle device 1X may temporarily store the captured image, or may transmit the captured image provided by the camera 2 to the map generating device 4 without storing it.
[0071] The map generating device 4 receives measurement data Im including a captured image from the vehicle-mounted device 1 and stores the received measurement data Im. Then, the map generating device 4 executes the normal calculation process by the vehicle-mounted device 1 described in the embodiment based on the stored measurement data Im. In this case, the map generating device 4 may calculate the normal vector of the object by processing the measurement data Im received from the vehicle-mounted device 1X in real time, or may calculate the normal vector at a predetermined timing based on a user input or the like. The map generating device 4 is an example of an "information processing device".
[0072] 10 is a block diagram showing the functional configuration of the map generating device 4. The map generating device 4 has an interface 41, a memory 42, and a controller 45. These elements are connected to each other via a bus line.
[0073] The interface 41 performs interface operations related to the transmission and reception of data between the map generating device 4 and an external device. The memory 42 is composed of various types of memory such as RAM, ROM, and other non-volatile memories (including hard disk drives, flash memories, etc.). The memory 42 stores programs for the controller 45 to execute predetermined processes. The memory 42 is also used as a working memory for the controller 45. The programs executed by the controller 45 may be stored in a storage medium other than the memory 42.
[0074] Moreover, the memory 42 functionally has a measurement data storage unit 46 and a normal line information storage unit 47. The measurement data storage unit 46 stores the measurement data Im received from the vehicle-mounted device 1. The normal line information storage unit 47 stores normal line information generated by the map generating device 4. At least one of the measurement data storage unit 46 and the normal line information storage unit 47 may be stored in a storage device external to the map generating device 4, such as a hard disk connected to the map generating device 4 via the interface 41. The above storage device may be a server device that communicates with the map generating device 4. The above storage device may be composed of multiple devices.
[0075] The controller 45 includes a processor such as a CPU, a GPU, etc., and controls the entire vehicle-mounted device 1. In this case, the controller 45 performs processing related to change point detection of feature objects by executing programs stored in the memory 42, etc. The controller 45 functions as an "image acquisition means," a "projection transformation means," a "normal calculation means," and a computer that executes programs, etc.
[0076] In this way, the normal calculation process may be performed by any device other than the vehicle-mounted device 1X. Instead of the vehicle-mounted device 1 and the map generating device 4 transferring the measurement data Im through data communication, the vehicle-mounted device 1 may acquire the measurement data Im by reading it from the storage medium. In this case, the storage medium is electrically connected to the vehicle-mounted device 1 during measurement, and the vehicle-mounted device 1 writes the measurement data Im into the storage medium.
[0077] (Variation 3) The target object is not limited to a feature such as a sign on or around a road, but may be any object having a planar shape.
[0078] In this case, the camera 2 may be provided on a moving body other than a vehicle. For example, the camera 2 may be provided on a self-propelled robot. Even in this case, the self-propelled robot or an information processing device that acquires the detection data of the camera 2 from the self-propelled robot executes the normal calculation process described in the embodiment to calculate the normal of the target object.
[0079] (Variation 4) The vehicle-mounted device 1 may recognize the shooting parameters (shooting position and shooting attitude) using the outputs of various sensors other than the camera.
[0080] In this case, the vehicle-mounted device 1 is electrically connected to the sensor unit, for example, by wire or wirelessly. The sensor unit is a group of sensors that detect information regarding the position and attitude (traveling direction) of the vehicle. The sensor unit includes a plurality of sensors, such as a GPS receiver, an acceleration sensor, a gyro sensor, and an IMU (Inertial Measurement Unit). The GPS receiver may generate highly accurate position information indicating the absolute position of the vehicle (e.g., a three-dimensional position of latitude, longitude, and altitude) based on an RTK positioning method (i.e., an interferometric positioning method). The sensor unit may be a sensor provided in the camera 2 so as to directly detect the position and shooting attitude of the camera 2. The sensor unit supplies information (also called "sensor information") that associates the detection result by the sensor with date and time data indicating the detection date and time to the vehicle-mounted device 1. The vehicle-mounted device 1 then calculates the shooting position and shooting attitude for the first image and the second image by referring to the sensor information including date and time data that matches the date and time indicated by the date and time data added to the first image and the second image. Here, when the referenced sensor information indicates the position and traveling direction of the vehicle, the vehicle-mounted device 1 further references camera installation information indicating the relative position and orientation of the camera 2 with respect to the vehicle, thereby identifying the position and shooting attitude of the camera 2 that captured the target image. The camera installation information is stored in advance, for example, in the memory 12. This embodiment also allows the vehicle-mounted device 1 to suitably calculate the shooting parameters.
[0081] In the above-described embodiment, the program can be stored using various types of non-transitory computer readable media and supplied to a controller or the like that is a computer. The non-transitory computer readable medium includes various types of tangible storage media. Examples of the non-transitory computer readable medium include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)).
[0082] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated herein by reference. [Explanation of symbols]
[0083] 1 On-vehicle device 2 Camera 4. Map Generation Device 16 Photographed image storage section 18, 47 Normal information storage section 46 Measurement data storage unit
Claims
1. an image capture means for capturing a first image including an object having a plane and a second image including the object; A relative photographing position and photographing attitude of the second image with respect to the first image; A distance from a photographing position of the first image to the object; and A provisional value of a normal of the object; A projection transformation means for projectively transforming the first image based on the projection transformation means; a normal calculation means for calculating the normal based on a correlation degree of the object between a projectively transformed image obtained by projectively transforming the first image and the second image; having the projective transformation means sets an initial value of the provisional value based on a vector determined from a shooting position of the first image and a shooting position of the second image; Information processing device.
2. an image capture means for capturing a first image including an object having a plane and a second image including the object; A relative photographing position and photographing attitude of the second image with respect to the first image; A distance from a photographing position of the first image to the object; and A provisional value of a normal of the object; A projection transformation means for projectively transforming the first image based on the projection transformation means; a normal calculation means for calculating the normal based on a correlation degree of the object between a projectively transformed image obtained by projectively transforming the first image and the second image; having The projective transformation means sets an initial value of the provisional value based on a vector determined from a photographing position of the first image or the second image and a position of the object. Information processing device.
3. an image capture means for capturing a first image including an object having a plane and a second image including the object; A geometric information acquisition means for acquiring geometric information that is geometric information of the object; A relative photographing position and photographing attitude of the second image with respect to the first image; A distance from a photographing position of the first image to the object; and A provisional value of a normal of the object; A projection transformation means for projectively transforming the first image based on the projection transformation means; a normal calculation means for calculating the normal based on a correlation degree of the object between a projectively transformed image obtained by projectively transforming the first image and the second image; having The normal calculation means determines an update value of the provisional value based on the geometric information. Information processing device.
4. The information processing apparatus according to claim 1 , wherein the normal calculation means calculates, as the normal, the provisional value that gives the highest correlation degree when the provisional value of the normal is changed and the correlation degree is calculated a plurality of times.
5. A geometric information acquisition means for acquiring geometric information that is geometric information of the object, The information processing device according to any one of claims 1 to 4, wherein the normal calculation means extracts a region of the object from the projectively transformed image and the second image based on the geometric information, and calculates the degree of correlation based on the regions extracted from each of the projectively transformed image and the second image.
6. The information processing device according to any one of claims 1 to 5, wherein the projective transformation means calculates the position of the object based on the first image and the second image, and calculates the distance based on the position of the object, the shooting position of the first image, and the provisional value.
7. the first image and the second image are generated by a camera that moves with the vehicle; The information processing device according to claim 1 , wherein the object is a sign.
8. By computer, acquiring a first image including an object having a plane and a second image including the object; A relative photographing position and photographing attitude of the second image with respect to the first image; A distance from a photographing position of the first image to the object; and A provisional value of a normal of the object; projecting the first image based on the Calculating the normal based on a correlation degree of the object between a projectively transformed image obtained by projectively transforming the first image and the second image; An initial value of the provisional value is set based on a vector determined from a photographing position of the first image or the second image and a position of the object. Control methods.
9. an image capture means for capturing a first image including an object having a plane and a second image including the object; A relative photographing position and photographing attitude of the second image with respect to the first image; A distance from a photographing position of the first image to the object; and A provisional value of a normal of the object; A projection transformation means for projectively transforming the first image based on the projection transformation means; a normal calculation means for calculating the normal based on a correlation degree of the object between a projectively transformed image obtained by projectively transforming the first image and the second image; The computer functions as The projective transformation means sets an initial value of the provisional value based on a vector determined from a photographing position of the first image or the second image and a position of the object. program.
10. A storage medium storing the program according to claim 9.
Citation Information
Patent Citations
Apparatus and method for object recognition
JP2004272694A
Image processing device, image processing method and image processing program
JP2017167817A
Object identification system
JP2019095345A
Normal line estimation device, normal line estimation method, and normal line estimation program
JP2018097795A
