Image registration device, image generation system, image registration method, and storage medium

Abstract

The image registration device is communicably connected to the distance measurement sensor (10) and communicably connected to the camera (20). The distance measurement sensor generates a reflected light image including distance information by a light receiving element sensing the reflected light reflected from an object through light irradiation, and generates a background light image in the same coordinate system as the reflected light image by the light receiving element sensing the background light with respect to the reflected light. The camera generates a camera image by a camera element detecting incident light from the outside. The image registration device includes: an image acquisition unit (41) that acquires the reflected light image, the background light image, and the camera image; and an image processing unit (42) that performs image registration of the reflected light image and the camera image in the same coordinate system as the background light image by determining the correspondence between the feature points of the background light image and the feature points of the camera image.

Description

[0001] Cross - reference to related applications

[0002] This application is based on Japanese Patent Application No. 2019 - 164860 filed on September 10, 2019, and the content of the base application is incorporated herein by reference in its entirety. Technical field

[0003] The disclosure of this specification relates to an image registration device, an image generation system, an image registration method, and an image registration program. Background art

[0004] A distance measuring sensor is disclosed in Patent Document 1. The distance measuring sensor can generate a reflected light image including distance information by a light receiving element sensing reflected light reflected from an object by light irradiation. A camera is disclosed in Patent Document 2. The camera can generate a high - resolution camera image by a camera element detecting incident light from the outside.

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2019 - 95452

[0006] Patent Document 2: Japanese Unexamined Patent Application Publication No. 2018 - 69878

[0007] The reflected light image and the camera image can be processed by an application. However, in the reflected light image and the camera image, there is a deviation Δt in the detection timing. If an object imaged in the reflected light image and the camera image moves during the deviation Δt, it is difficult to accurately associate the object imaged in the reflected light image with the object imaged in the camera image for processing. Therefore, even if the application uses both the reflected light image and the camera image, the information of these images cannot be utilized to the maximum extent, and thus the processing accuracy cannot be sufficiently improved. Summary of the invention

[0008] One of the objects of the disclosure of this specification is to provide an image registration device, an image generation system, an image registration method, and an image registration program that improve the processing accuracy of an application.

[0009] One aspect disclosed herein provides an image registration device communicably connected to a distance measuring sensor and communicably connected to a camera. The distance measuring sensor generates a reflected light image including distance information by a light receiving element sensing reflected light reflected from an object by light irradiation, and generates a background light image in the same coordinate system as the reflected light image by the light receiving element sensing background light with respect to the reflected light. The camera generates a camera image with higher resolution than the reflected light image and the background light image by a camera element detecting incident light from the outside. The image registration device includes:

[0010] An image acquisition unit that acquires a reflected light image, a background light image, and a camera image; and

[0011] An image processing unit that performs image registration of the reflected light image and the camera image in the same coordinate system as the background light image by determining the correspondence between the feature points of the background light image and the feature points of the camera image.

[0012] In this way, the acquired reflected light image and the camera image are subjected to image registration using the background light image in the same coordinate system as the reflected light image. That is, in the comparison between the feature points of the background light image, which is closer in nature to the camera image than the reflected light image, and the feature points of the camera image, it becomes easier to determine the correspondence between the feature points. By establishing such a correspondence, the coordinate system of the reflected light image can be made to coincide with the coordinate system of the camera image with high precision. Therefore, the processing accuracy of applications that use both the reflected light image and the camera image can be significantly improved.

[0013] In addition, another disclosed aspect provides an image generation system that generates an image for processing by an application. The image generation system includes:

[0014] A distance measurement sensor that generates a reflected light image including distance information by a light receiving element sensing the reflected light reflected from an object by light irradiation, and generates a background light image in the same coordinate system as the reflected light image by the light receiving element sensing the background light with respect to the reflected light;

[0015] A camera that generates a camera image with a higher resolution than the reflected light image and the background light image by a camera element detecting incident light from the outside; and

[0016] An image processing unit that performs image registration of the reflected light image and the camera image in the same coordinate system as the background light image by determining the correspondence between the feature points of the background light image and the feature points of the camera image, and generates a composite image that combines the distance information and the information of the camera image.

[0017] In this way, the reflected light image and the camera image are subjected to image registration using the background light image in the same coordinate system as the reflected light image. That is, in the comparison between the feature points of the background light image, which is closer in nature to the camera image than the reflected light image, and the feature points of the camera image, it becomes easier to determine the correspondence between the feature points. By establishing such a correspondence, the coordinate system of the reflected light image can be made to coincide with the coordinate system of the camera image with high precision. Moreover, the information from different image generation sources of the distance measurement sensor and the camera, that is, the distance information and the information of the camera image, can be provided in the form of a composite image that is easy for the application to process. Therefore, the processing accuracy of applications that use both the reflected light image and the camera image can be significantly improved.

[0018] In addition, as another disclosed method, an image registration method includes:

[0019] Prepare a reflected light image and a background light image, where the reflected light image and the background light image are images generated by a distance measuring sensor. The reflected light image containing distance information is generated by a light receiving element sensing the reflected light reflected from an object through light irradiation, and the background light image in the same coordinate system as the reflected light image is generated by the light receiving element sensing the background light relative to the reflected light.

[0020] Prepare a camera image, where the camera image is an image generated by a camera. The camera image with a higher resolution than the reflected light image and the background light image is generated by a camera element detecting incident light from the outside.

[0021] Detect feature points of the background light image and the camera image respectively.

[0022] Determine the correspondence between the detected feature points of the background light image and the feature points of the camera image; and

[0023] Based on the determination result of the correspondence, make each pixel of one of the background light image and the camera image correspond to each pixel of the other.

[0024] According to this method, detect feature points of the prepared background light image and the camera image respectively. Then, determine the correspondence between the detected feature points of the background light image and the feature points of the camera image. Then, based on the determination result of the correspondence, make each pixel of one of the background light image and the camera image correspond to each pixel of the other. In this way, image registration between the reflected light image and the camera image is implemented using the background light image in the same coordinate system as the reflected light image and with properties closer to the camera image than the reflected light image. Therefore, it becomes easy to determine the correspondence between feature points. By establishing such a correspondence, the coordinate systems of the reflected light image and the camera image can be made to coincide with high precision. Therefore, the processing accuracy of applications using both the reflected light image and the camera image can be significantly improved. Moreover, after determining the correspondence between feature points, use this result to make the coordinates of each pixel correspond. Therefore, compared with the case of overly determining the correspondence of each pixel, it is possible to suppress the processing amount or increase the processing speed, and high-precision image registration can be implemented.

[0025] In addition, another disclosed method provides an image registration program for implementing image registration between an image generated by a distance measuring sensor and an image generated by a camera, where

[0026] The image registration program causes at least one processing unit to execute the following processing:

[0027] Processing for obtaining a reflected light image and a background light image, wherein the reflected light image and the background light image are images generated by a distance measuring sensor. The reflected light image containing distance information is generated by a light receiving element sensing the reflected light reflected from an object by light irradiation, and the background light image in the same coordinate system as the reflected light image is generated by the light receiving element sensing the background light relative to the reflected light.

[0028] Processing for obtaining a camera image, wherein the camera image is an image generated by a camera. A camera image with a higher resolution than the reflected light image and the background light image is generated by a camera element detecting incident light from the outside.

[0029] Processing for separately detecting feature points of the background light image and feature points of the camera image.

[0030] Processing for determining the correspondence between the detected feature points of the background light image and the feature points of the camera image; and

[0031] Processing for causing each pixel of one of the background light image and the camera image to correspond to each pixel of the other based on the determination result of the correspondence.

[0032] In this way, the feature points of the obtained background light image and the feature points of the camera image are separately detected. Then, the correspondence between the detected feature points of the background light image and the feature points of the camera image is determined. Then, based on the determination result of the correspondence, each pixel of one of the background light image and the camera image is caused to correspond to each pixel of the other. In this way, image registration between the reflected light image and the camera image is performed using the background light image in the same coordinate system as the reflected light image and having properties closer to the camera image than the reflected light image, so that the determination of the correspondence between the feature points becomes easy. By establishing such a correspondence, the coordinate systems of the reflected light image and the camera image can be made to coincide with high precision, so that the processing accuracy of applications using both the reflected light image and the camera image can be significantly improved. Moreover, after the correspondence between the feature points is determined, this result is used to make the coordinates of the pixels correspond, so that compared with the case of overly determining the correspondence of each pixel, the processing amount can be suppressed or the processing speed can be increased, and high-precision image registration can be performed.

[0033] In addition, the reference numerals in parentheses in the claims, etc. illustratively represent the correspondence with parts of the following embodiments and are not intended to limit the technical scope. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a diagram showing an overall view of the image generation system and the driving assistance ECU of the first embodiment.

[0035] Figure 2This is a diagram showing the mounting states of the distance measurement sensor and the external camera of the first embodiment on the vehicle.

[0036] Figure 3 This is a block diagram showing the structure of the image processing ECU of the first embodiment.

[0037] Figure 4A This is a diagram for explaining the detection of feature points in the background light image of the first embodiment.

[0038] Figure 4B This is Figure 4A a diagram in which the background light image included is changed to a line graph.

[0039] Figure 5A This is a diagram for explaining the detection of feature points in the camera image of the first embodiment.

[0040] Figure 5B This is Figure 5A a diagram in which the camera image included is changed to a line graph.

[0041] Figure 6A This is a diagram for explaining the determination of the correspondence relationship of feature points in the first embodiment.

[0042] Figure 6B This is Figure 6A a diagram in which the included camera image and background light image shown are changed to line graphs.

[0043] Figure 7 This is a diagram for explaining the coordinate matching in the first embodiment.

[0044] Figure 8 This is a flowchart for explaining the processing of the image processing ECU in the first embodiment.

[0045] Figure 9 This is for the second embodiment corresponding to Figure 3 the diagram. Specific Embodiments

[0046] Hereinafter, a plurality of embodiments will be described based on the drawings. In addition, by assigning the same reference numerals to the corresponding components in each embodiment, redundant explanations may sometimes be omitted. When only a part of the structure is described in each embodiment, for the other parts of the structure, the structure of other embodiments described previously can be applied. In addition, not only the combinations of the structures explicitly shown in the description of each embodiment, but also the structures of multiple embodiments can be partially combined with each other without explicit indication as long as it does not particularly impede the combination.

[0047] (First Embodiment)

[0048] As Figure 1 shown, the image registration device according to the first embodiment of the present disclosure is used for a vehicle 1 as a moving body, and is configured as an image processing ECU (Electronic Control Unit) 30 mounted on the vehicle 1. The image processing ECU 30, together with the distance measurement sensor 10 and the external camera 20, constitutes an image generation system 100. The image generation system 100 of the present embodiment can generate peripheral monitoring image information integrating the measurement results of the distance measurement sensor 10 and the external camera 20, and provide it to the driving assistance ECU 50 and the like.

[0049] The image processing ECU 30 is communicably connected to the communication bus of the in-vehicle network mounted on the vehicle 1. The image processing ECU 30 is one of a plurality of nodes provided in the vehicle network. In addition to the distance measurement sensor 10 and the external camera 20, the driving assistance ECU 50 and the like are respectively connected as nodes to the communication bus of the in-vehicle network.

[0050] The driving assistance ECU 50 has a structure including a computer having a processor, a RAM (Random Access Memory), a storage unit, an input / output interface, and a bus connecting them as a main body. The driving assistance ECU 50 has at least one of a driving assistance function for assisting the driver's driving operation in the vehicle 1 and a driving substitution function for substituting the driver's driving operation. The driving assistance ECU 50 identifies the surrounding environment of the vehicle 1 based on the peripheral monitoring image information obtained from the image generation system 100 by executing a program stored in the storage unit by the processor. The driving assistance ECU 50 realizes the automatic driving or highly automated driving assistance of the vehicle 1 corresponding to the recognition result by executing a program stored in the storage unit by the processor.

[0051] Next, the details of each of the distance measurement sensor 10, the external camera 20, and the image processing ECU 30 included in the image generation system 100 will be described in sequence.

[0052] The distance measurement sensor 10 is, for example, a SPAD LiDAR (Single Photon Avalanche Diode Light Detection And Ranging) configured in front of the vehicle 1 or on the roof of the vehicle 1. The distance measurement sensor 10 can measure at least the front measurement range MA1 in the periphery of the vehicle 1.

[0053] The distance measurement sensor 10 has a structure including a light emitting unit 11, a light receiving unit 12, a control unit 13, etc. The light emitting unit 11 scans by using a movable optical component (for example, a polygon mirror) to face Figure 2The measurement range MA1 shown irradiates the light beam emitted from the light source. The light source is, for example, a semiconductor laser (Laser diode), and emits a light beam in the near-infrared region that cannot be visually confirmed by the occupant and people outside, according to the electrical signal from the control unit 13.

[0054] The light receiving unit 12 condenses, for example, the reflected light of the irradiated light beam reflected by an object within the measurement range MA1 or the background light with respect to the reflected light through a condenser lens, and makes it incident on the light receiving element 12a.

[0055] The light receiving element 12a is an element that converts light into an electrical signal through photoelectric conversion, and is a SPAD light receiving element that achieves high sensitivity by amplifying the detection voltage. In the light receiving element 12a, for example, in order to detect the reflected light in the near-infrared region, a CMOS sensor with a higher sensitivity in the near-infrared region than in the visible region is used. This sensitivity can also be adjusted by setting an optical filter in the light receiving unit 12. The light receiving element 12a has a plurality of light receiving pixels arranged in an array in a one-dimensional or two-dimensional direction.

[0056] The control unit 13 is a unit that controls the light emitting unit 11 and the light receiving unit 12. The control unit 13 is, for example, arranged on a substrate shared with the light receiving element 12a, and is mainly composed of a general-purpose processor such as a microcomputer or an FPGA (Field-Programmable Gate Array). The control unit 13 implements a scanning control function, a reflected light measurement function, and a background light measurement function.

[0057] The scanning control function is a function that controls the light beam scanning. The control unit 13 oscillates the light beam in a pulsed manner multiple times from the light source at the timing of the operation clock based on the clock oscillator provided in the distance measuring sensor 10, and makes the movable optical component operate.

[0058] The reflected light measurement function is a function that, in accordance with the timing of the light beam scanning, reads, for example, the voltage value based on the reflected light received by each light receiving pixel using a rolling shutter method, and measures the intensity of the reflected light. In the measurement of the reflected light, by detecting the time difference between the light emission timing of the light beam and the light reception timing of the reflected light, the distance from the distance measuring sensor 10 to the object that reflected the reflected light can be measured. Through the measurement of the reflected light, the control unit 13 can generate a reflected light image, which is image-like data associating the intensity of the reflected light and the distance information of the object that reflected the reflected light with the two-dimensional coordinates on the image plane corresponding to the measurement range MA1.

[0059] The background light measurement function is a function that reads the voltage value based on the background light received by each light-receiving pixel at a timing immediately before measuring the reflected light, and measures the intensity of the background light. Here, the background light refers to incident light that substantially does not contain reflected light and is incident on the light-receiving element 12a from the measurement range MA1 in the outside world. The incident light includes natural light, display light incident from the display in the outside world, and the like. By measuring the background light, the control unit 13 can generate a background light image ImL, which is image-like data associating the intensity of the background light with the two-dimensional coordinates on the image plane corresponding to the measurement range MA1.

[0060] The reflected light image and the background light image ImL are sensed by the shared light-receiving element 12a and obtained from the shared optical system including the light-receiving element 12a. Therefore, the coordinate system of the reflected light image and the coordinate system of the background light image ImL can be regarded as the same coordinate system that is mutually consistent. Moreover, it can be said that there is almost no deviation in the measurement timing (for example, less than 1 ns) between the reflected light image and the background light image ImL. Therefore, the reflected light image and the background light image ImL can be regarded as being obtained synchronously as well.

[0061] For example, in the present embodiment, corresponding to each pixel, integrated image data storing 3-channel data of the intensity of the reflected light, the distance of the object, and the intensity of the background light is sequentially output to the image processing ECU30 as a sensor image.

[0062] The outside camera 20 is, for example, a camera disposed inside the vehicle compartment on the front windshield of the vehicle 1. The outside camera 20 can measure at least the front measurement range MA2 in the outside of the vehicle 1, and more specifically, the measurement range MA2 that at least partially overlaps with the measurement range MA1 of the distance measurement sensor 10.

[0063] The outside camera 20 has a structure including a light-receiving unit 22 and a control unit 23. The light-receiving unit 22, for example, condenses incident light (background light) incident from the measurement range MA2 outside the camera through a light-receiving lens and makes it incident on the camera element 22a.

[0064] The camera element 22a is an element that converts light into an electrical signal through photoelectric conversion, and for example, a CCD sensor or a CMOS sensor can be used. In the camera element 22a, in order to efficiently receive natural light in the visible region, the sensitivity in the visible region is set higher than that in the near-infrared region. The camera element 22a has a plurality of light-receiving pixels (equivalent to so-called sub-pixels) arranged in an array in a two-dimensional direction. Color filters of red, green, and blue are arranged, for example, in adjacent light-receiving pixels. Each light-receiving pixel receives visible light of the color corresponding to the color filter arranged. By measuring the intensity of red, the intensity of green, and the intensity of blue respectively, the camera image ImC captured by the external camera 20 is an image with a higher resolution than the reflected light image and the background light image ImL, and can be a color image of the visible region.

[0065] The control unit 23 is a unit that controls the light-receiving unit 22. The control unit 23 is arranged, for example, on a substrate shared with the camera element 22a, and is mainly composed of a general-purpose processor such as a microcomputer or an FPGA. The control unit 23 implements the shooting function.

[0066] The shooting function is a function of shooting the above-mentioned color image. The control unit 23 reads out the voltage value based on the incident light received by each light-receiving pixel, for example, using the global shutter method, at the timing of the operation clock based on the clock oscillator provided in the external camera 20, and senses and measures the intensity of the incident light. This clock oscillator is provided separately and independently from the clock oscillator of the distance measurement sensor 10. The control unit 23 can generate a camera image ImC, which is image-like data associating the intensity of the incident light with the two-dimensional coordinates on the image plane corresponding to the measurement range MA2. Such a camera image ImC is sequentially output to the image processing ECU 30.

[0067] The distance measurement sensor 10 and the external camera 20 operate based on different clock oscillators, and the period of the measurement timing (i.e., the frame rate) is not limited to being the same, and in most cases is different. Therefore, a deviation Δt in the measurement timing occurs between the reflected light image and the background light image ImL and the camera image ImC. Δt can be more than 1000 times the deviation in the measurement timing between the reflected light image and the background light image ImL.

[0068] The image processing ECU 30 is an electronic control device that performs image processing on the reflected light image, the background light image ImL, and the camera image ImC in a composite manner. The image processing ECU 30 has a structure that includes a computer having a processing unit 31, a RAM 32, a storage unit 33, an input / output interface 34, and a bus connecting them as the main body. The processing unit 31 is hardware for arithmetic processing combined with the RAM 32. The processing unit 31 has a structure that includes at least one arithmetic core such as a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), and a RISC (Reduced Instruction Set Computer). The processing unit 31 may also have a structure that further includes an FPGA and an IP core having other dedicated functions. The RAM 32 may also have a structure that includes a video RAM for image generation. The processing unit 31 performs various processes for implementing the functions of the respective functional units described later by accessing the RAM 32. The storage unit 33 has a structure that includes a non-volatile storage medium. Various programs (such as an image registration program) executed by the processing unit 31 are stored in the storage unit 33.

[0069] The image processing ECU 30 has a plurality of functional units for performing image registration by executing the image registration program stored in the storage unit 33 by the processing unit 31. Specifically, as Figure 3 shown, in the image processing ECU 30, functional units such as an image acquisition unit 41 and an image processing unit 42 are constructed.

[0070] The image acquisition unit 41 acquires the reflected light image and the background light image ImL from the distance measurement sensor 10, and acquires the camera image ImC from the external camera 20. The image acquisition unit 41 sequentially supplies the latest set of the reflected light image and the background light image ImL and the latest camera image ImC to the image processing unit 42.

[0071] The image processing unit 42 performs image registration of the reflected light image having the same coordinate system as the background light image ImL and the camera image ImC by determining the correspondence between the feature point FPa of the background light image ImL and the feature point FPb of the camera image ImC. In particular, when the image processing unit 42 of the present embodiment is input with a sensor image storing three-channel data of the intensity of the reflected light, the distance of the object, and the intensity of the background light, and a camera image ImC that is a high-resolution image and a color image of the visible region, it outputs a composite image storing four-channel or more data of the intensity of the reflected light, the distance of the object, the intensity of the background light, and color information. In the present embodiment, the color information is composed of three-channel data of the intensity of red, the intensity of green, and the intensity of blue, so the composite image becomes an image storing six-channel data.

[0072] The image processing unit 42 has a feature point detection function, a correspondence determination function, and a coordinate matching function. In image registration, the feature point detection function implements the processing of the first stage, the correspondence determination function implements the processing of the second stage after the first stage, and the coordinate matching function implements the processing of the third stage after the second stage.

[0073] The feature point detection function is a function of respectively detecting the feature point FPa of the background light image ImL and the feature point FPb of the camera image ImC. The feature points FPa and FPb can be, for example, corners. The detection of the feature points FPa and FPb can adopt various feature point detection methods using a feature point detector. In particular, in the present embodiment, the Harris corner detection method based on Harris corner detectors 43a and 43b is adopted.

[0074] The Harris corner detectors 43a and 43b use the eigenvalues of the structure tensor to detect the feature points FPa and FPb when the weighted sum of squares of the differences in intensity accompanying the movement of pixels within the evaluation object region is represented by the structure tensor through approximation using Taylor expansion. The Harris corner detectors 43a and 43b can determine whether the evaluation object region is a corner (which corresponds to the feature points FPa and FPb), an edge, or flat through the evaluation of the matrix form and the sum of eigenvalues of the structure tensor.

[0075] As Figure 4A and Figure 4B shown, the Harris corner detector 43a detects multiple feature points FPa of the background light image ImL. As Figure 5A and Figure 5B shown, the Harris corner detector 43b detects multiple feature points FPb of the camera image ImC. In Figures 4A to 5BIn [description], the feature points FPa and FPb are schematically represented by the marks of a cross, but in reality, more feature points FPa and FPb are detected.

[0076] The Harris corner detectors 43a and 43b have more than one (two in this embodiment) parameters that affect the scale (in other words, parameters with relatively low invariance to scale). For example, the first parameter is the size of the evaluation target area. The second parameter is the kernel size of the gradient detection filter (e.g., the Sobel gradient detection filter).

[0077] For such parameters that affect the scale, since the resolution of the background light image ImL is different from that of the camera image ImC, the Harris corner detectors 43a and 43b usually detect different numbers of feature points FPa and FPb for the background light image ImL and the camera image ImC. Therefore, even if there is an overlapping range between the measurement range MA1 of the distance measurement sensor 10 and the measurement range MA2 of the external camera 20, it is not limited to detecting the same number of feature points FPa and FPb in this overlapping range.

[0078] In Figure 3 For convenience, one Harris corner detector 43a and 43b are respectively configured corresponding to the background light image ImL and the camera image ImC, but they can also be set commonly (through a common program) for the background light image ImL and the camera image ImC. It can also be that only a more general part of the processing of the Harris corner detectors 43a and 43b is set commonly for the background light image ImL and the camera image ImC.

[0079] The correspondence determination function determines the correspondence between the feature points FPa of the background light image ImL and the feature points FPb of the camera image ImC. In this embodiment, in addition to the different numbers of detected feature points FPa and FPb, the positional relationships of the multiple feature points FPa in the background light image ImL and the positional relationships of the multiple feature points FPb in the camera image ImC may be different, and there may be feature points FPa and FPb without a corresponding relationship, which increases the difficulty of determining the correspondence.

[0080] That is, as Figure 2 shown, the light receiving element 12a of the distance measurement sensor 10 and the camera element 22a of the external camera 20 are arranged at different positions in the vehicle 1, and the arranged orientations are also different. As a result, the positional relationships of the multiple feature points FPa in the background light image ImL and the positional relationships of the multiple feature points FPb in the camera image ImC are different as described above.

[0081] In addition, in the case of the vehicle 1 for high-speed movement as in the present embodiment, due to the deviation Δt of the measurement timing, the position of the object reflected in the background light image ImL may be significantly different from the position of the object reflected in the camera image ImC, and it is even possible that the object is reflected only in one of them. Therefore, it is easy to have a situation where the positional relationship of the plurality of feature points FPa in the background light image ImL is different from the positional relationship of the plurality of feature points FPb in the camera image ImC and there are feature points FPa and FPb that do not have a corresponding relationship.

[0082] In order to cope with the difficulty of determining such a correspondence relationship, first, the image processing unit 42 determines the correspondence relationship using feature amounts obtained from the peripheral regions including the respective feature points FPa and FPb and being feature amounts with relatively high scale invariance. As feature amounts with relatively high scale invariance, for example, information related to the direction of an edge, the average value or ratio of certain physical quantities in the peripheral region, etc. are listed. In the present embodiment, as the feature amount with relatively high scale invariance, information on the extreme value of the degree of smoothing in the case of applying a low-pass filter to the peripheral region is adopted.

[0083] For example, the image processing unit 42 of the present embodiment detects SIFT (Scale-Invariant Feature Transform) feature amounts (hereinafter, SIFT feature amounts) by SIFT feature amount detectors (hereinafter, feature amount detectors) 44a and 44b, and uses the detected SIFT feature amounts to determine the correspondence relationship. The feature amount detectors 44a and 44b apply a Gaussian filter as the above-mentioned low-pass filter to the peripheral regions including the respective feature points FPa and FPb detected by the Harris corner detectors 43a and 43b.

[0084] The feature amount detectors 44a and 44b change the weight coefficient σ corresponding to the standard deviation of the Gaussian filter, and search for local extrema in the peripheral region. The feature amount detectors 44a and 44b use at least a part of the σ in which local extrema are found and are promising (excluding edges) as SIFT feature amounts with relatively high scale invariance.

[0085] In this way, in the comparison of the SIFT feature amounts corresponding to the respective feature points FPa of the background light image ImL and the SIFT feature amounts corresponding to the respective feature points FPb of the camera image ImC, the matching accuracy between the feature points FPa and FPb can be improved.

[0086] In Figure 3In [the above], for convenience, one feature quantity detector 44a and 44b is respectively arranged corresponding to the background light image ImL and the camera image ImC, but it may also be set commonly (by a common program) with respect to the background light image ImL and the camera image ImC. It may also be that only a part with relatively high generality in the processing of the feature quantity detectors 44a and 44b is set commonly with respect to the background light image ImL and the camera image ImC.

[0087] Second, the image processing unit 42 determines the correspondence relationship in consideration of the difference in the position of the corresponding points appearing in the image based on the relative position between the distance measurement sensor 10 and the external camera 20. As Figure 6A and Figure 6B shown, the image processing unit 42 projects the epipolar line EL corresponding to the feature point FPb of the camera image ImC onto the background light image ImL through the epipolar line projector 45 based on the epipolar geometry. The epipolar line EL is the line where the epipolar plane intersects the image plane. The epipolar plane is the plane passing through the optical center of the distance measurement sensor 10, the optical center of the external camera 20, and the three-dimensional point of the object corresponding to the feature point FPb of the camera image ImC.

[0088] Actually, the epipolar line projector 45 stores an E matrix (Essential matrix) defined based on the positions of the light receiving element 12a and the camera element 22a. The E matrix is a matrix for mapping the points on the camera image ImC to the line (i.e., the epipolar line EL) on the background light image ImL.

[0089] Assuming that synchronization is obtained between the background light image ImL and the camera image ImC, the feature point FPb of the background light image ImL corresponding to a certain feature point FPb of the camera image ImC should exist on the epipolar line EL based on the epipolar line projector 45. However, in the present embodiment, there is a deviation Δt in the measurement timing between the background light image ImL and the camera image ImC, and there is a possibility that the object appearing in the background light image ImL and the camera image ImC moves during the deviation Δt.

[0090] Therefore, the image processing unit 42 uses a strip-shaped region with the epipolar line EL as the center line and is a determination region JA with a specified allowable width W to define the feature points FPa with a corresponding relationship. The allowable width W is set according to the offset assumed between the measurement timing of the background light image ImL and the measurement timing of the camera image ImC. Specifically, the image processing unit 42 defines the feature points FPa of the background light image ImL located inside the determination region JA as candidates for the points corresponding to the feature points FPb of the camera image ImC that is the projection source of the epipolar line EL. Moreover, among the defined feature points FPa, the feature point FPa with the most similar SIFT feature amount is determined as the corresponding point. In this way, the image processing unit 42 determines a one-to-one individual correspondence relationship based on each feature point FPb of the camera image ImC and each feature point FPa of the background light image ImL. When the number of feature points FPa and FPb detected by the Harris corner detectors 43a and 43b does not match between the background light image ImL and the camera image ImC, of course, there are feature points FPa and FPb for which no corresponding points are found in the other image, but such feature points FPa and FPb are not used for image registration as a result and are removed from subsequent processing.

[0091] The coordinate matching function is a function that makes each pixel of one of the background light image ImL and the camera image ImC correspond to each pixel of the other based on the result of determining the correspondence relationship of the feature points FPa and FPb. Specifically, as Figure 7 shown, the image processing unit 42 non-linearly and smoothly deforms at least one of the background light image ImL and the camera image ImC based on the positional relationship between each pair of the feature points FPa and FPb with a corresponding relationship, thereby obtaining the correspondence relationship between the coordinate systems of the background light image ImL and the camera image ImC.

[0092] When matching coordinates, the image processing unit 42 implements TPS (Thin Plate Spline) using, for example, the TPS model. The TPS model uses the coordinates of the feature points FPa and FPb in a corresponding relationship as covariants and implements TPS. The TPS model determines the correspondence relationship of each pixel that does not conform to the feature points FPa and FPb between the background light image ImL and the camera image ImC.

[0093] As a specific example for explaining the meaning of determining the correspondence of each pixel, consider the case where after the measurement timing Δt of the camera image ImC, the measurement timing of the background light image ImL follows, and another vehicle in the front moves away from vehicle 1 during the deviation Δt. In this case, in the background light image ImL, the ratio of the intervals between the feature points of the other vehicle reflected is smaller than the ratio of the intervals between the feature points of the other vehicle reflected in the camera image ImC with respect to the ratio of the intervals between the feature points of the scenery reflected. Therefore, by non-linearly deforming the background light image ImL in such a way that the area of the other vehicle reflected in the background light image ImL is enlarged with respect to the area of the scenery reflected, the coordinate system of the background light image ImL can be made to coincide with the coordinate system of the camera image ImC.

[0094] That is, the processing in the coordinate matching function can correct the deviation Δt in the measurement timing and perform the same processing on the background light image ImL and the camera image ImC as the data obtained synchronously with each other. As described above, the background light image ImL can be regarded as having the same coordinate system as the reflected light image and being obtained synchronously. As a result, the image processing unit 42 can perform the same processing on the reflected light image containing distance information and the camera image ImC which is a high-resolution and color image as the data obtained synchronously with each other. In the image registration of such a reflected light image and the camera image ImC, the background light image ImL functions like an adhesive for establishing the correspondence between the two images.

[0095] Moreover, the image processing unit 42 can output the above-described integrated image data, i.e., the composite image, by transforming the coordinates corresponding to each pixel of the background light image ImL into the coordinates on the camera image ImC. Since the composite image has a common coordinate system for each channel, the processing of an application program (hereinafter, referred to as an application) using this composite image can be simplified, the computational load can be reduced, and the processing accuracy of the application can be improved.

[0096] In the present embodiment, the composite image output by the image processing unit 42 is provided to the driving assistance ECU 50 as peripheral monitoring image information. In the driving assistance ECU 50, object recognition using the composite image is implemented by the processor executing an object recognition program which is an application for recognizing the peripheral environment of vehicle 1.

[0097] In the present embodiment, object recognition using semantic segmentation is implemented. In the storage unit of the driving assistance ECU 50, as a constituent element of the object recognition program, an object recognition model 51 mainly composed of a neural network is constructed. This neural network can adopt, for example, a structure called SegNet which combines an encoder and a decoder.

[0098] Next, useFigure 8 The flowchart is used to illustrate the details of the image registration method for performing image registration between the reflected light image and the camera image ImC based on the image registration program. For example, a series of image processing steps based on this flowchart are performed at regular intervals or whenever the distance measurement sensor 10 or the external camera 20 generates a new image.

[0099] First, in S11, the image acquisition unit 41 acquires the latest reflected light image and background light image ImL from the distance measurement sensor 10, and acquires the latest camera image ImC from the external camera 20. The image acquisition unit 41 provides these images to the image processing unit 42. After the processing in S11, the process moves to S12.

[0100] In S12, the image processing unit 42 respectively detects the feature points FPa of the background light image ImL and the feature points FPb of the camera image ImC. After the processing in S12, the process moves to S13.

[0101] In S13, the image processing unit 42 determines the correspondence relationship between the feature points FPa of the background light image ImL detected in S12 and the feature points FPb of the camera image ImC. After the processing in S13, the process moves to S14.

[0102] In S14, the image processing unit 42 establishes a correspondence relationship (coordinate matching) for the coordinates of each pixel that does not conform to the feature points FPa and FPb between the background light image ImL and the camera image ImC according to the positional relationship between the feature points FPa and FPb for which the correspondence relationship was determined in S13. After the processing in S14, the process moves to S15.

[0103] In S15, the image processing unit 42 transforms the coordinate systems of the background light image ImL and the reflected light image into the coordinate system of the camera image ImC, or performs the reverse process, thereby completing the image registration between the reflected light image and the camera image ImC. The series of processes end through S15.

[0104] (Function and effect)

[0105] The function and effect of the first embodiment described above are described again below.

[0106] The image processing ECU 30 according to the first embodiment performs image registration of the acquired reflected light image and the camera image ImC using the background light image ImL in the same coordinate system as the reflected light image. In the comparison between the feature point FPa of the background light image ImL, which is closer in nature to the camera image ImC than the reflected light image, and the feature point FPb of the camera image ImC, it becomes easier to determine the correspondence between the feature points FPa and FPb. By establishing such a correspondence, the coordinate system of the reflected light image can be made to coincide with the coordinate system of the camera image ImC with high precision, and thus the processing accuracy of applications using both the reflected light image and the camera image ImC can be significantly improved.

[0107] In addition, according to the first embodiment, in image registration, the feature point FPa of the background light image ImL and the feature point FPb of the camera image ImC are respectively detected. Then, the correspondence between the detected feature point FPa of the background light image ImL and the feature point FPb of the camera image ImC is determined. Then, based on the determination result of the correspondence, each pixel of one of the background light image ImL and the camera image ImC is made to correspond to each pixel of the other. That is, after determining the correspondence between the feature points FPa and FPb, this result is used to make the coordinates of each pixel correspond. Therefore, compared with the case of overly determining the correspondence of each pixel, it is possible to suppress the processing amount or increase the processing speed, and high-precision image registration can be performed.

[0108] In addition, according to the first embodiment, in the determination of the correspondence, the difference in the position of the corresponding points appearing in the image based on the relative position between the distance measuring sensor 10 and the external camera 20 is considered. By considering this, the determination accuracy of the correspondence between the feature points FPa and FPb can be improved.

[0109] In addition, according to the first embodiment, in the determination of the correspondence, the background light image ImL and the epipolar line EL corresponding to the feature point FPb of the projection source in the camera image ImC are projected onto the image of the projection destination. Moreover, the feature point FPa of the projection destination located within the strip-shaped determination region JA having a predetermined allowable width W along the epipolar line EL is determined as the point corresponding to the feature point FPb of the projection source. By making the determination with the allowable width W, errors such as the projection error between the background light image ImL and the camera image ImC can be absorbed in the determination of the correspondence between the feature points FPa and FPb, and the determination accuracy can be improved.

[0110] In addition, according to the first embodiment, the allowable width W is set based on the assumed offset between the measurement timing of the background light image ImL and the measurement timing of the camera image ImC. Even during the deviation Δt of the measurement timing, if the object constituting the feature points FPa and FPb moves in the background light image ImL and the camera image ImC, as long as the feature point FPa of the object is within the determination region JA having the allowable width W corresponding to the offset, the corresponding feature points FPa and FPb can be determined. Therefore, the accuracy of determining the correspondence can be improved.

[0111] In addition, according to the first embodiment, in determining the correspondence between the feature point FPa of the background light image ImL and the feature point FPb of the camera image ImC, the SIFT feature quantity is used. The SIFT feature quantity is a feature quantity obtained from the peripheral region including each of the feature points FPa and FPb and is a feature quantity with relatively high scale invariance. By using the SIFT feature quantity with relatively high scale invariance, even if there is a difference in the detection level (detection sensitivity) of the feature points FPa and FPb due to the higher resolution of the camera image ImC compared to the background light image ImL, false determination of the correspondence can be suppressed. Therefore, the accuracy of determining the correspondence can be improved.

[0112] In addition, the image generation system 100 according to the first embodiment performs image registration of the reflected light image and the camera image ImC using the background light image ImL having the same coordinate system as the reflected light image. That is, in comparing the feature point FPa of the background light image ImL, which is closer in nature to the camera image ImC than the reflected light image, with the feature point FPb of the camera image ImC, it is easy to determine the correspondence between the feature points FPa and FPb. By establishing such a correspondence, the coordinate system of the reflected light image can be made to coincide with the coordinate system of the camera image ImC with high precision. Moreover, information from different image generation sources of the distance measurement sensor 10 and the external camera 20, that is, distance information and information of the camera image ImC, can be provided in the form of a composite image that is easy to process in applications. Therefore, the processing efficiency of applications using both the reflected light image and the camera image ImC can be significantly improved.

[0113] In addition, according to the image registration method of the first embodiment, the feature points FPa of the prepared background light image ImL and the feature points FPb of the camera image ImC are respectively detected. Then, the correspondence relationship between the detected feature points FPa of the background light image ImL and the feature points FPb of the camera image ImC is determined. Then, based on the determination result of the correspondence relationship, each pixel of one of the background light image ImL and the camera image ImC is made to correspond to each pixel of the other. In this way, the image registration of the reflected light image and the camera image ImC is performed using the background light image ImL that has the same coordinate system as the reflected light image and is closer in nature to the camera image ImC than the reflected light image. Therefore, it becomes easier to determine the correspondence relationship between the feature points FPa and FPb. By establishing such a correspondence relationship, the coordinate system of the reflected light image can be made to coincide with the coordinate system of the camera image ImC with high precision. Therefore, the processing accuracy of applications using both the reflected light image and the camera image ImC can be significantly improved. Moreover, after the correspondence relationship between the feature points FPa and FPb is determined, this result is used to make the coordinates of each pixel correspond. Therefore, compared with the case where the correspondence relationship of each pixel is overly determined, it is possible to suppress the processing amount or increase the processing speed, and high-precision image registration can be performed.

[0114] In addition, according to the image registration program of the first embodiment, the feature points FPa of the acquired background light image ImL and the feature points FPb of the camera image ImC are respectively detected. Then, the correspondence relationship between the detected feature points FPa of the background light image ImL and the feature points FPb of the camera image ImC is determined. Then, based on the determination result of the correspondence relationship, each pixel of one of the background light image ImL and the camera image ImC is made to correspond to each pixel of the other. In this way, the image registration of the reflected light image and the camera image ImC is performed using the background light image ImL that has the same coordinate system as the reflected light image and is closer in nature to the camera image ImC than the reflected light image. Therefore, it becomes even easier to determine the correspondence relationship between the feature points FPa and FPb. By establishing such a correspondence relationship, the coordinate system of the reflected light image can be made to coincide with the coordinate system of the camera image ImC with high precision. Therefore, the processing accuracy of applications using both the reflected light image and the camera image ImC can be significantly improved. Moreover, after the correspondence relationship between the feature points FPa and FPb is determined, this result is used to make the coordinates of each pixel correspond. Therefore, compared with the case where the correspondence relationship of each pixel is overly determined, it is possible to suppress the processing amount or increase the processing speed, and high-precision image registration can be performed.

[0115] (Second Embodiment)

[0116] As Figure 9As shown, the second embodiment is a modification of the first embodiment. The second embodiment will be described centering on the points different from the first embodiment.

[0117] In the second embodiment, the functions of the image processing ECU 30 in the first embodiment and the functions of the driving assistance ECU 50 are integrated into one ECU, forming the driving assistance ECU 230. Therefore, in the second embodiment, the driving assistance ECU 230 is equivalent to the image registration device. In addition, in the driving assistance ECU 230 of the second embodiment, it can be said that the image registration function constitutes a part of the function for realizing a highly accurate surrounding recognition function. Therefore, this driving assistance ECU 230 is also equivalent to the surrounding environment recognition device for recognizing the surrounding environment of the vehicle 1. Similar to the image processing ECU 30 in the first embodiment, the driving assistance ECU 230 has a processing unit 31, a RAM 32, a storage unit 33, an input / output interface 34, etc.

[0118] Similar to the image processing ECU 30 in the first embodiment, the driving assistance ECU 230 in the second embodiment has a plurality of functional units by the processing unit 31 executing the image registration program and the object recognition program stored in the storage unit 33. Specifically, as Figure 9 shown, in the driving assistance ECU 230, functional units such as an image acquisition unit 41, an image processing unit 242, and an object recognition unit 48 are constructed.

[0119] The image acquisition unit 41 is the same as that in the first embodiment. The object recognition unit 48 uses the same object recognition model 48a as in the first embodiment and performs object recognition using semantic segmentation.

[0120] The image processing unit 242 in the second embodiment is the same as that in the first embodiment and has a feature point detection function, a correspondence determination function, and a coordinate matching function. However, it is different from the first embodiment in that the ratio of the resolution of the background light image ImL to the resolution of the camera image ImC (hereinafter, the resolution ratio) is considered in the feature point detection function, and the SIFT feature amount is not used in the correspondence determination function.

[0121] Specifically, a sensor system database (hereinafter referred to as sensor system DB) 243c is provided in the storage unit 33 of the driving assistance ECU 230. Information on various sensors and cameras mounted on the vehicle 1 is stored in the sensor system DB 243c. The information includes information related to the specifications of the light receiving element 12a of the ranging sensor 10 and information related to the specifications of the camera element 22a of the external camera 20. The information related to the specifications of the light receiving element 12a of the ranging sensor 10 includes information on the resolution of the light receiving element 12a, and the information related to the specifications of the camera element 22a of the external camera 20 includes information on the resolution of the camera element 22a. Based on this resolution information, the image processing unit 242 can grasp the resolution ratio.

[0122] The Harris corner detector 243a of the second embodiment makes the scale parameter when detecting the feature point FPa of the background light image ImL different from the scale parameter when detecting the feature point FPb of the camera image ImC based on the resolution ratio. Specifically, in the feature point detection of the camera image ImC with a higher resolution relative to the background light image ImL, at least one of the size of the evaluation target area as the scale parameter and the kernel size of the gradient detection filter is made smaller than in the case of the background light image ImL. In this way, the detection levels of the feature points FPa and FPb can be made closer between the background light image ImL and the camera image ImC.

[0123] As a result, even if SIFT is not used in the correspondence determination function, it becomes easier to determine the correspondence between the detected feature point FPa of the background light image ImL and the feature point FPb of the camera image ImC. In other words, the correspondence can be determined with high accuracy.

[0124] According to the second embodiment described above, the Harris corner detectors 243a and 243b, which are feature point detectors for detecting the feature point FPa of the background light image ImL and the feature point FPb of the camera image ImC, have scale parameters that affect the scale. In such a structure, based on the ratio of the resolution of the background light image ImL to the resolution of the camera image ImC that is grasped, the scale parameter used for detecting the feature point FPa of the background light image ImL is made different from the scale parameter used for detecting the feature point FPb of the camera image ImC. In this way, even if the resolution of the camera image ImC is higher than the resolution of the background light image ImL, the detection levels of the feature points FPa and FPb can be made closer between the background light image ImL and the camera image ImC. Since the feature points FPa and FPb detected at a similar level can be compared with each other, the accuracy of determining the correspondence can be improved.

[0125] (Other embodiments)

[0126] As described above, multiple embodiments have been explained. However, the present disclosure should not be construed as being limited to these embodiments, but can be applied to various embodiments and combinations without departing from the gist of the present disclosure.

[0127] Specifically, as Modification Example 1, the distance measurement sensor 10 and the external camera 20 may also form an integrated sensor unit. Further, an image registration device such as the image processing ECU 30 of the first embodiment may be included as a structural element of the sensor unit.

[0128] As Modification Example 2 related to the first embodiment, the image processing ECU 30 may include an object recognition unit 48 as in the second embodiment to recognize the surrounding environment of the vehicle 1. The analyzed information obtained by the image processing ECU 30 recognizing the surrounding environment of the vehicle 1 may also be provided to a driving assistance ECU 50 having a driving assistance function or the like.

[0129] As Modification Example 3, the image processing unit 42 may not synthesize the reflected light image, the background light image ImL, and the camera image ImC into a multi-channel composite image and output it. The image processing unit 42 may output the reflected light image, the background light image ImL, and the camera image ImC as respective image data, and in addition to these image data, also output coordinate correspondence data indicating the correspondence relationship of the coordinates of each image.

[0130] As Modification Example 4, the image processing unit 42 may only output the reflected light image and the camera image ImC after image registration, and may not output the background light image ImL.

[0131] As Modification Example 5, the camera image ImC may not be a color image but a grayscale image.

[0132] As Modification Example 6, the object recognition using the reflected light image and the camera image ImC after image registration may not be object recognition using semantic segmentation. The object recognition may, for example, also be object recognition using a bounding box.

[0133] As Modification Example 7, the reflected light image and the camera image ImC after image registration may also be used for applications other than object recognition in the vehicle 1. For example, the distance measurement sensor 10 and the camera 20 may be provided in a conference room, and the reflected light image and the camera image ImC after image registration may be used for video conferencing communication applications.

[0134] As Modification Example 8, information on a direction (Orientation) for having rotational invariance may be added to the detected SIFT feature amount. The information on the direction is useful, for example, in a situation where the inclination of the installation surface of the distance measurement sensor 10 is different from the inclination of the installation surface of the camera 20.

[0135] As a modification example 9, when projecting the epipolar line EL corresponding to the feature point FPb of the camera image ImC onto the background light image ImL, or when projecting the epipolar line EL corresponding to the feature point FPa of the background light image ImL onto the camera image ImC, the F matrix (Fundamental matrix) can be used instead of the E matrix. The F matrix is useful in a situation where the distance measurement sensor 10 and the camera 20 are not calibrated.

[0136] As a modification example 10, in addition to the images generated by the distance measurement sensor 10 and the camera 20, the image processing unit 42 can also perform image registration on additional images generated by a millimeter wave radar or the like.

[0137] As a modification example 11, each function provided by the image processing ECU 30 can also be provided by a combination of software and the hardware that executes the software, software only, hardware only, or their composite. Further, when such a function is provided by an electronic circuit as hardware, each function can also be provided by a digital circuit including a plurality of logic circuits or an analog circuit.

[0138] As a modification example 12, the form of the storage medium storing the anomaly detection program and the like capable of implementing the above-described anomaly detection method can also be appropriately changed. For example, the storage medium is not limited to a structure provided on a circuit board, and can also be a structure provided in the form of a memory card or the like, inserted into a socket portion, and electrically connected to the control circuit of the image processing ECU 30. Further, the storage medium can also be an optical disc and a hard disk that are the basis for copying the program of the image processing ECU 30.

[0139] The control unit and method described in the present disclosure can also be implemented by a dedicated computer, which constitutes a processor programmed to execute one or more functions embodied by a computer program. Alternatively, the device and method described in the present disclosure can also be implemented by dedicated hardware logic circuits. Alternatively, the device and method described in the present disclosure can also be implemented by one or more dedicated computers, which are constituted by a combination of a processor that executes a computer program and one or more hardware logic circuits. Further, the computer program can also be stored as instructions executed by a computer in a non-transitory tangible recording medium readable by the computer.

Claims

1. An image registration device is communicably connected to a distance measurement sensor and communicably connected to a camera. The distance measurement sensor generates a reflected light image including distance information by a light receiving element sensing reflected light reflected from an object through light irradiation, and generates a background light image in the same coordinate system as the reflected light image by the light receiving element sensing background light with respect to the reflected light. The camera generates a camera image with a higher resolution than the reflected light image and the background light image by a camera element detecting incident light from the outside, wherein, The image registration device includes: an image acquisition unit that acquires the reflected light image, the background light image, and the camera image; and an image processing unit that performs image registration of the reflected light image and the camera image in the same coordinate system as the background light image by determining the correspondence between the feature points of the background light image and the feature points of the camera image. The image processing unit respectively detects the feature points of the background light image and the feature points of the camera image. The image processing unit determines the correspondence between the detected feature points of the background light image and the feature points of the camera image. Based on the determination result of the correspondence, the image processing unit makes each pixel of one of the background light image and the camera image correspond to each pixel of the other.

2. The image registration device according to claim 1, wherein the image processing unit determines the correspondence in consideration of the difference in the position of the corresponding points appearing in the image based on the relative position between the distance measurement sensor and the camera.

3. The image registration device according to claim 2, wherein the image processing unit projects the epipolar line corresponding to the feature point of the projection source in the background light image and the camera image onto the image of the projection destination, and determines the feature point of the projection destination located within a strip-shaped determination region having a predetermined allowable width along the epipolar line as the point corresponding to the feature point of the projection source.

4. The image registration device according to claim 3, wherein the allowable width is set according to the offset assumed between the measurement timing of the background light image and the measurement timing of the camera image.

5. The image registration device according to any one of claims 1 to 4, wherein in determining the correspondence between the feature points of the background light image and the feature points of the camera image, the image processing unit uses a feature amount obtained from the peripheral region including each of the feature points, and the feature amount is a feature amount with high scale invariance.

6. The image registration device according to any one of claims 1 to 4, wherein the image processing unit uses a feature point detector having a parameter that affects the scale to respectively detect the feature points of the background light image and the feature points of the camera image. The image processing unit grasps the ratio of the resolution of the background light image to the resolution of the camera image, and based on the ratio, makes the parameter used for detecting the feature points of the background light image different from the parameter used for detecting the feature points of the camera image.

7. An image generation system that generates an image for processing by an application, wherein, The image generation system includes: a distance measurement sensor that generates a reflected light image including distance information by a light receiving element sensing reflected light reflected from an object by light irradiation, and generates a background light image in the same coordinate system as the reflected light image by the light receiving element sensing background light with respect to the reflected light; a camera that generates a camera image with a higher resolution than the reflected light image and the background light image by a camera element detecting incident light from the outside; and The image processing unit performs image registration of the reflected light image and the camera image in the same coordinate system as the background light image by determining the correspondence between the feature points of the background light image and the feature points of the camera image, and generates a composite image that combines the distance information and the information of the camera image. The image processing unit respectively detects the feature points of the background light image and the feature points of the camera image. The image processing unit determines the correspondence between the detected feature points of the background light image and the feature points of the camera image. Based on the determination result of the correspondence, the image processing unit makes each pixel of one of the background light image and the camera image correspond to each pixel of the other.

8. An image registration method, comprising: Preparing a reflected light image and a background light image, wherein the reflected light image and the background light image are images generated by a distance measuring sensor, and the reflected light image containing distance information is generated by a light receiving element sensing the reflected light reflected from an object through light irradiation, and the background light image in the same coordinate system as the reflected light image is generated by the light receiving element sensing the background light relative to the reflected light; Preparing a camera image, wherein the camera image is an image generated by a camera, and the camera image with a higher resolution than the reflected light image and the background light image is generated by a camera element detecting incident light from the outside; Respectively detecting the feature points of the background light image and the feature points of the camera image; Determining the correspondence between the detected feature points of the background light image and the feature points of the camera image; and Based on the determination result of the correspondence, making each pixel of one of the background light image and the camera image correspond to each pixel of the other.

9. A storage medium storing an image registration program for performing image registration of an image generated by a distance measuring sensor and an image generated by a camera, wherein The storage medium contains instructions executed by a computer, and the instructions are used to cause at least one processing unit to perform the following processing: Processing for acquiring a reflected light image and a background light image, wherein the reflected light image and the background light image are images generated by the distance measuring sensor, and the reflected light image containing distance information is generated by a light receiving element sensing the reflected light reflected from an object through light irradiation, and the background light image in the same coordinate system as the reflected light image is generated by the light receiving element sensing the background light relative to the reflected light; Processing for acquiring a camera image, wherein the camera image is an image generated by the camera, and the camera image with a higher resolution than the reflected light image and the background light image is generated by a camera element detecting incident light from the outside; Processing for respectively detecting the feature points of the background light image and the feature points of the camera image; Processing for determining the correspondence between the detected feature points of the background light image and the feature points of the camera image; and Processing for causing each pixel of one of the background light image and the camera image to correspond to each pixel of the other based on the determination result of the corresponding relationship.

Images